2024-08-19T12:33:12.759 INFO:root:teuthology version: 1.2.0 2024-08-19T12:33:12.781 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-19T12:33:12.897 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520 branch: main description: rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7859520' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request mon_bind_msgr2: false sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 964adda23da9f7b057cb34fdc0b82aeb7ed65324 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 1858 sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b sleep_before_teardown: 0 subset: 29813/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 964adda23da9f7b057cb34fdc0b82aeb7ed65324 targets: smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL5TalKRuBrQ5q6YCrqfV8a84+uYIIKsxmnp5aeu29gxTSCzm1Gv9o8FNYbLwIsXEa4cVWlB3Hg+6ULekUeBuCA= smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEUnLr0e9XB35qXcUAjFHGzTgLk1gtcI8PNKubiwJvbNn0fsFnyq38OLxh1sF2A50bNBvY+h5i2mAWsf7Cgz0b8= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-18_20:00:17 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-19T12:33:12.897 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa; will attempt to use it 2024-08-19T12:33:12.897 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks 2024-08-19T12:33:12.898 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-19T12:33:12.900 INFO:teuthology.task.internal:Checking packages... 2024-08-19T12:33:12.927 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '797669399b32d2915e64eb6d4b1257ae3c50f80b' 2024-08-19T12:33:12.927 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-19T12:33:12.927 INFO:teuthology.packaging:ref: None 2024-08-19T12:33:12.927 INFO:teuthology.packaging:tag: None 2024-08-19T12:33:12.927 INFO:teuthology.packaging:branch: main 2024-08-19T12:33:12.927 INFO:teuthology.packaging:sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T12:33:12.927 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-08-19T12:33:13.182 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5847.g79766939 2024-08-19T12:33:13.184 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-19T12:33:13.190 INFO:teuthology.task.internal:no buildpackages task found 2024-08-19T12:33:13.191 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-19T12:33:13.210 INFO:teuthology.task.internal:Saving configuration 2024-08-19T12:33:13.229 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-19T12:33:13.236 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-19T12:33:13.268 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-19 12:28:12.110701', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL5TalKRuBrQ5q6YCrqfV8a84+uYIIKsxmnp5aeu29gxTSCzm1Gv9o8FNYbLwIsXEa4cVWlB3Hg+6ULekUeBuCA='} 2024-08-19T12:33:13.301 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-19 12:28:12.108798', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEUnLr0e9XB35qXcUAjFHGzTgLk1gtcI8PNKubiwJvbNn0fsFnyq38OLxh1sF2A50bNBvY+h5i2mAWsf7Cgz0b8='} 2024-08-19T12:33:13.301 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-19T12:33:13.307 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2024-08-19T12:33:13.308 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-08-19T12:33:13.308 INFO:teuthology.run_tasks:Running task console_log... 2024-08-19T12:33:13.392 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f37a71a3ac0>, signals=[15]) 2024-08-19T12:33:13.393 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-19T12:33:13.399 INFO:teuthology.task.internal:Opening connections... 2024-08-19T12:33:13.399 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2024-08-19T12:33:13.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T12:33:13.490 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-08-19T12:33:13.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T12:33:13.575 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-19T12:33:13.582 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-08-19T12:33:13.601 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-08-19T12:33:13.601 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-08-19T12:33:13.659 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-08-19T12:33:13.659 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="9" 2024-08-19T12:33:13.660 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-08-19T12:33:13.660 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-08-19T12:33:13.660 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="9" 2024-08-19T12:33:13.660 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el9" 2024-08-19T12:33:13.660 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-19T12:33:13.660 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-08-19T12:33:13.660 INFO:teuthology.orchestra.run.smithi026.stdout:LOGO="fedora-logo-icon" 2024-08-19T12:33:13.660 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-19T12:33:13.660 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-08-19T12:33:13.660 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-19T12:33:13.660 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-19T12:33:13.660 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-19T12:33:13.661 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-08-19T12:33:13.691 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-08-19T12:33:13.709 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-08-19T12:33:13.710 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-08-19T12:33:13.767 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-08-19T12:33:13.767 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="9" 2024-08-19T12:33:13.767 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-08-19T12:33:13.767 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-08-19T12:33:13.767 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="9" 2024-08-19T12:33:13.767 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el9" 2024-08-19T12:33:13.768 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-19T12:33:13.768 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-08-19T12:33:13.768 INFO:teuthology.orchestra.run.smithi104.stdout:LOGO="fedora-logo-icon" 2024-08-19T12:33:13.768 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-19T12:33:13.768 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-08-19T12:33:13.768 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-19T12:33:13.768 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-19T12:33:13.768 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-19T12:33:13.768 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-08-19T12:33:13.799 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-19T12:33:13.809 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-19T12:33:13.816 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-19T12:33:13.817 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2024-08-19T12:33:13.819 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-08-19T12:33:13.836 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-19T12:33:13.842 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-19T12:33:13.842 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2024-08-19T12:33:13.877 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-08-19T12:33:13.908 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-19T12:33:13.980 INFO:teuthology.run_tasks:Running task kernel... 2024-08-19T12:33:13.995 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-19T12:33:13.996 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.f': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-19T12:33:13.996 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-19T12:33:13.997 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-19T12:33:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T12:33:14.014 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-08-19T12:33:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T12:33:14.015 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-08-19T12:33:14.072 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-480.el9.x86_64 2024-08-19T12:33:14.073 INFO:teuthology.task.kernel:Running kernel on smithi026: 5.14.0-480.el9.x86_64 2024-08-19T12:33:14.073 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum install -y kernel 2024-08-19T12:33:14.075 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-480.el9.x86_64 2024-08-19T12:33:14.075 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.14.0-480.el9.x86_64 2024-08-19T12:33:14.075 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum install -y kernel 2024-08-19T12:33:17.286 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.2 MB 00:01 2024-08-19T12:33:17.723 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-08-19T12:33:19.070 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-08-19T12:33:20.866 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 20 MB 00:02 2024-08-19T12:33:25.207 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.6 MB 00:00 2024-08-19T12:33:25.385 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.6 MB 00:02 2024-08-19T12:33:26.919 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 18 kB 00:00 2024-08-19T12:33:26.949 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 18 kB 00:00 2024-08-19T12:33:28.098 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 21 MB/s | 22 MB 00:01 2024-08-19T12:33:28.203 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 20 MB/s | 22 MB 00:01 2024-08-19T12:33:33.517 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-08-19T12:33:33.720 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-08-19T12:33:35.407 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-19T12:33:35.407 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-480.el9.x86_64 is already installed. 2024-08-19T12:33:35.488 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:33:35.489 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-08-19T12:33:35.489 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:33:35.601 DEBUG:teuthology.orchestra.run.smithi104:> echo no | sudo yum reinstall kernel || true 2024-08-19T12:33:35.852 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-19T12:33:35.852 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-480.el9.x86_64 is already installed. 2024-08-19T12:33:35.932 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:33:35.933 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-08-19T12:33:35.934 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:33:36.019 DEBUG:teuthology.orchestra.run.smithi026:> echo no | sudo yum reinstall kernel || true 2024-08-19T12:33:36.162 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:03 ago on Mon 19 Aug 2024 12:33:33 PM UTC. 2024-08-19T12:33:36.311 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:33:36.322 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:33:36.322 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-08-19T12:33:36.322 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:33:36.322 INFO:teuthology.orchestra.run.smithi104.stdout:Reinstalling: 2024-08-19T12:33:36.322 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T12:33:36.322 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:33:36.322 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:33:36.322 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:33:36.322 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:33:36.322 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 1.5 M 2024-08-19T12:33:36.323 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 0 2024-08-19T12:33:36.323 INFO:teuthology.orchestra.run.smithi104.stderr:Operation aborted. 2024-08-19T12:33:36.380 INFO:teuthology.orchestra.run.smithi104.stdout:Is this ok [y/N]: 2024-08-19T12:33:36.381 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum reinstall -y kernel || true 2024-08-19T12:33:36.554 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:03 ago on Mon 19 Aug 2024 12:33:33 PM UTC. 2024-08-19T12:33:36.709 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:33:36.713 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:33:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-08-19T12:33:36.713 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:33:36.713 INFO:teuthology.orchestra.run.smithi026.stdout:Reinstalling: 2024-08-19T12:33:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T12:33:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:33:36.713 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:33:36.713 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:33:36.713 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:33:36.714 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 1.5 M 2024-08-19T12:33:36.714 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 0 2024-08-19T12:33:36.714 INFO:teuthology.orchestra.run.smithi026.stderr:Operation aborted. 2024-08-19T12:33:36.773 INFO:teuthology.orchestra.run.smithi026.stdout:Is this ok [y/N]: 2024-08-19T12:33:36.774 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum reinstall -y kernel || true 2024-08-19T12:33:36.901 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:03 ago on Mon 19 Aug 2024 12:33:33 PM UTC. 2024-08-19T12:33:37.024 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:33:37.025 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:33:37.025 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-08-19T12:33:37.025 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:33:37.025 INFO:teuthology.orchestra.run.smithi104.stdout:Reinstalling: 2024-08-19T12:33:37.025 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T12:33:37.025 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:33:37.025 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:33:37.025 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:33:37.025 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:33:37.026 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 1.5 M 2024-08-19T12:33:37.026 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 0 2024-08-19T12:33:37.026 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:33:37.277 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-480.el9.x86_64.rpm 7.4 MB/s | 1.5 MB 00:00 2024-08-19T12:33:37.277 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:33:37.278 INFO:teuthology.orchestra.run.smithi104.stdout:Total 6.0 MB/s | 1.5 MB 00:00 2024-08-19T12:33:37.306 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:33:37.307 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:04 ago on Mon 19 Aug 2024 12:33:33 PM UTC. 2024-08-19T12:33:37.435 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:33:37.436 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:33:37.436 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-08-19T12:33:37.436 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:33:37.436 INFO:teuthology.orchestra.run.smithi026.stdout:Reinstalling: 2024-08-19T12:33:37.436 INFO:teuthology.orchestra.run.smithi026.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-19T12:33:37.436 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:33:37.436 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:33:37.437 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:33:37.437 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:33:37.437 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 1.5 M 2024-08-19T12:33:37.437 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 0 2024-08-19T12:33:37.437 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:33:37.555 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:33:37.555 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:33:37.580 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:33:37.581 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:33:37.680 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-480.el9.x86_64.rpm 7.6 MB/s | 1.5 MB 00:00 2024-08-19T12:33:37.681 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:33:37.681 INFO:teuthology.orchestra.run.smithi026.stdout:Total 6.2 MB/s | 1.5 MB 00:00 2024-08-19T12:33:37.711 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:33:37.718 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:33:37.812 INFO:teuthology.orchestra.run.smithi104.stdout: Reinstalling : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T12:33:37.813 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T12:33:37.930 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:33:37.930 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:33:37.956 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:33:37.956 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:33:38.126 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:33:38.340 INFO:teuthology.orchestra.run.smithi026.stdout: Reinstalling : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T12:33:38.341 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T12:33:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T12:33:38.576 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T12:33:39.030 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T12:33:39.030 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/2 2024-08-19T12:33:39.313 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T12:33:39.313 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:33:39.313 INFO:teuthology.orchestra.run.smithi104.stdout:Reinstalled: 2024-08-19T12:33:39.313 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-19T12:33:39.313 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:33:39.313 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:33:39.498 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-08-19T12:33:39.555 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-19T12:33:39.556 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-19T12:33:39.556 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-19T12:33:39.556 INFO:teuthology.task.kernel:Enabling kdb on mon.f... 2024-08-19T12:33:39.557 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-19T12:33:39.592 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-08-19T12:33:39.614 DEBUG:teuthology.parallel:result is None 2024-08-19T12:33:39.729 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 2/2 2024-08-19T12:33:39.729 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:33:39.729 INFO:teuthology.orchestra.run.smithi026.stdout:Reinstalled: 2024-08-19T12:33:39.729 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-19T12:33:39.730 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:33:39.730 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:33:39.893 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-08-19T12:33:39.951 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-19T12:33:39.951 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-19T12:33:39.951 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-19T12:33:39.951 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-19T12:33:39.952 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-19T12:33:39.991 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2024-08-19T12:33:40.009 DEBUG:teuthology.parallel:result is None 2024-08-19T12:33:40.010 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-19T12:33:40.019 INFO:teuthology.task.internal:Creating test directory... 2024-08-19T12:33:40.020 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-19T12:33:40.052 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-19T12:33:40.069 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-19T12:33:40.075 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-19T12:33:40.082 INFO:teuthology.task.internal:Creating archive directory... 2024-08-19T12:33:40.082 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-19T12:33:40.107 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-19T12:33:40.215 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-19T12:33:40.222 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-19T12:33:40.222 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-19T12:33:40.225 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-19T12:33:40.253 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T12:33:40.262 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T12:33:40.273 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T12:33:40.283 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-19T12:33:40.285 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-19T12:33:40.291 INFO:teuthology.task.internal:Configuring sudo... 2024-08-19T12:33:40.291 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-19T12:33:40.306 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-19T12:33:40.331 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-19T12:33:40.340 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-19T12:33:40.341 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-19T12:33:40.372 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-19T12:33:40.387 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T12:33:40.474 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T12:33:40.517 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T12:33:40.606 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T12:33:40.638 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:33:40.638 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-19T12:33:40.702 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T12:33:40.752 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-19T12:33:40.793 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T12:33:40.887 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-19T12:33:40.918 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:33:40.919 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-19T12:33:40.983 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2024-08-19T12:33:40.986 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-08-19T12:33:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T12:33:41.063 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T12:33:41.491 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-19T12:33:41.499 INFO:teuthology.task.internal:Starting timer... 2024-08-19T12:33:41.499 INFO:teuthology.run_tasks:Running task pcp... 2024-08-19T12:33:41.517 INFO:teuthology.run_tasks:Running task selinux... 2024-08-19T12:33:41.526 DEBUG:teuthology.orchestra.run.smithi026:> sudo service auditd rotate 2024-08-19T12:33:41.604 INFO:teuthology.orchestra.run.smithi026.stdout:Rotating logs: 2024-08-19T12:33:41.605 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-08-19T12:33:41.663 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-08-19T12:33:41.665 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-19T12:33:41.665 DEBUG:teuthology.orchestra.run.smithi026:> /usr/sbin/getenforce 2024-08-19T12:33:41.689 INFO:teuthology.orchestra.run.smithi026.stdout:Permissive 2024-08-19T12:33:41.689 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-08-19T12:33:41.710 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-08-19T12:33:41.710 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi026.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi104.front.sepia.ceph.com': 'permissive'} 2024-08-19T12:33:41.711 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-19T12:33:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T12:33:41.756 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-19T12:33:41.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T12:33:41.782 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-19T12:33:41.782 DEBUG:teuthology.orchestra.run.smithi026:> sudo /usr/sbin/setenforce permissive 2024-08-19T12:33:41.824 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-08-19T12:33:41.849 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-19T12:33:41.859 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-19T12:33:41.870 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-19T12:33:41.871 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi104.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-19T12:35:26.071 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com')] 2024-08-19T12:35:26.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-08-19T12:35:26.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T12:35:26.163 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-08-19T12:35:26.247 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-08-19T12:35:26.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-08-19T12:35:26.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-19T12:35:26.333 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-08-19T12:35:26.422 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-08-19T12:35:26.422 INFO:teuthology.run_tasks:Running task clock... 2024-08-19T12:35:26.433 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-19T12:35:26.433 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-19T12:35:26.433 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T12:35:26.436 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-19T12:35:26.436 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T12:35:26.466 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-19T12:35:26.482 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-19T12:35:26.493 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-19T12:35:26.507 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: ntpd: command not found 2024-08-19T12:35:26.508 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-19T12:35:26.519 INFO:teuthology.orchestra.run.smithi026.stdout:506 Cannot talk to daemon 2024-08-19T12:35:26.532 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-08-19T12:35:26.533 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-19T12:35:26.543 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-08-19T12:35:26.548 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-19T12:35:26.557 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-19T12:35:26.570 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-19T12:35:26.611 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-08-19T12:35:26.615 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T12:35:26.615 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-08-19T12:35:26.615 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T12:35:26.615 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T12:35:26.615 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T12:35:26.615 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T12:35:26.628 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-08-19T12:35:26.631 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T12:35:26.631 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-08-19T12:35:26.632 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T12:35:26.632 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T12:35:26.632 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T12:35:26.632 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T12:35:26.633 INFO:teuthology.run_tasks:Running task install... 2024-08-19T12:35:26.641 DEBUG:teuthology.task.install:project ceph 2024-08-19T12:35:26.641 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b'}} 2024-08-19T12:35:26.641 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b'} 2024-08-19T12:35:26.641 INFO:teuthology.task.install:Using flavor: default 2024-08-19T12:35:26.647 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-19T12:35:26.647 INFO:teuthology.task.install:extra packages: [] 2024-08-19T12:35:26.647 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b', 'tag': None, 'wait_for_package': False} 2024-08-19T12:35:26.648 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T12:35:26.650 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b', 'tag': None, 'wait_for_package': False} 2024-08-19T12:35:26.650 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T12:35:26.807 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/ 2024-08-19T12:35:26.807 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5847.g79766939 2024-08-19T12:35:26.819 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/ 2024-08-19T12:35:26.820 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5847.g79766939 2024-08-19T12:35:26.924 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-19T12:35:26.924 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:35:26.925 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-19T12:35:26.938 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-19T12:35:26.938 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:35:26.939 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-19T12:35:26.951 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-19T12:35:26.951 DEBUG:teuthology.orchestra.run.smithi026:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/797669399b32d2915e64eb6d4b1257ae3c50f80b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-19T12:35:26.964 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-19T12:35:26.965 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/797669399b32d2915e64eb6d4b1257ae3c50f80b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-19T12:35:27.019 DEBUG:teuthology.orchestra.run.smithi026:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-19T12:35:27.030 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-19T12:35:27.095 DEBUG:teuthology.orchestra.run.smithi026:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-19T12:35:27.105 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-19T12:35:27.159 INFO:teuthology.orchestra.run.smithi026.stdout:check_obsoletes = 1 2024-08-19T12:35:27.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum clean all 2024-08-19T12:35:27.169 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2024-08-19T12:35:27.171 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2024-08-19T12:35:27.391 INFO:teuthology.orchestra.run.smithi026.stdout:45 files removed 2024-08-19T12:35:27.396 INFO:teuthology.orchestra.run.smithi104.stdout:45 files removed 2024-08-19T12:35:27.418 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-radosgw 2024-08-19T12:35:27.423 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw 2024-08-19T12:35:27.966 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 312 kB/s | 84 kB 00:00 2024-08-19T12:35:27.966 INFO:teuthology.orchestra.run.smithi026.stdout:ceph packages for x86_64 312 kB/s | 84 kB 00:00 2024-08-19T12:35:28.258 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 96 kB/s | 24 kB 00:00 2024-08-19T12:35:28.258 INFO:teuthology.orchestra.run.smithi026.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-08-19T12:35:28.534 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-08-19T12:35:28.534 INFO:teuthology.orchestra.run.smithi026.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-08-19T12:35:29.226 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-08-19T12:35:29.258 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-08-19T12:35:31.110 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-08-19T12:35:33.892 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 20 MB 00:03 2024-08-19T12:35:36.528 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.6 MB 00:01 2024-08-19T12:35:38.203 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 18 kB 00:00 2024-08-19T12:35:38.293 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.6 MB 00:00 2024-08-19T12:35:39.724 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 16 MB/s | 22 MB 00:01 2024-08-19T12:35:39.952 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 18 kB 00:00 2024-08-19T12:35:41.039 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 23 MB/s | 22 MB 00:00 2024-08-19T12:35:45.372 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-08-19T12:35:46.395 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-08-19T12:35:47.692 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:35:47.695 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:35:47.695 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-08-19T12:35:47.695 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:35:47.695 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:35:47.695 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw x86_64 2:19.0.0-5847.g79766939.el9 ceph 11 M 2024-08-19T12:35:47.695 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrading: 2024-08-19T12:35:47.695 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 x86_64 2:19.0.0-5847.g79766939.el9 ceph 3.5 M 2024-08-19T12:35:47.695 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1 x86_64 2:19.0.0-5847.g79766939.el9 ceph 3.2 M 2024-08-19T12:35:47.695 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-08-19T12:35:47.696 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-19T12:35:47.696 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base x86_64 2:19.0.0-5847.g79766939.el9 ceph 5.5 M 2024-08-19T12:35:47.696 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common x86_64 2:19.0.0-5847.g79766939.el9 ceph 22 M 2024-08-19T12:35:47.696 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux x86_64 2:19.0.0-5847.g79766939.el9 ceph 25 k 2024-08-19T12:35:47.696 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-19T12:35:47.696 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-19T12:35:47.696 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-19T12:35:47.696 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2 x86_64 2:19.0.0-5847.g79766939.el9 ceph 755 k 2024-08-19T12:35:47.696 INFO:teuthology.orchestra.run.smithi026.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-19T12:35:47.696 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-19T12:35:47.696 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-19T12:35:47.696 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1 x86_64 2:19.0.0-5847.g79766939.el9 ceph 508 k 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2 x86_64 2:19.0.0-5847.g79766939.el9 ceph 5.5 M 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse x86_64 2:19.0.0-5847.g79766939.el9 ceph 45 k 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common x86_64 2:19.0.0-5847.g79766939.el9 ceph 138 k 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs x86_64 2:19.0.0-5847.g79766939.el9 ceph 188 k 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados x86_64 2:19.0.0-5847.g79766939.el9 ceph 371 k 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd x86_64 2:19.0.0-5847.g79766939.el9 ceph 356 k 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw x86_64 2:19.0.0-5847.g79766939.el9 ceph 116 k 2024-08-19T12:35:47.697 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-19T12:35:47.698 INFO:teuthology.orchestra.run.smithi026.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-19T12:35:47.698 INFO:teuthology.orchestra.run.smithi026.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-19T12:35:47.698 INFO:teuthology.orchestra.run.smithi026.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-19T12:35:47.698 INFO:teuthology.orchestra.run.smithi026.stdout:Installing weak dependencies: 2024-08-19T12:35:47.698 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-19T12:35:47.698 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:35:47.698 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:35:47.698 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:35:47.698 INFO:teuthology.orchestra.run.smithi026.stdout:Install 30 Packages 2024-08-19T12:35:47.698 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrade 2 Packages 2024-08-19T12:35:47.698 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:35:47.699 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 62 M 2024-08-19T12:35:47.699 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:35:48.604 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:35:48.606 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:35:48.606 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-08-19T12:35:48.606 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:35:48.606 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:35:48.606 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:19.0.0-5847.g79766939.el9 ceph 11 M 2024-08-19T12:35:48.606 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2024-08-19T12:35:48.606 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:19.0.0-5847.g79766939.el9 ceph 3.5 M 2024-08-19T12:35:48.606 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:19.0.0-5847.g79766939.el9 ceph 3.2 M 2024-08-19T12:35:48.607 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-08-19T12:35:48.607 INFO:teuthology.orchestra.run.smithi104.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-19T12:35:48.607 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:19.0.0-5847.g79766939.el9 ceph 5.5 M 2024-08-19T12:35:48.607 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:19.0.0-5847.g79766939.el9 ceph 22 M 2024-08-19T12:35:48.607 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:19.0.0-5847.g79766939.el9 ceph 25 k 2024-08-19T12:35:48.607 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-19T12:35:48.607 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-19T12:35:48.607 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-19T12:35:48.607 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:19.0.0-5847.g79766939.el9 ceph 755 k 2024-08-19T12:35:48.607 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-19T12:35:48.607 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-19T12:35:48.607 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:19.0.0-5847.g79766939.el9 ceph 508 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:19.0.0-5847.g79766939.el9 ceph 5.5 M 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:19.0.0-5847.g79766939.el9 ceph 45 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:19.0.0-5847.g79766939.el9 ceph 138 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:19.0.0-5847.g79766939.el9 ceph 188 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:19.0.0-5847.g79766939.el9 ceph 371 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:19.0.0-5847.g79766939.el9 ceph 356 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:19.0.0-5847.g79766939.el9 ceph 116 k 2024-08-19T12:35:48.608 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-19T12:35:48.609 INFO:teuthology.orchestra.run.smithi104.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-19T12:35:48.609 INFO:teuthology.orchestra.run.smithi104.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-19T12:35:48.609 INFO:teuthology.orchestra.run.smithi104.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-19T12:35:48.609 INFO:teuthology.orchestra.run.smithi104.stdout:Installing weak dependencies: 2024-08-19T12:35:48.609 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-19T12:35:48.609 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:35:48.609 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:35:48.609 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:35:48.609 INFO:teuthology.orchestra.run.smithi104.stdout:Install 30 Packages 2024-08-19T12:35:48.609 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 2 Packages 2024-08-19T12:35:48.609 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:35:48.610 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 62 M 2024-08-19T12:35:48.610 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:35:48.623 INFO:teuthology.orchestra.run.smithi026.stdout:(1/32): ceph-base-19.0.0-5847.g79766939.el9.x86 7.9 MB/s | 5.5 MB 00:00 2024-08-19T12:35:48.748 INFO:teuthology.orchestra.run.smithi026.stdout:(2/32): ceph-radosgw-19.0.0-5847.g79766939.el9. 13 MB/s | 11 MB 00:00 2024-08-19T12:35:48.773 INFO:teuthology.orchestra.run.smithi026.stdout:(3/32): ceph-selinux-19.0.0-5847.g79766939.el9. 168 kB/s | 25 kB 00:00 2024-08-19T12:35:48.807 INFO:teuthology.orchestra.run.smithi026.stdout:(4/32): libcephfs2-19.0.0-5847.g79766939.el9.x8 13 MB/s | 755 kB 00:00 2024-08-19T12:35:48.840 INFO:teuthology.orchestra.run.smithi026.stdout:(5/32): libradosstriper1-19.0.0-5847.g79766939. 7.4 MB/s | 508 kB 00:00 2024-08-19T12:35:48.899 INFO:teuthology.orchestra.run.smithi026.stdout:(6/32): python3-ceph-argparse-19.0.0-5847.g7976 775 kB/s | 45 kB 00:00 2024-08-19T12:35:49.141 INFO:teuthology.orchestra.run.smithi026.stdout:(7/32): ceph-common-19.0.0-5847.g79766939.el9.x 18 MB/s | 22 MB 00:01 2024-08-19T12:35:49.183 INFO:teuthology.orchestra.run.smithi026.stdout:(8/32): python3-ceph-common-19.0.0-5847.g797669 488 kB/s | 138 kB 00:00 2024-08-19T12:35:49.275 INFO:teuthology.orchestra.run.smithi026.stdout:(9/32): librgw2-19.0.0-5847.g79766939.el9.x86_6 12 MB/s | 5.5 MB 00:00 2024-08-19T12:35:49.300 INFO:teuthology.orchestra.run.smithi026.stdout:(10/32): python3-cephfs-19.0.0-5847.g79766939.e 1.2 MB/s | 188 kB 00:00 2024-08-19T12:35:49.350 INFO:teuthology.orchestra.run.smithi026.stdout:(11/32): python3-rados-19.0.0-5847.g79766939.el 2.2 MB/s | 371 kB 00:00 2024-08-19T12:35:49.362 INFO:teuthology.orchestra.run.smithi104.stdout:(1/32): ceph-base-19.0.0-5847.g79766939.el9.x86 11 MB/s | 5.5 MB 00:00 2024-08-19T12:35:49.392 INFO:teuthology.orchestra.run.smithi026.stdout:(12/32): python3-rgw-19.0.0-5847.g79766939.el9. 1.2 MB/s | 116 kB 00:00 2024-08-19T12:35:49.417 INFO:teuthology.orchestra.run.smithi026.stdout:(13/32): python3-rbd-19.0.0-5847.g79766939.el9. 2.4 MB/s | 356 kB 00:00 2024-08-19T12:35:49.421 INFO:teuthology.orchestra.run.smithi104.stdout:(2/32): ceph-selinux-19.0.0-5847.g79766939.el9. 433 kB/s | 25 kB 00:00 2024-08-19T12:35:49.521 INFO:teuthology.orchestra.run.smithi104.stdout:(3/32): libcephfs2-19.0.0-5847.g79766939.el9.x8 7.4 MB/s | 755 kB 00:00 2024-08-19T12:35:49.605 INFO:teuthology.orchestra.run.smithi104.stdout:(4/32): libradosstriper1-19.0.0-5847.g79766939. 6.0 MB/s | 508 kB 00:00 2024-08-19T12:35:49.759 INFO:teuthology.orchestra.run.smithi026.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 132 kB/s | 45 kB 00:00 2024-08-19T12:35:49.784 INFO:teuthology.orchestra.run.smithi026.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 246 kB/s | 107 kB 00:00 2024-08-19T12:35:49.810 INFO:teuthology.orchestra.run.smithi026.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 384 kB/s | 160 kB 00:00 2024-08-19T12:35:49.918 INFO:teuthology.orchestra.run.smithi026.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 391 kB/s | 42 kB 00:00 2024-08-19T12:35:49.938 INFO:teuthology.orchestra.run.smithi104.stdout:(5/32): ceph-radosgw-19.0.0-5847.g79766939.el9. 10 MB/s | 11 MB 00:01 2024-08-19T12:35:50.022 INFO:teuthology.orchestra.run.smithi104.stdout:(6/32): librgw2-19.0.0-5847.g79766939.el9.x86_6 13 MB/s | 5.5 MB 00:00 2024-08-19T12:35:50.047 INFO:teuthology.orchestra.run.smithi104.stdout:(7/32): python3-ceph-argparse-19.0.0-5847.g7976 416 kB/s | 45 kB 00:00 2024-08-19T12:35:50.052 INFO:teuthology.orchestra.run.smithi026.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 292 kB 00:00 2024-08-19T12:35:50.064 INFO:teuthology.orchestra.run.smithi104.stdout:(8/32): python3-ceph-common-19.0.0-5847.g797669 3.2 MB/s | 138 kB 00:00 2024-08-19T12:35:50.102 INFO:teuthology.orchestra.run.smithi026.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.2 MB/s | 221 kB 00:00 2024-08-19T12:35:50.152 INFO:teuthology.orchestra.run.smithi026.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 358 kB/s | 36 kB 00:00 2024-08-19T12:35:50.236 INFO:teuthology.orchestra.run.smithi026.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.6 MB/s | 309 kB 00:00 2024-08-19T12:35:50.262 INFO:teuthology.orchestra.run.smithi026.stdout:(22/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.3 MB/s | 662 kB 00:00 2024-08-19T12:35:50.264 INFO:teuthology.orchestra.run.smithi104.stdout:(9/32): ceph-common-19.0.0-5847.g79766939.el9.x 15 MB/s | 22 MB 00:01 2024-08-19T12:35:50.278 INFO:teuthology.orchestra.run.smithi026.stdout:(23/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 269 kB/s | 47 kB 00:00 2024-08-19T12:35:50.289 INFO:teuthology.orchestra.run.smithi104.stdout:(10/32): python3-cephfs-19.0.0-5847.g79766939.e 778 kB/s | 188 kB 00:00 2024-08-19T12:35:50.315 INFO:teuthology.orchestra.run.smithi104.stdout:(11/32): python3-rados-19.0.0-5847.g79766939.el 1.4 MB/s | 371 kB 00:00 2024-08-19T12:35:50.320 INFO:teuthology.orchestra.run.smithi026.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-08-19T12:35:50.338 INFO:teuthology.orchestra.run.smithi026.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 959 kB/s | 49 kB 00:00 2024-08-19T12:35:50.349 INFO:teuthology.orchestra.run.smithi104.stdout:(12/32): python3-rbd-19.0.0-5847.g79766939.el9. 4.2 MB/s | 356 kB 00:00 2024-08-19T12:35:50.363 INFO:teuthology.orchestra.run.smithi026.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-08-19T12:35:50.365 INFO:teuthology.orchestra.run.smithi104.stdout:(13/32): python3-rgw-19.0.0-5847.g79766939.el9. 1.5 MB/s | 116 kB 00:00 2024-08-19T12:35:50.405 INFO:teuthology.orchestra.run.smithi026.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-08-19T12:35:50.463 INFO:teuthology.orchestra.run.smithi026.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-08-19T12:35:50.530 INFO:teuthology.orchestra.run.smithi026.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-08-19T12:35:50.572 INFO:teuthology.orchestra.run.smithi026.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-08-19T12:35:50.699 INFO:teuthology.orchestra.run.smithi104.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 136 kB/s | 45 kB 00:00 2024-08-19T12:35:50.724 INFO:teuthology.orchestra.run.smithi104.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 261 kB/s | 107 kB 00:00 2024-08-19T12:35:50.739 INFO:teuthology.orchestra.run.smithi026.stdout:(31/32): librados2-19.0.0-5847.g79766939.el9.x8 13 MB/s | 3.5 MB 00:00 2024-08-19T12:35:50.767 INFO:teuthology.orchestra.run.smithi104.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 384 kB/s | 160 kB 00:00 2024-08-19T12:35:50.798 INFO:teuthology.orchestra.run.smithi026.stdout:(32/32): librbd1-19.0.0-5847.g79766939.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-08-19T12:35:50.803 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:35:50.803 INFO:teuthology.orchestra.run.smithi026.stdout:Total 20 MB/s | 62 MB 00:03 2024-08-19T12:35:50.858 INFO:teuthology.orchestra.run.smithi104.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 462 kB/s | 42 kB 00:00 2024-08-19T12:35:50.910 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:35:50.949 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:35:50.950 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:35:50.992 INFO:teuthology.orchestra.run.smithi104.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 292 kB 00:00 2024-08-19T12:35:51.042 INFO:teuthology.orchestra.run.smithi104.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.2 MB/s | 221 kB 00:00 2024-08-19T12:35:51.084 INFO:teuthology.orchestra.run.smithi104.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 391 kB/s | 36 kB 00:00 2024-08-19T12:35:51.126 INFO:teuthology.orchestra.run.smithi104.stdout:(21/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.5 MB/s | 662 kB 00:00 2024-08-19T12:35:51.143 INFO:teuthology.orchestra.run.smithi104.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 471 kB/s | 47 kB 00:00 2024-08-19T12:35:51.185 INFO:teuthology.orchestra.run.smithi104.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.0 MB/s | 309 kB 00:00 2024-08-19T12:35:51.201 INFO:teuthology.orchestra.run.smithi104.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 3.2 MB/s | 49 kB 00:00 2024-08-19T12:35:51.219 INFO:teuthology.orchestra.run.smithi104.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 380 kB/s | 26 kB 00:00 2024-08-19T12:35:51.236 INFO:teuthology.orchestra.run.smithi104.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-08-19T12:35:51.277 INFO:teuthology.orchestra.run.smithi104.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-08-19T12:35:51.328 INFO:teuthology.orchestra.run.smithi104.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-08-19T12:35:51.411 INFO:teuthology.orchestra.run.smithi104.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-08-19T12:35:51.450 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:35:51.450 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:35:51.478 INFO:teuthology.orchestra.run.smithi104.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-08-19T12:35:51.754 INFO:teuthology.orchestra.run.smithi104.stdout:(31/32): librados2-19.0.0-5847.g79766939.el9.x8 8.1 MB/s | 3.5 MB 00:00 2024-08-19T12:35:51.837 INFO:teuthology.orchestra.run.smithi104.stdout:(32/32): librbd1-19.0.0-5847.g79766939.el9.x86_ 7.6 MB/s | 3.2 MB 00:00 2024-08-19T12:35:51.842 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:35:51.843 INFO:teuthology.orchestra.run.smithi104.stdout:Total 19 MB/s | 62 MB 00:03 2024-08-19T12:35:51.947 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:35:51.983 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:35:51.984 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:35:52.461 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:35:52.461 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:35:52.923 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:35:53.015 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-19T12:35:53.052 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-08-19T12:35:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-08-19T12:35:53.100 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-19T12:35:53.123 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-19T12:35:53.406 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-19T12:35:53.473 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-19T12:35:53.837 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-08-19T12:35:53.912 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:35:53.954 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librados2-2:19.0.0-5847.g79766939.el9.x86_64 7/34 2024-08-19T12:35:54.003 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-19T12:35:54.042 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-08-19T12:35:54.051 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:19.0.0-5847.g79766939.el9.x86_64 7/34 2024-08-19T12:35:54.066 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-08-19T12:35:54.089 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-19T12:35:54.115 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-19T12:35:54.146 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 8/34 2024-08-19T12:35:54.172 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 9/34 2024-08-19T12:35:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 9/34 2024-08-19T12:35:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-08-19T12:35:54.302 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-argparse-2:19.0.0-5847.g79766939.el9. 11/34 2024-08-19T12:35:54.370 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 12/34 2024-08-19T12:35:54.370 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-19T12:35:54.400 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_6 13/34 2024-08-19T12:35:54.427 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-19T12:35:54.464 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_6 13/34 2024-08-19T12:35:54.498 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-08-19T12:35:54.609 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-08-19T12:35:54.640 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-08-19T12:35:54.673 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-08-19T12:35:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-08-19T12:35:54.735 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-08-19T12:35:54.767 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-08-19T12:35:54.809 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-08-19T12:35:54.877 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:19.0.0-5847.g79766939.el9.x86_64 7/34 2024-08-19T12:35:54.981 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:19.0.0-5847.g79766939.el9.x86_64 7/34 2024-08-19T12:35:55.076 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 8/34 2024-08-19T12:35:55.103 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 9/34 2024-08-19T12:35:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-08-19T12:35:55.147 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librbd1-2:19.0.0-5847.g79766939.el9.x86_64 22/34 2024-08-19T12:35:55.149 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 9/34 2024-08-19T12:35:55.178 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-08-19T12:35:55.224 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:19.0.0-5847.g79766939.el9. 11/34 2024-08-19T12:35:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:19.0.0-5847.g79766939.el9.x86_64 22/34 2024-08-19T12:35:55.268 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 23/34 2024-08-19T12:35:55.300 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 12/34 2024-08-19T12:35:55.305 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-19T12:35:55.322 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_6 13/34 2024-08-19T12:35:55.386 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_6 13/34 2024-08-19T12:35:55.421 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-08-19T12:35:55.530 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-08-19T12:35:55.564 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-08-19T12:35:55.629 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-08-19T12:35:55.662 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-08-19T12:35:55.691 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-08-19T12:35:55.723 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-08-19T12:35:55.824 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-19T12:35:55.849 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librgw2-2:19.0.0-5847.g79766939.el9.x86_64 26/34 2024-08-19T12:35:55.905 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librgw2-2:19.0.0-5847.g79766939.el9.x86_64 26/34 2024-08-19T12:35:55.985 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 27/34 2024-08-19T12:35:56.018 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-common-2:19.0.0-5847.g79766939.el9.x8 28/34 2024-08-19T12:35:56.034 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-08-19T12:35:56.087 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:19.0.0-5847.g79766939.el9.x86_64 22/34 2024-08-19T12:35:56.181 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:19.0.0-5847.g79766939.el9.x86_64 22/34 2024-08-19T12:35:56.208 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 23/34 2024-08-19T12:35:56.243 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-19T12:35:56.734 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-19T12:35:56.764 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:19.0.0-5847.g79766939.el9.x86_64 26/34 2024-08-19T12:35:56.836 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:19.0.0-5847.g79766939.el9.x86_64 26/34 2024-08-19T12:35:56.916 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 27/34 2024-08-19T12:35:56.943 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:19.0.0-5847.g79766939.el9.x8 28/34 2024-08-19T12:35:58.446 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 29/34 2024-08-19T12:35:58.483 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 29/34 2024-08-19T12:35:59.067 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 29/34 2024-08-19T12:35:59.103 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 30/34 2024-08-19T12:35:59.177 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 30/34 2024-08-19T12:35:59.178 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-19T12:35:59.178 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-19T12:35:59.178 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:35:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 31/34 2024-08-19T12:35:59.521 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 29/34 2024-08-19T12:35:59.556 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 29/34 2024-08-19T12:36:00.108 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 29/34 2024-08-19T12:36:00.143 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 30/34 2024-08-19T12:36:00.233 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 30/34 2024-08-19T12:36:00.233 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-19T12:36:00.233 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-19T12:36:00.233 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:00.260 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 31/34 2024-08-19T12:36:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 31/34 2024-08-19T12:36:09.192 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /sys 2024-08-19T12:36:09.192 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /proc 2024-08-19T12:36:09.192 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /mnt 2024-08-19T12:36:09.193 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /var/tmp 2024-08-19T12:36:09.193 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /home 2024-08-19T12:36:09.193 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /root 2024-08-19T12:36:09.193 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /tmp 2024-08-19T12:36:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:09.955 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 31/34 2024-08-19T12:36:09.955 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2024-08-19T12:36:09.955 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2024-08-19T12:36:09.955 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2024-08-19T12:36:09.955 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2024-08-19T12:36:09.956 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2024-08-19T12:36:09.956 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /root 2024-08-19T12:36:09.956 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2024-08-19T12:36:09.956 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 32/34 2024-08-19T12:36:10.255 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 32/34 2024-08-19T12:36:10.255 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:36:10.256 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-19T12:36:10.256 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T12:36:10.256 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T12:36:10.256 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:10.257 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-19T12:36:10.341 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-19T12:36:10.341 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T12:36:10.959 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 32/34 2024-08-19T12:36:10.984 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 32/34 2024-08-19T12:36:10.984 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:36:10.984 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-19T12:36:10.984 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T12:36:10.984 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-19T12:36:10.985 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:10.986 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-19T12:36:11.053 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-19T12:36:11.053 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T12:36:11.619 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T12:36:11.619 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 1/34 2024-08-19T12:36:11.619 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 2/34 2024-08-19T12:36:11.619 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 3/34 2024-08-19T12:36:11.619 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 4/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 5/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_6 6/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librgw2-2:19.0.0-5847.g79766939.el9.x86_64 7/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5847.g79766939.el9. 8/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-common-2:19.0.0-5847.g79766939.el9.x8 9/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 10/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 11/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 12/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 13/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-19T12:36:11.620 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-19T12:36:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-19T12:36:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-19T12:36:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-19T12:36:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-19T12:36:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-19T12:36:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-19T12:36:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-19T12:36:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-19T12:36:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-19T12:36:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-19T12:36:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-19T12:36:11.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-19T12:36:11.622 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-19T12:36:11.622 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:19.0.0-5847.g79766939.el9.x86_64 31/34 2024-08-19T12:36:11.622 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-19T12:36:11.622 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:19.0.0-5847.g79766939.el9.x86_64 33/34 2024-08-19T12:36:12.228 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T12:36:12.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 1/34 2024-08-19T12:36:12.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 2/34 2024-08-19T12:36:12.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 3/34 2024-08-19T12:36:12.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 4/34 2024-08-19T12:36:12.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 5/34 2024-08-19T12:36:12.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_6 6/34 2024-08-19T12:36:12.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:19.0.0-5847.g79766939.el9.x86_64 7/34 2024-08-19T12:36:12.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5847.g79766939.el9. 8/34 2024-08-19T12:36:12.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:19.0.0-5847.g79766939.el9.x8 9/34 2024-08-19T12:36:12.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 10/34 2024-08-19T12:36:12.229 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 11/34 2024-08-19T12:36:12.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 12/34 2024-08-19T12:36:12.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 13/34 2024-08-19T12:36:12.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-19T12:36:12.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-19T12:36:12.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-19T12:36:12.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-19T12:36:12.230 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-19T12:36:12.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:19.0.0-5847.g79766939.el9.x86_64 31/34 2024-08-19T12:36:12.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-19T12:36:12.232 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:19.0.0-5847.g79766939.el9.x86_64 33/34 2024-08-19T12:36:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T12:36:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:12.467 INFO:teuthology.orchestra.run.smithi026.stdout:Upgraded: 2024-08-19T12:36:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: librados2-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.468 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:36:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-19T12:36:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-19T12:36:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-19T12:36:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-19T12:36:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.468 INFO:teuthology.orchestra.run.smithi026.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-19T12:36:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-19T12:36:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-19T12:36:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-19T12:36:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-19T12:36:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-19T12:36:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-19T12:36:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.469 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-19T12:36:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-19T12:36:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-19T12:36:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-19T12:36:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: re2-1:20211101-3.el9.x86_64 2024-08-19T12:36:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-19T12:36:12.470 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:12.470 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:36:12.721 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-test 2024-08-19T12:36:13.049 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-19T12:36:13.049 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:13.049 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2024-08-19T12:36:13.049 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.049 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.049 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:36:13.049 INFO:teuthology.orchestra.run.smithi104.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-19T12:36:13.049 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.050 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.050 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.050 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.050 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-19T12:36:13.050 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-19T12:36:13.050 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-19T12:36:13.050 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.050 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-19T12:36:13.050 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-19T12:36:13.050 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-19T12:36:13.050 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.050 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-19T12:36:13.051 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-19T12:36:13.052 INFO:teuthology.orchestra.run.smithi104.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-19T12:36:13.052 INFO:teuthology.orchestra.run.smithi104.stdout: re2-1:20211101-3.el9.x86_64 2024-08-19T12:36:13.052 INFO:teuthology.orchestra.run.smithi104.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-19T12:36:13.052 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:13.052 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:36:13.309 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:28 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:36:13.380 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-test 2024-08-19T12:36:13.448 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:36:13.449 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:13.449 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-08-19T12:36:13.449 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:13.449 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:36:13.449 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test x86_64 2:19.0.0-5847.g79766939.el9 ceph 50 M 2024-08-19T12:36:13.449 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-08-19T12:36:13.449 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite x86_64 2:19.0.0-5847.g79766939.el9 ceph 163 k 2024-08-19T12:36:13.449 INFO:teuthology.orchestra.run.smithi026.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-19T12:36:13.449 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-19T12:36:13.449 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:13.449 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:36:13.449 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:13.450 INFO:teuthology.orchestra.run.smithi026.stdout:Install 4 Packages 2024-08-19T12:36:13.450 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:13.450 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 51 M 2024-08-19T12:36:13.450 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 216 M 2024-08-19T12:36:13.450 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:36:13.783 INFO:teuthology.orchestra.run.smithi026.stdout:(1/4): libcephsqlite-19.0.0-5847.g79766939.el9. 743 kB/s | 163 kB 00:00 2024-08-19T12:36:13.812 INFO:teuthology.orchestra.run.smithi026.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-08-19T12:36:13.941 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:27 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:36:13.943 INFO:teuthology.orchestra.run.smithi026.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 399 kB/s | 64 kB 00:00 2024-08-19T12:36:14.068 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:19.0.0-5847.g79766939.el9 ceph 50 M 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:19.0.0-5847.g79766939.el9 ceph 163 k 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:14.069 INFO:teuthology.orchestra.run.smithi104.stdout:Install 4 Packages 2024-08-19T12:36:14.070 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:14.070 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 51 M 2024-08-19T12:36:14.070 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 216 M 2024-08-19T12:36:14.070 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:36:14.317 INFO:teuthology.orchestra.run.smithi104.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-08-19T12:36:14.385 INFO:teuthology.orchestra.run.smithi104.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 952 kB/s | 64 kB 00:00 2024-08-19T12:36:14.603 INFO:teuthology.orchestra.run.smithi104.stdout:(3/4): libcephsqlite-19.0.0-5847.g79766939.el9. 336 kB/s | 163 kB 00:00 2024-08-19T12:36:15.236 INFO:teuthology.orchestra.run.smithi026.stdout:(4/4): ceph-test-19.0.0-5847.g79766939.el9.x86_ 30 MB/s | 50 MB 00:01 2024-08-19T12:36:15.238 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:36:15.238 INFO:teuthology.orchestra.run.smithi026.stdout:Total 28 MB/s | 51 MB 00:01 2024-08-19T12:36:15.267 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:36:15.289 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:36:15.289 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:36:15.687 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:36:15.688 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:36:16.097 INFO:teuthology.orchestra.run.smithi104.stdout:(4/4): ceph-test-19.0.0-5847.g79766939.el9.x86_ 25 MB/s | 50 MB 00:01 2024-08-19T12:36:16.099 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:36:16.099 INFO:teuthology.orchestra.run.smithi104.stdout:Total 25 MB/s | 51 MB 00:02 2024-08-19T12:36:16.127 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:36:16.146 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:36:16.147 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:36:16.357 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:36:16.421 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-19T12:36:16.460 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-19T12:36:16.488 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 3/4 2024-08-19T12:36:16.526 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:36:16.526 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:36:17.051 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:36:17.139 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-19T12:36:17.179 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-19T12:36:17.207 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 3/4 2024-08-19T12:36:21.479 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 3/4 2024-08-19T12:36:21.562 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 4/4 2024-08-19T12:36:21.909 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 3/4 2024-08-19T12:36:21.987 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 4/4 2024-08-19T12:36:24.075 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 4/4 2024-08-19T12:36:24.076 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 1/4 2024-08-19T12:36:24.076 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 2/4 2024-08-19T12:36:24.076 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-19T12:36:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-19T12:36:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:24.377 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:36:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-19T12:36:24.377 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-19T12:36:24.378 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:24.378 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:36:24.418 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 4/4 2024-08-19T12:36:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 1/4 2024-08-19T12:36:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 2/4 2024-08-19T12:36:24.419 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-19T12:36:24.665 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph 2024-08-19T12:36:24.704 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-19T12:36:24.704 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:24.704 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:36:24.704 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:24.704 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:24.705 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-19T12:36:24.705 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-19T12:36:24.705 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:24.705 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:36:24.997 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph 2024-08-19T12:36:25.221 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:40 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:36:25.360 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:36:25.362 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:25.362 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-08-19T12:36:25.362 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:25.362 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:36:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: ceph x86_64 2:19.0.0-5847.g79766939.el9 ceph 6.6 k 2024-08-19T12:36:25.363 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-08-19T12:36:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds x86_64 2:19.0.0-5847.g79766939.el9 ceph 2.4 M 2024-08-19T12:36:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr x86_64 2:19.0.0-5847.g79766939.el9 ceph 1.5 M 2024-08-19T12:36:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 284 k 2024-08-19T12:36:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon x86_64 2:19.0.0-5847.g79766939.el9 ceph 4.8 M 2024-08-19T12:36:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd x86_64 2:19.0.0-5847.g79766939.el9 ceph 17 M 2024-08-19T12:36:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-19T12:36:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-19T12:36:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-19T12:36:25.363 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-19T12:36:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-19T12:36:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-19T12:36:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-19T12:36:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-19T12:36:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-19T12:36:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-19T12:36:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections 2024-08-19T12:36:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-19T12:36:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-19T12:36:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-19T12:36:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-19T12:36:25.364 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-19T12:36:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-19T12:36:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-19T12:36:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-19T12:36:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-19T12:36:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-19T12:36:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-19T12:36:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-19T12:36:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-19T12:36:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-19T12:36:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-19T12:36:25.365 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-19T12:36:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-19T12:36:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-19T12:36:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-19T12:36:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-19T12:36:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-19T12:36:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:25.366 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:36:25.366 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:25.366 INFO:teuthology.orchestra.run.smithi026.stdout:Install 37 Packages 2024-08-19T12:36:25.366 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:25.367 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 31 M 2024-08-19T12:36:25.367 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 109 M 2024-08-19T12:36:25.367 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:36:25.554 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:39 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:36:25.691 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:36:25.694 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-08-19T12:36:25.694 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:25.694 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:36:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:19.0.0-5847.g79766939.el9 ceph 6.6 k 2024-08-19T12:36:25.694 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-08-19T12:36:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:19.0.0-5847.g79766939.el9 ceph 2.4 M 2024-08-19T12:36:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:19.0.0-5847.g79766939.el9 ceph 1.5 M 2024-08-19T12:36:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 284 k 2024-08-19T12:36:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:19.0.0-5847.g79766939.el9 ceph 4.8 M 2024-08-19T12:36:25.694 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:19.0.0-5847.g79766939.el9 ceph 17 M 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-19T12:36:25.695 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-19T12:36:25.696 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-19T12:36:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-19T12:36:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-19T12:36:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-19T12:36:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-19T12:36:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-19T12:36:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:25.697 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:36:25.697 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:25.697 INFO:teuthology.orchestra.run.smithi104.stdout:Install 37 Packages 2024-08-19T12:36:25.697 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:25.698 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 31 M 2024-08-19T12:36:25.698 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 109 M 2024-08-19T12:36:25.698 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:36:26.114 INFO:teuthology.orchestra.run.smithi104.stdout:(1/37): ceph-19.0.0-5847.g79766939.el9.x86_64.r 47 kB/s | 6.6 kB 00:00 2024-08-19T12:36:26.213 INFO:teuthology.orchestra.run.smithi026.stdout:(1/37): ceph-19.0.0-5847.g79766939.el9.x86_64.r 46 kB/s | 6.6 kB 00:00 2024-08-19T12:36:26.281 INFO:teuthology.orchestra.run.smithi104.stdout:(2/37): ceph-mgr-19.0.0-5847.g79766939.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-08-19T12:36:26.331 INFO:teuthology.orchestra.run.smithi104.stdout:(3/37): ceph-mds-19.0.0-5847.g79766939.el9.x86_ 6.6 MB/s | 2.4 MB 00:00 2024-08-19T12:36:26.405 INFO:teuthology.orchestra.run.smithi026.stdout:(2/37): ceph-mgr-19.0.0-5847.g79766939.el9.x86_ 4.6 MB/s | 1.5 MB 00:00 2024-08-19T12:36:26.415 INFO:teuthology.orchestra.run.smithi104.stdout:(4/37): ceph-mgr-modules-core-19.0.0-5847.g7976 3.3 MB/s | 284 kB 00:00 2024-08-19T12:36:26.480 INFO:teuthology.orchestra.run.smithi026.stdout:(3/37): ceph-mds-19.0.0-5847.g79766939.el9.x86_ 5.8 MB/s | 2.4 MB 00:00 2024-08-19T12:36:26.564 INFO:teuthology.orchestra.run.smithi026.stdout:(4/37): ceph-mgr-modules-core-19.0.0-5847.g7976 3.3 MB/s | 284 kB 00:00 2024-08-19T12:36:26.582 INFO:teuthology.orchestra.run.smithi104.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-08-19T12:36:26.857 INFO:teuthology.orchestra.run.smithi104.stdout:(6/37): ceph-mon-19.0.0-5847.g79766939.el9.x86_ 6.4 MB/s | 4.8 MB 00:00 2024-08-19T12:36:26.924 INFO:teuthology.orchestra.run.smithi104.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 3.6 MB/s | 1.2 MB 00:00 2024-08-19T12:36:26.947 INFO:teuthology.orchestra.run.smithi026.stdout:(5/37): ceph-mon-19.0.0-5847.g79766939.el9.x86_ 6.5 MB/s | 4.8 MB 00:00 2024-08-19T12:36:26.974 INFO:teuthology.orchestra.run.smithi104.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-08-19T12:36:27.158 INFO:teuthology.orchestra.run.smithi104.stdout:(9/37): ceph-osd-19.0.0-5847.g79766939.el9.x86_ 20 MB/s | 17 MB 00:00 2024-08-19T12:36:27.183 INFO:teuthology.orchestra.run.smithi104.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 327 kB/s | 106 kB 00:00 2024-08-19T12:36:27.208 INFO:teuthology.orchestra.run.smithi104.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 534 kB/s | 125 kB 00:00 2024-08-19T12:36:27.264 INFO:teuthology.orchestra.run.smithi026.stdout:(6/37): ceph-osd-19.0.0-5847.g79766939.el9.x86_ 20 MB/s | 17 MB 00:00 2024-08-19T12:36:27.267 INFO:teuthology.orchestra.run.smithi104.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-08-19T12:36:27.289 INFO:teuthology.orchestra.run.smithi026.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 349 kB/s | 253 kB 00:00 2024-08-19T12:36:27.373 INFO:teuthology.orchestra.run.smithi026.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 984 kB/s | 106 kB 00:00 2024-08-19T12:36:27.481 INFO:teuthology.orchestra.run.smithi026.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.1 MB/s | 125 kB 00:00 2024-08-19T12:36:27.715 INFO:teuthology.orchestra.run.smithi026.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 920 kB/s | 215 kB 00:00 2024-08-19T12:36:27.740 INFO:teuthology.orchestra.run.smithi026.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 300 kB/s | 135 kB 00:00 2024-08-19T12:36:27.867 INFO:teuthology.orchestra.run.smithi104.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 58 kB/s | 35 kB 00:00 2024-08-19T12:36:27.926 INFO:teuthology.orchestra.run.smithi104.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 240 kB/s | 172 kB 00:00 2024-08-19T12:36:27.941 INFO:teuthology.orchestra.run.smithi026.stdout:(12/37): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:00 2024-08-19T12:36:27.951 INFO:teuthology.orchestra.run.smithi104.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 245 kB/s | 188 kB 00:00 2024-08-19T12:36:28.166 INFO:teuthology.orchestra.run.smithi026.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 405 kB/s | 172 kB 00:00 2024-08-19T12:36:28.191 INFO:teuthology.orchestra.run.smithi026.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 396 kB/s | 188 kB 00:00 2024-08-19T12:36:28.216 INFO:teuthology.orchestra.run.smithi026.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 126 kB/s | 35 kB 00:00 2024-08-19T12:36:28.485 INFO:teuthology.orchestra.run.smithi026.stdout:(16/37): python3-toml-0.10.2-6.el9.noarch.rpm 156 kB/s | 42 kB 00:00 2024-08-19T12:36:28.509 INFO:teuthology.orchestra.run.smithi104.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 120 kB/s | 77 kB 00:00 2024-08-19T12:36:28.525 INFO:teuthology.orchestra.run.smithi026.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 153 kB/s | 51 kB 00:00 2024-08-19T12:36:28.534 INFO:teuthology.orchestra.run.smithi104.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 71 kB/s | 42 kB 00:00 2024-08-19T12:36:28.560 INFO:teuthology.orchestra.run.smithi104.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 81 kB/s | 51 kB 00:00 2024-08-19T12:36:28.575 INFO:teuthology.orchestra.run.smithi026.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-08-19T12:36:28.601 INFO:teuthology.orchestra.run.smithi026.stdout:(19/37): python3-packaging-20.9-5.el9.noarch.rp 177 kB/s | 77 kB 00:00 2024-08-19T12:36:28.601 INFO:teuthology.orchestra.run.smithi104.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-08-19T12:36:28.619 INFO:teuthology.orchestra.run.smithi104.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 207 kB/s | 22 kB 00:00 2024-08-19T12:36:28.626 INFO:teuthology.orchestra.run.smithi026.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 156 kB/s | 22 kB 00:00 2024-08-19T12:36:28.636 INFO:teuthology.orchestra.run.smithi104.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 576 kB/s | 43 kB 00:00 2024-08-19T12:36:28.659 INFO:teuthology.orchestra.run.smithi026.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 520 kB/s | 43 kB 00:00 2024-08-19T12:36:28.669 INFO:teuthology.orchestra.run.smithi104.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.6 MB/s | 172 kB 00:00 2024-08-19T12:36:28.686 INFO:teuthology.orchestra.run.smithi104.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 248 kB/s | 11 kB 00:00 2024-08-19T12:36:28.693 INFO:teuthology.orchestra.run.smithi026.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 453 kB/s | 11 kB 00:00 2024-08-19T12:36:28.703 INFO:teuthology.orchestra.run.smithi104.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 555 kB/s | 18 kB 00:00 2024-08-19T12:36:28.718 INFO:teuthology.orchestra.run.smithi026.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 776 kB/s | 18 kB 00:00 2024-08-19T12:36:28.720 INFO:teuthology.orchestra.run.smithi104.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 697 kB/s | 23 kB 00:00 2024-08-19T12:36:28.737 INFO:teuthology.orchestra.run.smithi104.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 583 kB/s | 19 kB 00:00 2024-08-19T12:36:28.743 INFO:teuthology.orchestra.run.smithi026.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-08-19T12:36:28.762 INFO:teuthology.orchestra.run.smithi104.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 469 kB/s | 20 kB 00:00 2024-08-19T12:36:28.769 INFO:teuthology.orchestra.run.smithi026.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-08-19T12:36:28.787 INFO:teuthology.orchestra.run.smithi104.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-08-19T12:36:28.794 INFO:teuthology.orchestra.run.smithi026.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-08-19T12:36:28.813 INFO:teuthology.orchestra.run.smithi104.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-08-19T12:36:28.820 INFO:teuthology.orchestra.run.smithi026.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-08-19T12:36:28.838 INFO:teuthology.orchestra.run.smithi104.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 656 kB/s | 16 kB 00:00 2024-08-19T12:36:28.845 INFO:teuthology.orchestra.run.smithi026.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-08-19T12:36:28.870 INFO:teuthology.orchestra.run.smithi026.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-08-19T12:36:28.872 INFO:teuthology.orchestra.run.smithi104.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-08-19T12:36:28.895 INFO:teuthology.orchestra.run.smithi026.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-08-19T12:36:28.897 INFO:teuthology.orchestra.run.smithi104.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-08-19T12:36:28.921 INFO:teuthology.orchestra.run.smithi026.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-08-19T12:36:28.922 INFO:teuthology.orchestra.run.smithi104.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 716 kB/s | 36 kB 00:00 2024-08-19T12:36:28.946 INFO:teuthology.orchestra.run.smithi026.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-08-19T12:36:28.947 INFO:teuthology.orchestra.run.smithi104.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-08-19T12:36:28.971 INFO:teuthology.orchestra.run.smithi026.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-08-19T12:36:28.972 INFO:teuthology.orchestra.run.smithi104.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 808 kB/s | 20 kB 00:00 2024-08-19T12:36:28.996 INFO:teuthology.orchestra.run.smithi026.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-08-19T12:36:28.997 INFO:teuthology.orchestra.run.smithi104.stdout:(36/37): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-08-19T12:36:29.022 INFO:teuthology.orchestra.run.smithi026.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 807 kB/s | 20 kB 00:00 2024-08-19T12:36:29.031 INFO:teuthology.orchestra.run.smithi104.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-08-19T12:36:29.037 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:36:29.037 INFO:teuthology.orchestra.run.smithi104.stdout:Total 9.2 MB/s | 31 MB 00:03 2024-08-19T12:36:29.047 INFO:teuthology.orchestra.run.smithi026.stdout:(36/37): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-19T12:36:29.072 INFO:teuthology.orchestra.run.smithi026.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-08-19T12:36:29.077 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:36:29.077 INFO:teuthology.orchestra.run.smithi026.stdout:Total 8.3 MB/s | 31 MB 00:03 2024-08-19T12:36:29.188 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:36:29.229 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:36:29.229 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:36:29.229 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:36:29.271 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:36:29.271 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:36:29.530 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:36:29.531 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:36:29.584 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:36:29.585 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:36:30.906 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:36:30.967 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-19T12:36:31.000 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-19T12:36:31.028 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-19T12:36:31.098 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-19T12:36:31.123 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:36:31.143 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-19T12:36:31.176 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-19T12:36:31.192 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-19T12:36:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-19T12:36:31.252 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-19T12:36:31.267 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-19T12:36:31.327 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-19T12:36:31.355 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-19T12:36:31.404 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-19T12:36:31.439 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-19T12:36:31.469 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-19T12:36:31.485 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-19T12:36:31.503 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-19T12:36:31.518 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-19T12:36:31.536 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-19T12:36:31.585 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-19T12:36:31.615 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-19T12:36:31.620 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-19T12:36:31.655 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-19T12:36:31.691 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-19T12:36:31.703 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-19T12:36:31.735 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-19T12:36:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-19T12:36:31.772 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-19T12:36:31.802 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-19T12:36:31.807 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-19T12:36:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-19T12:36:31.870 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-19T12:36:31.911 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-19T12:36:31.945 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-19T12:36:31.954 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-19T12:36:31.988 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-19T12:36:32.026 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-19T12:36:32.037 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-19T12:36:32.077 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-19T12:36:32.079 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-19T12:36:32.118 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-19T12:36:32.153 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-19T12:36:32.167 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-19T12:36:32.222 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-19T12:36:32.295 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-19T12:36:32.388 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-19T12:36:32.407 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-19T12:36:32.455 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-19T12:36:32.524 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-19T12:36:32.565 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-19T12:36:32.601 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-19T12:36:32.612 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-19T12:36:32.645 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-19T12:36:32.659 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-19T12:36:32.723 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-19T12:36:32.772 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-19T12:36:32.917 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-19T12:36:32.935 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5847.g79766939.el9. 32/37 2024-08-19T12:36:32.963 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-19T12:36:32.964 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 33/37 2024-08-19T12:36:32.990 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 33/37 2024-08-19T12:36:32.990 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:36:32.990 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-19T12:36:32.990 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T12:36:32.990 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T12:36:32.990 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:33.044 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-19T12:36:33.134 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-19T12:36:33.171 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-19T12:36:33.304 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-19T12:36:33.476 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5847.g79766939.el9. 32/37 2024-08-19T12:36:33.506 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 33/37 2024-08-19T12:36:33.531 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 33/37 2024-08-19T12:36:33.531 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:36:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-19T12:36:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T12:36:33.532 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-19T12:36:33.532 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:34.449 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 34/37 2024-08-19T12:36:34.475 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 34/37 2024-08-19T12:36:34.475 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:36:34.476 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-19T12:36:34.476 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T12:36:34.476 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T12:36:34.476 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:34.916 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 35/37 2024-08-19T12:36:34.939 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 35/37 2024-08-19T12:36:34.940 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:36:34.940 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-19T12:36:34.940 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T12:36:34.940 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T12:36:34.940 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:35.073 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 34/37 2024-08-19T12:36:35.100 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 34/37 2024-08-19T12:36:35.100 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:36:35.101 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-19T12:36:35.101 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T12:36:35.101 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-19T12:36:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:35.166 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 36/37 2024-08-19T12:36:35.189 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 36/37 2024-08-19T12:36:35.190 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:36:35.190 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-19T12:36:35.190 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T12:36:35.190 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T12:36:35.190 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:35.254 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:19.0.0-5847.g79766939.el9.x86_64 37/37 2024-08-19T12:36:35.574 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 35/37 2024-08-19T12:36:35.597 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 35/37 2024-08-19T12:36:35.597 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:36:35.597 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-19T12:36:35.597 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T12:36:35.597 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-19T12:36:35.597 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 36/37 2024-08-19T12:36:35.864 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 36/37 2024-08-19T12:36:35.865 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:36:35.865 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-19T12:36:35.865 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T12:36:35.865 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-19T12:36:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-2:19.0.0-5847.g79766939.el9.x86_64 37/37 2024-08-19T12:36:37.031 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-2:19.0.0-5847.g79766939.el9.x86_64 37/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:19.0.0-5847.g79766939.el9.x86_64 1/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 2/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 3/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 4/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 5/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5847.g79766939.el9. 6/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-19T12:36:37.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-19T12:36:37.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-19T12:36:37.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-19T12:36:37.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-19T12:36:37.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-19T12:36:37.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-19T12:36:37.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-19T12:36:37.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-19T12:36:37.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-19T12:36:37.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-19T12:36:37.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-19T12:36:37.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-19T12:36:37.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-19T12:36:37.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-19T12:36:37.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-19T12:36:37.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-19T12:36:37.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-19T12:36:37.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-19T12:36:37.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-19T12:36:37.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-19T12:36:37.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-19T12:36:37.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-19T12:36:37.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-19T12:36:37.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-19T12:36:37.546 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-2:19.0.0-5847.g79766939.el9.x86_64 37/37 2024-08-19T12:36:37.547 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-2:19.0.0-5847.g79766939.el9.x86_64 1/37 2024-08-19T12:36:37.547 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 2/37 2024-08-19T12:36:37.547 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 3/37 2024-08-19T12:36:37.547 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 4/37 2024-08-19T12:36:37.547 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 5/37 2024-08-19T12:36:37.547 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5847.g79766939.el9. 6/37 2024-08-19T12:36:37.547 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-19T12:36:37.547 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-19T12:36:37.547 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-19T12:36:37.547 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-19T12:36:37.548 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-19T12:36:37.549 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-19T12:36:37.549 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-19T12:36:37.550 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-19T12:36:37.550 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-19T12:36:37.550 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-19T12:36:37.550 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-19T12:36:37.550 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-19T12:36:37.550 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-19T12:36:37.550 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-19T12:36:37.550 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-19T12:36:37.550 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-19T12:36:37.551 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-19T12:36:37.551 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-19T12:36:37.985 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-19T12:36:37.985 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:37.985 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:36:37.985 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: lua-5.4.4-4.el9.x86_64 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-19T12:36:37.986 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply-3.11-14.el9.noarch 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-19T12:36:37.987 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-19T12:36:37.988 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-19T12:36:37.988 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-19T12:36:37.988 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-19T12:36:37.988 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-19T12:36:37.988 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-19T12:36:37.988 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-19T12:36:37.988 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-19T12:36:37.988 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-19T12:36:37.988 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-19T12:36:37.988 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:37.988 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:36:38.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-base 2024-08-19T12:36:38.472 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: lua-5.4.4-4.el9.x86_64 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-19T12:36:38.473 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-19T12:36:38.474 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply-3.11-14.el9.noarch 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:38.475 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:36:38.792 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-base 2024-08-19T12:36:38.879 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:52 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:36:38.962 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:36:39.003 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:36:39.004 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-08-19T12:36:39.004 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:36:39.067 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install cephadm 2024-08-19T12:36:39.344 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:54 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:36:39.431 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-base-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:36:39.474 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:36:39.476 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-08-19T12:36:39.476 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:36:39.544 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install cephadm 2024-08-19T12:36:39.615 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:53 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:36:39.741 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:36:39.742 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-08-19T12:36:39.742 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:39.742 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:36:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 772 k 2024-08-19T12:36:39.742 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:39.742 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:36:39.743 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:39.743 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-08-19T12:36:39.743 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:39.743 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 772 k 2024-08-19T12:36:39.743 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 779 k 2024-08-19T12:36:39.743 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:36:40.023 INFO:teuthology.orchestra.run.smithi104.stdout:cephadm-19.0.0-5847.g79766939.el9.noarch.rpm 2.7 MB/s | 772 kB 00:00 2024-08-19T12:36:40.023 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:36:40.024 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.7 MB/s | 772 kB 00:00 2024-08-19T12:36:40.024 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:36:40.029 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:36:40.029 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:36:40.035 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:36:40.035 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:36:40.084 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:55 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:36:40.104 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:36:40.214 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:36:40.215 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:40.215 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-08-19T12:36:40.216 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:40.216 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:36:40.216 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 772 k 2024-08-19T12:36:40.216 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:40.216 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:36:40.216 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:40.216 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-08-19T12:36:40.216 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:40.216 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 772 k 2024-08-19T12:36:40.216 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 779 k 2024-08-19T12:36:40.216 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:36:40.473 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm-19.0.0-5847.g79766939.el9.noarch.rpm 2.9 MB/s | 772 kB 00:00 2024-08-19T12:36:40.473 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:36:40.473 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.9 MB/s | 772 kB 00:00 2024-08-19T12:36:40.474 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:36:40.479 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:36:40.479 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:36:40.485 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:36:40.485 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:36:40.524 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:36:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:36:40.616 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:36:40.955 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:36:41.034 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:36:41.128 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:36:41.214 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:36:41.214 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:41.214 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:36:41.214 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:36:41.215 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:41.215 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:36:41.433 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-immutable-object-cache 2024-08-19T12:36:41.453 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:36:41.692 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:36:41.692 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:41.692 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:36:41.692 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:36:41.693 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:41.693 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:36:41.923 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-immutable-object-cache 2024-08-19T12:36:41.980 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:55 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:36:42.109 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:36:42.109 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:42.110 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-08-19T12:36:42.110 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:42.110 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:36:42.110 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5847.g79766939.el9 ceph 145 k 2024-08-19T12:36:42.110 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:42.110 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:36:42.110 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:42.110 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-08-19T12:36:42.110 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:42.110 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 145 k 2024-08-19T12:36:42.110 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 436 k 2024-08-19T12:36:42.110 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:36:42.298 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-immutable-object-cache-19.0.0-5847.g797669 775 kB/s | 145 kB 00:00 2024-08-19T12:36:42.298 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:36:42.299 INFO:teuthology.orchestra.run.smithi104.stdout:Total 767 kB/s | 145 kB 00:00 2024-08-19T12:36:42.299 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:36:42.307 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:36:42.307 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:36:42.365 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:36:42.365 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:36:42.475 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:57 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:36:42.513 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:36:42.586 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5847.g79766939. 1/1 2024-08-19T12:36:42.607 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:36:42.608 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:42.608 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-08-19T12:36:42.609 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:42.609 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:36:42.609 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5847.g79766939.el9 ceph 145 k 2024-08-19T12:36:42.609 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:42.609 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:36:42.609 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:42.610 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-08-19T12:36:42.610 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:42.610 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 145 k 2024-08-19T12:36:42.610 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 436 k 2024-08-19T12:36:42.610 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:36:42.612 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5847.g79766939. 1/1 2024-08-19T12:36:42.612 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:36:42.612 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-19T12:36:42.612 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:42.928 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-immutable-object-cache-19.0.0-5847.g797669 453 kB/s | 145 kB 00:00 2024-08-19T12:36:42.928 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:36:42.929 INFO:teuthology.orchestra.run.smithi026.stdout:Total 451 kB/s | 145 kB 00:00 2024-08-19T12:36:42.929 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:36:42.937 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:36:42.937 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:36:42.997 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:36:42.997 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:36:43.147 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:36:43.229 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5847.g79766939. 1/1 2024-08-19T12:36:43.254 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5847.g79766939. 1/1 2024-08-19T12:36:43.254 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:36:43.254 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-19T12:36:43.254 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:43.487 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5847.g79766939. 1/1 2024-08-19T12:36:43.487 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:43.488 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:36:43.488 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:43.488 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:43.488 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:36:43.695 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr 2024-08-19T12:36:44.141 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5847.g79766939. 1/1 2024-08-19T12:36:44.142 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:44.142 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:36:44.142 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:36:44.142 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:44.142 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:36:44.242 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:58 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:36:44.329 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:36:44.371 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:36:44.372 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-08-19T12:36:44.372 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:36:44.406 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr 2024-08-19T12:36:44.437 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-dashboard 2024-08-19T12:36:44.955 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:59 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:36:44.968 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:58 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:36:45.043 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-mgr-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:36:45.087 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:36:45.088 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-08-19T12:36:45.088 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:36:45.094 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:36:45.095 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:45.095 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-08-19T12:36:45.095 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:45.095 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:36:45.095 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 14 M 2024-08-19T12:36:45.095 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 34 k 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 16 k 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:45.096 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:36:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:36:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Install 12 Packages 2024-08-19T12:36:45.097 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 19 M 2024-08-19T12:36:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 158 M 2024-08-19T12:36:45.097 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:36:45.172 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-dashboard 2024-08-19T12:36:45.530 INFO:teuthology.orchestra.run.smithi104.stdout:(1/12): ceph-grafana-dashboards-19.0.0-5847.g79 229 kB/s | 34 kB 00:00 2024-08-19T12:36:45.556 INFO:teuthology.orchestra.run.smithi104.stdout:(2/12): ceph-prometheus-alerts-19.0.0-5847.g797 94 kB/s | 16 kB 00:00 2024-08-19T12:36:45.689 INFO:teuthology.orchestra.run.smithi104.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 269 kB 00:00 2024-08-19T12:36:45.723 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:00 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:36:45.739 INFO:teuthology.orchestra.run.smithi104.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-08-19T12:36:45.806 INFO:teuthology.orchestra.run.smithi104.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.0 MB/s | 548 kB 00:00 2024-08-19T12:36:45.832 INFO:teuthology.orchestra.run.smithi104.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 784 kB/s | 19 kB 00:00 2024-08-19T12:36:45.855 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:36:45.856 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:45.856 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-08-19T12:36:45.857 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:45.857 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:36:45.857 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 14 M 2024-08-19T12:36:45.857 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-08-19T12:36:45.857 INFO:teuthology.orchestra.run.smithi026.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-19T12:36:45.857 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 34 k 2024-08-19T12:36:45.857 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 16 k 2024-08-19T12:36:45.857 INFO:teuthology.orchestra.run.smithi026.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-19T12:36:45.857 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-19T12:36:45.857 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-19T12:36:45.857 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-19T12:36:45.857 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-19T12:36:45.858 INFO:teuthology.orchestra.run.smithi026.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-19T12:36:45.858 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-19T12:36:45.858 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-19T12:36:45.858 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:45.858 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:36:45.858 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:36:45.858 INFO:teuthology.orchestra.run.smithi026.stdout:Install 12 Packages 2024-08-19T12:36:45.858 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:45.859 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 19 M 2024-08-19T12:36:45.859 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 158 M 2024-08-19T12:36:45.859 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:36:45.932 INFO:teuthology.orchestra.run.smithi104.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 20 MB/s | 2.0 MB 00:00 2024-08-19T12:36:45.957 INFO:teuthology.orchestra.run.smithi104.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-08-19T12:36:45.982 INFO:teuthology.orchestra.run.smithi104.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-08-19T12:36:46.016 INFO:teuthology.orchestra.run.smithi104.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.6 MB/s | 863 kB 00:00 2024-08-19T12:36:46.041 INFO:teuthology.orchestra.run.smithi104.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-08-19T12:36:46.183 INFO:teuthology.orchestra.run.smithi104.stdout:(12/12): ceph-mgr-dashboard-19.0.0-5847.g797669 18 MB/s | 14 MB 00:00 2024-08-19T12:36:46.186 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:36:46.186 INFO:teuthology.orchestra.run.smithi104.stdout:Total 18 MB/s | 19 MB 00:01 2024-08-19T12:36:46.235 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:36:46.253 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:36:46.254 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:36:46.265 INFO:teuthology.orchestra.run.smithi026.stdout:(1/12): ceph-prometheus-alerts-19.0.0-5847.g797 109 kB/s | 16 kB 00:00 2024-08-19T12:36:46.290 INFO:teuthology.orchestra.run.smithi026.stdout:(2/12): ceph-grafana-dashboards-19.0.0-5847.g79 194 kB/s | 34 kB 00:00 2024-08-19T12:36:46.433 INFO:teuthology.orchestra.run.smithi026.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-08-19T12:36:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-08-19T12:36:46.474 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:36:46.474 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:36:46.525 INFO:teuthology.orchestra.run.smithi026.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-08-19T12:36:46.550 INFO:teuthology.orchestra.run.smithi026.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 780 kB/s | 19 kB 00:00 2024-08-19T12:36:46.754 INFO:teuthology.orchestra.run.smithi026.stdout:(7/12): ceph-mgr-dashboard-19.0.0-5847.g7976693 22 MB/s | 14 MB 00:00 2024-08-19T12:36:46.792 INFO:teuthology.orchestra.run.smithi026.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 8.4 MB/s | 2.0 MB 00:00 2024-08-19T12:36:46.826 INFO:teuthology.orchestra.run.smithi026.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.1 MB/s | 31 kB 00:00 2024-08-19T12:36:46.859 INFO:teuthology.orchestra.run.smithi026.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.0 MB/s | 863 kB 00:00 2024-08-19T12:36:46.884 INFO:teuthology.orchestra.run.smithi026.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.3 MB/s | 188 kB 00:00 2024-08-19T12:36:46.910 INFO:teuthology.orchestra.run.smithi026.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 909 kB/s | 144 kB 00:00 2024-08-19T12:36:46.913 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:36:46.913 INFO:teuthology.orchestra.run.smithi026.stdout:Total 18 MB/s | 19 MB 00:01 2024-08-19T12:36:46.962 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:36:46.982 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:36:46.982 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:36:47.054 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:36:47.133 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-08-19T12:36:47.168 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-08-19T12:36:47.212 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:36:47.213 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:36:47.215 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-08-19T12:36:47.245 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-08-19T12:36:47.323 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-08-19T12:36:47.463 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-08-19T12:36:47.557 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-08-19T12:36:47.600 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-08-19T12:36:47.628 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-08-19T12:36:47.661 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5847.g79766939.el9 10/12 2024-08-19T12:36:47.830 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:36:47.911 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-08-19T12:36:47.946 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-08-19T12:36:48.003 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-08-19T12:36:48.031 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-08-19T12:36:48.112 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-08-19T12:36:48.246 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-08-19T12:36:48.339 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-08-19T12:36:48.387 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-08-19T12:36:48.414 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-08-19T12:36:48.447 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5847.g79766939.el9 10/12 2024-08-19T12:36:50.845 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5847.g79766939.el 11/12 2024-08-19T12:36:50.898 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noa 12/12 2024-08-19T12:36:51.856 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5847.g79766939.el 11/12 2024-08-19T12:36:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noa 12/12 2024-08-19T12:36:52.768 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noa 12/12 2024-08-19T12:36:52.769 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5847.g79766939.el 1/12 2024-08-19T12:36:52.769 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noa 2/12 2024-08-19T12:36:52.769 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5847.g79766939.el9 3/12 2024-08-19T12:36:52.769 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-08-19T12:36:52.769 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-08-19T12:36:52.769 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-08-19T12:36:52.769 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-08-19T12:36:52.769 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-08-19T12:36:52.769 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-08-19T12:36:52.769 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-08-19T12:36:52.769 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-08-19T12:36:53.080 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-08-19T12:36:53.080 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:53.080 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:36:53.080 INFO:teuthology.orchestra.run.smithi104.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-19T12:36:53.080 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:36:53.081 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:36:53.081 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:36:53.081 INFO:teuthology.orchestra.run.smithi104.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-19T12:36:53.081 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-19T12:36:53.081 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-19T12:36:53.081 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-19T12:36:53.081 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-19T12:36:53.081 INFO:teuthology.orchestra.run.smithi104.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-19T12:36:53.081 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-19T12:36:53.081 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-19T12:36:53.081 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:53.081 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:36:53.347 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-19T12:36:53.870 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noa 12/12 2024-08-19T12:36:53.870 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5847.g79766939.el 1/12 2024-08-19T12:36:53.870 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noa 2/12 2024-08-19T12:36:53.870 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5847.g79766939.el9 3/12 2024-08-19T12:36:53.871 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-08-19T12:36:53.871 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-08-19T12:36:53.871 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-08-19T12:36:53.871 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-08-19T12:36:53.871 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-08-19T12:36:53.871 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-08-19T12:36:53.871 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-08-19T12:36:53.871 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-08-19T12:36:53.882 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:07 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:36:54.010 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:36:54.011 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-08-19T12:36:54.011 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-08-19T12:36:54.012 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-08-19T12:36:54.012 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:36:54.012 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 7.4 M 2024-08-19T12:36:54.012 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-08-19T12:36:54.012 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-19T12:36:54.012 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-19T12:36:54.012 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-19T12:36:54.012 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-19T12:36:54.012 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-19T12:36:54.012 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-19T12:36:54.012 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-19T12:36:54.012 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel x86_64 3.9.19-4.el9 appstream 244 k 2024-08-19T12:36:54.013 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-19T12:36:54.013 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-19T12:36:54.013 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-19T12:36:54.013 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:54.013 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:36:54.013 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-08-19T12:36:54.013 INFO:teuthology.orchestra.run.smithi104.stdout:Install 12 Packages 2024-08-19T12:36:54.013 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:36:54.014 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 38 M 2024-08-19T12:36:54.014 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 209 M 2024-08-19T12:36:54.014 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:36:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-08-19T12:36:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:54.182 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:36:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-19T12:36:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:36:54.182 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:36:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:36:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-19T12:36:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-19T12:36:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-19T12:36:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-19T12:36:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-19T12:36:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-19T12:36:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-19T12:36:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-19T12:36:54.183 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:54.183 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:36:54.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-19T12:36:54.614 INFO:teuthology.orchestra.run.smithi104.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 797 kB/s | 195 kB 00:00 2024-08-19T12:36:54.680 INFO:teuthology.orchestra.run.smithi104.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-08-19T12:36:54.973 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:09 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:36:55.110 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:36:55.111 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-08-19T12:36:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-08-19T12:36:55.111 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-08-19T12:36:55.111 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 7.4 M 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel x86_64 3.9.19-4.el9 appstream 244 k 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-19T12:36:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-19T12:36:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:55.113 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:36:55.113 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================= 2024-08-19T12:36:55.113 INFO:teuthology.orchestra.run.smithi026.stdout:Install 12 Packages 2024-08-19T12:36:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:36:55.113 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 38 M 2024-08-19T12:36:55.113 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 209 M 2024-08-19T12:36:55.114 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:36:55.164 INFO:teuthology.orchestra.run.smithi104.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 59 kB/s | 33 kB 00:00 2024-08-19T12:36:55.533 INFO:teuthology.orchestra.run.smithi026.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-08-19T12:36:55.600 INFO:teuthology.orchestra.run.smithi026.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.7 MB/s | 803 kB 00:00 2024-08-19T12:36:55.658 INFO:teuthology.orchestra.run.smithi026.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-08-19T12:36:55.672 INFO:teuthology.orchestra.run.smithi104.stdout:(4/12): ceph-mgr-diskprediction-local-19.0.0-58 5.7 MB/s | 7.4 MB 00:01 2024-08-19T12:36:55.689 INFO:teuthology.orchestra.run.smithi104.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 35 kB/s | 18 kB 00:00 2024-08-19T12:36:55.709 INFO:teuthology.orchestra.run.smithi026.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-08-19T12:36:55.739 INFO:teuthology.orchestra.run.smithi104.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.8 MB/s | 3.0 MB 00:01 2024-08-19T12:36:55.750 INFO:teuthology.orchestra.run.smithi026.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 848 kB/s | 35 kB 00:00 2024-08-19T12:36:55.842 INFO:teuthology.orchestra.run.smithi026.stdout:(6/12): ceph-mgr-diskprediction-local-19.0.0-58 16 MB/s | 7.4 MB 00:00 2024-08-19T12:36:56.101 INFO:teuthology.orchestra.run.smithi026.stdout:(7/12): python3-devel-3.9.19-4.el9.x86_64.rpm 945 kB/s | 244 kB 00:00 2024-08-19T12:36:56.151 INFO:teuthology.orchestra.run.smithi026.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.4 MB/s | 3.0 MB 00:00 2024-08-19T12:36:56.223 INFO:teuthology.orchestra.run.smithi104.stdout:(7/12): openblas-0.3.21-2.el9.x86_64.rpm 64 kB/s | 35 kB 00:00 2024-08-19T12:36:56.227 INFO:teuthology.orchestra.run.smithi026.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.3 MB/s | 250 kB 00:00 2024-08-19T12:36:56.498 INFO:teuthology.orchestra.run.smithi104.stdout:(8/12): python3-devel-3.9.19-4.el9.x86_64.rpm 322 kB/s | 244 kB 00:00 2024-08-19T12:36:56.552 INFO:teuthology.orchestra.run.smithi026.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-08-19T12:36:56.628 INFO:teuthology.orchestra.run.smithi026.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 5.4 MB/s | 4.7 MB 00:00 2024-08-19T12:36:56.807 INFO:teuthology.orchestra.run.smithi104.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 4.2 MB/s | 4.7 MB 00:01 2024-08-19T12:36:57.219 INFO:teuthology.orchestra.run.smithi026.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-08-19T12:36:57.220 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:36:57.220 INFO:teuthology.orchestra.run.smithi026.stdout:Total 18 MB/s | 38 MB 00:02 2024-08-19T12:36:57.282 INFO:teuthology.orchestra.run.smithi104.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 320 kB/s | 250 kB 00:00 2024-08-19T12:36:57.365 INFO:teuthology.orchestra.run.smithi104.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 5.1 MB 00:01 2024-08-19T12:36:57.580 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:36:57.607 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:36:57.607 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:36:58.037 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:36:58.037 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:36:58.257 INFO:teuthology.orchestra.run.smithi104.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-08-19T12:36:58.258 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:36:58.259 INFO:teuthology.orchestra.run.smithi104.stdout:Total 9.0 MB/s | 38 MB 00:04 2024-08-19T12:36:58.620 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:36:58.647 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:36:58.647 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:36:58.835 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:36:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-19T12:36:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-19T12:36:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-19T12:36:59.057 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:36:59.057 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:36:59.077 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-devel-3.9.19-4.el9.x86_64 4/12 2024-08-19T12:36:59.261 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-19T12:36:59.292 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-19T12:36:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-19T12:36:59.808 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:36:59.903 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-19T12:36:59.955 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-19T12:37:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-19T12:37:00.056 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-19T12:37:00.106 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-devel-3.9.19-4.el9.x86_64 4/12 2024-08-19T12:37:00.158 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-19T12:37:00.283 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-19T12:37:00.314 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-19T12:37:00.391 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-19T12:37:01.049 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-19T12:37:01.160 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-19T12:37:01.437 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-19T12:37:02.399 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-19T12:37:02.514 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-19T12:37:02.541 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766 12/12 2024-08-19T12:37:03.424 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-19T12:37:03.462 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766 12/12 2024-08-19T12:37:05.057 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766 12/12 2024-08-19T12:37:05.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766 1/12 2024-08-19T12:37:05.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-19T12:37:05.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-19T12:37:05.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-19T12:37:05.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-19T12:37:05.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-19T12:37:05.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-19T12:37:05.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-19T12:37:05.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-devel-3.9.19-4.el9.x86_64 9/12 2024-08-19T12:37:05.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-19T12:37:05.058 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-19T12:37:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-19T12:37:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:05.440 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:37:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:37:05.440 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-19T12:37:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-19T12:37:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-19T12:37:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-19T12:37:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-19T12:37:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-19T12:37:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-19T12:37:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel-3.9.19-4.el9.x86_64 2024-08-19T12:37:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-19T12:37:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-19T12:37:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-19T12:37:05.441 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:05.441 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:05.846 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-rook 2024-08-19T12:37:06.050 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766 12/12 2024-08-19T12:37:06.050 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766 1/12 2024-08-19T12:37:06.050 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-19T12:37:06.050 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-19T12:37:06.050 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-19T12:37:06.050 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-19T12:37:06.050 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-19T12:37:06.051 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-19T12:37:06.051 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-19T12:37:06.051 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-devel-3.9.19-4.el9.x86_64 9/12 2024-08-19T12:37:06.051 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-19T12:37:06.051 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-19T12:37:06.402 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:21 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:06.403 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-19T12:37:06.403 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:06.403 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:37:06.403 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:37:06.403 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-19T12:37:06.403 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-19T12:37:06.403 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-19T12:37:06.403 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-19T12:37:06.404 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-19T12:37:06.404 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-19T12:37:06.404 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-19T12:37:06.404 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel-3.9.19-4.el9.x86_64 2024-08-19T12:37:06.404 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-19T12:37:06.404 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-19T12:37:06.404 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-19T12:37:06.404 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:06.404 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:06.535 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:06.536 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-08-19T12:37:06.536 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:06.536 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:37:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 49 k 2024-08-19T12:37:06.536 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-08-19T12:37:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:06.537 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:37:06.538 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:06.538 INFO:teuthology.orchestra.run.smithi026.stdout:Install 13 Packages 2024-08-19T12:37:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:06.538 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 2.2 M 2024-08-19T12:37:06.538 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 26 M 2024-08-19T12:37:06.539 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:37:06.724 INFO:teuthology.orchestra.run.smithi026.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 176 kB/s | 19 kB 00:00 2024-08-19T12:37:06.749 INFO:teuthology.orchestra.run.smithi026.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 191 kB/s | 26 kB 00:00 2024-08-19T12:37:06.756 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-rook 2024-08-19T12:37:06.775 INFO:teuthology.orchestra.run.smithi026.stdout:(3/13): ceph-mgr-rook-19.0.0-5847.g79766939.el9 309 kB/s | 49 kB 00:00 2024-08-19T12:37:06.875 INFO:teuthology.orchestra.run.smithi026.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-08-19T12:37:06.900 INFO:teuthology.orchestra.run.smithi026.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-08-19T12:37:06.926 INFO:teuthology.orchestra.run.smithi026.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-08-19T12:37:06.968 INFO:teuthology.orchestra.run.smithi026.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 479 kB/s | 32 kB 00:00 2024-08-19T12:37:06.993 INFO:teuthology.orchestra.run.smithi026.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-08-19T12:37:07.018 INFO:teuthology.orchestra.run.smithi026.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-08-19T12:37:07.052 INFO:teuthology.orchestra.run.smithi026.stdout:(10/13): python3-google-auth-2.33.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-08-19T12:37:07.077 INFO:teuthology.orchestra.run.smithi026.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-08-19T12:37:07.102 INFO:teuthology.orchestra.run.smithi026.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-08-19T12:37:07.144 INFO:teuthology.orchestra.run.smithi026.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-08-19T12:37:07.146 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:07.147 INFO:teuthology.orchestra.run.smithi026.stdout:Total 3.6 MB/s | 2.2 MB 00:00 2024-08-19T12:37:07.210 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:37:07.226 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:37:07.227 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:37:07.298 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:21 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:07.318 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:37:07.319 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:37:07.425 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:07.426 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:07.426 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-08-19T12:37:07.427 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:07.427 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:37:07.427 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 49 k 2024-08-19T12:37:07.427 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-08-19T12:37:07.427 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-19T12:37:07.427 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-19T12:37:07.427 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-19T12:37:07.427 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-19T12:37:07.427 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-19T12:37:07.427 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-19T12:37:07.428 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-19T12:37:07.428 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-19T12:37:07.428 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-19T12:37:07.428 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-19T12:37:07.428 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-19T12:37:07.428 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-19T12:37:07.428 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:07.428 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:37:07.428 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:07.428 INFO:teuthology.orchestra.run.smithi104.stdout:Install 13 Packages 2024-08-19T12:37:07.428 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:07.429 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 2.2 M 2024-08-19T12:37:07.429 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 26 M 2024-08-19T12:37:07.429 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:37:07.853 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:37:07.911 INFO:teuthology.orchestra.run.smithi104.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 262 kB/s | 26 kB 00:00 2024-08-19T12:37:07.936 INFO:teuthology.orchestra.run.smithi104.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 156 kB/s | 19 kB 00:00 2024-08-19T12:37:07.995 INFO:teuthology.orchestra.run.smithi104.stdout:(3/13): ceph-mgr-rook-19.0.0-5847.g79766939.el9 271 kB/s | 49 kB 00:00 2024-08-19T12:37:08.040 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-19T12:37:08.070 INFO:teuthology.orchestra.run.smithi104.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-08-19T12:37:08.096 INFO:teuthology.orchestra.run.smithi104.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-08-19T12:37:08.122 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-19T12:37:08.129 INFO:teuthology.orchestra.run.smithi104.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 918 kB/s | 54 kB 00:00 2024-08-19T12:37:08.163 INFO:teuthology.orchestra.run.smithi104.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-08-19T12:37:08.177 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-19T12:37:08.188 INFO:teuthology.orchestra.run.smithi104.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 247 kB/s | 14 kB 00:00 2024-08-19T12:37:08.213 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-19T12:37:08.214 INFO:teuthology.orchestra.run.smithi104.stdout:(9/13): python3-google-auth-2.33.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-08-19T12:37:08.239 INFO:teuthology.orchestra.run.smithi104.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.1 MB/s | 279 kB 00:00 2024-08-19T12:37:08.255 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-19T12:37:08.264 INFO:teuthology.orchestra.run.smithi104.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-08-19T12:37:08.306 INFO:teuthology.orchestra.run.smithi104.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-08-19T12:37:08.339 INFO:teuthology.orchestra.run.smithi104.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-08-19T12:37:08.342 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:08.342 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-08-19T12:37:08.369 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-19T12:37:08.407 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:37:08.421 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:37:08.421 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:37:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-19T12:37:08.507 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:37:08.509 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:37:08.532 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-19T12:37:09.019 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:37:09.178 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-19T12:37:09.231 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-19T12:37:09.287 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-19T12:37:09.296 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-19T12:37:09.324 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-19T12:37:09.345 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-19T12:37:09.366 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-19T12:37:09.379 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-19T12:37:09.428 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-19T12:37:09.456 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 13/13 2024-08-19T12:37:09.468 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-19T12:37:09.568 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-19T12:37:09.617 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-19T12:37:10.195 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 13/13 2024-08-19T12:37:10.195 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 1/13 2024-08-19T12:37:10.196 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-19T12:37:10.196 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-19T12:37:10.196 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-19T12:37:10.196 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-19T12:37:10.196 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-19T12:37:10.196 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-19T12:37:10.196 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-19T12:37:10.196 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-19T12:37:10.196 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-19T12:37:10.196 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-19T12:37:10.196 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-19T12:37:10.328 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-19T12:37:10.381 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-19T12:37:10.423 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-19T12:37:10.472 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-19T12:37:10.501 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 13/13 2024-08-19T12:37:10.647 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-19T12:37:10.648 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:10.648 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:37:10.648 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:37:10.648 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-19T12:37:10.648 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-19T12:37:10.648 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-19T12:37:10.648 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-19T12:37:10.648 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-19T12:37:10.648 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-19T12:37:10.648 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-19T12:37:10.648 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-19T12:37:10.649 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-19T12:37:10.649 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-19T12:37:10.649 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-19T12:37:10.649 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-19T12:37:10.649 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:10.649 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:10.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-cephadm 2024-08-19T12:37:11.224 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 13/13 2024-08-19T12:37:11.224 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 1/13 2024-08-19T12:37:11.224 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-19T12:37:11.224 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-19T12:37:11.224 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-19T12:37:11.224 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-19T12:37:11.224 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-19T12:37:11.225 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-19T12:37:11.225 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-19T12:37:11.225 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-19T12:37:11.225 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-19T12:37:11.225 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-19T12:37:11.225 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-19T12:37:11.440 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:26 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:11.531 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-19T12:37:11.531 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:11.531 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:37:11.531 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:37:11.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-19T12:37:11.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-19T12:37:11.531 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-19T12:37:11.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-19T12:37:11.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-19T12:37:11.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-19T12:37:11.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-19T12:37:11.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-19T12:37:11.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-19T12:37:11.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-19T12:37:11.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-19T12:37:11.532 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-19T12:37:11.532 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:11.532 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:11.575 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:11.576 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:11.576 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-08-19T12:37:11.576 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 150 k 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-08-19T12:37:11.577 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:11.578 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 7.0 M 2024-08-19T12:37:11.578 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 33 M 2024-08-19T12:37:11.578 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:37:11.786 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-cephadm 2024-08-19T12:37:11.962 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-08-19T12:37:11.987 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): ceph-mgr-cephadm-19.0.0-5847.g79766939.e 675 kB/s | 150 kB 00:00 2024-08-19T12:37:12.029 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.4 MB/s | 58 kB 00:00 2024-08-19T12:37:12.054 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.2 MB/s | 74 kB 00:00 2024-08-19T12:37:12.088 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.3 MB/s | 548 kB 00:00 2024-08-19T12:37:12.188 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-08-19T12:37:12.191 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:12.191 INFO:teuthology.orchestra.run.smithi026.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-08-19T12:37:12.264 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:37:12.276 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:37:12.276 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:37:12.327 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:26 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:12.379 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:37:12.381 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:37:12.454 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:12.455 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:12.455 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-08-19T12:37:12.455 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:12.455 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:37:12.455 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 150 k 2024-08-19T12:37:12.455 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-08-19T12:37:12.455 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-19T12:37:12.455 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-19T12:37:12.455 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-19T12:37:12.456 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-19T12:37:12.456 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-19T12:37:12.456 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:12.456 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:37:12.456 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:12.456 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-08-19T12:37:12.456 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:12.457 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 7.0 M 2024-08-19T12:37:12.457 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 33 M 2024-08-19T12:37:12.457 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:37:12.690 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-08-19T12:37:12.695 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:37:12.740 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-08-19T12:37:12.766 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): ceph-mgr-cephadm-19.0.0-5847.g79766939.e 703 kB/s | 150 kB 00:00 2024-08-19T12:37:12.791 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-08-19T12:37:12.833 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-08-19T12:37:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-19T12:37:12.916 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-08-19T12:37:12.918 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:12.919 INFO:teuthology.orchestra.run.smithi104.stdout:Total 15 MB/s | 7.0 MB 00:00 2024-08-19T12:37:12.950 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-19T12:37:12.988 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:37:13.000 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:37:13.000 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:37:13.101 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:37:13.102 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:37:13.407 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:37:13.520 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-19T12:37:13.573 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-19T12:37:13.605 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-19T12:37:13.619 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-19T12:37:13.688 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-19T12:37:13.711 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 6/6 2024-08-19T12:37:14.130 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-19T12:37:14.222 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-19T12:37:14.291 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-19T12:37:14.325 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 6/6 2024-08-19T12:37:14.425 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 6/6 2024-08-19T12:37:14.425 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/6 2024-08-19T12:37:14.425 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-19T12:37:14.426 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-19T12:37:14.426 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-19T12:37:14.426 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-19T12:37:14.703 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-19T12:37:14.703 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:14.703 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:37:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:37:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-19T12:37:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-19T12:37:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-19T12:37:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-19T12:37:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-19T12:37:14.704 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:14.704 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:14.927 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-fuse 2024-08-19T12:37:15.110 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 6/6 2024-08-19T12:37:15.110 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 1/6 2024-08-19T12:37:15.110 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-19T12:37:15.110 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-19T12:37:15.110 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-19T12:37:15.110 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-19T12:37:15.373 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-19T12:37:15.373 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:15.374 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:37:15.374 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:37:15.374 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-19T12:37:15.374 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-19T12:37:15.374 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-19T12:37:15.374 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-19T12:37:15.374 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-19T12:37:15.374 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:15.374 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:15.481 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:30 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:15.614 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:15.615 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-08-19T12:37:15.615 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:15.615 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:37:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse x86_64 2:19.0.0-5847.g79766939.el9 ceph 901 k 2024-08-19T12:37:15.615 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-08-19T12:37:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-19T12:37:15.615 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:15.615 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:37:15.615 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:15.615 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-08-19T12:37:15.616 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:15.616 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 981 k 2024-08-19T12:37:15.616 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 2.9 M 2024-08-19T12:37:15.616 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:37:15.623 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-fuse 2024-08-19T12:37:16.065 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 414 kB/s | 80 kB 00:00 2024-08-19T12:37:16.158 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:30 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:16.165 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): ceph-fuse-19.0.0-5847.g79766939.el9.x86_ 3.0 MB/s | 901 kB 00:00 2024-08-19T12:37:16.166 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:16.166 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.7 MB/s | 981 kB 00:00 2024-08-19T12:37:16.179 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:37:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:37:16.202 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:37:16.285 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:16.286 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-08-19T12:37:16.286 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:16.286 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:37:16.286 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:19.0.0-5847.g79766939.el9 ceph 901 k 2024-08-19T12:37:16.287 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-08-19T12:37:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-19T12:37:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:16.287 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:37:16.287 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:16.287 INFO:teuthology.orchestra.run.smithi104.stdout:Install 2 Packages 2024-08-19T12:37:16.287 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:16.287 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 981 k 2024-08-19T12:37:16.287 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 2.9 M 2024-08-19T12:37:16.287 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:37:16.288 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:37:16.288 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:37:16.461 INFO:teuthology.orchestra.run.smithi104.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 516 kB/s | 80 kB 00:00 2024-08-19T12:37:16.487 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:37:16.619 INFO:teuthology.orchestra.run.smithi104.stdout:(2/2): ceph-fuse-19.0.0-5847.g79766939.el9.x86_ 2.8 MB/s | 901 kB 00:00 2024-08-19T12:37:16.620 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:16.620 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.9 MB/s | 981 kB 00:00 2024-08-19T12:37:16.632 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:37:16.639 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-19T12:37:16.653 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:37:16.653 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:37:16.710 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 2/2 2024-08-19T12:37:16.731 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:37:16.732 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:37:16.924 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:37:17.064 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-19T12:37:17.125 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 2/2 2024-08-19T12:37:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 2/2 2024-08-19T12:37:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/2 2024-08-19T12:37:17.628 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-19T12:37:17.628 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:17.628 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:37:17.629 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-19T12:37:17.629 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:17.629 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:17.742 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 2/2 2024-08-19T12:37:17.742 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/2 2024-08-19T12:37:17.899 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-volume 2024-08-19T12:37:17.998 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-19T12:37:17.999 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:17.999 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:37:17.999 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:19.0.0-5847.g79766939.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-19T12:37:17.999 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:17.999 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:18.294 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-volume 2024-08-19T12:37:18.457 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:33 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:18.593 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:18.594 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-08-19T12:37:18.594 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:18.594 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:37:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 285 k 2024-08-19T12:37:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:18.594 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:37:18.594 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:18.594 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-08-19T12:37:18.594 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:18.594 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 285 k 2024-08-19T12:37:18.595 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 1.3 M 2024-08-19T12:37:18.595 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:37:18.805 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-volume-19.0.0-5847.g79766939.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-08-19T12:37:18.805 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:18.805 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-08-19T12:37:18.806 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:37:18.815 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:37:18.815 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:37:18.837 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:37:18.837 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:37:18.845 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:32 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:18.978 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:18.979 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:18.979 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-08-19T12:37:18.979 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:18.979 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:37:18.979 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume noarch 2:19.0.0-5847.g79766939.el9 ceph-noarch 285 k 2024-08-19T12:37:18.980 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:18.980 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:37:18.980 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:18.980 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-08-19T12:37:18.980 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:18.980 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 285 k 2024-08-19T12:37:18.980 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 1.3 M 2024-08-19T12:37:18.980 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:37:19.058 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:37:19.115 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:37:19.131 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:37:19.131 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:37:19.131 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-19T12:37:19.131 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:19.215 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-volume-19.0.0-5847.g79766939.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-08-19T12:37:19.217 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:19.217 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-08-19T12:37:19.217 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:37:19.224 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:37:19.225 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:37:19.244 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:37:19.245 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:37:19.459 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:37:19.525 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:37:19.541 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:37:19.541 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:37:19.541 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-19T12:37:19.542 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:20.102 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:37:20.103 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:20.103 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:37:20.103 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:37:20.103 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:20.103 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:20.414 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados-devel 2024-08-19T12:37:20.506 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 1/1 2024-08-19T12:37:20.506 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:20.506 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:37:20.507 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume-2:19.0.0-5847.g79766939.el9.noarch 2024-08-19T12:37:20.507 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:20.507 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:20.835 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados-devel 2024-08-19T12:37:20.976 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:21.111 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:21.112 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-08-19T12:37:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:37:21.112 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel x86_64 2:19.0.0-5847.g79766939.el9 ceph 127 k 2024-08-19T12:37:21.112 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:37:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:21.112 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-08-19T12:37:21.112 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 127 k 2024-08-19T12:37:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 456 k 2024-08-19T12:37:21.113 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:37:21.342 INFO:teuthology.orchestra.run.smithi026.stdout:librados-devel-19.0.0-5847.g79766939.el9.x86_64 555 kB/s | 127 kB 00:00 2024-08-19T12:37:21.343 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:21.343 INFO:teuthology.orchestra.run.smithi026.stdout:Total 551 kB/s | 127 kB 00:00 2024-08-19T12:37:21.343 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:37:21.347 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:37:21.348 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:37:21.393 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:21.415 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:37:21.415 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:37:21.523 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:21.524 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:21.524 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-08-19T12:37:21.524 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:21.524 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:37:21.524 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:19.0.0-5847.g79766939.el9 ceph 127 k 2024-08-19T12:37:21.524 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:21.524 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:37:21.525 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:21.525 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-08-19T12:37:21.525 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:21.525 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 127 k 2024-08-19T12:37:21.525 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 456 k 2024-08-19T12:37:21.525 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:37:21.570 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:37:21.675 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:21.744 INFO:teuthology.orchestra.run.smithi104.stdout:librados-devel-19.0.0-5847.g79766939.el9.x86_64 577 kB/s | 127 kB 00:00 2024-08-19T12:37:21.745 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:21.745 INFO:teuthology.orchestra.run.smithi104.stdout:Total 573 kB/s | 127 kB 00:00 2024-08-19T12:37:21.745 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:37:21.751 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:37:21.751 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:37:21.813 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:37:21.814 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:37:21.964 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:37:22.065 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:22.096 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:22.361 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:22.361 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:22.361 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:37:22.361 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:37:22.361 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:22.361 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:22.505 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:22.612 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs2 2024-08-19T12:37:22.796 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:22.796 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:22.796 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:37:22.797 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:37:22.797 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:22.797 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:23.038 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs2 2024-08-19T12:37:23.167 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:23.255 INFO:teuthology.orchestra.run.smithi026.stdout:Package libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:23.301 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:23.302 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-08-19T12:37:23.302 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:23.378 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs-devel 2024-08-19T12:37:23.579 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:37 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:23.664 INFO:teuthology.orchestra.run.smithi104.stdout:Package libcephfs2-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:23.708 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:23.709 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-08-19T12:37:23.709 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:23.790 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs-devel 2024-08-19T12:37:23.926 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:24.059 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:24.059 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:24.059 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-08-19T12:37:24.060 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:24.060 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:37:24.060 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel x86_64 2:19.0.0-5847.g79766939.el9 ceph 32 k 2024-08-19T12:37:24.060 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:24.060 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:37:24.060 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:24.060 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-08-19T12:37:24.060 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:24.060 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 32 k 2024-08-19T12:37:24.060 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 142 k 2024-08-19T12:37:24.060 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:37:24.221 INFO:teuthology.orchestra.run.smithi026.stdout:libcephfs-devel-19.0.0-5847.g79766939.el9.x86_6 197 kB/s | 32 kB 00:00 2024-08-19T12:37:24.222 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:24.223 INFO:teuthology.orchestra.run.smithi026.stdout:Total 195 kB/s | 32 kB 00:00 2024-08-19T12:37:24.223 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:37:24.226 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:37:24.226 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:37:24.229 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:37:24.229 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:37:24.323 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:37:24.328 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:24.437 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:24.456 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:24.457 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-08-19T12:37:24.457 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:24.457 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:37:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:19.0.0-5847.g79766939.el9 ceph 32 k 2024-08-19T12:37:24.457 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:24.458 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:37:24.458 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:24.458 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-08-19T12:37:24.458 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:24.458 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 32 k 2024-08-19T12:37:24.458 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 142 k 2024-08-19T12:37:24.458 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:37:24.605 INFO:teuthology.orchestra.run.smithi104.stdout:libcephfs-devel-19.0.0-5847.g79766939.el9.x86_6 215 kB/s | 32 kB 00:00 2024-08-19T12:37:24.606 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:24.606 INFO:teuthology.orchestra.run.smithi104.stdout:Total 213 kB/s | 32 kB 00:00 2024-08-19T12:37:24.606 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:37:24.610 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:37:24.610 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:37:24.615 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:37:24.615 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:37:24.705 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:37:24.769 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:24.817 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:25.001 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:25.001 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:25.001 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:37:25.001 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:37:25.001 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:25.002 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:25.156 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:25.178 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados2 2024-08-19T12:37:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:25.370 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:37:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:37:25.370 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:25.371 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:25.536 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados2 2024-08-19T12:37:25.732 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:40 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:25.820 INFO:teuthology.orchestra.run.smithi026.stdout:Package librados2-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:25.865 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:25.866 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-08-19T12:37:25.866 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:25.947 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librbd1 2024-08-19T12:37:26.073 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:40 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:26.157 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:26.200 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:26.201 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-08-19T12:37:26.201 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:26.264 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librbd1 2024-08-19T12:37:26.511 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:41 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:26.600 INFO:teuthology.orchestra.run.smithi026.stdout:Package librbd1-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:26.647 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:26.648 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-08-19T12:37:26.648 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:26.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rados 2024-08-19T12:37:26.803 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:40 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:26.887 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:26.930 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:26.931 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-08-19T12:37:26.931 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:26.997 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rados 2024-08-19T12:37:27.292 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:42 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:27.380 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:27.425 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:27.426 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-08-19T12:37:27.426 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:27.503 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rgw 2024-08-19T12:37:27.535 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:41 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:27.620 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rados-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:27.664 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:27.665 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-08-19T12:37:27.665 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:27.736 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rgw 2024-08-19T12:37:28.052 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:43 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:28.140 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:28.187 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:28.187 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-08-19T12:37:28.188 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:28.263 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-cephfs 2024-08-19T12:37:28.273 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:42 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:28.357 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rgw-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:28.400 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:28.400 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-08-19T12:37:28.401 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:28.464 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-cephfs 2024-08-19T12:37:28.815 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:43 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:28.904 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:28.949 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:28.950 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-08-19T12:37:28.950 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:29.003 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:43 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:29.029 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rbd 2024-08-19T12:37:29.087 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-cephfs-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:29.130 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:29.131 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-08-19T12:37:29.131 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:29.194 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rbd 2024-08-19T12:37:29.579 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:44 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:29.666 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:29.712 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:29.713 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-08-19T12:37:29.713 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:29.734 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:43 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:29.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-fuse 2024-08-19T12:37:29.818 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rbd-2:19.0.0-5847.g79766939.el9.x86_64 is already installed. 2024-08-19T12:37:29.861 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:29.861 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-08-19T12:37:29.862 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:29.923 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-fuse 2024-08-19T12:37:30.337 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:45 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:30.462 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:44 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:30.470 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:30.471 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:30.472 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-08-19T12:37:30.472 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:30.472 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:37:30.472 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse x86_64 2:19.0.0-5847.g79766939.el9 ceph 85 k 2024-08-19T12:37:30.472 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:30.472 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:37:30.473 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:30.473 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-08-19T12:37:30.473 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:30.473 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 85 k 2024-08-19T12:37:30.473 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 231 k 2024-08-19T12:37:30.473 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:37:30.589 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:30.590 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-08-19T12:37:30.590 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:30.590 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:37:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:19.0.0-5847.g79766939.el9 ceph 85 k 2024-08-19T12:37:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:30.590 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:37:30.590 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:30.590 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-08-19T12:37:30.590 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:30.591 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 85 k 2024-08-19T12:37:30.591 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 231 k 2024-08-19T12:37:30.591 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:37:30.667 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-fuse-19.0.0-5847.g79766939.el9.x86_64.rpm 440 kB/s | 85 kB 00:00 2024-08-19T12:37:30.667 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:30.667 INFO:teuthology.orchestra.run.smithi026.stdout:Total 436 kB/s | 85 kB 00:00 2024-08-19T12:37:30.667 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:37:30.674 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:37:30.674 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:37:30.741 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:37:30.741 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:37:30.791 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-fuse-19.0.0-5847.g79766939.el9.x86_64.rpm 424 kB/s | 85 kB 00:00 2024-08-19T12:37:30.792 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:30.792 INFO:teuthology.orchestra.run.smithi104.stdout:Total 421 kB/s | 85 kB 00:00 2024-08-19T12:37:30.792 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:37:30.799 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:37:30.799 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:37:30.861 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:37:30.862 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:37:30.887 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:37:30.998 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:31.015 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:37:31.103 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:31.409 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:31.468 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:31.640 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:31.640 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:31.640 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:37:31.640 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:37:31.640 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:31.641 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:31.733 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:31.733 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:31.733 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:37:31.733 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:37:31.733 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:31.734 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:31.900 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-mirror 2024-08-19T12:37:32.009 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-mirror 2024-08-19T12:37:32.450 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:47 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:32.553 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:46 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:32.586 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:32.587 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:32.587 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-08-19T12:37:32.587 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:32.587 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:37:32.587 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror x86_64 2:19.0.0-5847.g79766939.el9 ceph 3.2 M 2024-08-19T12:37:32.587 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:32.587 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:37:32.587 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:32.587 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-08-19T12:37:32.587 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:32.587 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 3.2 M 2024-08-19T12:37:32.587 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 14 M 2024-08-19T12:37:32.588 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:37:32.682 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:32.683 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:32.683 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-08-19T12:37:32.683 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:32.683 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:37:32.683 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:19.0.0-5847.g79766939.el9 ceph 3.2 M 2024-08-19T12:37:32.683 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:32.683 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:37:32.684 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:32.684 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-08-19T12:37:32.684 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:32.684 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 3.2 M 2024-08-19T12:37:32.684 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 14 M 2024-08-19T12:37:32.684 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:37:33.661 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-mirror-19.0.0-5847.g79766939.el9.x86_64.rpm 2.9 MB/s | 3.2 MB 00:01 2024-08-19T12:37:33.661 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:33.662 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.9 MB/s | 3.2 MB 00:01 2024-08-19T12:37:33.662 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:37:33.673 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:37:33.673 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:37:33.727 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-mirror-19.0.0-5847.g79766939.el9.x86_64.rpm 3.0 MB/s | 3.2 MB 00:01 2024-08-19T12:37:33.728 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:33.728 INFO:teuthology.orchestra.run.smithi104.stdout:Total 3.0 MB/s | 3.2 MB 00:01 2024-08-19T12:37:33.729 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:37:33.738 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:37:33.738 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:37:33.760 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:37:33.760 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:37:33.822 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:37:33.823 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:37:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:37:34.287 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:34.301 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:37:34.312 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:34.312 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:37:34.312 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-19T12:37:34.312 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T12:37:34.312 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T12:37:34.312 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:34.353 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:34.379 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:34.379 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-19T12:37:34.380 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-19T12:37:34.380 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T12:37:34.380 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-19T12:37:34.380 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:35.434 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:35.434 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:35.434 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:37:35.434 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:37:35.434 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:35.434 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:35.491 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:37:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:37:35.491 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:35.491 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:35.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-nbd 2024-08-19T12:37:35.755 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-nbd 2024-08-19T12:37:36.236 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:51 ago on Mon 19 Aug 2024 12:35:45 PM UTC. 2024-08-19T12:37:36.312 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:50 ago on Mon 19 Aug 2024 12:35:46 PM UTC. 2024-08-19T12:37:36.371 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-08-19T12:37:36.372 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-08-19T12:37:36.372 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:36.372 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-08-19T12:37:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd x86_64 2:19.0.0-5847.g79766939.el9 ceph 171 k 2024-08-19T12:37:36.372 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:36.372 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-08-19T12:37:36.373 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-08-19T12:37:36.373 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-08-19T12:37:36.373 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:36.373 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 171 k 2024-08-19T12:37:36.373 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 490 k 2024-08-19T12:37:36.373 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-08-19T12:37:36.439 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-08-19T12:37:36.439 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:36.439 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-08-19T12:37:36.440 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:36.440 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-08-19T12:37:36.440 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:19.0.0-5847.g79766939.el9 ceph 171 k 2024-08-19T12:37:36.440 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:36.440 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-08-19T12:37:36.440 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-08-19T12:37:36.440 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-08-19T12:37:36.440 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:36.441 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 171 k 2024-08-19T12:37:36.441 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 490 k 2024-08-19T12:37:36.441 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-08-19T12:37:36.588 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-nbd-19.0.0-5847.g79766939.el9.x86_64.rpm 792 kB/s | 171 kB 00:00 2024-08-19T12:37:36.589 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:36.589 INFO:teuthology.orchestra.run.smithi026.stdout:Total 787 kB/s | 171 kB 00:00 2024-08-19T12:37:36.590 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-08-19T12:37:36.598 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-08-19T12:37:36.598 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-08-19T12:37:36.655 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-nbd-19.0.0-5847.g79766939.el9.x86_64.rpm 798 kB/s | 171 kB 00:00 2024-08-19T12:37:36.655 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-08-19T12:37:36.656 INFO:teuthology.orchestra.run.smithi104.stdout:Total 792 kB/s | 171 kB 00:00 2024-08-19T12:37:36.656 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-08-19T12:37:36.663 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-08-19T12:37:36.663 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-08-19T12:37:36.667 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-08-19T12:37:36.668 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-08-19T12:37:36.727 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-08-19T12:37:36.727 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-08-19T12:37:36.822 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-08-19T12:37:36.874 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-08-19T12:37:36.930 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:36.976 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:37.405 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:37.618 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:37.618 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:37.618 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-08-19T12:37:37.618 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:37:37.618 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:37.618 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-08-19T12:37:37.676 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 1/1 2024-08-19T12:37:37.676 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:37.676 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-08-19T12:37:37.676 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:19.0.0-5847.g79766939.el9.x86_64 2024-08-19T12:37:37.676 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:37.676 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-08-19T12:37:37.865 DEBUG:teuthology.parallel:result is None 2024-08-19T12:37:37.915 DEBUG:teuthology.parallel:result is None 2024-08-19T12:37:37.916 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T12:37:38.069 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-19T12:37:38.090 INFO:teuthology.orchestra.run.smithi026.stdout:19.0.0-5847.g79766939.el9 2024-08-19T12:37:38.091 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5847.g79766939.el9 2024-08-19T12:37:38.091 INFO:teuthology.task.install:The correct ceph version 19.0.0-5847.g79766939 is installed. 2024-08-19T12:37:38.093 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-19T12:37:38.245 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-19T12:37:38.266 INFO:teuthology.orchestra.run.smithi104.stdout:19.0.0-5847.g79766939.el9 2024-08-19T12:37:38.267 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5847.g79766939.el9 2024-08-19T12:37:38.267 INFO:teuthology.task.install:The correct ceph version 19.0.0-5847.g79766939 is installed. 2024-08-19T12:37:38.268 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-19T12:37:38.269 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:37:38.269 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-19T12:37:38.294 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:37:38.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-19T12:37:38.332 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-19T12:37:38.332 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:37:38.332 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2024-08-19T12:37:38.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-19T12:37:38.423 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:37:38.424 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-08-19T12:37:38.448 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-19T12:37:38.513 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-19T12:37:38.513 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:37:38.513 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-19T12:37:38.538 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-19T12:37:38.620 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:37:38.620 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-19T12:37:38.645 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-19T12:37:38.709 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-19T12:37:38.710 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:37:38.710 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/stdin-killer 2024-08-19T12:37:38.734 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-19T12:37:38.799 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:37:38.799 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-08-19T12:37:38.824 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-19T12:37:38.889 INFO:teuthology.run_tasks:Running task ceph... 2024-08-19T12:37:38.975 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-19T12:37:38.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /var/log/ceph 2024-08-19T12:37:38.978 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /var/log/ceph 2024-08-19T12:37:39.002 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-19T12:37:39.003 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-19T12:37:39.042 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-19T12:37:39.067 INFO:tasks.ceph:Creating extra log directories... 2024-08-19T12:37:39.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-19T12:37:39.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-19T12:37:39.142 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-08-19T12:37:39.142 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-08-19T12:37:39.142 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520', 'branch': 'main', 'description': 'rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_api_tests}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7859520', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '964adda23da9f7b057cb34fdc0b82aeb7ed65324'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'], ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 1858, 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b', 'sleep_before_teardown': 0, 'subset': '29813/100000', 'suite': 'rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '964adda23da9f7b057cb34fdc0b82aeb7ed65324', 'targets': {'smithi026.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL5TalKRuBrQ5q6YCrqfV8a84+uYIIKsxmnp5aeu29gxTSCzm1Gv9o8FNYbLwIsXEa4cVWlB3Hg+6ULekUeBuCA=', 'smithi104.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEUnLr0e9XB35qXcUAjFHGzTgLk1gtcI8PNKubiwJvbNn0fsFnyq38OLxh1sF2A50bNBvY+h5i2mAWsf7Cgz0b8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '67db276914d29505c9707d04f3a3b8fb9d8aee4b', 'timestamp': '2024-08-18_20:00:17', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-19T12:37:39.143 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-19T12:37:39.170 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-19T12:37:39.188 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/run/ceph 2024-08-19T12:37:39.226 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/run/ceph 2024-08-19T12:37:39.252 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:37:39.252 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-08-19T12:37:39.303 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-19T12:37:39.303 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 2024-08-19T12:37:39.357 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-19T12:37:39.358 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T12:37:39.358 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-08-19T12:37:39.358 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T12:37:39.358 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T12:37:39.358 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-08-19 12:37:34.578463547 +0000 2024-08-19T12:37:39.359 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-08-19 12:35:05.696998391 +0000 2024-08-19T12:37:39.359 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-08-19 12:35:05.696998391 +0000 2024-08-19T12:37:39.359 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-08-19 12:35:05.696998391 +0000 2024-08-19T12:37:39.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-19T12:37:39.420 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-08-19T12:37:39.420 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-08-19T12:37:39.420 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 9.9078e-05 s, 5.2 MB/s 2024-08-19T12:37:39.422 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-19T12:37:39.478 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 2024-08-19T12:37:39.532 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-19T12:37:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T12:37:39.533 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-08-19T12:37:39.533 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T12:37:39.533 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T12:37:39.533 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-08-19 12:37:34.578463547 +0000 2024-08-19T12:37:39.533 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-08-19 12:35:06.022993571 +0000 2024-08-19T12:37:39.533 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-08-19 12:35:06.022993571 +0000 2024-08-19T12:37:39.533 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-08-19 12:35:06.022993571 +0000 2024-08-19T12:37:39.533 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-19T12:37:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-08-19T12:37:39.595 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-08-19T12:37:39.596 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 8.8266e-05 s, 5.8 MB/s 2024-08-19T12:37:39.597 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-19T12:37:39.652 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 2024-08-19T12:37:39.707 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-19T12:37:39.708 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T12:37:39.708 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-08-19T12:37:39.708 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T12:37:39.708 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T12:37:39.708 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-08-19 12:37:34.579463532 +0000 2024-08-19T12:37:39.708 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-08-19 12:35:06.332988987 +0000 2024-08-19T12:37:39.708 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-08-19 12:35:06.332988987 +0000 2024-08-19T12:37:39.708 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-08-19 12:35:06.332988987 +0000 2024-08-19T12:37:39.708 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-19T12:37:39.770 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-08-19T12:37:39.771 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-08-19T12:37:39.771 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 9.9654e-05 s, 5.1 MB/s 2024-08-19T12:37:39.772 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-19T12:37:39.827 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 2024-08-19T12:37:39.882 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-19T12:37:39.883 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T12:37:39.883 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-08-19T12:37:39.883 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T12:37:39.883 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T12:37:39.883 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-08-19 12:37:34.579463532 +0000 2024-08-19T12:37:39.883 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-08-19 12:35:06.649984300 +0000 2024-08-19T12:37:39.883 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-08-19 12:35:06.649984300 +0000 2024-08-19T12:37:39.883 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-08-19 12:35:06.649984300 +0000 2024-08-19T12:37:39.884 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-19T12:37:39.945 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-08-19T12:37:39.945 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-08-19T12:37:39.945 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 9.3384e-05 s, 5.5 MB/s 2024-08-19T12:37:39.946 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-19T12:37:40.003 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-19T12:37:40.003 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:37:40.003 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-08-19T12:37:40.021 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-19T12:37:40.022 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-08-19T12:37:40.080 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-19T12:37:40.080 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T12:37:40.080 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-08-19T12:37:40.080 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T12:37:40.080 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T12:37:40.081 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-08-19 12:37:34.658253356 +0000 2024-08-19T12:37:40.081 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-08-19 12:35:05.642532772 +0000 2024-08-19T12:37:40.081 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-08-19 12:35:05.642532772 +0000 2024-08-19T12:37:40.081 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-08-19 12:35:05.642532772 +0000 2024-08-19T12:37:40.081 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-19T12:37:40.145 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-08-19T12:37:40.145 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-08-19T12:37:40.145 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000218425 s, 2.3 MB/s 2024-08-19T12:37:40.147 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-19T12:37:40.204 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-08-19T12:37:40.261 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-19T12:37:40.262 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T12:37:40.262 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-08-19T12:37:40.262 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T12:37:40.262 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T12:37:40.262 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-08-19 12:37:34.658253356 +0000 2024-08-19T12:37:40.262 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-08-19 12:35:05.979543545 +0000 2024-08-19T12:37:40.262 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-08-19 12:35:05.979543545 +0000 2024-08-19T12:37:40.262 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-08-19 12:35:05.979543545 +0000 2024-08-19T12:37:40.263 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-19T12:37:40.326 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-08-19T12:37:40.326 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-08-19T12:37:40.326 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000230292 s, 2.2 MB/s 2024-08-19T12:37:40.327 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-19T12:37:40.384 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-08-19T12:37:40.440 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-19T12:37:40.440 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T12:37:40.440 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-08-19T12:37:40.441 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T12:37:40.441 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T12:37:40.441 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-08-19 12:37:34.658253356 +0000 2024-08-19T12:37:40.441 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-08-19 12:35:06.307554030 +0000 2024-08-19T12:37:40.441 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-08-19 12:35:06.307554030 +0000 2024-08-19T12:37:40.441 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-08-19 12:35:06.307554030 +0000 2024-08-19T12:37:40.441 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-19T12:37:40.505 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-08-19T12:37:40.505 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-08-19T12:37:40.505 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000205356 s, 2.5 MB/s 2024-08-19T12:37:40.507 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-19T12:37:40.566 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-08-19T12:37:40.623 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-19T12:37:40.624 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-19T12:37:40.624 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-08-19T12:37:40.624 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-19T12:37:40.624 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-08-19T12:37:40.624 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-08-19 12:37:34.659253346 +0000 2024-08-19T12:37:40.624 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-08-19 12:35:06.645564834 +0000 2024-08-19T12:37:40.624 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-08-19 12:35:06.645564834 +0000 2024-08-19T12:37:40.624 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-08-19 12:35:06.645564834 +0000 2024-08-19T12:37:40.624 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-19T12:37:40.689 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-08-19T12:37:40.689 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-08-19T12:37:40.689 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000187834 s, 2.7 MB/s 2024-08-19T12:37:40.690 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-19T12:37:40.748 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-08-19T12:37:40.748 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-08-19T12:37:40.749 INFO:tasks.ceph:Generating config... 2024-08-19T12:37:40.750 INFO:tasks.ceph:[client] debug monc = 20 2024-08-19T12:37:40.750 INFO:tasks.ceph:[client] debug ms = 1 2024-08-19T12:37:40.750 INFO:tasks.ceph:[global] debug ms = 1 2024-08-19T12:37:40.750 INFO:tasks.ceph:[global] debug objecter = 20 2024-08-19T12:37:40.750 INFO:tasks.ceph:[global] debug rados = 20 2024-08-19T12:37:40.750 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-08-19T12:37:40.750 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-08-19T12:37:40.750 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-08-19T12:37:40.750 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-08-19T12:37:40.751 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-08-19T12:37:40.751 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-08-19T12:37:40.751 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-08-19T12:37:40.751 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-08-19T12:37:40.751 INFO:tasks.ceph:[global] ms type = async 2024-08-19T12:37:40.751 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-19T12:37:40.751 INFO:tasks.ceph:[mgr] debug monc = 10 2024-08-19T12:37:40.751 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-19T12:37:40.751 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-19T12:37:40.751 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-19T12:37:40.751 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-19T12:37:40.751 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-08-19T12:37:40.751 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-08-19T12:37:40.751 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-08-19T12:37:40.752 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-08-19T12:37:40.752 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-08-19T12:37:40.752 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-08-19T12:37:40.752 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-08-19T12:37:40.752 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-08-19T12:37:40.752 INFO:tasks.ceph:[mon] paxos min = 10 2024-08-19T12:37:40.752 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-08-19T12:37:40.752 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-08-19T12:37:40.752 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-08-19T12:37:40.752 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-08-19T12:37:40.752 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-08-19T12:37:40.752 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-08-19T12:37:40.752 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-08-19T12:37:40.753 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-08-19T12:37:40.753 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-08-19T12:37:40.753 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-08-19T12:37:40.753 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-19T12:37:40.753 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-19T12:37:40.753 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-08-19T12:37:40.753 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-08-19T12:37:40.753 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-08-19T12:37:40.753 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-08-19T12:37:40.753 INFO:tasks.ceph:[osd] osd class default list = * 2024-08-19T12:37:40.753 INFO:tasks.ceph:[osd] osd class load list = * 2024-08-19T12:37:40.754 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-08-19T12:37:40.754 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-08-19T12:37:40.754 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-08-19T12:37:40.754 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-08-19T12:37:40.754 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-08-19T12:37:40.754 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-08-19T12:37:40.754 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-08-19T12:37:40.754 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-08-19T12:37:40.754 INFO:tasks.ceph:Setting up mon.a... 2024-08-19T12:37:40.754 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-08-19T12:37:40.828 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/ceph.keyring 2024-08-19T12:37:40.831 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-08-19T12:37:40.875 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-19T12:37:40.941 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.26:6789]'), ('mon.b', '[v1:172.21.15.26:6790]'), ('mon.c', '[v1:172.21.15.26:6791]'), ('mon.d', '[v1:172.21.15.26:6792]'), ('mon.e', '[v1:172.21.15.26:6793]'), ('mon.f', '[v1:172.21.15.104:6789]'), ('mon.g', '[v1:172.21.15.104:6790]'), ('mon.h', '[v1:172.21.15.104:6791]'), ('mon.i', '[v1:172.21.15.104:6792]')] 2024-08-19T12:37:40.942 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.26:6789],[v1:172.21.15.26:6790],[v1:172.21.15.26:6791],[v1:172.21.15.26:6792],[v1:172.21.15.26:6793],[v1:172.21.15.104:6789],[v1:172.21.15.104:6790],[v1:172.21.15.104:6791],[v1:172.21.15.104:6792]', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mon.d': {}, 'mon.e': {}, 'mon.f': {}, 'mon.g': {}, 'mon.h': {}, 'mon.i': {}} 2024-08-19T12:37:40.942 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:37:40.943 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-19T12:37:40.997 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.26:6789]' --addv b '[v1:172.21.15.26:6790]' --addv c '[v1:172.21.15.26:6791]' --addv d '[v1:172.21.15.26:6792]' --addv e '[v1:172.21.15.26:6793]' --addv f '[v1:172.21.15.104:6789]' --addv g '[v1:172.21.15.104:6790]' --addv h '[v1:172.21.15.104:6791]' --addv i '[v1:172.21.15.104:6792]' --print /home/ubuntu/cephtest/ceph.monmap 2024-08-19T12:37:41.103 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-08-19T12:37:41.103 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: generated fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:37:41.104 INFO:teuthology.orchestra.run.smithi026.stdout:setting min_mon_release = quincy 2024-08-19T12:37:41.104 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 0 2024-08-19T12:37:41.104 INFO:teuthology.orchestra.run.smithi026.stdout:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:37:41.104 INFO:teuthology.orchestra.run.smithi026.stdout:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:37:41.104 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:37:41.104 INFO:teuthology.orchestra.run.smithi026.stdout:min_mon_release 17 (quincy) 2024-08-19T12:37:41.104 INFO:teuthology.orchestra.run.smithi026.stdout:election_strategy: 1 2024-08-19T12:37:41.104 INFO:teuthology.orchestra.run.smithi026.stdout:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:37:41.104 INFO:teuthology.orchestra.run.smithi026.stdout:1: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:37:41.104 INFO:teuthology.orchestra.run.smithi026.stdout:2: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:37:41.104 INFO:teuthology.orchestra.run.smithi026.stdout:3: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:37:41.105 INFO:teuthology.orchestra.run.smithi026.stdout:4: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:37:41.105 INFO:teuthology.orchestra.run.smithi026.stdout:5: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:37:41.105 INFO:teuthology.orchestra.run.smithi026.stdout:6: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:37:41.105 INFO:teuthology.orchestra.run.smithi026.stdout:7: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:37:41.105 INFO:teuthology.orchestra.run.smithi026.stdout:8: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:37:41.105 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (9 monitors) 2024-08-19T12:37:41.106 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-19T12:37:41.121 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 10d02fe0-13d8-499a-99b3-d3a725cba366... 2024-08-19T12:37:41.122 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-19T12:37:41.164 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-19T12:37:41.204 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-08-19T12:37:41.204 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2024-08-19T12:37:41.204 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-19T12:37:41.204 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2024-08-19T12:37:41.204 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.204 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2024-08-19T12:37:41.204 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.204 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2024-08-19T12:37:41.204 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.205 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2024-08-19T12:37:41.205 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2024-08-19T12:37:41.205 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.205 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2024-08-19T12:37:41.205 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2024-08-19T12:37:41.205 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.205 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-19T12:37:41.205 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2024-08-19T12:37:41.205 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.205 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2024-08-19T12:37:41.205 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-19T12:37:41.205 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.206 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2024-08-19T12:37:41.206 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.206 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2024-08-19T12:37:41.206 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2024-08-19T12:37:41.206 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2024-08-19T12:37:41.206 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2024-08-19T12:37:41.206 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2024-08-19T12:37:41.206 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2024-08-19T12:37:41.206 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2024-08-19T12:37:41.206 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-19T12:37:41.206 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-19T12:37:41.207 INFO:teuthology.orchestra.run.smithi026.stdout: mon_allow_pool_size_one = true 2024-08-19T12:37:41.207 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.207 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-19T12:37:41.207 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.207 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2024-08-19T12:37:41.207 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.207 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2024-08-19T12:37:41.207 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.207 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2024-08-19T12:37:41.207 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2024-08-19T12:37:41.208 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2024-08-19T12:37:41.208 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = "[v1:172.21.15.26:6789],[v1:172.21.15.26:6790],[v1:172.21.15.26:6791],[v1:172.21.15.26:6792],[v1:172.21.15.26:6793],[v1:172.21.15.104:6789],[v1:172.21.15.104:6790],[v1:172.21.15.104:6791],[v1:172.21.15.104:6792]" 2024-08-19T12:37:41.208 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-08-19T12:37:41.208 INFO:teuthology.orchestra.run.smithi026.stdout: debug objecter = 20 2024-08-19T12:37:41.208 INFO:teuthology.orchestra.run.smithi026.stdout: debug rados = 20 2024-08-19T12:37:41.208 INFO:teuthology.orchestra.run.smithi026.stdout: mon client directed command retry = 5 2024-08-19T12:37:41.208 INFO:teuthology.orchestra.run.smithi026.stdout: mon election default strategy = 1 2024-08-19T12:37:41.208 INFO:teuthology.orchestra.run.smithi026.stdout: ms bind msgr2 = False 2024-08-19T12:37:41.208 INFO:teuthology.orchestra.run.smithi026.stdout: ms inject delay max = 1 2024-08-19T12:37:41.208 INFO:teuthology.orchestra.run.smithi026.stdout: ms inject delay probability = 0.005 2024-08-19T12:37:41.208 INFO:teuthology.orchestra.run.smithi026.stdout: ms inject delay type = mon 2024-08-19T12:37:41.209 INFO:teuthology.orchestra.run.smithi026.stdout: ms inject internal delays = 0.002 2024-08-19T12:37:41.209 INFO:teuthology.orchestra.run.smithi026.stdout: ms inject socket failures = 2500 2024-08-19T12:37:41.209 INFO:teuthology.orchestra.run.smithi026.stdout: ms type = async 2024-08-19T12:37:41.209 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:37:41.209 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.209 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2024-08-19T12:37:41.209 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2024-08-19T12:37:41.209 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.209 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2024-08-19T12:37:41.209 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2024-08-19T12:37:41.209 INFO:teuthology.orchestra.run.smithi026.stdout: osd mclock profile = high_recovery_ops 2024-08-19T12:37:41.210 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.210 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2024-08-19T12:37:41.210 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2024-08-19T12:37:41.210 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.210 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2024-08-19T12:37:41.210 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2024-08-19T12:37:41.210 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2024-08-19T12:37:41.210 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug trim objects = true 2024-08-19T12:37:41.210 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.210 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2024-08-19T12:37:41.210 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2024-08-19T12:37:41.211 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.211 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2024-08-19T12:37:41.211 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.211 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2024-08-19T12:37:41.211 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.211 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2024-08-19T12:37:41.211 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.211 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2024-08-19T12:37:41.211 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2024-08-19T12:37:41.211 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.211 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2024-08-19T12:37:41.212 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2024-08-19T12:37:41.212 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore block size = 96636764160 2024-08-19T12:37:41.212 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore compression algorithm = zstd 2024-08-19T12:37:41.212 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore compression mode = aggressive 2024-08-19T12:37:41.212 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore fsck on mount = True 2024-08-19T12:37:41.212 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore write v2 random = True 2024-08-19T12:37:41.212 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore zero block detection = True 2024-08-19T12:37:41.212 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluefs = 20 2024-08-19T12:37:41.212 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluestore = 20 2024-08-19T12:37:41.212 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 20 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: debug rocksdb = 10 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd backfillfull_ratio = 0.85 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd full ratio = 0.9 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd nearfull ratio = 0.8 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: osd class default list = * 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: osd class load list = * 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify cached snaps = True 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify missing on start = True 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: osd failsafe full ratio = 0.95 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: osd mclock override recovery settings = True 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: osd objectstore = bluestore 2024-08-19T12:37:41.213 INFO:teuthology.orchestra.run.smithi026.stdout: osd op queue = debug_random 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout: osd op queue cut off = debug_random 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout: debug monc = 10 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.214 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: auth mon ticket ttl = 660 # 11m 2024-08-19T12:37:41.215 INFO:teuthology.orchestra.run.smithi026.stdout: auth service ticket ttl = 240 # 4m 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: # don't complain about insecure global_id in the test suite 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: # 1m isn't quite enough 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: mon_down_mkfs_grace = 2m 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_filestore_osds = false 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: mon min osdmap epochs = 25 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune interval = 2 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune min = 15 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune txsize = 2 2024-08-19T12:37:41.216 INFO:teuthology.orchestra.run.smithi026.stdout: mon scrub inject crc mismatch = 0.01 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: mon scrub inject missing keys = 0.05 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: mon scrub interval = 300 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on pool no app = False 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: paxos min = 10 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: paxos service trim min = 5 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: paxos trim min = 10 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-19T12:37:41.217 INFO:teuthology.orchestra.run.smithi026.stdout: debug monc = 20 2024-08-19T12:37:41.218 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-08-19T12:37:41.218 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2024-08-19T12:37:41.218 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.b] 2024-08-19T12:37:41.218 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.c] 2024-08-19T12:37:41.218 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.d] 2024-08-19T12:37:41.218 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.e] 2024-08-19T12:37:41.218 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.f] 2024-08-19T12:37:41.218 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.g] 2024-08-19T12:37:41.218 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.h] 2024-08-19T12:37:41.218 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.i] 2024-08-19T12:37:41.219 INFO:teuthology.orchestra.run.smithi104.stdout:[global] 2024-08-19T12:37:41.219 INFO:teuthology.orchestra.run.smithi104.stdout: chdir = "" 2024-08-19T12:37:41.219 INFO:teuthology.orchestra.run.smithi104.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-19T12:37:41.219 INFO:teuthology.orchestra.run.smithi104.stdout: auth supported = cephx 2024-08-19T12:37:41.219 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.219 INFO:teuthology.orchestra.run.smithi104.stdout: filestore xattr use omap = true 2024-08-19T12:37:41.219 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.219 INFO:teuthology.orchestra.run.smithi104.stdout: mon clock drift allowed = 1.000 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: osd crush chooseleaf type = 0 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: auth debug = true 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on old message = true 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on bug = true 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: mon pg warn max object skew = 0 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: # disable pg_autoscaler by default for new pools 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.220 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default size = 2 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow primary affinity = true 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow pg remap = true 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on legacy crush tunables = false 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on crush straw calc version zero = false 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on no sortbitwise = false 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on osd down out interval zero = false 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on too few osds = false 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: mon_allow_pool_size_one = true 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.221 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-19T12:37:41.222 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.222 INFO:teuthology.orchestra.run.smithi104.stdout: osd default data pool replay window = 5 2024-08-19T12:37:41.222 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.222 INFO:teuthology.orchestra.run.smithi104.stdout: mon allow pool delete = true 2024-08-19T12:37:41.222 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.222 INFO:teuthology.orchestra.run.smithi104.stdout: mon cluster log file level = debug 2024-08-19T12:37:41.222 INFO:teuthology.orchestra.run.smithi104.stdout: debug asserts on shutdown = true 2024-08-19T12:37:41.222 INFO:teuthology.orchestra.run.smithi104.stdout: mon health detail to clog = false 2024-08-19T12:37:41.222 INFO:teuthology.orchestra.run.smithi104.stdout: mon host = "[v1:172.21.15.26:6789],[v1:172.21.15.26:6790],[v1:172.21.15.26:6791],[v1:172.21.15.26:6792],[v1:172.21.15.26:6793],[v1:172.21.15.104:6789],[v1:172.21.15.104:6790],[v1:172.21.15.104:6791],[v1:172.21.15.104:6792]" 2024-08-19T12:37:41.222 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-08-19T12:37:41.222 INFO:teuthology.orchestra.run.smithi104.stdout: debug objecter = 20 2024-08-19T12:37:41.222 INFO:teuthology.orchestra.run.smithi104.stdout: debug rados = 20 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: mon client directed command retry = 5 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: mon election default strategy = 1 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: ms bind msgr2 = False 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject delay max = 1 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject delay probability = 0.005 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject delay type = mon 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject internal delays = 0.002 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject socket failures = 2500 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: ms type = async 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: fsid = 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout:[osd] 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: osd journal size = 100 2024-08-19T12:37:41.223 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub load threshold = 5.0 2024-08-19T12:37:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub max interval = 600 2024-08-19T12:37:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: osd mclock profile = high_recovery_ops 2024-08-19T12:37:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: osd recover clone overlap = true 2024-08-19T12:37:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: osd recovery max chunk = 1048576 2024-08-19T12:37:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug shutdown = true 2024-08-19T12:37:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug op order = true 2024-08-19T12:37:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify stray on activate = true 2024-08-19T12:37:41.224 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug trim objects = true 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: osd open classes on start = true 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug pg log writeout = true 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: osd deep scrub update digest min age = 30 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: osd map max advance = 10 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: journal zero on create = true 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: filestore ondisk finisher threads = 3 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: filestore apply finisher threads = 3 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.225 INFO:teuthology.orchestra.run.smithi104.stdout: bdev debug aio = true 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug misdirected ops = true 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore block size = 96636764160 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore compression algorithm = zstd 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore compression mode = aggressive 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore fsck on mount = True 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore write v2 random = True 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore zero block detection = True 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluefs = 20 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluestore = 20 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: debug osd = 20 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: debug rocksdb = 10 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd backfillfull_ratio = 0.85 2024-08-19T12:37:41.226 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd full ratio = 0.9 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd nearfull ratio = 0.8 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout: osd class default list = * 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout: osd class load list = * 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify cached snaps = True 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify missing on start = True 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout: osd failsafe full ratio = 0.95 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout: osd mclock override recovery settings = True 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout: osd objectstore = bluestore 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout: osd op queue = debug_random 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout: osd op queue cut off = debug_random 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr] 2024-08-19T12:37:41.227 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: debug mgr = 20 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: mgr/telemetry/nag = false 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: debug monc = 10 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout:[mon] 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: debug paxos = 20 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: mon data avail warn = 5 2024-08-19T12:37:41.228 INFO:teuthology.orchestra.run.smithi104.stdout: mon mgr mkfs grace = 240 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd reporter subtree level = osd 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd prime pg temp = true 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: auth mon ticket ttl = 660 # 11m 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: auth service ticket ttl = 240 # 4m 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: # don't complain about insecure global_id in the test suite 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.229 INFO:teuthology.orchestra.run.smithi104.stdout: # 1m isn't quite enough 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: mon_down_mkfs_grace = 2m 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_filestore_osds = false 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: mon min osdmap epochs = 25 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: mon osdmap full prune interval = 2 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: mon osdmap full prune min = 15 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: mon osdmap full prune txsize = 2 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: mon scrub inject crc mismatch = 0.01 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: mon scrub inject missing keys = 0.05 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: mon scrub interval = 300 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on pool no app = False 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: paxos min = 10 2024-08-19T12:37:41.230 INFO:teuthology.orchestra.run.smithi104.stdout: paxos service trim min = 5 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout: paxos trim min = 10 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout:[client] 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout: rgw cache enabled = true 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable ops log = true 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable usage log = true 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout: debug monc = 20 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.a] 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.b] 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.c] 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.d] 2024-08-19T12:37:41.231 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.e] 2024-08-19T12:37:41.232 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.f] 2024-08-19T12:37:41.232 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.g] 2024-08-19T12:37:41.232 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.h] 2024-08-19T12:37:41.232 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.i] 2024-08-19T12:37:41.233 INFO:tasks.ceph:Creating admin key on mon.a... 2024-08-19T12:37:41.233 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-08-19T12:37:41.327 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-19T12:37:41.327 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:37:41.327 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-08-19T12:37:41.342 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:37:41.342 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-08-19T12:37:41.397 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi026.front.sepia.ceph.com 2024-08-19T12:37:41.397 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:37:41.398 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-19T12:37:41.398 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-19T12:37:41.471 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:37:41.471 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-19T12:37:41.526 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi104.front.sepia.ceph.com 2024-08-19T12:37:41.526 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:37:41.526 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-19T12:37:41.526 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-19T12:37:41.566 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:37:41.566 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-19T12:37:41.623 INFO:tasks.ceph:Setting up mon nodes... 2024-08-19T12:37:41.623 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-19T12:37:41.623 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-08-19T12:37:41.728 INFO:teuthology.orchestra.run.smithi104.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-08-19T12:37:41.731 INFO:tasks.ceph:Setting up mds nodes... 2024-08-19T12:37:41.731 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-19T12:37:41.731 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-19T12:37:41.773 INFO:teuthology.orchestra.run.smithi104.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-08-19T12:37:41.786 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-19T12:37:41.786 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-08-19T12:37:41.787 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-08-19T12:37:41.812 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-19T12:37:41.812 INFO:tasks.ceph:role: osd.0 2024-08-19T12:37:41.812 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com 2024-08-19T12:37:41.812 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-19T12:37:41.876 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-19T12:37:41.876 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-19T12:37:41.877 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-19T12:37:41.877 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-19T12:37:41.877 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-19T12:37:41.877 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-08-19T12:37:41.877 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-19T12:37:41.877 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-19T12:37:41.877 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-19T12:37:41.877 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-19T12:37:41.909 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-08-19T12:37:41.911 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-08-19T12:37:41.911 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-08-19T12:37:41.979 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-08-19T12:37:42.047 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-08-19T12:37:42.113 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-19T12:37:42.113 INFO:tasks.ceph:role: osd.1 2024-08-19T12:37:42.113 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com 2024-08-19T12:37:42.113 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-19T12:37:42.178 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-19T12:37:42.178 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-19T12:37:42.178 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-19T12:37:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-19T12:37:42.179 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-19T12:37:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-08-19T12:37:42.179 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-19T12:37:42.179 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-19T12:37:42.179 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-19T12:37:42.179 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-19T12:37:42.212 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-08-19T12:37:42.213 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-08-19T12:37:42.214 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-08-19T12:37:42.283 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-08-19T12:37:42.351 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-08-19T12:37:42.416 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-19T12:37:42.416 INFO:tasks.ceph:role: osd.2 2024-08-19T12:37:42.416 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com 2024-08-19T12:37:42.417 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-19T12:37:42.481 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-19T12:37:42.482 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-19T12:37:42.482 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-19T12:37:42.482 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-19T12:37:42.482 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-19T12:37:42.482 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-08-19T12:37:42.482 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-19T12:37:42.482 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-19T12:37:42.482 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-19T12:37:42.482 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-19T12:37:42.514 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-08-19T12:37:42.516 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-08-19T12:37:42.517 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-08-19T12:37:42.586 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-08-19T12:37:42.654 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-19T12:37:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:37:42.748+0000 7f66a0a11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-08-19T12:37:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:37:42.749+0000 7f66a0a11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-08-19T12:37:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:37:42.749+0000 7f66a0a11640 -1 bdev(0x557c152a7800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-08-19T12:37:42.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:37:42.749+0000 7f66a0a11640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-08-19T12:37:46.411 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-19T12:37:46.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-19T12:37:46.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:37:46.531+0000 7f5f71955640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-08-19T12:37:46.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:37:46.531+0000 7f5f71955640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-08-19T12:37:46.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:37:46.531+0000 7f5f71955640 -1 bdev(0x560d660c7800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-08-19T12:37:46.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:37:46.531+0000 7f5f71955640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-08-19T12:37:49.445 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-19T12:37:49.474 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-19T12:37:49.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:37:49.563+0000 7f03da486640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-08-19T12:37:49.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:37:49.563+0000 7f03da486640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-08-19T12:37:49.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:37:49.563+0000 7f03da486640 -1 bdev(0x55a333b4f800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-08-19T12:37:49.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:37:49.563+0000 7f03da486640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-08-19T12:37:52.474 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-19T12:37:52.505 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-08-19T12:37:52.535 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-08-19T12:37:52.536 INFO:tasks.ceph:role: osd.3 2024-08-19T12:37:52.536 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com 2024-08-19T12:37:52.536 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-19T12:37:52.611 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-19T12:37:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-19T12:37:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-19T12:37:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-19T12:37:52.612 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-19T12:37:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-08-19T12:37:52.612 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-19T12:37:52.612 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-19T12:37:52.612 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-19T12:37:52.612 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-19T12:37:53.118 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-08-19T12:37:53.120 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-08-19T12:37:53.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-08-19T12:37:53.236 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-08-19T12:37:53.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-08-19T12:37:53.338 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-08-19T12:37:53.338 INFO:tasks.ceph:role: osd.4 2024-08-19T12:37:53.338 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com 2024-08-19T12:37:53.338 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-19T12:37:53.412 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-19T12:37:53.413 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-19T12:37:53.413 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-19T12:37:53.413 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-19T12:37:53.413 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-19T12:37:53.413 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-08-19T12:37:53.413 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-19T12:37:53.413 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-19T12:37:53.413 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-19T12:37:53.413 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-19T12:37:53.883 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-08-19T12:37:53.885 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-08-19T12:37:53.885 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-08-19T12:37:53.980 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-08-19T12:37:54.009 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-08-19T12:37:54.074 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-08-19T12:37:54.074 INFO:tasks.ceph:role: osd.5 2024-08-19T12:37:54.074 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com 2024-08-19T12:37:54.074 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-19T12:37:54.146 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-19T12:37:54.146 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-19T12:37:54.146 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-19T12:37:54.146 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-19T12:37:54.146 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-19T12:37:54.146 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-08-19T12:37:54.146 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-19T12:37:54.146 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-19T12:37:54.146 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-19T12:37:54.146 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-19T12:37:54.639 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-08-19T12:37:54.641 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-08-19T12:37:54.642 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-08-19T12:37:54.801 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-08-19T12:37:54.831 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-19T12:37:54.921 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:37:54.920+0000 7f5563613640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-08-19T12:37:54.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:37:54.923+0000 7f5563613640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-08-19T12:37:54.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:37:54.923+0000 7f5563613640 -1 bdev(0x557c25885800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-08-19T12:37:54.924 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:37:54.923+0000 7f5563613640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-08-19T12:37:58.085 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-19T12:37:58.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-19T12:37:58.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:37:58.203+0000 7fc573422640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-08-19T12:37:58.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:37:58.204+0000 7fc573422640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-08-19T12:37:58.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:37:58.204+0000 7fc573422640 -1 bdev(0x55733604b800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-08-19T12:37:58.205 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:37:58.204+0000 7fc573422640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-08-19T12:38:01.121 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-08-19T12:38:01.150 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-19T12:38:01.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:01.238+0000 7f90d54af640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-08-19T12:38:01.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:01.238+0000 7f90d54af640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-08-19T12:38:01.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:01.238+0000 7f90d54af640 -1 bdev(0x55f26d1a5800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-08-19T12:38:01.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:01.238+0000 7f90d54af640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-08-19T12:38:04.170 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-19T12:38:04.197 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-19T12:38:04.197 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:38:04.198 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-08-19T12:38:04.227 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:38:04.228 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-08-19T12:38:04.297 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:38:04.297 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-08-19T12:38:04.365 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:38:04.365 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-08-19T12:38:04.390 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:38:04.390 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-08-19T12:38:04.455 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:38:04.455 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-08-19T12:38:04.520 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:38:04.520 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-08-19T12:38:04.584 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:38:04.584 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-08-19T12:38:04.640 INFO:tasks.ceph:Adding keys to all mons... 2024-08-19T12:38:04.640 DEBUG:teuthology.orchestra.run.smithi026:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-19T12:38:04.642 DEBUG:teuthology.orchestra.run.smithi104:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-19T12:38:04.683 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2024-08-19T12:38:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCWPMNmyE6wLBAAbh1dSIx7MGsRc3HR1uMjCA== 2024-08-19T12:38:04.683 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2024-08-19T12:38:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCaPMNmyE62HxAA98U2zIYYn0BMAdpRhg3b6Q== 2024-08-19T12:38:04.683 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2024-08-19T12:38:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCdPMNmZ7CeIRAALvzrE2g0/FpDrpjSUO8TLA== 2024-08-19T12:38:04.683 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr.x] 2024-08-19T12:38:04.683 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCVPMNmWDhrKxAAzCdJx8daP60leZd5f0Cihw== 2024-08-19T12:38:04.683 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.3] 2024-08-19T12:38:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCiPMNmT9juNhAAyP4hAMIXuthgnBAIrGpI3g== 2024-08-19T12:38:04.684 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.4] 2024-08-19T12:38:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCmPMNm9DM2DBAAeKKN7Te7lz4V9oT1nyM5Pw== 2024-08-19T12:38:04.684 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.5] 2024-08-19T12:38:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCpPMNmoeFKDhAAzN533R0uUHcoT52qNfc0uw== 2024-08-19T12:38:04.684 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-08-19T12:38:04.684 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQCVPMNm99UgLhAA7caRG8aQFOf2ZXoLziXWCw== 2024-08-19T12:38:04.702 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.0] 2024-08-19T12:38:04.702 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCWPMNmyE6wLBAAbh1dSIx7MGsRc3HR1uMjCA== 2024-08-19T12:38:04.702 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.1] 2024-08-19T12:38:04.702 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCaPMNmyE62HxAA98U2zIYYn0BMAdpRhg3b6Q== 2024-08-19T12:38:04.702 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.2] 2024-08-19T12:38:04.703 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCdPMNmZ7CeIRAALvzrE2g0/FpDrpjSUO8TLA== 2024-08-19T12:38:04.703 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr.x] 2024-08-19T12:38:04.703 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCVPMNmWDhrKxAAzCdJx8daP60leZd5f0Cihw== 2024-08-19T12:38:04.703 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.3] 2024-08-19T12:38:04.703 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCiPMNmT9juNhAAyP4hAMIXuthgnBAIrGpI3g== 2024-08-19T12:38:04.703 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.4] 2024-08-19T12:38:04.703 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCmPMNm9DM2DBAAeKKN7Te7lz4V9oT1nyM5Pw== 2024-08-19T12:38:04.703 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.5] 2024-08-19T12:38:04.703 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCpPMNmoeFKDhAAzN533R0uUHcoT52qNfc0uw== 2024-08-19T12:38:04.703 INFO:teuthology.orchestra.run.smithi104.stdout:[client.0] 2024-08-19T12:38:04.703 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCVPMNm99UgLhAA7caRG8aQFOf2ZXoLziXWCw== 2024-08-19T12:38:04.704 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T12:38:04.728 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T12:38:04.795 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T12:38:04.797 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T12:38:04.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T12:38:04.850 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T12:38:04.901 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-19T12:38:04.904 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-19T12:38:04.955 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T12:38:04.957 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T12:38:05.008 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T12:38:05.010 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T12:38:05.061 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T12:38:05.063 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-19T12:38:05.113 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-19T12:38:05.116 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-19T12:38:05.169 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-19T12:38:05.170 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-08-19T12:38:05.194 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T12:38:05.291 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-08-19T12:38:05.316 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-08-19T12:38:05.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T12:38:05.475 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-08-19T12:38:05.499 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-08-19T12:38:05.564 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T12:38:05.659 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-08-19T12:38:05.683 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-d 2024-08-19T12:38:05.747 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i d --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T12:38:05.843 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-d 2024-08-19T12:38:05.867 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-e 2024-08-19T12:38:05.932 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i e --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T12:38:06.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-e 2024-08-19T12:38:06.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mon/ceph-f 2024-08-19T12:38:06.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i f --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T12:38:06.207 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-f 2024-08-19T12:38:06.234 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mon/ceph-g 2024-08-19T12:38:06.299 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i g --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T12:38:06.397 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-g 2024-08-19T12:38:06.422 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mon/ceph-h 2024-08-19T12:38:06.486 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i h --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T12:38:06.582 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-h 2024-08-19T12:38:06.608 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mon/ceph-i 2024-08-19T12:38:06.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i i --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-19T12:38:06.768 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-i 2024-08-19T12:38:06.793 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-19T12:38:06.795 DEBUG:teuthology.orchestra.run.smithi104:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-19T12:38:06.848 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-08-19T12:38:06.848 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-19T12:38:06.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-19T12:38:06.852 INFO:tasks.ceph.mon.a:Started 2024-08-19T12:38:06.852 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-19T12:38:06.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-08-19T12:38:06.854 INFO:tasks.ceph.mon.b:Started 2024-08-19T12:38:06.855 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-19T12:38:06.855 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-08-19T12:38:06.857 INFO:tasks.ceph.mon.c:Started 2024-08-19T12:38:06.857 INFO:tasks.ceph.mon.d:Restarting daemon 2024-08-19T12:38:06.857 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-08-19T12:38:06.859 INFO:tasks.ceph.mon.d:Started 2024-08-19T12:38:06.859 INFO:tasks.ceph.mon.e:Restarting daemon 2024-08-19T12:38:06.860 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-08-19T12:38:06.863 INFO:tasks.ceph.mon.e:Started 2024-08-19T12:38:06.863 INFO:tasks.ceph.mon.f:Restarting daemon 2024-08-19T12:38:06.863 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-08-19T12:38:06.890 INFO:tasks.ceph.mon.f:Started 2024-08-19T12:38:06.890 INFO:tasks.ceph.mon.g:Restarting daemon 2024-08-19T12:38:06.890 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-08-19T12:38:06.892 INFO:tasks.ceph.mon.g:Started 2024-08-19T12:38:06.893 INFO:tasks.ceph.mon.h:Restarting daemon 2024-08-19T12:38:06.893 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-08-19T12:38:06.895 INFO:tasks.ceph.mon.h:Started 2024-08-19T12:38:06.895 INFO:tasks.ceph.mon.i:Restarting daemon 2024-08-19T12:38:06.895 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-08-19T12:38:06.897 INFO:tasks.ceph.mon.i:Started 2024-08-19T12:38:06.897 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-08-19T12:38:06.897 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-08-19T12:38:06.897 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-08-19T12:38:06.900 INFO:tasks.ceph.mgr.x:Started 2024-08-19T12:38:06.900 DEBUG:tasks.ceph:set 0 configs 2024-08-19T12:38:06.900 INFO:tasks.ceph:Setting crush tunables to default 2024-08-19T12:38:06.900 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd crush tunables default 2024-08-19T12:38:07.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.029+0000 7fb24f70b640 10 monclient: get_monmap_and_config 2024-08-19T12:38:07.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.029+0000 7fb24f70b640 10 monclient: build_initial_monmap 2024-08-19T12:38:07.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.029+0000 7fb24f70b640 10 monclient: monmap: 2024-08-19T12:38:07.031 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:07.031 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:07.031 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:07.031502+0000 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:07.031502+0000 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:07.032 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:07.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.030+0000 7fb24f70b640 1 Processor -- start 2024-08-19T12:38:07.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.030+0000 7fb24f70b640 1 -- start start 2024-08-19T12:38:07.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.030+0000 7fb24f70b640 10 monclient: init 2024-08-19T12:38:07.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.030+0000 7fb24f70b640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:07.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.031+0000 7fb24f70b640 10 monclient: _add_conns ranks=[5,3,4,2,8,0,1,6,7] 2024-08-19T12:38:07.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.031+0000 7fb24f70b640 10 monclient(hunting): picked mon.noname-h con 0x7fb248073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:07.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.031+0000 7fb24f70b640 10 monclient(hunting): picked mon.noname-g con 0x7fb2480a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:07.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.031+0000 7fb24f70b640 10 monclient(hunting): picked mon.noname-c con 0x7fb24813f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:07.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.031+0000 7fb24f70b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb24809dae0 con 0x7fb2480a57a0 2024-08-19T12:38:07.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.031+0000 7fb24dca8640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fb24813f2e0 0x7fb2481416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:51058/0 (socket says 172.21.15.26:51058) 2024-08-19T12:38:07.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.031+0000 7fb24dca8640 1 -- 172.21.15.26:0/3598998427 learned_addr learned my addr 172.21.15.26:0/3598998427 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:07.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.031+0000 7fb24f70b640 1 -- 172.21.15.26:0/3598998427 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb248136910 con 0x7fb24813f2e0 2024-08-19T12:38:07.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.031+0000 7fb24f70b640 1 -- 172.21.15.26:0/3598998427 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb24809dd50 con 0x7fb248073470 2024-08-19T12:38:07.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.031+0000 7fb24f70b640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:07.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.031+0000 7fb24f70b640 10 monclient(hunting): authenticate will time out at 681.119352s 2024-08-19T12:38:07.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.033+0000 7fb24dca8640 1 -- 172.21.15.26:0/3598998427 >> v1:172.21.15.26:6791/0 conn(0x7fb24813f2e0 legacy=0x7fb2481416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:07.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.033+0000 7fb24cca6640 1 -- 172.21.15.26:0/3598998427 >> v1:172.21.15.104:6790/0 conn(0x7fb2480a57a0 legacy=0x7fb2480a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:07.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:07.033+0000 7fb24d4a7640 1 -- 172.21.15.26:0/3598998427 >> v1:172.21.15.104:6791/0 conn(0x7fb248073470 legacy=0x7fb248136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3224713341 0 0) 0x7fb24809dae0 con 0x7fb2480a57a0 2024-08-19T12:38:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 10 monclient(hunting): my global_id is 4107 2024-08-19T12:38:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb234003520 con 0x7fb2480a57a0 2024-08-19T12:38:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 350333106 0 0) 0x7fb24809dd50 con 0x7fb248073470 2024-08-19T12:38:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 10 monclient(hunting): my global_id is 4109 2024-08-19T12:38:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb24809dae0 con 0x7fb248073470 2024-08-19T12:38:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879402678 0 0) 0x7fb248136910 con 0x7fb24813f2e0 2024-08-19T12:38:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 10 monclient(hunting): my global_id is 4117 2024-08-19T12:38:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.858+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb24809dd50 con 0x7fb24813f2e0 2024-08-19T12:38:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4189498098 0 0) 0x7fb234003520 con 0x7fb2480a57a0 2024-08-19T12:38:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb24d4a7640 1 --1- 172.21.15.26:0/3598998427 >> v1:172.21.15.104:6791/0 conn(0x7fb248073470 0x7fb248136200 :-1 s=OPENED pgs=19 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fb2440019c0 mon_map magic: 0 2024-08-19T12:38:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 >> v1:172.21.15.104:6791/0 conn(0x7fb248073470 legacy=0x7fb248136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 >> v1:172.21.15.26:6791/0 conn(0x7fb24813f2e0 legacy=0x7fb2481416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 1 monclient: found mon.noname-g 2024-08-19T12:38:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:38:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24809e030 con 0x7fb2480a57a0 2024-08-19T12:38:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 10 monclient: _finish_auth 0 2024-08-19T12:38:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb238001b30 con 0x7fb2480a57a0 2024-08-19T12:38:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:38:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 10 monclient: dump: 2024-08-19T12:38:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb24f70b640 5 monclient: authenticate success, global_id 4107 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb24f70b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb238003610 con 0x7fb2480a57a0 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 1 -- 172.21.15.26:0/3598998427 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb238003b80 con 0x7fb2480a57a0 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb23ffff640 10 monclient: dump: 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:08.868 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb24f70b640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.859+0000 7fb24f70b640 4 set_mon_vals no callback set 2024-08-19T12:38:08.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.860+0000 7fb24f70b640 10 monclient: shutdown 2024-08-19T12:38:08.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.860+0000 7fb24f70b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:08.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.860+0000 7fb24f70b640 1 -- 172.21.15.26:0/3598998427 >> v1:172.21.15.104:6790/0 conn(0x7fb2480a57a0 legacy=0x7fb2480a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.860+0000 7fb24f70b640 1 -- 172.21.15.26:0/3598998427 shutdown_connections 2024-08-19T12:38:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.860+0000 7fb24f70b640 1 -- 172.21.15.26:0/3598998427 >> 172.21.15.26:0/3598998427 conn(0x7fb2480abff0 msgr2=0x7fb2480ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.860+0000 7fb24f70b640 1 -- 172.21.15.26:0/3598998427 shutdown_connections 2024-08-19T12:38:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.860+0000 7fb24f70b640 1 -- 172.21.15.26:0/3598998427 wait complete. 2024-08-19T12:38:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.861+0000 7fb24f70b640 10 monclient: build_initial_monmap 2024-08-19T12:38:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.861+0000 7fb24f70b640 10 monclient: monmap: 2024-08-19T12:38:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:08.874 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:08.875 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:08.875 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:08.875 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:08.875 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:08.875 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:08.875 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:08.875 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:08.875 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:08.875 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:08.876 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:08.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.862+0000 7fb24f70b640 1 librados: starting msgr at 2024-08-19T12:38:08.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.862+0000 7fb24f70b640 1 librados: starting objecter 2024-08-19T12:38:08.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.862+0000 7fb24f70b640 1 Processor -- start 2024-08-19T12:38:08.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.862+0000 7fb24f70b640 1 -- start start 2024-08-19T12:38:08.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.862+0000 7fb24f70b640 1 librados: setting wanted keys 2024-08-19T12:38:08.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.862+0000 7fb24f70b640 1 librados: calling monclient init 2024-08-19T12:38:08.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.862+0000 7fb24f70b640 10 monclient: init 2024-08-19T12:38:08.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.863+0000 7fb24f70b640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:08.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.863+0000 7fb24f70b640 10 monclient: _add_conns ranks=[3,4,7,2,1,5,8,0,6] 2024-08-19T12:38:08.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.863+0000 7fb24f70b640 10 monclient(hunting): picked mon.noname-g con 0x7fb248073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:08.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.863+0000 7fb24f70b640 10 monclient(hunting): picked mon.noname-c con 0x7fb2480a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.863+0000 7fb24f70b640 10 monclient(hunting): picked mon.noname-i con 0x7fb24813f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.863+0000 7fb24f70b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb248134f90 con 0x7fb248073470 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.863+0000 7fb24f70b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb248137730 con 0x7fb2480a57a0 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.863+0000 7fb24f70b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2480a3af0 con 0x7fb24813f2e0 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.863+0000 7fb24f70b640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.863+0000 7fb24f70b640 10 monclient(hunting): authenticate will time out at 682.951513s 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.863+0000 7fb24cca6640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fb2480a57a0 0x7fb2480a2760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:58654/0 (socket says 172.21.15.26:58654) 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.863+0000 7fb24cca6640 1 -- 172.21.15.26:0/2078571328 learned_addr learned my addr 172.21.15.26:0/2078571328 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.865+0000 7fb24d4a7640 1 -- 172.21.15.26:0/2078571328 >> v1:172.21.15.104:6790/0 conn(0x7fb248073470 legacy=0x7fb2480a2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.865+0000 7fb24cca6640 1 -- 172.21.15.26:0/2078571328 >> v1:172.21.15.26:6791/0 conn(0x7fb2480a57a0 legacy=0x7fb2480a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.865+0000 7fb23dffb640 10 client.?.objecter ms_handle_connect 0x7fb248073470 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.865+0000 7fb23dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:08.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.865+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.865+0000 7fb24dca8640 1 -- 172.21.15.26:0/2078571328 >> v1:172.21.15.104:6792/0 conn(0x7fb24813f2e0 legacy=0x7fb2480a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.867+0000 7fb23dffb640 10 client.?.objecter ms_handle_connect 0x7fb2480a57a0 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.867+0000 7fb23dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.867+0000 7fb23dffb640 10 client.?.objecter ms_handle_connect 0x7fb24813f2e0 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.867+0000 7fb23dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1972916604 0 0) 0x7fb2480a3af0 con 0x7fb24813f2e0 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): my global_id is 4120 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb22c004eb0 con 0x7fb24813f2e0 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1915264870 0 0) 0x7fb244001480 con 0x7fb248073470 2024-08-19T12:38:08.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): my global_id is 4116 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2480a3af0 con 0x7fb248073470 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 314257990 0 0) 0x7fb248137730 con 0x7fb2480a57a0 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): my global_id is 4135 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb244001480 con 0x7fb2480a57a0 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 672598973 0 0) 0x7fb22c004eb0 con 0x7fb24813f2e0 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb248137730 con 0x7fb24813f2e0 2024-08-19T12:38:08.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb240002020 con 0x7fb24813f2e0 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): dump: 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:08.880 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:08.881 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:08.881 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:08.881 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:08.881 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:08.881 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:08.881 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:08.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:08.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2174371050 0 0) 0x7fb2480a3af0 con 0x7fb248073470 2024-08-19T12:38:08.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb22c004eb0 con 0x7fb248073470 2024-08-19T12:38:08.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb244003950 con 0x7fb248073470 2024-08-19T12:38:08.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:08.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:08.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.868+0000 7fb23dffb640 10 monclient(hunting): dump: 2024-08-19T12:38:08.882 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:08.882 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:08.882 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:08.882 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:08.882 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:08.882 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:08.882 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:08.882 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:08.882 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1432873138 0 0) 0x7fb244001480 con 0x7fb2480a57a0 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2480a3af0 con 0x7fb2480a57a0 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2056549267 0 0) 0x7fb248137730 con 0x7fb24813f2e0 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 >> v1:172.21.15.26:6791/0 conn(0x7fb2480a57a0 legacy=0x7fb2480a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 >> v1:172.21.15.104:6790/0 conn(0x7fb248073470 legacy=0x7fb2480a2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:08.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 1 monclient: found mon.i 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb248136d50 con 0x7fb24813f2e0 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb24f70b640 5 monclient: authenticate success, global_id 4120 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb24f70b640 10 monclient: _renew_subs 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb24f70b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb24f70b640 1 -- 172.21.15.26:0/2078571328 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb24813ec60 con 0x7fb24813f2e0 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb240004e10 con 0x7fb24813f2e0 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb240005340 con 0x7fb24813f2e0 2024-08-19T12:38:08.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 10 monclient: dump: 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:08.885 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:08.886 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:08.886 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:08.886 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:08.886 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:08.886 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:08.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb240005c50 con 0x7fb24813f2e0 2024-08-19T12:38:08.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.869+0000 7fb23f7fe640 4 set_mon_vals no callback set 2024-08-19T12:38:08.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.870+0000 7fb24f70b640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.870+0000 7fb24f70b640 10 monclient: _renew_subs 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.870+0000 7fb24f70b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.870+0000 7fb24f70b640 1 -- 172.21.15.26:0/2078571328 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fb2481d7340 con 0x7fb24813f2e0 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.870+0000 7fb24f70b640 1 librados: init done 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.870+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1605025337 0 0) 0x7fb2400064b0 con 0x7fb24813f2e0 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.870+0000 7fb23dffb640 10 client.4120.objecter ms_dispatch 0x7fb248000b90 osd_map(1..1 src has 1..1) 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.870+0000 7fb23dffb640 3 client.4120.objecter handle_osd_map got epochs [1,1] > 0 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.870+0000 7fb23dffb640 3 client.4120.objecter handle_osd_map decoding full epoch 1 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.870+0000 7fb23dffb640 20 client.4120.objecter dump_active .. 0 homeless 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.871+0000 7fb24f70b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.871+0000 7fb24f70b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.871+0000 7fb24f70b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.871+0000 7fb24f70b640 1 -- 172.21.15.26:0/2078571328 --> v1:172.21.15.104:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2480008d0 con 0x7fb24813f2e0 2024-08-19T12:38:08.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.877+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.7 v1:172.21.15.104:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fb240006770 con 0x7fb24813f2e0 2024-08-19T12:38:08.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.877+0000 7fb23dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:08.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.877+0000 7fb23dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.945+0000 7fb24f70b640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-08-19T12:38:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.945+0000 7fb24f70b640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-08-19T12:38:08.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.945+0000 7fb24f70b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:08.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:08.945+0000 7fb24f70b640 1 -- 172.21.15.26:0/2078571328 --> v1:172.21.15.104:6792/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fb248142df0 con 0x7fb24813f2e0 2024-08-19T12:38:09.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.790+0000 7fb23dffb640 1 -- 172.21.15.26:0/2078571328 <== mon.7 v1:172.21.15.104:6792/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (unknown 2277712955 0 0) 0x7fb240007000 con 0x7fb24813f2e0 2024-08-19T12:38:09.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.790+0000 7fb23dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-08-19T12:38:09.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.790+0000 7fb23dffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-08-19T12:38:09.793 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2024-08-19T12:38:09.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.792+0000 7fb24f70b640 10 librados: watch_flush enter 2024-08-19T12:38:09.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.792+0000 7fb24f70b640 10 librados: watch_flush exit 2024-08-19T12:38:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.792+0000 7fb24f70b640 20 client.4120.objecter shutdown clearing up homeless session... 2024-08-19T12:38:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.792+0000 7fb24f70b640 10 client.4120.objecter successfully canceled tick 2024-08-19T12:38:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.792+0000 7fb24f70b640 10 monclient: shutdown 2024-08-19T12:38:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.792+0000 7fb24f70b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.792+0000 7fb24f70b640 1 -- 172.21.15.26:0/2078571328 >> v1:172.21.15.104:6792/0 conn(0x7fb24813f2e0 legacy=0x7fb2480a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.792+0000 7fb24f70b640 1 -- 172.21.15.26:0/2078571328 shutdown_connections 2024-08-19T12:38:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.792+0000 7fb24f70b640 1 -- 172.21.15.26:0/2078571328 >> 172.21.15.26:0/2078571328 conn(0x7fb2480abff0 msgr2=0x7fb2481338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.793+0000 7fb24f70b640 1 -- 172.21.15.26:0/2078571328 shutdown_connections 2024-08-19T12:38:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.793+0000 7fb24f70b640 1 -- 172.21.15.26:0/2078571328 wait complete. 2024-08-19T12:38:09.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:09.793+0000 7fb24f70b640 1 librados: shutdown 2024-08-19T12:38:09.806 INFO:tasks.ceph:check_enable_crimson: False 2024-08-19T12:38:09.807 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-08-19T12:38:09.807 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:38:09.807 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-08-19T12:38:09.835 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:38:09.835 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-08-19T12:38:09.901 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-08-19T12:38:09.901 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-08-19T12:38:09.969 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:38:09.969 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-08-19T12:38:09.995 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:38:09.995 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-08-19T12:38:10.059 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:38:10.059 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-08-19T12:38:10.125 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new ae69b71a-73f9-47e2-b81e-92e9cdd7df0b 0 2024-08-19T12:38:10.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.262+0000 7ffadf708640 10 monclient: get_monmap_and_config 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.262+0000 7ffadf708640 10 monclient: build_initial_monmap 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.262+0000 7ffadf708640 10 monclient: monmap: 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:38:10.263849+0000 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:38:10.263849+0000 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:10.264 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:10.265 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:10.265 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:10.265 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:10.265 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.263+0000 7ffadf708640 1 Processor -- start 2024-08-19T12:38:10.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.263+0000 7ffadf708640 1 -- start start 2024-08-19T12:38:10.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.263+0000 7ffadf708640 10 monclient: init 2024-08-19T12:38:10.266 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.264+0000 7ffadf708640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.264+0000 7ffadf708640 10 monclient: _add_conns ranks=[6,2,5,3,4,0,1,7,8] 2024-08-19T12:38:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.264+0000 7ffadf708640 10 monclient(hunting): picked mon.noname-d con 0x7ffad8073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.264+0000 7ffadf708640 10 monclient(hunting): picked mon.noname-b con 0x7ffad80a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.264+0000 7ffadf708640 10 monclient(hunting): picked mon.noname-h con 0x7ffad813f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.264+0000 7ffadf708640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad809dae0 con 0x7ffad80a57a0 2024-08-19T12:38:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.264+0000 7ffadf708640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad8136910 con 0x7ffad813f2e0 2024-08-19T12:38:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.264+0000 7ffadf708640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad809dd50 con 0x7ffad8073470 2024-08-19T12:38:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.264+0000 7ffadf708640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.264+0000 7ffadf708640 10 monclient(hunting): authenticate will time out at 654.293242s 2024-08-19T12:38:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.264+0000 7ffaddca5640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7ffad813f2e0 0x7ffad81416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.104:51296/0 (socket says 172.21.15.104:51296) 2024-08-19T12:38:10.267 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.264+0000 7ffaddca5640 1 -- 172.21.15.104:0/3447640315 learned_addr learned my addr 172.21.15.104:0/3447640315 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:38:10.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffaddca5640 1 -- 172.21.15.104:0/3447640315 >> v1:172.21.15.104:6791/0 conn(0x7ffad813f2e0 legacy=0x7ffad81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:10.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 1 -- 172.21.15.104:0/3447640315 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1940694376 0 0) 0x7ffad8136910 con 0x7ffad813f2e0 2024-08-19T12:38:10.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:10.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:10.275 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:10.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient(hunting): my global_id is 4118 2024-08-19T12:38:10.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 1 -- 172.21.15.104:0/3447640315 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffac4003520 con 0x7ffad813f2e0 2024-08-19T12:38:10.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffadd4a4640 1 -- 172.21.15.104:0/3447640315 >> v1:172.21.15.26:6792/0 conn(0x7ffad8073470 legacy=0x7ffad8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:10.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffadcca3640 1 -- 172.21.15.104:0/3447640315 >> v1:172.21.15.26:6790/0 conn(0x7ffad80a57a0 legacy=0x7ffad80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:10.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 1 -- 172.21.15.104:0/3447640315 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4003238756 0 0) 0x7ffac4003520 con 0x7ffad813f2e0 2024-08-19T12:38:10.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 1 -- 172.21.15.104:0/3447640315 >> v1:172.21.15.26:6792/0 conn(0x7ffad8073470 legacy=0x7ffad8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:10.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 1 -- 172.21.15.104:0/3447640315 >> v1:172.21.15.26:6790/0 conn(0x7ffad80a57a0 legacy=0x7ffad80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:10.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:10.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 1 monclient: found mon.noname-h 2024-08-19T12:38:10.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:10.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:38:10.276 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 1 -- 172.21.15.104:0/3447640315 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad809e030 con 0x7ffad813f2e0 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient: _finish_auth 0 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 1 -- 172.21.15.104:0/3447640315 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffad0001e60 con 0x7ffad813f2e0 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffadf708640 5 monclient: authenticate success, global_id 4118 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffadf708640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient: dump: 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.277 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:10.278 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:10.278 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffadf708640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 1 -- 172.21.15.104:0/3447640315 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffad0004c20 con 0x7ffad813f2e0 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffadf708640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 1 -- 172.21.15.104:0/3447640315 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffad0005190 con 0x7ffad813f2e0 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffacbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.266+0000 7ffadf708640 4 set_mon_vals no callback set 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.267+0000 7ffacbfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.267+0000 7ffacbfff640 10 monclient: dump: 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.279 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.267+0000 7ffadf708640 10 monclient: shutdown 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.267+0000 7ffadf708640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:10.280 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.267+0000 7ffadf708640 1 -- 172.21.15.104:0/3447640315 >> v1:172.21.15.104:6791/0 conn(0x7ffad813f2e0 legacy=0x7ffad81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.267+0000 7ffadf708640 1 -- 172.21.15.104:0/3447640315 shutdown_connections 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.267+0000 7ffadf708640 1 -- 172.21.15.104:0/3447640315 >> 172.21.15.104:0/3447640315 conn(0x7ffad80abff0 msgr2=0x7ffad80ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.267+0000 7ffadf708640 1 -- 172.21.15.104:0/3447640315 shutdown_connections 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.267+0000 7ffadf708640 1 -- 172.21.15.104:0/3447640315 wait complete. 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.268+0000 7ffadf708640 10 monclient: build_initial_monmap 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.268+0000 7ffadf708640 10 monclient: monmap: 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:10.281 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.268+0000 7ffadf708640 1 librados: starting msgr at 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.268+0000 7ffadf708640 1 librados: starting objecter 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.268+0000 7ffadf708640 1 Processor -- start 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.268+0000 7ffadf708640 1 -- start start 2024-08-19T12:38:10.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.268+0000 7ffadf708640 1 librados: setting wanted keys 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.268+0000 7ffadf708640 1 librados: calling monclient init 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.268+0000 7ffadf708640 10 monclient: init 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.269+0000 7ffadf708640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.269+0000 7ffadf708640 10 monclient: _add_conns ranks=[3,7,1,5,2,6,4,0,8] 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.269+0000 7ffadf708640 10 monclient(hunting): picked mon.noname-g con 0x7ffad8073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.269+0000 7ffadf708640 10 monclient(hunting): picked mon.noname-i con 0x7ffad80a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.269+0000 7ffadf708640 10 monclient(hunting): picked mon.noname-f con 0x7ffad813f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.269+0000 7ffadf708640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad80a3a70 con 0x7ffad813f2e0 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.269+0000 7ffadf708640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad8135400 con 0x7ffad8073470 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.269+0000 7ffadd4a4640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7ffad8073470 0x7ffad80a2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:56806/0 (socket says 172.21.15.104:56806) 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.269+0000 7ffadd4a4640 1 -- 172.21.15.104:0/1061992765 learned_addr learned my addr 172.21.15.104:0/1061992765 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.269+0000 7ffadf708640 1 -- 172.21.15.104:0/1061992765 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad81350b0 con 0x7ffad80a57a0 2024-08-19T12:38:10.283 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.269+0000 7ffadf708640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.269+0000 7ffadf708640 10 monclient(hunting): authenticate will time out at 654.298298s 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffadd4a4640 1 -- 172.21.15.104:0/1061992765 >> v1:172.21.15.104:6790/0 conn(0x7ffad8073470 legacy=0x7ffad80a2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 client.?.objecter ms_handle_connect 0x7ffad8073470 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 client.?.objecter ms_handle_connect 0x7ffad813f2e0 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffaddca5640 1 -- 172.21.15.104:0/1061992765 >> v1:172.21.15.104:6789/0 conn(0x7ffad813f2e0 legacy=0x7ffad81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 client.?.objecter ms_handle_connect 0x7ffad80a57a0 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffadcca3640 1 -- 172.21.15.104:0/1061992765 >> v1:172.21.15.104:6792/0 conn(0x7ffad80a57a0 legacy=0x7ffad80a2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2595531502 0 0) 0x7ffad8135400 con 0x7ffad8073470 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:10.284 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 monclient(hunting): my global_id is 4125 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffabc004eb0 con 0x7ffad8073470 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffaddca5640 1 --1- 172.21.15.104:0/1061992765 >> v1:172.21.15.104:6789/0 conn(0x7ffad813f2e0 0x7ffad81d6150 :-1 s=OPENED pgs=26 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7ffad80a3a70 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 640540405 0 0) 0x7ffad81350b0 con 0x7ffad80a57a0 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 monclient(hunting): my global_id is 4129 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffad8135400 con 0x7ffad80a57a0 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3830535311 0 0) 0x7ffad8135400 con 0x7ffad80a57a0 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffad81350b0 con 0x7ffad80a57a0 2024-08-19T12:38:10.285 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffad4003610 con 0x7ffad80a57a0 2024-08-19T12:38:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:38:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:10.286 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1997623294 0 0) 0x7ffabc004eb0 con 0x7ffad8073470 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.271+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffad8135400 con 0x7ffad8073470 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffacc003610 con 0x7ffad8073470 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:10.287 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:10.288 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2054160832 0 0) 0x7ffad81350b0 con 0x7ffad80a57a0 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 >> v1:172.21.15.104:6790/0 conn(0x7ffad8073470 legacy=0x7ffad80a2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 >> v1:172.21.15.104:6789/0 conn(0x7ffad813f2e0 legacy=0x7ffad81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 1 monclient: found mon.i 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad813edb0 con 0x7ffad80a57a0 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:10.289 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1588086216 0 0) 0x7ffad80a3a70 con 0x7ffad813f2e0 2024-08-19T12:38:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:38:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffadf708640 5 monclient: authenticate success, global_id 4129 2024-08-19T12:38:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffadf708640 10 monclient: _renew_subs 2024-08-19T12:38:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffadf708640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffadf708640 1 -- 172.21.15.104:0/1061992765 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffad8137210 con 0x7ffad80a57a0 2024-08-19T12:38:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffad4003b10 con 0x7ffad80a57a0 2024-08-19T12:38:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffad4004080 con 0x7ffad80a57a0 2024-08-19T12:38:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:10.290 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 monclient: dump: 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:10.291 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffacb7fe640 4 set_mon_vals no callback set 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffadf708640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffadf708640 10 monclient: _renew_subs 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffadf708640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffadf708640 1 -- 172.21.15.104:0/1061992765 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7ffad813e980 con 0x7ffad80a57a0 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7ffad4004320 con 0x7ffad80a57a0 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffadf708640 1 librados: init done 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (unknown 1041168815 0 0) 0x7ffad4005e90 con 0x7ffad80a57a0 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 10 client.4129.objecter ms_dispatch 0x7ffad8000b90 osd_map(2..2 src has 1..2) 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 3 client.4129.objecter handle_osd_map got epochs [2,2] > 0 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 3 client.4129.objecter handle_osd_map decoding full epoch 2 2024-08-19T12:38:10.292 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.272+0000 7ffac9ffb640 20 client.4129.objecter dump_active .. 0 homeless 2024-08-19T12:38:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.273+0000 7ffadf708640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.273+0000 7ffadf708640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.273+0000 7ffadf708640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.273+0000 7ffadf708640 1 -- 172.21.15.104:0/1061992765 --> v1:172.21.15.104:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad80008d0 con 0x7ffad80a57a0 2024-08-19T12:38:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.276+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.7 v1:172.21.15.104:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7ffad4006090 con 0x7ffad80a57a0 2024-08-19T12:38:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.276+0000 7ffac9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:10.293 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.276+0000 7ffac9ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:10.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.347+0000 7ffadf708640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ae69b71a-73f9-47e2-b81e-92e9cdd7df0b", "id": 0}] 2024-08-19T12:38:10.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.347+0000 7ffadf708640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ae69b71a-73f9-47e2-b81e-92e9cdd7df0b", "id": 0}] 2024-08-19T12:38:10.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.347+0000 7ffadf708640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:10.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.347+0000 7ffadf708640 1 -- 172.21.15.104:0/1061992765 --> v1:172.21.15.104:6792/0 -- mon_command({"prefix": "osd new", "uuid": "ae69b71a-73f9-47e2-b81e-92e9cdd7df0b", "id": 0} v 0) -- 0x7ffad8137630 con 0x7ffad80a57a0 2024-08-19T12:38:10.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.841+0000 7ffac9ffb640 1 -- 172.21.15.104:0/1061992765 <== mon.7 v1:172.21.15.104:6792/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ae69b71a-73f9-47e2-b81e-92e9cdd7df0b", "id": 0}]=0 v3) ==== 112+0+2 (unknown 1300875902 0 732707515) 0x7ffad4006270 con 0x7ffad80a57a0 2024-08-19T12:38:10.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.841+0000 7ffac9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ae69b71a-73f9-47e2-b81e-92e9cdd7df0b", "id": 0}] 2024-08-19T12:38:10.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.841+0000 7ffac9ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:10.843 INFO:teuthology.orchestra.run.smithi104.stdout:0 2024-08-19T12:38:10.843 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.842+0000 7ffadf708640 10 librados: watch_flush enter 2024-08-19T12:38:10.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.842+0000 7ffadf708640 10 librados: watch_flush exit 2024-08-19T12:38:10.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.842+0000 7ffadf708640 20 client.4129.objecter shutdown clearing up homeless session... 2024-08-19T12:38:10.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.842+0000 7ffadf708640 10 client.4129.objecter successfully canceled tick 2024-08-19T12:38:10.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.842+0000 7ffadf708640 10 monclient: shutdown 2024-08-19T12:38:10.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.842+0000 7ffadf708640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:10.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.842+0000 7ffadf708640 1 -- 172.21.15.104:0/1061992765 >> v1:172.21.15.104:6792/0 conn(0x7ffad80a57a0 legacy=0x7ffad80a2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:10.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.842+0000 7ffadf708640 1 -- 172.21.15.104:0/1061992765 shutdown_connections 2024-08-19T12:38:10.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.842+0000 7ffadf708640 1 -- 172.21.15.104:0/1061992765 >> 172.21.15.104:0/1061992765 conn(0x7ffad80abff0 msgr2=0x7ffad8133a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:10.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.842+0000 7ffadf708640 1 -- 172.21.15.104:0/1061992765 shutdown_connections 2024-08-19T12:38:10.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.842+0000 7ffadf708640 1 -- 172.21.15.104:0/1061992765 wait complete. 2024-08-19T12:38:10.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.842+0000 7ffadf708640 1 librados: shutdown 2024-08-19T12:38:10.854 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new fc6dcabb-211c-484a-b215-2ef5fe1cbd21 1 2024-08-19T12:38:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.951+0000 7fdb4ea56640 10 monclient: get_monmap_and_config 2024-08-19T12:38:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.951+0000 7fdb4ea56640 10 monclient: build_initial_monmap 2024-08-19T12:38:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.951+0000 7fdb4ea56640 10 monclient: monmap: 2024-08-19T12:38:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-08-19T12:38:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:38:10.952651+0000 2024-08-19T12:38:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:38:10.952651+0000 2024-08-19T12:38:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.955 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 1 Processor -- start 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 1 -- start start 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 10 monclient: init 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:10.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 10 monclient: _add_conns ranks=[2,0,5,1,7,3,4,8,6] 2024-08-19T12:38:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 10 monclient(hunting): picked mon.noname-b con 0x7fdb48073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 10 monclient(hunting): picked mon.noname-a con 0x7fdb480a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 10 monclient(hunting): picked mon.noname-h con 0x7fdb4813f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb4809dae0 con 0x7fdb480a57a0 2024-08-19T12:38:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb48136910 con 0x7fdb48073470 2024-08-19T12:38:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb4809dd50 con 0x7fdb4813f2e0 2024-08-19T12:38:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.952+0000 7fdb4ea56640 10 monclient(hunting): authenticate will time out at 654.981958s 2024-08-19T12:38:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.953+0000 7fdb4cff3640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7fdb4813f2e0 0x7fdb481416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.104:51298/0 (socket says 172.21.15.104:51298) 2024-08-19T12:38:10.957 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.953+0000 7fdb4cff3640 1 -- 172.21.15.104:0/3503939287 learned_addr learned my addr 172.21.15.104:0/3503939287 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:38:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb4cff3640 1 -- 172.21.15.104:0/3503939287 >> v1:172.21.15.104:6791/0 conn(0x7fdb4813f2e0 legacy=0x7fdb481416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 362243355 0 0) 0x7fdb4809dd50 con 0x7fdb4813f2e0 2024-08-19T12:38:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb477fe640 1 -- 172.21.15.104:0/3503939287 >> v1:172.21.15.26:6789/0 conn(0x7fdb480a57a0 legacy=0x7fdb480a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:10.965 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient(hunting): my global_id is 4127 2024-08-19T12:38:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb47fff640 1 -- 172.21.15.104:0/3503939287 >> v1:172.21.15.26:6790/0 conn(0x7fdb48073470 legacy=0x7fdb48136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb30003520 con 0x7fdb4813f2e0 2024-08-19T12:38:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3975210642 0 0) 0x7fdb48136910 con 0x7fdb48073470 2024-08-19T12:38:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient(hunting): my global_id is 4124 2024-08-19T12:38:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb4809dd50 con 0x7fdb48073470 2024-08-19T12:38:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 450696323 0 0) 0x7fdb4809dae0 con 0x7fdb480a57a0 2024-08-19T12:38:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:10.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient(hunting): my global_id is 4104 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb48136910 con 0x7fdb480a57a0 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 384144946 0 0) 0x7fdb30003520 con 0x7fdb4813f2e0 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 >> v1:172.21.15.26:6790/0 conn(0x7fdb48073470 legacy=0x7fdb48136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 >> v1:172.21.15.26:6789/0 conn(0x7fdb480a57a0 legacy=0x7fdb480a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 monclient: found mon.noname-h 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4809e030 con 0x7fdb4813f2e0 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:10.967 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb4ea56640 5 monclient: authenticate success, global_id 4127 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb4ea56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdb40001e60 con 0x7fdb4813f2e0 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient: dump: 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:10.968 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb4ea56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdb400020b0 con 0x7fdb4813f2e0 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb4ea56640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 1 -- 172.21.15.104:0/3503939287 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdb40004f60 con 0x7fdb4813f2e0 2024-08-19T12:38:10.969 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb4ea56640 4 set_mon_vals no callback set 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.955+0000 7fdb46ffd640 10 monclient: dump: 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:10.970 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.956+0000 7fdb4ea56640 10 monclient: shutdown 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.956+0000 7fdb4ea56640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.956+0000 7fdb4ea56640 1 -- 172.21.15.104:0/3503939287 >> v1:172.21.15.104:6791/0 conn(0x7fdb4813f2e0 legacy=0x7fdb481416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.956+0000 7fdb4ea56640 1 -- 172.21.15.104:0/3503939287 shutdown_connections 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.956+0000 7fdb4ea56640 1 -- 172.21.15.104:0/3503939287 >> 172.21.15.104:0/3503939287 conn(0x7fdb480abff0 msgr2=0x7fdb480ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.956+0000 7fdb4ea56640 1 -- 172.21.15.104:0/3503939287 shutdown_connections 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.956+0000 7fdb4ea56640 1 -- 172.21.15.104:0/3503939287 wait complete. 2024-08-19T12:38:10.971 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.957+0000 7fdb4ea56640 10 monclient: build_initial_monmap 2024-08-19T12:38:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.957+0000 7fdb4ea56640 10 monclient: monmap: 2024-08-19T12:38:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-08-19T12:38:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-08-19T12:38:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-08-19T12:38:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.972 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.957+0000 7fdb4ea56640 1 librados: starting msgr at 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.957+0000 7fdb4ea56640 1 librados: starting objecter 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.957+0000 7fdb4ea56640 1 Processor -- start 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.957+0000 7fdb4ea56640 1 -- start start 2024-08-19T12:38:10.973 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.957+0000 7fdb4ea56640 1 librados: setting wanted keys 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.957+0000 7fdb4ea56640 1 librados: calling monclient init 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.957+0000 7fdb4ea56640 10 monclient: init 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.958+0000 7fdb4ea56640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.958+0000 7fdb4ea56640 10 monclient: _add_conns ranks=[4,6,1,7,3,5,8,2,0] 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.958+0000 7fdb4ea56640 10 monclient(hunting): picked mon.noname-c con 0x7fdb48073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.958+0000 7fdb4ea56640 10 monclient(hunting): picked mon.noname-d con 0x7fdb480a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.958+0000 7fdb4ea56640 10 monclient(hunting): picked mon.noname-f con 0x7fdb4813f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.958+0000 7fdb4ea56640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb48135430 con 0x7fdb4813f2e0 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.958+0000 7fdb4ea56640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb4806d8f0 con 0x7fdb48073470 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.958+0000 7fdb4ea56640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb4813e720 con 0x7fdb480a57a0 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.958+0000 7fdb4ea56640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.958+0000 7fdb4ea56640 10 monclient(hunting): authenticate will time out at 654.987152s 2024-08-19T12:38:10.974 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.958+0000 7fdb4cff3640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdb4813f2e0 0x7fdb481d6c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53318/0 (socket says 172.21.15.104:53318) 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.958+0000 7fdb4cff3640 1 -- 172.21.15.104:0/2154886848 learned_addr learned my addr 172.21.15.104:0/2154886848 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb4cff3640 1 -- 172.21.15.104:0/2154886848 >> v1:172.21.15.104:6789/0 conn(0x7fdb4813f2e0 legacy=0x7fdb481d6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 client.?.objecter ms_handle_connect 0x7fdb4813f2e0 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 client.?.objecter ms_handle_connect 0x7fdb48073470 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb47fff640 1 -- 172.21.15.104:0/2154886848 >> v1:172.21.15.26:6791/0 conn(0x7fdb48073470 legacy=0x7fdb48132590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 client.?.objecter ms_handle_connect 0x7fdb480a57a0 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb477fe640 1 -- 172.21.15.104:0/2154886848 >> v1:172.21.15.26:6792/0 conn(0x7fdb480a57a0 legacy=0x7fdb481d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3776566814 0 0) 0x7fdb4809e030 con 0x7fdb4813f2e0 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:10.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): my global_id is 4114 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb34004eb0 con 0x7fdb4813f2e0 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 113893381 0 0) 0x7fdb4809dd50 con 0x7fdb48073470 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): my global_id is 4144 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb4809e030 con 0x7fdb48073470 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2486582493 0 0) 0x7fdb4813e720 con 0x7fdb480a57a0 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:10.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): my global_id is 4119 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb4809dd50 con 0x7fdb480a57a0 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4087432324 0 0) 0x7fdb34004eb0 con 0x7fdb4813f2e0 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb4813e720 con 0x7fdb4813f2e0 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdb40004ea0 con 0x7fdb4813f2e0 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.977 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:10.978 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.978 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:10.978 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:10.978 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:10.978 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:10.978 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:10.978 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:10.978 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:10.978 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:10.978 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3438305690 0 0) 0x7fdb4809e030 con 0x7fdb48073470 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb34004eb0 con 0x7fdb48073470 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdb38001d40 con 0x7fdb48073470 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.960+0000 7fdb44ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.979 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 350256586 0 0) 0x7fdb4809dd50 con 0x7fdb480a57a0 2024-08-19T12:38:10.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb4809e030 con 0x7fdb480a57a0 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdb3c003610 con 0x7fdb480a57a0 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:10.981 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4002303988 0 0) 0x7fdb4813e720 con 0x7fdb4813f2e0 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 >> v1:172.21.15.26:6792/0 conn(0x7fdb480a57a0 legacy=0x7fdb481d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 >> v1:172.21.15.26:6791/0 conn(0x7fdb48073470 legacy=0x7fdb48132590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 1 monclient: found mon.f 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:10.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb48137420 con 0x7fdb4813f2e0 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 10 monclient: _finish_auth 0 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb4ea56640 5 monclient: authenticate success, global_id 4114 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb4ea56640 10 monclient: _renew_subs 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb4ea56640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb4ea56640 1 -- 172.21.15.104:0/2154886848 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdb480a48c0 con 0x7fdb4813f2e0 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb4ea56640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb4ea56640 10 monclient: _renew_subs 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb4ea56640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb4ea56640 1 -- 172.21.15.104:0/2154886848 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdb480a32d0 con 0x7fdb4813f2e0 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb4ea56640 1 librados: init done 2024-08-19T12:38:10.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdb400052f0 con 0x7fdb4813f2e0 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdb40005860 con 0x7fdb4813f2e0 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 10 monclient: dump: 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:10.984 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb467fc640 4 set_mon_vals no callback set 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdb40005f10 con 0x7fdb4813f2e0 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (unknown 375172481 0 0) 0x7fdb40007450 con 0x7fdb4813f2e0 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 10 client.4114.objecter ms_dispatch 0x7fdb48000b90 osd_map(3..3 src has 1..3) 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 3 client.4114.objecter handle_osd_map got epochs [3,3] > 0 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 3 client.4114.objecter handle_osd_map decoding full epoch 3 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.961+0000 7fdb44ff9640 20 client.4114.objecter dump_active .. 0 homeless 2024-08-19T12:38:10.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.962+0000 7fdb4ea56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:10.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.962+0000 7fdb4ea56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:10.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.962+0000 7fdb4ea56640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:10.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.962+0000 7fdb4ea56640 1 -- 172.21.15.104:0/2154886848 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb480a3d60 con 0x7fdb4813f2e0 2024-08-19T12:38:10.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.966+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.1 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fdb48135430 con 0x7fdb4813f2e0 2024-08-19T12:38:10.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.966+0000 7fdb44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:10.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:10.966+0000 7fdb44ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:11.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.038+0000 7fdb4ea56640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fc6dcabb-211c-484a-b215-2ef5fe1cbd21", "id": 1}] 2024-08-19T12:38:11.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.038+0000 7fdb4ea56640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fc6dcabb-211c-484a-b215-2ef5fe1cbd21", "id": 1}] 2024-08-19T12:38:11.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.038+0000 7fdb4ea56640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:11.039 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.038+0000 7fdb4ea56640 1 -- 172.21.15.104:0/2154886848 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd new", "uuid": "fc6dcabb-211c-484a-b215-2ef5fe1cbd21", "id": 1} v 0) -- 0x7fdb4806c840 con 0x7fdb4813f2e0 2024-08-19T12:38:11.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.041+0000 7fdb44ff9640 1 -- 172.21.15.104:0/2154886848 <== mon.1 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fc6dcabb-211c-484a-b215-2ef5fe1cbd21", "id": 1}]=0 v4) ==== 112+0+2 (unknown 1740944478 0 940483276) 0x7fdb40007650 con 0x7fdb4813f2e0 2024-08-19T12:38:11.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.041+0000 7fdb44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fc6dcabb-211c-484a-b215-2ef5fe1cbd21", "id": 1}] 2024-08-19T12:38:11.042 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.041+0000 7fdb44ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:11.043 INFO:teuthology.orchestra.run.smithi104.stdout:1 2024-08-19T12:38:11.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.042+0000 7fdb4ea56640 10 librados: watch_flush enter 2024-08-19T12:38:11.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.042+0000 7fdb4ea56640 10 librados: watch_flush exit 2024-08-19T12:38:11.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.042+0000 7fdb4ea56640 20 client.4114.objecter shutdown clearing up homeless session... 2024-08-19T12:38:11.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.042+0000 7fdb4ea56640 10 client.4114.objecter successfully canceled tick 2024-08-19T12:38:11.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.042+0000 7fdb4ea56640 10 monclient: shutdown 2024-08-19T12:38:11.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.042+0000 7fdb4ea56640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:11.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.042+0000 7fdb4ea56640 1 -- 172.21.15.104:0/2154886848 >> v1:172.21.15.104:6789/0 conn(0x7fdb4813f2e0 legacy=0x7fdb481d6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.044 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.042+0000 7fdb4ea56640 1 -- 172.21.15.104:0/2154886848 shutdown_connections 2024-08-19T12:38:11.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.042+0000 7fdb4ea56640 1 -- 172.21.15.104:0/2154886848 >> 172.21.15.104:0/2154886848 conn(0x7fdb480abff0 msgr2=0x7fdb48076880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:11.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.043+0000 7fdb4ea56640 1 -- 172.21.15.104:0/2154886848 shutdown_connections 2024-08-19T12:38:11.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.043+0000 7fdb4ea56640 1 -- 172.21.15.104:0/2154886848 wait complete. 2024-08-19T12:38:11.045 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.043+0000 7fdb4ea56640 1 librados: shutdown 2024-08-19T12:38:11.053 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 88a4da36-5efe-4f64-84c3-28413eba4011 2 2024-08-19T12:38:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.150+0000 7f0c69fd9640 10 monclient: get_monmap_and_config 2024-08-19T12:38:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.150+0000 7f0c69fd9640 10 monclient: build_initial_monmap 2024-08-19T12:38:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.150+0000 7f0c69fd9640 10 monclient: monmap: 2024-08-19T12:38:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-08-19T12:38:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:38:11.151109+0000 2024-08-19T12:38:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:38:11.151109+0000 2024-08-19T12:38:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:11.151 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:11.152 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:11.152 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:11.152 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:11.152 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:11.152 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:11.152 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:11.152 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.150+0000 7f0c69fd9640 1 Processor -- start 2024-08-19T12:38:11.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c69fd9640 1 -- start start 2024-08-19T12:38:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c69fd9640 10 monclient: init 2024-08-19T12:38:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c69fd9640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c69fd9640 10 monclient: _add_conns ranks=[1,8,4,2,3,6,7,0,5] 2024-08-19T12:38:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c69fd9640 10 monclient(hunting): picked mon.noname-f con 0x7f0c64073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c69fd9640 10 monclient(hunting): picked mon.noname-e con 0x7f0c640a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c69fd9640 10 monclient(hunting): picked mon.noname-c con 0x7f0c6413f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c69fd9640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6409dae0 con 0x7f0c64073470 2024-08-19T12:38:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c69fd9640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c64136910 con 0x7f0c6413f2e0 2024-08-19T12:38:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c69fd9640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6409dd50 con 0x7f0c640a57a0 2024-08-19T12:38:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c69fd9640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c69fd9640 10 monclient(hunting): authenticate will time out at 655.180578s 2024-08-19T12:38:11.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c637fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0c64073470 0x7f0c64136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53328/0 (socket says 172.21.15.104:53328) 2024-08-19T12:38:11.155 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.151+0000 7f0c637fe640 1 -- 172.21.15.104:0/3775336174 learned_addr learned my addr 172.21.15.104:0/3775336174 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:38:11.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.153+0000 7f0c637fe640 1 -- 172.21.15.104:0/3775336174 >> v1:172.21.15.104:6789/0 conn(0x7f0c64073470 legacy=0x7f0c64136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.153+0000 7f0c62ffd640 1 -- 172.21.15.104:0/3775336174 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 727562905 0 0) 0x7f0c6409dae0 con 0x7f0c64073470 2024-08-19T12:38:11.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient(hunting): my global_id is 4123 2024-08-19T12:38:11.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 1 -- 172.21.15.104:0/3775336174 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c50003520 con 0x7f0c64073470 2024-08-19T12:38:11.159 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c63fff640 1 -- 172.21.15.104:0/3775336174 >> v1:172.21.15.26:6791/0 conn(0x7f0c6413f2e0 legacy=0x7f0c641416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c5affd640 1 -- 172.21.15.104:0/3775336174 >> v1:172.21.15.26:6793/0 conn(0x7f0c640a57a0 legacy=0x7f0c640a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 1 -- 172.21.15.104:0/3775336174 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2139103237 0 0) 0x7f0c64136910 con 0x7f0c6413f2e0 2024-08-19T12:38:11.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.161 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient(hunting): my global_id is 4153 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 1 -- 172.21.15.104:0/3775336174 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c6409dae0 con 0x7f0c6413f2e0 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 1 -- 172.21.15.104:0/3775336174 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3021084676 0 0) 0x7f0c50003520 con 0x7f0c64073470 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 1 -- 172.21.15.104:0/3775336174 >> v1:172.21.15.26:6793/0 conn(0x7f0c640a57a0 legacy=0x7f0c640a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 1 -- 172.21.15.104:0/3775336174 >> v1:172.21.15.26:6791/0 conn(0x7f0c6413f2e0 legacy=0x7f0c641416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 1 monclient: found mon.noname-f 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 1 -- 172.21.15.104:0/3775336174 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6409e030 con 0x7f0c64073470 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:11.162 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 1 -- 172.21.15.104:0/3775336174 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c54001ae0 con 0x7f0c64073470 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c69fd9640 5 monclient: authenticate success, global_id 4123 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c69fd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient: dump: 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.163 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c69fd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 1 -- 172.21.15.104:0/3775336174 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c540048a0 con 0x7f0c64073470 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:11.164 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 1 -- 172.21.15.104:0/3775336174 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c54004e10 con 0x7f0c64073470 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c69fd9640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c69fd9640 4 set_mon_vals no callback set 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c62ffd640 10 monclient: dump: 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.165 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c69fd9640 10 monclient: shutdown 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c69fd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.154+0000 7f0c69fd9640 1 -- 172.21.15.104:0/3775336174 >> v1:172.21.15.104:6789/0 conn(0x7f0c64073470 legacy=0x7f0c64136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.155+0000 7f0c69fd9640 1 -- 172.21.15.104:0/3775336174 shutdown_connections 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.155+0000 7f0c69fd9640 1 -- 172.21.15.104:0/3775336174 >> 172.21.15.104:0/3775336174 conn(0x7f0c640abff0 msgr2=0x7f0c640ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:11.166 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.155+0000 7f0c69fd9640 1 -- 172.21.15.104:0/3775336174 shutdown_connections 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.155+0000 7f0c69fd9640 1 -- 172.21.15.104:0/3775336174 wait complete. 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.155+0000 7f0c69fd9640 10 monclient: build_initial_monmap 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.155+0000 7f0c69fd9640 10 monclient: monmap: 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:11.167 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.155+0000 7f0c69fd9640 1 librados: starting msgr at 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.155+0000 7f0c69fd9640 1 librados: starting objecter 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.155+0000 7f0c69fd9640 1 Processor -- start 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 1 -- start start 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 1 librados: setting wanted keys 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 1 librados: calling monclient init 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 10 monclient: init 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:11.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 10 monclient: _add_conns ranks=[5,7,1,4,3,6,8,0,2] 2024-08-19T12:38:11.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 10 monclient(hunting): picked mon.noname-h con 0x7f0c64073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:11.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 10 monclient(hunting): picked mon.noname-i con 0x7f0c640a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:11.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 10 monclient(hunting): picked mon.noname-f con 0x7f0c6413f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:11.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c640a47a0 con 0x7f0c6413f2e0 2024-08-19T12:38:11.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c641370b0 con 0x7f0c64073470 2024-08-19T12:38:11.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c64137390 con 0x7f0c640a57a0 2024-08-19T12:38:11.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c637fe640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f0c64073470 0x7f0c640a22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.104:51314/0 (socket says 172.21.15.104:51314) 2024-08-19T12:38:11.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c5affd640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f0c640a57a0 0x7f0c640a29f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.104:39368/0 (socket says 172.21.15.104:39368) 2024-08-19T12:38:11.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c637fe640 1 -- 172.21.15.104:0/1896266844 learned_addr learned my addr 172.21.15.104:0/1896266844 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:38:11.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:11.169 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.156+0000 7f0c69fd9640 10 monclient(hunting): authenticate will time out at 655.185855s 2024-08-19T12:38:11.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.158+0000 7f0c637fe640 1 -- 172.21.15.104:0/1896266844 >> v1:172.21.15.104:6791/0 conn(0x7f0c64073470 legacy=0x7f0c640a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.158+0000 7f0c63fff640 1 -- 172.21.15.104:0/1896266844 >> v1:172.21.15.104:6789/0 conn(0x7f0c6413f2e0 legacy=0x7f0c641d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.158+0000 7f0c60ff9640 10 client.?.objecter ms_handle_connect 0x7f0c64073470 2024-08-19T12:38:11.173 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.158+0000 7f0c60ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.158+0000 7f0c60ff9640 10 client.?.objecter ms_handle_connect 0x7f0c6413f2e0 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.158+0000 7f0c60ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.158+0000 7f0c60ff9640 10 client.?.objecter ms_handle_connect 0x7f0c640a57a0 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.158+0000 7f0c60ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.158+0000 7f0c5affd640 1 -- 172.21.15.104:0/1896266844 >> v1:172.21.15.104:6792/0 conn(0x7f0c640a57a0 legacy=0x7f0c640a29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240307966 0 0) 0x7f0c641370b0 con 0x7f0c64073470 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): my global_id is 4136 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c44004eb0 con 0x7f0c64073470 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1470408663 0 0) 0x7f0c64137390 con 0x7f0c640a57a0 2024-08-19T12:38:11.174 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): my global_id is 4138 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c641370b0 con 0x7f0c640a57a0 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 419507560 0 0) 0x7f0c6409dae0 con 0x7f0c6413f2e0 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): my global_id is 4132 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c64137390 con 0x7f0c6413f2e0 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 565553051 0 0) 0x7f0c44004eb0 con 0x7f0c64073470 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c6409dae0 con 0x7f0c64073470 2024-08-19T12:38:11.175 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4125161127 0 0) 0x7f0c641370b0 con 0x7f0c640a57a0 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c44004eb0 con 0x7f0c640a57a0 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c54004d90 con 0x7f0c64073470 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.176 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c4c001d10 con 0x7f0c640a57a0 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.177 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.178 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2865699696 0 0) 0x7f0c64137390 con 0x7f0c6413f2e0 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c641370b0 con 0x7f0c6413f2e0 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c5c002070 con 0x7f0c6413f2e0 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.179 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2914221748 0 0) 0x7f0c6409dae0 con 0x7f0c64073470 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 >> v1:172.21.15.104:6792/0 conn(0x7f0c640a57a0 legacy=0x7f0c640a29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.180 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 >> v1:172.21.15.104:6789/0 conn(0x7f0c6413f2e0 legacy=0x7f0c641d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 monclient: found mon.h 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6406d8f0 con 0x7f0c64073470 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient: _finish_auth 0 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c69fd9640 5 monclient: authenticate success, global_id 4136 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c69fd9640 10 monclient: _renew_subs 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c69fd9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c69fd9640 1 -- 172.21.15.104:0/1896266844 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c640a5430 con 0x7f0c64073470 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c69fd9640 10 client.4136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:11.181 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c69fd9640 10 monclient: _renew_subs 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c69fd9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c69fd9640 1 -- 172.21.15.104:0/1896266844 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c640a32e0 con 0x7f0c64073470 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c69fd9640 1 librados: init done 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c54001a80 con 0x7f0c64073470 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c540048a0 con 0x7f0c64073470 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c627fc640 4 set_mon_vals no callback set 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.159+0000 7f0c60ff9640 10 monclient: dump: 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.182 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.160+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0c54005ad0 con 0x7f0c64073470 2024-08-19T12:38:11.183 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.160+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (unknown 3202988577 0 0) 0x7f0c54006dc0 con 0x7f0c64073470 2024-08-19T12:38:11.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.160+0000 7f0c60ff9640 10 client.4136.objecter ms_dispatch 0x7f0c64000b90 osd_map(4..4 src has 1..4) 2024-08-19T12:38:11.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.160+0000 7f0c60ff9640 3 client.4136.objecter handle_osd_map got epochs [4,4] > 0 2024-08-19T12:38:11.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.160+0000 7f0c60ff9640 3 client.4136.objecter handle_osd_map decoding full epoch 4 2024-08-19T12:38:11.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.160+0000 7f0c60ff9640 20 client.4136.objecter dump_active .. 0 homeless 2024-08-19T12:38:11.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.160+0000 7f0c69fd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:11.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.160+0000 7f0c69fd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:11.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.160+0000 7f0c69fd9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:11.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.160+0000 7f0c69fd9640 1 -- 172.21.15.104:0/1896266844 --> v1:172.21.15.104:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c48005080 con 0x7f0c64073470 2024-08-19T12:38:11.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.162+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.5 v1:172.21.15.104:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f0c540069a0 con 0x7f0c64073470 2024-08-19T12:38:11.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.162+0000 7f0c60ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:11.184 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.162+0000 7f0c60ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:11.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.233+0000 7f0c69fd9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "88a4da36-5efe-4f64-84c3-28413eba4011", "id": 2}] 2024-08-19T12:38:11.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.233+0000 7f0c69fd9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "88a4da36-5efe-4f64-84c3-28413eba4011", "id": 2}] 2024-08-19T12:38:11.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.233+0000 7f0c69fd9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:11.234 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.233+0000 7f0c69fd9640 1 -- 172.21.15.104:0/1896266844 --> v1:172.21.15.104:6791/0 -- mon_command({"prefix": "osd new", "uuid": "88a4da36-5efe-4f64-84c3-28413eba4011", "id": 2} v 0) -- 0x7f0c48005370 con 0x7f0c64073470 2024-08-19T12:38:11.259 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.259+0000 7f0c60ff9640 1 -- 172.21.15.104:0/1896266844 <== mon.5 v1:172.21.15.104:6791/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "88a4da36-5efe-4f64-84c3-28413eba4011", "id": 2}]=0 v5) ==== 112+0+2 (unknown 1334693771 0 216599125) 0x7f0c54005dc0 con 0x7f0c64073470 2024-08-19T12:38:11.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.259+0000 7f0c60ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "88a4da36-5efe-4f64-84c3-28413eba4011", "id": 2}] 2024-08-19T12:38:11.260 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.259+0000 7f0c60ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:11.260 INFO:teuthology.orchestra.run.smithi104.stdout:2 2024-08-19T12:38:11.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.259+0000 7f0c69fd9640 10 librados: watch_flush enter 2024-08-19T12:38:11.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.259+0000 7f0c69fd9640 10 librados: watch_flush exit 2024-08-19T12:38:11.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.259+0000 7f0c69fd9640 20 client.4136.objecter shutdown clearing up homeless session... 2024-08-19T12:38:11.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.259+0000 7f0c69fd9640 10 client.4136.objecter successfully canceled tick 2024-08-19T12:38:11.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.259+0000 7f0c69fd9640 10 monclient: shutdown 2024-08-19T12:38:11.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.259+0000 7f0c69fd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:11.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.259+0000 7f0c69fd9640 1 -- 172.21.15.104:0/1896266844 >> v1:172.21.15.104:6791/0 conn(0x7f0c64073470 legacy=0x7f0c640a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.261 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.260+0000 7f0c69fd9640 1 -- 172.21.15.104:0/1896266844 shutdown_connections 2024-08-19T12:38:11.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.260+0000 7f0c69fd9640 1 -- 172.21.15.104:0/1896266844 >> 172.21.15.104:0/1896266844 conn(0x7f0c640abff0 msgr2=0x7f0c64134970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:11.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.260+0000 7f0c69fd9640 1 -- 172.21.15.104:0/1896266844 shutdown_connections 2024-08-19T12:38:11.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.260+0000 7f0c69fd9640 1 -- 172.21.15.104:0/1896266844 wait complete. 2024-08-19T12:38:11.262 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.260+0000 7f0c69fd9640 1 librados: shutdown 2024-08-19T12:38:11.271 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 98c08a65-4fb2-4e5f-a9bc-e8899aed1103 3 2024-08-19T12:38:11.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.368+0000 7fc229cba640 10 monclient: get_monmap_and_config 2024-08-19T12:38:11.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.368+0000 7fc229cba640 10 monclient: build_initial_monmap 2024-08-19T12:38:11.369 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.368+0000 7fc229cba640 10 monclient: monmap: 2024-08-19T12:38:11.369 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-08-19T12:38:11.369 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.369 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:38:11.369113+0000 2024-08-19T12:38:11.369 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:38:11.369113+0000 2024-08-19T12:38:11.369 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:11.369 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.369 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:11.370 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:11.370 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:11.370 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:11.370 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:11.370 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:11.370 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:11.370 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:11.370 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:11.370 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 1 Processor -- start 2024-08-19T12:38:11.371 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 1 -- start start 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 10 monclient: init 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 10 monclient: _add_conns ranks=[2,8,0,6,7,5,3,1,4] 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 10 monclient(hunting): picked mon.noname-b con 0x7fc224073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 10 monclient(hunting): picked mon.noname-e con 0x7fc2240a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 10 monclient(hunting): picked mon.noname-a con 0x7fc22413f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22409dae0 con 0x7fc22413f2e0 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc224136910 con 0x7fc224073470 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22409dd50 con 0x7fc2240a57a0 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc229cba640 10 monclient(hunting): authenticate will time out at 655.398654s 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc2237fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc224073470 0x7fc224136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.104:36732/0 (socket says 172.21.15.104:36732) 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc223fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc22413f2e0 0x7fc2241416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.104:58940/0 (socket says 172.21.15.104:58940) 2024-08-19T12:38:11.372 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc21bfff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fc2240a57a0 0x7fc2240a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.104:47988/0 (socket says 172.21.15.104:47988) 2024-08-19T12:38:11.373 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.369+0000 7fc223fff640 1 -- 172.21.15.104:0/1564170642 learned_addr learned my addr 172.21.15.104:0/1564170642 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:38:11.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc223fff640 1 -- 172.21.15.104:0/1564170642 >> v1:172.21.15.26:6789/0 conn(0x7fc22413f2e0 legacy=0x7fc2241416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc21bfff640 1 -- 172.21.15.104:0/1564170642 >> v1:172.21.15.26:6793/0 conn(0x7fc2240a57a0 legacy=0x7fc2240a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc2237fe640 1 -- 172.21.15.104:0/1564170642 >> v1:172.21.15.26:6790/0 conn(0x7fc224073470 legacy=0x7fc224136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2020180056 0 0) 0x7fc22409dd50 con 0x7fc2240a57a0 2024-08-19T12:38:11.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.376 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient(hunting): my global_id is 4121 2024-08-19T12:38:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc210003520 con 0x7fc2240a57a0 2024-08-19T12:38:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2667171144 0 0) 0x7fc22409dae0 con 0x7fc22413f2e0 2024-08-19T12:38:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient(hunting): my global_id is 4113 2024-08-19T12:38:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc22409dd50 con 0x7fc22413f2e0 2024-08-19T12:38:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 745206976 0 0) 0x7fc224136910 con 0x7fc224073470 2024-08-19T12:38:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.377 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient(hunting): my global_id is 4133 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc22409dae0 con 0x7fc224073470 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2846421356 0 0) 0x7fc210003520 con 0x7fc2240a57a0 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 >> v1:172.21.15.26:6790/0 conn(0x7fc224073470 legacy=0x7fc224136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 >> v1:172.21.15.26:6789/0 conn(0x7fc22413f2e0 legacy=0x7fc2241416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 monclient: found mon.noname-e 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22409e030 con 0x7fc2240a57a0 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:11.378 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc20c001a10 con 0x7fc2240a57a0 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient: dump: 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.379 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc229cba640 5 monclient: authenticate success, global_id 4121 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc229cba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc20c001ce0 con 0x7fc2240a57a0 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc229cba640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc229cba640 4 set_mon_vals no callback set 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 1 -- 172.21.15.104:0/1564170642 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc20c004b60 con 0x7fc2240a57a0 2024-08-19T12:38:11.380 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:11.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:11.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.372+0000 7fc222ffd640 10 monclient: dump: 2024-08-19T12:38:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.382 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.373+0000 7fc229cba640 10 monclient: shutdown 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.373+0000 7fc229cba640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.373+0000 7fc229cba640 1 -- 172.21.15.104:0/1564170642 >> v1:172.21.15.26:6793/0 conn(0x7fc2240a57a0 legacy=0x7fc2240a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.373+0000 7fc229cba640 1 -- 172.21.15.104:0/1564170642 shutdown_connections 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.373+0000 7fc229cba640 1 -- 172.21.15.104:0/1564170642 >> 172.21.15.104:0/1564170642 conn(0x7fc2240abff0 msgr2=0x7fc2240ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.373+0000 7fc229cba640 1 -- 172.21.15.104:0/1564170642 shutdown_connections 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.373+0000 7fc229cba640 1 -- 172.21.15.104:0/1564170642 wait complete. 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 10 monclient: build_initial_monmap 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 10 monclient: monmap: 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-08-19T12:38:11.383 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:11.384 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 1 librados: starting msgr at 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 1 librados: starting objecter 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 1 Processor -- start 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 1 -- start start 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 1 librados: setting wanted keys 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 1 librados: calling monclient init 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 10 monclient: init 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 10 monclient: _add_conns ranks=[3,0,5,7,8,2,1,4,6] 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 10 monclient(hunting): picked mon.noname-g con 0x7fc224073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 10 monclient(hunting): picked mon.noname-a con 0x7fc2240a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 10 monclient(hunting): picked mon.noname-h con 0x7fc22413f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:11.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22413eb90 con 0x7fc2240a57a0 2024-08-19T12:38:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc224137650 con 0x7fc224073470 2024-08-19T12:38:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22406d8f0 con 0x7fc22413f2e0 2024-08-19T12:38:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc229cba640 10 monclient(hunting): authenticate will time out at 655.403851s 2024-08-19T12:38:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc223fff640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7fc22413f2e0 0x7fc2241d6a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.104:51316/0 (socket says 172.21.15.104:51316) 2024-08-19T12:38:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc2237fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fc224073470 0x7fc2241ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:56810/0 (socket says 172.21.15.104:56810) 2024-08-19T12:38:11.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.374+0000 7fc223fff640 1 -- 172.21.15.104:0/1250570228 learned_addr learned my addr 172.21.15.104:0/1250570228 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:38:11.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc223fff640 1 -- 172.21.15.104:0/1250570228 >> v1:172.21.15.104:6791/0 conn(0x7fc22413f2e0 legacy=0x7fc2241d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.390 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 client.?.objecter ms_handle_connect 0x7fc22413f2e0 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc2237fe640 1 -- 172.21.15.104:0/1250570228 >> v1:172.21.15.104:6790/0 conn(0x7fc224073470 legacy=0x7fc2241ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 client.?.objecter ms_handle_connect 0x7fc224073470 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 client.?.objecter ms_handle_connect 0x7fc2240a57a0 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc21bfff640 1 -- 172.21.15.104:0/1250570228 >> v1:172.21.15.26:6789/0 conn(0x7fc2240a57a0 legacy=0x7fc2241d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1758427210 0 0) 0x7fc224137650 con 0x7fc224073470 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): my global_id is 4134 2024-08-19T12:38:11.391 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc204004eb0 con 0x7fc224073470 2024-08-19T12:38:11.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4283960137 0 0) 0x7fc22406d8f0 con 0x7fc22413f2e0 2024-08-19T12:38:11.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): my global_id is 4145 2024-08-19T12:38:11.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc224137650 con 0x7fc22413f2e0 2024-08-19T12:38:11.392 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2421528046 0 0) 0x7fc22413eb90 con 0x7fc2240a57a0 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): my global_id is 4122 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc22406d8f0 con 0x7fc2240a57a0 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 445257402 0 0) 0x7fc204004eb0 con 0x7fc224073470 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc22413eb90 con 0x7fc224073470 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc214001cc0 con 0x7fc224073470 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.393 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.394 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.395 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2078530604 0 0) 0x7fc224137650 con 0x7fc22413f2e0 2024-08-19T12:38:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc204004eb0 con 0x7fc22413f2e0 2024-08-19T12:38:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc21c002090 con 0x7fc22413f2e0 2024-08-19T12:38:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.395 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3673724299 0 0) 0x7fc22406d8f0 con 0x7fc2240a57a0 2024-08-19T12:38:11.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc224137650 con 0x7fc2240a57a0 2024-08-19T12:38:11.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc20c001990 con 0x7fc2240a57a0 2024-08-19T12:38:11.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:11.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:11.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:11.397 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.397 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.397 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.397 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.397 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.397 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.398 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.398 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.398 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.398 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.398 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.398 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.398 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.398 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.398 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.398 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1743161617 0 0) 0x7fc22413eb90 con 0x7fc224073470 2024-08-19T12:38:11.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 >> v1:172.21.15.104:6791/0 conn(0x7fc22413f2e0 legacy=0x7fc2241d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 >> v1:172.21.15.26:6789/0 conn(0x7fc2240a57a0 legacy=0x7fc2241d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:38:11.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 monclient: found mon.g 2024-08-19T12:38:11.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:11.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:11.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc224137370 con 0x7fc224073470 2024-08-19T12:38:11.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient: _finish_auth 0 2024-08-19T12:38:11.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:38:11.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:11.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc229cba640 5 monclient: authenticate success, global_id 4134 2024-08-19T12:38:11.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc229cba640 10 monclient: _renew_subs 2024-08-19T12:38:11.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc229cba640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc229cba640 1 -- 172.21.15.104:0/1250570228 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc2240a4880 con 0x7fc224073470 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc229cba640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc229cba640 10 monclient: _renew_subs 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc229cba640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc229cba640 1 -- 172.21.15.104:0/1250570228 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc2240a35d0 con 0x7fc224073470 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc229cba640 1 librados: init done 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.3 v1:172.21.15.104:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc214004c00 con 0x7fc224073470 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.377+0000 7fc220ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.378+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.3 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc214005170 con 0x7fc224073470 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.378+0000 7fc220ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.378+0000 7fc2227fc640 4 set_mon_vals no callback set 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.378+0000 7fc220ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:11.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.378+0000 7fc220ff9640 10 monclient: dump: 2024-08-19T12:38:11.401 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.401 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.401 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.401 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.401 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.401 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.401 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.401 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.401 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.401 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.402 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.402 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.402 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.402 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.402 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.402 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.378+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.3 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc214005810 con 0x7fc224073470 2024-08-19T12:38:11.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.378+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.3 v1:172.21.15.104:6790/0 8 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (unknown 1941796338 0 0) 0x7fc214006d70 con 0x7fc224073470 2024-08-19T12:38:11.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.378+0000 7fc220ff9640 10 client.4134.objecter ms_dispatch 0x7fc224000b90 osd_map(5..5 src has 1..5) 2024-08-19T12:38:11.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.378+0000 7fc220ff9640 3 client.4134.objecter handle_osd_map got epochs [5,5] > 0 2024-08-19T12:38:11.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.378+0000 7fc220ff9640 3 client.4134.objecter handle_osd_map decoding full epoch 5 2024-08-19T12:38:11.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.378+0000 7fc220ff9640 20 client.4134.objecter dump_active .. 0 homeless 2024-08-19T12:38:11.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.379+0000 7fc229cba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:11.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.379+0000 7fc229cba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:11.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.379+0000 7fc229cba640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:11.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.379+0000 7fc229cba640 1 -- 172.21.15.104:0/1250570228 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc208005080 con 0x7fc224073470 2024-08-19T12:38:11.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.381+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.3 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fc2140068e0 con 0x7fc224073470 2024-08-19T12:38:11.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.381+0000 7fc220ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:11.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.381+0000 7fc220ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.451+0000 7fc229cba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "98c08a65-4fb2-4e5f-a9bc-e8899aed1103", "id": 3}] 2024-08-19T12:38:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.451+0000 7fc229cba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "98c08a65-4fb2-4e5f-a9bc-e8899aed1103", "id": 3}] 2024-08-19T12:38:11.452 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.451+0000 7fc229cba640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:11.453 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.451+0000 7fc229cba640 1 -- 172.21.15.104:0/1250570228 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd new", "uuid": "98c08a65-4fb2-4e5f-a9bc-e8899aed1103", "id": 3} v 0) -- 0x7fc208005370 con 0x7fc224073470 2024-08-19T12:38:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.458+0000 7fc220ff9640 1 -- 172.21.15.104:0/1250570228 <== mon.3 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "98c08a65-4fb2-4e5f-a9bc-e8899aed1103", "id": 3}]=0 v6) ==== 112+0+2 (unknown 774739038 0 525046306) 0x7fc214005c40 con 0x7fc224073470 2024-08-19T12:38:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.458+0000 7fc220ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "98c08a65-4fb2-4e5f-a9bc-e8899aed1103", "id": 3}] 2024-08-19T12:38:11.459 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.458+0000 7fc220ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:11.460 INFO:teuthology.orchestra.run.smithi104.stdout:3 2024-08-19T12:38:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.458+0000 7fc229cba640 10 librados: watch_flush enter 2024-08-19T12:38:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.458+0000 7fc229cba640 10 librados: watch_flush exit 2024-08-19T12:38:11.462 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.458+0000 7fc229cba640 20 client.4134.objecter shutdown clearing up homeless session... 2024-08-19T12:38:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.458+0000 7fc229cba640 10 client.4134.objecter successfully canceled tick 2024-08-19T12:38:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.459+0000 7fc229cba640 10 monclient: shutdown 2024-08-19T12:38:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.459+0000 7fc229cba640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.459+0000 7fc229cba640 1 -- 172.21.15.104:0/1250570228 >> v1:172.21.15.104:6790/0 conn(0x7fc224073470 legacy=0x7fc2241ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.459+0000 7fc229cba640 1 -- 172.21.15.104:0/1250570228 shutdown_connections 2024-08-19T12:38:11.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.459+0000 7fc229cba640 1 -- 172.21.15.104:0/1250570228 >> 172.21.15.104:0/1250570228 conn(0x7fc2240abff0 msgr2=0x7fc2240a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.459+0000 7fc229cba640 1 -- 172.21.15.104:0/1250570228 shutdown_connections 2024-08-19T12:38:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.459+0000 7fc229cba640 1 -- 172.21.15.104:0/1250570228 wait complete. 2024-08-19T12:38:11.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.459+0000 7fc229cba640 1 librados: shutdown 2024-08-19T12:38:11.472 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new a4c308b6-5e55-4328-b7b0-1db11fd51d73 4 2024-08-19T12:38:11.569 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.568+0000 7fca13e20640 10 monclient: get_monmap_and_config 2024-08-19T12:38:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.568+0000 7fca13e20640 10 monclient: build_initial_monmap 2024-08-19T12:38:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.568+0000 7fca13e20640 10 monclient: monmap: 2024-08-19T12:38:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-08-19T12:38:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:38:11.569513+0000 2024-08-19T12:38:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:38:11.569513+0000 2024-08-19T12:38:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:11.570 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:11.571 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 1 Processor -- start 2024-08-19T12:38:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 1 -- start start 2024-08-19T12:38:11.571 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 10 monclient: init 2024-08-19T12:38:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 10 monclient: _add_conns ranks=[5,6,3,7,2,1,4,8,0] 2024-08-19T12:38:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 10 monclient(hunting): picked mon.noname-h con 0x7fca0c073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 10 monclient(hunting): picked mon.noname-d con 0x7fca0c0a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 10 monclient(hunting): picked mon.noname-g con 0x7fca0c13f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca0c09dae0 con 0x7fca0c13f2e0 2024-08-19T12:38:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca0c136910 con 0x7fca0c073470 2024-08-19T12:38:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca0c09dd50 con 0x7fca0c0a57a0 2024-08-19T12:38:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:11.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca13e20640 10 monclient(hunting): authenticate will time out at 655.598820s 2024-08-19T12:38:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca123bd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fca0c13f2e0 0x7fca0c1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:56824/0 (socket says 172.21.15.104:56824) 2024-08-19T12:38:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca11bbc640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7fca0c073470 0x7fca0c136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.104:51322/0 (socket says 172.21.15.104:51322) 2024-08-19T12:38:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.569+0000 7fca123bd640 1 -- 172.21.15.104:0/126443167 learned_addr learned my addr 172.21.15.104:0/126443167 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:38:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca123bd640 1 -- 172.21.15.104:0/126443167 >> v1:172.21.15.104:6790/0 conn(0x7fca0c13f2e0 legacy=0x7fca0c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca11bbc640 1 -- 172.21.15.104:0/126443167 >> v1:172.21.15.104:6791/0 conn(0x7fca0c073470 legacy=0x7fca0c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca113bb640 1 -- 172.21.15.104:0/126443167 >> v1:172.21.15.26:6792/0 conn(0x7fca0c0a57a0 legacy=0x7fca0c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 1 -- 172.21.15.104:0/126443167 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 860883581 0 0) 0x7fca0c09dae0 con 0x7fca0c13f2e0 2024-08-19T12:38:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient(hunting): my global_id is 4143 2024-08-19T12:38:11.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 1 -- 172.21.15.104:0/126443167 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9f8003520 con 0x7fca0c13f2e0 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 1 -- 172.21.15.104:0/126443167 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 896464581 0 0) 0x7fca0c136910 con 0x7fca0c073470 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient(hunting): my global_id is 4154 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 1 -- 172.21.15.104:0/126443167 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca0c09dae0 con 0x7fca0c073470 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca11bbc640 1 -- 172.21.15.104:0/126443167 >> v1:172.21.15.104:6791/0 conn(0x7fca0c073470 legacy=0x7fca0c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca11bbc640 1 -- 172.21.15.104:0/126443167 >> v1:172.21.15.104:6791/0 conn(0x7fca0c073470 legacy=0x7fca0c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca11bbc640 1 --1- 172.21.15.104:0/126443167 >> v1:172.21.15.104:6791/0 conn(0x7fca0c073470 0x7fca0c136200 :-1 s=OPENED pgs=38 cs=1 l=1).handle_message read tag failed 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca11bbc640 1 --1- 172.21.15.104:0/126443167 >> v1:172.21.15.104:6791/0 conn(0x7fca0c073470 0x7fca0c136200 :-1 s=OPENED pgs=38 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.104:6791/0 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 1 -- 172.21.15.104:0/126443167 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4040293013 0 0) 0x7fc9f8003520 con 0x7fca0c13f2e0 2024-08-19T12:38:11.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 1 -- 172.21.15.104:0/126443167 >> v1:172.21.15.26:6792/0 conn(0x7fca0c0a57a0 legacy=0x7fca0c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 1 -- 172.21.15.104:0/126443167 >> v1:172.21.15.104:6791/0 conn(0x7fca0c073470 legacy=0x7fca0c136200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient: _finish_hunting 0 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 1 monclient: found mon.noname-g 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 1 -- 172.21.15.104:0/126443167 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0c09e030 con 0x7fca0c13f2e0 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient: _finish_auth 0 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient: _check_auth_tickets 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 1 -- 172.21.15.104:0/126443167 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fca04001e60 con 0x7fca0c13f2e0 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:38:11.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient: dump: 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.581 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca13e20640 5 monclient: authenticate success, global_id 4143 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca13e20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 1 -- 172.21.15.104:0/126443167 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fca04004c20 con 0x7fca0c13f2e0 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 1 -- 172.21.15.104:0/126443167 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fca04005190 con 0x7fca0c13f2e0 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca10bba640 10 monclient: dump: 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.582 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.583 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca13e20640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.572+0000 7fca13e20640 4 set_mon_vals no callback set 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.573+0000 7fca13e20640 10 monclient: shutdown 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.573+0000 7fca13e20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.573+0000 7fca13e20640 1 -- 172.21.15.104:0/126443167 >> v1:172.21.15.104:6790/0 conn(0x7fca0c13f2e0 legacy=0x7fca0c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.573+0000 7fca13e20640 1 -- 172.21.15.104:0/126443167 shutdown_connections 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.573+0000 7fca13e20640 1 -- 172.21.15.104:0/126443167 >> 172.21.15.104:0/126443167 conn(0x7fca0c0abff0 msgr2=0x7fca0c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.573+0000 7fca13e20640 1 -- 172.21.15.104:0/126443167 shutdown_connections 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.573+0000 7fca13e20640 1 -- 172.21.15.104:0/126443167 wait complete. 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.574+0000 7fca13e20640 10 monclient: build_initial_monmap 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.574+0000 7fca13e20640 10 monclient: monmap: 2024-08-19T12:38:11.584 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:11.585 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.574+0000 7fca13e20640 1 librados: starting msgr at 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.574+0000 7fca13e20640 1 librados: starting objecter 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 1 Processor -- start 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 1 -- start start 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 1 librados: setting wanted keys 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 1 librados: calling monclient init 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 10 monclient: init 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 10 monclient: _add_conns ranks=[1,8,3,6,2,7,5,4,0] 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 10 monclient(hunting): picked mon.noname-f con 0x7fca0c073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:11.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 10 monclient(hunting): picked mon.noname-e con 0x7fca0c0a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:11.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 10 monclient(hunting): picked mon.noname-g con 0x7fca0c13f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:11.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca0c133200 con 0x7fca0c073470 2024-08-19T12:38:11.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca0c12ed70 con 0x7fca0c13f2e0 2024-08-19T12:38:11.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca0c131130 con 0x7fca0c0a57a0 2024-08-19T12:38:11.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:11.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca13e20640 10 monclient(hunting): authenticate will time out at 655.604948s 2024-08-19T12:38:11.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca11bbc640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fca0c073470 0x7fca0c12df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53352/0 (socket says 172.21.15.104:53352) 2024-08-19T12:38:11.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca123bd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fca0c13f2e0 0x7fca0c1d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:56826/0 (socket says 172.21.15.104:56826) 2024-08-19T12:38:11.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.575+0000 7fca11bbc640 1 -- 172.21.15.104:0/2924702839 learned_addr learned my addr 172.21.15.104:0/2924702839 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:38:11.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca11bbc640 1 -- 172.21.15.104:0/2924702839 >> v1:172.21.15.104:6789/0 conn(0x7fca0c073470 legacy=0x7fca0c12df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 client.?.objecter ms_handle_connect 0x7fca0c073470 2024-08-19T12:38:11.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:11.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 client.?.objecter ms_handle_connect 0x7fca0c13f2e0 2024-08-19T12:38:11.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:11.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca123bd640 1 -- 172.21.15.104:0/2924702839 >> v1:172.21.15.104:6790/0 conn(0x7fca0c13f2e0 legacy=0x7fca0c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.591 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca11bbc640 1 --1- 172.21.15.104:0/2924702839 >> v1:172.21.15.104:6789/0 conn(0x7fca0c073470 0x7fca0c12df50 :-1 s=OPENED pgs=30 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fca0c133200 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1468390494 0 0) 0x7fca0c12ed70 con 0x7fca0c13f2e0 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient(hunting): my global_id is 4152 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9f0004eb0 con 0x7fca0c13f2e0 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca113bb640 1 -- 172.21.15.104:0/2924702839 >> v1:172.21.15.26:6793/0 conn(0x7fca0c0a57a0 legacy=0x7fca0c12e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 client.?.objecter ms_handle_connect 0x7fca0c0a57a0 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1200537291 0 0) 0x7fc9f0004eb0 con 0x7fca0c13f2e0 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca0c12ed70 con 0x7fca0c13f2e0 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fca040020b0 con 0x7fca0c13f2e0 2024-08-19T12:38:11.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient(hunting): dump: 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.593 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.594 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.594 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.594 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.594 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.594 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:11.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 201523913 0 0) 0x7fca0c131130 con 0x7fca0c0a57a0 2024-08-19T12:38:11.594 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient(hunting): my global_id is 4130 2024-08-19T12:38:11.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9f0004eb0 con 0x7fca0c0a57a0 2024-08-19T12:38:11.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1870543826 0 0) 0x7fca0c12ed70 con 0x7fca0c13f2e0 2024-08-19T12:38:11.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 >> v1:172.21.15.26:6793/0 conn(0x7fca0c0a57a0 legacy=0x7fca0c12e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 >> v1:172.21.15.104:6789/0 conn(0x7fca0c073470 legacy=0x7fca0c12df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:11.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 monclient: found mon.g 2024-08-19T12:38:11.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:11.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:11.595 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0c131410 con 0x7fca0c13f2e0 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2692257995 0 0) 0x7fca0c133200 con 0x7fca0c073470 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca027fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca13e20640 5 monclient: authenticate success, global_id 4152 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca13e20640 10 monclient: _renew_subs 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca13e20640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.578+0000 7fca13e20640 1 -- 172.21.15.104:0/2924702839 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fca0c1316f0 con 0x7fca0c13f2e0 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca13e20640 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca13e20640 10 monclient: _renew_subs 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca13e20640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:11.596 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 <== mon.3 v1:172.21.15.104:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fca040053c0 con 0x7fca0c13f2e0 2024-08-19T12:38:11.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca13e20640 1 -- 172.21.15.104:0/2924702839 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fca0c12fa10 con 0x7fca0c13f2e0 2024-08-19T12:38:11.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca13e20640 1 librados: init done 2024-08-19T12:38:11.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca027fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:11.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 <== mon.3 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fca040058b0 con 0x7fca0c13f2e0 2024-08-19T12:38:11.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:11.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca027fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:11.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca03fff640 4 set_mon_vals no callback set 2024-08-19T12:38:11.597 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca027fc640 10 monclient: dump: 2024-08-19T12:38:11.597 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.597 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.598 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.598 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.598 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.598 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.598 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.598 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.598 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.598 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.598 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.598 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.598 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 <== mon.3 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fca04005de0 con 0x7fca0c13f2e0 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 <== mon.3 v1:172.21.15.104:6790/0 8 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (unknown 1397248079 0 0) 0x7fca04007170 con 0x7fca0c13f2e0 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca027fc640 10 client.4152.objecter ms_dispatch 0x7fca0c000b90 osd_map(6..6 src has 1..6) 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca027fc640 3 client.4152.objecter handle_osd_map got epochs [6,6] > 0 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca027fc640 3 client.4152.objecter handle_osd_map decoding full epoch 6 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca027fc640 20 client.4152.objecter dump_active .. 0 homeless 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca13e20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca13e20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca13e20640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:11.599 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.579+0000 7fca13e20640 1 -- 172.21.15.104:0/2924702839 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f4005080 con 0x7fca0c13f2e0 2024-08-19T12:38:11.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.581+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 <== mon.3 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fca04006c60 con 0x7fca0c13f2e0 2024-08-19T12:38:11.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.581+0000 7fca027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:11.600 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.581+0000 7fca027fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.652+0000 7fca13e20640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a4c308b6-5e55-4328-b7b0-1db11fd51d73", "id": 4}] 2024-08-19T12:38:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.652+0000 7fca13e20640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a4c308b6-5e55-4328-b7b0-1db11fd51d73", "id": 4}] 2024-08-19T12:38:11.653 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.652+0000 7fca13e20640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:11.654 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.652+0000 7fca13e20640 1 -- 172.21.15.104:0/2924702839 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd new", "uuid": "a4c308b6-5e55-4328-b7b0-1db11fd51d73", "id": 4} v 0) -- 0x7fc9f4005370 con 0x7fca0c13f2e0 2024-08-19T12:38:11.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.861+0000 7fca027fc640 1 -- 172.21.15.104:0/2924702839 <== mon.3 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a4c308b6-5e55-4328-b7b0-1db11fd51d73", "id": 4}]=0 v7) ==== 112+0+2 (unknown 895536148 0 1697012583) 0x7fca04006ee0 con 0x7fca0c13f2e0 2024-08-19T12:38:11.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.862+0000 7fca027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a4c308b6-5e55-4328-b7b0-1db11fd51d73", "id": 4}] 2024-08-19T12:38:11.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.862+0000 7fca027fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:11.863 INFO:teuthology.orchestra.run.smithi104.stdout:4 2024-08-19T12:38:11.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.863+0000 7fca13e20640 10 librados: watch_flush enter 2024-08-19T12:38:11.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.863+0000 7fca13e20640 10 librados: watch_flush exit 2024-08-19T12:38:11.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.863+0000 7fca13e20640 20 client.4152.objecter shutdown clearing up homeless session... 2024-08-19T12:38:11.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.863+0000 7fca13e20640 10 client.4152.objecter successfully canceled tick 2024-08-19T12:38:11.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.863+0000 7fca13e20640 10 monclient: shutdown 2024-08-19T12:38:11.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.863+0000 7fca13e20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:11.864 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.863+0000 7fca13e20640 1 -- 172.21.15.104:0/2924702839 >> v1:172.21.15.104:6790/0 conn(0x7fca0c13f2e0 legacy=0x7fca0c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.863+0000 7fca13e20640 1 -- 172.21.15.104:0/2924702839 shutdown_connections 2024-08-19T12:38:11.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.863+0000 7fca13e20640 1 -- 172.21.15.104:0/2924702839 >> 172.21.15.104:0/2924702839 conn(0x7fca0c0abff0 msgr2=0x7fca0c133e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:11.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.863+0000 7fca13e20640 1 -- 172.21.15.104:0/2924702839 shutdown_connections 2024-08-19T12:38:11.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.864+0000 7fca13e20640 1 -- 172.21.15.104:0/2924702839 wait complete. 2024-08-19T12:38:11.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.864+0000 7fca13e20640 1 librados: shutdown 2024-08-19T12:38:11.879 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd new 99117114-07ce-4b9b-8df6-39f5fb1b610b 5 2024-08-19T12:38:11.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.977+0000 7fb514b41640 10 monclient: get_monmap_and_config 2024-08-19T12:38:11.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.977+0000 7fb514b41640 10 monclient: build_initial_monmap 2024-08-19T12:38:11.980 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.977+0000 7fb514b41640 10 monclient: monmap: 2024-08-19T12:38:11.980 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-08-19T12:38:11.980 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:38:11.978147+0000 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:38:11.978147+0000 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:11.981 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.977+0000 7fb514b41640 1 Processor -- start 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.977+0000 7fb514b41640 1 -- start start 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.977+0000 7fb514b41640 10 monclient: init 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.978+0000 7fb514b41640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.978+0000 7fb514b41640 10 monclient: _add_conns ranks=[6,3,8,1,4,7,5,0,2] 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.978+0000 7fb514b41640 10 monclient(hunting): picked mon.noname-d con 0x7fb510073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.978+0000 7fb514b41640 10 monclient(hunting): picked mon.noname-g con 0x7fb5100a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.978+0000 7fb514b41640 10 monclient(hunting): picked mon.noname-e con 0x7fb51013f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.978+0000 7fb514b41640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb51009dae0 con 0x7fb5100a57a0 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.978+0000 7fb514b41640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb510136910 con 0x7fb510073470 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.978+0000 7fb514b41640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb51009dd50 con 0x7fb51013f2e0 2024-08-19T12:38:11.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.978+0000 7fb514b41640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:11.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.978+0000 7fb514b41640 10 monclient(hunting): authenticate will time out at 656.007451s 2024-08-19T12:38:11.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.978+0000 7fb507fff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fb5100a57a0 0x7fb5100a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:56840/0 (socket says 172.21.15.104:56840) 2024-08-19T12:38:11.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.978+0000 7fb507fff640 1 -- 172.21.15.104:0/729164807 learned_addr learned my addr 172.21.15.104:0/729164807 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:38:11.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.980+0000 7fb507fff640 1 -- 172.21.15.104:0/729164807 >> v1:172.21.15.104:6790/0 conn(0x7fb5100a57a0 legacy=0x7fb5100a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.980+0000 7fb50ed9d640 1 -- 172.21.15.104:0/729164807 >> v1:172.21.15.26:6793/0 conn(0x7fb51013f2e0 legacy=0x7fb5101416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.980+0000 7fb50e59c640 1 -- 172.21.15.104:0/729164807 >> v1:172.21.15.26:6792/0 conn(0x7fb510073470 legacy=0x7fb510136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:11.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.980+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 935210755 0 0) 0x7fb51009dae0 con 0x7fb5100a57a0 2024-08-19T12:38:11.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.980+0000 7fb50dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.980+0000 7fb50dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.980+0000 7fb50dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.980+0000 7fb50dd9b640 10 monclient(hunting): my global_id is 4161 2024-08-19T12:38:11.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.980+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4fc003520 con 0x7fb5100a57a0 2024-08-19T12:38:11.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 938261596 0 0) 0x7fb510136910 con 0x7fb510073470 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient(hunting): my global_id is 4137 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb51009dae0 con 0x7fb510073470 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 781730653 0 0) 0x7fb51009dd50 con 0x7fb51013f2e0 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient(hunting): my global_id is 4139 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb510136910 con 0x7fb51013f2e0 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 212519924 0 0) 0x7fb4fc003520 con 0x7fb5100a57a0 2024-08-19T12:38:11.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 >> v1:172.21.15.26:6793/0 conn(0x7fb51013f2e0 legacy=0x7fb5101416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 >> v1:172.21.15.26:6792/0 conn(0x7fb510073470 legacy=0x7fb510136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient: _finish_hunting 0 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 1 monclient: found mon.noname-g 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51009e030 con 0x7fb5100a57a0 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient: _finish_auth 0 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient: _check_auth_tickets 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb4f8001ae0 con 0x7fb5100a57a0 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:38:11.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient: dump: 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.994 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb514b41640 5 monclient: authenticate success, global_id 4161 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb514b41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb4f80048a0 con 0x7fb5100a57a0 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 1 -- 172.21.15.104:0/729164807 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb4f8004e10 con 0x7fb5100a57a0 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb514b41640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb514b41640 4 set_mon_vals no callback set 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.981+0000 7fb50dd9b640 10 monclient: dump: 2024-08-19T12:38:11.995 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:11.996 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.982+0000 7fb514b41640 10 monclient: shutdown 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.982+0000 7fb514b41640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.982+0000 7fb514b41640 1 -- 172.21.15.104:0/729164807 >> v1:172.21.15.104:6790/0 conn(0x7fb5100a57a0 legacy=0x7fb5100a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.982+0000 7fb514b41640 1 -- 172.21.15.104:0/729164807 shutdown_connections 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.982+0000 7fb514b41640 1 -- 172.21.15.104:0/729164807 >> 172.21.15.104:0/729164807 conn(0x7fb5100abff0 msgr2=0x7fb5100ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.982+0000 7fb514b41640 1 -- 172.21.15.104:0/729164807 shutdown_connections 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.982+0000 7fb514b41640 1 -- 172.21.15.104:0/729164807 wait complete. 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.983+0000 7fb514b41640 10 monclient: build_initial_monmap 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.983+0000 7fb514b41640 10 monclient: monmap: 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-08-19T12:38:11.997 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.983+0000 7fb514b41640 1 librados: starting msgr at 2024-08-19T12:38:11.998 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.983+0000 7fb514b41640 1 librados: starting objecter 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.983+0000 7fb514b41640 1 Processor -- start 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.983+0000 7fb514b41640 1 -- start start 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.983+0000 7fb514b41640 1 librados: setting wanted keys 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.983+0000 7fb514b41640 1 librados: calling monclient init 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.983+0000 7fb514b41640 10 monclient: init 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.984+0000 7fb514b41640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.984+0000 7fb514b41640 10 monclient: _add_conns ranks=[1,7,3,2,5,8,4,6,0] 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.984+0000 7fb514b41640 10 monclient(hunting): picked mon.noname-f con 0x7fb510073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.984+0000 7fb514b41640 10 monclient(hunting): picked mon.noname-i con 0x7fb5100a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.984+0000 7fb514b41640 10 monclient(hunting): picked mon.noname-g con 0x7fb51013f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.984+0000 7fb514b41640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb510136d30 con 0x7fb510073470 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.984+0000 7fb514b41640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb51013f070 con 0x7fb51013f2e0 2024-08-19T12:38:11.999 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.984+0000 7fb514b41640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb51006d8f0 con 0x7fb5100a57a0 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.984+0000 7fb514b41640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.984+0000 7fb514b41640 10 monclient(hunting): authenticate will time out at 656.013760s 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.984+0000 7fb50e59c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb510073470 0x7fb5100a2580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:53362/0 (socket says 172.21.15.104:53362) 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.984+0000 7fb50e59c640 1 -- 172.21.15.104:0/2533453812 learned_addr learned my addr 172.21.15.104:0/2533453812 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.986+0000 7fb50ed9d640 1 -- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6790/0 conn(0x7fb51013f2e0 legacy=0x7fb5101d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.986+0000 7fb50e59c640 1 -- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6789/0 conn(0x7fb510073470 legacy=0x7fb5100a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.986+0000 7fb506ffd640 10 client.?.objecter ms_handle_connect 0x7fb51013f2e0 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.986+0000 7fb506ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.986+0000 7fb506ffd640 10 client.?.objecter ms_handle_connect 0x7fb510073470 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.986+0000 7fb506ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.986+0000 7fb506ffd640 10 client.?.objecter ms_handle_connect 0x7fb5100a57a0 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.986+0000 7fb506ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:12.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb507fff640 1 -- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6792/0 conn(0x7fb5100a57a0 legacy=0x7fb5100a2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2805751093 0 0) 0x7fb51013f070 con 0x7fb51013f2e0 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): my global_id is 4170 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4f0004ee0 con 0x7fb51013f2e0 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 674161088 0 0) 0x7fb51006d8f0 con 0x7fb5100a57a0 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): my global_id is 4147 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb51013f070 con 0x7fb5100a57a0 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 239729372 0 0) 0x7fb510136d30 con 0x7fb510073470 2024-08-19T12:38:12.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): my global_id is 4150 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb51006d8f0 con 0x7fb510073470 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb507fff640 1 -- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6792/0 conn(0x7fb5100a57a0 legacy=0x7fb5100a2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb507fff640 1 -- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6792/0 conn(0x7fb5100a57a0 legacy=0x7fb5100a2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb50e59c640 1 -- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6789/0 conn(0x7fb510073470 legacy=0x7fb5100a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb507fff640 1 --1- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6792/0 conn(0x7fb5100a57a0 0x7fb5100a2c90 :-1 s=OPENED pgs=32 cs=1 l=1).handle_message read tag failed 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb50e59c640 1 -- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6789/0 conn(0x7fb510073470 legacy=0x7fb5100a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb507fff640 1 --1- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6792/0 conn(0x7fb5100a57a0 0x7fb5100a2c90 :-1 s=OPENED pgs=32 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb50e59c640 1 --1- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6789/0 conn(0x7fb510073470 0x7fb5100a2580 :-1 s=OPENED pgs=31 cs=1 l=1).handle_message read tag failed 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb50e59c640 1 --1- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6789/0 conn(0x7fb510073470 0x7fb5100a2580 :-1 s=OPENED pgs=31 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:38:12.002 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.104:6792/0 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.104:6789/0 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3920567043 0 0) 0x7fb4f0004ee0 con 0x7fb51013f2e0 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb510136d30 con 0x7fb51013f2e0 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb508002070 con 0x7fb51013f2e0 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:12.003 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 171369157 0 0) 0x7fb510136d30 con 0x7fb51013f2e0 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6792/0 conn(0x7fb5100a57a0 legacy=0x7fb5100a2c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:12.004 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6789/0 conn(0x7fb510073470 legacy=0x7fb5100a2580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 monclient: found mon.g 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5100a4020 con 0x7fb51013f2e0 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb514b41640 5 monclient: authenticate success, global_id 4170 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb514b41640 10 monclient: _renew_subs 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb514b41640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb514b41640 1 -- 172.21.15.104:0/2533453812 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb5100a33a0 con 0x7fb51013f2e0 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 <== mon.3 v1:172.21.15.104:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb508003d30 con 0x7fb51013f2e0 2024-08-19T12:38:12.005 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb514b41640 10 client.4170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 <== mon.3 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb5080042a0 con 0x7fb51013f2e0 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb514b41640 10 monclient: _renew_subs 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb514b41640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb514b41640 1 -- 172.21.15.104:0/2533453812 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb5101379b0 con 0x7fb51013f2e0 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb50d59a640 4 set_mon_vals no callback set 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb514b41640 1 librados: init done 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 monclient: dump: 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:12.006 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-08-19T12:38:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:12.007 INFO:teuthology.orchestra.run.smithi104.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 <== mon.3 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb508005b40 con 0x7fb51013f2e0 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 <== mon.3 v1:172.21.15.104:6790/0 8 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (unknown 1923007335 0 0) 0x7fb5080063b0 con 0x7fb51013f2e0 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 10 client.4170.objecter ms_dispatch 0x7fb510000b90 osd_map(7..7 src has 1..7) 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 3 client.4170.objecter handle_osd_map got epochs [7,7] > 0 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 3 client.4170.objecter handle_osd_map decoding full epoch 7 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.987+0000 7fb506ffd640 20 client.4170.objecter dump_active .. 0 homeless 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.989+0000 7fb514b41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.989+0000 7fb514b41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.989+0000 7fb514b41640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:12.008 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.989+0000 7fb514b41640 1 -- 172.21.15.104:0/2533453812 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5100008d0 con 0x7fb51013f2e0 2024-08-19T12:38:12.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.991+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 <== mon.3 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fb5080065b0 con 0x7fb51013f2e0 2024-08-19T12:38:12.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.991+0000 7fb506ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:12.009 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:11.991+0000 7fb506ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:12.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.062+0000 7fb514b41640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "99117114-07ce-4b9b-8df6-39f5fb1b610b", "id": 5}] 2024-08-19T12:38:12.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.062+0000 7fb514b41640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "99117114-07ce-4b9b-8df6-39f5fb1b610b", "id": 5}] 2024-08-19T12:38:12.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.062+0000 7fb514b41640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:38:12.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.062+0000 7fb514b41640 1 -- 172.21.15.104:0/2533453812 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd new", "uuid": "99117114-07ce-4b9b-8df6-39f5fb1b610b", "id": 5} v 0) -- 0x7fb510142bd0 con 0x7fb51013f2e0 2024-08-19T12:38:12.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.067+0000 7fb506ffd640 1 -- 172.21.15.104:0/2533453812 <== mon.3 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "99117114-07ce-4b9b-8df6-39f5fb1b610b", "id": 5}]=0 v8) ==== 112+0+2 (unknown 2854250029 0 1988412176) 0x7fb508005e30 con 0x7fb51013f2e0 2024-08-19T12:38:12.067 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.067+0000 7fb506ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "99117114-07ce-4b9b-8df6-39f5fb1b610b", "id": 5}] 2024-08-19T12:38:12.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.067+0000 7fb506ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:12.068 INFO:teuthology.orchestra.run.smithi104.stdout:5 2024-08-19T12:38:12.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.067+0000 7fb514b41640 10 librados: watch_flush enter 2024-08-19T12:38:12.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.068+0000 7fb514b41640 10 librados: watch_flush exit 2024-08-19T12:38:12.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.068+0000 7fb514b41640 20 client.4170.objecter shutdown clearing up homeless session... 2024-08-19T12:38:12.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.068+0000 7fb514b41640 10 client.4170.objecter successfully canceled tick 2024-08-19T12:38:12.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.068+0000 7fb514b41640 10 monclient: shutdown 2024-08-19T12:38:12.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.068+0000 7fb514b41640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:12.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.068+0000 7fb514b41640 1 -- 172.21.15.104:0/2533453812 >> v1:172.21.15.104:6790/0 conn(0x7fb51013f2e0 legacy=0x7fb5101d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:12.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.068+0000 7fb514b41640 1 -- 172.21.15.104:0/2533453812 shutdown_connections 2024-08-19T12:38:12.070 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.068+0000 7fb514b41640 1 -- 172.21.15.104:0/2533453812 >> 172.21.15.104:0/2533453812 conn(0x7fb5100abff0 msgr2=0x7fb510133a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:12.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.068+0000 7fb514b41640 1 -- 172.21.15.104:0/2533453812 shutdown_connections 2024-08-19T12:38:12.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.068+0000 7fb514b41640 1 -- 172.21.15.104:0/2533453812 wait complete. 2024-08-19T12:38:12.071 INFO:teuthology.orchestra.run.smithi104.stderr:2024-08-19T12:38:12.068+0000 7fb514b41640 1 librados: shutdown 2024-08-19T12:38:12.079 INFO:tasks.ceph.osd.0:Restarting daemon 2024-08-19T12:38:12.079 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-08-19T12:38:12.082 INFO:tasks.ceph.osd.0:Started 2024-08-19T12:38:12.082 INFO:tasks.ceph.osd.1:Restarting daemon 2024-08-19T12:38:12.082 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-08-19T12:38:12.084 INFO:tasks.ceph.osd.1:Started 2024-08-19T12:38:12.084 INFO:tasks.ceph.osd.2:Restarting daemon 2024-08-19T12:38:12.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-08-19T12:38:12.086 INFO:tasks.ceph.osd.2:Started 2024-08-19T12:38:12.086 INFO:tasks.ceph.osd.3:Restarting daemon 2024-08-19T12:38:12.086 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-08-19T12:38:12.088 INFO:tasks.ceph.osd.3:Started 2024-08-19T12:38:12.089 INFO:tasks.ceph.osd.4:Restarting daemon 2024-08-19T12:38:12.089 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-08-19T12:38:12.091 INFO:tasks.ceph.osd.4:Started 2024-08-19T12:38:12.091 INFO:tasks.ceph.osd.5:Restarting daemon 2024-08-19T12:38:12.091 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-08-19T12:38:12.093 INFO:tasks.ceph.osd.5:Started 2024-08-19T12:38:12.093 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-19T12:38:12.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.204+0000 7f9738ff2640 10 monclient: get_monmap_and_config 2024-08-19T12:38:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.204+0000 7f9738ff2640 10 monclient: build_initial_monmap 2024-08-19T12:38:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.204+0000 7f9738ff2640 10 monclient: monmap: 2024-08-19T12:38:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:12.206365+0000 2024-08-19T12:38:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:12.206365+0000 2024-08-19T12:38:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:12.207 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:12.208 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:12.208 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:12.208 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:12.208 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:12.208 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:12.208 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:12.208 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:12.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.205+0000 7f9738ff2640 1 Processor -- start 2024-08-19T12:38:12.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.205+0000 7f9738ff2640 1 -- start start 2024-08-19T12:38:12.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.205+0000 7f9738ff2640 10 monclient: init 2024-08-19T12:38:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.205+0000 7f9738ff2640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.205+0000 7f9738ff2640 10 monclient: _add_conns ranks=[8,7,3,1,0,2,5,6,4] 2024-08-19T12:38:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.205+0000 7f9738ff2640 10 monclient(hunting): picked mon.noname-e con 0x7f9734073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.205+0000 7f9738ff2640 10 monclient(hunting): picked mon.noname-i con 0x7f97340a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.205+0000 7f9738ff2640 10 monclient(hunting): picked mon.noname-g con 0x7f973413f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.206+0000 7f9738ff2640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f973409dae0 con 0x7f973413f2e0 2024-08-19T12:38:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.206+0000 7f9738ff2640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9734136910 con 0x7f97340a57a0 2024-08-19T12:38:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.206+0000 7f9738ff2640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f973409dd50 con 0x7f9734073470 2024-08-19T12:38:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.206+0000 7f9738ff2640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.206+0000 7f9738ff2640 10 monclient(hunting): authenticate will time out at 686.294087s 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.206+0000 7f973259c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f9734073470 0x7f9734136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:54864/0 (socket says 172.21.15.26:54864) 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.206+0000 7f973259c640 1 -- 172.21.15.26:0/1210330544 learned_addr learned my addr 172.21.15.26:0/1210330544 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973259c640 1 -- 172.21.15.26:0/1210330544 >> v1:172.21.15.26:6793/0 conn(0x7f9734073470 legacy=0x7f9734136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f9732d9d640 1 -- 172.21.15.26:0/1210330544 >> v1:172.21.15.104:6790/0 conn(0x7f973413f2e0 legacy=0x7f97341416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f9731d9b640 1 -- 172.21.15.26:0/1210330544 >> v1:172.21.15.104:6792/0 conn(0x7f97340a57a0 legacy=0x7f97340a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 124467725 0 0) 0x7f973409dd50 con 0x7f9734073470 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient(hunting): my global_id is 4157 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f971c003520 con 0x7f9734073470 2024-08-19T12:38:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288223448 0 0) 0x7f973409dae0 con 0x7f973413f2e0 2024-08-19T12:38:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient(hunting): my global_id is 4197 2024-08-19T12:38:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f973409dd50 con 0x7f973413f2e0 2024-08-19T12:38:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2865377964 0 0) 0x7f971c003520 con 0x7f9734073470 2024-08-19T12:38:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 >> v1:172.21.15.104:6792/0 conn(0x7f97340a57a0 legacy=0x7f97340a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 >> v1:172.21.15.104:6790/0 conn(0x7f973413f2e0 legacy=0x7f97341416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: _finish_hunting 0 2024-08-19T12:38:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 monclient: found mon.noname-e 2024-08-19T12:38:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:12.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973409e030 con 0x7f9734073470 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: _finish_auth 0 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: _check_auth_tickets 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9724001ae0 con 0x7f9734073470 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: dump: 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.217 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:12.218 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f9738ff2640 5 monclient: authenticate success, global_id 4157 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2656365835 0 0) 0x7f973409dd50 con 0x7f973413f2e0 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f9738ff2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f972c001e90 con 0x7f973413f2e0 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97240048a0 con 0x7f9734073470 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 1 -- 172.21.15.26:0/1210330544 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9724004e10 con 0x7f9734073470 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f973159a640 10 monclient: dump: 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f9738ff2640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.208+0000 7f9738ff2640 4 set_mon_vals no callback set 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.209+0000 7f9738ff2640 10 monclient: shutdown 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.209+0000 7f9738ff2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.209+0000 7f9738ff2640 1 -- 172.21.15.26:0/1210330544 >> v1:172.21.15.26:6793/0 conn(0x7f9734073470 legacy=0x7f9734136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.209+0000 7f9738ff2640 1 -- 172.21.15.26:0/1210330544 shutdown_connections 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.209+0000 7f9738ff2640 1 -- 172.21.15.26:0/1210330544 >> 172.21.15.26:0/1210330544 conn(0x7f97340abff0 msgr2=0x7f97340ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.209+0000 7f9738ff2640 1 -- 172.21.15.26:0/1210330544 shutdown_connections 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.209+0000 7f9738ff2640 1 -- 172.21.15.26:0/1210330544 wait complete. 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 10 monclient: build_initial_monmap 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 10 monclient: monmap: 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:12.221 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:12.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 1 librados: starting msgr at 2024-08-19T12:38:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 1 librados: starting objecter 2024-08-19T12:38:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 1 Processor -- start 2024-08-19T12:38:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 1 -- start start 2024-08-19T12:38:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 1 librados: setting wanted keys 2024-08-19T12:38:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 1 librados: calling monclient init 2024-08-19T12:38:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 10 monclient: init 2024-08-19T12:38:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 10 monclient: _add_conns ranks=[0,3,2,6,4,5,8,1,7] 2024-08-19T12:38:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 10 monclient(hunting): picked mon.noname-a con 0x7f9734073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 10 monclient(hunting): picked mon.noname-g con 0x7f97340a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 10 monclient(hunting): picked mon.noname-b con 0x7f973413f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:12.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.210+0000 7f9738ff2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97341cef20 con 0x7f9734073470 2024-08-19T12:38:12.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.211+0000 7f9738ff2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9734135200 con 0x7f973413f2e0 2024-08-19T12:38:12.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.211+0000 7f9738ff2640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f973413f090 con 0x7f97340a57a0 2024-08-19T12:38:12.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.211+0000 7f9738ff2640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:12.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.211+0000 7f9738ff2640 10 monclient(hunting): authenticate will time out at 686.299037s 2024-08-19T12:38:12.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.211+0000 7f973259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9734073470 0x7f97341ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57300/0 (socket says 172.21.15.26:57300) 2024-08-19T12:38:12.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.211+0000 7f973259c640 1 -- 172.21.15.26:0/226681121 learned_addr learned my addr 172.21.15.26:0/226681121 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9732d9d640 1 -- 172.21.15.26:0/226681121 >> v1:172.21.15.26:6790/0 conn(0x7f973413f2e0 legacy=0x7f97341d63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f973259c640 1 -- 172.21.15.26:0/226681121 >> v1:172.21.15.26:6789/0 conn(0x7f9734073470 legacy=0x7f97341ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 client.?.objecter ms_handle_connect 0x7f973413f2e0 2024-08-19T12:38:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 client.?.objecter ms_handle_connect 0x7f9734073470 2024-08-19T12:38:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 client.?.objecter ms_handle_connect 0x7f97340a57a0 2024-08-19T12:38:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9731d9b640 1 -- 172.21.15.26:0/226681121 >> v1:172.21.15.104:6790/0 conn(0x7f97340a57a0 legacy=0x7f97341ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4290097926 0 0) 0x7f9734135200 con 0x7f973413f2e0 2024-08-19T12:38:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): my global_id is 4160 2024-08-19T12:38:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9714004eb0 con 0x7f973413f2e0 2024-08-19T12:38:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2915469772 0 0) 0x7f973413f090 con 0x7f97340a57a0 2024-08-19T12:38:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): my global_id is 4206 2024-08-19T12:38:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9734135200 con 0x7f97340a57a0 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3598230470 0 0) 0x7f97341cef20 con 0x7f9734073470 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): my global_id is 4149 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f973413f090 con 0x7f9734073470 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1495279191 0 0) 0x7f9714004eb0 con 0x7f973413f2e0 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97341cef20 con 0x7f973413f2e0 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f972c003990 con 0x7f973413f2e0 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1175056552 0 0) 0x7f973413f090 con 0x7f9734073470 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9714004eb0 con 0x7f9734073470 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9724004d90 con 0x7f9734073470 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2729618738 0 0) 0x7f97341cef20 con 0x7f973413f2e0 2024-08-19T12:38:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 >> v1:172.21.15.104:6790/0 conn(0x7f97340a57a0 legacy=0x7f97341ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 >> v1:172.21.15.26:6789/0 conn(0x7f9734073470 legacy=0x7f97341ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 monclient: found mon.b 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9734131a40 con 0x7f973413f2e0 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.213+0000 7f9722ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 5 monclient: authenticate success, global_id 4160 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 10 monclient: _renew_subs 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 1 -- 172.21.15.26:0/226681121 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9734131d20 con 0x7f973413f2e0 2024-08-19T12:38:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 10 client.4160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 10 monclient: _renew_subs 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 1 -- 172.21.15.26:0/226681121 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f97341324b0 con 0x7f973413f2e0 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 1 librados: init done 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f972c003e60 con 0x7f973413f2e0 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9722ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f972c0043d0 con 0x7f973413f2e0 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9722ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9723fff640 4 set_mon_vals no callback set 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9722ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9722ffd640 10 monclient: dump: 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:12.235 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f972c005b40 con 0x7f973413f2e0 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 4290134622 0 0) 0x7f972c007380 con 0x7f973413f2e0 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9722ffd640 10 client.4160.objecter ms_dispatch 0x7f9734000b90 osd_map(8..8 src has 1..8) 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9722ffd640 3 client.4160.objecter handle_osd_map got epochs [8,8] > 0 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9722ffd640 3 client.4160.objecter handle_osd_map decoding full epoch 8 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9722ffd640 20 client.4160.objecter dump_active .. 0 homeless 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.214+0000 7f9738ff2640 1 -- 172.21.15.26:0/226681121 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9718005080 con 0x7f973413f2e0 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.217+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f972c006480 con 0x7f973413f2e0 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.217+0000 7f9722ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.217+0000 7f9722ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.277+0000 7f9738ff2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.277+0000 7f9738ff2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.277+0000 7f9738ff2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.277+0000 7f9738ff2640 1 -- 172.21.15.26:0/226681121 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9718005370 con 0x7f973413f2e0 2024-08-19T12:38:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.277+0000 7f9722ffd640 1 -- 172.21.15.26:0/226681121 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 1740574615) 0x7f972c005e30 con 0x7f973413f2e0 2024-08-19T12:38:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.277+0000 7f9722ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:12.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.277+0000 7f9722ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:12.280 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:38:12.280 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":8,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","created":"2024-08-19T12:38:08.215243+0000","modified":"2024-08-19T12:38:12.064517+0000","last_up_change":"0.000000","last_in_change":"2024-08-19T12:38:12.064517+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae69b71a-73f9-47e2-b81e-92e9cdd7df0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fc6dcabb-211c-484a-b215-2ef5fe1cbd21","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"88a4da36-5efe-4f64-84c3-28413eba4011","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98c08a65-4fb2-4e5f-a9bc-e8899aed1103","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a4c308b6-5e55-4328-b7b0-1db11fd51d73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"99117114-07ce-4b9b-8df6-39f5fb1b610b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T12:38:12.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.278+0000 7f9738ff2640 10 librados: watch_flush enter 2024-08-19T12:38:12.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.278+0000 7f9738ff2640 10 librados: watch_flush exit 2024-08-19T12:38:12.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.278+0000 7f9738ff2640 20 client.4160.objecter shutdown clearing up homeless session... 2024-08-19T12:38:12.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.278+0000 7f9738ff2640 10 client.4160.objecter successfully canceled tick 2024-08-19T12:38:12.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.278+0000 7f9738ff2640 10 monclient: shutdown 2024-08-19T12:38:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.278+0000 7f9738ff2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.278+0000 7f9738ff2640 1 -- 172.21.15.26:0/226681121 >> v1:172.21.15.26:6790/0 conn(0x7f973413f2e0 legacy=0x7f97341d63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.278+0000 7f9738ff2640 1 -- 172.21.15.26:0/226681121 shutdown_connections 2024-08-19T12:38:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.278+0000 7f9738ff2640 1 -- 172.21.15.26:0/226681121 >> 172.21.15.26:0/226681121 conn(0x7f97340abff0 msgr2=0x7f973412c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.278+0000 7f9738ff2640 1 -- 172.21.15.26:0/226681121 shutdown_connections 2024-08-19T12:38:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.278+0000 7f9738ff2640 1 -- 172.21.15.26:0/226681121 wait complete. 2024-08-19T12:38:12.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:12.278+0000 7f9738ff2640 1 librados: shutdown 2024-08-19T12:38:12.290 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-08-19T12:38:12.290 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-19T12:38:12.897 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-19T12:38:12.994 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.992+0000 7f69ab656640 10 monclient: get_monmap_and_config 2024-08-19T12:38:12.995 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.992+0000 7f69ab656640 10 monclient: build_initial_monmap 2024-08-19T12:38:12.995 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.992+0000 7f69ab656640 10 monclient: monmap: 2024-08-19T12:38:12.995 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-08-19T12:38:12.995 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:12.995 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:38:12.994680+0000 2024-08-19T12:38:12.995 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:38:12.994680+0000 2024-08-19T12:38:12.995 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:12.995 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:12.995 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:12.995 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:12.995 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:12.996 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:12.996 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:12.996 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:12.996 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:12.996 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:12.996 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:12.996 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:12.996 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.993+0000 7f69ab656640 1 Processor -- start 2024-08-19T12:38:12.997 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.993+0000 7f69ab656640 1 -- start start 2024-08-19T12:38:12.998 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.993+0000 7f69ab656640 10 monclient: init 2024-08-19T12:38:12.998 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.994+0000 7f69ab656640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:12.998 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.994+0000 7f69ab656640 10 monclient: _add_conns ranks=[2,8,7,6,4,5,0,3,1] 2024-08-19T12:38:12.998 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.994+0000 7f69ab656640 10 monclient(hunting): picked mon.noname-b con 0x7f69a40a6630 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:12.998 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.994+0000 7f69ab656640 10 monclient(hunting): picked mon.noname-e con 0x7f69a40a44d0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:12.998 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.994+0000 7f69ab656640 10 monclient(hunting): picked mon.noname-i con 0x7f69a40a7120 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:12.998 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.994+0000 7f69ab656640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69a40a3aa0 con 0x7f69a40a6630 2024-08-19T12:38:12.998 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.994+0000 7f69ab656640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69a40a3240 con 0x7f69a40a7120 2024-08-19T12:38:12.998 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.994+0000 7f69ab656640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69a40a37c0 con 0x7f69a40a44d0 2024-08-19T12:38:12.998 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.994+0000 7f69ab656640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:12.998 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.994+0000 7f69ab656640 10 monclient(hunting): authenticate will time out at 687.082355s 2024-08-19T12:38:12.999 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.994+0000 7f69a8bf1640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f69a40a44d0 0x7f69a413e840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:54876/0 (socket says 172.21.15.26:54876) 2024-08-19T12:38:12.999 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.994+0000 7f69a8bf1640 1 -- 172.21.15.26:0/77471171 learned_addr learned my addr 172.21.15.26:0/77471171 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:13.011 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f69a8bf1640 1 -- 172.21.15.26:0/77471171 >> v1:172.21.15.26:6793/0 conn(0x7f69a40a44d0 legacy=0x7f69a413e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:13.011 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f69a93f2640 1 -- 172.21.15.26:0/77471171 >> v1:172.21.15.26:6790/0 conn(0x7f69a40a6630 legacy=0x7f69a40a6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:13.011 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f69a9bf3640 1 -- 172.21.15.26:0/77471171 >> v1:172.21.15.104:6792/0 conn(0x7f69a40a7120 legacy=0x7f69a4141f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:13.011 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2490043287 0 0) 0x7f69a40a37c0 con 0x7f69a40a44d0 2024-08-19T12:38:13.011 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:13.011 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:13.012 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:13.012 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 10 monclient(hunting): my global_id is 4166 2024-08-19T12:38:13.012 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6990003520 con 0x7f69a40a44d0 2024-08-19T12:38:13.012 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 330749134 0 0) 0x7f69a40a3aa0 con 0x7f69a40a6630 2024-08-19T12:38:13.012 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:13.012 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:13.012 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:13.012 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 10 monclient(hunting): my global_id is 4169 2024-08-19T12:38:13.012 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69a40a37c0 con 0x7f69a40a6630 2024-08-19T12:38:13.012 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1135248709 0 0) 0x7f69a40a3240 con 0x7f69a40a7120 2024-08-19T12:38:13.012 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:13.013 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:13.013 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:13.013 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 10 monclient(hunting): my global_id is 4156 2024-08-19T12:38:13.013 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.996+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69a40a3aa0 con 0x7f69a40a7120 2024-08-19T12:38:13.013 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3925493930 0 0) 0x7f6990003520 con 0x7f69a40a44d0 2024-08-19T12:38:13.013 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 >> v1:172.21.15.104:6792/0 conn(0x7f69a40a7120 legacy=0x7f69a4141f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:13.013 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 >> v1:172.21.15.26:6790/0 conn(0x7f69a40a6630 legacy=0x7f69a40a6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:13.013 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:13.013 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 1 monclient: found mon.noname-e 2024-08-19T12:38:13.013 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:13.013 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:13.014 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a4069750 con 0x7f69a40a44d0 2024-08-19T12:38:13.014 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 10 monclient: _finish_auth 0 2024-08-19T12:38:13.014 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:13.014 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:13.014 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6994001ae0 con 0x7f69a40a44d0 2024-08-19T12:38:13.014 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f69ab656640 5 monclient: authenticate success, global_id 4166 2024-08-19T12:38:13.014 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f69ab656640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:13.014 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:13.015 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:13.015 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 10 monclient: dump: 2024-08-19T12:38:13.015 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:13.015 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:13.015 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:13.015 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:13.015 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:13.015 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:13.015 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:13.015 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f69ab656640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69940048a0 con 0x7f69a40a44d0 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 1 -- 172.21.15.26:0/77471171 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6994004e10 con 0x7f69a40a44d0 2024-08-19T12:38:13.016 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f69ab656640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f69ab656640 4 set_mon_vals no callback set 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f699bfff640 10 monclient: dump: 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:13.017 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:13.018 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:13.018 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:13.018 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:13.018 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:13.018 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:13.018 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:13.018 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f69ab656640 10 monclient: shutdown 2024-08-19T12:38:13.018 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f69ab656640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:13.018 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f69ab656640 1 -- 172.21.15.26:0/77471171 >> v1:172.21.15.26:6793/0 conn(0x7f69a40a44d0 legacy=0x7f69a413e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:13.019 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f69ab656640 1 -- 172.21.15.26:0/77471171 shutdown_connections 2024-08-19T12:38:13.019 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f69ab656640 1 -- 172.21.15.26:0/77471171 >> 172.21.15.26:0/77471171 conn(0x7f69a40a8e20 msgr2=0x7f69a4068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:13.019 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.997+0000 7f69ab656640 1 -- 172.21.15.26:0/77471171 shutdown_connections 2024-08-19T12:38:13.019 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.998+0000 7f69ab656640 1 -- 172.21.15.26:0/77471171 wait complete. 2024-08-19T12:38:13.019 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.998+0000 7f69ab656640 10 monclient: build_initial_monmap 2024-08-19T12:38:13.019 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.998+0000 7f69ab656640 10 monclient: monmap: 2024-08-19T12:38:13.019 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-08-19T12:38:13.019 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:13.019 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:13.019 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-08-19T12:38:13.019 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:13.019 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:13.020 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:13.020 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:13.020 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:13.020 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:13.020 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:13.020 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:13.020 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:13.020 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:13.020 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:13.020 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:13.020 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.998+0000 7f69ab656640 1 librados: starting msgr at 2024-08-19T12:38:13.020 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.998+0000 7f69ab656640 1 librados: starting objecter 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.998+0000 7f69ab656640 1 Processor -- start 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.998+0000 7f69ab656640 1 -- start start 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.998+0000 7f69ab656640 1 librados: setting wanted keys 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.998+0000 7f69ab656640 1 librados: calling monclient init 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.998+0000 7f69ab656640 10 monclient: init 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.999+0000 7f69ab656640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.999+0000 7f69ab656640 10 monclient: _add_conns ranks=[4,0,2,8,1,6,3,7,5] 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.999+0000 7f69ab656640 10 monclient(hunting): picked mon.noname-c con 0x7f69a40a44d0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.999+0000 7f69ab656640 10 monclient(hunting): picked mon.noname-a con 0x7f69a40a6630 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.999+0000 7f69ab656640 10 monclient(hunting): picked mon.noname-b con 0x7f69a40a7120 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.999+0000 7f69ab656640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69a4073700 con 0x7f69a40a6630 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.999+0000 7f69ab656640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69a4126f10 con 0x7f69a40a7120 2024-08-19T12:38:13.021 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.999+0000 7f69ab656640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69a4124bb0 con 0x7f69a40a44d0 2024-08-19T12:38:13.022 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.999+0000 7f69ab656640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:13.022 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.999+0000 7f69a8bf1640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f69a40a6630 0x7f69a41d2bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57302/0 (socket says 172.21.15.26:57302) 2024-08-19T12:38:13.022 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.999+0000 7f69ab656640 10 monclient(hunting): authenticate will time out at 687.087494s 2024-08-19T12:38:13.022 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:12.999+0000 7f69a8bf1640 1 -- 172.21.15.26:0/743576756 learned_addr learned my addr 172.21.15.26:0/743576756 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:13.022 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f69a93f2640 1 -- 172.21.15.26:0/743576756 >> v1:172.21.15.26:6791/0 conn(0x7f69a40a44d0 legacy=0x7f69a4139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:13.022 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 client.?.objecter ms_handle_connect 0x7f69a40a44d0 2024-08-19T12:38:13.022 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:13.022 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 client.?.objecter ms_handle_connect 0x7f69a40a6630 2024-08-19T12:38:13.022 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:13.022 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f69a8bf1640 1 -- 172.21.15.26:0/743576756 >> v1:172.21.15.26:6789/0 conn(0x7f69a40a6630 legacy=0x7f69a41d2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:13.022 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 client.?.objecter ms_handle_connect 0x7f69a40a7120 2024-08-19T12:38:13.022 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f69a9bf3640 1 -- 172.21.15.26:0/743576756 >> v1:172.21.15.26:6790/0 conn(0x7f69a40a7120 legacy=0x7f69a41d6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3270061411 0 0) 0x7f69a4124bb0 con 0x7f69a40a44d0 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 monclient(hunting): my global_id is 4171 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6988004eb0 con 0x7f69a40a44d0 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577221483 0 0) 0x7f69a4069750 con 0x7f69a40a6630 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 monclient(hunting): my global_id is 4158 2024-08-19T12:38:13.023 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69a4124bb0 con 0x7f69a40a6630 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3987690519 0 0) 0x7f69a4126f10 con 0x7f69a40a7120 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 10 monclient(hunting): my global_id is 4178 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.001+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69a4069750 con 0x7f69a40a7120 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3586123105 0 0) 0x7f6988004eb0 con 0x7f69a40a44d0 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69a4126f10 con 0x7f69a40a44d0 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f69a0003950 con 0x7f69a40a44d0 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:13.024 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:13.025 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:13.026 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:13.026 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:13.026 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:13.026 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4072901163 0 0) 0x7f69a4124bb0 con 0x7f69a40a6630 2024-08-19T12:38:13.026 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6988004eb0 con 0x7f69a40a6630 2024-08-19T12:38:13.026 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3311234786 0 0) 0x7f69a4069750 con 0x7f69a40a7120 2024-08-19T12:38:13.026 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69a4124bb0 con 0x7f69a40a7120 2024-08-19T12:38:13.026 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6994004d90 con 0x7f69a40a6630 2024-08-19T12:38:13.026 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:13.026 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:13.026 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:13.026 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:13.027 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:13.027 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:13.027 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:13.027 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:13.027 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:13.027 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:13.027 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:13.027 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f699c002020 con 0x7f69a40a7120 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:13.028 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:13.029 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 149261941 0 0) 0x7f6988004eb0 con 0x7f69a40a6630 2024-08-19T12:38:13.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 >> v1:172.21.15.26:6791/0 conn(0x7f69a40a44d0 legacy=0x7f69a4139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:13.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 >> v1:172.21.15.26:6790/0 conn(0x7f69a40a7120 legacy=0x7f69a41d6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:13.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:13.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 monclient: found mon.a 2024-08-19T12:38:13.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:13.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:13.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a4126670 con 0x7f69a40a6630 2024-08-19T12:38:13.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:13.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:13.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:13.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f69ab656640 5 monclient: authenticate success, global_id 4158 2024-08-19T12:38:13.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f69ab656640 10 monclient: _renew_subs 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f69ab656640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f69ab656640 1 -- 172.21.15.26:0/743576756 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f69a4139c40 con 0x7f69a40a6630 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6994001b00 con 0x7f69a40a6630 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6994005740 con 0x7f69a40a6630 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 monclient: dump: 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:13.031 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6994005c70 con 0x7f69a40a6630 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f699affd640 4 set_mon_vals no callback set 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f69ab656640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:13.032 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f69ab656640 10 monclient: _renew_subs 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f69ab656640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f69ab656640 1 -- 172.21.15.26:0/743576756 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f69a4124330 con 0x7f69a40a6630 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f69ab656640 1 librados: init done 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 4290134622 0 0) 0x7f69940065b0 con 0x7f69a40a6630 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 10 client.4158.objecter ms_dispatch 0x7f69a4000b90 osd_map(8..8 src has 1..8) 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 3 client.4158.objecter handle_osd_map got epochs [8,8] > 0 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.002+0000 7f6999ffb640 3 client.4158.objecter handle_osd_map decoding full epoch 8 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.003+0000 7f6999ffb640 20 client.4158.objecter dump_active .. 0 homeless 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.003+0000 7f69ab656640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.003+0000 7f69ab656640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.003+0000 7f69ab656640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.003+0000 7f69ab656640 1 -- 172.21.15.26:0/743576756 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69a409f0e0 con 0x7f69a40a6630 2024-08-19T12:38:13.033 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.005+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f69a4073700 con 0x7f69a40a6630 2024-08-19T12:38:13.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.005+0000 7f6999ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:13.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.005+0000 7f6999ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:13.069 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.067+0000 7f69ab656640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:13.069 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.067+0000 7f69ab656640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:13.069 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.067+0000 7f69ab656640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:13.070 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.067+0000 7f69ab656640 1 -- 172.21.15.26:0/743576756 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69a406c8d0 con 0x7f69a40a6630 2024-08-19T12:38:13.070 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.067+0000 7f6999ffb640 1 -- 172.21.15.26:0/743576756 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 1740574615) 0x7f6994005f60 con 0x7f69a40a6630 2024-08-19T12:38:13.070 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.067+0000 7f6999ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:13.070 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.067+0000 7f6999ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:13.070 INFO:teuthology.misc.health.smithi026.stdout: 2024-08-19T12:38:13.070 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":8,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","created":"2024-08-19T12:38:08.215243+0000","modified":"2024-08-19T12:38:12.064517+0000","last_up_change":"0.000000","last_in_change":"2024-08-19T12:38:12.064517+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae69b71a-73f9-47e2-b81e-92e9cdd7df0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fc6dcabb-211c-484a-b215-2ef5fe1cbd21","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"88a4da36-5efe-4f64-84c3-28413eba4011","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98c08a65-4fb2-4e5f-a9bc-e8899aed1103","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a4c308b6-5e55-4328-b7b0-1db11fd51d73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"99117114-07ce-4b9b-8df6-39f5fb1b610b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T12:38:13.072 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.068+0000 7f69ab656640 10 librados: watch_flush enter 2024-08-19T12:38:13.072 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.068+0000 7f69ab656640 10 librados: watch_flush exit 2024-08-19T12:38:13.072 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.068+0000 7f69ab656640 20 client.4158.objecter shutdown clearing up homeless session... 2024-08-19T12:38:13.072 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.068+0000 7f69ab656640 10 client.4158.objecter successfully canceled tick 2024-08-19T12:38:13.072 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.068+0000 7f69ab656640 10 monclient: shutdown 2024-08-19T12:38:13.072 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.068+0000 7f69ab656640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:13.072 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.068+0000 7f69ab656640 1 -- 172.21.15.26:0/743576756 >> v1:172.21.15.26:6789/0 conn(0x7f69a40a6630 legacy=0x7f69a41d2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:13.072 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.068+0000 7f69ab656640 1 -- 172.21.15.26:0/743576756 shutdown_connections 2024-08-19T12:38:13.072 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.068+0000 7f69ab656640 1 -- 172.21.15.26:0/743576756 >> 172.21.15.26:0/743576756 conn(0x7f69a40a8e20 msgr2=0x7f69a40aa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:13.072 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.068+0000 7f69ab656640 1 -- 172.21.15.26:0/743576756 shutdown_connections 2024-08-19T12:38:13.073 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.068+0000 7f69ab656640 1 -- 172.21.15.26:0/743576756 wait complete. 2024-08-19T12:38:13.073 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:13.068+0000 7f69ab656640 1 librados: shutdown 2024-08-19T12:38:13.079 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-08-19T12:38:13.281 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-08-19T12:38:13.279+0000 7f8b1920d640 -1 Falling back to public interface 2024-08-19T12:38:13.289 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-08-19T12:38:13.287+0000 7fa6c4818640 -1 Falling back to public interface 2024-08-19T12:38:13.556 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-08-19T12:38:13.555+0000 7f038129a640 -1 Falling back to public interface 2024-08-19T12:38:13.583 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-08-19T12:38:13.582+0000 7f9287a0d640 -1 Falling back to public interface 2024-08-19T12:38:13.766 INFO:tasks.ceph.osd.3.smithi104.stderr:2024-08-19T12:38:13.765+0000 7f731e3ba640 -1 Falling back to public interface 2024-08-19T12:38:13.782 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-08-19T12:38:13.780+0000 7f318f325640 -1 Falling back to public interface 2024-08-19T12:38:14.947 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-08-19T12:38:14.945+0000 7f7a29131640 -1 mgr.server handle_report got status from non-daemon mon.f 2024-08-19T12:38:14.947 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-08-19T12:38:14.947+0000 7f7a29131640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-08-19T12:38:14.950 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-08-19T12:38:14.949+0000 7f7a29131640 -1 mgr.server handle_report got status from non-daemon mon.g 2024-08-19T12:38:14.950 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-08-19T12:38:14.950+0000 7f7a29131640 -1 mgr.server handle_report got status from non-daemon mon.d 2024-08-19T12:38:14.951 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-08-19T12:38:14.950+0000 7f7a29131640 -1 mgr.server handle_report got status from non-daemon mon.h 2024-08-19T12:38:14.955 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-08-19T12:38:14.954+0000 7f7a29131640 -1 mgr.server handle_report got status from non-daemon mon.i 2024-08-19T12:38:14.955 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-08-19T12:38:14.954+0000 7f7a29131640 -1 mgr.server handle_report got status from non-daemon mon.e 2024-08-19T12:38:14.955 INFO:tasks.ceph.mgr.x.smithi104.stderr:2024-08-19T12:38:14.955+0000 7f7a29131640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-08-19T12:38:17.085 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-08-19T12:38:17.083+0000 7f8b1920d640 -1 osd.0 0 log_to_monitors true 2024-08-19T12:38:17.291 INFO:tasks.ceph.osd.3.smithi104.stderr:2024-08-19T12:38:17.290+0000 7f731e3ba640 -1 osd.3 0 log_to_monitors true 2024-08-19T12:38:17.332 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-08-19T12:38:17.332+0000 7f038129a640 -1 osd.5 0 log_to_monitors true 2024-08-19T12:38:17.582 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-08-19T12:38:17.580+0000 7fa6c4818640 -1 osd.1 0 log_to_monitors true 2024-08-19T12:38:17.616 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-08-19T12:38:17.616+0000 7f9287a0d640 -1 osd.4 0 log_to_monitors true 2024-08-19T12:38:17.800 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-08-19T12:38:17.798+0000 7f318f325640 -1 osd.2 0 log_to_monitors true 2024-08-19T12:38:19.284 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-08-19T12:38:19.282+0000 7f8b15307640 -1 osd.0 0 waiting for initial osdmap 2024-08-19T12:38:19.287 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-08-19T12:38:19.285+0000 7f8b0c912640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T12:38:19.683 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-19T12:38:19.702 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-08-19T12:38:19.702+0000 7f318b2bb640 -1 osd.2 0 waiting for initial osdmap 2024-08-19T12:38:19.709 INFO:tasks.ceph.osd.2.smithi026.stderr:2024-08-19T12:38:19.708+0000 7f31870e2640 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T12:38:19.798 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.797+0000 7f84e9f6b640 10 monclient: get_monmap_and_config 2024-08-19T12:38:19.798 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.797+0000 7f84e9f6b640 10 monclient: build_initial_monmap 2024-08-19T12:38:19.798 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.797+0000 7f84e9f6b640 10 monclient: monmap: 2024-08-19T12:38:19.798 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-08-19T12:38:19.798 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:19.798 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:38:19.798142+0000 2024-08-19T12:38:19.798 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:38:19.798142+0000 2024-08-19T12:38:19.798 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:19.798 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:19.798 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:19.799 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:19.799 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:19.799 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:19.799 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:19.799 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:19.799 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:19.799 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:19.799 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:19.799 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:19.800 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 1 Processor -- start 2024-08-19T12:38:19.800 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 1 -- start start 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 10 monclient: init 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 10 monclient: _add_conns ranks=[6,8,1,2,0,3,4,5,7] 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 10 monclient(hunting): picked mon.noname-d con 0x7f84e40a6630 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 10 monclient(hunting): picked mon.noname-e con 0x7f84e40a44d0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 10 monclient(hunting): picked mon.noname-f con 0x7f84e40a7120 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e40a3aa0 con 0x7f84e40a7120 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e40a3240 con 0x7f84e40a6630 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e40a37c0 con 0x7f84e40a44d0 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e9f6b640 10 monclient(hunting): authenticate will time out at 693.885771s 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e2ffd640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f84e40a44d0 0x7f84e413e840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:43728/0 (socket says 172.21.15.26:43728) 2024-08-19T12:38:19.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.798+0000 7f84e2ffd640 1 -- 172.21.15.26:0/809174041 learned_addr learned my addr 172.21.15.26:0/809174041 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:19.805 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e2ffd640 1 -- 172.21.15.26:0/809174041 >> v1:172.21.15.26:6793/0 conn(0x7f84e40a44d0 legacy=0x7f84e413e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:19.805 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e37fe640 1 -- 172.21.15.26:0/809174041 >> v1:172.21.15.26:6792/0 conn(0x7f84e40a6630 legacy=0x7f84e40a6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e3fff640 1 -- 172.21.15.26:0/809174041 >> v1:172.21.15.104:6789/0 conn(0x7f84e40a7120 legacy=0x7f84e4141f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3406590804 0 0) 0x7f84e40a37c0 con 0x7f84e40a44d0 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient(hunting): my global_id is 4229 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84cc003520 con 0x7f84e40a44d0 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 882559026 0 0) 0x7f84e40a3240 con 0x7f84e40a6630 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient(hunting): my global_id is 4218 2024-08-19T12:38:19.806 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84e40a37c0 con 0x7f84e40a6630 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 90305930 0 0) 0x7f84e40a3aa0 con 0x7f84e40a7120 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient(hunting): my global_id is 4195 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84e40a3240 con 0x7f84e40a7120 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 702008507 0 0) 0x7f84cc003520 con 0x7f84e40a44d0 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 >> v1:172.21.15.26:6792/0 conn(0x7f84e40a6630 legacy=0x7f84e40a6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 >> v1:172.21.15.104:6789/0 conn(0x7f84e40a7120 legacy=0x7f84e4141f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 monclient: found mon.noname-e 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:19.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e4069750 con 0x7f84e40a44d0 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f84d8001ae0 con 0x7f84e40a44d0 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient: dump: 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:19.808 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e9f6b640 5 monclient: authenticate success, global_id 4229 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e9f6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f84d80048a0 con 0x7f84e40a44d0 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:19.809 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 1 -- 172.21.15.26:0/809174041 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f84d8004e10 con 0x7f84e40a44d0 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e9f6b640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e9f6b640 4 set_mon_vals no callback set 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.801+0000 7f84e27fc640 10 monclient: dump: 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:19.810 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.802+0000 7f84e9f6b640 10 monclient: shutdown 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.802+0000 7f84e9f6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.802+0000 7f84e9f6b640 1 -- 172.21.15.26:0/809174041 >> v1:172.21.15.26:6793/0 conn(0x7f84e40a44d0 legacy=0x7f84e413e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.802+0000 7f84e9f6b640 1 -- 172.21.15.26:0/809174041 shutdown_connections 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.802+0000 7f84e9f6b640 1 -- 172.21.15.26:0/809174041 >> 172.21.15.26:0/809174041 conn(0x7f84e40a8e20 msgr2=0x7f84e4068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.802+0000 7f84e9f6b640 1 -- 172.21.15.26:0/809174041 shutdown_connections 2024-08-19T12:38:19.811 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.802+0000 7f84e9f6b640 1 -- 172.21.15.26:0/809174041 wait complete. 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.803+0000 7f84e9f6b640 10 monclient: build_initial_monmap 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.803+0000 7f84e9f6b640 10 monclient: monmap: 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:19.812 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.803+0000 7f84e9f6b640 1 librados: starting msgr at 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.803+0000 7f84e9f6b640 1 librados: starting objecter 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.803+0000 7f84e9f6b640 1 Processor -- start 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.803+0000 7f84e9f6b640 1 -- start start 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.803+0000 7f84e9f6b640 1 librados: setting wanted keys 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.803+0000 7f84e9f6b640 1 librados: calling monclient init 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.803+0000 7f84e9f6b640 10 monclient: init 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.804+0000 7f84e9f6b640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.804+0000 7f84e9f6b640 10 monclient: _add_conns ranks=[4,8,5,0,1,3,2,6,7] 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.804+0000 7f84e9f6b640 10 monclient(hunting): picked mon.noname-c con 0x7f84e40a44d0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:19.813 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.804+0000 7f84e9f6b640 10 monclient(hunting): picked mon.noname-e con 0x7f84e40a6630 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:19.814 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.804+0000 7f84e9f6b640 10 monclient(hunting): picked mon.noname-h con 0x7f84e40a7120 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:19.814 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.804+0000 7f84e9f6b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e4124db0 con 0x7f84e40a44d0 2024-08-19T12:38:19.814 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.804+0000 7f84e9f6b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e40a1fd0 con 0x7f84e40a7120 2024-08-19T12:38:19.814 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.804+0000 7f84e9f6b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e4073990 con 0x7f84e40a6630 2024-08-19T12:38:19.814 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.804+0000 7f84e9f6b640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:19.814 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.804+0000 7f84e9f6b640 10 monclient(hunting): authenticate will time out at 693.891454s 2024-08-19T12:38:19.814 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.804+0000 7f84e2ffd640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f84e40a6630 0x7f84e41d2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:43736/0 (socket says 172.21.15.26:43736) 2024-08-19T12:38:19.814 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.804+0000 7f84e2ffd640 1 -- 172.21.15.26:0/442503413 learned_addr learned my addr 172.21.15.26:0/442503413 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:19.817 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84e2ffd640 1 -- 172.21.15.26:0/442503413 >> v1:172.21.15.26:6793/0 conn(0x7f84e40a6630 legacy=0x7f84e41d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84e37fe640 1 -- 172.21.15.26:0/442503413 >> v1:172.21.15.26:6791/0 conn(0x7f84e40a44d0 legacy=0x7f84e4139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 client.?.objecter ms_handle_connect 0x7f84e40a6630 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 client.?.objecter ms_handle_connect 0x7f84e40a44d0 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84e3fff640 1 -- 172.21.15.26:0/442503413 >> v1:172.21.15.104:6791/0 conn(0x7f84e40a7120 legacy=0x7f84e41d66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 client.?.objecter ms_handle_connect 0x7f84e40a7120 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700993256 0 0) 0x7f84e4073990 con 0x7f84e40a6630 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 monclient(hunting): my global_id is 4238 2024-08-19T12:38:19.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84d0004eb0 con 0x7f84e40a6630 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1862551282 0 0) 0x7f84e4124db0 con 0x7f84e40a44d0 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 10 monclient(hunting): my global_id is 4216 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.806+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84e4073990 con 0x7f84e40a44d0 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633352065 0 0) 0x7f84e40a1fd0 con 0x7f84e40a7120 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): my global_id is 4208 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84e4124db0 con 0x7f84e40a7120 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3611977962 0 0) 0x7f84d0004eb0 con 0x7f84e40a6630 2024-08-19T12:38:19.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f84e40a1fd0 con 0x7f84e40a6630 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f84d8004d90 con 0x7f84e40a6630 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): dump: 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:19.820 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1732070782 0 0) 0x7f84e4073990 con 0x7f84e40a44d0 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f84d0004eb0 con 0x7f84e40a44d0 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f84d4003610 con 0x7f84e40a44d0 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): dump: 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:19.821 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:19.822 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 875595122 0 0) 0x7f84e4124db0 con 0x7f84e40a7120 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f84e4073990 con 0x7f84e40a7120 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f84dc002070 con 0x7f84e40a7120 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient(hunting): dump: 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:19.823 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3936996849 0 0) 0x7f84e40a1fd0 con 0x7f84e40a6630 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 >> v1:172.21.15.104:6791/0 conn(0x7f84e40a7120 legacy=0x7f84e41d66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 >> v1:172.21.15.26:6791/0 conn(0x7f84e40a44d0 legacy=0x7f84e4139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 monclient: found mon.e 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e4127110 con 0x7f84e40a6630 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient: _finish_auth 0 2024-08-19T12:38:19.824 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84e9f6b640 5 monclient: authenticate success, global_id 4238 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84e9f6b640 10 monclient: _renew_subs 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84e9f6b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84e9f6b640 1 -- 172.21.15.26:0/442503413 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f84e4126870 con 0x7f84e40a6630 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84e9f6b640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84e9f6b640 10 monclient: _renew_subs 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84e9f6b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84e9f6b640 1 -- 172.21.15.26:0/442503413 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f84e4125af0 con 0x7f84e40a6630 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84e9f6b640 1 librados: init done 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f84d8001d30 con 0x7f84e40a6630 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f84d8001a80 con 0x7f84e40a6630 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:19.825 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84e1ffb640 4 set_mon_vals no callback set 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.807+0000 7f84cbfff640 10 monclient: dump: 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:19.826 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.808+0000 7f84e9f6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.808+0000 7f84e9f6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.808+0000 7f84e9f6b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.808+0000 7f84e9f6b640 1 -- 172.21.15.26:0/442503413 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84c4005080 con 0x7f84e40a6630 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.809+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 4) ==== 49789+0+0 (unknown 356348996 0 0) 0x7f84d80119d0 con 0x7f84e40a6630 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.809+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(11..11 src has 1..11) ==== 2067+0+0 (unknown 3180048920 0 0) 0x7f84d804d410 con 0x7f84e40a6630 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.809+0000 7f84cbfff640 10 client.4238.objecter ms_dispatch 0x7f84e4000b90 osd_map(11..11 src has 1..11) 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.809+0000 7f84cbfff640 3 client.4238.objecter handle_osd_map got epochs [11,11] > 0 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.809+0000 7f84cbfff640 3 client.4238.objecter handle_osd_map decoding full epoch 11 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.809+0000 7f84cbfff640 20 client.4238.objecter dump_active .. 0 homeless 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.809+0000 7f84e37fe640 10 monclient: get_auth_request con 0x7f84d0041930 auth_method 0 2024-08-19T12:38:19.827 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.812+0000 7f84cbfff640 10 client.4238.objecter ms_handle_connect 0x7f84d0041930 2024-08-19T12:38:19.828 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.813+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f84d8017560 con 0x7f84e40a6630 2024-08-19T12:38:19.828 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.813+0000 7f84cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:19.828 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:19.813+0000 7f84cbfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:20.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.029+0000 7f84e9f6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:20.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.029+0000 7f84e9f6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:20.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.029+0000 7f84e9f6b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:20.030 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.029+0000 7f84e9f6b640 1 -- 172.21.15.26:0/442503413 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84c4005370 con 0x7f84e40a6630 2024-08-19T12:38:20.031 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.030+0000 7f84cbfff640 1 -- 172.21.15.26:0/442503413 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+5863 (unknown 238102980 0 2258439336) 0x7f84d801b400 con 0x7f84e40a6630 2024-08-19T12:38:20.031 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.030+0000 7f84cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:20.031 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.030+0000 7f84cbfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:20.031 INFO:teuthology.misc.health.smithi026.stdout: 2024-08-19T12:38:20.031 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":11,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","created":"2024-08-19T12:38:08.215243+0000","modified":"2024-08-19T12:38:19.694249+0000","last_up_change":"2024-08-19T12:38:19.694249+0000","last_in_change":"2024-08-19T12:38:12.064517+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ae69b71a-73f9-47e2-b81e-92e9cdd7df0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6800","nonce":3531533167}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6801","nonce":3531533167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6803","nonce":3531533167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6802","nonce":3531533167}]},"public_addr":"172.21.15.26:6800/3531533167","cluster_addr":"172.21.15.26:6801/3531533167","heartbeat_back_addr":"172.21.15.26:6803/3531533167","heartbeat_front_addr":"172.21.15.26:6802/3531533167","state":["exists","up"]},{"osd":1,"uuid":"fc6dcabb-211c-484a-b215-2ef5fe1cbd21","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"88a4da36-5efe-4f64-84c3-28413eba4011","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98c08a65-4fb2-4e5f-a9bc-e8899aed1103","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a4c308b6-5e55-4328-b7b0-1db11fd51d73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"99117114-07ce-4b9b-8df6-39f5fb1b610b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T12:38:20.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.032+0000 7f84e9f6b640 10 librados: watch_flush enter 2024-08-19T12:38:20.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.032+0000 7f84e9f6b640 10 librados: watch_flush exit 2024-08-19T12:38:20.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.032+0000 7f84e9f6b640 20 client.4238.objecter shutdown clearing up homeless session... 2024-08-19T12:38:20.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.032+0000 7f84e9f6b640 10 client.4238.objecter successfully canceled tick 2024-08-19T12:38:20.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.032+0000 7f84e9f6b640 1 -- 172.21.15.26:0/442503413 >> v1:172.21.15.104:6800/899887506 conn(0x7f84d0041930 legacy=0x7f84d0043df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:20.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.032+0000 7f84e9f6b640 10 monclient: shutdown 2024-08-19T12:38:20.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.032+0000 7f84e9f6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:20.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.032+0000 7f84e9f6b640 1 -- 172.21.15.26:0/442503413 >> v1:172.21.15.26:6793/0 conn(0x7f84e40a6630 legacy=0x7f84e41d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:20.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.032+0000 7f84e9f6b640 1 -- 172.21.15.26:0/442503413 shutdown_connections 2024-08-19T12:38:20.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.032+0000 7f84e9f6b640 1 -- 172.21.15.26:0/442503413 >> 172.21.15.26:0/442503413 conn(0x7f84e40a8e20 msgr2=0x7f84e40aa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:20.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.033+0000 7f84e9f6b640 1 -- 172.21.15.26:0/442503413 shutdown_connections 2024-08-19T12:38:20.034 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.033+0000 7f84e9f6b640 1 -- 172.21.15.26:0/442503413 wait complete. 2024-08-19T12:38:20.035 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:20.033+0000 7f84e9f6b640 1 librados: shutdown 2024-08-19T12:38:20.045 DEBUG:teuthology.misc:1 of 6 OSDs are up 2024-08-19T12:38:20.316 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-08-19T12:38:20.315+0000 7fa6c01e7640 -1 osd.1 0 waiting for initial osdmap 2024-08-19T12:38:20.319 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-08-19T12:38:20.318+0000 7fa6b8006640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T12:38:20.593 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-08-19T12:38:20.592+0000 7f9283b38640 -1 osd.4 0 waiting for initial osdmap 2024-08-19T12:38:20.596 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-08-19T12:38:20.596+0000 7f927b143640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T12:38:20.614 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-08-19T12:38:20.612+0000 7f037d230640 -1 osd.5 0 waiting for initial osdmap 2024-08-19T12:38:20.617 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-08-19T12:38:20.616+0000 7f037504f640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T12:38:20.633 INFO:tasks.ceph.osd.3.smithi104.stderr:2024-08-19T12:38:20.633+0000 7f731a350640 -1 osd.3 0 waiting for initial osdmap 2024-08-19T12:38:20.637 INFO:tasks.ceph.osd.3.smithi104.stderr:2024-08-19T12:38:20.636+0000 7f731216f640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-19T12:38:26.652 DEBUG:teuthology.orchestra.run.smithi026:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-19T12:38:26.755 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.753+0000 7ff9cf511640 10 monclient: get_monmap_and_config 2024-08-19T12:38:26.755 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.753+0000 7ff9cf511640 10 monclient: build_initial_monmap 2024-08-19T12:38:26.756 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.753+0000 7ff9cf511640 10 monclient: monmap: 2024-08-19T12:38:26.756 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-08-19T12:38:26.756 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:26.756 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:38:26.754808+0000 2024-08-19T12:38:26.756 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:38:26.754808+0000 2024-08-19T12:38:26.756 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:26.756 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:26.756 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:26.757 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:26.757 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:26.757 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:26.757 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:26.758 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:26.758 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:26.758 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:26.758 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:26.758 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:26.758 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.754+0000 7ff9cf511640 1 Processor -- start 2024-08-19T12:38:26.759 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.754+0000 7ff9cf511640 1 -- start start 2024-08-19T12:38:26.759 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.754+0000 7ff9cf511640 10 monclient: init 2024-08-19T12:38:26.776 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9cf511640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:26.776 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9cf511640 10 monclient: _add_conns ranks=[7,4,6,8,2,3,1,5,0] 2024-08-19T12:38:26.776 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9cf511640 10 monclient(hunting): picked mon.noname-i con 0x7ff9c80a65a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:26.776 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9cf511640 10 monclient(hunting): picked mon.noname-c con 0x7ff9c80a7090 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:26.776 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9cf511640 10 monclient(hunting): picked mon.noname-d con 0x7ff9c813fed0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:26.776 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9cf511640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c806ef70 con 0x7ff9c80a7090 2024-08-19T12:38:26.777 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9cf511640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c806ed30 con 0x7ff9c813fed0 2024-08-19T12:38:26.777 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9cf511640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c80a7740 con 0x7ff9c80a65a0 2024-08-19T12:38:26.777 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9cf511640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:26.777 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9cf511640 10 monclient(hunting): authenticate will time out at 700.842397s 2024-08-19T12:38:26.777 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9ccaac640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7ff9c80a7090 0x7ff9c813e7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:50724/0 (socket says 172.21.15.26:50724) 2024-08-19T12:38:26.777 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9cdaae640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7ff9c813fed0 0x7ff9c81422c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:44542/0 (socket says 172.21.15.26:44542) 2024-08-19T12:38:26.777 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.755+0000 7ff9ccaac640 1 -- 172.21.15.26:0/3325348635 learned_addr learned my addr 172.21.15.26:0/3325348635 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:26.778 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.757+0000 7ff9cdaae640 1 -- 172.21.15.26:0/3325348635 >> v1:172.21.15.26:6792/0 conn(0x7ff9c813fed0 legacy=0x7ff9c81422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:26.778 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.757+0000 7ff9ccaac640 1 -- 172.21.15.26:0/3325348635 >> v1:172.21.15.26:6791/0 conn(0x7ff9c80a7090 legacy=0x7ff9c813e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:26.778 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9cd2ad640 1 -- 172.21.15.26:0/3325348635 >> v1:172.21.15.104:6792/0 conn(0x7ff9c80a65a0 legacy=0x7ff9c80a6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:26.778 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 605777510 0 0) 0x7ff9c806ef70 con 0x7ff9c80a7090 2024-08-19T12:38:26.778 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:26.778 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:26.778 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:26.779 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient(hunting): my global_id is 4243 2024-08-19T12:38:26.779 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9b4003520 con 0x7ff9c80a7090 2024-08-19T12:38:26.779 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 248846205 0 0) 0x7ff9c806ed30 con 0x7ff9c813fed0 2024-08-19T12:38:26.779 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:26.779 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:26.779 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:26.779 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient(hunting): my global_id is 4245 2024-08-19T12:38:26.779 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9c806ef70 con 0x7ff9c813fed0 2024-08-19T12:38:26.780 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3965752797 0 0) 0x7ff9c80a7740 con 0x7ff9c80a65a0 2024-08-19T12:38:26.780 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:26.780 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:26.780 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:26.780 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient(hunting): my global_id is 4201 2024-08-19T12:38:26.781 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9c806ed30 con 0x7ff9c80a65a0 2024-08-19T12:38:26.781 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 522151301 0 0) 0x7ff9b4003520 con 0x7ff9c80a7090 2024-08-19T12:38:26.781 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 >> v1:172.21.15.104:6792/0 conn(0x7ff9c80a65a0 legacy=0x7ff9c80a6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:26.781 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 >> v1:172.21.15.26:6792/0 conn(0x7ff9c813fed0 legacy=0x7ff9c81422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:26.781 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:26.781 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 monclient: found mon.noname-c 2024-08-19T12:38:26.782 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:26.782 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:38:26.782 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c809dbe0 con 0x7ff9c80a7090 2024-08-19T12:38:26.782 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient: _finish_auth 0 2024-08-19T12:38:26.782 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:26.782 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:26.783 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff9b8001a10 con 0x7ff9c80a7090 2024-08-19T12:38:26.783 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:26.783 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:26.783 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient: dump: 2024-08-19T12:38:26.783 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:26.784 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:26.784 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:26.784 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:26.784 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:26.784 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:26.784 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:26.785 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:26.785 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:26.785 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:26.785 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:26.785 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:26.786 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:26.786 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:26.786 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:26.786 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:26.786 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9cf511640 5 monclient: authenticate success, global_id 4243 2024-08-19T12:38:26.786 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9cf511640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:26.787 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff9b8001ce0 con 0x7ff9c80a7090 2024-08-19T12:38:26.787 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:26.787 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 1 -- 172.21.15.26:0/3325348635 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff9b8004be0 con 0x7ff9c80a7090 2024-08-19T12:38:26.787 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9cf511640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:26.787 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9cf511640 4 set_mon_vals no callback set 2024-08-19T12:38:26.787 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:26.788 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:26.788 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.758+0000 7ff9bffff640 10 monclient: dump: 2024-08-19T12:38:26.788 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:26.788 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:26.788 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:26.788 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:26.789 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:26.789 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:26.789 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:26.789 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:26.789 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:26.789 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:26.790 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:26.790 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:26.790 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:26.790 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:26.790 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:26.791 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:26.791 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.759+0000 7ff9cf511640 10 monclient: shutdown 2024-08-19T12:38:26.791 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.759+0000 7ff9cf511640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:26.791 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.759+0000 7ff9cf511640 1 -- 172.21.15.26:0/3325348635 >> v1:172.21.15.26:6791/0 conn(0x7ff9c80a7090 legacy=0x7ff9c813e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:26.791 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.759+0000 7ff9cf511640 1 -- 172.21.15.26:0/3325348635 shutdown_connections 2024-08-19T12:38:26.791 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.759+0000 7ff9cf511640 1 -- 172.21.15.26:0/3325348635 >> 172.21.15.26:0/3325348635 conn(0x7ff9c80a86a0 msgr2=0x7ff9c80a8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:26.792 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.759+0000 7ff9cf511640 1 -- 172.21.15.26:0/3325348635 shutdown_connections 2024-08-19T12:38:26.792 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.759+0000 7ff9cf511640 1 -- 172.21.15.26:0/3325348635 wait complete. 2024-08-19T12:38:26.792 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.759+0000 7ff9cf511640 10 monclient: build_initial_monmap 2024-08-19T12:38:26.792 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.759+0000 7ff9cf511640 10 monclient: monmap: 2024-08-19T12:38:26.792 INFO:teuthology.misc.health.smithi026.stderr:epoch 0 2024-08-19T12:38:26.792 INFO:teuthology.misc.health.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:26.792 INFO:teuthology.misc.health.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:26.793 INFO:teuthology.misc.health.smithi026.stderr:created 0.000000 2024-08-19T12:38:26.793 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:26.793 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:26.793 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:26.793 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:26.793 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:26.793 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:26.794 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:26.794 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:26.794 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:26.794 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:26.794 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:26.794 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:26.795 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 1 librados: starting msgr at 2024-08-19T12:38:26.795 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 1 librados: starting objecter 2024-08-19T12:38:26.795 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 1 Processor -- start 2024-08-19T12:38:26.795 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 1 -- start start 2024-08-19T12:38:26.795 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 1 librados: setting wanted keys 2024-08-19T12:38:26.795 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 1 librados: calling monclient init 2024-08-19T12:38:26.796 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 10 monclient: init 2024-08-19T12:38:26.796 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:26.796 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 10 monclient: _add_conns ranks=[7,3,8,5,1,2,6,4,0] 2024-08-19T12:38:26.796 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 10 monclient(hunting): picked mon.noname-i con 0x7ff9c80a65a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:26.796 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 10 monclient(hunting): picked mon.noname-g con 0x7ff9c80a7090 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:26.796 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 10 monclient(hunting): picked mon.noname-e con 0x7ff9c813fed0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:26.797 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c806e470 con 0x7ff9c80a7090 2024-08-19T12:38:26.797 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c80a1430 con 0x7ff9c80a65a0 2024-08-19T12:38:26.797 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c80a1710 con 0x7ff9c813fed0 2024-08-19T12:38:26.797 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:26.797 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cf511640 10 monclient(hunting): authenticate will time out at 700.847858s 2024-08-19T12:38:26.797 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cdaae640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7ff9c813fed0 0x7ff9c81d60a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:43742/0 (socket says 172.21.15.26:43742) 2024-08-19T12:38:26.797 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.760+0000 7ff9cdaae640 1 -- 172.21.15.26:0/1809242404 learned_addr learned my addr 172.21.15.26:0/1809242404 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:26.798 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9cdaae640 1 -- 172.21.15.26:0/1809242404 >> v1:172.21.15.26:6793/0 conn(0x7ff9c813fed0 legacy=0x7ff9c81d60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:26.798 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 client.?.objecter ms_handle_connect 0x7ff9c813fed0 2024-08-19T12:38:26.798 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:26.798 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9cd2ad640 1 -- 172.21.15.26:0/1809242404 >> v1:172.21.15.104:6792/0 conn(0x7ff9c80a65a0 legacy=0x7ff9c809e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:26.798 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 client.?.objecter ms_handle_connect 0x7ff9c80a65a0 2024-08-19T12:38:26.799 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:26.799 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9ccaac640 1 -- 172.21.15.26:0/1809242404 >> v1:172.21.15.104:6790/0 conn(0x7ff9c80a7090 legacy=0x7ff9c809efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:26.799 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 283131808 0 0) 0x7ff9c80a1710 con 0x7ff9c813fed0 2024-08-19T12:38:26.799 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:26.799 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:26.799 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:26.800 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): my global_id is 4256 2024-08-19T12:38:26.800 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9ac004eb0 con 0x7ff9c813fed0 2024-08-19T12:38:26.800 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 client.?.objecter ms_handle_connect 0x7ff9c80a7090 2024-08-19T12:38:26.800 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:26.800 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 275287303 0 0) 0x7ff9c80a1430 con 0x7ff9c80a65a0 2024-08-19T12:38:26.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:26.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:26.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:26.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): my global_id is 4210 2024-08-19T12:38:26.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9c80a1710 con 0x7ff9c80a65a0 2024-08-19T12:38:26.801 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4272349444 0 0) 0x7ff9c809dbe0 con 0x7ff9c80a7090 2024-08-19T12:38:26.802 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:26.802 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:26.802 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:26.802 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): my global_id is 4260 2024-08-19T12:38:26.802 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9c80a1430 con 0x7ff9c80a7090 2024-08-19T12:38:26.803 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4266474389 0 0) 0x7ff9ac004eb0 con 0x7ff9c813fed0 2024-08-19T12:38:26.803 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9c809dbe0 con 0x7ff9c813fed0 2024-08-19T12:38:26.803 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff9c0002070 con 0x7ff9c813fed0 2024-08-19T12:38:26.803 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:26.803 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:26.804 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): dump: 2024-08-19T12:38:26.804 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:26.804 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:26.804 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:26.804 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:26.805 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:26.805 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:26.805 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:26.805 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:26.805 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:26.805 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:26.806 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:26.806 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:26.806 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:26.806 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:26.806 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:26.807 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:26.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:26.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2049310605 0 0) 0x7ff9c80a1710 con 0x7ff9c80a65a0 2024-08-19T12:38:26.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9ac004eb0 con 0x7ff9c80a65a0 2024-08-19T12:38:26.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff9c4003610 con 0x7ff9c80a65a0 2024-08-19T12:38:26.807 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:26.808 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:26.808 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): dump: 2024-08-19T12:38:26.808 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:26.808 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:26.808 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:26.809 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:26.809 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:26.809 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:26.809 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:26.809 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:26.809 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:26.810 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:26.810 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:26.810 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:26.810 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:26.810 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:26.811 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:26.811 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:26.811 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 896818620 0 0) 0x7ff9c80a1430 con 0x7ff9c80a7090 2024-08-19T12:38:26.811 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9c80a1710 con 0x7ff9c80a7090 2024-08-19T12:38:26.811 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff9b8004b20 con 0x7ff9c80a7090 2024-08-19T12:38:26.811 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:26.811 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:26.812 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient(hunting): dump: 2024-08-19T12:38:26.812 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:26.812 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:26.812 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:26.812 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:26.812 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:26.812 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:26.813 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:26.813 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:26.813 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:26.813 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:26.813 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:26.813 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:26.813 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:26.814 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:26.814 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:26.814 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:26.814 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1506812140 0 0) 0x7ff9c809dbe0 con 0x7ff9c813fed0 2024-08-19T12:38:26.814 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 >> v1:172.21.15.104:6792/0 conn(0x7ff9c80a65a0 legacy=0x7ff9c809e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:26.814 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 >> v1:172.21.15.104:6790/0 conn(0x7ff9c80a7090 legacy=0x7ff9c809efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:26.815 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:26.815 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 monclient: found mon.e 2024-08-19T12:38:26.815 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:26.815 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:26.815 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c809f6c0 con 0x7ff9c813fed0 2024-08-19T12:38:26.815 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:26.815 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:26.816 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:26.816 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9cf511640 5 monclient: authenticate success, global_id 4256 2024-08-19T12:38:26.816 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9cf511640 10 monclient: _renew_subs 2024-08-19T12:38:26.816 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9cf511640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:26.816 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.763+0000 7ff9cf511640 1 -- 172.21.15.26:0/1809242404 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff9c813f330 con 0x7ff9c813fed0 2024-08-19T12:38:26.816 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff9c0004e80 con 0x7ff9c813fed0 2024-08-19T12:38:26.816 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9bdffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:26.817 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff9c0005370 con 0x7ff9c813fed0 2024-08-19T12:38:26.817 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:26.817 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9bdffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:26.817 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9bdffb640 10 monclient: dump: 2024-08-19T12:38:26.817 INFO:teuthology.misc.health.smithi026.stderr:epoch 1 2024-08-19T12:38:26.817 INFO:teuthology.misc.health.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:26.817 INFO:teuthology.misc.health.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:26.817 INFO:teuthology.misc.health.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:26.817 INFO:teuthology.misc.health.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:26.817 INFO:teuthology.misc.health.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:26.817 INFO:teuthology.misc.health.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:26.817 INFO:teuthology.misc.health.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr: 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9beffd640 4 set_mon_vals no callback set 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9cf511640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9cf511640 10 monclient: _renew_subs 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9cf511640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:26.818 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9cf511640 1 -- 172.21.15.26:0/1809242404 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7ff9c80a0e70 con 0x7ff9c813fed0 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9cf511640 1 librados: init done 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.764+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff9c00118d0 con 0x7ff9c813fed0 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.765+0000 7ff9cf511640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.765+0000 7ff9cf511640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.765+0000 7ff9cf511640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.765+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (unknown 3485197666 0 0) 0x7ff9c004dba0 con 0x7ff9c813fed0 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.765+0000 7ff9bdffb640 10 client.4256.objecter ms_dispatch 0x7ff9c8000b90 osd_map(15..15 src has 1..15) 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.765+0000 7ff9cf511640 1 -- 172.21.15.26:0/1809242404 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9c8051b00 con 0x7ff9c813fed0 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.765+0000 7ff9bdffb640 3 client.4256.objecter handle_osd_map got epochs [15,15] > 0 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.765+0000 7ff9bdffb640 3 client.4256.objecter handle_osd_map decoding full epoch 15 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.765+0000 7ff9bdffb640 20 client.4256.objecter dump_active .. 0 homeless 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.765+0000 7ff9cd2ad640 10 monclient: get_auth_request con 0x7ff9ac041980 auth_method 0 2024-08-19T12:38:26.819 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.767+0000 7ff9bdffb640 10 client.4256.objecter ms_handle_connect 0x7ff9ac041980 2024-08-19T12:38:26.820 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.770+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7ff9c0017830 con 0x7ff9c813fed0 2024-08-19T12:38:26.820 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.770+0000 7ff9bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:26.820 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.770+0000 7ff9bdffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:26.956 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.955+0000 7ff9cf511640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:26.956 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.955+0000 7ff9cf511640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:26.956 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.955+0000 7ff9cf511640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:26.956 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.955+0000 7ff9cf511640 1 -- 172.21.15.26:0/1809242404 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9c80a01f0 con 0x7ff9c813fed0 2024-08-19T12:38:26.956 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.955+0000 7ff9bdffb640 1 -- 172.21.15.26:0/1809242404 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+9049 (unknown 4221056075 0 4002032883) 0x7ff9c0017130 con 0x7ff9c813fed0 2024-08-19T12:38:26.956 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.955+0000 7ff9bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:26.957 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.955+0000 7ff9bdffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:26.957 INFO:teuthology.misc.health.smithi026.stdout: 2024-08-19T12:38:26.957 INFO:teuthology.misc.health.smithi026.stdout:{"epoch":15,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","created":"2024-08-19T12:38:08.215243+0000","modified":"2024-08-19T12:38:23.732884+0000","last_up_change":"2024-08-19T12:38:20.704140+0000","last_in_change":"2024-08-19T12:38:12.064517+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T12:38:21.116424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae69b71a-73f9-47e2-b81e-92e9cdd7df0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6800","nonce":3531533167}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6801","nonce":3531533167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6803","nonce":3531533167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6802","nonce":3531533167}]},"public_addr":"172.21.15.26:6800/3531533167","cluster_addr":"172.21.15.26:6801/3531533167","heartbeat_back_addr":"172.21.15.26:6803/3531533167","heartbeat_front_addr":"172.21.15.26:6802/3531533167","state":["exists","up"]},{"osd":1,"uuid":"fc6dcabb-211c-484a-b215-2ef5fe1cbd21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6804","nonce":2594673920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6805","nonce":2594673920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6807","nonce":2594673920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6806","nonce":2594673920}]},"public_addr":"172.21.15.26:6804/2594673920","cluster_addr":"172.21.15.26:6805/2594673920","heartbeat_back_addr":"172.21.15.26:6807/2594673920","heartbeat_front_addr":"172.21.15.26:6806/2594673920","state":["exists","up"]},{"osd":2,"uuid":"88a4da36-5efe-4f64-84c3-28413eba4011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6808","nonce":1443966639}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6809","nonce":1443966639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6811","nonce":1443966639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6810","nonce":1443966639}]},"public_addr":"172.21.15.26:6808/1443966639","cluster_addr":"172.21.15.26:6809/1443966639","heartbeat_back_addr":"172.21.15.26:6811/1443966639","heartbeat_front_addr":"172.21.15.26:6810/1443966639","state":["exists","up"]},{"osd":3,"uuid":"98c08a65-4fb2-4e5f-a9bc-e8899aed1103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6809","nonce":847838248}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6810","nonce":847838248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6812","nonce":847838248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6811","nonce":847838248}]},"public_addr":"172.21.15.104:6809/847838248","cluster_addr":"172.21.15.104:6810/847838248","heartbeat_back_addr":"172.21.15.104:6812/847838248","heartbeat_front_addr":"172.21.15.104:6811/847838248","state":["exists","up"]},{"osd":4,"uuid":"a4c308b6-5e55-4328-b7b0-1db11fd51d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6805","nonce":1352976736}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6806","nonce":1352976736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6808","nonce":1352976736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6807","nonce":1352976736}]},"public_addr":"172.21.15.104:6805/1352976736","cluster_addr":"172.21.15.104:6806/1352976736","heartbeat_back_addr":"172.21.15.104:6808/1352976736","heartbeat_front_addr":"172.21.15.104:6807/1352976736","state":["exists","up"]},{"osd":5,"uuid":"99117114-07ce-4b9b-8df6-39f5fb1b610b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6801","nonce":3713183117}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6802","nonce":3713183117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6804","nonce":3713183117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6803","nonce":3713183117}]},"public_addr":"172.21.15.104:6801/3713183117","cluster_addr":"172.21.15.104:6802/3713183117","heartbeat_back_addr":"172.21.15.104:6804/3713183117","heartbeat_front_addr":"172.21.15.104:6803/3713183117","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.063585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.566627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.758447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.327856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.657329+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.299879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T12:38:26.959 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 10 librados: watch_flush enter 2024-08-19T12:38:26.959 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 10 librados: watch_flush exit 2024-08-19T12:38:26.959 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 20 client.4256.objecter shutdown clearing up homeless session... 2024-08-19T12:38:26.959 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 10 client.4256.objecter successfully canceled tick 2024-08-19T12:38:26.959 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 1 -- 172.21.15.26:0/1809242404 >> v1:172.21.15.104:6800/899887506 conn(0x7ff9ac041980 legacy=0x7ff9ac043e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:26.959 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 10 monclient: shutdown 2024-08-19T12:38:26.959 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:26.959 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 1 -- 172.21.15.26:0/1809242404 >> v1:172.21.15.26:6793/0 conn(0x7ff9c813fed0 legacy=0x7ff9c81d60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:26.959 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 1 -- 172.21.15.26:0/1809242404 shutdown_connections 2024-08-19T12:38:26.959 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 1 -- 172.21.15.26:0/1809242404 >> 172.21.15.26:0/1809242404 conn(0x7ff9c80a86a0 msgr2=0x7ff9c8135330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:26.959 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 1 -- 172.21.15.26:0/1809242404 shutdown_connections 2024-08-19T12:38:26.959 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 1 -- 172.21.15.26:0/1809242404 wait complete. 2024-08-19T12:38:26.960 INFO:teuthology.misc.health.smithi026.stderr:2024-08-19T12:38:26.957+0000 7ff9cf511640 1 librados: shutdown 2024-08-19T12:38:26.968 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-08-19T12:38:26.968 INFO:tasks.ceph:Creating RBD pool 2024-08-19T12:38:26.968 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd pool create rbd 8 2024-08-19T12:38:27.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.073+0000 7f556a01e640 10 monclient: get_monmap_and_config 2024-08-19T12:38:27.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.073+0000 7f556a01e640 10 monclient: build_initial_monmap 2024-08-19T12:38:27.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.073+0000 7f556a01e640 10 monclient: monmap: 2024-08-19T12:38:27.074 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:27.074 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:27.075 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:27.074360+0000 2024-08-19T12:38:27.075 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:27.074360+0000 2024-08-19T12:38:27.075 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:27.075 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:27.075 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:27.075 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:27.075 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:27.075 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:27.075 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:27.075 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:27.076 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:27.076 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:27.076 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:27.076 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:27.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 1 Processor -- start 2024-08-19T12:38:27.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 1 -- start start 2024-08-19T12:38:27.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 10 monclient: init 2024-08-19T12:38:27.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:27.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 10 monclient: _add_conns ranks=[4,8,7,5,0,6,2,1,3] 2024-08-19T12:38:27.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 10 monclient(hunting): picked mon.noname-c con 0x7f556406e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:27.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 10 monclient(hunting): picked mon.noname-e con 0x7f55640a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:27.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 10 monclient(hunting): picked mon.noname-i con 0x7f55640a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:27.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5564136790 con 0x7f556406e7b0 2024-08-19T12:38:27.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556409ddf0 con 0x7f55640a5d50 2024-08-19T12:38:27.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556409e0d0 con 0x7f55640a5920 2024-08-19T12:38:27.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:27.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f556a01e640 10 monclient(hunting): authenticate will time out at 701.161874s 2024-08-19T12:38:27.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f55637fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f556406e7b0 0x7f5564136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:50728/0 (socket says 172.21.15.26:50728) 2024-08-19T12:38:27.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.074+0000 7f55637fe640 1 -- 172.21.15.26:0/3626230054 learned_addr learned my addr 172.21.15.26:0/3626230054 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:27.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55637fe640 1 -- 172.21.15.26:0/3626230054 >> v1:172.21.15.26:6791/0 conn(0x7f556406e7b0 legacy=0x7f5564136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:27.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f5562ffd640 1 -- 172.21.15.26:0/3626230054 >> v1:172.21.15.26:6793/0 conn(0x7f55640a5920 legacy=0x7f556413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:27.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 1 -- 172.21.15.26:0/3626230054 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 869603198 0 0) 0x7f5564136790 con 0x7f556406e7b0 2024-08-19T12:38:27.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:27.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:27.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:27.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient(hunting): my global_id is 4252 2024-08-19T12:38:27.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 1 -- 172.21.15.26:0/3626230054 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f554c003520 con 0x7f556406e7b0 2024-08-19T12:38:27.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 1 -- 172.21.15.26:0/3626230054 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026698791 0 0) 0x7f556409e0d0 con 0x7f55640a5920 2024-08-19T12:38:27.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:27.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:27.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:27.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient(hunting): my global_id is 4265 2024-08-19T12:38:27.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 1 -- 172.21.15.26:0/3626230054 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5564136790 con 0x7f55640a5920 2024-08-19T12:38:27.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f5563fff640 1 -- 172.21.15.26:0/3626230054 >> v1:172.21.15.104:6792/0 conn(0x7f55640a5d50 legacy=0x7f5564141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:27.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 1 -- 172.21.15.26:0/3626230054 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 883665892 0 0) 0x7f5564136790 con 0x7f55640a5920 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 1 -- 172.21.15.26:0/3626230054 >> v1:172.21.15.104:6792/0 conn(0x7f55640a5d50 legacy=0x7f5564141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 1 -- 172.21.15.26:0/3626230054 >> v1:172.21.15.26:6791/0 conn(0x7f556406e7b0 legacy=0x7f5564136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 1 monclient: found mon.noname-e 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 1 -- 172.21.15.26:0/3626230054 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556409e3b0 con 0x7f55640a5920 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 1 -- 172.21.15.26:0/3626230054 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5558001b10 con 0x7f55640a5920 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f556a01e640 5 monclient: authenticate success, global_id 4265 2024-08-19T12:38:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f556a01e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient: dump: 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:27.086 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f556a01e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 1 -- 172.21.15.26:0/3626230054 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5558003610 con 0x7f55640a5920 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f556a01e640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 1 -- 172.21.15.26:0/3626230054 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5558003b80 con 0x7f55640a5920 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:27.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f556a01e640 4 set_mon_vals no callback set 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.077+0000 7f55627fc640 10 monclient: dump: 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:27.088 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:27.089 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.078+0000 7f556a01e640 10 monclient: shutdown 2024-08-19T12:38:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.078+0000 7f556a01e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.078+0000 7f556a01e640 1 -- 172.21.15.26:0/3626230054 >> v1:172.21.15.26:6793/0 conn(0x7f55640a5920 legacy=0x7f556413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.078+0000 7f556a01e640 1 -- 172.21.15.26:0/3626230054 shutdown_connections 2024-08-19T12:38:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.078+0000 7f556a01e640 1 -- 172.21.15.26:0/3626230054 >> 172.21.15.26:0/3626230054 conn(0x7f55640a8740 msgr2=0x7f55640a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.078+0000 7f556a01e640 1 -- 172.21.15.26:0/3626230054 shutdown_connections 2024-08-19T12:38:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.078+0000 7f556a01e640 1 -- 172.21.15.26:0/3626230054 wait complete. 2024-08-19T12:38:27.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.078+0000 7f556a01e640 10 monclient: build_initial_monmap 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.078+0000 7f556a01e640 10 monclient: monmap: 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:27.090 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 1 librados: starting msgr at 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 1 librados: starting objecter 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 1 Processor -- start 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 1 -- start start 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 1 librados: setting wanted keys 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 1 librados: calling monclient init 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 10 monclient: init 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 10 monclient: _add_conns ranks=[6,2,5,7,4,0,1,3,8] 2024-08-19T12:38:27.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 10 monclient(hunting): picked mon.noname-d con 0x7f556406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:27.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 10 monclient(hunting): picked mon.noname-b con 0x7f55640a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:27.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 10 monclient(hunting): picked mon.noname-h con 0x7f55640a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:27.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556413b450 con 0x7f55640a5920 2024-08-19T12:38:27.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55641390f0 con 0x7f55640a5d50 2024-08-19T12:38:27.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5564139c50 con 0x7f556406e7b0 2024-08-19T12:38:27.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:27.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f556a01e640 10 monclient(hunting): authenticate will time out at 701.166862s 2024-08-19T12:38:27.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f55637fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f556406e7b0 0x7f55641ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:44544/0 (socket says 172.21.15.26:44544) 2024-08-19T12:38:27.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.079+0000 7f55637fe640 1 -- 172.21.15.26:0/2135294961 learned_addr learned my addr 172.21.15.26:0/2135294961 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:27.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f55637fe640 1 -- 172.21.15.26:0/2135294961 >> v1:172.21.15.26:6792/0 conn(0x7f556406e7b0 legacy=0x7f55641ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:27.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 client.?.objecter ms_handle_connect 0x7f556406e7b0 2024-08-19T12:38:27.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:27.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 client.?.objecter ms_handle_connect 0x7f55640a5920 2024-08-19T12:38:27.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:27.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f5562ffd640 1 -- 172.21.15.26:0/2135294961 >> v1:172.21.15.26:6790/0 conn(0x7f55640a5920 legacy=0x7f55641d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:27.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3052623489 0 0) 0x7f5564139c50 con 0x7f556406e7b0 2024-08-19T12:38:27.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): my global_id is 4254 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5550004eb0 con 0x7f556406e7b0 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2579901235 0 0) 0x7f556413b450 con 0x7f55640a5920 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): my global_id is 4205 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5564139c50 con 0x7f55640a5920 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f5563fff640 1 -- 172.21.15.26:0/2135294961 >> v1:172.21.15.104:6791/0 conn(0x7f55640a5d50 legacy=0x7f55641d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 client.?.objecter ms_handle_connect 0x7f55640a5d50 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:27.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2903540248 0 0) 0x7f5564139c50 con 0x7f55640a5920 2024-08-19T12:38:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f556413b450 con 0x7f55640a5920 2024-08-19T12:38:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5558003b00 con 0x7f55640a5920 2024-08-19T12:38:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): dump: 2024-08-19T12:38:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:27.098 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 292370987 0 0) 0x7f5550004eb0 con 0x7f556406e7b0 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5564139c50 con 0x7f556406e7b0 2024-08-19T12:38:27.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5554001a10 con 0x7f556406e7b0 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): dump: 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:27.100 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3171276991 0 0) 0x7f55641390f0 con 0x7f55640a5d50 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient(hunting): my global_id is 4235 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5550004eb0 con 0x7f55640a5d50 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3140392507 0 0) 0x7f556413b450 con 0x7f55640a5920 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 >> v1:172.21.15.26:6792/0 conn(0x7f556406e7b0 legacy=0x7f55641ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 >> v1:172.21.15.104:6791/0 conn(0x7f55640a5d50 legacy=0x7f55641d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:27.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 monclient: found mon.b 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556413abb0 con 0x7f55640a5920 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient: _finish_auth 0 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f556a01e640 5 monclient: authenticate success, global_id 4205 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f556a01e640 10 monclient: _renew_subs 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f556a01e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f556a01e640 1 -- 172.21.15.26:0/2135294961 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f556413ae90 con 0x7f55640a5920 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f556a01e640 10 client.4205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:27.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f556a01e640 10 monclient: _renew_subs 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f556a01e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f556a01e640 1 -- 172.21.15.26:0/2135294961 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5564138690 con 0x7f55640a5920 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f556a01e640 1 librados: init done 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5558001370 con 0x7f55640a5920 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5558003610 con 0x7f55640a5920 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f5561ffb640 4 set_mon_vals no callback set 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.082+0000 7f554bfff640 10 monclient: dump: 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:27.103 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.083+0000 7f556a01e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.083+0000 7f556a01e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:27.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.083+0000 7f556a01e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.083+0000 7f556a01e640 1 -- 172.21.15.26:0/2135294961 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5544005080 con 0x7f55640a5920 2024-08-19T12:38:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.083+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5558011a80 con 0x7f55640a5920 2024-08-19T12:38:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.084+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (unknown 3485197666 0 0) 0x7f555804d790 con 0x7f55640a5920 2024-08-19T12:38:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.084+0000 7f554bfff640 10 client.4205.objecter ms_dispatch 0x7f5564000b90 osd_map(15..15 src has 1..15) 2024-08-19T12:38:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.084+0000 7f554bfff640 3 client.4205.objecter handle_osd_map got epochs [15,15] > 0 2024-08-19T12:38:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.084+0000 7f554bfff640 3 client.4205.objecter handle_osd_map decoding full epoch 15 2024-08-19T12:38:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.084+0000 7f554bfff640 20 client.4205.objecter dump_active .. 0 homeless 2024-08-19T12:38:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.084+0000 7f55637fe640 10 monclient: get_auth_request con 0x7f5550041840 auth_method 0 2024-08-19T12:38:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.086+0000 7f554bfff640 10 client.4205.objecter ms_handle_connect 0x7f5550041840 2024-08-19T12:38:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.088+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f5558017480 con 0x7f55640a5920 2024-08-19T12:38:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.088+0000 7f554bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.088+0000 7f554bfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:27.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.272+0000 7f556a01e640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-08-19T12:38:27.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.272+0000 7f556a01e640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-08-19T12:38:27.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.272+0000 7f556a01e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:27.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:27.272+0000 7f556a01e640 1 -- 172.21.15.26:0/2135294961 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f5544005c40 con 0x7f55640a5920 2024-08-19T12:38:28.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.928+0000 7f554bfff640 1 -- 172.21.15.26:0/2135294961 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (unknown 231814880 0 0) 0x7f555801fd10 con 0x7f55640a5920 2024-08-19T12:38:28.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.928+0000 7f554bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-08-19T12:38:28.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.928+0000 7f554bfff640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-08-19T12:38:28.930 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'rbd' created 2024-08-19T12:38:28.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 10 librados: watch_flush enter 2024-08-19T12:38:28.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 10 librados: watch_flush exit 2024-08-19T12:38:28.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 20 client.4205.objecter shutdown clearing up homeless session... 2024-08-19T12:38:28.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 10 client.4205.objecter successfully canceled tick 2024-08-19T12:38:28.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 1 -- 172.21.15.26:0/2135294961 >> v1:172.21.15.104:6800/899887506 conn(0x7f5550041840 legacy=0x7f5550043d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:28.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 10 monclient: shutdown 2024-08-19T12:38:28.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:28.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 1 -- 172.21.15.26:0/2135294961 >> v1:172.21.15.26:6790/0 conn(0x7f55640a5920 legacy=0x7f55641d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:28.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 1 -- 172.21.15.26:0/2135294961 shutdown_connections 2024-08-19T12:38:28.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 1 -- 172.21.15.26:0/2135294961 >> 172.21.15.26:0/2135294961 conn(0x7f55640a8740 msgr2=0x7f55640ab120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:28.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 1 -- 172.21.15.26:0/2135294961 shutdown_connections 2024-08-19T12:38:28.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 1 -- 172.21.15.26:0/2135294961 wait complete. 2024-08-19T12:38:28.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.932+0000 7f556a01e640 1 librados: shutdown 2024-08-19T12:38:28.945 DEBUG:teuthology.orchestra.run.smithi026:> rbd --cluster ceph pool init rbd 2024-08-19T12:38:28.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.982+0000 7f427518df00 10 monclient: get_monmap_and_config 2024-08-19T12:38:28.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.982+0000 7f427518df00 10 monclient: build_initial_monmap 2024-08-19T12:38:28.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.983+0000 7f427518df00 10 monclient: monmap: 2024-08-19T12:38:28.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:28.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:28.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:28.983826+0000 2024-08-19T12:38:28.984 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:28.983826+0000 2024-08-19T12:38:28.984 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:28.984 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:28.984 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:28.984 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:28.984 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:28.985 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:28.985 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:28.985 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:28.985 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:28.985 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:28.985 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:28.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:28.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.983+0000 7f427518df00 1 Processor -- start 2024-08-19T12:38:28.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.983+0000 7f427518df00 1 -- start start 2024-08-19T12:38:28.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.983+0000 7f427518df00 10 monclient: init 2024-08-19T12:38:28.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.984+0000 7f427518df00 10 monclient: _reopen_session rank -1 2024-08-19T12:38:28.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.984+0000 7f427518df00 10 monclient: _add_conns ranks=[2,1,7,0,4,6,8,3,5] 2024-08-19T12:38:28.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.984+0000 7f427518df00 10 monclient(hunting): picked mon.noname-b con 0x556644841870 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:28.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.984+0000 7f427518df00 10 monclient(hunting): picked mon.noname-f con 0x55664483fc30 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:28.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.984+0000 7f427518df00 10 monclient(hunting): picked mon.noname-i con 0x556644842360 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:28.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.984+0000 7f427518df00 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5566446ce9a0 con 0x55664483fc30 2024-08-19T12:38:28.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.984+0000 7f427518df00 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5566446a7010 con 0x556644841870 2024-08-19T12:38:28.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.984+0000 7f427518df00 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55664468c0c0 con 0x556644842360 2024-08-19T12:38:28.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.984+0000 7f427518df00 10 monclient(hunting): _renew_subs 2024-08-19T12:38:28.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.984+0000 7f427518df00 10 monclient(hunting): authenticate will time out at 703.071288s 2024-08-19T12:38:28.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.984+0000 7f4273c7a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x556644841870 0x556644841c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50222/0 (socket says 172.21.15.26:50222) 2024-08-19T12:38:28.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.984+0000 7f4273c7a640 1 -- 172.21.15.26:0/4126757672 learned_addr learned my addr 172.21.15.26:0/4126757672 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:28.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.986+0000 7f4273c7a640 1 -- 172.21.15.26:0/4126757672 >> v1:172.21.15.26:6790/0 conn(0x556644841870 legacy=0x556644841c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.986+0000 7f4273479640 1 -- 172.21.15.26:0/4126757672 >> v1:172.21.15.104:6789/0 conn(0x55664483fc30 legacy=0x5566448d9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.986+0000 7f4274edc640 1 -- 172.21.15.26:0/4126757672 >> v1:172.21.15.104:6792/0 conn(0x556644842360 legacy=0x5566448dd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 338598807 0 0) 0x5566446a7010 con 0x556644841870 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient(hunting): _negotiate 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient(hunting): my global_id is 4214 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4260001e10 con 0x556644841870 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2384672774 0 0) 0x5566446ce9a0 con 0x55664483fc30 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient(hunting): _negotiate 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:28.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient(hunting): my global_id is 4213 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x5566446a7010 con 0x55664483fc30 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931802606 0 0) 0x55664468c0c0 con 0x556644842360 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient(hunting): _negotiate 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient(hunting): my global_id is 4219 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x5566446ce9a0 con 0x556644842360 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3383312381 0 0) 0x7f4260001e10 con 0x556644841870 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 >> v1:172.21.15.104:6792/0 conn(0x556644842360 legacy=0x5566448dd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 >> v1:172.21.15.104:6789/0 conn(0x55664483fc30 legacy=0x5566448d9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient: _finish_hunting 0 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 monclient: found mon.noname-b 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:28.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55664483f900 con 0x556644841870 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient: _finish_auth 0 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient: _check_auth_tickets 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4268001ae0 con 0x556644841870 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f427518df00 5 monclient: authenticate success, global_id 4214 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f427518df00 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient: dump: 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:28.994 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:28.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f427518df00 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4268003190 con 0x556644841870 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 1 -- 172.21.15.26:0/4126757672 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4268003700 con 0x556644841870 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f427518df00 10 monclient: get_monmap_and_config success 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f427518df00 4 set_mon_vals no callback set 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.987+0000 7f4272c78640 10 monclient: dump: 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:28.996 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.988+0000 7f427518df00 10 monclient: shutdown 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.988+0000 7f427518df00 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.988+0000 7f427518df00 1 -- 172.21.15.26:0/4126757672 >> v1:172.21.15.26:6790/0 conn(0x556644841870 legacy=0x556644841c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:28.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.988+0000 7f427518df00 1 -- 172.21.15.26:0/4126757672 shutdown_connections 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.988+0000 7f427518df00 1 -- 172.21.15.26:0/4126757672 >> 172.21.15.26:0/4126757672 conn(0x556644823830 msgr2=0x556644823c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.988+0000 7f427518df00 1 -- 172.21.15.26:0/4126757672 shutdown_connections 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.988+0000 7f427518df00 1 -- 172.21.15.26:0/4126757672 wait complete. 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.988+0000 7f427518df00 10 monclient: build_initial_monmap 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.988+0000 7f427518df00 10 monclient: monmap: 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:28.998 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.988+0000 7f427518df00 1 librados: starting msgr at 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.988+0000 7f427518df00 1 librados: starting objecter 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 1 Processor -- start 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 1 -- start start 2024-08-19T12:38:28.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 1 librados: setting wanted keys 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 1 librados: calling monclient init 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 10 monclient: init 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 10 monclient: _reopen_session rank -1 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 10 monclient: _add_conns ranks=[2,1,7,0,3,5,4,8,6] 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 10 monclient(hunting): picked mon.noname-b con 0x55664483fc30 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 10 monclient(hunting): picked mon.noname-f con 0x556644841870 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 10 monclient(hunting): picked mon.noname-i con 0x556644842360 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5566448c6780 con 0x556644841870 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5566448c6a60 con 0x55664483fc30 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5566448c59e0 con 0x556644842360 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 10 monclient(hunting): _renew_subs 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f427518df00 10 monclient(hunting): authenticate will time out at 703.076685s 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f4273c7a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x55664483fc30 0x55664483d360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50236/0 (socket says 172.21.15.26:50236) 2024-08-19T12:38:29.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.989+0000 7f4273c7a640 1 -- 172.21.15.26:0/2983727097 learned_addr learned my addr 172.21.15.26:0/2983727097 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:29.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.991+0000 7f4273c7a640 1 -- 172.21.15.26:0/2983727097 >> v1:172.21.15.26:6790/0 conn(0x55664483fc30 legacy=0x55664483d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:29.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.991+0000 7f425bfff640 10 client.?.objecter ms_handle_connect 0x55664483fc30 2024-08-19T12:38:29.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:29.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 client.?.objecter ms_handle_connect 0x556644841870 2024-08-19T12:38:29.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:29.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f4273479640 1 -- 172.21.15.26:0/2983727097 >> v1:172.21.15.104:6789/0 conn(0x556644841870 legacy=0x55664483da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:29.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f4274edc640 1 -- 172.21.15.26:0/2983727097 >> v1:172.21.15.104:6792/0 conn(0x556644842360 legacy=0x5566448ed9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:29.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 client.?.objecter ms_handle_connect 0x556644842360 2024-08-19T12:38:29.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:29.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2223377366 0 0) 0x5566448c6780 con 0x556644841870 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 monclient(hunting): my global_id is 4222 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4254002fe0 con 0x556644841870 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 941975365 0 0) 0x5566448c6a60 con 0x55664483fc30 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 monclient(hunting): my global_id is 4223 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x5566448c6780 con 0x55664483fc30 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 190360655 0 0) 0x5566448c59e0 con 0x556644842360 2024-08-19T12:38:29.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 10 monclient(hunting): my global_id is 4228 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x5566448c6a60 con 0x556644842360 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.992+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1735776026 0 0) 0x7f4254002fe0 con 0x556644841870 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x5566448c59e0 con 0x556644841870 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f425c001f00 con 0x556644841870 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient(hunting): dump: 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:29.006 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:29.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1054071339 0 0) 0x5566448c6780 con 0x55664483fc30 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4254002fe0 con 0x55664483fc30 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4268003680 con 0x55664483fc30 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient(hunting): dump: 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:29.008 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 105684806 0 0) 0x5566448c6a60 con 0x556644842360 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x5566448c6780 con 0x556644842360 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4264002070 con 0x556644842360 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient(hunting): dump: 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:29.011 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:29.011 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:29.011 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:29.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 339183538 0 0) 0x5566448c59e0 con 0x556644841870 2024-08-19T12:38:29.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 >> v1:172.21.15.104:6792/0 conn(0x556644842360 legacy=0x5566448ed9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:29.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 >> v1:172.21.15.26:6790/0 conn(0x55664483fc30 legacy=0x55664483d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:29.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:29.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 monclient: found mon.f 2024-08-19T12:38:29.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55664483ed00 con 0x556644841870 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient: _finish_auth 0 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f427518df00 5 monclient: authenticate success, global_id 4222 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f427518df00 10 monclient: _renew_subs 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f427518df00 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x5566448c5260 con 0x556644841870 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f427518df00 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f427518df00 10 monclient: _renew_subs 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f427518df00 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x55664483e180 con 0x556644841870 2024-08-19T12:38:29.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f427518df00 1 librados: init done 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f427518df00 10 librados: wait_for_osdmap waiting 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f425c002450 con 0x556644841870 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f425c0029c0 con 0x556644841870 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f4272477640 4 set_mon_vals no callback set 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.993+0000 7f425bfff640 10 monclient: dump: 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:29.013 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.994+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f425c002b60 con 0x556644841870 2024-08-19T12:38:29.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.994+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(16..16 src has 1..16) ==== 3495+0+0 (unknown 1240527274 0 0) 0x7f425c04c320 con 0x556644841870 2024-08-19T12:38:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.994+0000 7f425bfff640 10 client.4222.objecter ms_dispatch 0x556644725a00 osd_map(16..16 src has 1..16) 2024-08-19T12:38:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.994+0000 7f425bfff640 3 client.4222.objecter handle_osd_map got epochs [16,16] > 0 2024-08-19T12:38:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.994+0000 7f425bfff640 3 client.4222.objecter handle_osd_map decoding full epoch 16 2024-08-19T12:38:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.995+0000 7f425bfff640 20 client.4222.objecter dump_active .. 0 homeless 2024-08-19T12:38:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.995+0000 7f427518df00 10 librados: wait_for_osdmap done waiting 2024-08-19T12:38:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.995+0000 7f427518df00 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-08-19T12:38:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.995+0000 7f427518df00 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-08-19T12:38:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.995+0000 7f427518df00 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.995+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x5566448c0b60 con 0x556644841870 2024-08-19T12:38:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.995+0000 7f4273c7a640 10 monclient: get_auth_request con 0x7f4254048270 auth_method 0 2024-08-19T12:38:29.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:28.997+0000 7f425bfff640 10 client.4222.objecter ms_handle_connect 0x7f4254048270 2024-08-19T12:38:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.946+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 141+0+0 (unknown 1851909647 0 0) 0x7f425c015e60 con 0x556644841870 2024-08-19T12:38:29.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.946+0000 7f425bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-08-19T12:38:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.946+0000 7f425bfff640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-08-19T12:38:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.946+0000 7f427518df00 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.946+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4268003190 con 0x556644841870 2024-08-19T12:38:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 10 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7f425c015110 con 0x556644841870 2024-08-19T12:38:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f425bfff640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-08-19T12:38:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f4272477640 10 client.4222.objecter _get_latest_version latest 17, waiting 2024-08-19T12:38:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f4272477640 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f4272477640 10 monclient: _renew_subs 2024-08-19T12:38:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f4272477640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f4272477640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=17}) -- 0x7f424c003d70 con 0x556644841870 2024-08-19T12:38:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 11 ==== osd_map(17..17 src has 1..17) ==== 658+0+0 (unknown 4127354426 0 0) 0x7f425c0159c0 con 0x556644841870 2024-08-19T12:38:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f425bfff640 10 client.4222.objecter ms_dispatch 0x556644725a00 osd_map(17..17 src has 1..17) 2024-08-19T12:38:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f425bfff640 3 client.4222.objecter handle_osd_map got epochs [17,17] > 16 2024-08-19T12:38:29.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f425bfff640 3 client.4222.objecter handle_osd_map decoding incremental epoch 17 2024-08-19T12:38:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f425bfff640 20 client.4222.objecter dump_active .. 0 homeless 2024-08-19T12:38:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 10 librados: create oid=rbd_trash nspace= 2024-08-19T12:38:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 10 client.4222.objecter _op_submit op 0x5566448c1150 2024-08-19T12:38:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 20 client.4222.objecter _calc_target epoch 17 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T12:38:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 20 client.4222.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-08-19T12:38:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 10 client.4222.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [5,2] primary 5 2024-08-19T12:38:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 20 client.4222.objecter _get_session s=0x55664483a6b0 osd=5 3 2024-08-19T12:38:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 10 client.4222.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.5 2024-08-19T12:38:29.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 20 client.4222.objecter get_session s=0x55664483a6b0 osd=5 3 2024-08-19T12:38:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 15 client.4222.objecter _session_op_assign 5 1 2024-08-19T12:38:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 15 client.4222.objecter _send_op 1 to 2.3 on osd.5 2024-08-19T12:38:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6801/3713183117 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x5566448c4610 con 0x5566448c1b10 2024-08-19T12:38:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 20 client.4222.objecter put_session s=0x55664483a6b0 osd=5 4 2024-08-19T12:38:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f427518df00 5 client.4222.objecter 1 in flight 2024-08-19T12:38:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.953+0000 7f4274edc640 10 monclient: get_auth_request con 0x5566448c1b10 auth_method 0 2024-08-19T12:38:29.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.958+0000 7f425bfff640 10 client.4222.objecter ms_handle_connect 0x5566448c1b10 2024-08-19T12:38:29.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.959+0000 7f4274edc640 1 -- 172.21.15.26:0/2983727097 <== osd.5 v1:172.21.15.104:6801/3713183117 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) ==== 153+0+0 (unknown 1703302865 0 0) 0x7f4264004530 con 0x5566448c1b10 2024-08-19T12:38:29.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.959+0000 7f4274edc640 10 client.4222.objecter ms_dispatch 0x556644725a00 osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) 2024-08-19T12:38:29.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.959+0000 7f4274edc640 10 client.4222.objecter in handle_osd_op_reply 2024-08-19T12:38:29.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.959+0000 7f4274edc640 7 client.4222.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-08-19T12:38:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.959+0000 7f4274edc640 10 client.4222.objecter op 0 rval 0 len 0 2024-08-19T12:38:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.959+0000 7f4274edc640 15 client.4222.objecter handle_osd_op_reply completed tid 1 2024-08-19T12:38:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.959+0000 7f4274edc640 15 client.4222.objecter _finish_op 1 2024-08-19T12:38:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.959+0000 7f4274edc640 20 client.4222.objecter put_session s=0x55664483a6b0 osd=5 4 2024-08-19T12:38:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.959+0000 7f4274edc640 15 client.4222.objecter _session_op_remove 5 1 2024-08-19T12:38:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.959+0000 7f4274edc640 5 client.4222.objecter 0 in flight 2024-08-19T12:38:29.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.959+0000 7f427518df00 10 librados: Objecter returned from create r=0 2024-08-19T12:38:29.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 10 client.4222.objecter _op_submit op 0x556644941630 2024-08-19T12:38:29.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 20 client.4222.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 20 client.4222.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 10 client.4222.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 20 client.4222.objecter _get_session s=0x55664483a6b0 osd=5 3 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 10 client.4222.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.5 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 20 client.4222.objecter get_session s=0x55664483a6b0 osd=5 3 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 15 client.4222.objecter _session_op_assign 5 2 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 15 client.4222.objecter _send_op 2 to 2.2 on osd.5 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6801/3713183117 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x556644941b80 con 0x5566448c1b10 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 20 client.4222.objecter put_session s=0x55664483a6b0 osd=5 4 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 5 client.4222.objecter 1 in flight 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f4274edc640 1 -- 172.21.15.26:0/2983727097 <== osd.5 v1:172.21.15.104:6801/3713183117 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 4274215525 0 0) 0x7f4264004530 con 0x5566448c1b10 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f4274edc640 10 client.4222.objecter ms_dispatch 0x556644725a00 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T12:38:29.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f4274edc640 10 client.4222.objecter in handle_osd_op_reply 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f4274edc640 7 client.4222.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f4274edc640 10 client.4222.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f4274edc640 10 client.4222.objecter op 0 rval 0 len 0 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f4274edc640 15 client.4222.objecter handle_osd_op_reply completed tid 2 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f4274edc640 15 client.4222.objecter _finish_op 2 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f4274edc640 20 client.4222.objecter put_session s=0x55664483a6b0 osd=5 4 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f4274edc640 15 client.4222.objecter _session_op_remove 5 2 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f4274edc640 5 client.4222.objecter 0 in flight 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 10 client.4222.objecter _op_submit op 0x5566449407c0 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 20 client.4222.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 20 client.4222.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 10 client.4222.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 20 client.4222.objecter _get_session s=0x55664483a6b0 osd=5 3 2024-08-19T12:38:29.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 10 client.4222.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.5 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 20 client.4222.objecter get_session s=0x55664483a6b0 osd=5 3 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 15 client.4222.objecter _session_op_assign 5 3 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 15 client.4222.objecter _send_op 3 to 2.2 on osd.5 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6801/3713183117 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x556644940c70 con 0x5566448c1b10 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 20 client.4222.objecter put_session s=0x55664483a6b0 osd=5 4 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.962+0000 7f427518df00 5 client.4222.objecter 1 in flight 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4274edc640 1 -- 172.21.15.26:0/2983727097 <== osd.5 v1:172.21.15.104:6801/3713183117 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 4287864937 0 0) 0x7f4264004530 con 0x5566448c1b10 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4274edc640 10 client.4222.objecter ms_dispatch 0x556644725a00 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4274edc640 10 client.4222.objecter in handle_osd_op_reply 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4274edc640 7 client.4222.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4274edc640 10 client.4222.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4274edc640 10 client.4222.objecter op 0 rval 0 len 0 2024-08-19T12:38:29.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4274edc640 15 client.4222.objecter handle_osd_op_reply completed tid 3 2024-08-19T12:38:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4274edc640 15 client.4222.objecter _finish_op 3 2024-08-19T12:38:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4274edc640 20 client.4222.objecter put_session s=0x55664483a6b0 osd=5 4 2024-08-19T12:38:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4274edc640 15 client.4222.objecter _session_op_remove 5 3 2024-08-19T12:38:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4274edc640 5 client.4222.objecter 0 in flight 2024-08-19T12:38:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4271c76640 10 client.4222.objecter allocate_selfmanaged_snap; pool: 2 2024-08-19T12:38:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4271c76640 10 client.4222.objecter pool_op_submit 4 2024-08-19T12:38:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4271c76640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:29.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:29.963+0000 7f4271c76640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6789/0 -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f42500030d0 con 0x556644841870 2024-08-19T12:38:30.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 12 ==== pool_op_reply(tid 4 (0) Success v18) ==== 55+0+0 (unknown 2698192564 0 0) 0x7f425c015760 con 0x556644841870 2024-08-19T12:38:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 10 client.4222.objecter ms_dispatch 0x556644725a00 pool_op_reply(tid 4 (0) Success v18) 2024-08-19T12:38:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 10 client.4222.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v18) 2024-08-19T12:38:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 10 client.4222.objecter have request 4 at 0x7f4250003060 Op: create unmanaged snap 2024-08-19T12:38:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 20 client.4222.objecter waiting for client to reach epoch 18 before calling back 2024-08-19T12:38:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 10 monclient: _renew_subs 2024-08-19T12:38:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=18}) -- 0x5566448c59e0 con 0x556644841870 2024-08-19T12:38:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 10 client.4222.objecter done 2024-08-19T12:38:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 13 ==== osd_map(18..18 src has 1..18) ==== 626+0+0 (unknown 4210562932 0 0) 0x7f425c014a20 con 0x556644841870 2024-08-19T12:38:30.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 10 client.4222.objecter ms_dispatch 0x556644725a00 osd_map(18..18 src has 1..18) 2024-08-19T12:38:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 3 client.4222.objecter handle_osd_map got epochs [18,18] > 17 2024-08-19T12:38:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 3 client.4222.objecter handle_osd_map decoding incremental epoch 18 2024-08-19T12:38:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f425bfff640 20 client.4222.objecter dump_active .. 0 homeless 2024-08-19T12:38:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 20 librados: queue_aio_write 0x55664493cbc0 completion 0x55664470cc00 write_seq 1 2024-08-19T12:38:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 10 client.4222.objecter _op_submit op 0x7f424c006460 2024-08-19T12:38:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 20 client.4222.objecter _calc_target epoch 18 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T12:38:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 20 client.4222.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-08-19T12:38:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 10 client.4222.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-08-19T12:38:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 20 client.4222.objecter _get_session s=0x55664483a6b0 osd=5 3 2024-08-19T12:38:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 10 client.4222.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.5 2024-08-19T12:38:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 20 client.4222.objecter get_session s=0x55664483a6b0 osd=5 3 2024-08-19T12:38:30.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 15 client.4222.objecter _session_op_assign 5 5 2024-08-19T12:38:30.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 15 client.4222.objecter _send_op 5 to 2.2 on osd.5 2024-08-19T12:38:30.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6801/3713183117 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7f424c006a80 con 0x5566448c1b10 2024-08-19T12:38:30.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 20 client.4222.objecter put_session s=0x55664483a6b0 osd=5 4 2024-08-19T12:38:30.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.947+0000 7f4272477640 5 client.4222.objecter 1 in flight 2024-08-19T12:38:30.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4274edc640 1 -- 172.21.15.26:0/2983727097 <== osd.5 v1:172.21.15.104:6801/3713183117 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) ==== 194+0+0 (unknown 2910594948 0 0) 0x7f4264004530 con 0x5566448c1b10 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4274edc640 10 client.4222.objecter ms_dispatch 0x556644725a00 osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4274edc640 10 client.4222.objecter in handle_osd_op_reply 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4274edc640 7 client.4222.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4274edc640 10 client.4222.objecter op 0 rval 0 len 0 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4274edc640 10 client.4222.objecter op 1 rval 0 len 0 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4274edc640 15 client.4222.objecter handle_osd_op_reply completed tid 5 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4274edc640 15 client.4222.objecter _finish_op 5 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4274edc640 20 client.4222.objecter put_session s=0x55664483a6b0 osd=5 4 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4274edc640 15 client.4222.objecter _session_op_remove 5 5 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4274edc640 5 client.4222.objecter 0 in flight 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4274edc640 20 librados: complete_aio_write 0x55664470cc00 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4271c76640 10 client.4222.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-08-19T12:38:30.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4271c76640 10 client.4222.objecter pool_op_submit 6 2024-08-19T12:38:30.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4271c76640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:30.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:30.949+0000 7f4271c76640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6789/0 -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) -- 0x7f4250004540 con 0x556644841870 2024-08-19T12:38:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.959+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 14 ==== pool_op_reply(tid 6 (0) Success v19) ==== 43+0+0 (unknown 4289217514 0 0) 0x7f425c01e750 con 0x556644841870 2024-08-19T12:38:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.959+0000 7f425bfff640 10 client.4222.objecter ms_dispatch 0x556644725a00 pool_op_reply(tid 6 (0) Success v19) 2024-08-19T12:38:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.959+0000 7f425bfff640 10 client.4222.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v19) 2024-08-19T12:38:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.959+0000 7f425bfff640 10 client.4222.objecter have request 6 at 0x7f4250003060 Op: delete unmanaged snap 2024-08-19T12:38:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.959+0000 7f425bfff640 20 client.4222.objecter waiting for client to reach epoch 19 before calling back 2024-08-19T12:38:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.959+0000 7f425bfff640 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.959+0000 7f425bfff640 10 monclient: _renew_subs 2024-08-19T12:38:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.959+0000 7f425bfff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:31.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.959+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=19}) -- 0x5566448c6a60 con 0x556644841870 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.959+0000 7f425bfff640 10 client.4222.objecter done 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f425bfff640 1 -- 172.21.15.26:0/2983727097 <== mon.1 v1:172.21.15.104:6789/0 15 ==== osd_map(19..19 src has 1..19) ==== 654+0+0 (unknown 4175944276 0 0) 0x7f425c011d80 con 0x556644841870 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f425bfff640 10 client.4222.objecter ms_dispatch 0x556644725a00 osd_map(19..19 src has 1..19) 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f425bfff640 3 client.4222.objecter handle_osd_map got epochs [19,19] > 18 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f425bfff640 3 client.4222.objecter handle_osd_map decoding incremental epoch 19 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f425bfff640 20 client.4222.objecter dump_active .. 0 homeless 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 20 librados: queue_aio_write 0x55664493cbc0 completion 0x7f4250001d60 write_seq 2 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 10 client.4222.objecter _op_submit op 0x7f424c006460 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 20 client.4222.objecter _calc_target epoch 19 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 20 client.4222.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 10 client.4222.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 20 client.4222.objecter _get_session s=0x55664483a6b0 osd=5 3 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 10 client.4222.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.5 2024-08-19T12:38:31.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 20 client.4222.objecter get_session s=0x55664483a6b0 osd=5 3 2024-08-19T12:38:31.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 15 client.4222.objecter _session_op_assign 5 7 2024-08-19T12:38:31.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 15 client.4222.objecter _send_op 7 to 2.2 on osd.5 2024-08-19T12:38:31.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 1 -- 172.21.15.26:0/2983727097 --> v1:172.21.15.104:6801/3713183117 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7f424c006e80 con 0x5566448c1b10 2024-08-19T12:38:31.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 20 client.4222.objecter put_session s=0x55664483a6b0 osd=5 4 2024-08-19T12:38:31.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.960+0000 7f4272477640 5 client.4222.objecter 1 in flight 2024-08-19T12:38:31.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.961+0000 7f4274edc640 1 -- 172.21.15.26:0/2983727097 <== osd.5 v1:172.21.15.104:6801/3713183117 5 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) ==== 152+0+0 (unknown 900577457 0 0) 0x7f4264004530 con 0x5566448c1b10 2024-08-19T12:38:31.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.961+0000 7f4274edc640 10 client.4222.objecter ms_dispatch 0x556644725a00 osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) 2024-08-19T12:38:31.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.961+0000 7f4274edc640 10 client.4222.objecter in handle_osd_op_reply 2024-08-19T12:38:31.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.961+0000 7f4274edc640 7 client.4222.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-08-19T12:38:31.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.961+0000 7f4274edc640 10 client.4222.objecter op 0 rval 0 len 0 2024-08-19T12:38:31.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.961+0000 7f4274edc640 15 client.4222.objecter handle_osd_op_reply completed tid 7 2024-08-19T12:38:31.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.961+0000 7f4274edc640 15 client.4222.objecter _finish_op 7 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.961+0000 7f4274edc640 20 client.4222.objecter put_session s=0x55664483a6b0 osd=5 4 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.961+0000 7f4274edc640 15 client.4222.objecter _session_op_remove 5 7 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.961+0000 7f4274edc640 5 client.4222.objecter 0 in flight 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.961+0000 7f4274edc640 20 librados: complete_aio_write 0x7f4250001d60 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 10 librados: watch_flush enter 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 10 librados: watch_flush exit 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 10 client.4222.objecter close_session for osd.5 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 >> v1:172.21.15.104:6801/3713183117 conn(0x5566448c1b10 legacy=0x5566448c3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 20 client.4222.objecter put_session s=0x55664483a6b0 osd=5 1 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 20 client.4222.objecter shutdown clearing up homeless session... 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 10 client.4222.objecter successfully canceled tick 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 >> v1:172.21.15.104:6800/899887506 conn(0x7f4254048270 legacy=0x7f425404a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 10 monclient: shutdown 2024-08-19T12:38:31.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:31.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 >> v1:172.21.15.104:6789/0 conn(0x556644841870 legacy=0x55664483da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:31.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f4274edc640 1 -- 172.21.15.26:0/2983727097 reap_dead start 2024-08-19T12:38:31.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 shutdown_connections 2024-08-19T12:38:31.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 >> 172.21.15.26:0/2983727097 conn(0x556644823830 msgr2=0x556644843d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:31.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 shutdown_connections 2024-08-19T12:38:31.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 1 -- 172.21.15.26:0/2983727097 wait complete. 2024-08-19T12:38:31.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:31.962+0000 7f427518df00 1 librados: shutdown 2024-08-19T12:38:31.975 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-08-19T12:38:31.976 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-08-19T12:38:31.976 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-19T12:38:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.122+0000 7f919ed3c640 10 monclient: get_monmap_and_config 2024-08-19T12:38:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.122+0000 7f919ed3c640 10 monclient: build_initial_monmap 2024-08-19T12:38:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.122+0000 7f919ed3c640 10 monclient: monmap: 2024-08-19T12:38:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:32.123389+0000 2024-08-19T12:38:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:32.123389+0000 2024-08-19T12:38:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:32.125 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 1 Processor -- start 2024-08-19T12:38:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 1 -- start start 2024-08-19T12:38:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 10 monclient: init 2024-08-19T12:38:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 10 monclient: _add_conns ranks=[8,4,1,3,0,2,6,5,7] 2024-08-19T12:38:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 10 monclient(hunting): picked mon.noname-e con 0x7f919806e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 10 monclient(hunting): picked mon.noname-c con 0x7f91980a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 10 monclient(hunting): picked mon.noname-f con 0x7f91980a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9198136790 con 0x7f91980a5d50 2024-08-19T12:38:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919809ddf0 con 0x7f91980a5920 2024-08-19T12:38:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919809e0d0 con 0x7f919806e7b0 2024-08-19T12:38:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919ed3c640 10 monclient(hunting): authenticate will time out at 706.210834s 2024-08-19T12:38:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919cad8640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f919806e7b0 0x7f9198136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:33900/0 (socket says 172.21.15.26:33900) 2024-08-19T12:38:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f919cad8640 1 -- 172.21.15.26:0/561846944 learned_addr learned my addr 172.21.15.26:0/561846944 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.123+0000 7f9197fff640 1 --1- 172.21.15.26:0/561846944 >> v1:172.21.15.26:6791/0 conn(0x7f91980a5920 0x7f919813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57760/0 (socket says 172.21.15.26:57760) 2024-08-19T12:38:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f9197fff640 1 -- 172.21.15.26:0/561846944 >> v1:172.21.15.26:6791/0 conn(0x7f91980a5920 legacy=0x7f919813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f919cad8640 1 -- 172.21.15.26:0/561846944 >> v1:172.21.15.26:6793/0 conn(0x7f919806e7b0 legacy=0x7f9198136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 1 -- 172.21.15.26:0/561846944 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1687790983 0 0) 0x7f9184001450 con 0x7f91980a5920 2024-08-19T12:38:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient(hunting): my global_id is 4261 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 1 -- 172.21.15.26:0/561846944 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9188003520 con 0x7f91980a5920 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 1 -- 172.21.15.26:0/561846944 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 705629680 0 0) 0x7f919809e0d0 con 0x7f919806e7b0 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient(hunting): my global_id is 4274 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 1 -- 172.21.15.26:0/561846944 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9184001450 con 0x7f919806e7b0 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f919d2d9640 1 -- 172.21.15.26:0/561846944 >> v1:172.21.15.104:6789/0 conn(0x7f91980a5d50 legacy=0x7f9198141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 1 -- 172.21.15.26:0/561846944 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1236743016 0 0) 0x7f9184001450 con 0x7f919806e7b0 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 1 -- 172.21.15.26:0/561846944 >> v1:172.21.15.26:6791/0 conn(0x7f91980a5920 legacy=0x7f919813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 1 -- 172.21.15.26:0/561846944 >> v1:172.21.15.104:6789/0 conn(0x7f91980a5d50 legacy=0x7f9198141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 1 monclient: found mon.noname-e 2024-08-19T12:38:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 1 -- 172.21.15.26:0/561846944 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919809e3b0 con 0x7f919806e7b0 2024-08-19T12:38:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 1 -- 172.21.15.26:0/561846944 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f918c001a40 con 0x7f919806e7b0 2024-08-19T12:38:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f919ed3c640 5 monclient: authenticate success, global_id 4274 2024-08-19T12:38:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f919ed3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient: dump: 2024-08-19T12:38:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f919ed3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 1 -- 172.21.15.26:0/561846944 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f918c001d10 con 0x7f919806e7b0 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 1 -- 172.21.15.26:0/561846944 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f918c004be0 con 0x7f919806e7b0 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f919ed3c640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f919ed3c640 4 set_mon_vals no callback set 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.126+0000 7f91977fe640 10 monclient: dump: 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.127+0000 7f919ed3c640 10 monclient: shutdown 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.127+0000 7f919ed3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.127+0000 7f919ed3c640 1 -- 172.21.15.26:0/561846944 >> v1:172.21.15.26:6793/0 conn(0x7f919806e7b0 legacy=0x7f9198136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.127+0000 7f919ed3c640 1 -- 172.21.15.26:0/561846944 shutdown_connections 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.127+0000 7f919ed3c640 1 -- 172.21.15.26:0/561846944 >> 172.21.15.26:0/561846944 conn(0x7f91980a8740 msgr2=0x7f91980a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.127+0000 7f919ed3c640 1 -- 172.21.15.26:0/561846944 shutdown_connections 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.127+0000 7f919ed3c640 1 -- 172.21.15.26:0/561846944 wait complete. 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.127+0000 7f919ed3c640 10 monclient: build_initial_monmap 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.127+0000 7f919ed3c640 10 monclient: monmap: 2024-08-19T12:38:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:32.139 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 1 librados: starting msgr at 2024-08-19T12:38:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 1 librados: starting objecter 2024-08-19T12:38:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 1 Processor -- start 2024-08-19T12:38:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 1 -- start start 2024-08-19T12:38:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 1 librados: setting wanted keys 2024-08-19T12:38:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 1 librados: calling monclient init 2024-08-19T12:38:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 10 monclient: init 2024-08-19T12:38:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 10 monclient: _add_conns ranks=[0,7,8,6,2,3,4,5,1] 2024-08-19T12:38:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 10 monclient(hunting): picked mon.noname-a con 0x7f919806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 10 monclient(hunting): picked mon.noname-i con 0x7f91980a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 10 monclient(hunting): picked mon.noname-e con 0x7f91980a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919813a870 con 0x7f919806e7b0 2024-08-19T12:38:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9198139bd0 con 0x7f91980a5920 2024-08-19T12:38:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91981345e0 con 0x7f91980a5d50 2024-08-19T12:38:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.128+0000 7f919ed3c640 10 monclient(hunting): authenticate will time out at 706.215934s 2024-08-19T12:38:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.129+0000 7f919cad8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f919806e7b0 0x7f91981379b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58328/0 (socket says 172.21.15.26:58328) 2024-08-19T12:38:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.129+0000 7f919cad8640 1 -- 172.21.15.26:0/3413127982 learned_addr learned my addr 172.21.15.26:0/3413127982 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.129+0000 7f919d2d9640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f91980a5d50 0x7f91981d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:33904/0 (socket says 172.21.15.26:33904) 2024-08-19T12:38:32.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f919cad8640 1 -- 172.21.15.26:0/3413127982 >> v1:172.21.15.26:6789/0 conn(0x7f919806e7b0 legacy=0x7f91981379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f919d2d9640 1 -- 172.21.15.26:0/3413127982 >> v1:172.21.15.26:6793/0 conn(0x7f91980a5d50 legacy=0x7f91981d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 client.?.objecter ms_handle_connect 0x7f919806e7b0 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 client.?.objecter ms_handle_connect 0x7f91980a5d50 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2619884315 0 0) 0x7f91981345e0 con 0x7f91980a5d50 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): my global_id is 4283 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f917c004eb0 con 0x7f91980a5d50 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 247769452 0 0) 0x7f919813a870 con 0x7f919806e7b0 2024-08-19T12:38:32.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): my global_id is 4203 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91981345e0 con 0x7f919806e7b0 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f9197fff640 1 -- 172.21.15.26:0/3413127982 >> v1:172.21.15.104:6792/0 conn(0x7f91980a5920 legacy=0x7f91981380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 client.?.objecter ms_handle_connect 0x7f91980a5920 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2979666067 0 0) 0x7f917c004eb0 con 0x7f91980a5d50 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f919813a870 con 0x7f91980a5d50 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9190002030 con 0x7f91980a5d50 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:32.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): dump: 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 597047265 0 0) 0x7f9198139bd0 con 0x7f91980a5920 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): my global_id is 4237 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f917c004eb0 con 0x7f91980a5920 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 712399225 0 0) 0x7f91981345e0 con 0x7f919806e7b0 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9198139bd0 con 0x7f919806e7b0 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f918c004b20 con 0x7f919806e7b0 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient(hunting): dump: 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 757565300 0 0) 0x7f919813a870 con 0x7f91980a5d50 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 >> v1:172.21.15.104:6792/0 conn(0x7f91980a5920 legacy=0x7f91981380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 >> v1:172.21.15.26:6789/0 conn(0x7f919806e7b0 legacy=0x7f91981379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient: _finish_hunting 0 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 monclient: found mon.e 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919813ab30 con 0x7f91980a5d50 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient: _finish_auth 0 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 10 monclient: _check_auth_tickets 2024-08-19T12:38:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f91957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f919ed3c640 5 monclient: authenticate success, global_id 4283 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f919ed3c640 10 monclient: _renew_subs 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f919ed3c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f919ed3c640 1 -- 172.21.15.26:0/3413127982 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f91981390b0 con 0x7f91980a5d50 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f919ed3c640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f919ed3c640 10 monclient: _renew_subs 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f919ed3c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f919ed3c640 1 -- 172.21.15.26:0/3413127982 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f91981389b0 con 0x7f91980a5d50 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.131+0000 7f919ed3c640 1 librados: init done 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.132+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9190003d70 con 0x7f91980a5d50 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.132+0000 7f91957fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.132+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f91900042a0 con 0x7f91980a5d50 2024-08-19T12:38:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.132+0000 7f91957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.132+0000 7f9196ffd640 4 set_mon_vals no callback set 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.132+0000 7f91957fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.132+0000 7f91957fa640 10 monclient: dump: 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.132+0000 7f919ed3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.132+0000 7f919ed3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.132+0000 7f919ed3c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.132+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f9190004440 con 0x7f91980a5d50 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.132+0000 7f919ed3c640 1 -- 172.21.15.26:0/3413127982 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9178005080 con 0x7f91980a5d50 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.133+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f919004dc70 con 0x7f91980a5d50 2024-08-19T12:38:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.133+0000 7f91957fa640 10 client.4283.objecter ms_dispatch 0x7f9198000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.133+0000 7f91957fa640 3 client.4283.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.133+0000 7f91957fa640 3 client.4283.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.133+0000 7f91957fa640 20 client.4283.objecter dump_active .. 0 homeless 2024-08-19T12:38:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.133+0000 7f919cad8640 10 monclient: get_auth_request con 0x7f917c041720 auth_method 0 2024-08-19T12:38:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.135+0000 7f91957fa640 10 client.4283.objecter ms_handle_connect 0x7f917c041720 2024-08-19T12:38:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.137+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f91900170e0 con 0x7f91980a5d50 2024-08-19T12:38:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.137+0000 7f91957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.137+0000 7f91957fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:32.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.323+0000 7f919ed3c640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-08-19T12:38:32.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.323+0000 7f919ed3c640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-08-19T12:38:32.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.323+0000 7f919ed3c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:32.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.323+0000 7f919ed3c640 1 -- 172.21.15.26:0/3413127982 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f9178005370 con 0x7f91980a5d50 2024-08-19T12:38:32.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.323+0000 7f91957fa640 1 -- 172.21.15.26:0/3413127982 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v6) ==== 86+0+159 (unknown 2698850854 0 4258242034) 0x7f919001b5d0 con 0x7f91980a5d50 2024-08-19T12:38:32.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.323+0000 7f91957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-08-19T12:38:32.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.323+0000 7f91957fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:38:32.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.325+0000 7f919ed3c640 10 librados: watch_flush enter 2024-08-19T12:38:32.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.325+0000 7f919ed3c640 10 librados: watch_flush exit 2024-08-19T12:38:32.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.325+0000 7f919ed3c640 20 client.4283.objecter shutdown clearing up homeless session... 2024-08-19T12:38:32.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.325+0000 7f919ed3c640 10 client.4283.objecter successfully canceled tick 2024-08-19T12:38:32.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.325+0000 7f919ed3c640 1 -- 172.21.15.26:0/3413127982 >> v1:172.21.15.104:6800/899887506 conn(0x7f917c041720 legacy=0x7f917c043be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.325+0000 7f919ed3c640 10 monclient: shutdown 2024-08-19T12:38:32.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.325+0000 7f919ed3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:32.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.325+0000 7f919ed3c640 1 -- 172.21.15.26:0/3413127982 >> v1:172.21.15.26:6793/0 conn(0x7f91980a5d50 legacy=0x7f91981d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.325+0000 7f919ed3c640 1 -- 172.21.15.26:0/3413127982 shutdown_connections 2024-08-19T12:38:32.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.325+0000 7f919ed3c640 1 -- 172.21.15.26:0/3413127982 >> 172.21.15.26:0/3413127982 conn(0x7f91980a8740 msgr2=0x7f91980aa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:32.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.325+0000 7f919ed3c640 1 -- 172.21.15.26:0/3413127982 shutdown_connections 2024-08-19T12:38:32.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.325+0000 7f919ed3c640 1 -- 172.21.15.26:0/3413127982 wait complete. 2024-08-19T12:38:32.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.326+0000 7f919ed3c640 1 librados: shutdown 2024-08-19T12:38:32.336 INFO:teuthology.orchestra.run.smithi026.stdout:[{"version":6,"timestamp":"2024-08-19T12:38:20.634661+0000","name":"","changes":[{"name":"osd.3/osd_mclock_max_capacity_iops_ssd","new_value":"5847.866536"}]}] 2024-08-19T12:38:32.337 INFO:tasks.ceph_manager:config epoch is 6 2024-08-19T12:38:32.337 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-19T12:38:32.337 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-08-19T12:38:32.337 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-08-19T12:38:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.435+0000 7f68fb1f2640 10 monclient: get_monmap_and_config 2024-08-19T12:38:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.435+0000 7f68fb1f2640 10 monclient: build_initial_monmap 2024-08-19T12:38:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.435+0000 7f68fb1f2640 10 monclient: monmap: 2024-08-19T12:38:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:32.436065+0000 2024-08-19T12:38:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:32.436065+0000 2024-08-19T12:38:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:32.436 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:32.437 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:32.437 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:32.437 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:32.437 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:32.437 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:32.437 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:32.437 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:32.437 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.435+0000 7f68fb1f2640 1 Processor -- start 2024-08-19T12:38:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.435+0000 7f68fb1f2640 1 -- start start 2024-08-19T12:38:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.435+0000 7f68fb1f2640 10 monclient: init 2024-08-19T12:38:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68fb1f2640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68fb1f2640 10 monclient: _add_conns ranks=[7,0,2,4,3,8,6,5,1] 2024-08-19T12:38:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68fb1f2640 10 monclient(hunting): picked mon.noname-i con 0x7f68f406e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68fb1f2640 10 monclient(hunting): picked mon.noname-a con 0x7f68f40a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68fb1f2640 10 monclient(hunting): picked mon.noname-b con 0x7f68f40a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68fb1f2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f4136790 con 0x7f68f40a5920 2024-08-19T12:38:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68fb1f2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f409ddf0 con 0x7f68f40a5d50 2024-08-19T12:38:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68fb1f2640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f409e0d0 con 0x7f68f406e7b0 2024-08-19T12:38:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68fb1f2640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:32.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68fb1f2640 10 monclient(hunting): authenticate will time out at 706.523517s 2024-08-19T12:38:32.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68f978f640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f68f40a5d50 0x7f68f4141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50240/0 (socket says 172.21.15.26:50240) 2024-08-19T12:38:32.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68f978f640 1 -- 172.21.15.26:0/1115625807 learned_addr learned my addr 172.21.15.26:0/1115625807 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:32.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.436+0000 7f68f3fff640 1 --1- 172.21.15.26:0/1115625807 >> v1:172.21.15.26:6789/0 conn(0x7f68f40a5920 0x7f68f413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58340/0 (socket says 172.21.15.26:58340) 2024-08-19T12:38:32.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.438+0000 7f68f978f640 1 -- 172.21.15.26:0/1115625807 >> v1:172.21.15.26:6790/0 conn(0x7f68f40a5d50 legacy=0x7f68f4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.438+0000 7f68f3fff640 1 -- 172.21.15.26:0/1115625807 >> v1:172.21.15.26:6789/0 conn(0x7f68f40a5920 legacy=0x7f68f413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.438+0000 7f68f37fe640 1 -- 172.21.15.26:0/1115625807 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506534027 0 0) 0x7f68e40017d0 con 0x7f68f40a5d50 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.438+0000 7f68f37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.438+0000 7f68f37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.438+0000 7f68f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.438+0000 7f68f37fe640 10 monclient(hunting): my global_id is 4232 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.438+0000 7f68f37fe640 1 -- 172.21.15.26:0/1115625807 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68e0003520 con 0x7f68f40a5d50 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 1 -- 172.21.15.26:0/1115625807 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2884434368 0 0) 0x7f68f4136790 con 0x7f68f40a5920 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient(hunting): my global_id is 4212 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f8f8e640 1 -- 172.21.15.26:0/1115625807 >> v1:172.21.15.104:6792/0 conn(0x7f68f406e7b0 legacy=0x7f68f4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 1 -- 172.21.15.26:0/1115625807 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68e40017d0 con 0x7f68f40a5920 2024-08-19T12:38:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 1 -- 172.21.15.26:0/1115625807 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2169039133 0 0) 0x7f68e0003520 con 0x7f68f40a5d50 2024-08-19T12:38:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 1 -- 172.21.15.26:0/1115625807 >> v1:172.21.15.104:6792/0 conn(0x7f68f406e7b0 legacy=0x7f68f4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 1 -- 172.21.15.26:0/1115625807 >> v1:172.21.15.26:6789/0 conn(0x7f68f40a5920 legacy=0x7f68f413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 1 monclient: found mon.noname-b 2024-08-19T12:38:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 1 -- 172.21.15.26:0/1115625807 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f409e3b0 con 0x7f68f40a5d50 2024-08-19T12:38:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68fb1f2640 5 monclient: authenticate success, global_id 4232 2024-08-19T12:38:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 1 -- 172.21.15.26:0/1115625807 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f68e4004c20 con 0x7f68f40a5d50 2024-08-19T12:38:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68fb1f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient: dump: 2024-08-19T12:38:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.446 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68fb1f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f978f640 1 --1- 172.21.15.26:0/1115625807 >> v1:172.21.15.26:6790/0 conn(0x7f68f40a5d50 0x7f68f4141a20 :-1 s=OPENED pgs=61 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f68e4004e40 config(0 keys) 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 1 -- 172.21.15.26:0/1115625807 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f68e4004e40 con 0x7f68f40a5d50 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:32.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68fb1f2640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68fb1f2640 4 set_mon_vals no callback set 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68fb1f2640 10 monclient: shutdown 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68fb1f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1115625807 >> v1:172.21.15.26:6790/0 conn(0x7f68f40a5d50 legacy=0x7f68f4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 1 -- 172.21.15.26:0/1115625807 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f68e4005370 con 0x7f68f40a5d50 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68f37fe640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1115625807 shutdown_connections 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1115625807 >> 172.21.15.26:0/1115625807 conn(0x7f68f40a8740 msgr2=0x7f68f40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1115625807 shutdown_connections 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.439+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1115625807 wait complete. 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.440+0000 7f68fb1f2640 10 monclient: build_initial_monmap 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.440+0000 7f68fb1f2640 10 monclient: monmap: 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.440+0000 7f68fb1f2640 1 librados: starting msgr at 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.440+0000 7f68fb1f2640 1 librados: starting objecter 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.440+0000 7f68fb1f2640 1 Processor -- start 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.440+0000 7f68fb1f2640 1 -- start start 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.440+0000 7f68fb1f2640 1 librados: setting wanted keys 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.440+0000 7f68fb1f2640 1 librados: calling monclient init 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.440+0000 7f68fb1f2640 10 monclient: init 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.441+0000 7f68fb1f2640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.441+0000 7f68fb1f2640 10 monclient: _add_conns ranks=[3,6,4,7,2,5,1,8,0] 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.441+0000 7f68fb1f2640 10 monclient(hunting): picked mon.noname-g con 0x7f68f406e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.441+0000 7f68fb1f2640 10 monclient(hunting): picked mon.noname-d con 0x7f68f40a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.441+0000 7f68fb1f2640 10 monclient(hunting): picked mon.noname-c con 0x7f68f40a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.441+0000 7f68fb1f2640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f413b160 con 0x7f68f406e7b0 2024-08-19T12:38:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.441+0000 7f68fb1f2640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f4138180 con 0x7f68f40a5d50 2024-08-19T12:38:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.441+0000 7f68fb1f2640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f413ae80 con 0x7f68f40a5920 2024-08-19T12:38:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.441+0000 7f68fb1f2640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.441+0000 7f68fb1f2640 10 monclient(hunting): authenticate will time out at 706.528462s 2024-08-19T12:38:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.441+0000 7f68f3fff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f68f40a5920 0x7f68f4137a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58446/0 (socket says 172.21.15.26:58446) 2024-08-19T12:38:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.441+0000 7f68f3fff640 1 -- 172.21.15.26:0/1747401056 learned_addr learned my addr 172.21.15.26:0/1747401056 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f3fff640 1 -- 172.21.15.26:0/1747401056 >> v1:172.21.15.26:6792/0 conn(0x7f68f40a5920 legacy=0x7f68f4137a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 client.?.objecter ms_handle_connect 0x7f68f40a5920 2024-08-19T12:38:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 client.?.objecter ms_handle_connect 0x7f68f40a5d50 2024-08-19T12:38:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f978f640 1 -- 172.21.15.26:0/1747401056 >> v1:172.21.15.26:6791/0 conn(0x7f68f40a5d50 legacy=0x7f68f41d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3140474535 0 0) 0x7f68f413ae80 con 0x7f68f40a5920 2024-08-19T12:38:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 monclient(hunting): my global_id is 4263 2024-08-19T12:38:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68d4004eb0 con 0x7f68f40a5920 2024-08-19T12:38:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2967438996 0 0) 0x7f68f4138180 con 0x7f68f40a5d50 2024-08-19T12:38:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 monclient(hunting): my global_id is 4270 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68f413ae80 con 0x7f68f40a5d50 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f17fa640 10 client.?.objecter ms_handle_connect 0x7f68f406e7b0 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.443+0000 7f68f8f8e640 1 -- 172.21.15.26:0/1747401056 >> v1:172.21.15.104:6790/0 conn(0x7f68f406e7b0 legacy=0x7f68f4137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 398032375 0 0) 0x7f68f413ae80 con 0x7f68f40a5d50 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68f4138180 con 0x7f68f40a5d50 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f68e40020b0 con 0x7f68f40a5d50 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient(hunting): dump: 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2207681915 0 0) 0x7f68d4004eb0 con 0x7f68f40a5920 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68f413ae80 con 0x7f68f40a5920 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f68dc0047e0 con 0x7f68f40a5920 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient(hunting): dump: 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1685237461 0 0) 0x7f68f413b160 con 0x7f68f406e7b0 2024-08-19T12:38:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient(hunting): my global_id is 4269 2024-08-19T12:38:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68d4004eb0 con 0x7f68f406e7b0 2024-08-19T12:38:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2935295014 0 0) 0x7f68f4138180 con 0x7f68f40a5d50 2024-08-19T12:38:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 >> v1:172.21.15.26:6792/0 conn(0x7f68f40a5920 legacy=0x7f68f4137a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 >> v1:172.21.15.104:6790/0 conn(0x7f68f406e7b0 legacy=0x7f68f4137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient: _finish_hunting 0 2024-08-19T12:38:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 monclient: found mon.c 2024-08-19T12:38:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f4139bd0 con 0x7f68f40a5d50 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient: _finish_auth 0 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient: _check_auth_tickets 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68fb1f2640 5 monclient: authenticate success, global_id 4270 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68fb1f2640 10 monclient: _renew_subs 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68fb1f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1747401056 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f68f41428d0 con 0x7f68f40a5d50 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68fb1f2640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68fb1f2640 10 monclient: _renew_subs 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68fb1f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1747401056 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f68f41d7cb0 con 0x7f68f40a5d50 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68fb1f2640 1 librados: init done 2024-08-19T12:38:32.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f68e40052e0 con 0x7f68f40a5d50 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f68e4004ca0 con 0x7f68f40a5d50 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f2ffd640 4 set_mon_vals no callback set 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.444+0000 7f68f17fa640 10 monclient: dump: 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.463 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.445+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f68e4011bc0 con 0x7f68f40a5d50 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.445+0000 7f68fb1f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.445+0000 7f68fb1f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.445+0000 7f68fb1f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:32.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.445+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1747401056 --> v1:172.21.15.26:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68d8005080 con 0x7f68f40a5d50 2024-08-19T12:38:32.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.445+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f68e404db20 con 0x7f68f40a5d50 2024-08-19T12:38:32.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.445+0000 7f68f17fa640 10 client.4270.objecter ms_dispatch 0x7f68f4000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:32.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.445+0000 7f68f17fa640 3 client.4270.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:32.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.445+0000 7f68f17fa640 3 client.4270.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:32.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.445+0000 7f68f17fa640 20 client.4270.objecter dump_active .. 0 homeless 2024-08-19T12:38:32.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.445+0000 7f68f8f8e640 10 monclient: get_auth_request con 0x7f68d40418a0 auth_method 0 2024-08-19T12:38:32.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.448+0000 7f68f17fa640 10 client.4270.objecter ms_handle_connect 0x7f68d40418a0 2024-08-19T12:38:32.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.450+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.4 v1:172.21.15.26:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f68f409ddf0 con 0x7f68f40a5d50 2024-08-19T12:38:32.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.450+0000 7f68f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:32.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.450+0000 7f68f17fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:32.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.681+0000 7f68fb1f2640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-08-19T12:38:32.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.681+0000 7f68fb1f2640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-08-19T12:38:32.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.681+0000 7f68fb1f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:32.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.681+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1747401056 --> v1:172.21.15.26:6791/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f68d8005370 con 0x7f68f40a5d50 2024-08-19T12:38:32.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.682+0000 7f68f17fa640 1 -- 172.21.15.26:0/1747401056 <== mon.4 v1:172.21.15.26:6791/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+96226 (unknown 481731318 0 685372166) 0x7f68e4017090 con 0x7f68f40a5d50 2024-08-19T12:38:32.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.682+0000 7f68f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-08-19T12:38:32.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.682+0000 7f68f17fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:32.688 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:38:32.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.684+0000 7f68fb1f2640 10 librados: watch_flush enter 2024-08-19T12:38:32.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.684+0000 7f68fb1f2640 10 librados: watch_flush exit 2024-08-19T12:38:32.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.684+0000 7f68fb1f2640 20 client.4270.objecter shutdown clearing up homeless session... 2024-08-19T12:38:32.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.684+0000 7f68fb1f2640 10 client.4270.objecter successfully canceled tick 2024-08-19T12:38:32.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.684+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1747401056 >> v1:172.21.15.104:6800/899887506 conn(0x7f68d40418a0 legacy=0x7f68d4043d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.684+0000 7f68fb1f2640 10 monclient: shutdown 2024-08-19T12:38:32.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.684+0000 7f68fb1f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:32.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.684+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1747401056 >> v1:172.21.15.26:6791/0 conn(0x7f68f40a5d50 legacy=0x7f68f41d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.684+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1747401056 shutdown_connections 2024-08-19T12:38:32.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.684+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1747401056 >> 172.21.15.26:0/1747401056 conn(0x7f68f40a8740 msgr2=0x7f68f406dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:32.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.685+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1747401056 shutdown_connections 2024-08-19T12:38:32.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.685+0000 7f68fb1f2640 1 -- 172.21.15.26:0/1747401056 wait complete. 2024-08-19T12:38:32.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.685+0000 7f68fb1f2640 1 librados: shutdown 2024-08-19T12:38:32.696 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":5,"flags":0,"active_gid":4115,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6800","nonce":899887506}]},"active_addr":"172.21.15.104:6800/899887506","active_change":"2024-08-19T12:38:13.084488+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":2512405282}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":3366954181}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":3299363273}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.104:0","nonce":1142013871}]}]} 2024-08-19T12:38:32.698 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-08-19T12:38:32.698 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-08-19T12:38:32.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-19T12:38:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.796+0000 7f28b3e7e640 10 monclient: get_monmap_and_config 2024-08-19T12:38:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.796+0000 7f28b3e7e640 10 monclient: build_initial_monmap 2024-08-19T12:38:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.796+0000 7f28b3e7e640 10 monclient: monmap: 2024-08-19T12:38:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:32.796998+0000 2024-08-19T12:38:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:32.796998+0000 2024-08-19T12:38:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:32.798 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:32.798 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.796+0000 7f28b3e7e640 1 Processor -- start 2024-08-19T12:38:32.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.796+0000 7f28b3e7e640 1 -- start start 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.796+0000 7f28b3e7e640 10 monclient: init 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.797+0000 7f28b3e7e640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.797+0000 7f28b3e7e640 10 monclient: _add_conns ranks=[1,3,2,0,4,5,8,7,6] 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.797+0000 7f28b3e7e640 10 monclient(hunting): picked mon.noname-f con 0x7f28ac073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.797+0000 7f28b3e7e640 10 monclient(hunting): picked mon.noname-g con 0x7f28ac0a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.797+0000 7f28b3e7e640 10 monclient(hunting): picked mon.noname-b con 0x7f28ac13f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.797+0000 7f28b3e7e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28ac09dae0 con 0x7f28ac073470 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.797+0000 7f28b3e7e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28ac136910 con 0x7f28ac13f2e0 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.797+0000 7f28b3e7e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28ac09dd50 con 0x7f28ac0a57a0 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.797+0000 7f28b3e7e640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.797+0000 7f28b3e7e640 10 monclient(hunting): authenticate will time out at 706.884545s 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.797+0000 7f28b241b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f28ac13f2e0 0x7f28ac1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50242/0 (socket says 172.21.15.26:50242) 2024-08-19T12:38:32.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.797+0000 7f28b241b640 1 -- 172.21.15.26:0/3115263823 learned_addr learned my addr 172.21.15.26:0/3115263823 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:32.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.799+0000 7f28b241b640 1 -- 172.21.15.26:0/3115263823 >> v1:172.21.15.26:6790/0 conn(0x7f28ac13f2e0 legacy=0x7f28ac1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.799+0000 7f28b1419640 1 -- 172.21.15.26:0/3115263823 >> v1:172.21.15.104:6790/0 conn(0x7f28ac0a57a0 legacy=0x7f28ac0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.799+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2377769062 0 0) 0x7f28ac136910 con 0x7f28ac13f2e0 2024-08-19T12:38:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.799+0000 7f28b0c18640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.799+0000 7f28b0c18640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.799+0000 7f28b0c18640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.799+0000 7f28b0c18640 10 monclient(hunting): my global_id is 4241 2024-08-19T12:38:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.799+0000 7f28b1c1a640 1 -- 172.21.15.26:0/3115263823 >> v1:172.21.15.104:6789/0 conn(0x7f28ac073470 legacy=0x7f28ac136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2898003520 con 0x7f28ac13f2e0 2024-08-19T12:38:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3499443156 0 0) 0x7f28ac09dd50 con 0x7f28ac0a57a0 2024-08-19T12:38:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient(hunting): my global_id is 4278 2024-08-19T12:38:32.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28ac136910 con 0x7f28ac0a57a0 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 939306549 0 0) 0x7f28ac09dae0 con 0x7f28ac073470 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient(hunting): my global_id is 4240 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28ac09dd50 con 0x7f28ac073470 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1890274728 0 0) 0x7f2898003520 con 0x7f28ac13f2e0 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 >> v1:172.21.15.104:6790/0 conn(0x7f28ac0a57a0 legacy=0x7f28ac0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 >> v1:172.21.15.104:6789/0 conn(0x7f28ac073470 legacy=0x7f28ac136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient: _finish_hunting 0 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 monclient: found mon.noname-b 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:32.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ac09e030 con 0x7f28ac13f2e0 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient: _finish_auth 0 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient: _check_auth_tickets 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f28a4001e60 con 0x7f28ac13f2e0 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b3e7e640 5 monclient: authenticate success, global_id 4241 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b3e7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient: dump: 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.807 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b3e7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f28a4004c20 con 0x7f28ac13f2e0 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 1 -- 172.21.15.26:0/3115263823 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f28a4005190 con 0x7f28ac13f2e0 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b3e7e640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b3e7e640 4 set_mon_vals no callback set 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b0c18640 10 monclient: dump: 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.809 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b3e7e640 10 monclient: shutdown 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b3e7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b3e7e640 1 -- 172.21.15.26:0/3115263823 >> v1:172.21.15.26:6790/0 conn(0x7f28ac13f2e0 legacy=0x7f28ac1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b3e7e640 1 -- 172.21.15.26:0/3115263823 shutdown_connections 2024-08-19T12:38:32.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b3e7e640 1 -- 172.21.15.26:0/3115263823 >> 172.21.15.26:0/3115263823 conn(0x7f28ac0abff0 msgr2=0x7f28ac0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b3e7e640 1 -- 172.21.15.26:0/3115263823 shutdown_connections 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.800+0000 7f28b3e7e640 1 -- 172.21.15.26:0/3115263823 wait complete. 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.801+0000 7f28b3e7e640 10 monclient: build_initial_monmap 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.801+0000 7f28b3e7e640 10 monclient: monmap: 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:32.811 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.801+0000 7f28b3e7e640 1 librados: starting msgr at 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.801+0000 7f28b3e7e640 1 librados: starting objecter 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.801+0000 7f28b3e7e640 1 Processor -- start 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.801+0000 7f28b3e7e640 1 -- start start 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.801+0000 7f28b3e7e640 1 librados: setting wanted keys 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.801+0000 7f28b3e7e640 1 librados: calling monclient init 2024-08-19T12:38:32.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.801+0000 7f28b3e7e640 10 monclient: init 2024-08-19T12:38:32.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.802+0000 7f28b3e7e640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:32.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.802+0000 7f28b3e7e640 10 monclient: _add_conns ranks=[5,2,0,6,7,3,8,4,1] 2024-08-19T12:38:32.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.802+0000 7f28b3e7e640 10 monclient(hunting): picked mon.noname-h con 0x7f28ac073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:32.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.802+0000 7f28b3e7e640 10 monclient(hunting): picked mon.noname-b con 0x7f28ac0a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:32.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.802+0000 7f28b3e7e640 10 monclient(hunting): picked mon.noname-a con 0x7f28ac13f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:32.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.802+0000 7f28b3e7e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28ac135360 con 0x7f28ac13f2e0 2024-08-19T12:38:32.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.802+0000 7f28b3e7e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28ac137280 con 0x7f28ac0a57a0 2024-08-19T12:38:32.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.802+0000 7f28b3e7e640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28ac13efb0 con 0x7f28ac073470 2024-08-19T12:38:32.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.802+0000 7f28b3e7e640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:32.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.802+0000 7f28b3e7e640 10 monclient(hunting): authenticate will time out at 706.889316s 2024-08-19T12:38:32.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.802+0000 7f28b1419640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f28ac0a57a0 0x7f28ac0a2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50254/0 (socket says 172.21.15.26:50254) 2024-08-19T12:38:32.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.802+0000 7f28b1419640 1 -- 172.21.15.26:0/2982398281 learned_addr learned my addr 172.21.15.26:0/2982398281 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:32.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28b1419640 1 -- 172.21.15.26:0/2982398281 >> v1:172.21.15.26:6790/0 conn(0x7f28ac0a57a0 legacy=0x7f28ac0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 client.?.objecter ms_handle_connect 0x7f28ac0a57a0 2024-08-19T12:38:32.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:32.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 client.?.objecter ms_handle_connect 0x7f28ac13f2e0 2024-08-19T12:38:32.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:32.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28b241b640 1 -- 172.21.15.26:0/2982398281 >> v1:172.21.15.26:6789/0 conn(0x7f28ac13f2e0 legacy=0x7f28ac1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28b1c1a640 1 -- 172.21.15.26:0/2982398281 >> v1:172.21.15.104:6791/0 conn(0x7f28ac073470 legacy=0x7f28ac0a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:32.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 client.?.objecter ms_handle_connect 0x7f28ac073470 2024-08-19T12:38:32.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:32.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2314094148 0 0) 0x7f28ac137280 con 0x7f28ac0a57a0 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 monclient(hunting): my global_id is 4250 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2890004eb0 con 0x7f28ac0a57a0 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4217333768 0 0) 0x7f28ac09e030 con 0x7f28ac13f2e0 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 monclient(hunting): my global_id is 4221 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28ac137280 con 0x7f28ac13f2e0 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 864131328 0 0) 0x7f28ac13efb0 con 0x7f28ac073470 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:32.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 10 monclient(hunting): my global_id is 4244 2024-08-19T12:38:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.804+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28ac09e030 con 0x7f28ac073470 2024-08-19T12:38:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1198629512 0 0) 0x7f2890004eb0 con 0x7f28ac0a57a0 2024-08-19T12:38:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f28ac13efb0 con 0x7f28ac0a57a0 2024-08-19T12:38:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f289c003610 con 0x7f28ac0a57a0 2024-08-19T12:38:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient(hunting): dump: 2024-08-19T12:38:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:32.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1369734467 0 0) 0x7f28ac137280 con 0x7f28ac13f2e0 2024-08-19T12:38:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2890004eb0 con 0x7f28ac13f2e0 2024-08-19T12:38:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1363217817 0 0) 0x7f28ac09e030 con 0x7f28ac073470 2024-08-19T12:38:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f28ac137280 con 0x7f28ac073470 2024-08-19T12:38:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f28a40020b0 con 0x7f28ac13f2e0 2024-08-19T12:38:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient(hunting): dump: 2024-08-19T12:38:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.822 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f28a8001cc0 con 0x7f28ac073470 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:32.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient(hunting): dump: 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.824 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 793120969 0 0) 0x7f28ac13efb0 con 0x7f28ac0a57a0 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 >> v1:172.21.15.104:6791/0 conn(0x7f28ac073470 legacy=0x7f28ac0a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 >> v1:172.21.15.26:6789/0 conn(0x7f28ac13f2e0 legacy=0x7f28ac1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 monclient: found mon.b 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ac136fa0 con 0x7f28ac0a57a0 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:32.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28b3e7e640 5 monclient: authenticate success, global_id 4250 2024-08-19T12:38:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28b3e7e640 10 monclient: _renew_subs 2024-08-19T12:38:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28b3e7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28b3e7e640 1 -- 172.21.15.26:0/2982398281 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f28ac13e100 con 0x7f28ac0a57a0 2024-08-19T12:38:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28b3e7e640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28b3e7e640 10 monclient: _renew_subs 2024-08-19T12:38:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28b3e7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28b3e7e640 1 -- 172.21.15.26:0/2982398281 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f28ac0a3170 con 0x7f28ac0a57a0 2024-08-19T12:38:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28b3e7e640 1 librados: init done 2024-08-19T12:38:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f289c003be0 con 0x7f28ac0a57a0 2024-08-19T12:38:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:32.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f289c004110 con 0x7f28ac0a57a0 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a37fe640 4 set_mon_vals no callback set 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.805+0000 7f28a27fc640 10 monclient: dump: 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:32.827 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.806+0000 7f28b3e7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.806+0000 7f28b3e7e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.806+0000 7f28b3e7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.806+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f289c0042b0 con 0x7f28ac0a57a0 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.806+0000 7f28b3e7e640 1 -- 172.21.15.26:0/2982398281 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2894005080 con 0x7f28ac0a57a0 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.806+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f289c04dbd0 con 0x7f28ac0a57a0 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.806+0000 7f28a27fc640 10 client.4250.objecter ms_dispatch 0x7f28ac000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:32.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.806+0000 7f28a27fc640 3 client.4250.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:32.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.806+0000 7f28a27fc640 3 client.4250.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:32.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.806+0000 7f28a27fc640 20 client.4250.objecter dump_active .. 0 homeless 2024-08-19T12:38:32.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.806+0000 7f28b1c1a640 10 monclient: get_auth_request con 0x7f2890041b90 auth_method 0 2024-08-19T12:38:32.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.809+0000 7f28a27fc640 10 client.4250.objecter ms_handle_connect 0x7f2890041b90 2024-08-19T12:38:32.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.811+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f289c017690 con 0x7f28ac0a57a0 2024-08-19T12:38:32.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.811+0000 7f28a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:32.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.811+0000 7f28a27fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:33.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.999+0000 7f28b3e7e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:33.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.999+0000 7f28b3e7e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:33.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.999+0000 7f28b3e7e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:33.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:32.999+0000 7f28b3e7e640 1 -- 172.21.15.26:0/2982398281 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2894005370 con 0x7f28ac0a57a0 2024-08-19T12:38:33.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.000+0000 7f28a27fc640 1 -- 172.21.15.26:0/2982398281 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10953 (unknown 941590788 0 2621968770) 0x7f289c01b590 con 0x7f28ac0a57a0 2024-08-19T12:38:33.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.000+0000 7f28a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:33.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.000+0000 7f28a27fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:33.001 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:38:33.001 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":19,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","created":"2024-08-19T12:38:08.215243+0000","modified":"2024-08-19T12:38:31.949225+0000","last_up_change":"2024-08-19T12:38:20.704140+0000","last_in_change":"2024-08-19T12:38:12.064517+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T12:38:21.116424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-19T12:38:27.276571+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae69b71a-73f9-47e2-b81e-92e9cdd7df0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6800","nonce":3531533167}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6801","nonce":3531533167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6803","nonce":3531533167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6802","nonce":3531533167}]},"public_addr":"172.21.15.26:6800/3531533167","cluster_addr":"172.21.15.26:6801/3531533167","heartbeat_back_addr":"172.21.15.26:6803/3531533167","heartbeat_front_addr":"172.21.15.26:6802/3531533167","state":["exists","up"]},{"osd":1,"uuid":"fc6dcabb-211c-484a-b215-2ef5fe1cbd21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6804","nonce":2594673920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6805","nonce":2594673920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6807","nonce":2594673920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6806","nonce":2594673920}]},"public_addr":"172.21.15.26:6804/2594673920","cluster_addr":"172.21.15.26:6805/2594673920","heartbeat_back_addr":"172.21.15.26:6807/2594673920","heartbeat_front_addr":"172.21.15.26:6806/2594673920","state":["exists","up"]},{"osd":2,"uuid":"88a4da36-5efe-4f64-84c3-28413eba4011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6808","nonce":1443966639}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6809","nonce":1443966639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6811","nonce":1443966639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6810","nonce":1443966639}]},"public_addr":"172.21.15.26:6808/1443966639","cluster_addr":"172.21.15.26:6809/1443966639","heartbeat_back_addr":"172.21.15.26:6811/1443966639","heartbeat_front_addr":"172.21.15.26:6810/1443966639","state":["exists","up"]},{"osd":3,"uuid":"98c08a65-4fb2-4e5f-a9bc-e8899aed1103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6809","nonce":847838248}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6810","nonce":847838248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6812","nonce":847838248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6811","nonce":847838248}]},"public_addr":"172.21.15.104:6809/847838248","cluster_addr":"172.21.15.104:6810/847838248","heartbeat_back_addr":"172.21.15.104:6812/847838248","heartbeat_front_addr":"172.21.15.104:6811/847838248","state":["exists","up"]},{"osd":4,"uuid":"a4c308b6-5e55-4328-b7b0-1db11fd51d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6805","nonce":1352976736}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6806","nonce":1352976736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6808","nonce":1352976736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6807","nonce":1352976736}]},"public_addr":"172.21.15.104:6805/1352976736","cluster_addr":"172.21.15.104:6806/1352976736","heartbeat_back_addr":"172.21.15.104:6808/1352976736","heartbeat_front_addr":"172.21.15.104:6807/1352976736","state":["exists","up"]},{"osd":5,"uuid":"99117114-07ce-4b9b-8df6-39f5fb1b610b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6801","nonce":3713183117}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6802","nonce":3713183117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6804","nonce":3713183117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6803","nonce":3713183117}]},"public_addr":"172.21.15.104:6801/3713183117","cluster_addr":"172.21.15.104:6802/3713183117","heartbeat_back_addr":"172.21.15.104:6804/3713183117","heartbeat_front_addr":"172.21.15.104:6803/3713183117","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.063585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.566627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.758447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.327856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.657329+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.299879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T12:38:33.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 10 librados: watch_flush enter 2024-08-19T12:38:33.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 10 librados: watch_flush exit 2024-08-19T12:38:33.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 20 client.4250.objecter shutdown clearing up homeless session... 2024-08-19T12:38:33.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 10 client.4250.objecter successfully canceled tick 2024-08-19T12:38:33.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 1 -- 172.21.15.26:0/2982398281 >> v1:172.21.15.104:6800/899887506 conn(0x7f2890041b90 legacy=0x7f2890044050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 10 monclient: shutdown 2024-08-19T12:38:33.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 1 -- 172.21.15.26:0/2982398281 >> v1:172.21.15.26:6790/0 conn(0x7f28ac0a57a0 legacy=0x7f28ac0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 1 -- 172.21.15.26:0/2982398281 shutdown_connections 2024-08-19T12:38:33.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 1 -- 172.21.15.26:0/2982398281 >> 172.21.15.26:0/2982398281 conn(0x7f28ac0abff0 msgr2=0x7f28ac134ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 1 -- 172.21.15.26:0/2982398281 shutdown_connections 2024-08-19T12:38:33.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 1 -- 172.21.15.26:0/2982398281 wait complete. 2024-08-19T12:38:33.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.002+0000 7f28b3e7e640 1 librados: shutdown 2024-08-19T12:38:33.013 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-08-19T12:38:33.014 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-19T12:38:33.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.112+0000 7f8329174640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.112+0000 7f8329174640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.112+0000 7f8329174640 10 monclient: monmap: 2024-08-19T12:38:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.112872+0000 2024-08-19T12:38:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.112872+0000 2024-08-19T12:38:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.113 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.114 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.114 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.114 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.114 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.114 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.114 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.112+0000 7f8329174640 1 Processor -- start 2024-08-19T12:38:33.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.112+0000 7f8329174640 1 -- start start 2024-08-19T12:38:33.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.112+0000 7f8329174640 10 monclient: init 2024-08-19T12:38:33.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f8329174640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f8329174640 10 monclient: _add_conns ranks=[3,6,0,4,7,2,1,5,8] 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f8329174640 10 monclient(hunting): picked mon.noname-g con 0x7f832406e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f8329174640 10 monclient(hunting): picked mon.noname-d con 0x7f83240a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f8329174640 10 monclient(hunting): picked mon.noname-a con 0x7f83240a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f8329174640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8324136790 con 0x7f83240a5d50 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f8329174640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f832409ddf0 con 0x7f832406e7b0 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f8329174640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f832409e0d0 con 0x7f83240a5920 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f8329174640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f8329174640 10 monclient(hunting): authenticate will time out at 707.200189s 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f832259c640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f83240a5920 0x7f832413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58456/0 (socket says 172.21.15.26:58456) 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f832359e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f83240a5d50 0x7f8324141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58354/0 (socket says 172.21.15.26:58354) 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.113+0000 7f832259c640 1 -- 172.21.15.26:0/3544891578 learned_addr learned my addr 172.21.15.26:0/3544891578 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f832359e640 1 -- 172.21.15.26:0/3544891578 >> v1:172.21.15.26:6789/0 conn(0x7f83240a5d50 legacy=0x7f8324141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f832259c640 1 -- 172.21.15.26:0/3544891578 >> v1:172.21.15.26:6792/0 conn(0x7f83240a5920 legacy=0x7f832413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 1 -- 172.21.15.26:0/3544891578 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1043188786 0 0) 0x7f8324136790 con 0x7f83240a5d50 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient(hunting): my global_id is 4230 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 1 -- 172.21.15.26:0/3544891578 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8310003520 con 0x7f83240a5d50 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 1 -- 172.21.15.26:0/3544891578 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1187310985 0 0) 0x7f832409e0d0 con 0x7f83240a5920 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient(hunting): my global_id is 4272 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 1 -- 172.21.15.26:0/3544891578 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8324136790 con 0x7f83240a5920 2024-08-19T12:38:33.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8322d9d640 1 -- 172.21.15.26:0/3544891578 >> v1:172.21.15.104:6790/0 conn(0x7f832406e7b0 legacy=0x7f8324136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 1 -- 172.21.15.26:0/3544891578 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 290187077 0 0) 0x7f8324136790 con 0x7f83240a5920 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 1 -- 172.21.15.26:0/3544891578 >> v1:172.21.15.104:6790/0 conn(0x7f832406e7b0 legacy=0x7f8324136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 1 -- 172.21.15.26:0/3544891578 >> v1:172.21.15.26:6789/0 conn(0x7f83240a5d50 legacy=0x7f8324141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 1 monclient: found mon.noname-d 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 1 -- 172.21.15.26:0/3544891578 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832409e3b0 con 0x7f83240a5920 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 1 -- 172.21.15.26:0/3544891578 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f830c001b80 con 0x7f83240a5920 2024-08-19T12:38:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8329174640 5 monclient: authenticate success, global_id 4272 2024-08-19T12:38:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8329174640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8321d9b640 10 monclient: dump: 2024-08-19T12:38:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.122 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.115+0000 7f8329174640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f832259c640 1 --1- 172.21.15.26:0/3544891578 >> v1:172.21.15.26:6792/0 conn(0x7f83240a5920 0x7f832413e2f0 :-1 s=OPENED pgs=56 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f830c0048a0 config(0 keys) 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8321d9b640 1 -- 172.21.15.26:0/3544891578 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f830c0048a0 con 0x7f83240a5920 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8321d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8329174640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8329174640 4 set_mon_vals no callback set 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8329174640 10 monclient: shutdown 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8329174640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8329174640 1 -- 172.21.15.26:0/3544891578 >> v1:172.21.15.26:6792/0 conn(0x7f83240a5920 legacy=0x7f832413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8321d9b640 1 -- 172.21.15.26:0/3544891578 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f830c004e10 con 0x7f83240a5920 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8321d9b640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8329174640 1 -- 172.21.15.26:0/3544891578 shutdown_connections 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8329174640 1 -- 172.21.15.26:0/3544891578 >> 172.21.15.26:0/3544891578 conn(0x7f83240a8740 msgr2=0x7f83240a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8329174640 1 -- 172.21.15.26:0/3544891578 shutdown_connections 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.116+0000 7f8329174640 1 -- 172.21.15.26:0/3544891578 wait complete. 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.117+0000 7f8329174640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.117+0000 7f8329174640 10 monclient: monmap: 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.124 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.125 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.117+0000 7f8329174640 1 librados: starting msgr at 2024-08-19T12:38:33.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.117+0000 7f8329174640 1 librados: starting objecter 2024-08-19T12:38:33.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.117+0000 7f8329174640 1 Processor -- start 2024-08-19T12:38:33.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.117+0000 7f8329174640 1 -- start start 2024-08-19T12:38:33.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.117+0000 7f8329174640 1 librados: setting wanted keys 2024-08-19T12:38:33.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.117+0000 7f8329174640 1 librados: calling monclient init 2024-08-19T12:38:33.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.117+0000 7f8329174640 10 monclient: init 2024-08-19T12:38:33.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.118+0000 7f8329174640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.118+0000 7f8329174640 10 monclient: _add_conns ranks=[6,4,8,5,0,7,3,1,2] 2024-08-19T12:38:33.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.118+0000 7f8329174640 10 monclient(hunting): picked mon.noname-d con 0x7f832406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.118+0000 7f8329174640 10 monclient(hunting): picked mon.noname-c con 0x7f83240a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.118+0000 7f8329174640 10 monclient(hunting): picked mon.noname-e con 0x7f83240a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.118+0000 7f8329174640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f832413a880 con 0x7f83240a5920 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.118+0000 7f8329174640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8324142410 con 0x7f832406e7b0 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.118+0000 7f8329174640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f832413b3e0 con 0x7f83240a5d50 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.118+0000 7f8329174640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.118+0000 7f8329174640 10 monclient(hunting): authenticate will time out at 707.205463s 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.118+0000 7f8322d9d640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f832406e7b0 0x7f83241ce660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58464/0 (socket says 172.21.15.26:58464) 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.118+0000 7f8322d9d640 1 -- 172.21.15.26:0/1972954383 learned_addr learned my addr 172.21.15.26:0/1972954383 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f8322d9d640 1 -- 172.21.15.26:0/1972954383 >> v1:172.21.15.26:6792/0 conn(0x7f832406e7b0 legacy=0x7f83241ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 client.?.objecter ms_handle_connect 0x7f832406e7b0 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 client.?.objecter ms_handle_connect 0x7f83240a5920 2024-08-19T12:38:33.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f832259c640 1 -- 172.21.15.26:0/1972954383 >> v1:172.21.15.26:6791/0 conn(0x7f83240a5920 legacy=0x7f83241d35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f832359e640 1 -- 172.21.15.26:0/1972954383 >> v1:172.21.15.26:6793/0 conn(0x7f83240a5d50 legacy=0x7f83241d6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 client.?.objecter ms_handle_connect 0x7f83240a5d50 2024-08-19T12:38:33.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1561033035 0 0) 0x7f83140018e0 con 0x7f832406e7b0 2024-08-19T12:38:33.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 monclient(hunting): my global_id is 4281 2024-08-19T12:38:33.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8304004eb0 con 0x7f832406e7b0 2024-08-19T12:38:33.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1358826872 0 0) 0x7f832413a880 con 0x7f83240a5920 2024-08-19T12:38:33.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 monclient(hunting): my global_id is 4279 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83140018e0 con 0x7f83240a5920 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1224175699 0 0) 0x7f832413b3e0 con 0x7f83240a5d50 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 10 monclient(hunting): my global_id is 4292 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.120+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f832413a880 con 0x7f83240a5d50 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3323083753 0 0) 0x7f8304004eb0 con 0x7f832406e7b0 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f832413b3e0 con 0x7f832406e7b0 2024-08-19T12:38:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8314003cd0 con 0x7f832406e7b0 2024-08-19T12:38:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient(hunting): dump: 2024-08-19T12:38:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 35950628 0 0) 0x7f83140018e0 con 0x7f83240a5920 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8304004eb0 con 0x7f83240a5920 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 803275496 0 0) 0x7f832413a880 con 0x7f83240a5d50 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83140018e0 con 0x7f83240a5d50 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f830c001b40 con 0x7f83240a5920 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient(hunting): dump: 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.136 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f831c001320 con 0x7f83240a5d50 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient(hunting): dump: 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.137 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3181072707 0 0) 0x7f832413b3e0 con 0x7f832406e7b0 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 >> v1:172.21.15.26:6793/0 conn(0x7f83240a5d50 legacy=0x7f83241d6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 >> v1:172.21.15.26:6791/0 conn(0x7f83240a5920 legacy=0x7f83241d35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 monclient: found mon.d 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8324142e30 con 0x7f832406e7b0 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f8329174640 5 monclient: authenticate success, global_id 4281 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f8329174640 10 monclient: _renew_subs 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f8329174640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f8329174640 1 -- 172.21.15.26:0/1972954383 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8324134670 con 0x7f832406e7b0 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f8329174640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f8329174640 10 monclient: _renew_subs 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f8329174640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f8329174640 1 -- 172.21.15.26:0/1972954383 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f832413b100 con 0x7f832406e7b0 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f8329174640 1 librados: init done 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8314002080 con 0x7f832406e7b0 2024-08-19T12:38:33.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8314004590 con 0x7f832406e7b0 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f832159a640 4 set_mon_vals no callback set 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.121+0000 7f831b7fe640 10 monclient: dump: 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.140 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.122+0000 7f8329174640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.122+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f8314011e00 con 0x7f832406e7b0 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.122+0000 7f8329174640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.122+0000 7f8329174640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.122+0000 7f8329174640 1 -- 172.21.15.26:0/1972954383 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8308005080 con 0x7f832406e7b0 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.122+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f831404e0e0 con 0x7f832406e7b0 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.122+0000 7f831b7fe640 10 client.4281.objecter ms_dispatch 0x7f8324000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:33.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.122+0000 7f831b7fe640 3 client.4281.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:33.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.122+0000 7f831b7fe640 3 client.4281.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:33.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.122+0000 7f831b7fe640 20 client.4281.objecter dump_active .. 0 homeless 2024-08-19T12:38:33.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.123+0000 7f832259c640 10 monclient: get_auth_request con 0x7f8304041c60 auth_method 0 2024-08-19T12:38:33.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.125+0000 7f831b7fe640 10 client.4281.objecter ms_handle_connect 0x7f8304041c60 2024-08-19T12:38:33.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.127+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f8324142410 con 0x7f832406e7b0 2024-08-19T12:38:33.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.127+0000 7f831b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.127+0000 7f831b7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:33.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.312+0000 7f8329174640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:33.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.312+0000 7f8329174640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:33.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.312+0000 7f8329174640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.312+0000 7f8329174640 1 -- 172.21.15.26:0/1972954383 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8308005370 con 0x7f832406e7b0 2024-08-19T12:38:33.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.312+0000 7f831b7fe640 1 -- 172.21.15.26:0/1972954383 <== mon.6 v1:172.21.15.26:6792/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10953 (unknown 941590788 0 2621968770) 0x7f832409ddf0 con 0x7f832406e7b0 2024-08-19T12:38:33.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.312+0000 7f831b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:33.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.312+0000 7f831b7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:33.314 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:38:33.314 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":19,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","created":"2024-08-19T12:38:08.215243+0000","modified":"2024-08-19T12:38:31.949225+0000","last_up_change":"2024-08-19T12:38:20.704140+0000","last_in_change":"2024-08-19T12:38:12.064517+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T12:38:21.116424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-19T12:38:27.276571+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae69b71a-73f9-47e2-b81e-92e9cdd7df0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6800","nonce":3531533167}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6801","nonce":3531533167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6803","nonce":3531533167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6802","nonce":3531533167}]},"public_addr":"172.21.15.26:6800/3531533167","cluster_addr":"172.21.15.26:6801/3531533167","heartbeat_back_addr":"172.21.15.26:6803/3531533167","heartbeat_front_addr":"172.21.15.26:6802/3531533167","state":["exists","up"]},{"osd":1,"uuid":"fc6dcabb-211c-484a-b215-2ef5fe1cbd21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6804","nonce":2594673920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6805","nonce":2594673920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6807","nonce":2594673920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6806","nonce":2594673920}]},"public_addr":"172.21.15.26:6804/2594673920","cluster_addr":"172.21.15.26:6805/2594673920","heartbeat_back_addr":"172.21.15.26:6807/2594673920","heartbeat_front_addr":"172.21.15.26:6806/2594673920","state":["exists","up"]},{"osd":2,"uuid":"88a4da36-5efe-4f64-84c3-28413eba4011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6808","nonce":1443966639}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6809","nonce":1443966639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6811","nonce":1443966639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6810","nonce":1443966639}]},"public_addr":"172.21.15.26:6808/1443966639","cluster_addr":"172.21.15.26:6809/1443966639","heartbeat_back_addr":"172.21.15.26:6811/1443966639","heartbeat_front_addr":"172.21.15.26:6810/1443966639","state":["exists","up"]},{"osd":3,"uuid":"98c08a65-4fb2-4e5f-a9bc-e8899aed1103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6809","nonce":847838248}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6810","nonce":847838248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6812","nonce":847838248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6811","nonce":847838248}]},"public_addr":"172.21.15.104:6809/847838248","cluster_addr":"172.21.15.104:6810/847838248","heartbeat_back_addr":"172.21.15.104:6812/847838248","heartbeat_front_addr":"172.21.15.104:6811/847838248","state":["exists","up"]},{"osd":4,"uuid":"a4c308b6-5e55-4328-b7b0-1db11fd51d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6805","nonce":1352976736}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6806","nonce":1352976736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6808","nonce":1352976736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6807","nonce":1352976736}]},"public_addr":"172.21.15.104:6805/1352976736","cluster_addr":"172.21.15.104:6806/1352976736","heartbeat_back_addr":"172.21.15.104:6808/1352976736","heartbeat_front_addr":"172.21.15.104:6807/1352976736","state":["exists","up"]},{"osd":5,"uuid":"99117114-07ce-4b9b-8df6-39f5fb1b610b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6801","nonce":3713183117}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6802","nonce":3713183117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6804","nonce":3713183117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6803","nonce":3713183117}]},"public_addr":"172.21.15.104:6801/3713183117","cluster_addr":"172.21.15.104:6802/3713183117","heartbeat_back_addr":"172.21.15.104:6804/3713183117","heartbeat_front_addr":"172.21.15.104:6803/3713183117","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.063585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.566627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.758447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.327856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.657329+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.299879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T12:38:33.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.314+0000 7f8329174640 10 librados: watch_flush enter 2024-08-19T12:38:33.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.314+0000 7f8329174640 10 librados: watch_flush exit 2024-08-19T12:38:33.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.314+0000 7f8329174640 20 client.4281.objecter shutdown clearing up homeless session... 2024-08-19T12:38:33.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.314+0000 7f8329174640 10 client.4281.objecter successfully canceled tick 2024-08-19T12:38:33.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.315+0000 7f8329174640 1 -- 172.21.15.26:0/1972954383 >> v1:172.21.15.104:6800/899887506 conn(0x7f8304041c60 legacy=0x7f8304044120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.315+0000 7f8329174640 10 monclient: shutdown 2024-08-19T12:38:33.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.315+0000 7f8329174640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.315+0000 7f8329174640 1 -- 172.21.15.26:0/1972954383 >> v1:172.21.15.26:6792/0 conn(0x7f832406e7b0 legacy=0x7f83241ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.315+0000 7f8329174640 1 -- 172.21.15.26:0/1972954383 shutdown_connections 2024-08-19T12:38:33.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.315+0000 7f8329174640 1 -- 172.21.15.26:0/1972954383 >> 172.21.15.26:0/1972954383 conn(0x7f83240a8740 msgr2=0x7f83240aa4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.315+0000 7f8329174640 1 -- 172.21.15.26:0/1972954383 shutdown_connections 2024-08-19T12:38:33.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.315+0000 7f8329174640 1 -- 172.21.15.26:0/1972954383 wait complete. 2024-08-19T12:38:33.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.315+0000 7f8329174640 1 librados: shutdown 2024-08-19T12:38:33.327 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-08-19T12:38:33.327 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-08-19T12:38:33.328 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-08-19T12:38:33.328 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-08-19T12:38:33.328 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-08-19T12:38:33.329 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-08-19T12:38:33.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.441+0000 7fcf6c63f640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.441+0000 7fcf6c63f640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.441+0000 7fcf6c63f640 10 monclient: monmap: 2024-08-19T12:38:33.442 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.442 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.442 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.442012+0000 2024-08-19T12:38:33.442 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.442012+0000 2024-08-19T12:38:33.442 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.442 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.442 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.443 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.443 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.443 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.443 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.443 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.443 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.443 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.443 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.443 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.441+0000 7fa93959f640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.441+0000 7fa93959f640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.441+0000 7fa93959f640 10 monclient: monmap: 2024-08-19T12:38:33.444 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.444 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.444 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.442202+0000 2024-08-19T12:38:33.444 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.442202+0000 2024-08-19T12:38:33.444 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.444 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.444 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.444 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.444 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.445 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.445 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.445 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.445 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.445 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.445 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.445 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.441+0000 7fcf6c63f640 1 Processor -- start 2024-08-19T12:38:33.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.441+0000 7fcf6c63f640 1 -- start start 2024-08-19T12:38:33.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.441+0000 7fcf6c63f640 10 monclient: init 2024-08-19T12:38:33.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 1 Processor -- start 2024-08-19T12:38:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 1 -- start start 2024-08-19T12:38:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 10 monclient: init 2024-08-19T12:38:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 10 monclient: _add_conns ranks=[8,4,6,0,1,5,3,2,7] 2024-08-19T12:38:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 10 monclient(hunting): picked mon.noname-e con 0x7fa9340a78f0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 10 monclient(hunting): picked mon.noname-c con 0x7fa9340a83e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 10 monclient(hunting): picked mon.noname-d con 0x7fa93413c8f0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa93406fe10 con 0x7fa9340a83e0 2024-08-19T12:38:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa934071bd0 con 0x7fa93413c8f0 2024-08-19T12:38:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa934070010 con 0x7fa9340a78f0 2024-08-19T12:38:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa93959f640 10 monclient(hunting): authenticate will time out at 707.529650s 2024-08-19T12:38:33.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fcf6c63f640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa932ffd640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fa9340a78f0 0x7fa9340a7cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:33918/0 (socket says 172.21.15.26:33918) 2024-08-19T12:38:33.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fa932ffd640 1 -- 172.21.15.26:0/2159257846 learned_addr learned my addr 172.21.15.26:0/2159257846 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fcf6c63f640 10 monclient: _add_conns ranks=[3,1,5,8,4,2,0,6,7] 2024-08-19T12:38:33.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fcf6c63f640 10 monclient(hunting): picked mon.noname-g con 0x7fcf64073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:33.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fcf6c63f640 10 monclient(hunting): picked mon.noname-f con 0x7fcf640a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:33.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fcf6c63f640 10 monclient(hunting): picked mon.noname-h con 0x7fcf6413f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:33.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fcf6c63f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf6409dae0 con 0x7fcf640a57a0 2024-08-19T12:38:33.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fcf6c63f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf64136910 con 0x7fcf64073470 2024-08-19T12:38:33.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fcf6c63f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf6409dd50 con 0x7fcf6413f2e0 2024-08-19T12:38:33.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fcf6c63f640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.442+0000 7fcf6c63f640 10 monclient(hunting): authenticate will time out at 707.529996s 2024-08-19T12:38:33.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.443+0000 7fcf6a3db640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fcf64073470 0x7fcf64136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:39930/0 (socket says 172.21.15.26:39930) 2024-08-19T12:38:33.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.443+0000 7fcf6a3db640 1 -- 172.21.15.26:0/903862639 learned_addr learned my addr 172.21.15.26:0/903862639 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.444+0000 7fa932ffd640 1 -- 172.21.15.26:0/2159257846 >> v1:172.21.15.26:6793/0 conn(0x7fa9340a78f0 legacy=0x7fa9340a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa9337fe640 1 -- 172.21.15.26:0/2159257846 >> v1:172.21.15.26:6792/0 conn(0x7fa93413c8f0 legacy=0x7fa93413ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3943010549 0 0) 0x7fa934070010 con 0x7fa9340a78f0 2024-08-19T12:38:33.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient(hunting): my global_id is 4301 2024-08-19T12:38:33.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa920003520 con 0x7fa9340a78f0 2024-08-19T12:38:33.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa9327fc640 1 -- 172.21.15.26:0/2159257846 >> v1:172.21.15.26:6791/0 conn(0x7fa9340a83e0 legacy=0x7fa93413b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf6a3db640 1 -- 172.21.15.26:0/903862639 >> v1:172.21.15.104:6790/0 conn(0x7fcf64073470 legacy=0x7fcf64136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf6abdc640 1 -- 172.21.15.26:0/903862639 >> v1:172.21.15.104:6791/0 conn(0x7fcf6413f2e0 legacy=0x7fcf641416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf69bda640 1 -- 172.21.15.26:0/903862639 >> v1:172.21.15.104:6789/0 conn(0x7fcf640a57a0 legacy=0x7fcf640a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306533941 0 0) 0x7fa934071bd0 con 0x7fa93413c8f0 2024-08-19T12:38:33.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient(hunting): my global_id is 4290 2024-08-19T12:38:33.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa934070010 con 0x7fa93413c8f0 2024-08-19T12:38:33.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 1 -- 172.21.15.26:0/903862639 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016204118 0 0) 0x7fcf64136910 con 0x7fcf64073470 2024-08-19T12:38:33.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 10 monclient(hunting): my global_id is 4296 2024-08-19T12:38:33.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3142312840 0 0) 0x7fa93406fe10 con 0x7fa9340a83e0 2024-08-19T12:38:33.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient(hunting): my global_id is 4288 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa934071bd0 con 0x7fa9340a83e0 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2109479332 0 0) 0x7fa934070010 con 0x7fa93413c8f0 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 >> v1:172.21.15.26:6793/0 conn(0x7fa9340a78f0 legacy=0x7fa9340a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 >> v1:172.21.15.26:6791/0 conn(0x7fa9340a83e0 legacy=0x7fa93413b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 1 monclient: found mon.noname-d 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9340a5ec0 con 0x7fa93413c8f0 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa92c001b10 con 0x7fa93413c8f0 2024-08-19T12:38:33.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:33.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa931ffb640 10 monclient: dump: 2024-08-19T12:38:33.456 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.456 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.456 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.456 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.456 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.457 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.457 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.457 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.457 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.457 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.457 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.457 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.457 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.457 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.457 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.457 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa93959f640 5 monclient: authenticate success, global_id 4290 2024-08-19T12:38:33.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fa93959f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 1 -- 172.21.15.26:0/903862639 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf50003520 con 0x7fcf64073470 2024-08-19T12:38:33.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 1 -- 172.21.15.26:0/903862639 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931586832 0 0) 0x7fcf6409dae0 con 0x7fcf640a57a0 2024-08-19T12:38:33.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 10 monclient(hunting): my global_id is 4249 2024-08-19T12:38:33.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 1 -- 172.21.15.26:0/903862639 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf64136910 con 0x7fcf640a57a0 2024-08-19T12:38:33.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.445+0000 7fcf693d9640 1 -- 172.21.15.26:0/903862639 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2322989588 0 0) 0x7fcf50003520 con 0x7fcf64073470 2024-08-19T12:38:33.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa92c003610 con 0x7fa93413c8f0 2024-08-19T12:38:33.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fa931ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fa931ffb640 1 -- 172.21.15.26:0/2159257846 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa92c003b80 con 0x7fa93413c8f0 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fa931ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fa931ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fa931ffb640 10 monclient: dump: 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.461 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fa93959f640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fa93959f640 4 set_mon_vals no callback set 2024-08-19T12:38:33.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 1 -- 172.21.15.26:0/903862639 >> v1:172.21.15.104:6791/0 conn(0x7fcf6413f2e0 legacy=0x7fcf641416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 1 -- 172.21.15.26:0/903862639 >> v1:172.21.15.104:6789/0 conn(0x7fcf640a57a0 legacy=0x7fcf640a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 1 monclient: found mon.noname-g 2024-08-19T12:38:33.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:38:33.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 1 -- 172.21.15.26:0/903862639 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf6409e030 con 0x7fcf64073470 2024-08-19T12:38:33.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf6c63f640 5 monclient: authenticate success, global_id 4296 2024-08-19T12:38:33.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf6c63f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 1 -- 172.21.15.26:0/903862639 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcf60001ae0 con 0x7fcf64073470 2024-08-19T12:38:33.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:38:33.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 10 monclient: dump: 2024-08-19T12:38:33.465 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.465 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.465 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.465 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.465 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.465 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.466 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.466 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.466 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.466 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.466 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.466 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.466 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.466 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.466 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.466 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf6c63f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 1 -- 172.21.15.26:0/903862639 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf600048a0 con 0x7fcf64073470 2024-08-19T12:38:33.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf693d9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fa93959f640 10 monclient: shutdown 2024-08-19T12:38:33.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fa93959f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fa93959f640 1 -- 172.21.15.26:0/2159257846 >> v1:172.21.15.26:6792/0 conn(0x7fa93413c8f0 legacy=0x7fa93413ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fa93959f640 1 -- 172.21.15.26:0/2159257846 shutdown_connections 2024-08-19T12:38:33.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fa93959f640 1 -- 172.21.15.26:0/2159257846 >> 172.21.15.26:0/2159257846 conn(0x7fa9340a9010 msgr2=0x7fa9340a54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf6c63f640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.446+0000 7fcf6c63f640 4 set_mon_vals no callback set 2024-08-19T12:38:33.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fcf693d9640 1 -- 172.21.15.26:0/903862639 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcf60004d90 con 0x7fcf64073470 2024-08-19T12:38:33.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fcf693d9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fcf693d9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fcf693d9640 10 monclient: dump: 2024-08-19T12:38:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.469 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.470 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fa93959f640 1 -- 172.21.15.26:0/2159257846 shutdown_connections 2024-08-19T12:38:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fa93959f640 1 -- 172.21.15.26:0/2159257846 wait complete. 2024-08-19T12:38:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fcf6c63f640 10 monclient: shutdown 2024-08-19T12:38:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fcf6c63f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fcf6c63f640 1 -- 172.21.15.26:0/903862639 >> v1:172.21.15.104:6790/0 conn(0x7fcf64073470 legacy=0x7fcf64136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fcf6c63f640 1 -- 172.21.15.26:0/903862639 shutdown_connections 2024-08-19T12:38:33.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fcf6c63f640 1 -- 172.21.15.26:0/903862639 >> 172.21.15.26:0/903862639 conn(0x7fcf640abff0 msgr2=0x7fcf640ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.447+0000 7fcf6c63f640 1 -- 172.21.15.26:0/903862639 shutdown_connections 2024-08-19T12:38:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fcf6c63f640 1 -- 172.21.15.26:0/903862639 wait complete. 2024-08-19T12:38:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fa93959f640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fa93959f640 10 monclient: monmap: 2024-08-19T12:38:33.472 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.473 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.474 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fa93959f640 1 librados: starting msgr at 2024-08-19T12:38:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fa93959f640 1 librados: starting objecter 2024-08-19T12:38:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fa93959f640 1 Processor -- start 2024-08-19T12:38:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fcf6c63f640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fcf6c63f640 10 monclient: monmap: 2024-08-19T12:38:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:33.474 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.475 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.476 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fa93959f640 1 -- start start 2024-08-19T12:38:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fa93959f640 1 librados: setting wanted keys 2024-08-19T12:38:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fa93959f640 1 librados: calling monclient init 2024-08-19T12:38:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fa93959f640 10 monclient: init 2024-08-19T12:38:33.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fcf6c63f640 1 librados: starting msgr at 2024-08-19T12:38:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.448+0000 7fcf6c63f640 1 librados: starting objecter 2024-08-19T12:38:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 1 Processor -- start 2024-08-19T12:38:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 1 -- start start 2024-08-19T12:38:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 1 librados: setting wanted keys 2024-08-19T12:38:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 1 librados: calling monclient init 2024-08-19T12:38:33.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 10 monclient: init 2024-08-19T12:38:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fa93959f640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fa93959f640 10 monclient: _add_conns ranks=[5,0,6,2,7,1,8,3,4] 2024-08-19T12:38:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fa93959f640 10 monclient(hunting): picked mon.noname-h con 0x7fa9340a83e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fa93959f640 10 monclient(hunting): picked mon.noname-a con 0x7fa93413c8f0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fa93959f640 10 monclient(hunting): picked mon.noname-d con 0x7fa9341cdc80 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fa93959f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9341cebd0 con 0x7fa93413c8f0 2024-08-19T12:38:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fa93959f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa934126b00 con 0x7fa9340a83e0 2024-08-19T12:38:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fa93959f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa934126de0 con 0x7fa9341cdc80 2024-08-19T12:38:33.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fa93959f640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fa93959f640 10 monclient(hunting): authenticate will time out at 707.536441s 2024-08-19T12:38:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fa9337fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fa9341cdc80 0x7fa9341d6730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58486/0 (socket says 172.21.15.26:58486) 2024-08-19T12:38:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fa9337fe640 1 -- 172.21.15.26:0/4022087026 learned_addr learned my addr 172.21.15.26:0/4022087026 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 10 monclient: _add_conns ranks=[8,2,4,0,6,3,7,5,1] 2024-08-19T12:38:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 10 monclient(hunting): picked mon.noname-e con 0x7fcf64073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 10 monclient(hunting): picked mon.noname-b con 0x7fcf640a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 10 monclient(hunting): picked mon.noname-c con 0x7fcf6413f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf640a5210 con 0x7fcf640a57a0 2024-08-19T12:38:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf640a0760 con 0x7fcf6413f2e0 2024-08-19T12:38:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf6409f410 con 0x7fcf64073470 2024-08-19T12:38:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.449+0000 7fcf6c63f640 10 monclient(hunting): authenticate will time out at 707.536872s 2024-08-19T12:38:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.450+0000 7fcf6abdc640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fcf6413f2e0 0x7fcf641da1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57818/0 (socket says 172.21.15.26:57818) 2024-08-19T12:38:33.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.450+0000 7fcf6abdc640 1 -- 172.21.15.26:0/2636182587 learned_addr learned my addr 172.21.15.26:0/2636182587 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.451+0000 7fa9337fe640 1 -- 172.21.15.26:0/4022087026 >> v1:172.21.15.26:6792/0 conn(0x7fa9341cdc80 legacy=0x7fa9341d6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.451+0000 7fa92b7fe640 10 client.?.objecter ms_handle_connect 0x7fa9341cdc80 2024-08-19T12:38:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.451+0000 7fa92b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.451+0000 7fa9327fc640 1 -- 172.21.15.26:0/4022087026 >> v1:172.21.15.26:6789/0 conn(0x7fa93413c8f0 legacy=0x7fa9341d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.451+0000 7fa92b7fe640 10 client.?.objecter ms_handle_connect 0x7fa93413c8f0 2024-08-19T12:38:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.451+0000 7fa932ffd640 1 -- 172.21.15.26:0/4022087026 >> v1:172.21.15.104:6791/0 conn(0x7fa9340a83e0 legacy=0x7fa934138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.451+0000 7fa92b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.451+0000 7fa92b7fe640 10 client.?.objecter ms_handle_connect 0x7fa9340a83e0 2024-08-19T12:38:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.451+0000 7fa92b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4186521183 0 0) 0x7fa934126de0 con 0x7fa9341cdc80 2024-08-19T12:38:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): my global_id is 4299 2024-08-19T12:38:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa914004eb0 con 0x7fa9341cdc80 2024-08-19T12:38:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651710655 0 0) 0x7fa934126b00 con 0x7fa9340a83e0 2024-08-19T12:38:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf6abdc640 1 -- 172.21.15.26:0/2636182587 >> v1:172.21.15.26:6791/0 conn(0x7fcf6413f2e0 legacy=0x7fcf641da1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 client.?.objecter ms_handle_connect 0x7fcf6413f2e0 2024-08-19T12:38:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 client.?.objecter ms_handle_connect 0x7fcf640a57a0 2024-08-19T12:38:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf69bda640 1 -- 172.21.15.26:0/2636182587 >> v1:172.21.15.26:6790/0 conn(0x7fcf640a57a0 legacy=0x7fcf6409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): my global_id is 4262 2024-08-19T12:38:33.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa934126de0 con 0x7fa9340a83e0 2024-08-19T12:38:33.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3953041971 0 0) 0x7fa934071bd0 con 0x7fa93413c8f0 2024-08-19T12:38:33.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): my global_id is 4239 2024-08-19T12:38:33.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa934126b00 con 0x7fa93413c8f0 2024-08-19T12:38:33.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf6a3db640 1 -- 172.21.15.26:0/2636182587 >> v1:172.21.15.26:6793/0 conn(0x7fcf64073470 legacy=0x7fcf6409e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 client.?.objecter ms_handle_connect 0x7fcf64073470 2024-08-19T12:38:33.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 318070248 0 0) 0x7fcf6409dd50 con 0x7fcf6413f2e0 2024-08-19T12:38:33.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 monclient(hunting): my global_id is 4297 2024-08-19T12:38:33.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf48004eb0 con 0x7fcf6413f2e0 2024-08-19T12:38:33.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 984655123 0 0) 0x7fa914004eb0 con 0x7fa9341cdc80 2024-08-19T12:38:33.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa934071bd0 con 0x7fa9341cdc80 2024-08-19T12:38:33.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1256821006 0 0) 0x7fa934126de0 con 0x7fa9340a83e0 2024-08-19T12:38:33.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 798775135 0 0) 0x7fcf640a5210 con 0x7fcf640a57a0 2024-08-19T12:38:33.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 monclient(hunting): my global_id is 4259 2024-08-19T12:38:33.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf6409dd50 con 0x7fcf640a57a0 2024-08-19T12:38:33.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa914004eb0 con 0x7fa9340a83e0 2024-08-19T12:38:33.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3901042414 0 0) 0x7fa934126b00 con 0x7fa93413c8f0 2024-08-19T12:38:33.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa934126de0 con 0x7fa93413c8f0 2024-08-19T12:38:33.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa92c003b00 con 0x7fa9341cdc80 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): dump: 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.490 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.491 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.491 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.491 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.491 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.491 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.491 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:33.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2652052165 0 0) 0x7fcf6409f410 con 0x7fcf64073470 2024-08-19T12:38:33.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa9240047e0 con 0x7fa9340a83e0 2024-08-19T12:38:33.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:33.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): dump: 2024-08-19T12:38:33.492 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.492 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.493 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.494 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa91c003990 con 0x7fa93413c8f0 2024-08-19T12:38:33.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:33.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fa92b7fe640 10 monclient(hunting): dump: 2024-08-19T12:38:33.494 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.494 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.494 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.494 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.494 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.494 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.494 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2517916709 0 0) 0x7fa914004eb0 con 0x7fa9340a83e0 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 >> v1:172.21.15.26:6792/0 conn(0x7fa9341cdc80 legacy=0x7fa9341d6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 >> v1:172.21.15.26:6789/0 conn(0x7fa93413c8f0 legacy=0x7fa9341d3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 1 monclient: found mon.h 2024-08-19T12:38:33.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:33.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9340a72e0 con 0x7fa9340a83e0 2024-08-19T12:38:33.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 10 monclient(hunting): my global_id is 4310 2024-08-19T12:38:33.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.452+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf640a5210 con 0x7fcf64073470 2024-08-19T12:38:33.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4129901986 0 0) 0x7fcf640a5210 con 0x7fcf64073470 2024-08-19T12:38:33.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf6409f410 con 0x7fcf64073470 2024-08-19T12:38:33.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2220256535 0 0) 0x7fcf6409dd50 con 0x7fcf640a57a0 2024-08-19T12:38:33.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf640a5210 con 0x7fcf640a57a0 2024-08-19T12:38:33.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2945469690 0 0) 0x7fcf48004eb0 con 0x7fcf6413f2e0 2024-08-19T12:38:33.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf6409dd50 con 0x7fcf6413f2e0 2024-08-19T12:38:33.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcf60001a60 con 0x7fcf64073470 2024-08-19T12:38:33.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:33.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 10 monclient(hunting): dump: 2024-08-19T12:38:33.502 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.502 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.502 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.502 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.502 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.502 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.502 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.502 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.502 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.502 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.503 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.503 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.503 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.503 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.505 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.505 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:33.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcf54001280 con 0x7fcf640a57a0 2024-08-19T12:38:33.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:33.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 10 monclient(hunting): dump: 2024-08-19T12:38:33.506 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.506 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.506 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.506 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcf5c003990 con 0x7fcf6413f2e0 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fcf5affd640 10 monclient(hunting): dump: 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.508 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1415375054 0 0) 0x7fcf6409f410 con 0x7fcf64073470 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 >> v1:172.21.15.26:6791/0 conn(0x7fcf6413f2e0 legacy=0x7fcf641da1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 >> v1:172.21.15.26:6790/0 conn(0x7fcf640a57a0 legacy=0x7fcf6409ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 1 monclient: found mon.e 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf640a04a0 con 0x7fcf64073470 2024-08-19T12:38:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf60001320 con 0x7fcf64073470 2024-08-19T12:38:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcf60004db0 con 0x7fcf64073470 2024-08-19T12:38:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:33.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf5affd640 10 monclient: dump: 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.512 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.512 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.454+0000 7fcf68bd8640 4 set_mon_vals no callback set 2024-08-19T12:38:33.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa93959f640 5 monclient: authenticate success, global_id 4262 2024-08-19T12:38:33.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa93959f640 10 monclient: _renew_subs 2024-08-19T12:38:33.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa93959f640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:33.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa93959f640 1 -- 172.21.15.26:0/4022087026 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa934126820 con 0x7fa9340a83e0 2024-08-19T12:38:33.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fcf6c63f640 5 monclient: authenticate success, global_id 4310 2024-08-19T12:38:33.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fcf6c63f640 10 monclient: _renew_subs 2024-08-19T12:38:33.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fcf6c63f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fcf6c63f640 1 -- 172.21.15.26:0/2636182587 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcf640a1000 con 0x7fcf64073470 2024-08-19T12:38:33.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fcf6c63f640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fcf6c63f640 10 monclient: _renew_subs 2024-08-19T12:38:33.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fcf6c63f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fcf6c63f640 1 -- 172.21.15.26:0/2636182587 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fcf64071ae0 con 0x7fcf64073470 2024-08-19T12:38:33.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fcf6c63f640 1 librados: init done 2024-08-19T12:38:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa93959f640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa93959f640 10 monclient: _renew_subs 2024-08-19T12:38:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa93959f640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:33.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa93959f640 1 -- 172.21.15.26:0/4022087026 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fa934138b20 con 0x7fa9340a83e0 2024-08-19T12:38:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa93959f640 1 librados: init done 2024-08-19T12:38:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa924004c30 con 0x7fa9340a83e0 2024-08-19T12:38:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa924005120 con 0x7fa9340a83e0 2024-08-19T12:38:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa92b7fe640 10 monclient: dump: 2024-08-19T12:38:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.516 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.517 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.453+0000 7fa9317fa640 4 set_mon_vals no callback set 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa924011580 con 0x7fa9340a83e0 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7fa92404d970 con 0x7fa9340a83e0 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fa92b7fe640 10 client.4262.objecter ms_dispatch 0x7fa934000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fa92b7fe640 3 client.4262.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fa92b7fe640 3 client.4262.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.455+0000 7fa92b7fe640 20 client.4262.objecter dump_active .. 0 homeless 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa93959f640 10 client.4262.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa9327fc640 10 monclient: get_auth_request con 0x7fa914041900 auth_method 0 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa93959f640 15 client.4262.objecter _session_command_op_assign -1 1 2024-08-19T12:38:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa93959f640 20 client.4262.objecter _get_session s=0x7fa9180028d0 osd=2 3 2024-08-19T12:38:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa93959f640 20 client.4262.objecter put_session s=0x7fa9180028d0 osd=2 3 2024-08-19T12:38:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa93959f640 20 client.4262.objecter _get_session s=0x7fa9180028d0 osd=2 3 2024-08-19T12:38:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa93959f640 15 client.4262.objecter _session_command_op_remove -1 1 2024-08-19T12:38:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa93959f640 20 client.4262.objecter get_session s=0x7fa9180028d0 osd=2 3 2024-08-19T12:38:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa93959f640 15 client.4262.objecter _session_command_op_assign 2 1 2024-08-19T12:38:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa93959f640 20 client.4262.objecter put_session s=0x7fa9180028d0 osd=2 4 2024-08-19T12:38:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa93959f640 10 client.4262.objecter _send_command 1 2024-08-19T12:38:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa93959f640 1 -- 172.21.15.26:0/4022087026 --> v1:172.21.15.26:6808/1443966639 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa918006d30 con 0x7fa918002fc0 2024-08-19T12:38:33.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fa9337fe640 10 monclient: get_auth_request con 0x7fa918002fc0 auth_method 0 2024-08-19T12:38:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fcf6c63f640 10 client.4310.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fcf6c63f640 15 client.4310.objecter _session_command_op_assign -1 1 2024-08-19T12:38:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fcf6c63f640 20 client.4310.objecter _get_session osd=-1 returning homeless 2024-08-19T12:38:33.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fcf6c63f640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fcf6c63f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fcf6c63f640 1 -- 172.21.15.26:0/2636182587 --> v1:172.21.15.26:6793/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fcf64142ba0 con 0x7fcf64073470 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.456+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fcf60011900 con 0x7fcf64073470 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7fcf6004d970 con 0x7fcf64073470 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 10 client.4310.objecter ms_dispatch 0x7fcf64000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 3 client.4310.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 3 client.4310.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 10 client.4310.objecter checking command 1 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 20 client.4310.objecter _get_session s=0x7fcf48047f00 osd=4 3 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 20 client.4310.objecter put_session s=0x7fcf48047f00 osd=4 3 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 15 client.4310.objecter _session_command_op_remove -1 1 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 20 client.4310.objecter _get_session s=0x7fcf48047f00 osd=4 3 2024-08-19T12:38:33.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 20 client.4310.objecter get_session s=0x7fcf48047f00 osd=4 3 2024-08-19T12:38:33.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 15 client.4310.objecter _session_command_op_assign 4 1 2024-08-19T12:38:33.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 20 client.4310.objecter put_session s=0x7fcf48047f00 osd=4 4 2024-08-19T12:38:33.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 10 client.4310.objecter _send_command 1 2024-08-19T12:38:33.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 --> v1:172.21.15.104:6805/1352976736 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf600048a0 con 0x7fcf480485b0 2024-08-19T12:38:33.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 20 client.4310.objecter dump_active .. 0 homeless 2024-08-19T12:38:33.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7fcf6004dde0 con 0x7fcf64073470 2024-08-19T12:38:33.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf5affd640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-08-19T12:38:33.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf69bda640 10 monclient: get_auth_request con 0x7fcf480419f0 auth_method 0 2024-08-19T12:38:33.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.457+0000 7fcf6abdc640 10 monclient: get_auth_request con 0x7fcf480485b0 auth_method 0 2024-08-19T12:38:33.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.458+0000 7fa92b7fe640 10 client.4262.objecter ms_handle_connect 0x7fa914041900 2024-08-19T12:38:33.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.459+0000 7fcf5affd640 10 client.4310.objecter ms_handle_connect 0x7fcf480419f0 2024-08-19T12:38:33.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.460+0000 7fa92b7fe640 10 client.4262.objecter ms_handle_connect 0x7fa918002fc0 2024-08-19T12:38:33.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.461+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== osd.2 v1:172.21.15.26:6808/1443966639 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa918006d30 con 0x7fa918002fc0 2024-08-19T12:38:33.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.461+0000 7fa92b7fe640 10 client.4262.objecter ms_dispatch 0x7fa934000b90 command_reply(tid 1: 0 ) 2024-08-19T12:38:33.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.461+0000 7fa92b7fe640 10 client.4262.objecter _finish_command 1 = system:0 2024-08-19T12:38:33.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.461+0000 7fa92b7fe640 20 client.4262.objecter put_session s=0x7fa9180028d0 osd=2 4 2024-08-19T12:38:33.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.461+0000 7fa92b7fe640 15 client.4262.objecter _session_command_op_remove 2 1 2024-08-19T12:38:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.461+0000 7fcf5affd640 10 client.4310.objecter ms_handle_connect 0x7fcf480485b0 2024-08-19T12:38:33.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.462+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== osd.4 v1:172.21.15.104:6805/1352976736 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fcf600048a0 con 0x7fcf480485b0 2024-08-19T12:38:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.462+0000 7fcf5affd640 10 client.4310.objecter ms_dispatch 0x7fcf64000b90 command_reply(tid 1: 0 ) 2024-08-19T12:38:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.462+0000 7fcf5affd640 10 client.4310.objecter _finish_command 1 = system:0 2024-08-19T12:38:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.462+0000 7fcf5affd640 20 client.4310.objecter put_session s=0x7fcf48047f00 osd=4 4 2024-08-19T12:38:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.462+0000 7fcf5affd640 15 client.4310.objecter _session_command_op_remove 4 1 2024-08-19T12:38:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.476+0000 7fcf6c63f640 10 client.4310.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-19T12:38:33.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.476+0000 7fcf6c63f640 15 client.4310.objecter _session_command_op_assign -1 2 2024-08-19T12:38:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.476+0000 7fcf6c63f640 20 client.4310.objecter _get_session s=0x7fcf48047f00 osd=4 3 2024-08-19T12:38:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.476+0000 7fcf6c63f640 20 client.4310.objecter put_session s=0x7fcf48047f00 osd=4 3 2024-08-19T12:38:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.476+0000 7fcf6c63f640 20 client.4310.objecter _get_session s=0x7fcf48047f00 osd=4 3 2024-08-19T12:38:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.476+0000 7fcf6c63f640 15 client.4310.objecter _session_command_op_remove -1 2 2024-08-19T12:38:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.476+0000 7fcf6c63f640 20 client.4310.objecter get_session s=0x7fcf48047f00 osd=4 3 2024-08-19T12:38:33.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.476+0000 7fcf6c63f640 15 client.4310.objecter _session_command_op_assign 4 2 2024-08-19T12:38:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.476+0000 7fcf6c63f640 20 client.4310.objecter put_session s=0x7fcf48047f00 osd=4 4 2024-08-19T12:38:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.476+0000 7fcf6c63f640 10 client.4310.objecter _send_command 2 2024-08-19T12:38:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.476+0000 7fcf6c63f640 1 -- 172.21.15.26:0/2636182587 --> v1:172.21.15.104:6805/1352976736 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcf64142df0 con 0x7fcf480485b0 2024-08-19T12:38:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.477+0000 7fcf5affd640 1 -- 172.21.15.26:0/2636182587 <== osd.4 v1:172.21.15.104:6805/1352976736 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7fcf64142df0 con 0x7fcf480485b0 2024-08-19T12:38:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.477+0000 7fcf5affd640 10 client.4310.objecter ms_dispatch 0x7fcf64000b90 command_reply(tid 2: 0 ) 2024-08-19T12:38:33.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.477+0000 7fcf5affd640 10 client.4310.objecter _finish_command 2 = system:0 2024-08-19T12:38:33.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.477+0000 7fcf5affd640 20 client.4310.objecter put_session s=0x7fcf48047f00 osd=4 4 2024-08-19T12:38:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.477+0000 7fcf5affd640 15 client.4310.objecter _session_command_op_remove 4 2 2024-08-19T12:38:33.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.477+0000 7fcf6c63f640 10 librados: watch_flush enter 2024-08-19T12:38:33.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.477+0000 7fcf6c63f640 10 librados: watch_flush exit 2024-08-19T12:38:33.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.477+0000 7fcf6c63f640 10 client.4310.objecter close_session for osd.4 2024-08-19T12:38:33.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.477+0000 7fcf6c63f640 1 -- 172.21.15.26:0/2636182587 >> v1:172.21.15.104:6805/1352976736 conn(0x7fcf480485b0 legacy=0x7fcf4804a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.477+0000 7fcf6c63f640 20 client.4310.objecter put_session s=0x7fcf48047f00 osd=4 1 2024-08-19T12:38:33.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.477+0000 7fcf6c63f640 20 client.4310.objecter shutdown clearing up homeless session... 2024-08-19T12:38:33.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.477+0000 7fcf6c63f640 10 client.4310.objecter successfully canceled tick 2024-08-19T12:38:33.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.478+0000 7fcf6c63f640 1 -- 172.21.15.26:0/2636182587 >> v1:172.21.15.104:6800/899887506 conn(0x7fcf480419f0 legacy=0x7fcf48043eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.478+0000 7fcf6c63f640 10 monclient: shutdown 2024-08-19T12:38:33.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.478+0000 7fcf6c63f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.478+0000 7fcf6c63f640 1 -- 172.21.15.26:0/2636182587 >> v1:172.21.15.26:6793/0 conn(0x7fcf64073470 legacy=0x7fcf6409e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.478+0000 7fcf6abdc640 1 -- 172.21.15.26:0/2636182587 reap_dead start 2024-08-19T12:38:33.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.478+0000 7fcf6c63f640 1 -- 172.21.15.26:0/2636182587 shutdown_connections 2024-08-19T12:38:33.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.478+0000 7fcf6c63f640 1 -- 172.21.15.26:0/2636182587 >> 172.21.15.26:0/2636182587 conn(0x7fcf640abff0 msgr2=0x7fcf64132d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.478+0000 7fcf6c63f640 1 -- 172.21.15.26:0/2636182587 shutdown_connections 2024-08-19T12:38:33.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.478+0000 7fcf6c63f640 1 -- 172.21.15.26:0/2636182587 wait complete. 2024-08-19T12:38:33.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.478+0000 7fcf6c63f640 1 librados: shutdown 2024-08-19T12:38:33.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa93959f640 10 client.4262.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-19T12:38:33.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa93959f640 15 client.4262.objecter _session_command_op_assign -1 2 2024-08-19T12:38:33.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa93959f640 20 client.4262.objecter _get_session s=0x7fa9180028d0 osd=2 3 2024-08-19T12:38:33.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa93959f640 20 client.4262.objecter put_session s=0x7fa9180028d0 osd=2 3 2024-08-19T12:38:33.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa93959f640 20 client.4262.objecter _get_session s=0x7fa9180028d0 osd=2 3 2024-08-19T12:38:33.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa93959f640 15 client.4262.objecter _session_command_op_remove -1 2 2024-08-19T12:38:33.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa93959f640 20 client.4262.objecter get_session s=0x7fa9180028d0 osd=2 3 2024-08-19T12:38:33.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa93959f640 15 client.4262.objecter _session_command_op_assign 2 2 2024-08-19T12:38:33.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa93959f640 20 client.4262.objecter put_session s=0x7fa9180028d0 osd=2 4 2024-08-19T12:38:33.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa93959f640 10 client.4262.objecter _send_command 2 2024-08-19T12:38:33.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa93959f640 1 -- 172.21.15.26:0/4022087026 --> v1:172.21.15.26:6808/1443966639 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa918001210 con 0x7fa918002fc0 2024-08-19T12:38:33.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa92b7fe640 1 -- 172.21.15.26:0/4022087026 <== osd.2 v1:172.21.15.26:6808/1443966639 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7fa918001210 con 0x7fa918002fc0 2024-08-19T12:38:33.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa92b7fe640 10 client.4262.objecter ms_dispatch 0x7fa934000b90 command_reply(tid 2: 0 ) 2024-08-19T12:38:33.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa92b7fe640 10 client.4262.objecter _finish_command 2 = system:0 2024-08-19T12:38:33.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa92b7fe640 20 client.4262.objecter put_session s=0x7fa9180028d0 osd=2 4 2024-08-19T12:38:33.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.480+0000 7fa92b7fe640 15 client.4262.objecter _session_command_op_remove 2 2 2024-08-19T12:38:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.481+0000 7fa93959f640 10 librados: watch_flush enter 2024-08-19T12:38:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.481+0000 7fa93959f640 10 librados: watch_flush exit 2024-08-19T12:38:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.481+0000 7fa93959f640 10 client.4262.objecter close_session for osd.2 2024-08-19T12:38:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.481+0000 7fa93959f640 1 -- 172.21.15.26:0/4022087026 >> v1:172.21.15.26:6808/1443966639 conn(0x7fa918002fc0 legacy=0x7fa918005480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.481+0000 7fa93959f640 20 client.4262.objecter put_session s=0x7fa9180028d0 osd=2 1 2024-08-19T12:38:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.481+0000 7fa93959f640 20 client.4262.objecter shutdown clearing up homeless session... 2024-08-19T12:38:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.481+0000 7fa93959f640 10 client.4262.objecter successfully canceled tick 2024-08-19T12:38:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.481+0000 7fa93959f640 1 -- 172.21.15.26:0/4022087026 >> v1:172.21.15.104:6800/899887506 conn(0x7fa914041900 legacy=0x7fa914043dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.481+0000 7fa93959f640 10 monclient: shutdown 2024-08-19T12:38:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.481+0000 7fa93959f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.481+0000 7fa93959f640 1 -- 172.21.15.26:0/4022087026 >> v1:172.21.15.104:6791/0 conn(0x7fa9340a83e0 legacy=0x7fa934138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.481+0000 7fa9337fe640 1 -- 172.21.15.26:0/4022087026 reap_dead start 2024-08-19T12:38:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.482+0000 7fa93959f640 1 -- 172.21.15.26:0/4022087026 shutdown_connections 2024-08-19T12:38:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.482+0000 7fa93959f640 1 -- 172.21.15.26:0/4022087026 >> 172.21.15.26:0/4022087026 conn(0x7fa9340a9010 msgr2=0x7fa934072850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.482+0000 7fa93959f640 1 -- 172.21.15.26:0/4022087026 shutdown_connections 2024-08-19T12:38:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.482+0000 7fa93959f640 1 -- 172.21.15.26:0/4022087026 wait complete. 2024-08-19T12:38:33.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.482+0000 7fa93959f640 1 librados: shutdown 2024-08-19T12:38:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.482+0000 7efd58c98640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.482+0000 7efd58c98640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.482+0000 7efd58c98640 10 monclient: monmap: 2024-08-19T12:38:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.544 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.483387+0000 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.483387+0000 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.545 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.483+0000 7efd58c98640 1 Processor -- start 2024-08-19T12:38:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.483+0000 7efd58c98640 1 -- start start 2024-08-19T12:38:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.483+0000 7efd58c98640 10 monclient: init 2024-08-19T12:38:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.484+0000 7efd58c98640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.484+0000 7efd58c98640 10 monclient: _add_conns ranks=[8,4,2,7,1,5,3,6,0] 2024-08-19T12:38:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.484+0000 7efd58c98640 10 monclient(hunting): picked mon.noname-e con 0x7efd54073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.484+0000 7efd58c98640 10 monclient(hunting): picked mon.noname-c con 0x7efd540a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.484+0000 7efd58c98640 10 monclient(hunting): picked mon.noname-b con 0x7efd5413f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.484+0000 7efd58c98640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd5409dae0 con 0x7efd5413f2e0 2024-08-19T12:38:33.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.484+0000 7efd58c98640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd54136910 con 0x7efd540a57a0 2024-08-19T12:38:33.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.484+0000 7efd58c98640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd5409dd50 con 0x7efd54073470 2024-08-19T12:38:33.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.484+0000 7efd58c98640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.484+0000 7efd58c98640 10 monclient(hunting): authenticate will time out at 707.571549s 2024-08-19T12:38:33.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.484+0000 7efd51d9b640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7efd540a57a0 0x7efd540a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57822/0 (socket says 172.21.15.26:57822) 2024-08-19T12:38:33.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.484+0000 7efd51d9b640 1 -- 172.21.15.26:0/659349664 learned_addr learned my addr 172.21.15.26:0/659349664 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.485+0000 7f2def186640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.485+0000 7f2def186640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.485+0000 7f2def186640 10 monclient: monmap: 2024-08-19T12:38:33.548 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.548 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.548 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.486512+0000 2024-08-19T12:38:33.548 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.486512+0000 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.486+0000 7f2def186640 1 Processor -- start 2024-08-19T12:38:33.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.486+0000 7f2def186640 1 -- start start 2024-08-19T12:38:33.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.486+0000 7f2def186640 10 monclient: init 2024-08-19T12:38:33.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.486+0000 7efd51d9b640 1 -- 172.21.15.26:0/659349664 >> v1:172.21.15.26:6791/0 conn(0x7efd540a57a0 legacy=0x7efd540a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.486+0000 7efd5259c640 1 -- 172.21.15.26:0/659349664 >> v1:172.21.15.26:6793/0 conn(0x7efd54073470 legacy=0x7efd54136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.486+0000 7efd52d9d640 1 -- 172.21.15.26:0/659349664 >> v1:172.21.15.26:6790/0 conn(0x7efd5413f2e0 legacy=0x7efd541416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2100408324 0 0) 0x7efd54136910 con 0x7efd540a57a0 2024-08-19T12:38:33.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient(hunting): my global_id is 4306 2024-08-19T12:38:33.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd3c003520 con 0x7efd540a57a0 2024-08-19T12:38:33.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457389757 0 0) 0x7efd5409dd50 con 0x7efd54073470 2024-08-19T12:38:33.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient(hunting): my global_id is 4319 2024-08-19T12:38:33.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd54136910 con 0x7efd54073470 2024-08-19T12:38:33.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7f2def186640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7f2def186640 10 monclient: _add_conns ranks=[3,0,5,2,8,4,1,7,6] 2024-08-19T12:38:33.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7f2def186640 10 monclient(hunting): picked mon.noname-g con 0x7f2de8073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:33.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7f2f47d6b640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7f2f47d6b640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7f2f47d6b640 10 monclient: monmap: 2024-08-19T12:38:33.552 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.488749+0000 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.488749+0000 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.553 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.554 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7f2def186640 10 monclient(hunting): picked mon.noname-a con 0x7f2de80a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:33.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1867991064 0 0) 0x7efd5409dae0 con 0x7efd5413f2e0 2024-08-19T12:38:33.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient(hunting): my global_id is 4268 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd5409dd50 con 0x7efd5413f2e0 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2245875546 0 0) 0x7efd54136910 con 0x7efd54073470 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 >> v1:172.21.15.26:6791/0 conn(0x7efd540a57a0 legacy=0x7efd540a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 >> v1:172.21.15.26:6790/0 conn(0x7efd5413f2e0 legacy=0x7efd541416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 monclient: found mon.noname-e 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5409e030 con 0x7efd54073470 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd44001a80 con 0x7efd54073470 2024-08-19T12:38:33.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:33.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient: dump: 2024-08-19T12:38:33.556 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.556 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.556 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.556 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.556 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.556 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.556 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.556 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.557 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd58c98640 5 monclient: authenticate success, global_id 4319 2024-08-19T12:38:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd58c98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5259c640 1 --1- 172.21.15.26:0/659349664 >> v1:172.21.15.26:6793/0 conn(0x7efd54073470 0x7efd54136200 :-1 s=OPENED pgs=59 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7efd44003950 mon_map magic: 0 2024-08-19T12:38:33.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd44001ca0 con 0x7efd54073470 2024-08-19T12:38:33.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd5159a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd58c98640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.487+0000 7efd58c98640 4 set_mon_vals no callback set 2024-08-19T12:38:33.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7f2def186640 10 monclient(hunting): picked mon.noname-h con 0x7f2de813f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:33.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7f2decf22640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f2de8073470 0x7f2de8136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:39936/0 (socket says 172.21.15.26:39936) 2024-08-19T12:38:33.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7f2decf22640 1 -- 172.21.15.26:0/950831496 learned_addr learned my addr 172.21.15.26:0/950831496 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7f2def186640 1 -- 172.21.15.26:0/950831496 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de809dae0 con 0x7f2de80a57a0 2024-08-19T12:38:33.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7f2def186640 1 -- 172.21.15.26:0/950831496 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de8136910 con 0x7f2de8073470 2024-08-19T12:38:33.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7f2def186640 1 -- 172.21.15.26:0/950831496 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de809dd50 con 0x7f2de813f2e0 2024-08-19T12:38:33.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7f2def186640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7f2def186640 10 monclient(hunting): authenticate will time out at 707.575335s 2024-08-19T12:38:33.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7f2f47d6b640 1 Processor -- start 2024-08-19T12:38:33.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7efd58c98640 10 monclient: shutdown 2024-08-19T12:38:33.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7efd58c98640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7efd58c98640 1 -- 172.21.15.26:0/659349664 >> v1:172.21.15.26:6793/0 conn(0x7efd54073470 legacy=0x7efd54136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7efd5159a640 1 -- 172.21.15.26:0/659349664 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd44003950 con 0x7efd54073470 2024-08-19T12:38:33.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.489+0000 7f2f47d6b640 1 -- start start 2024-08-19T12:38:33.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.488+0000 7efd5159a640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:38:33.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.489+0000 7f2f47d6b640 10 monclient: init 2024-08-19T12:38:33.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.489+0000 7efd58c98640 1 -- 172.21.15.26:0/659349664 shutdown_connections 2024-08-19T12:38:33.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.489+0000 7efd58c98640 1 -- 172.21.15.26:0/659349664 >> 172.21.15.26:0/659349664 conn(0x7efd540abff0 msgr2=0x7efd540ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.489+0000 7efd58c98640 1 -- 172.21.15.26:0/659349664 shutdown_connections 2024-08-19T12:38:33.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.489+0000 7efd58c98640 1 -- 172.21.15.26:0/659349664 wait complete. 2024-08-19T12:38:33.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.489+0000 7f2f47d6b640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.489+0000 7f2f47d6b640 10 monclient: _add_conns ranks=[2,6,3,5,8,0,7,1,4] 2024-08-19T12:38:33.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2f47d6b640 10 monclient(hunting): picked mon.noname-b con 0x7f2f400a44d0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:33.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2decf22640 1 -- 172.21.15.26:0/950831496 >> v1:172.21.15.104:6790/0 conn(0x7f2de8073470 legacy=0x7f2de8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de7fff640 1 -- 172.21.15.26:0/950831496 >> v1:172.21.15.26:6789/0 conn(0x7f2de80a57a0 legacy=0x7f2de80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7efd58c98640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2f45306640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f2f400a4fc0 0x7f2f4013b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58490/0 (socket says 172.21.15.26:58490) 2024-08-19T12:38:33.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 184854469 0 0) 0x7f2de8136910 con 0x7f2de8073470 2024-08-19T12:38:33.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7efd58c98640 10 monclient: monmap: 2024-08-19T12:38:33.566 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.566 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:33.566 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:33.566 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:33.566 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.566 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.566 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.566 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.566 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.567 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.567 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.567 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.567 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.567 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.567 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.567 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2ded723640 1 -- 172.21.15.26:0/950831496 >> v1:172.21.15.104:6791/0 conn(0x7f2de813f2e0 legacy=0x7f2de81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient(hunting): my global_id is 4305 2024-08-19T12:38:33.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dd8003520 con 0x7f2de8073470 2024-08-19T12:38:33.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2f45306640 1 -- 172.21.15.26:0/2339083742 learned_addr learned my addr 172.21.15.26:0/2339083742 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1522689328 0 0) 0x7f2de809dae0 con 0x7f2de80a57a0 2024-08-19T12:38:33.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient(hunting): my global_id is 4248 2024-08-19T12:38:33.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2de8136910 con 0x7f2de80a57a0 2024-08-19T12:38:33.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 595057806 0 0) 0x7f2de809dd50 con 0x7f2de813f2e0 2024-08-19T12:38:33.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient(hunting): my global_id is 4271 2024-08-19T12:38:33.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2de809dae0 con 0x7f2de813f2e0 2024-08-19T12:38:33.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2f47d6b640 10 monclient(hunting): picked mon.noname-d con 0x7f2f400a4fc0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:33.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3694702461 0 0) 0x7f2dd8003520 con 0x7f2de8073470 2024-08-19T12:38:33.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 >> v1:172.21.15.104:6791/0 conn(0x7f2de813f2e0 legacy=0x7f2de81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 >> v1:172.21.15.26:6789/0 conn(0x7f2de80a57a0 legacy=0x7f2de80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 1 monclient: found mon.noname-g 2024-08-19T12:38:33.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7efd58c98640 1 librados: starting msgr at 2024-08-19T12:38:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2f47d6b640 10 monclient(hunting): picked mon.noname-g con 0x7f2f4013c9d0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7f2de77fe640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:38:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de809e030 con 0x7f2de8073470 2024-08-19T12:38:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ddc001ae0 con 0x7f2de8073470 2024-08-19T12:38:33.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.490+0000 7efd58c98640 1 librados: starting objecter 2024-08-19T12:38:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2f47d6b640 1 -- 172.21.15.26:0/2339083742 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f40071350 con 0x7f2f400a44d0 2024-08-19T12:38:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:38:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 10 monclient: dump: 2024-08-19T12:38:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.573 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.574 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.575 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2def186640 5 monclient: authenticate success, global_id 4305 2024-08-19T12:38:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2def186640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2f47d6b640 1 -- 172.21.15.26:0/2339083742 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f400704f0 con 0x7f2f4013c9d0 2024-08-19T12:38:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ddc0048a0 con 0x7f2de8073470 2024-08-19T12:38:33.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7efd58c98640 1 Processor -- start 2024-08-19T12:38:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2f47d6b640 1 -- 172.21.15.26:0/2339083742 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f40074310 con 0x7f2f400a4fc0 2024-08-19T12:38:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2f47d6b640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 1 -- 172.21.15.26:0/950831496 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ddc004d90 con 0x7f2de8073470 2024-08-19T12:38:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2def186640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2def186640 4 set_mon_vals no callback set 2024-08-19T12:38:33.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2f47d6b640 10 monclient(hunting): authenticate will time out at 707.578335s 2024-08-19T12:38:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7efd58c98640 1 -- start start 2024-08-19T12:38:33.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7efd58c98640 1 librados: setting wanted keys 2024-08-19T12:38:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7f2de77fe640 10 monclient: dump: 2024-08-19T12:38:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.579 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.580 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7efd58c98640 1 librados: calling monclient init 2024-08-19T12:38:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7efd58c98640 10 monclient: init 2024-08-19T12:38:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7efd58c98640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7efd58c98640 10 monclient: _add_conns ranks=[8,3,0,6,7,5,2,4,1] 2024-08-19T12:38:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7fe693cc5640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7fe693cc5640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.491+0000 7fe693cc5640 10 monclient: monmap: 2024-08-19T12:38:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.492772+0000 2024-08-19T12:38:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.492772+0000 2024-08-19T12:38:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.581 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7efd58c98640 10 monclient(hunting): picked mon.noname-e con 0x7efd54073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7f2def186640 10 monclient: shutdown 2024-08-19T12:38:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7f2def186640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7efd58c98640 10 monclient(hunting): picked mon.noname-g con 0x7efd540a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7efd5259c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7efd54073470 0x7efd5413e020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:33934/0 (socket says 172.21.15.26:33934) 2024-08-19T12:38:33.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7efd5259c640 1 -- 172.21.15.26:0/1866162104 learned_addr learned my addr 172.21.15.26:0/1866162104 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7f2def186640 1 -- 172.21.15.26:0/950831496 >> v1:172.21.15.104:6790/0 conn(0x7f2de8073470 legacy=0x7f2de8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7efd58c98640 10 monclient(hunting): picked mon.noname-a con 0x7efd5413f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd5413efb0 con 0x7efd5413f2e0 2024-08-19T12:38:33.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7f2def186640 1 -- 172.21.15.26:0/950831496 shutdown_connections 2024-08-19T12:38:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd540a3240 con 0x7efd540a57a0 2024-08-19T12:38:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7f2def186640 1 -- 172.21.15.26:0/950831496 >> 172.21.15.26:0/950831496 conn(0x7f2de80abff0 msgr2=0x7f2de80ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd540a52c0 con 0x7efd54073470 2024-08-19T12:38:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7efd58c98640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7efd58c98640 10 monclient(hunting): authenticate will time out at 707.579535s 2024-08-19T12:38:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7f2def186640 1 -- 172.21.15.26:0/950831496 shutdown_connections 2024-08-19T12:38:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7f2def186640 1 -- 172.21.15.26:0/950831496 wait complete. 2024-08-19T12:38:33.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7fe693cc5640 1 Processor -- start 2024-08-19T12:38:33.587 INFO:teuthology.orchestra.run.smithi026.stdout:51539607556 2024-08-19T12:38:33.587 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-19T12:38:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7fe693cc5640 1 -- start start 2024-08-19T12:38:33.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7fe693cc5640 10 monclient: init 2024-08-19T12:38:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.492+0000 7f2f45306640 1 -- 172.21.15.26:0/2339083742 >> v1:172.21.15.26:6792/0 conn(0x7f2f400a4fc0 legacy=0x7f2f4013b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1010395329 0 0) 0x7f2f40074310 con 0x7f2f400a4fc0 2024-08-19T12:38:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient(hunting): my global_id is 4308 2024-08-19T12:38:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f2c003520 con 0x7f2f400a4fc0 2024-08-19T12:38:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f45b07640 1 -- 172.21.15.26:0/2339083742 >> v1:172.21.15.26:6790/0 conn(0x7f2f400a44d0 legacy=0x7f2f400a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f46308640 1 -- 172.21.15.26:0/2339083742 >> v1:172.21.15.104:6790/0 conn(0x7f2f4013c9d0 legacy=0x7f2f4013edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1752339917 0 0) 0x7f2f40071350 con 0x7f2f400a44d0 2024-08-19T12:38:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2def186640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7fe693cc5640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7fe693cc5640 10 monclient: _add_conns ranks=[5,6,3,8,7,2,4,0,1] 2024-08-19T12:38:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7fe693cc5640 10 monclient(hunting): picked mon.noname-h con 0x7fe68c068650 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7fe693cc5640 10 monclient(hunting): picked mon.noname-d con 0x7fe68c09e6d0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7fe693cc5640 10 monclient(hunting): picked mon.noname-g con 0x7fe68c069140 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7fe693cc5640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68c09d400 con 0x7fe68c069140 2024-08-19T12:38:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7fe693cc5640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68c09d6e0 con 0x7fe68c068650 2024-08-19T12:38:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7fe693cc5640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68c09d9c0 con 0x7fe68c09e6d0 2024-08-19T12:38:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7fe693cc5640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7fe693cc5640 10 monclient(hunting): authenticate will time out at 707.580675s 2024-08-19T12:38:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2def186640 10 monclient: monmap: 2024-08-19T12:38:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:33.592 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.593 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7fe691260640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fe68c09e6d0 0x7fe68c13aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58498/0 (socket says 172.21.15.26:58498) 2024-08-19T12:38:33.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7fe691260640 1 -- 172.21.15.26:0/2065817150 learned_addr learned my addr 172.21.15.26:0/2065817150 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient(hunting): my global_id is 4277 2024-08-19T12:38:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f40074310 con 0x7f2f400a44d0 2024-08-19T12:38:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3028820993 0 0) 0x7f2f400704f0 con 0x7f2f4013c9d0 2024-08-19T12:38:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient(hunting): my global_id is 4314 2024-08-19T12:38:33.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f2c003a10 con 0x7f2f4013c9d0 2024-08-19T12:38:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2274302898 0 0) 0x7f2f2c003520 con 0x7f2f400a4fc0 2024-08-19T12:38:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 >> v1:172.21.15.104:6790/0 conn(0x7f2f4013c9d0 legacy=0x7f2f4013edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 >> v1:172.21.15.26:6790/0 conn(0x7f2f400a44d0 legacy=0x7f2f400a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 monclient: found mon.noname-d 2024-08-19T12:38:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7f2def186640 1 librados: starting msgr at 2024-08-19T12:38:33.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7f2def186640 1 librados: starting objecter 2024-08-19T12:38:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f400a5fa0 con 0x7f2f400a4fc0 2024-08-19T12:38:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2f30001ae0 con 0x7f2f400a4fc0 2024-08-19T12:38:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient: dump: 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.598 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.599 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd5259c640 1 -- 172.21.15.26:0/1866162104 >> v1:172.21.15.26:6793/0 conn(0x7efd54073470 legacy=0x7efd5413e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 client.?.objecter ms_handle_connect 0x7efd54073470 2024-08-19T12:38:33.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f300048a0 con 0x7f2f400a4fc0 2024-08-19T12:38:33.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f47d6b640 5 monclient: authenticate success, global_id 4308 2024-08-19T12:38:33.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f47d6b640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f47d6b640 4 set_mon_vals no callback set 2024-08-19T12:38:33.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 1 -- 172.21.15.26:0/2339083742 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2f30004e50 con 0x7f2f400a4fc0 2024-08-19T12:38:33.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:33.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.493+0000 7f2f44b05640 10 monclient: dump: 2024-08-19T12:38:33.600 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.601 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.602 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7f2def186640 1 Processor -- start 2024-08-19T12:38:33.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd51d9b640 1 -- 172.21.15.26:0/1866162104 >> v1:172.21.15.104:6790/0 conn(0x7efd540a57a0 legacy=0x7efd541cea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 client.?.objecter ms_handle_connect 0x7efd540a57a0 2024-08-19T12:38:33.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7f2def186640 1 -- start start 2024-08-19T12:38:33.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7f2def186640 1 librados: setting wanted keys 2024-08-19T12:38:33.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7f2def186640 1 librados: calling monclient init 2024-08-19T12:38:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7f2def186640 10 monclient: init 2024-08-19T12:38:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2053905099 0 0) 0x7efd540a52c0 con 0x7efd54073470 2024-08-19T12:38:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 monclient(hunting): my global_id is 4328 2024-08-19T12:38:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd34004eb0 con 0x7efd54073470 2024-08-19T12:38:33.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7f2f47d6b640 10 monclient: shutdown 2024-08-19T12:38:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7f2f47d6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7f2f47d6b640 1 -- 172.21.15.26:0/2339083742 >> v1:172.21.15.26:6792/0 conn(0x7f2f400a4fc0 legacy=0x7f2f4013b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1044107888 0 0) 0x7efd540a3240 con 0x7efd540a57a0 2024-08-19T12:38:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 monclient(hunting): my global_id is 4323 2024-08-19T12:38:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd52d9d640 1 -- 172.21.15.26:0/1866162104 >> v1:172.21.15.26:6789/0 conn(0x7efd5413f2e0 legacy=0x7efd541d2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd540a52c0 con 0x7efd540a57a0 2024-08-19T12:38:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 client.?.objecter ms_handle_connect 0x7efd5413f2e0 2024-08-19T12:38:33.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7efd42ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7f2f47d6b640 1 -- 172.21.15.26:0/2339083742 shutdown_connections 2024-08-19T12:38:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.494+0000 7f2f47d6b640 1 -- 172.21.15.26:0/2339083742 >> 172.21.15.26:0/2339083742 conn(0x7f2f4006ce70 msgr2=0x7f2f400a55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7f2f47d6b640 1 -- 172.21.15.26:0/2339083742 shutdown_connections 2024-08-19T12:38:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3764647739 0 0) 0x7efd34004eb0 con 0x7efd54073470 2024-08-19T12:38:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd540a3240 con 0x7efd54073470 2024-08-19T12:38:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd44001a40 con 0x7efd54073470 2024-08-19T12:38:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.607 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.608 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 975851352 0 0) 0x7efd5413efb0 con 0x7efd5413f2e0 2024-08-19T12:38:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient(hunting): my global_id is 4257 2024-08-19T12:38:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd34004eb0 con 0x7efd5413f2e0 2024-08-19T12:38:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4132533630 0 0) 0x7efd540a52c0 con 0x7efd540a57a0 2024-08-19T12:38:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd5413efb0 con 0x7efd540a57a0 2024-08-19T12:38:33.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7f2f47d6b640 1 -- 172.21.15.26:0/2339083742 wait complete. 2024-08-19T12:38:33.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd480047e0 con 0x7efd540a57a0 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.611 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1655670056 0 0) 0x7efd540a3240 con 0x7efd54073470 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 >> v1:172.21.15.104:6790/0 conn(0x7efd540a57a0 legacy=0x7efd541cea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 >> v1:172.21.15.26:6789/0 conn(0x7efd5413f2e0 legacy=0x7efd541d2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 monclient: found mon.e 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5413e730 con 0x7efd54073470 2024-08-19T12:38:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7f2def186640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd58c98640 5 monclient: authenticate success, global_id 4328 2024-08-19T12:38:33.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd58c98640 10 monclient: _renew_subs 2024-08-19T12:38:33.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd58c98640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efd540a3da0 con 0x7efd54073470 2024-08-19T12:38:33.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7f2def186640 10 monclient: _add_conns ranks=[8,4,7,6,3,0,5,2,1] 2024-08-19T12:38:33.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd44003fb0 con 0x7efd54073470 2024-08-19T12:38:33.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd58c98640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd58c98640 10 monclient: _renew_subs 2024-08-19T12:38:33.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd58c98640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7efd541c98f0 con 0x7efd54073470 2024-08-19T12:38:33.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd58c98640 1 librados: init done 2024-08-19T12:38:33.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7f2def186640 10 monclient(hunting): picked mon.noname-e con 0x7f2de8073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:33.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd43fff640 4 set_mon_vals no callback set 2024-08-19T12:38:33.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd44001be0 con 0x7efd54073470 2024-08-19T12:38:33.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7f2decf22640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f2de8073470 0x7f2de81ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:33946/0 (socket says 172.21.15.26:33946) 2024-08-19T12:38:33.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7f2def186640 10 monclient(hunting): picked mon.noname-c con 0x7f2de80a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:33.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7efd42ffd640 10 monclient: dump: 2024-08-19T12:38:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.617 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.618 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.618 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7f2def186640 10 monclient(hunting): picked mon.noname-i con 0x7f2de813f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7f2decf22640 1 -- 172.21.15.26:0/2860903982 learned_addr learned my addr 172.21.15.26:0/2860903982 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de806d8f0 con 0x7f2de80a57a0 2024-08-19T12:38:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.495+0000 7fe691260640 1 -- 172.21.15.26:0/2065817150 >> v1:172.21.15.26:6792/0 conn(0x7fe68c09e6d0 legacy=0x7fe68c13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de80a3a40 con 0x7f2de813f2e0 2024-08-19T12:38:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de8136eb0 con 0x7f2de8073470 2024-08-19T12:38:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 1 -- 172.21.15.26:0/2065817150 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1626326126 0 0) 0x7fe68c09d9c0 con 0x7fe68c09e6d0 2024-08-19T12:38:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient(hunting): my global_id is 4317 2024-08-19T12:38:33.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 1 -- 172.21.15.26:0/2065817150 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe678005730 con 0x7fe68c09e6d0 2024-08-19T12:38:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2def186640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2def186640 10 monclient(hunting): authenticate will time out at 707.583348s 2024-08-19T12:38:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 1 -- 172.21.15.26:0/2065817150 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4051252372 0 0) 0x7fe678005730 con 0x7fe68c09e6d0 2024-08-19T12:38:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 1 -- 172.21.15.26:0/2065817150 >> v1:172.21.15.104:6791/0 conn(0x7fe68c068650 legacy=0x7fe68c068a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T12:38:33.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 1 -- 172.21.15.26:0/2065817150 >> v1:172.21.15.104:6790/0 conn(0x7fe68c069140 legacy=0x7fe68c13e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 1 monclient: found mon.noname-d 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 1 -- 172.21.15.26:0/2065817150 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c0a8ef0 con 0x7fe68c09e6d0 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 1 -- 172.21.15.26:0/2065817150 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe67c001ae0 con 0x7fe68c09e6d0 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient: dump: 2024-08-19T12:38:33.622 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.623 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.624 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe693cc5640 5 monclient: authenticate success, global_id 4317 2024-08-19T12:38:33.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe693cc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2f47d6b640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2f47d6b640 10 monclient: monmap: 2024-08-19T12:38:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.625 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2f47d6b640 1 librados: starting msgr at 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2f47d6b640 1 librados: starting objecter 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2f47d6b640 1 Processor -- start 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2f47d6b640 1 -- start start 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2f47d6b640 1 librados: setting wanted keys 2024-08-19T12:38:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2f47d6b640 1 librados: calling monclient init 2024-08-19T12:38:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 1 -- 172.21.15.26:0/2065817150 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe67c0048a0 con 0x7fe68c09e6d0 2024-08-19T12:38:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 1 -- 172.21.15.26:0/2065817150 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe67c004d90 con 0x7fe68c09e6d0 2024-08-19T12:38:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe690a5f640 10 monclient: dump: 2024-08-19T12:38:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.627 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.628 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.628 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.628 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.628 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.628 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.628 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.628 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.629 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.629 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.629 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.629 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.629 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.629 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7f2f47d6b640 10 monclient: init 2024-08-19T12:38:33.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe693cc5640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7fe693cc5640 4 set_mon_vals no callback set 2024-08-19T12:38:33.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7efd58c98640 10 client.4328.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7efd58c98640 15 client.4328.objecter _session_command_op_assign -1 1 2024-08-19T12:38:33.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7efd58c98640 20 client.4328.objecter _get_session osd=-1 returning homeless 2024-08-19T12:38:33.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7efd58c98640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7efd58c98640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.496+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.26:6793/0 -- mon_get_version(what=osdmap handle=1) -- 0x7efd38002910 con 0x7efd54073470 2024-08-19T12:38:33.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7efd44011a80 con 0x7efd54073470 2024-08-19T12:38:33.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7f2f47d6b640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7f2f47d6b640 10 monclient: _add_conns ranks=[0,3,7,8,6,1,5,4,2] 2024-08-19T12:38:33.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7f2f47d6b640 10 monclient(hunting): picked mon.noname-a con 0x7f2f400a44d0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:33.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7f2f47d6b640 10 monclient(hunting): picked mon.noname-g con 0x7f2f400a4fc0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:33.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7fe693cc5640 10 monclient: shutdown 2024-08-19T12:38:33.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7f2f47d6b640 10 monclient(hunting): picked mon.noname-i con 0x7f2f4013c9d0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:33.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7fe693cc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7f2f47d6b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f40068ed0 con 0x7f2f400a44d0 2024-08-19T12:38:33.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7f2f45b07640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2f400a44d0 0x7f2f401daef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58392/0 (socket says 172.21.15.26:58392) 2024-08-19T12:38:33.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7f2f45b07640 1 -- 172.21.15.26:0/1509160170 learned_addr learned my addr 172.21.15.26:0/1509160170 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f40138a10 con 0x7f2f400a4fc0 2024-08-19T12:38:33.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7efd4404db70 con 0x7efd54073470 2024-08-19T12:38:33.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 10 client.4328.objecter ms_dispatch 0x7efd54000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:33.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7fe693cc5640 1 -- 172.21.15.26:0/2065817150 >> v1:172.21.15.26:6792/0 conn(0x7fe68c09e6d0 legacy=0x7fe68c13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f40133a10 con 0x7f2f4013c9d0 2024-08-19T12:38:33.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7f2f47d6b640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 3 client.4328.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:33.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7f2f47d6b640 10 monclient(hunting): authenticate will time out at 707.584772s 2024-08-19T12:38:33.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 3 client.4328.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:33.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7fe693cc5640 1 -- 172.21.15.26:0/2065817150 shutdown_connections 2024-08-19T12:38:33.641 INFO:teuthology.orchestra.run.smithi026.stdout:51539607556 2024-08-19T12:38:33.641 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-19T12:38:33.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 10 client.4328.objecter checking command 1 2024-08-19T12:38:33.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 20 client.4328.objecter _get_session s=0x7efd34047b90 osd=3 3 2024-08-19T12:38:33.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 20 client.4328.objecter put_session s=0x7efd34047b90 osd=3 3 2024-08-19T12:38:33.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 15 client.4328.objecter _session_command_op_remove -1 1 2024-08-19T12:38:33.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 20 client.4328.objecter _get_session s=0x7efd34047b90 osd=3 3 2024-08-19T12:38:33.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 20 client.4328.objecter get_session s=0x7efd34047b90 osd=3 3 2024-08-19T12:38:33.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 15 client.4328.objecter _session_command_op_assign 3 1 2024-08-19T12:38:33.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 20 client.4328.objecter put_session s=0x7efd34047b90 osd=3 4 2024-08-19T12:38:33.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 10 client.4328.objecter _send_command 1 2024-08-19T12:38:33.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7fe693cc5640 1 -- 172.21.15.26:0/2065817150 >> 172.21.15.26:0/2065817150 conn(0x7fe68c0a8160 msgr2=0x7fe68c0a8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.104:6809/847838248 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd3404bef0 con 0x7efd34048240 2024-08-19T12:38:33.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 20 client.4328.objecter dump_active .. 0 homeless 2024-08-19T12:38:33.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7efd4404dfe0 con 0x7efd54073470 2024-08-19T12:38:33.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd42ffd640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-08-19T12:38:33.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7efd51d9b640 10 monclient: get_auth_request con 0x7efd34041640 auth_method 0 2024-08-19T12:38:33.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.497+0000 7fe693cc5640 1 -- 172.21.15.26:0/2065817150 shutdown_connections 2024-08-19T12:38:33.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7fe693cc5640 1 -- 172.21.15.26:0/2065817150 wait complete. 2024-08-19T12:38:33.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2decf22640 1 -- 172.21.15.26:0/2860903982 >> v1:172.21.15.26:6793/0 conn(0x7f2de8073470 legacy=0x7f2de81ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7efd52d9d640 10 monclient: get_auth_request con 0x7efd34048240 auth_method 0 2024-08-19T12:38:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 client.?.objecter ms_handle_connect 0x7f2de8073470 2024-08-19T12:38:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1666822297 0 0) 0x7f2de8136eb0 con 0x7f2de8073470 2024-08-19T12:38:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 monclient(hunting): my global_id is 4337 2024-08-19T12:38:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dcc004ee0 con 0x7f2de8073470 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7fe693cc5640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7fe693cc5640 10 monclient: monmap: 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.649 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.649 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.649 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.649 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.649 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 448251830 0 0) 0x7f2dcc004ee0 con 0x7f2de8073470 2024-08-19T12:38:33.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2de8136eb0 con 0x7f2de8073470 2024-08-19T12:38:33.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7fe693cc5640 1 librados: starting msgr at 2024-08-19T12:38:33.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 client.?.objecter ms_handle_connect 0x7f2de80a57a0 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ddc001a60 con 0x7f2de8073470 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de7fff640 1 -- 172.21.15.26:0/2860903982 >> v1:172.21.15.26:6791/0 conn(0x7f2de80a57a0 legacy=0x7f2de81d33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 monclient(hunting): dump: 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.651 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.652 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7fe693cc5640 1 librados: starting objecter 2024-08-19T12:38:33.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7fe693cc5640 1 Processor -- start 2024-08-19T12:38:33.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2ded723640 1 -- 172.21.15.26:0/2860903982 >> v1:172.21.15.104:6792/0 conn(0x7f2de813f2e0 legacy=0x7f2de81d6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 client.?.objecter ms_handle_connect 0x7f2de813f2e0 2024-08-19T12:38:33.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3146353261 0 0) 0x7f2de8136eb0 con 0x7f2de8073470 2024-08-19T12:38:33.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.498+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 >> v1:172.21.15.104:6792/0 conn(0x7f2de813f2e0 legacy=0x7f2de81d6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 1 -- start start 2024-08-19T12:38:33.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 1 librados: setting wanted keys 2024-08-19T12:38:33.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 1 librados: calling monclient init 2024-08-19T12:38:33.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 10 monclient: init 2024-08-19T12:38:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 >> v1:172.21.15.26:6791/0 conn(0x7f2de80a57a0 legacy=0x7f2de81d33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 1 monclient: found mon.e 2024-08-19T12:38:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de8137190 con 0x7f2de8073470 2024-08-19T12:38:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2def186640 5 monclient: authenticate success, global_id 4337 2024-08-19T12:38:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2def186640 10 monclient: _renew_subs 2024-08-19T12:38:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2def186640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2de813ed70 con 0x7f2de8073470 2024-08-19T12:38:33.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2def186640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2def186640 10 monclient: _renew_subs 2024-08-19T12:38:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2def186640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f2de80a5140 con 0x7f2de8073470 2024-08-19T12:38:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2def186640 1 librados: init done 2024-08-19T12:38:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ddc001d30 con 0x7f2de8073470 2024-08-19T12:38:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ddc005640 con 0x7f2de8073470 2024-08-19T12:38:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de6ffd640 4 set_mon_vals no callback set 2024-08-19T12:38:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:33.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2de57fa640 10 monclient: dump: 2024-08-19T12:38:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.658 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.659 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.659 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.659 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.659 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.659 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.659 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.659 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.659 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.659 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 10 monclient: _add_conns ranks=[6,5,8,2,3,1,0,4,7] 2024-08-19T12:38:33.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 10 monclient(hunting): picked mon.noname-d con 0x7fe68c068650 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:33.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 10 monclient(hunting): picked mon.noname-h con 0x7fe68c069140 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:33.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 10 monclient(hunting): picked mon.noname-e con 0x7fe68c09e6d0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:33.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68c134d10 con 0x7fe68c069140 2024-08-19T12:38:33.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68c0a2010 con 0x7fe68c068650 2024-08-19T12:38:33.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68c0a2300 con 0x7fe68c09e6d0 2024-08-19T12:38:33.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe693cc5640 10 monclient(hunting): authenticate will time out at 707.586493s 2024-08-19T12:38:33.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe691a61640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fe68c068650 0x7fe68c133630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58510/0 (socket says 172.21.15.26:58510) 2024-08-19T12:38:33.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7fe691a61640 1 -- 172.21.15.26:0/1525925952 learned_addr learned my addr 172.21.15.26:0/1525925952 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2f45b07640 1 -- 172.21.15.26:0/1509160170 >> v1:172.21.15.26:6789/0 conn(0x7f2f400a44d0 legacy=0x7f2f401daef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2f367fc640 10 client.?.objecter ms_handle_connect 0x7f2f400a44d0 2024-08-19T12:38:33.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.499+0000 7f2f367fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2f45306640 1 -- 172.21.15.26:0/1509160170 >> v1:172.21.15.104:6790/0 conn(0x7f2f400a4fc0 legacy=0x7f2f40131130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2f367fc640 10 client.?.objecter ms_handle_connect 0x7f2f400a4fc0 2024-08-19T12:38:33.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2f367fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2def186640 10 client.4337.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2def186640 15 client.4337.objecter _session_command_op_assign -1 1 2024-08-19T12:38:33.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2def186640 20 client.4337.objecter _get_session osd=-1 returning homeless 2024-08-19T12:38:33.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2def186640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2def186640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 --> v1:172.21.15.26:6793/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f2dc8002910 con 0x7f2de8073470 2024-08-19T12:38:33.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2f45306640 1 --1- 172.21.15.26:0/1509160170 >> v1:172.21.15.104:6790/0 conn(0x7f2f400a4fc0 0x7f2f40131130 :-1 s=OPENED pgs=62 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f2f40138a10 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:38:33.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7efd42ffd640 10 client.4328.objecter ms_handle_connect 0x7efd34041640 2024-08-19T12:38:33.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2ddc011aa0 con 0x7f2de8073470 2024-08-19T12:38:33.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f2ddc04db90 con 0x7f2de8073470 2024-08-19T12:38:33.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 10 client.4337.objecter ms_dispatch 0x7f2de8000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:33.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 3 client.4337.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:33.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 3 client.4337.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:33.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 10 client.4337.objecter checking command 1 2024-08-19T12:38:33.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 20 client.4337.objecter _get_session s=0x7f2dcc047410 osd=1 3 2024-08-19T12:38:33.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 20 client.4337.objecter put_session s=0x7f2dcc047410 osd=1 3 2024-08-19T12:38:33.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 15 client.4337.objecter _session_command_op_remove -1 1 2024-08-19T12:38:33.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 20 client.4337.objecter _get_session s=0x7f2dcc047410 osd=1 3 2024-08-19T12:38:33.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 20 client.4337.objecter get_session s=0x7f2dcc047410 osd=1 3 2024-08-19T12:38:33.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 15 client.4337.objecter _session_command_op_assign 1 1 2024-08-19T12:38:33.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 20 client.4337.objecter put_session s=0x7f2dcc047410 osd=1 4 2024-08-19T12:38:33.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 10 client.4337.objecter _send_command 1 2024-08-19T12:38:33.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 --> v1:172.21.15.26:6804/2594673920 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ddc0048a0 con 0x7f2dcc047ac0 2024-08-19T12:38:33.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 20 client.4337.objecter dump_active .. 0 homeless 2024-08-19T12:38:33.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f2ddc04e000 con 0x7f2de8073470 2024-08-19T12:38:33.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de57fa640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-08-19T12:38:33.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2de7fff640 10 monclient: get_auth_request con 0x7f2dcc040ec0 auth_method 0 2024-08-19T12:38:33.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.500+0000 7f2ded723640 10 monclient: get_auth_request con 0x7f2dcc047ac0 auth_method 0 2024-08-19T12:38:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 client.?.objecter ms_handle_connect 0x7f2f4013c9d0 2024-08-19T12:38:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f46308640 1 -- 172.21.15.26:0/1509160170 >> v1:172.21.15.104:6792/0 conn(0x7f2f4013c9d0 legacy=0x7f2f401e30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1200213657 0 0) 0x7f2f40133a10 con 0x7f2f4013c9d0 2024-08-19T12:38:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient(hunting): my global_id is 4264 2024-08-19T12:38:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f24004eb0 con 0x7f2f4013c9d0 2024-08-19T12:38:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2209578827 0 0) 0x7f2f24004eb0 con 0x7f2f4013c9d0 2024-08-19T12:38:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f40133a10 con 0x7f2f4013c9d0 2024-08-19T12:38:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2f380020b0 con 0x7f2f4013c9d0 2024-08-19T12:38:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient(hunting): dump: 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.671 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.671 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.671 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.671 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:33.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7fe691a61640 1 -- 172.21.15.26:0/1525925952 >> v1:172.21.15.26:6792/0 conn(0x7fe68c068650 legacy=0x7fe68c133630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7fe6827fc640 10 client.?.objecter ms_handle_connect 0x7fe68c068650 2024-08-19T12:38:33.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7fe6827fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7fe691260640 1 -- 172.21.15.26:0/1525925952 >> v1:172.21.15.104:6791/0 conn(0x7fe68c069140 legacy=0x7fe68c133d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7fe6827fc640 10 client.?.objecter ms_handle_connect 0x7fe68c069140 2024-08-19T12:38:33.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7fe6827fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3227592232 0 0) 0x7f2f40133a10 con 0x7f2f4013c9d0 2024-08-19T12:38:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 >> v1:172.21.15.104:6790/0 conn(0x7f2f400a4fc0 legacy=0x7f2f40131130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 >> v1:172.21.15.26:6789/0 conn(0x7f2f400a44d0 legacy=0x7f2f401daef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 1 monclient: found mon.i 2024-08-19T12:38:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:33.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f40133730 con 0x7f2f4013c9d0 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017693538 0 0) 0x7f2f40138a10 con 0x7f2f400a4fc0 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.501+0000 7f2f367fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 5 monclient: authenticate success, global_id 4264 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 10 monclient: _renew_subs 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2f40132e60 con 0x7f2f4013c9d0 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 10 monclient: _renew_subs 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:33.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f2f401db7e0 con 0x7f2f4013c9d0 2024-08-19T12:38:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 1 librados: init done 2024-08-19T12:38:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 client.?.objecter ms_handle_connect 0x7fe68c09e6d0 2024-08-19T12:38:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe692262640 1 -- 172.21.15.26:0/1525925952 >> v1:172.21.15.26:6793/0 conn(0x7fe68c09e6d0 legacy=0x7fe68c1d5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f38003e80 con 0x7f2f4013c9d0 2024-08-19T12:38:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f367fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2f380043b0 con 0x7f2f4013c9d0 2024-08-19T12:38:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f377fe640 4 set_mon_vals no callback set 2024-08-19T12:38:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f367fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f367fc640 10 monclient: dump: 2024-08-19T12:38:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.677 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 65939145 0 0) 0x7fe688003c00 con 0x7fe68c068650 2024-08-19T12:38:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): my global_id is 4326 2024-08-19T12:38:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe670004eb0 con 0x7fe68c068650 2024-08-19T12:38:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 208127531 0 0) 0x7fe68c0a2300 con 0x7fe68c09e6d0 2024-08-19T12:38:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): my global_id is 4346 2024-08-19T12:38:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe688003c00 con 0x7fe68c09e6d0 2024-08-19T12:38:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7efd42ffd640 10 client.4328.objecter ms_handle_connect 0x7efd34048240 2024-08-19T12:38:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4061896867 0 0) 0x7fe670004eb0 con 0x7fe68c068650 2024-08-19T12:38:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe68c0a2300 con 0x7fe68c068650 2024-08-19T12:38:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe688001210 con 0x7fe68c068650 2024-08-19T12:38:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): dump: 2024-08-19T12:38:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4247254843 0 0) 0x7fe688003c00 con 0x7fe68c09e6d0 2024-08-19T12:38:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe670004eb0 con 0x7fe68c09e6d0 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe684004810 con 0x7fe68c09e6d0 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient(hunting): dump: 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.683 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3471477820 0 0) 0x7fe68c0a2300 con 0x7fe68c068650 2024-08-19T12:38:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 >> v1:172.21.15.26:6793/0 conn(0x7fe68c09e6d0 legacy=0x7fe68c1d5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 >> v1:172.21.15.104:6791/0 conn(0x7fe68c069140 legacy=0x7fe68c133d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 monclient: found mon.d 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c134450 con 0x7fe68c068650 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe693cc5640 5 monclient: authenticate success, global_id 4326 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe693cc5640 10 monclient: _renew_subs 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe693cc5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe693cc5640 1 -- 172.21.15.26:0/1525925952 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe68c09bfb0 con 0x7fe68c068650 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe693cc5640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe693cc5640 10 monclient: _renew_subs 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe693cc5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe693cc5640 1 -- 172.21.15.26:0/1525925952 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe68c08f000 con 0x7fe68c068650 2024-08-19T12:38:33.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe693cc5640 1 librados: init done 2024-08-19T12:38:33.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 10 client.4264.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 15 client.4264.objecter _session_command_op_assign -1 1 2024-08-19T12:38:33.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 20 client.4264.objecter _get_session osd=-1 returning homeless 2024-08-19T12:38:33.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:33.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 --> v1:172.21.15.104:6792/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f2f28002910 con 0x7f2f4013c9d0 2024-08-19T12:38:33.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe688004420 con 0x7fe68c068650 2024-08-19T12:38:33.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe688001ad0 con 0x7fe68c068650 2024-08-19T12:38:33.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.502+0000 7fe683fff640 4 set_mon_vals no callback set 2024-08-19T12:38:33.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7fe6827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7fe6827fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:33.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2de57fa640 10 client.4337.objecter ms_handle_connect 0x7f2dcc040ec0 2024-08-19T12:38:33.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7fe6827fc640 10 monclient: dump: 2024-08-19T12:38:33.687 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.687 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.687 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.687 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.687 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.687 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.688 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2f38004550 con 0x7f2f4013c9d0 2024-08-19T12:38:33.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== osd.3 v1:172.21.15.104:6809/847838248 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7efd3404bef0 con 0x7efd34048240 2024-08-19T12:38:33.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7efd42ffd640 10 client.4328.objecter ms_dispatch 0x7efd54000b90 command_reply(tid 1: 0 ) 2024-08-19T12:38:33.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7efd42ffd640 10 client.4328.objecter _finish_command 1 = system:0 2024-08-19T12:38:33.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7efd42ffd640 20 client.4328.objecter put_session s=0x7efd34047b90 osd=3 4 2024-08-19T12:38:33.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7efd42ffd640 15 client.4328.objecter _session_command_op_remove 3 1 2024-08-19T12:38:33.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f2f3804dec0 con 0x7f2f4013c9d0 2024-08-19T12:38:33.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 10 client.4264.objecter ms_dispatch 0x7f2f40000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:33.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 3 client.4264.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:33.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 3 client.4264.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:33.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 10 client.4264.objecter checking command 1 2024-08-19T12:38:33.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe688011a80 con 0x7fe68c068650 2024-08-19T12:38:33.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 20 client.4264.objecter _get_session s=0x7f2f24047490 osd=0 3 2024-08-19T12:38:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 20 client.4264.objecter put_session s=0x7f2f24047490 osd=0 3 2024-08-19T12:38:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7fe693cc5640 10 client.4326.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 15 client.4264.objecter _session_command_op_remove -1 1 2024-08-19T12:38:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 20 client.4264.objecter _get_session s=0x7f2f24047490 osd=0 3 2024-08-19T12:38:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 20 client.4264.objecter get_session s=0x7f2f24047490 osd=0 3 2024-08-19T12:38:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 15 client.4264.objecter _session_command_op_assign 0 1 2024-08-19T12:38:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 20 client.4264.objecter put_session s=0x7f2f24047490 osd=0 4 2024-08-19T12:38:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 10 client.4264.objecter _send_command 1 2024-08-19T12:38:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 --> v1:172.21.15.26:6800/3531533167 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f380020b0 con 0x7f2f24047b40 2024-08-19T12:38:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 20 client.4264.objecter dump_active .. 0 homeless 2024-08-19T12:38:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 <== mon.7 v1:172.21.15.104:6792/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f2f3804e330 con 0x7f2f4013c9d0 2024-08-19T12:38:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f367fc640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-08-19T12:38:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f45b07640 10 monclient: get_auth_request con 0x7f2f24040f90 auth_method 0 2024-08-19T12:38:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7fe693cc5640 15 client.4326.objecter _session_command_op_assign -1 1 2024-08-19T12:38:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7fe693cc5640 20 client.4326.objecter _get_session osd=-1 returning homeless 2024-08-19T12:38:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7f2f45306640 10 monclient: get_auth_request con 0x7f2f24047b40 auth_method 0 2024-08-19T12:38:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7fe693cc5640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7fe693cc5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.503+0000 7fe693cc5640 1 -- 172.21.15.26:0/1525925952 --> v1:172.21.15.26:6792/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe674002910 con 0x7fe68c068650 2024-08-19T12:38:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7fe68804dcc0 con 0x7fe68c068650 2024-08-19T12:38:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 10 client.4326.objecter ms_dispatch 0x7fe68c000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 3 client.4326.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 3 client.4326.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 10 client.4326.objecter checking command 1 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 20 client.4326.objecter _get_session s=0x7fe670047c20 osd=5 3 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 20 client.4326.objecter put_session s=0x7fe670047c20 osd=5 3 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 15 client.4326.objecter _session_command_op_remove -1 1 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe691260640 10 monclient: get_auth_request con 0x7fe670041720 auth_method 0 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 20 client.4326.objecter _get_session s=0x7fe670047c20 osd=5 3 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 20 client.4326.objecter get_session s=0x7fe670047c20 osd=5 3 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 15 client.4326.objecter _session_command_op_assign 5 1 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 20 client.4326.objecter put_session s=0x7fe670047c20 osd=5 4 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 10 client.4326.objecter _send_command 1 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 --> v1:172.21.15.104:6801/3713183117 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe684004810 con 0x7fe6700482d0 2024-08-19T12:38:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 20 client.4326.objecter dump_active .. 0 homeless 2024-08-19T12:38:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7fe688016e90 con 0x7fe68c068650 2024-08-19T12:38:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe6827fc640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-08-19T12:38:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.504+0000 7fe692262640 10 monclient: get_auth_request con 0x7fe6700482d0 auth_method 0 2024-08-19T12:38:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.505+0000 7f2de57fa640 10 client.4337.objecter ms_handle_connect 0x7f2dcc047ac0 2024-08-19T12:38:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.506+0000 7f2f367fc640 10 client.4264.objecter ms_handle_connect 0x7f2f24040f90 2024-08-19T12:38:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.506+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 <== osd.1 v1:172.21.15.26:6804/2594673920 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2ddc0048a0 con 0x7f2dcc047ac0 2024-08-19T12:38:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.506+0000 7f2de57fa640 10 client.4337.objecter ms_dispatch 0x7f2de8000b90 command_reply(tid 1: 0 ) 2024-08-19T12:38:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.506+0000 7f2de57fa640 10 client.4337.objecter _finish_command 1 = system:0 2024-08-19T12:38:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.506+0000 7f2de57fa640 20 client.4337.objecter put_session s=0x7f2dcc047410 osd=1 4 2024-08-19T12:38:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.506+0000 7f2de57fa640 15 client.4337.objecter _session_command_op_remove 1 1 2024-08-19T12:38:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.506+0000 7fe6827fc640 10 client.4326.objecter ms_handle_connect 0x7fe670041720 2024-08-19T12:38:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.508+0000 7f2f367fc640 10 client.4264.objecter ms_handle_connect 0x7f2f24047b40 2024-08-19T12:38:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.508+0000 7fe6827fc640 10 client.4326.objecter ms_handle_connect 0x7fe6700482d0 2024-08-19T12:38:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.508+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 <== osd.0 v1:172.21.15.26:6800/3531533167 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2f380020b0 con 0x7f2f24047b40 2024-08-19T12:38:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.508+0000 7f2f367fc640 10 client.4264.objecter ms_dispatch 0x7f2f40000b90 command_reply(tid 1: 0 ) 2024-08-19T12:38:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.508+0000 7f2f367fc640 10 client.4264.objecter _finish_command 1 = system:0 2024-08-19T12:38:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.508+0000 7f2f367fc640 20 client.4264.objecter put_session s=0x7f2f24047490 osd=0 4 2024-08-19T12:38:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.508+0000 7f2f367fc640 15 client.4264.objecter _session_command_op_remove 0 1 2024-08-19T12:38:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.509+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== osd.5 v1:172.21.15.104:6801/3713183117 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe684004810 con 0x7fe6700482d0 2024-08-19T12:38:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.509+0000 7fe6827fc640 10 client.4326.objecter ms_dispatch 0x7fe68c000b90 command_reply(tid 1: 0 ) 2024-08-19T12:38:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.509+0000 7fe6827fc640 10 client.4326.objecter _finish_command 1 = system:0 2024-08-19T12:38:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.509+0000 7fe6827fc640 20 client.4326.objecter put_session s=0x7fe670047c20 osd=5 4 2024-08-19T12:38:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.510+0000 7fe6827fc640 15 client.4326.objecter _session_command_op_remove 5 1 2024-08-19T12:38:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 10 client.4328.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-19T12:38:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 15 client.4328.objecter _session_command_op_assign -1 2 2024-08-19T12:38:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 20 client.4328.objecter _get_session s=0x7efd34047b90 osd=3 3 2024-08-19T12:38:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 20 client.4328.objecter put_session s=0x7efd34047b90 osd=3 3 2024-08-19T12:38:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 20 client.4328.objecter _get_session s=0x7efd34047b90 osd=3 3 2024-08-19T12:38:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 15 client.4328.objecter _session_command_op_remove -1 2 2024-08-19T12:38:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 20 client.4328.objecter get_session s=0x7efd34047b90 osd=3 3 2024-08-19T12:38:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 15 client.4328.objecter _session_command_op_assign 3 2 2024-08-19T12:38:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 20 client.4328.objecter put_session s=0x7efd34047b90 osd=3 4 2024-08-19T12:38:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 10 client.4328.objecter _send_command 2 2024-08-19T12:38:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 --> v1:172.21.15.104:6809/847838248 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efd38001220 con 0x7efd34048240 2024-08-19T12:38:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd42ffd640 1 -- 172.21.15.26:0/1866162104 <== osd.3 v1:172.21.15.104:6809/847838248 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3543570501) 0x7efd38001220 con 0x7efd34048240 2024-08-19T12:38:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd42ffd640 10 client.4328.objecter ms_dispatch 0x7efd54000b90 command_reply(tid 2: 0 ) 2024-08-19T12:38:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd42ffd640 10 client.4328.objecter _finish_command 2 = system:0 2024-08-19T12:38:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd42ffd640 20 client.4328.objecter put_session s=0x7efd34047b90 osd=3 4 2024-08-19T12:38:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd42ffd640 15 client.4328.objecter _session_command_op_remove 3 2 2024-08-19T12:38:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 10 librados: watch_flush enter 2024-08-19T12:38:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 10 librados: watch_flush exit 2024-08-19T12:38:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 10 client.4328.objecter close_session for osd.3 2024-08-19T12:38:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 >> v1:172.21.15.104:6809/847838248 conn(0x7efd34048240 legacy=0x7efd3404a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 20 client.4328.objecter put_session s=0x7efd34047b90 osd=3 1 2024-08-19T12:38:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 20 client.4328.objecter shutdown clearing up homeless session... 2024-08-19T12:38:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 10 client.4328.objecter successfully canceled tick 2024-08-19T12:38:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 >> v1:172.21.15.104:6800/899887506 conn(0x7efd34041640 legacy=0x7efd34043b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 10 monclient: shutdown 2024-08-19T12:38:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 >> v1:172.21.15.26:6793/0 conn(0x7efd54073470 legacy=0x7efd5413e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.518+0000 7efd52d9d640 1 -- 172.21.15.26:0/1866162104 reap_dead start 2024-08-19T12:38:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.519+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 shutdown_connections 2024-08-19T12:38:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.519+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 >> 172.21.15.26:0/1866162104 conn(0x7efd540abff0 msgr2=0x7efd54133cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.519+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 shutdown_connections 2024-08-19T12:38:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.519+0000 7efd58c98640 1 -- 172.21.15.26:0/1866162104 wait complete. 2024-08-19T12:38:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.519+0000 7efd58c98640 1 librados: shutdown 2024-08-19T12:38:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.523+0000 7f2f47d6b640 10 client.4264.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-19T12:38:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.523+0000 7f2f47d6b640 15 client.4264.objecter _session_command_op_assign -1 2 2024-08-19T12:38:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.523+0000 7f2f47d6b640 20 client.4264.objecter _get_session s=0x7f2f24047490 osd=0 3 2024-08-19T12:38:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.523+0000 7f2f47d6b640 20 client.4264.objecter put_session s=0x7f2f24047490 osd=0 3 2024-08-19T12:38:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.523+0000 7f2f47d6b640 20 client.4264.objecter _get_session s=0x7f2f24047490 osd=0 3 2024-08-19T12:38:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.523+0000 7f2f47d6b640 15 client.4264.objecter _session_command_op_remove -1 2 2024-08-19T12:38:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.523+0000 7f2f47d6b640 20 client.4264.objecter get_session s=0x7f2f24047490 osd=0 3 2024-08-19T12:38:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.523+0000 7f2f47d6b640 15 client.4264.objecter _session_command_op_assign 0 2 2024-08-19T12:38:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.523+0000 7f2f47d6b640 20 client.4264.objecter put_session s=0x7f2f24047490 osd=0 4 2024-08-19T12:38:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.523+0000 7f2f47d6b640 10 client.4264.objecter _send_command 2 2024-08-19T12:38:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.523+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 --> v1:172.21.15.26:6800/3531533167 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2f28001220 con 0x7f2f24047b40 2024-08-19T12:38:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f367fc640 1 -- 172.21.15.26:0/1509160170 <== osd.0 v1:172.21.15.26:6800/3531533167 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3580782990) 0x7f2f28001220 con 0x7f2f24047b40 2024-08-19T12:38:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f367fc640 10 client.4264.objecter ms_dispatch 0x7f2f40000b90 command_reply(tid 2: 0 ) 2024-08-19T12:38:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f367fc640 10 client.4264.objecter _finish_command 2 = system:0 2024-08-19T12:38:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f367fc640 20 client.4264.objecter put_session s=0x7f2f24047490 osd=0 4 2024-08-19T12:38:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f367fc640 15 client.4264.objecter _session_command_op_remove 0 2 2024-08-19T12:38:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f47d6b640 10 librados: watch_flush enter 2024-08-19T12:38:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f47d6b640 10 librados: watch_flush exit 2024-08-19T12:38:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f47d6b640 10 client.4264.objecter close_session for osd.0 2024-08-19T12:38:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 >> v1:172.21.15.26:6800/3531533167 conn(0x7f2f24047b40 legacy=0x7f2f24049f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f47d6b640 20 client.4264.objecter put_session s=0x7f2f24047490 osd=0 1 2024-08-19T12:38:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f47d6b640 20 client.4264.objecter shutdown clearing up homeless session... 2024-08-19T12:38:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f47d6b640 10 client.4264.objecter successfully canceled tick 2024-08-19T12:38:33.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 >> v1:172.21.15.104:6800/899887506 conn(0x7f2f24040f90 legacy=0x7f2f24043450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f47d6b640 10 monclient: shutdown 2024-08-19T12:38:33.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f47d6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 >> v1:172.21.15.104:6792/0 conn(0x7f2f4013c9d0 legacy=0x7f2f401e30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.526+0000 7f2f46308640 1 -- 172.21.15.26:0/1509160170 reap_dead start 2024-08-19T12:38:33.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.527+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 shutdown_connections 2024-08-19T12:38:33.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.527+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 >> 172.21.15.26:0/1509160170 conn(0x7f2f4006ce70 msgr2=0x7f2f400a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.527+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 shutdown_connections 2024-08-19T12:38:33.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.527+0000 7f2f47d6b640 1 -- 172.21.15.26:0/1509160170 wait complete. 2024-08-19T12:38:33.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.527+0000 7f2f47d6b640 1 librados: shutdown 2024-08-19T12:38:33.710 INFO:teuthology.orchestra.run.smithi026.stdout:51539607557 2024-08-19T12:38:33.710 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-19T12:38:33.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2def186640 10 client.4337.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-19T12:38:33.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2def186640 15 client.4337.objecter _session_command_op_assign -1 2 2024-08-19T12:38:33.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2def186640 20 client.4337.objecter _get_session s=0x7f2dcc047410 osd=1 3 2024-08-19T12:38:33.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2def186640 20 client.4337.objecter put_session s=0x7f2dcc047410 osd=1 3 2024-08-19T12:38:33.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2def186640 20 client.4337.objecter _get_session s=0x7f2dcc047410 osd=1 3 2024-08-19T12:38:33.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2def186640 15 client.4337.objecter _session_command_op_remove -1 2 2024-08-19T12:38:33.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2def186640 20 client.4337.objecter get_session s=0x7f2dcc047410 osd=1 3 2024-08-19T12:38:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2def186640 15 client.4337.objecter _session_command_op_assign 1 2 2024-08-19T12:38:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2def186640 20 client.4337.objecter put_session s=0x7f2dcc047410 osd=1 4 2024-08-19T12:38:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2def186640 10 client.4337.objecter _send_command 2 2024-08-19T12:38:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 --> v1:172.21.15.26:6804/2594673920 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2dc8001220 con 0x7f2dcc047ac0 2024-08-19T12:38:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2de57fa640 1 -- 172.21.15.26:0/2860903982 <== osd.1 v1:172.21.15.26:6804/2594673920 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7f2dc8001220 con 0x7f2dcc047ac0 2024-08-19T12:38:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2de57fa640 10 client.4337.objecter ms_dispatch 0x7f2de8000b90 command_reply(tid 2: 0 ) 2024-08-19T12:38:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2de57fa640 10 client.4337.objecter _finish_command 2 = system:0 2024-08-19T12:38:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2de57fa640 20 client.4337.objecter put_session s=0x7f2dcc047410 osd=1 4 2024-08-19T12:38:33.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2de57fa640 15 client.4337.objecter _session_command_op_remove 1 2 2024-08-19T12:38:33.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.533+0000 7f2def186640 10 librados: watch_flush enter 2024-08-19T12:38:33.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 10 librados: watch_flush exit 2024-08-19T12:38:33.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 10 client.4337.objecter close_session for osd.1 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 >> v1:172.21.15.26:6804/2594673920 conn(0x7f2dcc047ac0 legacy=0x7f2dcc049f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 20 client.4337.objecter put_session s=0x7f2dcc047410 osd=1 1 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 20 client.4337.objecter shutdown clearing up homeless session... 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 10 client.4337.objecter successfully canceled tick 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 >> v1:172.21.15.104:6800/899887506 conn(0x7f2dcc040ec0 legacy=0x7f2dcc043380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 10 monclient: shutdown 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 >> v1:172.21.15.26:6793/0 conn(0x7f2de8073470 legacy=0x7f2de81ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2ded723640 1 -- 172.21.15.26:0/2860903982 reap_dead start 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 shutdown_connections 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 >> 172.21.15.26:0/2860903982 conn(0x7f2de80abff0 msgr2=0x7f2de8133a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 shutdown_connections 2024-08-19T12:38:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 1 -- 172.21.15.26:0/2860903982 wait complete. 2024-08-19T12:38:33.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.534+0000 7f2def186640 1 librados: shutdown 2024-08-19T12:38:33.715 INFO:teuthology.orchestra.run.smithi026.stdout:47244640261 2024-08-19T12:38:33.715 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-19T12:38:33.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.539+0000 7fe693cc5640 10 client.4326.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-19T12:38:33.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.539+0000 7fe693cc5640 15 client.4326.objecter _session_command_op_assign -1 2 2024-08-19T12:38:33.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.539+0000 7fe693cc5640 20 client.4326.objecter _get_session s=0x7fe670047c20 osd=5 3 2024-08-19T12:38:33.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.539+0000 7fe693cc5640 20 client.4326.objecter put_session s=0x7fe670047c20 osd=5 3 2024-08-19T12:38:33.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.539+0000 7fe693cc5640 20 client.4326.objecter _get_session s=0x7fe670047c20 osd=5 3 2024-08-19T12:38:33.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.539+0000 7fe693cc5640 15 client.4326.objecter _session_command_op_remove -1 2 2024-08-19T12:38:33.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.539+0000 7fe693cc5640 20 client.4326.objecter get_session s=0x7fe670047c20 osd=5 3 2024-08-19T12:38:33.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.539+0000 7fe693cc5640 15 client.4326.objecter _session_command_op_assign 5 2 2024-08-19T12:38:33.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.539+0000 7fe693cc5640 20 client.4326.objecter put_session s=0x7fe670047c20 osd=5 4 2024-08-19T12:38:33.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.539+0000 7fe693cc5640 10 client.4326.objecter _send_command 2 2024-08-19T12:38:33.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.539+0000 7fe693cc5640 1 -- 172.21.15.26:0/1525925952 --> v1:172.21.15.104:6801/3713183117 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe674001240 con 0x7fe6700482d0 2024-08-19T12:38:33.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe6827fc640 1 -- 172.21.15.26:0/1525925952 <== osd.5 v1:172.21.15.104:6801/3713183117 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7fe674001240 con 0x7fe6700482d0 2024-08-19T12:38:33.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe6827fc640 10 client.4326.objecter ms_dispatch 0x7fe68c000b90 command_reply(tid 2: 0 ) 2024-08-19T12:38:33.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe6827fc640 10 client.4326.objecter _finish_command 2 = system:0 2024-08-19T12:38:33.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe6827fc640 20 client.4326.objecter put_session s=0x7fe670047c20 osd=5 4 2024-08-19T12:38:33.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe6827fc640 15 client.4326.objecter _session_command_op_remove 5 2 2024-08-19T12:38:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 10 librados: watch_flush enter 2024-08-19T12:38:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 10 librados: watch_flush exit 2024-08-19T12:38:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 10 client.4326.objecter close_session for osd.5 2024-08-19T12:38:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 1 -- 172.21.15.26:0/1525925952 >> v1:172.21.15.104:6801/3713183117 conn(0x7fe6700482d0 legacy=0x7fe67004a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 20 client.4326.objecter put_session s=0x7fe670047c20 osd=5 1 2024-08-19T12:38:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 20 client.4326.objecter shutdown clearing up homeless session... 2024-08-19T12:38:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 10 client.4326.objecter successfully canceled tick 2024-08-19T12:38:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 1 -- 172.21.15.26:0/1525925952 >> v1:172.21.15.104:6800/899887506 conn(0x7fe670041720 legacy=0x7fe670043be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 10 monclient: shutdown 2024-08-19T12:38:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 1 -- 172.21.15.26:0/1525925952 >> v1:172.21.15.26:6792/0 conn(0x7fe68c068650 legacy=0x7fe68c133630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe692262640 1 -- 172.21.15.26:0/1525925952 reap_dead start 2024-08-19T12:38:33.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 1 -- 172.21.15.26:0/1525925952 shutdown_connections 2024-08-19T12:38:33.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 1 -- 172.21.15.26:0/1525925952 >> 172.21.15.26:0/1525925952 conn(0x7fe68c0a8160 msgr2=0x7fe68c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 1 -- 172.21.15.26:0/1525925952 shutdown_connections 2024-08-19T12:38:33.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 1 -- 172.21.15.26:0/1525925952 wait complete. 2024-08-19T12:38:33.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.540+0000 7fe693cc5640 1 librados: shutdown 2024-08-19T12:38:33.720 INFO:teuthology.orchestra.run.smithi026.stdout:51539607556 2024-08-19T12:38:33.721 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-19T12:38:33.722 INFO:teuthology.orchestra.run.smithi026.stdout:51539607556 2024-08-19T12:38:33.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-19T12:38:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.863+0000 7f47ecfde640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.863+0000 7fd403de9640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.863+0000 7fd403de9640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.863+0000 7fd403de9640 10 monclient: monmap: 2024-08-19T12:38:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.863978+0000 2024-08-19T12:38:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.863978+0000 2024-08-19T12:38:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.865 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.863+0000 7f47ecfde640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.863+0000 7f47ecfde640 10 monclient: monmap: 2024-08-19T12:38:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.863977+0000 2024-08-19T12:38:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.863977+0000 2024-08-19T12:38:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.867 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.863+0000 7fd403de9640 1 Processor -- start 2024-08-19T12:38:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.863+0000 7f47ecfde640 1 Processor -- start 2024-08-19T12:38:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.863+0000 7f47ecfde640 1 -- start start 2024-08-19T12:38:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.863+0000 7f47ecfde640 10 monclient: init 2024-08-19T12:38:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7fd403de9640 1 -- start start 2024-08-19T12:38:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7fd403de9640 10 monclient: init 2024-08-19T12:38:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7fd403de9640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7fd403de9640 10 monclient: _add_conns ranks=[8,4,1,2,3,5,6,0,7] 2024-08-19T12:38:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7f47ecfde640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7f47ecfde640 10 monclient: _add_conns ranks=[0,7,8,3,4,2,5,1,6] 2024-08-19T12:38:33.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7fd403de9640 10 monclient(hunting): picked mon.noname-e con 0x7fd3fc06e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:33.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7fd403de9640 10 monclient(hunting): picked mon.noname-c con 0x7fd3fc0a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:33.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7fd403de9640 10 monclient(hunting): picked mon.noname-f con 0x7fd3fc0a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:33.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7f47ecfde640 10 monclient(hunting): picked mon.noname-a con 0x7f47e8073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7fd403de9640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3fc136790 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7fd401384640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fd3fc0a5920 0x7fd3fc13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57852/0 (socket says 172.21.15.26:57852) 2024-08-19T12:38:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7f47ecfde640 10 monclient(hunting): picked mon.noname-i con 0x7f47e80a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7f47ecfde640 10 monclient(hunting): picked mon.noname-e con 0x7f47e813f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7f47ecfde640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e809dae0 con 0x7f47e8073470 2024-08-19T12:38:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7f47ecfde640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e8136910 con 0x7f47e80a57a0 2024-08-19T12:38:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7f47ecfde640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e809dd50 con 0x7f47e813f2e0 2024-08-19T12:38:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7f47ecfde640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.864+0000 7f47ecfde640 10 monclient(hunting): authenticate will time out at 707.951835s 2024-08-19T12:38:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.865+0000 7fd401384640 1 -- 172.21.15.26:0/1332909056 learned_addr learned my addr 172.21.15.26:0/1332909056 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.865+0000 7f47e6d9d640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f47e813f2e0 0x7f47e81416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:33976/0 (socket says 172.21.15.26:33976) 2024-08-19T12:38:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.865+0000 7f47e6d9d640 1 -- 172.21.15.26:0/739198350 learned_addr learned my addr 172.21.15.26:0/739198350 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.865+0000 7fd401b85640 1 --1- 172.21.15.26:0/1332909056 >> v1:172.21.15.26:6793/0 conn(0x7fd3fc06e7b0 0x7fd3fc136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:33970/0 (socket says 172.21.15.26:33970) 2024-08-19T12:38:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.865+0000 7fd403de9640 1 -- 172.21.15.26:0/1332909056 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3fc09ddf0 con 0x7fd3fc0a5920 2024-08-19T12:38:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.865+0000 7fd403de9640 1 -- 172.21.15.26:0/1332909056 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3fc09e0d0 con 0x7fd3fc06e7b0 2024-08-19T12:38:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.865+0000 7fd403de9640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.865+0000 7fd403de9640 10 monclient(hunting): authenticate will time out at 707.952223s 2024-08-19T12:38:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd401b85640 1 -- 172.21.15.26:0/1332909056 >> v1:172.21.15.26:6793/0 conn(0x7fd3fc06e7b0 legacy=0x7fd3fc136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd401384640 1 -- 172.21.15.26:0/1332909056 >> v1:172.21.15.26:6791/0 conn(0x7fd3fc0a5920 legacy=0x7fd3fc13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e659c640 1 -- 172.21.15.26:0/739198350 >> v1:172.21.15.26:6789/0 conn(0x7f47e8073470 legacy=0x7f47e8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd402386640 1 -- 172.21.15.26:0/1332909056 >> v1:172.21.15.104:6789/0 conn(0x7fd3fc0a5d50 legacy=0x7fd3fc141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 1 -- 172.21.15.26:0/1332909056 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2928727567 0 0) 0x7fd3ec001450 con 0x7fd3fc0a5920 2024-08-19T12:38:33.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 10 monclient(hunting): my global_id is 4324 2024-08-19T12:38:33.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 1 -- 172.21.15.26:0/1332909056 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3e8003520 con 0x7fd3fc0a5920 2024-08-19T12:38:33.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 1 -- 172.21.15.26:0/1332909056 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1809938490 0 0) 0x7fd3fc09e0d0 con 0x7fd3fc06e7b0 2024-08-19T12:38:33.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 10 monclient(hunting): my global_id is 4355 2024-08-19T12:38:33.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e6d9d640 1 -- 172.21.15.26:0/739198350 >> v1:172.21.15.26:6793/0 conn(0x7f47e813f2e0 legacy=0x7f47e81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 1 -- 172.21.15.26:0/1332909056 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3ec001450 con 0x7fd3fc06e7b0 2024-08-19T12:38:33.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 1 -- 172.21.15.26:0/1332909056 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 262009363 0 0) 0x7fd3e8003520 con 0x7fd3fc0a5920 2024-08-19T12:38:33.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 1 -- 172.21.15.26:0/1332909056 >> v1:172.21.15.26:6793/0 conn(0x7fd3fc06e7b0 legacy=0x7fd3fc136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e5d9b640 1 -- 172.21.15.26:0/739198350 >> v1:172.21.15.104:6792/0 conn(0x7f47e80a57a0 legacy=0x7f47e80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 532848814 0 0) 0x7f47e809dae0 con 0x7f47e8073470 2024-08-19T12:38:33.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 1 -- 172.21.15.26:0/1332909056 >> v1:172.21.15.104:6789/0 conn(0x7fd3fc0a5d50 legacy=0x7fd3fc141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e559a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e559a640 10 monclient(hunting): my global_id is 4275 2024-08-19T12:38:33.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47d0003520 con 0x7f47e8073470 2024-08-19T12:38:33.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2542755234 0 0) 0x7f47e809dd50 con 0x7f47e813f2e0 2024-08-19T12:38:33.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e559a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e559a640 10 monclient(hunting): my global_id is 4364 2024-08-19T12:38:33.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47e809dae0 con 0x7f47e813f2e0 2024-08-19T12:38:33.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 1 monclient: found mon.noname-c 2024-08-19T12:38:33.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:38:33.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1393752088 0 0) 0x7f47e8136910 con 0x7f47e80a57a0 2024-08-19T12:38:33.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 10 monclient(hunting): my global_id is 4273 2024-08-19T12:38:33.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47e809dd50 con 0x7f47e80a57a0 2024-08-19T12:38:33.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 506498144 0 0) 0x7f47e809dae0 con 0x7f47e813f2e0 2024-08-19T12:38:33.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 >> v1:172.21.15.104:6792/0 conn(0x7f47e80a57a0 legacy=0x7f47e80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.867+0000 7fd400b83640 1 -- 172.21.15.26:0/1332909056 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3fc09e3b0 con 0x7fd3fc0a5920 2024-08-19T12:38:33.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd401384640 1 --1- 172.21.15.26:0/1332909056 >> v1:172.21.15.26:6791/0 conn(0x7fd3fc0a5920 0x7fd3fc13e2f0 :-1 s=OPENED pgs=80 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fd3ec004f40 mon_map magic: 0 2024-08-19T12:38:33.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd400b83640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd400b83640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd400b83640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd400b83640 1 -- 172.21.15.26:0/1332909056 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3ec001cb0 con 0x7fd3fc0a5920 2024-08-19T12:38:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd400b83640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd400b83640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd400b83640 10 monclient: dump: 2024-08-19T12:38:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.882 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd400b83640 1 -- 172.21.15.26:0/1332909056 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd3ec004a10 con 0x7fd3fc0a5920 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd400b83640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd403de9640 5 monclient: authenticate success, global_id 4324 2024-08-19T12:38:33.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd403de9640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7fd403de9640 4 set_mon_vals no callback set 2024-08-19T12:38:33.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 >> v1:172.21.15.26:6789/0 conn(0x7f47e8073470 legacy=0x7f47e8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 1 monclient: found mon.noname-e 2024-08-19T12:38:33.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e809e030 con 0x7f47e813f2e0 2024-08-19T12:38:33.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47ecfde640 5 monclient: authenticate success, global_id 4364 2024-08-19T12:38:33.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47ecfde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f47d80019f0 con 0x7f47e813f2e0 2024-08-19T12:38:33.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 10 monclient: dump: 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.888 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.888 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.888 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.888 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.888 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47ecfde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f47d80048a0 con 0x7f47e813f2e0 2024-08-19T12:38:33.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47e559a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47ecfde640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.868+0000 7f47ecfde640 4 set_mon_vals no callback set 2024-08-19T12:38:33.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7fd403de9640 10 monclient: shutdown 2024-08-19T12:38:33.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7fd403de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7f47e559a640 1 -- 172.21.15.26:0/739198350 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f47d8001d40 con 0x7f47e813f2e0 2024-08-19T12:38:33.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7f47e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7f47e559a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:33.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7f47e559a640 10 monclient: dump: 2024-08-19T12:38:33.889 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.889 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.890 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.891 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7fd403de9640 1 -- 172.21.15.26:0/1332909056 >> v1:172.21.15.26:6791/0 conn(0x7fd3fc0a5920 legacy=0x7fd3fc13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7fd400b83640 1 -- 172.21.15.26:0/1332909056 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3ec004f40 con 0x7fd3fc0a5920 2024-08-19T12:38:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7fd400b83640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:38:33.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7fd403de9640 1 -- 172.21.15.26:0/1332909056 shutdown_connections 2024-08-19T12:38:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7fd403de9640 1 -- 172.21.15.26:0/1332909056 >> 172.21.15.26:0/1332909056 conn(0x7fd3fc0a8740 msgr2=0x7fd3fc0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7f47ecfde640 10 monclient: shutdown 2024-08-19T12:38:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7fd403de9640 1 -- 172.21.15.26:0/1332909056 shutdown_connections 2024-08-19T12:38:33.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7fd403de9640 1 -- 172.21.15.26:0/1332909056 wait complete. 2024-08-19T12:38:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7f47ecfde640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7f47ecfde640 1 -- 172.21.15.26:0/739198350 >> v1:172.21.15.26:6793/0 conn(0x7f47e813f2e0 legacy=0x7f47e81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7f47ecfde640 1 -- 172.21.15.26:0/739198350 shutdown_connections 2024-08-19T12:38:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7f47ecfde640 1 -- 172.21.15.26:0/739198350 >> 172.21.15.26:0/739198350 conn(0x7f47e80abff0 msgr2=0x7f47e80ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7f47ecfde640 1 -- 172.21.15.26:0/739198350 shutdown_connections 2024-08-19T12:38:33.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.869+0000 7f47ecfde640 1 -- 172.21.15.26:0/739198350 wait complete. 2024-08-19T12:38:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.870+0000 7fd403de9640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.870+0000 7fd403de9640 10 monclient: monmap: 2024-08-19T12:38:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:33.894 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.870+0000 7fd403de9640 1 librados: starting msgr at 2024-08-19T12:38:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.870+0000 7fd403de9640 1 librados: starting objecter 2024-08-19T12:38:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.870+0000 7fd403de9640 1 Processor -- start 2024-08-19T12:38:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.870+0000 7f47ecfde640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.870+0000 7f47ecfde640 10 monclient: monmap: 2024-08-19T12:38:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.897 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.870+0000 7fd403de9640 1 -- start start 2024-08-19T12:38:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.870+0000 7fd403de9640 1 librados: setting wanted keys 2024-08-19T12:38:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.870+0000 7fd403de9640 1 librados: calling monclient init 2024-08-19T12:38:33.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.870+0000 7fd403de9640 10 monclient: init 2024-08-19T12:38:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 1 librados: starting msgr at 2024-08-19T12:38:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 1 librados: starting objecter 2024-08-19T12:38:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 1 Processor -- start 2024-08-19T12:38:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 1 -- start start 2024-08-19T12:38:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 1 librados: setting wanted keys 2024-08-19T12:38:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 1 librados: calling monclient init 2024-08-19T12:38:33.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 10 monclient: init 2024-08-19T12:38:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7fd403de9640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7fd403de9640 10 monclient: _add_conns ranks=[3,7,8,0,1,2,6,5,4] 2024-08-19T12:38:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7fd403de9640 10 monclient(hunting): picked mon.noname-g con 0x7fd3fc06e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7fd403de9640 10 monclient(hunting): picked mon.noname-i con 0x7fd3fc0a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7fd403de9640 10 monclient(hunting): picked mon.noname-e con 0x7fd3fc0a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7fd403de9640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3fc13b1a0 con 0x7fd3fc06e7b0 2024-08-19T12:38:33.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7fd403de9640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3fc1381c0 con 0x7fd3fc0a5920 2024-08-19T12:38:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7fd403de9640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3fc13aec0 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7fd403de9640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7fd403de9640 10 monclient(hunting): authenticate will time out at 707.958577s 2024-08-19T12:38:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7fd402386640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fd3fc0a5d50 0x7fd3fc1d63b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:33984/0 (socket says 172.21.15.26:33984) 2024-08-19T12:38:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7fd402386640 1 -- 172.21.15.26:0/2490016665 learned_addr learned my addr 172.21.15.26:0/2490016665 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 10 monclient: _add_conns ranks=[6,1,2,3,7,0,8,5,4] 2024-08-19T12:38:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 10 monclient(hunting): picked mon.noname-d con 0x7f47e8073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 10 monclient(hunting): picked mon.noname-f con 0x7f47e80a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 10 monclient(hunting): picked mon.noname-b con 0x7f47e813f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e8132620 con 0x7f47e80a57a0 2024-08-19T12:38:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e8142bc0 con 0x7f47e813f2e0 2024-08-19T12:38:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e81319f0 con 0x7f47e8073470 2024-08-19T12:38:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47ecfde640 10 monclient(hunting): authenticate will time out at 707.958809s 2024-08-19T12:38:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47e6d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f47e813f2e0 0x7f47e81d61c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50316/0 (socket says 172.21.15.26:50316) 2024-08-19T12:38:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.871+0000 7f47e6d9d640 1 -- 172.21.15.26:0/1325725496 learned_addr learned my addr 172.21.15.26:0/1325725496 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.873+0000 7fd402386640 1 -- 172.21.15.26:0/2490016665 >> v1:172.21.15.26:6793/0 conn(0x7fd3fc0a5d50 legacy=0x7fd3fc1d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.873+0000 7fd3f27fc640 10 client.?.objecter ms_handle_connect 0x7fd3fc0a5d50 2024-08-19T12:38:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.873+0000 7fd3f27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.873+0000 7fd401b85640 1 -- 172.21.15.26:0/2490016665 >> v1:172.21.15.104:6790/0 conn(0x7fd3fc06e7b0 legacy=0x7fd3fc1373a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.873+0000 7fd3f27fc640 10 client.?.objecter ms_handle_connect 0x7fd3fc06e7b0 2024-08-19T12:38:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.873+0000 7fd3f27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47e6d9d640 1 -- 172.21.15.26:0/1325725496 >> v1:172.21.15.26:6790/0 conn(0x7f47e813f2e0 legacy=0x7f47e81d61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 client.?.objecter ms_handle_connect 0x7f47e813f2e0 2024-08-19T12:38:33.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.873+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 904379164 0 0) 0x7fd3fc13aec0 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47e659c640 1 -- 172.21.15.26:0/1325725496 >> v1:172.21.15.26:6792/0 conn(0x7f47e8073470 legacy=0x7f47e812dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 client.?.objecter ms_handle_connect 0x7f47e8073470 2024-08-19T12:38:33.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 10 monclient(hunting): my global_id is 4373 2024-08-19T12:38:33.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3e0004eb0 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1019989348 0 0) 0x7fd3fc13b1a0 con 0x7fd3fc06e7b0 2024-08-19T12:38:33.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 10 monclient(hunting): my global_id is 4341 2024-08-19T12:38:33.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3fc13aec0 con 0x7fd3fc06e7b0 2024-08-19T12:38:33.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3782400779 0 0) 0x7fd3e0004eb0 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47e5d9b640 1 -- 172.21.15.26:0/1325725496 >> v1:172.21.15.104:6789/0 conn(0x7f47e80a57a0 legacy=0x7f47e812e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 client.?.objecter ms_handle_connect 0x7f47e80a57a0 2024-08-19T12:38:33.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:33.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3fc13b1a0 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1738852004 0 0) 0x7f47e81319f0 con 0x7f47e8073470 2024-08-19T12:38:33.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): my global_id is 4335 2024-08-19T12:38:33.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47c8004eb0 con 0x7f47e8073470 2024-08-19T12:38:33.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288830077 0 0) 0x7f47e8132620 con 0x7f47e80a57a0 2024-08-19T12:38:33.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3f4003990 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): my global_id is 4267 2024-08-19T12:38:33.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47e81319f0 con 0x7f47e80a57a0 2024-08-19T12:38:33.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 10 monclient(hunting): dump: 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.912 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.912 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.912 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.912 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.912 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:33.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2656196476 0 0) 0x7fd3fc13aec0 con 0x7fd3fc06e7b0 2024-08-19T12:38:33.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3e0004eb0 con 0x7fd3fc06e7b0 2024-08-19T12:38:33.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1215249237 0 0) 0x7f47e809e030 con 0x7f47e813f2e0 2024-08-19T12:38:33.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3f80047e0 con 0x7fd3fc06e7b0 2024-08-19T12:38:33.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:33.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 10 monclient(hunting): dump: 2024-08-19T12:38:33.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.914 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.917 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): my global_id is 4286 2024-08-19T12:38:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3669270973 0 0) 0x7fd3fc13b1a0 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47e8132620 con 0x7f47e813f2e0 2024-08-19T12:38:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 645977593 0 0) 0x7f47e81319f0 con 0x7f47e80a57a0 2024-08-19T12:38:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47e809e030 con 0x7f47e80a57a0 2024-08-19T12:38:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f47dc003610 con 0x7f47e80a57a0 2024-08-19T12:38:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:33.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.874+0000 7f47d6ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.919 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2451573441 0 0) 0x7f47e809e030 con 0x7f47e80a57a0 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 >> v1:172.21.15.26:6792/0 conn(0x7f47e8073470 legacy=0x7f47e812dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 >> v1:172.21.15.26:6790/0 conn(0x7f47e813f2e0 legacy=0x7f47e81d61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 1 monclient: found mon.f 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e812f690 con 0x7f47e80a57a0 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 >> v1:172.21.15.104:6792/0 conn(0x7fd3fc0a5920 legacy=0x7fd3fc137ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 >> v1:172.21.15.104:6790/0 conn(0x7fd3fc06e7b0 legacy=0x7fd3fc1373a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 1 monclient: found mon.e 2024-08-19T12:38:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3fc139c10 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47ecfde640 5 monclient: authenticate success, global_id 4267 2024-08-19T12:38:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47ecfde640 10 monclient: _renew_subs 2024-08-19T12:38:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47ecfde640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:33.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47ecfde640 1 -- 172.21.15.26:0/1325725496 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f47e812eb60 con 0x7f47e80a57a0 2024-08-19T12:38:33.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47ecfde640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47ecfde640 10 monclient: _renew_subs 2024-08-19T12:38:33.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47ecfde640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:33.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47ecfde640 1 -- 172.21.15.26:0/1325725496 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f47e8130460 con 0x7f47e80a57a0 2024-08-19T12:38:33.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47ecfde640 1 librados: init done 2024-08-19T12:38:33.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd3f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f47dc003b10 con 0x7f47e80a57a0 2024-08-19T12:38:33.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f47dc004000 con 0x7f47e80a57a0 2024-08-19T12:38:33.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:33.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47d6ffd640 10 monclient: dump: 2024-08-19T12:38:33.924 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.924 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.924 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.924 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7f47e4d99640 4 set_mon_vals no callback set 2024-08-19T12:38:33.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd403de9640 5 monclient: authenticate success, global_id 4373 2024-08-19T12:38:33.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd403de9640 10 monclient: _renew_subs 2024-08-19T12:38:33.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd403de9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd403de9640 1 -- 172.21.15.26:0/2490016665 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd3fc142910 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd403de9640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:33.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.876+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f47dc0041a0 con 0x7f47e80a57a0 2024-08-19T12:38:33.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd403de9640 10 monclient: _renew_subs 2024-08-19T12:38:33.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd403de9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd403de9640 1 -- 172.21.15.26:0/2490016665 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fd3fc1d7cb0 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.875+0000 7fd403de9640 1 librados: init done 2024-08-19T12:38:33.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.876+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd3f4003e60 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.876+0000 7fd3f27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.876+0000 7fd3f3fff640 4 set_mon_vals no callback set 2024-08-19T12:38:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f47dc04da50 con 0x7f47e80a57a0 2024-08-19T12:38:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7f47d6ffd640 10 client.4267.objecter ms_dispatch 0x7f47e8000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7f47d6ffd640 3 client.4267.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7f47d6ffd640 3 client.4267.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7f47ecfde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7f47ecfde640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7f47ecfde640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7f47d6ffd640 20 client.4267.objecter dump_active .. 0 homeless 2024-08-19T12:38:33.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7f47ecfde640 1 -- 172.21.15.26:0/1325725496 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47cc005080 con 0x7f47e80a57a0 2024-08-19T12:38:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3f4004390 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7fd3f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7fd3f27fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.877+0000 7fd3f27fc640 10 monclient: dump: 2024-08-19T12:38:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.930 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.878+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd3f40045e0 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.878+0000 7f47e659c640 10 monclient: get_auth_request con 0x7f47c80411a0 auth_method 0 2024-08-19T12:38:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.878+0000 7fd403de9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.878+0000 7fd403de9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.878+0000 7fd403de9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.878+0000 7fd403de9640 1 -- 172.21.15.26:0/2490016665 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3e4005080 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.878+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7fd3f404ddb0 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.878+0000 7fd3f27fc640 10 client.4373.objecter ms_dispatch 0x7fd3fc000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.878+0000 7fd3f27fc640 3 client.4373.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.878+0000 7fd3f27fc640 3 client.4373.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.878+0000 7fd3f27fc640 20 client.4373.objecter dump_active .. 0 homeless 2024-08-19T12:38:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.878+0000 7fd401b85640 10 monclient: get_auth_request con 0x7fd3e0041540 auth_method 0 2024-08-19T12:38:33.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.880+0000 7f47d6ffd640 10 client.4267.objecter ms_handle_connect 0x7f47c80411a0 2024-08-19T12:38:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.880+0000 7fd3f27fc640 10 client.4373.objecter ms_handle_connect 0x7fd3e0041540 2024-08-19T12:38:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.882+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 <== mon.1 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f47dc017550 con 0x7f47e80a57a0 2024-08-19T12:38:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.882+0000 7f47d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.882+0000 7f47d6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.882+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fd3f4017340 con 0x7fd3fc0a5d50 2024-08-19T12:38:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.883+0000 7fd3f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:33.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.883+0000 7fd3f27fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:33.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.904+0000 7f6c74b86640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.904+0000 7f6c74b86640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.904+0000 7f6c74b86640 10 monclient: monmap: 2024-08-19T12:38:33.937 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.937 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.937 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.905396+0000 2024-08-19T12:38:33.937 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.905396+0000 2024-08-19T12:38:33.937 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.937 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.937 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.937 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.905+0000 7f6c74b86640 1 Processor -- start 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.905+0000 7f6c74b86640 1 -- start start 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.905+0000 7f6c74b86640 10 monclient: init 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c74b86640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c74b86640 10 monclient: _add_conns ranks=[7,6,0,5,4,3,8,2,1] 2024-08-19T12:38:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c74b86640 10 monclient(hunting): picked mon.noname-i con 0x7f6c70073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c74b86640 10 monclient(hunting): picked mon.noname-d con 0x7f6c700a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c6dd9b640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f6c700a57a0 0x7f6c700a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58516/0 (socket says 172.21.15.26:58516) 2024-08-19T12:38:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c74b86640 10 monclient(hunting): picked mon.noname-a con 0x7f6c7013f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c6e59c640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f6c70073470 0x7f6c70136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:43286/0 (socket says 172.21.15.26:43286) 2024-08-19T12:38:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c74b86640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c7009dae0 con 0x7f6c7013f2e0 2024-08-19T12:38:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c6e59c640 1 -- 172.21.15.26:0/3018649412 learned_addr learned my addr 172.21.15.26:0/3018649412 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c74b86640 1 -- 172.21.15.26:0/3018649412 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c70136910 con 0x7f6c700a57a0 2024-08-19T12:38:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c74b86640 1 -- 172.21.15.26:0/3018649412 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c7009dd50 con 0x7f6c70073470 2024-08-19T12:38:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c74b86640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.906+0000 7f6c74b86640 10 monclient(hunting): authenticate will time out at 707.993537s 2024-08-19T12:38:33.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.907+0000 7f210b5e7640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.907+0000 7f210b5e7640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.907+0000 7f210b5e7640 10 monclient: monmap: 2024-08-19T12:38:33.940 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.940 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.940 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.908462+0000 2024-08-19T12:38:33.940 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.908462+0000 2024-08-19T12:38:33.940 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f210b5e7640 1 Processor -- start 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f210b5e7640 1 -- start start 2024-08-19T12:38:33.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f210b5e7640 10 monclient: init 2024-08-19T12:38:33.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6ed9d640 1 -- 172.21.15.26:0/3018649412 >> v1:172.21.15.26:6789/0 conn(0x7f6c7013f2e0 legacy=0x7f6c701416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6e59c640 1 -- 172.21.15.26:0/3018649412 >> v1:172.21.15.104:6792/0 conn(0x7f6c70073470 legacy=0x7f6c70136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6dd9b640 1 -- 172.21.15.26:0/3018649412 >> v1:172.21.15.26:6792/0 conn(0x7f6c700a57a0 legacy=0x7f6c700a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056977518 0 0) 0x7f6c7009dd50 con 0x7f6c70073470 2024-08-19T12:38:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6d59a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6d59a640 10 monclient(hunting): my global_id is 4282 2024-08-19T12:38:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c58003520 con 0x7f6c70073470 2024-08-19T12:38:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3183661030 0 0) 0x7f6c7009dae0 con 0x7f6c7013f2e0 2024-08-19T12:38:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6d59a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6d59a640 10 monclient(hunting): my global_id is 4284 2024-08-19T12:38:33.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.908+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c580048e0 con 0x7f6c7013f2e0 2024-08-19T12:38:33.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 506262694 0 0) 0x7f6c70136910 con 0x7f6c700a57a0 2024-08-19T12:38:33.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 10 monclient(hunting): my global_id is 4344 2024-08-19T12:38:33.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c7009dae0 con 0x7f6c700a57a0 2024-08-19T12:38:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f210b5e7640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f210b5e7640 10 monclient: _add_conns ranks=[3,2,4,8,0,6,1,5,7] 2024-08-19T12:38:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f210b5e7640 10 monclient(hunting): picked mon.noname-g con 0x7f2104132cf0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f210b5e7640 10 monclient(hunting): picked mon.noname-b con 0x7f210413a0e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f2109383640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f2104132cf0 0x7f21041330d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:39980/0 (socket says 172.21.15.26:39980) 2024-08-19T12:38:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f2108b82640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f210413a0e0 0x7f210413c4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50320/0 (socket says 172.21.15.26:50320) 2024-08-19T12:38:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f210b5e7640 10 monclient(hunting): picked mon.noname-c con 0x7f210413dbf0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f210b5e7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f210409f060 con 0x7f210413a0e0 2024-08-19T12:38:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f210b5e7640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21040773d0 con 0x7f2104132cf0 2024-08-19T12:38:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f210b5e7640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21040a8f20 con 0x7f210413dbf0 2024-08-19T12:38:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f210b5e7640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f2109383640 1 -- 172.21.15.26:0/3429098668 learned_addr learned my addr 172.21.15.26:0/3429098668 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f210b5e7640 10 monclient(hunting): authenticate will time out at 707.996458s 2024-08-19T12:38:33.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6ed9d640 1 --1- 172.21.15.26:0/3018649412 >> v1:172.21.15.26:6789/0 conn(0x7f6c7013f2e0 0x7f6c701416d0 :-1 s=OPENED pgs=90 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f6c68001e10 mon_map magic: 0 2024-08-19T12:38:33.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4193962216 0 0) 0x7f6c58003520 con 0x7f6c70073470 2024-08-19T12:38:33.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 >> v1:172.21.15.26:6792/0 conn(0x7f6c700a57a0 legacy=0x7f6c700a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 >> v1:172.21.15.26:6789/0 conn(0x7f6c7013f2e0 legacy=0x7f6c701416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 1 monclient: found mon.noname-i 2024-08-19T12:38:33.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7009e030 con 0x7f6c70073470 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6c600019f0 con 0x7f6c70073470 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 10 monclient: dump: 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.948 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c74b86640 5 monclient: authenticate success, global_id 4282 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c74b86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6c68001e10 con 0x7f6c7013f2e0 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.909+0000 7f6c6d59a640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:38:33.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.910+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c60001cc0 con 0x7f6c70073470 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.910+0000 7f6c6d59a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.910+0000 7f6c6d59a640 1 -- 172.21.15.26:0/3018649412 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6c60004c20 con 0x7f6c70073470 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.910+0000 7f6c6d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.910+0000 7f6c6d59a640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.910+0000 7f6c6d59a640 10 monclient: dump: 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.950 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.951 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.951 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.951 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.951 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.951 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.951 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.951 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.951 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.910+0000 7f6c74b86640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.910+0000 7f6c74b86640 4 set_mon_vals no callback set 2024-08-19T12:38:33.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.910+0000 7f6c74b86640 10 monclient: shutdown 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.910+0000 7f1018912640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.910+0000 7f1018912640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7f1018912640 10 monclient: monmap: 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.911860+0000 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.911860+0000 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.952 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.953 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.953 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.953 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.953 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.953 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7f6c74b86640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7f6c74b86640 1 -- 172.21.15.26:0/3018649412 >> v1:172.21.15.104:6792/0 conn(0x7f6c70073470 legacy=0x7f6c70136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7f6c74b86640 1 -- 172.21.15.26:0/3018649412 shutdown_connections 2024-08-19T12:38:33.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7f6c74b86640 1 -- 172.21.15.26:0/3018649412 >> 172.21.15.26:0/3018649412 conn(0x7f6c700abff0 msgr2=0x7f6c700ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7f1018912640 1 Processor -- start 2024-08-19T12:38:33.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7f2109383640 1 -- 172.21.15.26:0/3429098668 >> v1:172.21.15.104:6790/0 conn(0x7f2104132cf0 legacy=0x7f21041330d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7f1018912640 1 -- start start 2024-08-19T12:38:33.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7f1018912640 10 monclient: init 2024-08-19T12:38:33.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7fe2359c3640 10 monclient: get_monmap_and_config 2024-08-19T12:38:33.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7fe2359c3640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7fe2359c3640 10 monclient: monmap: 2024-08-19T12:38:33.955 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.955 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:33.912653+0000 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:33.912653+0000 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.956 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7f2108b82640 1 -- 172.21.15.26:0/3429098668 >> v1:172.21.15.26:6790/0 conn(0x7f210413a0e0 legacy=0x7f210413c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.911+0000 7f2109b84640 1 -- 172.21.15.26:0/3429098668 >> v1:172.21.15.26:6791/0 conn(0x7f210413dbf0 legacy=0x7f210413ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 1 -- 172.21.15.26:0/3429098668 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1754448793 0 0) 0x7f21040773d0 con 0x7f2104132cf0 2024-08-19T12:38:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient(hunting): my global_id is 4350 2024-08-19T12:38:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 1 -- 172.21.15.26:0/3429098668 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20f0003520 con 0x7f2104132cf0 2024-08-19T12:38:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 1 -- 172.21.15.26:0/3429098668 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2884955627 0 0) 0x7f210409f060 con 0x7f210413a0e0 2024-08-19T12:38:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f1018912640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f1018912640 10 monclient: _add_conns ranks=[0,7,3,8,2,1,6,4,5] 2024-08-19T12:38:33.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f1018912640 10 monclient(hunting): picked mon.noname-a con 0x7f10140a78f0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:33.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f1018912640 10 monclient(hunting): picked mon.noname-i con 0x7f10140a83e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:33.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f1018912640 10 monclient(hunting): picked mon.noname-g con 0x7f101413c8f0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:33.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f1018912640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f101406fe10 con 0x7f10140a78f0 2024-08-19T12:38:33.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient(hunting): my global_id is 4295 2024-08-19T12:38:33.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f1018912640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1014071bd0 con 0x7f101413c8f0 2024-08-19T12:38:33.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f1018912640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1014070010 con 0x7f10140a83e0 2024-08-19T12:38:33.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f1018912640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f1018912640 10 monclient(hunting): authenticate will time out at 707.999572s 2024-08-19T12:38:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 1 -- 172.21.15.26:0/3429098668 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21040773d0 con 0x7f210413a0e0 2024-08-19T12:38:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 1 -- 172.21.15.26:0/3429098668 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3611085332 0 0) 0x7f21040773d0 con 0x7f210413a0e0 2024-08-19T12:38:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 1 -- 172.21.15.26:0/3429098668 >> v1:172.21.15.26:6791/0 conn(0x7f210413dbf0 legacy=0x7f210413ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 1 -- 172.21.15.26:0/3429098668 >> v1:172.21.15.104:6790/0 conn(0x7f2104132cf0 legacy=0x7f21041330d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 1 monclient: found mon.noname-b 2024-08-19T12:38:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 1 -- 172.21.15.26:0/3429098668 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21040a6da0 con 0x7f210413a0e0 2024-08-19T12:38:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 1 -- 172.21.15.26:0/3429098668 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20f4001a40 con 0x7f210413a0e0 2024-08-19T12:38:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient: dump: 2024-08-19T12:38:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.963 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.964 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f210b5e7640 5 monclient: authenticate success, global_id 4295 2024-08-19T12:38:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f210b5e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7fe2359c3640 1 Processor -- start 2024-08-19T12:38:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f1012d9d640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f101413c8f0 0x7f101413ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:39992/0 (socket says 172.21.15.26:39992) 2024-08-19T12:38:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f1012d9d640 1 -- 172.21.15.26:0/1367218869 learned_addr learned my addr 172.21.15.26:0/1367218869 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7fe2359c3640 1 -- start start 2024-08-19T12:38:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7fe2359c3640 10 monclient: init 2024-08-19T12:38:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f6c74b86640 1 -- 172.21.15.26:0/3018649412 shutdown_connections 2024-08-19T12:38:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 1 -- 172.21.15.26:0/3429098668 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20f4001d10 con 0x7f210413a0e0 2024-08-19T12:38:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 1 -- 172.21.15.26:0/3429098668 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20f4004b20 con 0x7f210413a0e0 2024-08-19T12:38:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f20fbfff640 10 monclient: dump: 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.967 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.968 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.968 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.968 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f210b5e7640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.912+0000 7f210b5e7640 4 set_mon_vals no callback set 2024-08-19T12:38:33.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7f6c74b86640 1 -- 172.21.15.26:0/3018649412 wait complete. 2024-08-19T12:38:33.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7fe2359c3640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7fe2359c3640 10 monclient: _add_conns ranks=[2,5,3,4,0,7,1,8,6] 2024-08-19T12:38:33.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7fe2359c3640 10 monclient(hunting): picked mon.noname-b con 0x7fe2300a44d0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:33.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7fe2359c3640 10 monclient(hunting): picked mon.noname-h con 0x7fe2300a4fc0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:33.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7fe2359c3640 10 monclient(hunting): picked mon.noname-g con 0x7fe23013c9d0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:33.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7fe2359c3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe230071350 con 0x7fe2300a44d0 2024-08-19T12:38:33.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7fe2359c3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2300704f0 con 0x7fe23013c9d0 2024-08-19T12:38:33.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7fe2359c3640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe230074310 con 0x7fe2300a4fc0 2024-08-19T12:38:33.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7fe2359c3640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7fe2359c3640 10 monclient(hunting): authenticate will time out at 708.000610s 2024-08-19T12:38:33.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7fe22effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe2300a44d0 0x7fe2300a48b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50326/0 (socket says 172.21.15.26:50326) 2024-08-19T12:38:33.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7fe22effd640 1 -- 172.21.15.26:0/2620129243 learned_addr learned my addr 172.21.15.26:0/2620129243 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7f210b5e7640 10 monclient: shutdown 2024-08-19T12:38:33.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7f210b5e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7f210b5e7640 1 -- 172.21.15.26:0/3429098668 >> v1:172.21.15.26:6790/0 conn(0x7f210413a0e0 legacy=0x7f210413c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7f210b5e7640 1 -- 172.21.15.26:0/3429098668 shutdown_connections 2024-08-19T12:38:33.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.913+0000 7f210b5e7640 1 -- 172.21.15.26:0/3429098668 >> 172.21.15.26:0/3429098668 conn(0x7f21040a4b60 msgr2=0x7f21040a4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.914+0000 7f210b5e7640 1 -- 172.21.15.26:0/3429098668 shutdown_connections 2024-08-19T12:38:33.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.914+0000 7f210b5e7640 1 -- 172.21.15.26:0/3429098668 wait complete. 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.914+0000 7f6c74b86640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.914+0000 7f6c74b86640 10 monclient: monmap: 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.973 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.914+0000 7f210b5e7640 10 monclient: build_initial_monmap 2024-08-19T12:38:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.914+0000 7f210b5e7640 10 monclient: monmap: 2024-08-19T12:38:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:33.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.914+0000 7f6c74b86640 1 librados: starting msgr at 2024-08-19T12:38:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.914+0000 7f6c74b86640 1 librados: starting objecter 2024-08-19T12:38:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101259c640 1 -- 172.21.15.26:0/1367218869 >> v1:172.21.15.26:6789/0 conn(0x7f10140a78f0 legacy=0x7f10140a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f1011d9b640 1 -- 172.21.15.26:0/1367218869 >> v1:172.21.15.104:6792/0 conn(0x7f10140a83e0 legacy=0x7f101413b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f1012d9d640 1 -- 172.21.15.26:0/1367218869 >> v1:172.21.15.104:6790/0 conn(0x7f101413c8f0 legacy=0x7f101413ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f6c74b86640 1 Processor -- start 2024-08-19T12:38:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 830451828 0 0) 0x7f1014071bd0 con 0x7f101413c8f0 2024-08-19T12:38:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient(hunting): my global_id is 4359 2024-08-19T12:38:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ffc003520 con 0x7f101413c8f0 2024-08-19T12:38:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2423526614 0 0) 0x7f1014070010 con 0x7f10140a83e0 2024-08-19T12:38:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient(hunting): my global_id is 4291 2024-08-19T12:38:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1014071bd0 con 0x7f10140a83e0 2024-08-19T12:38:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1345312500 0 0) 0x7f101406fe10 con 0x7f10140a78f0 2024-08-19T12:38:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient(hunting): my global_id is 4293 2024-08-19T12:38:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1014070010 con 0x7f10140a78f0 2024-08-19T12:38:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f6c74b86640 1 -- start start 2024-08-19T12:38:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f6c74b86640 1 librados: setting wanted keys 2024-08-19T12:38:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f6c74b86640 1 librados: calling monclient init 2024-08-19T12:38:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f6c74b86640 10 monclient: init 2024-08-19T12:38:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f210b5e7640 1 librados: starting msgr at 2024-08-19T12:38:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f210b5e7640 1 librados: starting objecter 2024-08-19T12:38:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f210b5e7640 1 Processor -- start 2024-08-19T12:38:33.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 252923943 0 0) 0x7f0ffc003520 con 0x7f101413c8f0 2024-08-19T12:38:33.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 >> v1:172.21.15.104:6792/0 conn(0x7f10140a83e0 legacy=0x7f101413b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f210b5e7640 1 -- start start 2024-08-19T12:38:33.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 >> v1:172.21.15.26:6789/0 conn(0x7f10140a78f0 legacy=0x7f10140a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 monclient: found mon.noname-g 2024-08-19T12:38:33.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f210b5e7640 1 librados: setting wanted keys 2024-08-19T12:38:33.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f210b5e7640 1 librados: calling monclient init 2024-08-19T12:38:33.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f210b5e7640 10 monclient: init 2024-08-19T12:38:33.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:38:33.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10140a5ec0 con 0x7f101413c8f0 2024-08-19T12:38:33.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f100c001e60 con 0x7f101413c8f0 2024-08-19T12:38:33.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f1018912640 5 monclient: authenticate success, global_id 4359 2024-08-19T12:38:33.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:38:33.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient: dump: 2024-08-19T12:38:33.981 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.981 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.982 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.983 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f1018912640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7fe22effd640 1 -- 172.21.15.26:0/2620129243 >> v1:172.21.15.26:6790/0 conn(0x7fe2300a44d0 legacy=0x7fe2300a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f100c004c20 con 0x7f101413c8f0 2024-08-19T12:38:33.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 1 -- 172.21.15.26:0/1367218869 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f100c005110 con 0x7f101413c8f0 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f101159a640 10 monclient: dump: 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f1018912640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.915+0000 7f1018912640 4 set_mon_vals no callback set 2024-08-19T12:38:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22e7fc640 1 -- 172.21.15.26:0/2620129243 >> v1:172.21.15.104:6791/0 conn(0x7fe2300a4fc0 legacy=0x7fe23013b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22f7fe640 1 -- 172.21.15.26:0/2620129243 >> v1:172.21.15.104:6790/0 conn(0x7fe23013c9d0 legacy=0x7fe23013edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 1 -- 172.21.15.26:0/2620129243 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845724806 0 0) 0x7fe230071350 con 0x7fe2300a44d0 2024-08-19T12:38:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient(hunting): my global_id is 4304 2024-08-19T12:38:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 1 -- 172.21.15.26:0/2620129243 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe218003520 con 0x7fe2300a44d0 2024-08-19T12:38:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f6c74b86640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f6c74b86640 10 monclient: _add_conns ranks=[5,0,3,4,7,2,6,8,1] 2024-08-19T12:38:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f6c74b86640 10 monclient(hunting): picked mon.noname-h con 0x7f6c70073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 1 -- 172.21.15.26:0/2620129243 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3015290683 0 0) 0x7fe2300704f0 con 0x7fe23013c9d0 2024-08-19T12:38:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient(hunting): my global_id is 4368 2024-08-19T12:38:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 1 -- 172.21.15.26:0/2620129243 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe230071350 con 0x7fe23013c9d0 2024-08-19T12:38:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f210b5e7640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 1 -- 172.21.15.26:0/2620129243 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2009719426 0 0) 0x7fe218003520 con 0x7fe2300a44d0 2024-08-19T12:38:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 1 -- 172.21.15.26:0/2620129243 >> v1:172.21.15.104:6791/0 conn(0x7fe2300a4fc0 legacy=0x7fe23013b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 1 -- 172.21.15.26:0/2620129243 >> v1:172.21.15.104:6790/0 conn(0x7fe23013c9d0 legacy=0x7fe23013edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 1 monclient: found mon.noname-b 2024-08-19T12:38:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f210b5e7640 10 monclient: _add_conns ranks=[4,2,7,0,1,3,5,6,8] 2024-08-19T12:38:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 1 -- 172.21.15.26:0/2620129243 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2300a5fa0 con 0x7fe2300a44d0 2024-08-19T12:38:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 1 -- 172.21.15.26:0/2620129243 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe220001ae0 con 0x7fe2300a44d0 2024-08-19T12:38:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient: dump: 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.992 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.992 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f6c6dd9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6c700a57a0 0x7f6c700a2660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58432/0 (socket says 172.21.15.26:58432) 2024-08-19T12:38:33.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe2359c3640 5 monclient: authenticate success, global_id 4304 2024-08-19T12:38:33.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe2359c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:33.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f6c6dd9b640 1 -- 172.21.15.26:0/79235612 learned_addr learned my addr 172.21.15.26:0/79235612 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f6c74b86640 10 monclient(hunting): picked mon.noname-a con 0x7f6c700a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:33.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f210b5e7640 10 monclient(hunting): picked mon.noname-c con 0x7f2104132cf0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:33.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f2109383640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f2104132cf0 0x7f210412ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57860/0 (socket says 172.21.15.26:57860) 2024-08-19T12:38:33.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 1 -- 172.21.15.26:0/2620129243 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe220006ab0 con 0x7fe2300a44d0 2024-08-19T12:38:33.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:33.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f2109383640 1 -- 172.21.15.26:0/3448608152 learned_addr learned my addr 172.21.15.26:0/3448608152 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:33.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 1 -- 172.21.15.26:0/2620129243 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe220006fa0 con 0x7fe2300a44d0 2024-08-19T12:38:33.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:33.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f6c74b86640 10 monclient(hunting): picked mon.noname-g con 0x7f6c7013f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe22dffb640 10 monclient: dump: 2024-08-19T12:38:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:33.995 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:33.996 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f1018912640 10 monclient: shutdown 2024-08-19T12:38:33.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f1018912640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe2359c3640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c7013def0 con 0x7f6c700a57a0 2024-08-19T12:38:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f1018912640 1 -- 172.21.15.26:0/1367218869 >> v1:172.21.15.104:6790/0 conn(0x7f101413c8f0 legacy=0x7f101413ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7fe2359c3640 4 set_mon_vals no callback set 2024-08-19T12:38:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c700a5500 con 0x7f6c7013f2e0 2024-08-19T12:38:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.916+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c70142b10 con 0x7f6c70073470 2024-08-19T12:38:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f6c74b86640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f6c74b86640 10 monclient(hunting): authenticate will time out at 708.004057s 2024-08-19T12:38:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f1018912640 1 -- 172.21.15.26:0/1367218869 shutdown_connections 2024-08-19T12:38:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f1018912640 1 -- 172.21.15.26:0/1367218869 >> 172.21.15.26:0/1367218869 conn(0x7f10140a9010 msgr2=0x7f10140a54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f210b5e7640 10 monclient(hunting): picked mon.noname-b con 0x7f210413a0e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f1018912640 1 -- 172.21.15.26:0/1367218869 shutdown_connections 2024-08-19T12:38:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f1018912640 1 -- 172.21.15.26:0/1367218869 wait complete. 2024-08-19T12:38:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f210b5e7640 10 monclient(hunting): picked mon.noname-i con 0x7f210413dbf0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f210413d840 con 0x7f210413a0e0 2024-08-19T12:38:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f210413bd90 con 0x7f2104132cf0 2024-08-19T12:38:34.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2104135e50 con 0x7f210413dbf0 2024-08-19T12:38:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f210b5e7640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7f210b5e7640 10 monclient(hunting): authenticate will time out at 708.004514s 2024-08-19T12:38:34.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7fe2359c3640 10 monclient: shutdown 2024-08-19T12:38:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7fe2359c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7fe2359c3640 1 -- 172.21.15.26:0/2620129243 >> v1:172.21.15.26:6790/0 conn(0x7fe2300a44d0 legacy=0x7fe2300a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7fe2359c3640 1 -- 172.21.15.26:0/2620129243 shutdown_connections 2024-08-19T12:38:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7fe2359c3640 1 -- 172.21.15.26:0/2620129243 >> 172.21.15.26:0/2620129243 conn(0x7fe23006ce70 msgr2=0x7fe2300a55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.917+0000 7fe2359c3640 1 -- 172.21.15.26:0/2620129243 shutdown_connections 2024-08-19T12:38:34.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7fe2359c3640 1 -- 172.21.15.26:0/2620129243 wait complete. 2024-08-19T12:38:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7f1018912640 10 monclient: build_initial_monmap 2024-08-19T12:38:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7f1018912640 10 monclient: monmap: 2024-08-19T12:38:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7f1018912640 1 librados: starting msgr at 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7f1018912640 1 librados: starting objecter 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7f1018912640 1 Processor -- start 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7f1018912640 1 -- start start 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7f1018912640 1 librados: setting wanted keys 2024-08-19T12:38:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7f1018912640 1 librados: calling monclient init 2024-08-19T12:38:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7f1018912640 10 monclient: init 2024-08-19T12:38:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7fe2359c3640 10 monclient: build_initial_monmap 2024-08-19T12:38:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7fe2359c3640 10 monclient: monmap: 2024-08-19T12:38:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:34.006 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7f6c6dd9b640 1 -- 172.21.15.26:0/79235612 >> v1:172.21.15.26:6789/0 conn(0x7f6c700a57a0 legacy=0x7f6c700a2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7f6c5effd640 10 client.?.objecter ms_handle_connect 0x7f6c700a57a0 2024-08-19T12:38:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.918+0000 7f6c5effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f2109383640 1 -- 172.21.15.26:0/3448608152 >> v1:172.21.15.26:6791/0 conn(0x7f2104132cf0 legacy=0x7f210412ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f20f9ffb640 10 client.?.objecter ms_handle_connect 0x7f2104132cf0 2024-08-19T12:38:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f20f9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c6e59c640 1 -- 172.21.15.26:0/79235612 >> v1:172.21.15.104:6791/0 conn(0x7f6c70073470 legacy=0x7f6c700a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 client.?.objecter ms_handle_connect 0x7f6c70073470 2024-08-19T12:38:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1937483919 0 0) 0x7f6c70142b10 con 0x7f6c70073470 2024-08-19T12:38:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 monclient(hunting): my global_id is 4289 2024-08-19T12:38:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c50004ee0 con 0x7f6c70073470 2024-08-19T12:38:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3445885313 0 0) 0x7f6c7013def0 con 0x7f6c700a57a0 2024-08-19T12:38:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 monclient(hunting): my global_id is 4302 2024-08-19T12:38:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c70142b10 con 0x7f6c700a57a0 2024-08-19T12:38:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7fe2359c3640 1 librados: starting msgr at 2024-08-19T12:38:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7fe2359c3640 1 librados: starting objecter 2024-08-19T12:38:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f2108b82640 1 -- 172.21.15.26:0/3448608152 >> v1:172.21.15.26:6790/0 conn(0x7f210413a0e0 legacy=0x7f21041d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f20f9ffb640 10 client.?.objecter ms_handle_connect 0x7f210413a0e0 2024-08-19T12:38:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f20f9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f1018912640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f1018912640 10 monclient: _add_conns ranks=[7,3,0,6,1,5,2,8,4] 2024-08-19T12:38:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7fe2359c3640 1 Processor -- start 2024-08-19T12:38:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f1018912640 10 monclient(hunting): picked mon.noname-i con 0x7f10140a78f0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f2109b84640 1 -- 172.21.15.26:0/3448608152 >> v1:172.21.15.104:6792/0 conn(0x7f210413dbf0 legacy=0x7f21041d69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1735618580 0 0) 0x7f210413d840 con 0x7f210413a0e0 2024-08-19T12:38:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f20f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f1018912640 10 monclient(hunting): picked mon.noname-g con 0x7f10140a83e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f20f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f20f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f20f9ffb640 10 monclient(hunting): my global_id is 4313 2024-08-19T12:38:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20e8004eb0 con 0x7f210413a0e0 2024-08-19T12:38:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f20f9ffb640 10 client.?.objecter ms_handle_connect 0x7f210413dbf0 2024-08-19T12:38:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f20f9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1009445921 0 0) 0x7f6c50004ee0 con 0x7f6c70073470 2024-08-19T12:38:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c7013def0 con 0x7f6c70073470 2024-08-19T12:38:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6c60004b20 con 0x7f6c70073470 2024-08-19T12:38:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:38:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 monclient(hunting): dump: 2024-08-19T12:38:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:34.014 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:34.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f101259c640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f10140a78f0 0x7f10141c9a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:43312/0 (socket says 172.21.15.26:43312) 2024-08-19T12:38:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2354189832 0 0) 0x7f6c70142b10 con 0x7f6c700a57a0 2024-08-19T12:38:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7fe2359c3640 1 -- start start 2024-08-19T12:38:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7fe2359c3640 1 librados: setting wanted keys 2024-08-19T12:38:34.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.919+0000 7f1018912640 10 monclient(hunting): picked mon.noname-a con 0x7f101413c8f0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f1012d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f101413c8f0 0x7f10141d1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58444/0 (socket says 172.21.15.26:58444) 2024-08-19T12:38:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f1012d9d640 1 -- 172.21.15.26:0/4111316388 learned_addr learned my addr 172.21.15.26:0/4111316388 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f1011d9b640 1 --1- 172.21.15.26:0/4111316388 >> v1:172.21.15.104:6790/0 conn(0x7f10140a83e0 0x7f10141ca170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:40024/0 (socket says 172.21.15.26:40024) 2024-08-19T12:38:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7fe2359c3640 1 librados: calling monclient init 2024-08-19T12:38:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7fe2359c3640 10 monclient: init 2024-08-19T12:38:34.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10140a4d60 con 0x7f101413c8f0 2024-08-19T12:38:34.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10140a73c0 con 0x7f10140a83e0 2024-08-19T12:38:34.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1014125b90 con 0x7f10140a78f0 2024-08-19T12:38:34.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f1018912640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:34.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f1018912640 10 monclient(hunting): authenticate will time out at 708.007156s 2024-08-19T12:38:34.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1901183913 0 0) 0x7f20e8004eb0 con 0x7f210413a0e0 2024-08-19T12:38:34.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f210413d840 con 0x7f210413a0e0 2024-08-19T12:38:34.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20f40019c0 con 0x7f210413a0e0 2024-08-19T12:38:34.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:34.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:34.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:34.018 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:34.018 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:34.018 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.018 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.018 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2209925104 0 0) 0x7f2104135e50 con 0x7f210413dbf0 2024-08-19T12:38:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient(hunting): my global_id is 4300 2024-08-19T12:38:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20e8004eb0 con 0x7f210413dbf0 2024-08-19T12:38:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6c50004ee0 con 0x7f6c700a57a0 2024-08-19T12:38:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6c640047e0 con 0x7f6c700a57a0 2024-08-19T12:38:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 10 monclient(hunting): dump: 2024-08-19T12:38:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3857227530 0 0) 0x7f6c7013def0 con 0x7f6c70073470 2024-08-19T12:38:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 >> v1:172.21.15.104:6790/0 conn(0x7f6c7013f2e0 legacy=0x7f6c701d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 >> v1:172.21.15.26:6789/0 conn(0x7f6c700a57a0 legacy=0x7f6c700a2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 1 monclient: found mon.h 2024-08-19T12:38:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c70136bc0 con 0x7f6c70073470 2024-08-19T12:38:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 10 monclient: _finish_auth 0 2024-08-19T12:38:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c74b86640 5 monclient: authenticate success, global_id 4289 2024-08-19T12:38:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1396851786 0 0) 0x7f210413d840 con 0x7f210413a0e0 2024-08-19T12:38:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c74b86640 10 monclient: _renew_subs 2024-08-19T12:38:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 >> v1:172.21.15.104:6792/0 conn(0x7f210413dbf0 legacy=0x7f21041d69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c74b86640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 >> v1:172.21.15.26:6791/0 conn(0x7f2104132cf0 legacy=0x7f210412ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 1 monclient: found mon.b 2024-08-19T12:38:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6c700a3a40 con 0x7f6c70073470 2024-08-19T12:38:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2104135570 con 0x7f210413a0e0 2024-08-19T12:38:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c74b86640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c74b86640 10 monclient: _renew_subs 2024-08-19T12:38:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c74b86640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6c60001a10 con 0x7f6c70073470 2024-08-19T12:38:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f6c700a4840 con 0x7f6c70073470 2024-08-19T12:38:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c74b86640 1 librados: init done 2024-08-19T12:38:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6c60005510 con 0x7f6c70073470 2024-08-19T12:38:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f210b5e7640 5 monclient: authenticate success, global_id 4313 2024-08-19T12:38:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20f40050d0 con 0x7f210413a0e0 2024-08-19T12:38:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c6cd99640 4 set_mon_vals no callback set 2024-08-19T12:38:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f6c5effd640 10 monclient: dump: 2024-08-19T12:38:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:34.029 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7fe2359c3640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7fe2359c3640 10 monclient: _add_conns ranks=[3,6,4,8,7,0,1,5,2] 2024-08-19T12:38:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7fe2359c3640 10 monclient(hunting): picked mon.noname-g con 0x7fe2300a44d0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7fe22effd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fe2300a44d0 0x7fe23013b800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:40028/0 (socket says 172.21.15.26:40028) 2024-08-19T12:38:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7fe22e7fc640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fe2300a4fc0 0x7fe2301ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58524/0 (socket says 172.21.15.26:58524) 2024-08-19T12:38:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7fe22effd640 1 -- 172.21.15.26:0/1203463265 learned_addr learned my addr 172.21.15.26:0/1203463265 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20f4004be0 con 0x7f210413a0e0 2024-08-19T12:38:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7fe2359c3640 10 monclient(hunting): picked mon.noname-d con 0x7fe2300a4fc0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20f9ffb640 10 monclient: dump: 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:34.033 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7fe2359c3640 10 monclient(hunting): picked mon.noname-c con 0x7fe23013c9d0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.920+0000 7f20fb7fe640 4 set_mon_vals no callback set 2024-08-19T12:38:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe230125320 con 0x7fe2300a44d0 2024-08-19T12:38:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe23013bf10 con 0x7fe23013c9d0 2024-08-19T12:38:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe230126190 con 0x7fe2300a4fc0 2024-08-19T12:38:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7fe2359c3640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7fe2359c3640 10 monclient(hunting): authenticate will time out at 708.008684s 2024-08-19T12:38:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7f210b5e7640 10 monclient: _renew_subs 2024-08-19T12:38:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7f210b5e7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f210413c8f0 con 0x7f210413a0e0 2024-08-19T12:38:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6c60011970 con 0x7f6c70073470 2024-08-19T12:38:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7f210b5e7640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7f210b5e7640 10 monclient: _renew_subs 2024-08-19T12:38:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7f210b5e7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7f6c74b86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f6c74b86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f6c74b86640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f210413b4d0 con 0x7f210413a0e0 2024-08-19T12:38:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 --> v1:172.21.15.104:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c54005080 con 0x7f6c70073470 2024-08-19T12:38:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.921+0000 7f210b5e7640 1 librados: init done 2024-08-19T12:38:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f1011d9b640 1 -- 172.21.15.26:0/4111316388 >> v1:172.21.15.104:6790/0 conn(0x7f10140a83e0 legacy=0x7f10141ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:38:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f101259c640 1 -- 172.21.15.26:0/4111316388 >> v1:172.21.15.104:6792/0 conn(0x7f10140a78f0 legacy=0x7f10141c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f1012d9d640 1 -- 172.21.15.26:0/4111316388 >> v1:172.21.15.26:6789/0 conn(0x7f101413c8f0 legacy=0x7f10141d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:34.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f6c6dd9b640 10 monclient: get_auth_request con 0x7f6c50041400 auth_method 0 2024-08-19T12:38:34.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f6c6004d8f0 con 0x7f6c70073470 2024-08-19T12:38:34.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f6c5effd640 10 client.4289.objecter ms_dispatch 0x7f6c70000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:34.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f6c5effd640 3 client.4289.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:34.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f6c5effd640 3 client.4289.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:34.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f6c5effd640 20 client.4289.objecter dump_active .. 0 homeless 2024-08-19T12:38:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f20f4011880 con 0x7f210413a0e0 2024-08-19T12:38:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f210b5e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f210b5e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f210b5e7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.922+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f210413b860 con 0x7f210413a0e0 2024-08-19T12:38:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f20f404da00 con 0x7f210413a0e0 2024-08-19T12:38:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7f20f9ffb640 10 client.4313.objecter ms_dispatch 0x7f2104000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7f20f9ffb640 3 client.4313.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7f20f9ffb640 3 client.4313.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7f20f9ffb640 20 client.4313.objecter dump_active .. 0 homeless 2024-08-19T12:38:34.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7f2109383640 10 monclient: get_auth_request con 0x7f20e8040f40 auth_method 0 2024-08-19T12:38:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe22effd640 1 -- 172.21.15.26:0/1203463265 >> v1:172.21.15.104:6790/0 conn(0x7fe2300a44d0 legacy=0x7fe23013b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe22e7fc640 1 -- 172.21.15.26:0/1203463265 >> v1:172.21.15.26:6792/0 conn(0x7fe2300a4fc0 legacy=0x7fe2301ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 client.?.objecter ms_handle_connect 0x7fe2300a44d0 2024-08-19T12:38:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 client.?.objecter ms_handle_connect 0x7fe2300a4fc0 2024-08-19T12:38:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3648634063 0 0) 0x7fe230126190 con 0x7fe2300a4fc0 2024-08-19T12:38:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 monclient(hunting): my global_id is 4353 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe21c004eb0 con 0x7fe2300a4fc0 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 228983682 0 0) 0x7fe230125320 con 0x7fe2300a44d0 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 monclient(hunting): my global_id is 4386 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe230126190 con 0x7fe2300a44d0 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 client.?.objecter ms_handle_connect 0x7fe23013c9d0 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2359851149 0 0) 0x7fe21c004eb0 con 0x7fe2300a4fc0 2024-08-19T12:38:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe230125320 con 0x7fe2300a4fc0 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe22f7fe640 1 -- 172.21.15.26:0/1203463265 >> v1:172.21.15.26:6791/0 conn(0x7fe23013c9d0 legacy=0x7fe2301d24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe224001c80 con 0x7fe2300a4fc0 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 monclient(hunting): dump: 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.923+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1388520138 0 0) 0x7fe230126190 con 0x7fe2300a44d0 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe21c004eb0 con 0x7fe2300a44d0 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe220001a60 con 0x7fe2300a44d0 2024-08-19T12:38:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 10 monclient(hunting): dump: 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 963094048 0 0) 0x7fe230125320 con 0x7fe2300a4fc0 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 >> v1:172.21.15.26:6791/0 conn(0x7fe23013c9d0 legacy=0x7fe2301d24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 >> v1:172.21.15.104:6790/0 conn(0x7fe2300a44d0 legacy=0x7fe23013b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 1 monclient: found mon.d 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2300a7c30 con 0x7fe2300a4fc0 2024-08-19T12:38:34.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 10 monclient: _finish_auth 0 2024-08-19T12:38:34.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:34.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:34.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe2359c3640 5 monclient: authenticate success, global_id 4353 2024-08-19T12:38:34.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe2359c3640 10 monclient: _renew_subs 2024-08-19T12:38:34.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe2240039a0 con 0x7fe2300a4fc0 2024-08-19T12:38:34.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe2359c3640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:34.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe230068e20 con 0x7fe2300a4fc0 2024-08-19T12:38:34.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 client.?.objecter ms_handle_connect 0x7f10140a83e0 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 client.?.objecter ms_handle_connect 0x7f10140a78f0 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 client.?.objecter ms_handle_connect 0x7f101413c8f0 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3954369374 0 0) 0x7f1008001480 con 0x7f10140a83e0 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): my global_id is 4377 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ff4004ee0 con 0x7f10140a83e0 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308423477 0 0) 0x7f10140a5ec0 con 0x7f101413c8f0 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:34.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): my global_id is 4311 2024-08-19T12:38:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1008001480 con 0x7f101413c8f0 2024-08-19T12:38:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1613504678 0 0) 0x7f1014125b90 con 0x7f10140a78f0 2024-08-19T12:38:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): my global_id is 4309 2024-08-19T12:38:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10140a5ec0 con 0x7f10140a78f0 2024-08-19T12:38:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3228278904 0 0) 0x7f1008001480 con 0x7f101413c8f0 2024-08-19T12:38:34.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1014125b90 con 0x7f101413c8f0 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe224003ed0 con 0x7fe2300a4fc0 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe217fff640 10 monclient: dump: 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:34.052 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f6c5effd640 10 client.4289.objecter ms_handle_connect 0x7f6c50041400 2024-08-19T12:38:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 157269795 0 0) 0x7f0ff4004ee0 con 0x7f10140a83e0 2024-08-19T12:38:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1008001480 con 0x7f10140a83e0 2024-08-19T12:38:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1008003950 con 0x7f10140a83e0 2024-08-19T12:38:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:38:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f100c001de0 con 0x7f101413c8f0 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7f1002ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2277417623 0 0) 0x7f1014125b90 con 0x7f101413c8f0 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 >> v1:172.21.15.104:6792/0 conn(0x7f10140a78f0 legacy=0x7f10141c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 >> v1:172.21.15.104:6790/0 conn(0x7f10140a83e0 legacy=0x7f10141ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 1 monclient: found mon.a 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10141ca880 con 0x7f101413c8f0 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1018912640 5 monclient: authenticate success, global_id 4311 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1018912640 10 monclient: _renew_subs 2024-08-19T12:38:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1018912640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1014126320 con 0x7f101413c8f0 2024-08-19T12:38:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.924+0000 7fe22d7fa640 4 set_mon_vals no callback set 2024-08-19T12:38:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1018912640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1018912640 10 monclient: _renew_subs 2024-08-19T12:38:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1018912640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1014125000 con 0x7f101413c8f0 2024-08-19T12:38:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1018912640 1 librados: init done 2024-08-19T12:38:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f100c0020b0 con 0x7f101413c8f0 2024-08-19T12:38:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f100c005980 con 0x7f101413c8f0 2024-08-19T12:38:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7f1010d99640 4 set_mon_vals no callback set 2024-08-19T12:38:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7fe2359c3640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7fe2359c3640 10 monclient: _renew_subs 2024-08-19T12:38:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7fe2359c3640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe2301ca8e0 con 0x7fe2300a4fc0 2024-08-19T12:38:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7fe2359c3640 1 librados: init done 2024-08-19T12:38:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.925+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe2240041a0 con 0x7fe2300a4fc0 2024-08-19T12:38:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7f20f9ffb640 10 client.4313.objecter ms_handle_connect 0x7f20e8040f40 2024-08-19T12:38:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7f1002ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7f1002ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7f1002ffd640 10 monclient: dump: 2024-08-19T12:38:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:34.062 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:34.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7fe22404c960 con 0x7fe2300a4fc0 2024-08-19T12:38:34.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7fe217fff640 10 client.4353.objecter ms_dispatch 0x7fe230000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:34.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7fe217fff640 3 client.4353.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:34.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7fe217fff640 3 client.4353.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:34.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7fe217fff640 20 client.4353.objecter dump_active .. 0 homeless 2024-08-19T12:38:34.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7fe2359c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:34.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7fe2359c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:34.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7fe2359c3640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:34.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2300a69a0 con 0x7fe2300a4fc0 2024-08-19T12:38:34.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7fe22effd640 10 monclient: get_auth_request con 0x7fe21c049d10 auth_method 0 2024-08-19T12:38:34.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.926+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f100c011de0 con 0x7f101413c8f0 2024-08-19T12:38:34.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.927+0000 7f1018912640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.927+0000 7f1018912640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.927+0000 7f1018912640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.927+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ff8005080 con 0x7f101413c8f0 2024-08-19T12:38:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.927+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f100c04dd70 con 0x7f101413c8f0 2024-08-19T12:38:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.927+0000 7f1002ffd640 10 client.4311.objecter ms_dispatch 0x7f1014000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.927+0000 7f1002ffd640 3 client.4311.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.927+0000 7f1002ffd640 3 client.4311.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.927+0000 7f1002ffd640 20 client.4311.objecter dump_active .. 0 homeless 2024-08-19T12:38:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.927+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.5 v1:172.21.15.104:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f6c60017450 con 0x7f6c70073470 2024-08-19T12:38:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.927+0000 7f6c5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:34.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.927+0000 7f6c5effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:34.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.929+0000 7f101259c640 10 monclient: get_auth_request con 0x7f0ff40417b0 auth_method 0 2024-08-19T12:38:34.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.929+0000 7fe217fff640 10 client.4353.objecter ms_handle_connect 0x7fe21c049d10 2024-08-19T12:38:34.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.931+0000 7f1002ffd640 10 client.4311.objecter ms_handle_connect 0x7f0ff40417b0 2024-08-19T12:38:34.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.931+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fe2240175a0 con 0x7fe2300a4fc0 2024-08-19T12:38:34.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.931+0000 7fe217fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:34.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.931+0000 7fe217fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:34.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.932+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f20f4017500 con 0x7f210413a0e0 2024-08-19T12:38:34.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.932+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f10140a4d60 con 0x7f101413c8f0 2024-08-19T12:38:34.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.932+0000 7f20f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:34.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.932+0000 7f1002ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:34.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.932+0000 7f1002ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:34.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:33.932+0000 7f20f9ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:34.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7f1018912640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T12:38:34.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7f1018912640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T12:38:34.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7f1018912640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:34.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f0ff8005370 con 0x7f101413c8f0 2024-08-19T12:38:34.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7fe2359c3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T12:38:34.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7fe2359c3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T12:38:34.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7fe2359c3640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:34.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fe230071bf0 con 0x7fe2300a4fc0 2024-08-19T12:38:34.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7f1002ffd640 1 -- 172.21.15.26:0/4111316388 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 956180462) 0x7f100c017990 con 0x7f101413c8f0 2024-08-19T12:38:34.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7f1002ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T12:38:34.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7f1002ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:34.172 INFO:teuthology.orchestra.run.smithi026.stdout:51539607555 2024-08-19T12:38:34.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7fe217fff640 1 -- 172.21.15.26:0/1203463265 <== mon.6 v1:172.21.15.26:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 1444633209) 0x7fe224016ea0 con 0x7fe2300a4fc0 2024-08-19T12:38:34.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7fe217fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T12:38:34.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.169+0000 7fe217fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:34.172 INFO:teuthology.orchestra.run.smithi026.stdout:47244640260 2024-08-19T12:38:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.172+0000 7f1018912640 10 librados: watch_flush enter 2024-08-19T12:38:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.172+0000 7f1018912640 10 librados: watch_flush exit 2024-08-19T12:38:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.172+0000 7f1018912640 20 client.4311.objecter shutdown clearing up homeless session... 2024-08-19T12:38:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.172+0000 7f1018912640 10 client.4311.objecter successfully canceled tick 2024-08-19T12:38:34.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 >> v1:172.21.15.104:6800/899887506 conn(0x7f0ff40417b0 legacy=0x7f0ff4043c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 10 librados: watch_flush enter 2024-08-19T12:38:34.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7f1018912640 10 monclient: shutdown 2024-08-19T12:38:34.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7f1018912640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:34.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 10 librados: watch_flush exit 2024-08-19T12:38:34.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 20 client.4353.objecter shutdown clearing up homeless session... 2024-08-19T12:38:34.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 >> v1:172.21.15.26:6789/0 conn(0x7f101413c8f0 legacy=0x7f10141d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 10 client.4353.objecter successfully canceled tick 2024-08-19T12:38:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 shutdown_connections 2024-08-19T12:38:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 >> 172.21.15.26:0/4111316388 conn(0x7f10140a9010 msgr2=0x7f1014068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 shutdown_connections 2024-08-19T12:38:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 >> v1:172.21.15.104:6800/899887506 conn(0x7fe21c049d10 legacy=0x7fe21c04c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 10 monclient: shutdown 2024-08-19T12:38:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 >> v1:172.21.15.26:6792/0 conn(0x7fe2300a4fc0 legacy=0x7fe2301ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 shutdown_connections 2024-08-19T12:38:34.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 >> 172.21.15.26:0/1203463265 conn(0x7fe23006ce70 msgr2=0x7fe2300a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:34.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7f1018912640 1 -- 172.21.15.26:0/4111316388 wait complete. 2024-08-19T12:38:34.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7f1018912640 1 librados: shutdown 2024-08-19T12:38:34.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 shutdown_connections 2024-08-19T12:38:34.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 1 -- 172.21.15.26:0/1203463265 wait complete. 2024-08-19T12:38:34.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.173+0000 7fe2359c3640 1 librados: shutdown 2024-08-19T12:38:34.189 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.2 2024-08-19T12:38:34.190 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.0 2024-08-19T12:38:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.206+0000 7f47ecfde640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T12:38:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.206+0000 7f47ecfde640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T12:38:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.206+0000 7f47ecfde640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.206+0000 7f47ecfde640 1 -- 172.21.15.26:0/1325725496 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f47cc005370 con 0x7f47e80a57a0 2024-08-19T12:38:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.207+0000 7f47d6ffd640 1 -- 172.21.15.26:0/1325725496 <== mon.1 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 203001719) 0x7f47dc016e50 con 0x7f47e80a57a0 2024-08-19T12:38:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.207+0000 7f47d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T12:38:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.207+0000 7f47d6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:34.208 INFO:teuthology.orchestra.run.smithi026.stdout:51539607556 2024-08-19T12:38:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 10 librados: watch_flush enter 2024-08-19T12:38:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 10 librados: watch_flush exit 2024-08-19T12:38:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 20 client.4267.objecter shutdown clearing up homeless session... 2024-08-19T12:38:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 10 client.4267.objecter successfully canceled tick 2024-08-19T12:38:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 1 -- 172.21.15.26:0/1325725496 >> v1:172.21.15.104:6800/899887506 conn(0x7f47c80411a0 legacy=0x7f47c8043660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 10 monclient: shutdown 2024-08-19T12:38:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 1 -- 172.21.15.26:0/1325725496 >> v1:172.21.15.104:6789/0 conn(0x7f47e80a57a0 legacy=0x7f47e812e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 1 -- 172.21.15.26:0/1325725496 shutdown_connections 2024-08-19T12:38:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 1 -- 172.21.15.26:0/1325725496 >> 172.21.15.26:0/1325725496 conn(0x7f47e80abff0 msgr2=0x7f47e80a9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 1 -- 172.21.15.26:0/1325725496 shutdown_connections 2024-08-19T12:38:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 1 -- 172.21.15.26:0/1325725496 wait complete. 2024-08-19T12:38:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.209+0000 7f47ecfde640 1 librados: shutdown 2024-08-19T12:38:34.220 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.3 2024-08-19T12:38:34.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.270+0000 7f6c74b86640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T12:38:34.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.270+0000 7f6c74b86640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T12:38:34.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.270+0000 7f6c74b86640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:34.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.270+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 --> v1:172.21.15.104:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6c54005370 con 0x7f6c70073470 2024-08-19T12:38:34.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.270+0000 7f6c5effd640 1 -- 172.21.15.26:0/79235612 <== mon.5 v1:172.21.15.104:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 956180462) 0x7f6c60016d50 con 0x7f6c70073470 2024-08-19T12:38:34.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.270+0000 7f6c5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T12:38:34.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.270+0000 7f6c5effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:34.272 INFO:teuthology.orchestra.run.smithi026.stdout:51539607555 2024-08-19T12:38:34.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.272+0000 7f210b5e7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T12:38:34.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.272+0000 7f210b5e7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T12:38:34.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.272+0000 7f210b5e7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:34.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.272+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f2104069660 con 0x7f210413a0e0 2024-08-19T12:38:34.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.272+0000 7f20f9ffb640 1 -- 172.21.15.26:0/3448608152 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 956180462) 0x7f20f4016e00 con 0x7f210413a0e0 2024-08-19T12:38:34.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.272+0000 7f20f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T12:38:34.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.272+0000 7f20f9ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:34.274 INFO:teuthology.orchestra.run.smithi026.stdout:51539607555 2024-08-19T12:38:34.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7fd403de9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T12:38:34.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7fd403de9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T12:38:34.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 10 librados: watch_flush enter 2024-08-19T12:38:34.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7fd403de9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:34.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7fd403de9640 1 -- 172.21.15.26:0/2490016665 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd3e4005370 con 0x7fd3fc0a5d50 2024-08-19T12:38:34.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 10 librados: watch_flush exit 2024-08-19T12:38:34.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 20 client.4289.objecter shutdown clearing up homeless session... 2024-08-19T12:38:34.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 10 client.4289.objecter successfully canceled tick 2024-08-19T12:38:34.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 >> v1:172.21.15.104:6800/899887506 conn(0x7f6c50041400 legacy=0x7f6c500438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 10 monclient: shutdown 2024-08-19T12:38:34.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:34.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 >> v1:172.21.15.104:6791/0 conn(0x7f6c70073470 legacy=0x7f6c700a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 shutdown_connections 2024-08-19T12:38:34.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 >> 172.21.15.26:0/79235612 conn(0x7f6c700abff0 msgr2=0x7f6c70134960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:34.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7fd3f27fc640 1 -- 172.21.15.26:0/2490016665 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 956180462) 0x7fd3f4020280 con 0x7fd3fc0a5d50 2024-08-19T12:38:34.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7fd3f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T12:38:34.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7fd3f27fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:34.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 shutdown_connections 2024-08-19T12:38:34.277 INFO:teuthology.orchestra.run.smithi026.stdout:51539607555 2024-08-19T12:38:34.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 1 -- 172.21.15.26:0/79235612 wait complete. 2024-08-19T12:38:34.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.273+0000 7f6c74b86640 1 librados: shutdown 2024-08-19T12:38:34.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 10 librados: watch_flush enter 2024-08-19T12:38:34.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 10 librados: watch_flush exit 2024-08-19T12:38:34.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 20 client.4313.objecter shutdown clearing up homeless session... 2024-08-19T12:38:34.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 10 client.4313.objecter successfully canceled tick 2024-08-19T12:38:34.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 >> v1:172.21.15.104:6800/899887506 conn(0x7f20e8040f40 legacy=0x7f20e8043400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 10 monclient: shutdown 2024-08-19T12:38:34.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:34.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 >> v1:172.21.15.26:6790/0 conn(0x7f210413a0e0 legacy=0x7f21041d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 shutdown_connections 2024-08-19T12:38:34.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 >> 172.21.15.26:0/3448608152 conn(0x7f21040a4b60 msgr2=0x7f2104131670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:34.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 shutdown_connections 2024-08-19T12:38:34.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 1 -- 172.21.15.26:0/3448608152 wait complete. 2024-08-19T12:38:34.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7fd403de9640 10 librados: watch_flush enter 2024-08-19T12:38:34.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7fd403de9640 10 librados: watch_flush exit 2024-08-19T12:38:34.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7fd403de9640 20 client.4373.objecter shutdown clearing up homeless session... 2024-08-19T12:38:34.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7fd403de9640 10 client.4373.objecter successfully canceled tick 2024-08-19T12:38:34.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7f210b5e7640 1 librados: shutdown 2024-08-19T12:38:34.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7fd403de9640 1 -- 172.21.15.26:0/2490016665 >> v1:172.21.15.104:6800/899887506 conn(0x7fd3e0041540 legacy=0x7fd3e0043a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7fd403de9640 10 monclient: shutdown 2024-08-19T12:38:34.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7fd403de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:34.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7fd403de9640 1 -- 172.21.15.26:0/2490016665 >> v1:172.21.15.26:6793/0 conn(0x7fd3fc0a5d50 legacy=0x7fd3fc1d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:34.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7fd403de9640 1 -- 172.21.15.26:0/2490016665 shutdown_connections 2024-08-19T12:38:34.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7fd403de9640 1 -- 172.21.15.26:0/2490016665 >> 172.21.15.26:0/2490016665 conn(0x7fd3fc0a8740 msgr2=0x7fd3fc0ab050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:34.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7fd403de9640 1 -- 172.21.15.26:0/2490016665 shutdown_connections 2024-08-19T12:38:34.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.275+0000 7fd403de9640 1 -- 172.21.15.26:0/2490016665 wait complete. 2024-08-19T12:38:34.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:34.276+0000 7fd403de9640 1 librados: shutdown 2024-08-19T12:38:34.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.1 2024-08-19T12:38:34.288 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.5 2024-08-19T12:38:34.288 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.4 2024-08-19T12:38:35.190 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-19T12:38:35.190 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-19T12:38:35.222 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-19T12:38:35.287 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-19T12:38:35.288 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-19T12:38:35.289 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-19T12:38:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.310+0000 7ff161bf4640 10 monclient: get_monmap_and_config 2024-08-19T12:38:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.310+0000 7ff161bf4640 10 monclient: build_initial_monmap 2024-08-19T12:38:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.310+0000 7ff161bf4640 10 monclient: monmap: 2024-08-19T12:38:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:35.311117+0000 2024-08-19T12:38:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:35.311117+0000 2024-08-19T12:38:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:35.312 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.312 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:35.312 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:35.312 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:35.312 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:35.312 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:35.312 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:35.312 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:35.312 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:35.312 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:35.312 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.311+0000 7ff161bf4640 1 Processor -- start 2024-08-19T12:38:35.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.311+0000 7ff161bf4640 1 -- start start 2024-08-19T12:38:35.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.311+0000 7ff161bf4640 10 monclient: init 2024-08-19T12:38:35.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.311+0000 7ff161bf4640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:35.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.311+0000 7ff161bf4640 10 monclient: _add_conns ranks=[6,8,1,4,7,2,0,3,5] 2024-08-19T12:38:35.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.311+0000 7ff161bf4640 10 monclient(hunting): picked mon.noname-d con 0x7ff15c073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:35.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.311+0000 7ff161bf4640 10 monclient(hunting): picked mon.noname-e con 0x7ff15c0a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:35.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.311+0000 7ff161bf4640 10 monclient(hunting): picked mon.noname-f con 0x7ff15c13f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:35.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.311+0000 7ff161bf4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff15c09dae0 con 0x7ff15c13f2e0 2024-08-19T12:38:35.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.312+0000 7ff161bf4640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff15c136910 con 0x7ff15c073470 2024-08-19T12:38:35.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.312+0000 7ff15b7fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7ff15c073470 0x7ff15c136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58534/0 (socket says 172.21.15.26:58534) 2024-08-19T12:38:35.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.312+0000 7ff15b7fe640 1 -- 172.21.15.26:0/3120155745 learned_addr learned my addr 172.21.15.26:0/3120155745 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:35.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.312+0000 7ff161bf4640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff15c09dd50 con 0x7ff15c0a57a0 2024-08-19T12:38:35.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.312+0000 7ff161bf4640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:35.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.312+0000 7ff161bf4640 10 monclient(hunting): authenticate will time out at 709.399230s 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.314+0000 7ff15b7fe640 1 -- 172.21.15.26:0/3120155745 >> v1:172.21.15.26:6792/0 conn(0x7ff15c073470 legacy=0x7ff15c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.314+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1031038361 0 0) 0x7ff15c136910 con 0x7ff15c073470 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.314+0000 7ff15affd640 1 -- 172.21.15.26:0/3120155745 >> v1:172.21.15.26:6793/0 conn(0x7ff15c0a57a0 legacy=0x7ff15c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.314+0000 7ff15bfff640 1 -- 172.21.15.26:0/3120155745 >> v1:172.21.15.104:6789/0 conn(0x7ff15c13f2e0 legacy=0x7ff15c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.314+0000 7ff15a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.314+0000 7ff15a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.314+0000 7ff15a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.314+0000 7ff15a7fc640 10 monclient(hunting): my global_id is 4362 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.314+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff144003520 con 0x7ff15c073470 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.314+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1934637022 0 0) 0x7ff15c09dae0 con 0x7ff15c13f2e0 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.314+0000 7ff15a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient(hunting): my global_id is 4276 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff15c136910 con 0x7ff15c13f2e0 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2108158742 0 0) 0x7ff15c09dd50 con 0x7ff15c0a57a0 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient(hunting): my global_id is 4382 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff15c09dae0 con 0x7ff15c0a57a0 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2086998044 0 0) 0x7ff144003520 con 0x7ff15c073470 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 >> v1:172.21.15.26:6793/0 conn(0x7ff15c0a57a0 legacy=0x7ff15c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 >> v1:172.21.15.104:6789/0 conn(0x7ff15c13f2e0 legacy=0x7ff15c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 1 monclient: found mon.noname-d 2024-08-19T12:38:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15c09e030 con 0x7ff15c073470 2024-08-19T12:38:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff14c001ae0 con 0x7ff15c073470 2024-08-19T12:38:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient: dump: 2024-08-19T12:38:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.320 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff161bf4640 5 monclient: authenticate success, global_id 4362 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff161bf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff14c001d30 con 0x7ff15c073470 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 1 -- 172.21.15.26:0/3120155745 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff14c004be0 con 0x7ff15c073470 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff15a7fc640 10 monclient: dump: 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.322 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7fdd8b59e640 10 monclient: get_monmap_and_config 2024-08-19T12:38:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7fdd8b59e640 10 monclient: build_initial_monmap 2024-08-19T12:38:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff161bf4640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7ff161bf4640 4 set_mon_vals no callback set 2024-08-19T12:38:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.315+0000 7fdd8b59e640 10 monclient: monmap: 2024-08-19T12:38:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:35.316531+0000 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:35.316531+0000 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:35.325 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.316+0000 7fdd8b59e640 1 Processor -- start 2024-08-19T12:38:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.316+0000 7ff161bf4640 10 monclient: shutdown 2024-08-19T12:38:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.316+0000 7ff161bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.316+0000 7ff161bf4640 1 -- 172.21.15.26:0/3120155745 >> v1:172.21.15.26:6792/0 conn(0x7ff15c073470 legacy=0x7ff15c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.316+0000 7fdd8b59e640 1 -- start start 2024-08-19T12:38:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.316+0000 7fdd8b59e640 10 monclient: init 2024-08-19T12:38:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.316+0000 7ff161bf4640 1 -- 172.21.15.26:0/3120155745 shutdown_connections 2024-08-19T12:38:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7ff161bf4640 1 -- 172.21.15.26:0/3120155745 >> 172.21.15.26:0/3120155745 conn(0x7ff15c0abff0 msgr2=0x7ff15c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7ff161bf4640 1 -- 172.21.15.26:0/3120155745 shutdown_connections 2024-08-19T12:38:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7ff161bf4640 1 -- 172.21.15.26:0/3120155745 wait complete. 2024-08-19T12:38:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7fdd8b59e640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7fdd8b59e640 10 monclient: _add_conns ranks=[0,7,4,5,1,6,8,3,2] 2024-08-19T12:38:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7fdd8b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fdd8c133a90 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7fdd8b59e640 10 monclient(hunting): picked mon.noname-i con 0x7fdd8c0a17b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7fdd8b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fdd8c13d6d0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7fdd8b59e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8c069540 con 0x7fdd8c133a90 2024-08-19T12:38:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7fdd8b59e640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8c0a3950 con 0x7fdd8c13d6d0 2024-08-19T12:38:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7fdd8b59e640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8c0a2050 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7fdd8b59e640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7fdd8b59e640 10 monclient(hunting): authenticate will time out at 709.404518s 2024-08-19T12:38:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7fdd8ad9d640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fdd8c13d6d0 0x7fdd8c13fac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57882/0 (socket says 172.21.15.26:57882) 2024-08-19T12:38:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.317+0000 7fdd8ad9d640 1 -- 172.21.15.26:0/3938398396 learned_addr learned my addr 172.21.15.26:0/3938398396 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.318+0000 7ff161bf4640 10 monclient: build_initial_monmap 2024-08-19T12:38:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.318+0000 7ff161bf4640 10 monclient: monmap: 2024-08-19T12:38:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.318+0000 7ff161bf4640 1 librados: starting msgr at 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.318+0000 7ff161bf4640 1 librados: starting objecter 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.318+0000 7ff161bf4640 1 Processor -- start 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.318+0000 7ff161bf4640 1 -- start start 2024-08-19T12:38:35.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.318+0000 7ff161bf4640 1 librados: setting wanted keys 2024-08-19T12:38:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.318+0000 7ff161bf4640 1 librados: calling monclient init 2024-08-19T12:38:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.318+0000 7ff161bf4640 10 monclient: init 2024-08-19T12:38:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7ff161bf4640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7ff161bf4640 10 monclient: _add_conns ranks=[6,3,0,4,2,8,5,7,1] 2024-08-19T12:38:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7ff161bf4640 10 monclient(hunting): picked mon.noname-d con 0x7ff15c073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7ff161bf4640 10 monclient(hunting): picked mon.noname-g con 0x7ff15c0a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7ff161bf4640 10 monclient(hunting): picked mon.noname-a con 0x7ff15c13f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7ff161bf4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff15c0a4760 con 0x7ff15c13f2e0 2024-08-19T12:38:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7ff161bf4640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff15c142ba0 con 0x7ff15c0a57a0 2024-08-19T12:38:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7ff15b7fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7ff15c073470 0x7ff15c1ce3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58544/0 (socket says 172.21.15.26:58544) 2024-08-19T12:38:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7ff15b7fe640 1 -- 172.21.15.26:0/3672565718 learned_addr learned my addr 172.21.15.26:0/3672565718 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7fdd8ad9d640 1 -- 172.21.15.26:0/3938398396 >> v1:172.21.15.26:6791/0 conn(0x7fdd8c13d6d0 legacy=0x7fdd8c13fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7ff161bf4640 1 -- 172.21.15.26:0/3672565718 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff15c0a3c00 con 0x7ff15c073470 2024-08-19T12:38:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7ff161bf4640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.319+0000 7ff161bf4640 10 monclient(hunting): authenticate will time out at 709.406950s 2024-08-19T12:38:35.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd81d9b640 1 -- 172.21.15.26:0/3938398396 >> v1:172.21.15.104:6792/0 conn(0x7fdd8c0a17b0 legacy=0x7fdd8c13bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd8a59c640 1 -- 172.21.15.26:0/3938398396 >> v1:172.21.15.26:6789/0 conn(0x7fdd8c133a90 legacy=0x7fdd8c133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 1 -- 172.21.15.26:0/3938398396 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2496068417 0 0) 0x7fdd8c0a3950 con 0x7fdd8c13d6d0 2024-08-19T12:38:35.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 10 monclient(hunting): my global_id is 4360 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 1 -- 172.21.15.26:0/3938398396 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd78003520 con 0x7fdd8c13d6d0 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 1 -- 172.21.15.26:0/3938398396 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4092010744 0 0) 0x7fdd7c0013d0 con 0x7fdd8c133a90 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 10 monclient(hunting): my global_id is 4320 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 1 -- 172.21.15.26:0/3938398396 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd8c0a3950 con 0x7fdd8c133a90 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 1 -- 172.21.15.26:0/3938398396 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2166101713 0 0) 0x7fdd8c0a3950 con 0x7fdd8c133a90 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 1 -- 172.21.15.26:0/3938398396 >> v1:172.21.15.104:6792/0 conn(0x7fdd8c0a17b0 legacy=0x7fdd8c13bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 1 -- 172.21.15.26:0/3938398396 >> v1:172.21.15.26:6791/0 conn(0x7fdd8c13d6d0 legacy=0x7fdd8c13fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 1 monclient: found mon.noname-a 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 1 -- 172.21.15.26:0/3938398396 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8c077370 con 0x7fdd8c133a90 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 10 monclient: _finish_auth 0 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.320+0000 7fdd89d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd8b59e640 5 monclient: authenticate success, global_id 4320 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd8b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd89d9b640 1 -- 172.21.15.26:0/3938398396 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdd7c001c10 con 0x7fdd8c133a90 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd89d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd89d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:38:35.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd89d9b640 10 monclient: dump: 2024-08-19T12:38:35.339 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.339 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.339 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.339 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.339 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.339 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.339 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.339 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.339 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.339 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.339 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.339 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.340 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.340 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.340 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.340 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd8b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:35.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd89d9b640 1 -- 172.21.15.26:0/3938398396 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd7c003650 con 0x7fdd8c133a90 2024-08-19T12:38:35.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff15b7fe640 1 -- 172.21.15.26:0/3672565718 >> v1:172.21.15.26:6792/0 conn(0x7ff15c073470 legacy=0x7ff15c1ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd89d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff15bfff640 1 -- 172.21.15.26:0/3672565718 >> v1:172.21.15.26:6789/0 conn(0x7ff15c13f2e0 legacy=0x7ff15c1d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd89d9b640 1 -- 172.21.15.26:0/3938398396 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdd7c003b40 con 0x7fdd8c133a90 2024-08-19T12:38:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd8b59e640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd8b59e640 4 set_mon_vals no callback set 2024-08-19T12:38:35.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff15affd640 1 -- 172.21.15.26:0/3672565718 >> v1:172.21.15.104:6790/0 conn(0x7ff15c0a57a0 legacy=0x7ff15c1d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd89d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 10 client.?.objecter ms_handle_connect 0x7ff15c073470 2024-08-19T12:38:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 10 client.?.objecter ms_handle_connect 0x7ff15c13f2e0 2024-08-19T12:38:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 10 client.?.objecter ms_handle_connect 0x7ff15c0a57a0 2024-08-19T12:38:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd89d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.321+0000 7fdd89d9b640 10 monclient: dump: 2024-08-19T12:38:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.344 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.344 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1448317532 0 0) 0x7ff15c0a3c00 con 0x7ff15c073470 2024-08-19T12:38:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7fdd8b59e640 10 monclient: shutdown 2024-08-19T12:38:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7fdd8b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3938398396 >> v1:172.21.15.26:6789/0 conn(0x7fdd8c133a90 legacy=0x7fdd8c133e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3938398396 shutdown_connections 2024-08-19T12:38:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3938398396 >> 172.21.15.26:0/3938398396 conn(0x7fdd8c0a53c0 msgr2=0x7fdd8c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3938398396 shutdown_connections 2024-08-19T12:38:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3938398396 wait complete. 2024-08-19T12:38:35.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 10 monclient(hunting): my global_id is 4371 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff148004eb0 con 0x7ff15c073470 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1147554943 0 0) 0x7ff15c142ba0 con 0x7ff15c0a57a0 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.322+0000 7ff143fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 10 monclient(hunting): my global_id is 4395 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff15c0a3c00 con 0x7ff15c0a57a0 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2204600338 0 0) 0x7ff15c0a4760 con 0x7ff15c13f2e0 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 10 monclient(hunting): my global_id is 4329 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff15c142ba0 con 0x7ff15c13f2e0 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3542414445 0 0) 0x7ff148004eb0 con 0x7ff15c073470 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff15c0a4760 con 0x7ff15c073470 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff14c004b20 con 0x7ff15c073470 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 10 monclient(hunting): dump: 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.351 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 693158913 0 0) 0x7ff15c0a3c00 con 0x7ff15c0a57a0 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff148004eb0 con 0x7ff15c0a57a0 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2148804500 0 0) 0x7ff15c0a4760 con 0x7ff15c073470 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 >> v1:172.21.15.104:6790/0 conn(0x7ff15c0a57a0 legacy=0x7ff15c1d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 >> v1:172.21.15.26:6789/0 conn(0x7ff15c13f2e0 legacy=0x7ff15c1d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 monclient: found mon.d 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15c0a5300 con 0x7ff15c073470 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff143fff640 10 monclient: _finish_auth 0 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff143fff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff143fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff14c004cc0 con 0x7ff15c073470 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff143fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff14c005510 con 0x7ff15c073470 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff143fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff143fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:35.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff143fff640 10 monclient: dump: 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.354 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.355 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.355 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.355 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff161bf4640 5 monclient: authenticate success, global_id 4371 2024-08-19T12:38:35.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7fdd8b59e640 10 monclient: build_initial_monmap 2024-08-19T12:38:35.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7fdd8b59e640 10 monclient: monmap: 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:35.357 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7fdd8b59e640 1 librados: starting msgr at 2024-08-19T12:38:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.323+0000 7fdd8b59e640 1 librados: starting objecter 2024-08-19T12:38:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7fdd8b59e640 1 Processor -- start 2024-08-19T12:38:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7fdd8b59e640 1 -- start start 2024-08-19T12:38:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7fdd8b59e640 1 librados: setting wanted keys 2024-08-19T12:38:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7fdd8b59e640 1 librados: calling monclient init 2024-08-19T12:38:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7fdd8b59e640 10 monclient: init 2024-08-19T12:38:35.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff161bf4640 10 monclient: _renew_subs 2024-08-19T12:38:35.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff161bf4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:35.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff161bf4640 1 -- 172.21.15.26:0/3672565718 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff15c137d00 con 0x7ff15c073470 2024-08-19T12:38:35.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7ff159ffb640 4 set_mon_vals no callback set 2024-08-19T12:38:35.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7ff161bf4640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:35.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7ff161bf4640 10 monclient: _renew_subs 2024-08-19T12:38:35.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7ff161bf4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:35.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7ff161bf4640 1 -- 172.21.15.26:0/3672565718 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7ff15c0a3350 con 0x7ff15c073470 2024-08-19T12:38:35.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7fdd8b59e640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:35.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7ff161bf4640 1 librados: init done 2024-08-19T12:38:35.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.324+0000 7fdd8b59e640 10 monclient: _add_conns ranks=[2,6,7,1,5,4,0,8,3] 2024-08-19T12:38:35.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7fdd8b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fdd8c0a17b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:35.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7fdd8b59e640 10 monclient(hunting): picked mon.noname-d con 0x7fdd8c13d6d0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:35.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7fdd8b59e640 10 monclient(hunting): picked mon.noname-i con 0x7fdd8c139520 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:35.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7fdd8b59e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8c13a8c0 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7fdd8b59e640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8c13a4a0 con 0x7fdd8c13d6d0 2024-08-19T12:38:35.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7fdd8b59e640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8c13cb40 con 0x7fdd8c139520 2024-08-19T12:38:35.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7fdd8b59e640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:35.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7fdd8b59e640 10 monclient(hunting): authenticate will time out at 709.412111s 2024-08-19T12:38:35.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7fdd81d9b640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fdd8c13d6d0 0x7fdd8c136e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58556/0 (socket says 172.21.15.26:58556) 2024-08-19T12:38:35.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7fdd81d9b640 1 -- 172.21.15.26:0/3580148894 learned_addr learned my addr 172.21.15.26:0/3580148894 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:35.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.325+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff14c011970 con 0x7ff15c073470 2024-08-19T12:38:35.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.326+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7ff14c04dca0 con 0x7ff15c073470 2024-08-19T12:38:35.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.326+0000 7ff143fff640 10 client.4371.objecter ms_dispatch 0x7ff15c000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:35.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.326+0000 7ff143fff640 3 client.4371.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:35.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.326+0000 7ff143fff640 3 client.4371.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:35.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.326+0000 7ff15affd640 10 monclient: get_auth_request con 0x7ff1480413f0 auth_method 0 2024-08-19T12:38:35.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.326+0000 7ff143fff640 20 client.4371.objecter dump_active .. 0 homeless 2024-08-19T12:38:35.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.326+0000 7ff161bf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.326+0000 7ff161bf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.326+0000 7ff161bf4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:35.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.326+0000 7ff161bf4640 1 -- 172.21.15.26:0/3672565718 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff15c0a93f0 con 0x7ff15c073470 2024-08-19T12:38:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd81d9b640 1 -- 172.21.15.26:0/3580148894 >> v1:172.21.15.26:6792/0 conn(0x7fdd8c13d6d0 legacy=0x7fdd8c136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd837fe640 10 client.?.objecter ms_handle_connect 0x7fdd8c13d6d0 2024-08-19T12:38:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd837fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd8a59c640 1 -- 172.21.15.26:0/3580148894 >> v1:172.21.15.26:6790/0 conn(0x7fdd8c0a17b0 legacy=0x7fdd8c136730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd8ad9d640 1 -- 172.21.15.26:0/3580148894 >> v1:172.21.15.104:6792/0 conn(0x7fdd8c139520 legacy=0x7fdd8c1da910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd837fe640 10 client.?.objecter ms_handle_connect 0x7fdd8c0a17b0 2024-08-19T12:38:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd837fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd837fe640 10 client.?.objecter ms_handle_connect 0x7fdd8c139520 2024-08-19T12:38:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd837fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1053957437 0 0) 0x7fdd8c077370 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd837fe640 10 monclient(hunting): my global_id is 4322 2024-08-19T12:38:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.327+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd6c004eb0 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2477360507 0 0) 0x7fdd6c004eb0 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd8c077370 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdd7c003b40 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:35.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 10 monclient(hunting): dump: 2024-08-19T12:38:35.369 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.369 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.369 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.369 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2888829091 0 0) 0x7fdd8c077370 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 >> v1:172.21.15.104:6792/0 conn(0x7fdd8c139520 legacy=0x7fdd8c1da910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 >> v1:172.21.15.26:6792/0 conn(0x7fdd8c13d6d0 legacy=0x7fdd8c136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 1 monclient: found mon.b 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8c0a72e0 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd8b59e640 5 monclient: authenticate success, global_id 4322 2024-08-19T12:38:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd8b59e640 10 monclient: _renew_subs 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd8b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3580148894 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdd8c0a6dd0 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd8b59e640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd8b59e640 10 monclient: _renew_subs 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd8b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3580148894 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdd8c13a1c0 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd8b59e640 1 librados: init done 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd7c003650 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.328+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdd7c004200 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.329+0000 7fdd837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.329+0000 7fdd837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.329+0000 7fdd837fe640 10 monclient: dump: 2024-08-19T12:38:35.372 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.373 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.374 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.329+0000 7fdd8959a640 4 set_mon_vals no callback set 2024-08-19T12:38:35.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.330+0000 7ff143fff640 10 client.4371.objecter ms_handle_connect 0x7ff1480413f0 2024-08-19T12:38:35.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.330+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fdd7c011a80 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.330+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7fdd7c04d9c0 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.330+0000 7fdd837fe640 10 client.4322.objecter ms_dispatch 0x7fdd8c000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.330+0000 7fdd837fe640 3 client.4322.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.330+0000 7fdd837fe640 3 client.4322.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.330+0000 7fdd8b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.330+0000 7fdd8b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.330+0000 7fdd8b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.330+0000 7fdd837fe640 20 client.4322.objecter dump_active .. 0 homeless 2024-08-19T12:38:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.330+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3580148894 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd70005080 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.331+0000 7fdd81d9b640 10 monclient: get_auth_request con 0x7fdd6c040eb0 auth_method 0 2024-08-19T12:38:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.333+0000 7fdd837fe640 10 client.4322.objecter ms_handle_connect 0x7fdd6c040eb0 2024-08-19T12:38:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.335+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7ff15c0a93f0 con 0x7ff15c073470 2024-08-19T12:38:35.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.335+0000 7ff143fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.335+0000 7ff143fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:35.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.338+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fdd8c13a8c0 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.338+0000 7fdd837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.338+0000 7fdd837fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.364+0000 7f2068638640 10 monclient: get_monmap_and_config 2024-08-19T12:38:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.364+0000 7f2068638640 10 monclient: build_initial_monmap 2024-08-19T12:38:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.364+0000 7f2068638640 10 monclient: monmap: 2024-08-19T12:38:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:35.365436+0000 2024-08-19T12:38:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:35.365436+0000 2024-08-19T12:38:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.365+0000 7f2068638640 1 Processor -- start 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.365+0000 7f2068638640 1 -- start start 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.365+0000 7f2068638640 10 monclient: init 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.366+0000 7f2068638640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.366+0000 7f2068638640 10 monclient: _add_conns ranks=[6,1,5,2,8,4,7,3,0] 2024-08-19T12:38:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.366+0000 7f2068638640 10 monclient(hunting): picked mon.noname-d con 0x7f2060073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.366+0000 7f2068638640 10 monclient(hunting): picked mon.noname-f con 0x7f20600a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.366+0000 7f2068638640 10 monclient(hunting): picked mon.noname-h con 0x7f206013f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.366+0000 7f2068638640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f206009dae0 con 0x7f20600a57a0 2024-08-19T12:38:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.366+0000 7f2068638640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2060136910 con 0x7f206013f2e0 2024-08-19T12:38:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.366+0000 7f2068638640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f206009dd50 con 0x7f2060073470 2024-08-19T12:38:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.366+0000 7f2068638640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.366+0000 7f2068638640 10 monclient(hunting): authenticate will time out at 709.453256s 2024-08-19T12:38:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.366+0000 7f20663d4640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f2060073470 0x7f2060136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58562/0 (socket says 172.21.15.26:58562) 2024-08-19T12:38:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.366+0000 7f20663d4640 1 -- 172.21.15.26:0/953960038 learned_addr learned my addr 172.21.15.26:0/953960038 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.368+0000 7f20663d4640 1 -- 172.21.15.26:0/953960038 >> v1:172.21.15.26:6792/0 conn(0x7f2060073470 legacy=0x7f2060136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.368+0000 7f2065bd3640 1 -- 172.21.15.26:0/953960038 >> v1:172.21.15.104:6789/0 conn(0x7f20600a57a0 legacy=0x7f20600a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.368+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545557997 0 0) 0x7f206009dd50 con 0x7f2060073470 2024-08-19T12:38:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.368+0000 7f2066bd5640 1 -- 172.21.15.26:0/953960038 >> v1:172.21.15.104:6791/0 conn(0x7f206013f2e0 legacy=0x7f20601416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.368+0000 7f20653d2640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.368+0000 7f20653d2640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.368+0000 7f20653d2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.368+0000 7f20653d2640 10 monclient(hunting): my global_id is 4380 2024-08-19T12:38:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f204c003520 con 0x7f2060073470 2024-08-19T12:38:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1087458952 0 0) 0x7f206009dae0 con 0x7f20600a57a0 2024-08-19T12:38:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 10 monclient(hunting): my global_id is 4285 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f206009dd50 con 0x7f20600a57a0 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1300813570 0 0) 0x7f2060136910 con 0x7f206013f2e0 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 10 monclient(hunting): my global_id is 4298 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.369+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f206009dae0 con 0x7f206013f2e0 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3387402357 0 0) 0x7f204c003520 con 0x7f2060073470 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 >> v1:172.21.15.104:6791/0 conn(0x7f206013f2e0 legacy=0x7f20601416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 >> v1:172.21.15.104:6789/0 conn(0x7f20600a57a0 legacy=0x7f20600a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 10 monclient: _finish_hunting 0 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 1 monclient: found mon.noname-d 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206009e030 con 0x7f2060073470 2024-08-19T12:38:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 10 monclient: _finish_auth 0 2024-08-19T12:38:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 10 monclient: _check_auth_tickets 2024-08-19T12:38:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f2068638640 5 monclient: authenticate success, global_id 4380 2024-08-19T12:38:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f2068638640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f205c001b20 con 0x7f2060073470 2024-08-19T12:38:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.370+0000 7f20653d2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.371+0000 7f20653d2640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.371+0000 7f20653d2640 10 monclient: dump: 2024-08-19T12:38:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.397 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.371+0000 7f2068638640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.371+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f205c0048a0 con 0x7f2060073470 2024-08-19T12:38:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.371+0000 7f20653d2640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.371+0000 7f2068638640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.371+0000 7f2068638640 4 set_mon_vals no callback set 2024-08-19T12:38:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.371+0000 7f20653d2640 1 -- 172.21.15.26:0/953960038 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f205c004e10 con 0x7f2060073470 2024-08-19T12:38:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.371+0000 7f20653d2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.371+0000 7f20653d2640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.371+0000 7f20653d2640 10 monclient: dump: 2024-08-19T12:38:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.399 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.372+0000 7f2068638640 10 monclient: shutdown 2024-08-19T12:38:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.372+0000 7f2068638640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.372+0000 7f2068638640 1 -- 172.21.15.26:0/953960038 >> v1:172.21.15.26:6792/0 conn(0x7f2060073470 legacy=0x7f2060136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.372+0000 7f2068638640 1 -- 172.21.15.26:0/953960038 shutdown_connections 2024-08-19T12:38:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.372+0000 7f2068638640 1 -- 172.21.15.26:0/953960038 >> 172.21.15.26:0/953960038 conn(0x7f20600abff0 msgr2=0x7f20600ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.372+0000 7f2068638640 1 -- 172.21.15.26:0/953960038 shutdown_connections 2024-08-19T12:38:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.372+0000 7f2068638640 1 -- 172.21.15.26:0/953960038 wait complete. 2024-08-19T12:38:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.372+0000 7f2068638640 10 monclient: build_initial_monmap 2024-08-19T12:38:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.372+0000 7f2068638640 10 monclient: monmap: 2024-08-19T12:38:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.373+0000 7f2068638640 1 librados: starting msgr at 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.373+0000 7f2068638640 1 librados: starting objecter 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.373+0000 7f2068638640 1 Processor -- start 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.373+0000 7f2068638640 1 -- start start 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.373+0000 7f2068638640 1 librados: setting wanted keys 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.373+0000 7f2068638640 1 librados: calling monclient init 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.373+0000 7f2068638640 10 monclient: init 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.373+0000 7f2068638640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.373+0000 7f2068638640 10 monclient: _add_conns ranks=[1,6,5,3,8,7,4,2,0] 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.373+0000 7f2068638640 10 monclient(hunting): picked mon.noname-f con 0x7f2060073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.373+0000 7f2068638640 10 monclient(hunting): picked mon.noname-d con 0x7f20600a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.374+0000 7f2068638640 10 monclient(hunting): picked mon.noname-h con 0x7f206013f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.374+0000 7f2068638640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f206012f810 con 0x7f2060073470 2024-08-19T12:38:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.374+0000 7f2068638640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2060135290 con 0x7f206013f2e0 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.374+0000 7f2065bd3640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f20600a57a0 0x7f206012e4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58568/0 (socket says 172.21.15.26:58568) 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.374+0000 7f2065bd3640 1 -- 172.21.15.26:0/4254881291 learned_addr learned my addr 172.21.15.26:0/4254881291 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.374+0000 7f2068638640 1 -- 172.21.15.26:0/4254881291 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2060131b70 con 0x7f20600a57a0 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.374+0000 7f2068638640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.374+0000 7f2068638640 10 monclient(hunting): authenticate will time out at 709.461211s 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2065bd3640 1 -- 172.21.15.26:0/4254881291 >> v1:172.21.15.26:6792/0 conn(0x7f20600a57a0 legacy=0x7f206012e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 client.?.objecter ms_handle_connect 0x7f20600a57a0 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 client.?.objecter ms_handle_connect 0x7f2060073470 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f20663d4640 1 -- 172.21.15.26:0/4254881291 >> v1:172.21.15.104:6789/0 conn(0x7f2060073470 legacy=0x7f206012dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2066bd5640 1 -- 172.21.15.26:0/4254881291 >> v1:172.21.15.104:6791/0 conn(0x7f206013f2e0 legacy=0x7f20601d61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 client.?.objecter ms_handle_connect 0x7f206013f2e0 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3566904765 0 0) 0x7f2060131b70 con 0x7f20600a57a0 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 monclient(hunting): my global_id is 4389 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2044004eb0 con 0x7f20600a57a0 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2913621847 0 0) 0x7f206012f810 con 0x7f2060073470 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 10 monclient(hunting): my global_id is 4294 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2060131b70 con 0x7f2060073470 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.376+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3133551854 0 0) 0x7f206009dae0 con 0x7f206013f2e0 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient(hunting): my global_id is 4307 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f206012f810 con 0x7f206013f2e0 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 172364279 0 0) 0x7f2044004eb0 con 0x7f20600a57a0 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f206009dae0 con 0x7f20600a57a0 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2050003950 con 0x7f20600a57a0 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:35.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2931020767 0 0) 0x7f2060131b70 con 0x7f2060073470 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2044004eb0 con 0x7f2060073470 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f205c001a60 con 0x7f2060073470 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.408 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 560287380 0 0) 0x7f206009dae0 con 0x7f20600a57a0 2024-08-19T12:38:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 >> v1:172.21.15.104:6791/0 conn(0x7f206013f2e0 legacy=0x7f20601d61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 >> v1:172.21.15.104:6789/0 conn(0x7f2060073470 legacy=0x7f206012dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 1 monclient: found mon.d 2024-08-19T12:38:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206006d8f0 con 0x7f20600a57a0 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2056ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2068638640 5 monclient: authenticate success, global_id 4389 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2068638640 10 monclient: _renew_subs 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2068638640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2068638640 1 -- 172.21.15.26:0/4254881291 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2060131890 con 0x7f20600a57a0 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2068638640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2068638640 10 monclient: _renew_subs 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2068638640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2068638640 1 -- 172.21.15.26:0/4254881291 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f2060132300 con 0x7f20600a57a0 2024-08-19T12:38:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.377+0000 7f2068638640 1 librados: init done 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.378+0000 7f2054ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.378+0000 7f2054ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.378+0000 7f2054ff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.378+0000 7f2054ff9640 1 -- 172.21.15.26:0/4254881291 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20600008d0 con 0x7f20600a57a0 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2050001b40 con 0x7f20600a57a0 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2056ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2050004210 con 0x7f20600a57a0 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2056ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2056ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2056ffd640 10 monclient: dump: 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.411 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2050011a80 con 0x7f20600a57a0 2024-08-19T12:38:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2064bd1640 4 set_mon_vals no callback set 2024-08-19T12:38:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f205004dca0 con 0x7f20600a57a0 2024-08-19T12:38:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2056ffd640 10 client.4389.objecter ms_dispatch 0x7f2060000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2056ffd640 3 client.4389.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2056ffd640 3 client.4389.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.380+0000 7f2056ffd640 20 client.4389.objecter dump_active .. 0 homeless 2024-08-19T12:38:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.381+0000 7f20663d4640 10 monclient: get_auth_request con 0x7f20440417b0 auth_method 0 2024-08-19T12:38:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.383+0000 7f2056ffd640 10 client.4389.objecter ms_handle_connect 0x7f20440417b0 2024-08-19T12:38:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.384+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f20500177a0 con 0x7f20600a57a0 2024-08-19T12:38:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.384+0000 7f2056ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.384+0000 7f2056ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:35.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.417+0000 7f69cdd53640 10 monclient: get_monmap_and_config 2024-08-19T12:38:35.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.417+0000 7f69cdd53640 10 monclient: build_initial_monmap 2024-08-19T12:38:35.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.417+0000 7f69cdd53640 10 monclient: monmap: 2024-08-19T12:38:35.418 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:35.418 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.418 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:35.418186+0000 2024-08-19T12:38:35.418 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:35.418186+0000 2024-08-19T12:38:35.418 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:35.418 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.419 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:35.419 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:35.419 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:35.419 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:35.419 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:35.419 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:35.419 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:35.419 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:35.419 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:35.419 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.418+0000 7f69cdd53640 1 Processor -- start 2024-08-19T12:38:35.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.418+0000 7f69cdd53640 1 -- start start 2024-08-19T12:38:35.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.418+0000 7f69cdd53640 10 monclient: init 2024-08-19T12:38:35.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.418+0000 7f69cdd53640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:35.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.418+0000 7f69cdd53640 10 monclient: _add_conns ranks=[1,5,8,2,4,3,7,0,6] 2024-08-19T12:38:35.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.418+0000 7f69cdd53640 10 monclient(hunting): picked mon.noname-f con 0x7f69c8073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:35.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.418+0000 7f69cdd53640 10 monclient(hunting): picked mon.noname-h con 0x7f69c80a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:35.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.418+0000 7f69cdd53640 10 monclient(hunting): picked mon.noname-e con 0x7f69c813f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:35.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.418+0000 7f69cdd53640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c809dae0 con 0x7f69c8073470 2024-08-19T12:38:35.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.418+0000 7f69cdd53640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c8136910 con 0x7f69c80a57a0 2024-08-19T12:38:35.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.419+0000 7f69cdd53640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c809dd50 con 0x7f69c813f2e0 2024-08-19T12:38:35.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.419+0000 7f69cdd53640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:35.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.419+0000 7f69cdd53640 10 monclient(hunting): authenticate will time out at 709.506040s 2024-08-19T12:38:35.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.419+0000 7f69c7fff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f69c813f2e0 0x7f69c81416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34000/0 (socket says 172.21.15.26:34000) 2024-08-19T12:38:35.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.419+0000 7f69c7fff640 1 -- 172.21.15.26:0/3169674109 learned_addr learned my addr 172.21.15.26:0/3169674109 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:35.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.421+0000 7f69c7fff640 1 -- 172.21.15.26:0/3169674109 >> v1:172.21.15.26:6793/0 conn(0x7f69c813f2e0 legacy=0x7f69c81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.421+0000 7f69c77fe640 1 -- 172.21.15.26:0/3169674109 >> v1:172.21.15.104:6789/0 conn(0x7f69c8073470 legacy=0x7f69c8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.421+0000 7f69c6ffd640 1 -- 172.21.15.26:0/3169674109 >> v1:172.21.15.104:6791/0 conn(0x7f69c80a57a0 legacy=0x7f69c80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.421+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1692256612 0 0) 0x7f69c809dd50 con 0x7f69c813f2e0 2024-08-19T12:38:35.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.421+0000 7f69c67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.421+0000 7f69c67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.421+0000 7f69c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.421+0000 7f69c67fc640 10 monclient(hunting): my global_id is 4391 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.421+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69b0003520 con 0x7f69c813f2e0 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2632415808 0 0) 0x7f69c809dae0 con 0x7f69c8073470 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient(hunting): my global_id is 4303 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69c809dd50 con 0x7f69c8073470 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3701526015 0 0) 0x7f69c8136910 con 0x7f69c80a57a0 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient(hunting): my global_id is 4316 2024-08-19T12:38:35.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69c809dae0 con 0x7f69c80a57a0 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3838990119 0 0) 0x7f69b0003520 con 0x7f69c813f2e0 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 >> v1:172.21.15.104:6791/0 conn(0x7f69c80a57a0 legacy=0x7f69c80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 >> v1:172.21.15.104:6789/0 conn(0x7f69c8073470 legacy=0x7f69c8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 1 monclient: found mon.noname-e 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c809e030 con 0x7f69c813f2e0 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f69c0001e60 con 0x7f69c813f2e0 2024-08-19T12:38:35.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient: dump: 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.438 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69c0004c20 con 0x7f69c813f2e0 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 1 -- 172.21.15.26:0/3169674109 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f69c0005150 con 0x7f69c813f2e0 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69c67fc640 10 monclient: dump: 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.439 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69cdd53640 5 monclient: authenticate success, global_id 4391 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69cdd53640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:35.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.422+0000 7f69cdd53640 4 set_mon_vals no callback set 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.423+0000 7f69cdd53640 10 monclient: shutdown 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.423+0000 7f69cdd53640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.423+0000 7f69cdd53640 1 -- 172.21.15.26:0/3169674109 >> v1:172.21.15.26:6793/0 conn(0x7f69c813f2e0 legacy=0x7f69c81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.423+0000 7f69cdd53640 1 -- 172.21.15.26:0/3169674109 shutdown_connections 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.423+0000 7f69cdd53640 1 -- 172.21.15.26:0/3169674109 >> 172.21.15.26:0/3169674109 conn(0x7f69c80abff0 msgr2=0x7f69c80ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.423+0000 7f69cdd53640 1 -- 172.21.15.26:0/3169674109 shutdown_connections 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.423+0000 7f69cdd53640 1 -- 172.21.15.26:0/3169674109 wait complete. 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.424+0000 7f69cdd53640 10 monclient: build_initial_monmap 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.424+0000 7f69cdd53640 10 monclient: monmap: 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:35.441 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.424+0000 7f69cdd53640 1 librados: starting msgr at 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.424+0000 7f69cdd53640 1 librados: starting objecter 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.424+0000 7f69cdd53640 1 Processor -- start 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 1 -- start start 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 1 librados: setting wanted keys 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 1 librados: calling monclient init 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 10 monclient: init 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 10 monclient: _add_conns ranks=[2,8,5,4,7,1,0,3,6] 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 10 monclient(hunting): picked mon.noname-b con 0x7f69c8073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 10 monclient(hunting): picked mon.noname-e con 0x7f69c80a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69c6ffd640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f69c80a57a0 0x7f69c812e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34006/0 (socket says 172.21.15.26:34006) 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69c77fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f69c8073470 0x7f69c812db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50336/0 (socket says 172.21.15.26:50336) 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 10 monclient(hunting): picked mon.noname-h con 0x7f69c813f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c81316f0 con 0x7f69c8073470 2024-08-19T12:38:35.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c813edb0 con 0x7f69c813f2e0 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c812e990 con 0x7f69c80a57a0 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69cdd53640 10 monclient(hunting): authenticate will time out at 709.512890s 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.425+0000 7f69c6ffd640 1 -- 172.21.15.26:0/2399563671 learned_addr learned my addr 172.21.15.26:0/2399563671 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69c6ffd640 1 -- 172.21.15.26:0/2399563671 >> v1:172.21.15.26:6793/0 conn(0x7f69c80a57a0 legacy=0x7f69c812e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69c77fe640 1 -- 172.21.15.26:0/2399563671 >> v1:172.21.15.26:6790/0 conn(0x7f69c8073470 legacy=0x7f69c812db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 client.?.objecter ms_handle_connect 0x7f69c80a57a0 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 client.?.objecter ms_handle_connect 0x7f69c8073470 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1441579512 0 0) 0x7f69c812e990 con 0x7f69c80a57a0 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): my global_id is 4400 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69c7fff640 1 -- 172.21.15.26:0/2399563671 >> v1:172.21.15.104:6791/0 conn(0x7f69c813f2e0 legacy=0x7f69c81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69b4004eb0 con 0x7f69c80a57a0 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 client.?.objecter ms_handle_connect 0x7f69c813f2e0 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3691800312 0 0) 0x7f69c81316f0 con 0x7f69c8073470 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): my global_id is 4331 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69c812e990 con 0x7f69c8073470 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2971379143 0 0) 0x7f69c813edb0 con 0x7f69c813f2e0 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): my global_id is 4325 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69c81316f0 con 0x7f69c813f2e0 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2111861179 0 0) 0x7f69b4004eb0 con 0x7f69c80a57a0 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69c813edb0 con 0x7f69c80a57a0 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f69bc001cc0 con 0x7f69c80a57a0 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): dump: 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.446 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.428+0000 7f69affff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:35.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2825899132 0 0) 0x7f69c81316f0 con 0x7f69c813f2e0 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69b4004eb0 con 0x7f69c813f2e0 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f69c00020b0 con 0x7f69c813f2e0 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 10 monclient(hunting): dump: 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.448 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 157949439 0 0) 0x7f69c812e990 con 0x7f69c8073470 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69c81316f0 con 0x7f69c8073470 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f69b8003610 con 0x7f69c8073470 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 10 monclient(hunting): dump: 2024-08-19T12:38:35.449 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.450 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.450 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.450 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.450 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.450 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.450 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.450 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.450 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.450 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.450 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.450 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.450 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3783530712 0 0) 0x7f69c813edb0 con 0x7f69c80a57a0 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 >> v1:172.21.15.104:6791/0 conn(0x7f69c813f2e0 legacy=0x7f69c81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 >> v1:172.21.15.26:6790/0 conn(0x7f69c8073470 legacy=0x7f69c812db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 1 monclient: found mon.e 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c80aa1b0 con 0x7f69c80a57a0 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 10 monclient: _finish_auth 0 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:35.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69cdd53640 5 monclient: authenticate success, global_id 4400 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69cdd53640 10 monclient: _renew_subs 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69cdd53640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69cdd53640 1 -- 172.21.15.26:0/2399563671 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f69c81301c0 con 0x7f69c80a57a0 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69cdd53640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69cdd53640 10 monclient: _renew_subs 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69cdd53640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69cdd53640 1 -- 172.21.15.26:0/2399563671 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f69c812fef0 con 0x7f69c80a57a0 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.429+0000 7f69cdd53640 1 librados: init done 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69bc003a30 con 0x7f69c80a57a0 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69affff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f69bc003f60 con 0x7f69c80a57a0 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69affff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69c5ffb640 4 set_mon_vals no callback set 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69affff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69affff640 10 monclient: dump: 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.453 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f69bc004180 con 0x7f69c80a57a0 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f69bc04d9e0 con 0x7f69c80a57a0 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69affff640 10 client.4400.objecter ms_dispatch 0x7f69c8000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69affff640 3 client.4400.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69affff640 3 client.4400.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69affff640 20 client.4400.objecter dump_active .. 0 homeless 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.432+0000 7f69c77fe640 10 monclient: get_auth_request con 0x7f69b404a500 auth_method 0 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.431+0000 7f69cdd53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.432+0000 7f69cdd53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.432+0000 7f69cdd53640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.432+0000 7f69cdd53640 1 -- 172.21.15.26:0/2399563671 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69c80a9d10 con 0x7f69c80a57a0 2024-08-19T12:38:35.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.435+0000 7f69affff640 10 client.4400.objecter ms_handle_connect 0x7f69b404a500 2024-08-19T12:38:35.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.437+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f69bc017540 con 0x7f69c80a57a0 2024-08-19T12:38:35.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.437+0000 7f69affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.437+0000 7f69affff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:35.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.449+0000 7f936e027640 10 monclient: get_monmap_and_config 2024-08-19T12:38:35.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.449+0000 7f936e027640 10 monclient: build_initial_monmap 2024-08-19T12:38:35.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.449+0000 7f936e027640 10 monclient: monmap: 2024-08-19T12:38:35.456 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:35.456 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.456 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:35.450762+0000 2024-08-19T12:38:35.456 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:35.450762+0000 2024-08-19T12:38:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:35.457 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.450+0000 7f6bde022640 10 monclient: get_monmap_and_config 2024-08-19T12:38:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.450+0000 7f6bde022640 10 monclient: build_initial_monmap 2024-08-19T12:38:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.450+0000 7f6bde022640 10 monclient: monmap: 2024-08-19T12:38:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:35.451307+0000 2024-08-19T12:38:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:35.451307+0000 2024-08-19T12:38:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.459 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:35.459 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:35.459 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:35.459 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:35.459 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:35.459 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:35.459 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:35.459 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:35.459 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:35.459 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.450+0000 7f936e027640 1 Processor -- start 2024-08-19T12:38:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.450+0000 7f936e027640 1 -- start start 2024-08-19T12:38:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.450+0000 7f936e027640 10 monclient: init 2024-08-19T12:38:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f6bde022640 1 Processor -- start 2024-08-19T12:38:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f6bde022640 1 -- start start 2024-08-19T12:38:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f6bde022640 10 monclient: init 2024-08-19T12:38:35.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f936e027640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:35.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f936e027640 10 monclient: _add_conns ranks=[6,4,5,7,2,8,1,0,3] 2024-08-19T12:38:35.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f936e027640 10 monclient(hunting): picked mon.noname-d con 0x7f9368073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:35.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f936e027640 10 monclient(hunting): picked mon.noname-c con 0x7f93680a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:35.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f936e027640 10 monclient(hunting): picked mon.noname-h con 0x7f936813f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:35.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f936e027640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f936809dae0 con 0x7f93680a57a0 2024-08-19T12:38:35.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f9366ffd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f93680a57a0 0x7f93680a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57894/0 (socket says 172.21.15.26:57894) 2024-08-19T12:38:35.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f9366ffd640 1 -- 172.21.15.26:0/2988427941 learned_addr learned my addr 172.21.15.26:0/2988427941 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:35.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f936e027640 1 -- 172.21.15.26:0/2988427941 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9368136910 con 0x7f936813f2e0 2024-08-19T12:38:35.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f936e027640 1 -- 172.21.15.26:0/2988427941 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f936809dd50 con 0x7f9368073470 2024-08-19T12:38:35.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f936e027640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:35.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.451+0000 7f936e027640 10 monclient(hunting): authenticate will time out at 709.538837s 2024-08-19T12:38:35.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.452+0000 7f6bde022640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:35.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.452+0000 7f6bde022640 10 monclient: _add_conns ranks=[7,6,1,5,2,0,8,3,4] 2024-08-19T12:38:35.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.452+0000 7f6bde022640 10 monclient(hunting): picked mon.noname-i con 0x7f6bd00262e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:35.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.452+0000 7f6bde022640 10 monclient(hunting): picked mon.noname-d con 0x7f6bd0020a40 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:35.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.452+0000 7f6bde022640 10 monclient(hunting): picked mon.noname-f con 0x7f6bd00201f0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:35.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.452+0000 7f6bde022640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd0022390 con 0x7f6bd00201f0 2024-08-19T12:38:35.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.452+0000 7f6bde022640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd0021df0 con 0x7f6bd0020a40 2024-08-19T12:38:35.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.452+0000 7f6bde022640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd0021710 con 0x7f6bd00262e0 2024-08-19T12:38:35.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.452+0000 7f6bde022640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:35.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.452+0000 7f6bde022640 10 monclient(hunting): authenticate will time out at 709.539240s 2024-08-19T12:38:35.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.452+0000 7f6bdc81f640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f6bd0020a40 0x7f6bd00be3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58586/0 (socket says 172.21.15.26:58586) 2024-08-19T12:38:35.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.452+0000 7f6bdc81f640 1 -- 172.21.15.26:0/2118116888 learned_addr learned my addr 172.21.15.26:0/2118116888 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f9366ffd640 1 -- 172.21.15.26:0/2988427941 >> v1:172.21.15.26:6791/0 conn(0x7f93680a57a0 legacy=0x7f93680a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f9367fff640 1 -- 172.21.15.26:0/2988427941 >> v1:172.21.15.104:6791/0 conn(0x7f936813f2e0 legacy=0x7f93681416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93677fe640 1 -- 172.21.15.26:0/2988427941 >> v1:172.21.15.26:6792/0 conn(0x7f9368073470 legacy=0x7f9368136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2417465002 0 0) 0x7f9368136910 con 0x7f936813f2e0 2024-08-19T12:38:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f6bdc81f640 1 -- 172.21.15.26:0/2118116888 >> v1:172.21.15.26:6792/0 conn(0x7f6bd0020a40 legacy=0x7f6bd00be3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient(hunting): my global_id is 4334 2024-08-19T12:38:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9350003520 con 0x7f936813f2e0 2024-08-19T12:38:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3731312207 0 0) 0x7f936809dae0 con 0x7f93680a57a0 2024-08-19T12:38:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient(hunting): my global_id is 4369 2024-08-19T12:38:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9368136910 con 0x7f93680a57a0 2024-08-19T12:38:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4246190111 0 0) 0x7f936809dd50 con 0x7f9368073470 2024-08-19T12:38:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient(hunting): my global_id is 4398 2024-08-19T12:38:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f936809dae0 con 0x7f9368073470 2024-08-19T12:38:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f6bdd020640 1 -- 172.21.15.26:0/2118116888 >> v1:172.21.15.104:6792/0 conn(0x7f6bd00262e0 legacy=0x7f6bd00266c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f6bdd821640 1 -- 172.21.15.26:0/2118116888 >> v1:172.21.15.104:6789/0 conn(0x7f6bd00201f0 legacy=0x7f6bd00c1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2407905476 0 0) 0x7f9350003520 con 0x7f936813f2e0 2024-08-19T12:38:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 >> v1:172.21.15.26:6792/0 conn(0x7f9368073470 legacy=0x7f9368136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 >> v1:172.21.15.26:6791/0 conn(0x7f93680a57a0 legacy=0x7f93680a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 1 monclient: found mon.noname-h 2024-08-19T12:38:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:38:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f936809e030 con 0x7f936813f2e0 2024-08-19T12:38:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9360001eb0 con 0x7f936813f2e0 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f93667fc640 10 monclient: dump: 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.471 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f936e027640 5 monclient: authenticate success, global_id 4334 2024-08-19T12:38:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f936e027640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f6bcffff640 1 -- 172.21.15.26:0/2118116888 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 210612820 0 0) 0x7f6bd0022390 con 0x7f6bd00201f0 2024-08-19T12:38:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f6bcffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f6bcffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f6bcffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f6bcffff640 10 monclient(hunting): my global_id is 4312 2024-08-19T12:38:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f6bcffff640 1 -- 172.21.15.26:0/2118116888 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bc4003520 con 0x7f6bd00201f0 2024-08-19T12:38:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.454+0000 7f9367fff640 1 --1- 172.21.15.26:0/2988427941 >> v1:172.21.15.104:6791/0 conn(0x7f936813f2e0 0x7f93681416d0 :-1 s=OPENED pgs=76 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f9360003e80 mon_map magic: 0 2024-08-19T12:38:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.455+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9360003990 con 0x7f936813f2e0 2024-08-19T12:38:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.455+0000 7f93667fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.455+0000 7f936e027640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.455+0000 7f936e027640 4 set_mon_vals no callback set 2024-08-19T12:38:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f936e027640 10 monclient: shutdown 2024-08-19T12:38:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 1 -- 172.21.15.26:0/2118116888 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2983386627 0 0) 0x7f6bd0021df0 con 0x7f6bd0020a40 2024-08-19T12:38:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient(hunting): my global_id is 4407 2024-08-19T12:38:35.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 1 -- 172.21.15.26:0/2118116888 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bd0022390 con 0x7f6bd0020a40 2024-08-19T12:38:35.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 1 -- 172.21.15.26:0/2118116888 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 621539303 0 0) 0x7f6bc4003520 con 0x7f6bd00201f0 2024-08-19T12:38:35.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 1 -- 172.21.15.26:0/2118116888 >> v1:172.21.15.104:6792/0 conn(0x7f6bd00262e0 legacy=0x7f6bd00266c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 1 -- 172.21.15.26:0/2118116888 >> v1:172.21.15.26:6792/0 conn(0x7f6bd0020a40 legacy=0x7f6bd00be3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f936e027640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:35.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f936e027640 1 -- 172.21.15.26:0/2988427941 >> v1:172.21.15.104:6791/0 conn(0x7f936813f2e0 legacy=0x7f93681416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:35.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 1 monclient: found mon.noname-f 2024-08-19T12:38:35.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:35.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:38:35.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 1 -- 172.21.15.26:0/2118116888 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd0006ee0 con 0x7f6bd00201f0 2024-08-19T12:38:35.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient: _finish_auth 0 2024-08-19T12:38:35.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:35.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:35.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 1 -- 172.21.15.26:0/2118116888 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6bd806bac0 con 0x7f6bd00201f0 2024-08-19T12:38:35.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:38:35.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient: dump: 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.476 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.477 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f93667fc640 1 -- 172.21.15.26:0/2988427941 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9360003e80 con 0x7f936813f2e0 2024-08-19T12:38:35.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f93667fc640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:38:35.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bde022640 5 monclient: authenticate success, global_id 4312 2024-08-19T12:38:35.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bde022640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:35.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f936e027640 1 -- 172.21.15.26:0/2988427941 shutdown_connections 2024-08-19T12:38:35.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f936e027640 1 -- 172.21.15.26:0/2988427941 >> 172.21.15.26:0/2988427941 conn(0x7f93680abff0 msgr2=0x7f93680ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:35.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 1 -- 172.21.15.26:0/2118116888 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6bd8051b00 con 0x7f6bd00201f0 2024-08-19T12:38:35.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:35.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 1 -- 172.21.15.26:0/2118116888 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6bd806c030 con 0x7f6bd00201f0 2024-08-19T12:38:35.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bde022640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:35.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f936e027640 1 -- 172.21.15.26:0/2988427941 shutdown_connections 2024-08-19T12:38:35.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bde022640 4 set_mon_vals no callback set 2024-08-19T12:38:35.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:35.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f6bcffff640 10 monclient: dump: 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.481 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.456+0000 7f936e027640 1 -- 172.21.15.26:0/2988427941 wait complete. 2024-08-19T12:38:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f936e027640 10 monclient: build_initial_monmap 2024-08-19T12:38:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f6bde022640 10 monclient: shutdown 2024-08-19T12:38:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f6bde022640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f6bde022640 1 -- 172.21.15.26:0/2118116888 >> v1:172.21.15.104:6789/0 conn(0x7f6bd00201f0 legacy=0x7f6bd00c1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f936e027640 10 monclient: monmap: 2024-08-19T12:38:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:35.483 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f6bde022640 1 -- 172.21.15.26:0/2118116888 shutdown_connections 2024-08-19T12:38:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f936e027640 1 librados: starting msgr at 2024-08-19T12:38:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f6bde022640 1 -- 172.21.15.26:0/2118116888 >> 172.21.15.26:0/2118116888 conn(0x7f6bd0007970 msgr2=0x7f6bd0007d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f936e027640 1 librados: starting objecter 2024-08-19T12:38:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f936e027640 1 Processor -- start 2024-08-19T12:38:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f6bde022640 1 -- 172.21.15.26:0/2118116888 shutdown_connections 2024-08-19T12:38:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f6bde022640 1 -- 172.21.15.26:0/2118116888 wait complete. 2024-08-19T12:38:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f936e027640 1 -- start start 2024-08-19T12:38:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f936e027640 1 librados: setting wanted keys 2024-08-19T12:38:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f936e027640 1 librados: calling monclient init 2024-08-19T12:38:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.457+0000 7f936e027640 10 monclient: init 2024-08-19T12:38:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f936e027640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f936e027640 10 monclient: _add_conns ranks=[8,3,4,2,6,1,7,5,0] 2024-08-19T12:38:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f936e027640 10 monclient(hunting): picked mon.noname-e con 0x7f9368073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f936e027640 10 monclient(hunting): picked mon.noname-g con 0x7f93680a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f936e027640 10 monclient(hunting): picked mon.noname-c con 0x7f936813f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f936e027640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9368136f30 con 0x7f93680a57a0 2024-08-19T12:38:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f936e027640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9368137210 con 0x7f936813f2e0 2024-08-19T12:38:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f936e027640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93681353e0 con 0x7f9368073470 2024-08-19T12:38:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f936e027640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f936e027640 10 monclient(hunting): authenticate will time out at 709.545442s 2024-08-19T12:38:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f93677fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f9368073470 0x7f93680a2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34010/0 (socket says 172.21.15.26:34010) 2024-08-19T12:38:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f93677fe640 1 -- 172.21.15.26:0/271140412 learned_addr learned my addr 172.21.15.26:0/271140412 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f6bde022640 10 monclient: build_initial_monmap 2024-08-19T12:38:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f6bde022640 10 monclient: monmap: 2024-08-19T12:38:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:35.489 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f6bde022640 1 librados: starting msgr at 2024-08-19T12:38:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.458+0000 7f6bde022640 1 librados: starting objecter 2024-08-19T12:38:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 1 Processor -- start 2024-08-19T12:38:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 1 -- start start 2024-08-19T12:38:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 1 librados: setting wanted keys 2024-08-19T12:38:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 1 librados: calling monclient init 2024-08-19T12:38:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 10 monclient: init 2024-08-19T12:38:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 10 monclient: _add_conns ranks=[8,3,7,4,6,1,0,2,5] 2024-08-19T12:38:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 10 monclient(hunting): picked mon.noname-e con 0x7f6bd00201f0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 10 monclient(hunting): picked mon.noname-g con 0x7f6bd0020a40 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 10 monclient(hunting): picked mon.noname-i con 0x7f6bd00262e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd00a7490 con 0x7f6bd0020a40 2024-08-19T12:38:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd00a5130 con 0x7f6bd00262e0 2024-08-19T12:38:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd0029ff0 con 0x7f6bd00201f0 2024-08-19T12:38:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bde022640 10 monclient(hunting): authenticate will time out at 709.546722s 2024-08-19T12:38:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bdd821640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f6bd00262e0 0x7f6bd01555b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:43354/0 (socket says 172.21.15.26:43354) 2024-08-19T12:38:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.459+0000 7f6bdd821640 1 -- 172.21.15.26:0/151073713 learned_addr learned my addr 172.21.15.26:0/151073713 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f93677fe640 1 -- 172.21.15.26:0/271140412 >> v1:172.21.15.26:6793/0 conn(0x7f9368073470 legacy=0x7f93680a2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f9367fff640 1 -- 172.21.15.26:0/271140412 >> v1:172.21.15.26:6791/0 conn(0x7f936813f2e0 legacy=0x7f93681d62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f934ffff640 10 client.?.objecter ms_handle_connect 0x7f9368073470 2024-08-19T12:38:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f934ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f934ffff640 10 client.?.objecter ms_handle_connect 0x7f936813f2e0 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f934ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f9366ffd640 1 -- 172.21.15.26:0/271140412 >> v1:172.21.15.104:6790/0 conn(0x7f93680a57a0 legacy=0x7f93680a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f934ffff640 10 client.?.objecter ms_handle_connect 0x7f93680a57a0 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f934ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1655785369 0 0) 0x7f93681353e0 con 0x7f9368073470 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f934ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f934ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f934ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f934ffff640 10 monclient(hunting): my global_id is 4409 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.460+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9354004ee0 con 0x7f9368073470 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3939347532 0 0) 0x7f9368137210 con 0x7f936813f2e0 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): my global_id is 4378 2024-08-19T12:38:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93681353e0 con 0x7f936813f2e0 2024-08-19T12:38:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.462+0000 7f6bdd821640 1 -- 172.21.15.26:0/151073713 >> v1:172.21.15.104:6792/0 conn(0x7f6bd00262e0 legacy=0x7f6bd01555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.462+0000 7f6bcdffb640 10 client.?.objecter ms_handle_connect 0x7f6bd00262e0 2024-08-19T12:38:35.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404975944 0 0) 0x7f9368136f30 con 0x7f93680a57a0 2024-08-19T12:38:35.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): my global_id is 4404 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9368137210 con 0x7f93680a57a0 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2996509608 0 0) 0x7f9354004ee0 con 0x7f9368073470 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9368136f30 con 0x7f9368073470 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3878202545 0 0) 0x7f9368137210 con 0x7f93680a57a0 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9354004ee0 con 0x7f93680a57a0 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f935c0047e0 con 0x7f93680a57a0 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): dump: 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.497 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.498 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.498 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9358003610 con 0x7f9368073470 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient(hunting): dump: 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.499 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3037509211 0 0) 0x7f9368136f30 con 0x7f9368073470 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 >> v1:172.21.15.26:6791/0 conn(0x7f936813f2e0 legacy=0x7f93681d62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 >> v1:172.21.15.104:6790/0 conn(0x7f93680a57a0 legacy=0x7f93680a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:35.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 monclient: found mon.e 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93680a4570 con 0x7f9368073470 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient: _finish_auth 0 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f934ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f936e027640 5 monclient: authenticate success, global_id 4409 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f936e027640 10 monclient: _renew_subs 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f936e027640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f936e027640 1 -- 172.21.15.26:0/271140412 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9368136c50 con 0x7f9368073470 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f936e027640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f936e027640 10 monclient: _renew_subs 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f936e027640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f936e027640 1 -- 172.21.15.26:0/271140412 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f93680a3290 con 0x7f9368073470 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.461+0000 7f936e027640 1 librados: init done 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.462+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9358003b20 con 0x7f9368073470 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.462+0000 7f934ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.462+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9358004050 con 0x7f9368073470 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.462+0000 7f934ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.462+0000 7f934ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.462+0000 7f934ffff640 10 monclient: dump: 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.502 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.503 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.462+0000 7f6bcdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.462+0000 7f9365ffb640 4 set_mon_vals no callback set 2024-08-19T12:38:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.462+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f93580041f0 con 0x7f9368073470 2024-08-19T12:38:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f935804db40 con 0x7f9368073470 2024-08-19T12:38:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f934ffff640 10 client.4409.objecter ms_dispatch 0x7f9368000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f934ffff640 3 client.4409.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:35.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f934ffff640 3 client.4409.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:35.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f936e027640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f934ffff640 20 client.4409.objecter dump_active .. 0 homeless 2024-08-19T12:38:35.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bdd020640 1 -- 172.21.15.26:0/151073713 >> v1:172.21.15.26:6793/0 conn(0x7f6bd00201f0 legacy=0x7f6bd00a3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 client.?.objecter ms_handle_connect 0x7f6bd00201f0 2024-08-19T12:38:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bdc81f640 1 -- 172.21.15.26:0/151073713 >> v1:172.21.15.104:6790/0 conn(0x7f6bd0020a40 legacy=0x7f6bd00a3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3243142811 0 0) 0x7f6bd0029ff0 con 0x7f6bd00201f0 2024-08-19T12:38:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient(hunting): my global_id is 4418 2024-08-19T12:38:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bbc004eb0 con 0x7f6bd00201f0 2024-08-19T12:38:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 client.?.objecter ms_handle_connect 0x7f6bd0020a40 2024-08-19T12:38:35.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 616216777 0 0) 0x7f6bd00a7490 con 0x7f6bd0020a40 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient(hunting): my global_id is 4413 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bd0029ff0 con 0x7f6bd0020a40 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 843246756 0 0) 0x7f6bbc004eb0 con 0x7f6bd00201f0 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bd00a7490 con 0x7f6bd00201f0 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6bd4001d30 con 0x7f6bd00201f0 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient(hunting): dump: 2024-08-19T12:38:35.507 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.508 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.509 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f936e027640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f936e027640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f936e027640 1 -- 172.21.15.26:0/271140412 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9348005080 con 0x7f9368073470 2024-08-19T12:38:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1887713657 0 0) 0x7f6bd00a7490 con 0x7f6bd00201f0 2024-08-19T12:38:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 >> v1:172.21.15.104:6792/0 conn(0x7f6bd00262e0 legacy=0x7f6bd01555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 >> v1:172.21.15.104:6790/0 conn(0x7f6bd0020a40 legacy=0x7f6bd00a3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.464+0000 7f9366ffd640 10 monclient: get_auth_request con 0x7f9354041560 auth_method 0 2024-08-19T12:38:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:35.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 1 monclient: found mon.e 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd0007ee0 con 0x7f6bd00201f0 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bde022640 5 monclient: authenticate success, global_id 4418 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bde022640 10 monclient: _renew_subs 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bde022640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.463+0000 7f6bde022640 1 -- 172.21.15.26:0/151073713 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd00a6620 con 0x7f6bd00201f0 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.464+0000 7f6bde022640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.464+0000 7f6bde022640 10 monclient: _renew_subs 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.464+0000 7f6bde022640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.464+0000 7f6bde022640 1 -- 172.21.15.26:0/151073713 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f6bd00b9fe0 con 0x7f6bd00201f0 2024-08-19T12:38:35.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.464+0000 7f6bde022640 1 librados: init done 2024-08-19T12:38:35.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.465+0000 7f6bde022640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.465+0000 7f6bde022640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.465+0000 7f6bde022640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.465+0000 7f6bde022640 1 -- 172.21.15.26:0/151073713 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bd001fe40 con 0x7f6bd00201f0 2024-08-19T12:38:35.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.466+0000 7f934ffff640 10 client.4409.objecter ms_handle_connect 0x7f9354041560 2024-08-19T12:38:35.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.473+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6bd4003940 con 0x7f6bd00201f0 2024-08-19T12:38:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.473+0000 7f6bcdffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.473+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6bd4003e30 con 0x7f6bd00201f0 2024-08-19T12:38:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.473+0000 7f6bcdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.473+0000 7f6bcdffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.473+0000 7f6bcdffb640 10 monclient: dump: 2024-08-19T12:38:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:35.516 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.473+0000 7f6bcf7fe640 4 set_mon_vals no callback set 2024-08-19T12:38:35.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.473+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f9358017580 con 0x7f9368073470 2024-08-19T12:38:35.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.473+0000 7f934ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.473+0000 7f934ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.473+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6bd40040d0 con 0x7f6bd00201f0 2024-08-19T12:38:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.474+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f6bd404da20 con 0x7f6bd00201f0 2024-08-19T12:38:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.474+0000 7f6bcdffb640 10 client.4418.objecter ms_dispatch 0x7f6bd8000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.474+0000 7f6bcdffb640 3 client.4418.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.474+0000 7f6bcdffb640 3 client.4418.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.474+0000 7f6bcdffb640 20 client.4418.objecter dump_active .. 0 homeless 2024-08-19T12:38:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.474+0000 7f6bdc81f640 10 monclient: get_auth_request con 0x7f6bbc045520 auth_method 0 2024-08-19T12:38:35.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.477+0000 7f6bcdffb640 10 client.4418.objecter ms_handle_connect 0x7f6bbc045520 2024-08-19T12:38:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.482+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f6bd001fe40 con 0x7f6bd00201f0 2024-08-19T12:38:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.482+0000 7f6bcdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:35.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.482+0000 7f6bcdffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:35.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.587+0000 7f2054ff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T12:38:35.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.587+0000 7f2054ff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T12:38:35.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.587+0000 7f2054ff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:35.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.587+0000 7f2054ff9640 1 -- 172.21.15.26:0/4254881291 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f2060130e80 con 0x7f20600a57a0 2024-08-19T12:38:35.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.588+0000 7f2056ffd640 1 -- 172.21.15.26:0/4254881291 <== mon.6 v1:172.21.15.26:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 203001719) 0x7f20500170a0 con 0x7f20600a57a0 2024-08-19T12:38:35.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.588+0000 7f2056ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T12:38:35.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.588+0000 7f2056ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:35.589 INFO:teuthology.orchestra.run.smithi026.stdout:51539607556 2024-08-19T12:38:35.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 10 librados: watch_flush enter 2024-08-19T12:38:35.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 10 librados: watch_flush exit 2024-08-19T12:38:35.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 20 client.4389.objecter shutdown clearing up homeless session... 2024-08-19T12:38:35.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 10 client.4389.objecter successfully canceled tick 2024-08-19T12:38:35.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 1 -- 172.21.15.26:0/4254881291 >> v1:172.21.15.104:6800/899887506 conn(0x7f20440417b0 legacy=0x7f2044043c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 10 monclient: shutdown 2024-08-19T12:38:35.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:35.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 1 -- 172.21.15.26:0/4254881291 >> v1:172.21.15.26:6792/0 conn(0x7f20600a57a0 legacy=0x7f206012e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 1 -- 172.21.15.26:0/4254881291 shutdown_connections 2024-08-19T12:38:35.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 1 -- 172.21.15.26:0/4254881291 >> 172.21.15.26:0/4254881291 conn(0x7f20600abff0 msgr2=0x7f20601354a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:35.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 1 -- 172.21.15.26:0/4254881291 shutdown_connections 2024-08-19T12:38:35.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 1 -- 172.21.15.26:0/4254881291 wait complete. 2024-08-19T12:38:35.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.590+0000 7f2054ff9640 1 librados: shutdown 2024-08-19T12:38:35.602 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.3 2024-08-19T12:38:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.646+0000 7f69cdd53640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T12:38:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.646+0000 7f69cdd53640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T12:38:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.646+0000 7f69cdd53640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.646+0000 7f69cdd53640 1 -- 172.21.15.26:0/2399563671 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f69c806c840 con 0x7f69c80a57a0 2024-08-19T12:38:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.647+0000 7f69affff640 1 -- 172.21.15.26:0/2399563671 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 956180462) 0x7f69bc016e40 con 0x7f69c80a57a0 2024-08-19T12:38:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.647+0000 7f69affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T12:38:35.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.647+0000 7f69affff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:35.648 INFO:teuthology.orchestra.run.smithi026.stdout:51539607555 2024-08-19T12:38:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.649+0000 7f69cdd53640 10 librados: watch_flush enter 2024-08-19T12:38:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.649+0000 7f69cdd53640 10 librados: watch_flush exit 2024-08-19T12:38:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.649+0000 7f69cdd53640 20 client.4400.objecter shutdown clearing up homeless session... 2024-08-19T12:38:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.649+0000 7f69cdd53640 10 client.4400.objecter successfully canceled tick 2024-08-19T12:38:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.649+0000 7f69cdd53640 1 -- 172.21.15.26:0/2399563671 >> v1:172.21.15.104:6800/899887506 conn(0x7f69b404a500 legacy=0x7f69b404c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.649+0000 7f69cdd53640 10 monclient: shutdown 2024-08-19T12:38:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.649+0000 7f69cdd53640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.649+0000 7f69cdd53640 1 -- 172.21.15.26:0/2399563671 >> v1:172.21.15.26:6793/0 conn(0x7f69c80a57a0 legacy=0x7f69c812e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.649+0000 7f69cdd53640 1 -- 172.21.15.26:0/2399563671 shutdown_connections 2024-08-19T12:38:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.649+0000 7f69cdd53640 1 -- 172.21.15.26:0/2399563671 >> 172.21.15.26:0/2399563671 conn(0x7f69c80abff0 msgr2=0x7f69c80a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.650+0000 7f69cdd53640 1 -- 172.21.15.26:0/2399563671 shutdown_connections 2024-08-19T12:38:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.650+0000 7f69cdd53640 1 -- 172.21.15.26:0/2399563671 wait complete. 2024-08-19T12:38:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.650+0000 7f69cdd53640 1 librados: shutdown 2024-08-19T12:38:35.661 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.5 2024-08-19T12:38:35.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.737+0000 7ff161bf4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T12:38:35.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.737+0000 7ff161bf4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T12:38:35.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.737+0000 7ff161bf4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:35.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.737+0000 7ff161bf4640 1 -- 172.21.15.26:0/3672565718 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff15c136cd0 con 0x7ff15c073470 2024-08-19T12:38:35.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.737+0000 7ff143fff640 1 -- 172.21.15.26:0/3672565718 <== mon.6 v1:172.21.15.26:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 1444633209) 0x7ff14c0178a0 con 0x7ff15c073470 2024-08-19T12:38:35.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.737+0000 7ff143fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T12:38:35.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.737+0000 7ff143fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:35.739 INFO:teuthology.orchestra.run.smithi026.stdout:47244640260 2024-08-19T12:38:35.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.740+0000 7ff161bf4640 10 librados: watch_flush enter 2024-08-19T12:38:35.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.740+0000 7ff161bf4640 10 librados: watch_flush exit 2024-08-19T12:38:35.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.740+0000 7ff161bf4640 20 client.4371.objecter shutdown clearing up homeless session... 2024-08-19T12:38:35.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.740+0000 7ff161bf4640 10 client.4371.objecter successfully canceled tick 2024-08-19T12:38:35.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.740+0000 7ff161bf4640 1 -- 172.21.15.26:0/3672565718 >> v1:172.21.15.104:6800/899887506 conn(0x7ff1480413f0 legacy=0x7ff1480438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.740+0000 7ff161bf4640 10 monclient: shutdown 2024-08-19T12:38:35.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.740+0000 7ff161bf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:35.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.740+0000 7ff161bf4640 1 -- 172.21.15.26:0/3672565718 >> v1:172.21.15.26:6792/0 conn(0x7ff15c073470 legacy=0x7ff15c1ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.741+0000 7ff161bf4640 1 -- 172.21.15.26:0/3672565718 shutdown_connections 2024-08-19T12:38:35.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.741+0000 7ff161bf4640 1 -- 172.21.15.26:0/3672565718 >> 172.21.15.26:0/3672565718 conn(0x7ff15c0abff0 msgr2=0x7ff15c1338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:35.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.741+0000 7ff161bf4640 1 -- 172.21.15.26:0/3672565718 shutdown_connections 2024-08-19T12:38:35.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.741+0000 7ff161bf4640 1 -- 172.21.15.26:0/3672565718 wait complete. 2024-08-19T12:38:35.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.741+0000 7ff161bf4640 1 librados: shutdown 2024-08-19T12:38:35.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.741+0000 7fdd8b59e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T12:38:35.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.741+0000 7fdd8b59e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T12:38:35.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.741+0000 7fdd8b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:35.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.741+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3580148894 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fdd70005370 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.741+0000 7fdd837fe640 1 -- 172.21.15.26:0/3580148894 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 956180462) 0x7fdd8c069540 con 0x7fdd8c0a17b0 2024-08-19T12:38:35.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.741+0000 7fdd837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T12:38:35.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.741+0000 7fdd837fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:35.745 INFO:teuthology.orchestra.run.smithi026.stdout:51539607555 2024-08-19T12:38:35.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.743+0000 7fdd8b59e640 10 librados: watch_flush enter 2024-08-19T12:38:35.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.743+0000 7fdd8b59e640 10 librados: watch_flush exit 2024-08-19T12:38:35.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.743+0000 7fdd8b59e640 20 client.4322.objecter shutdown clearing up homeless session... 2024-08-19T12:38:35.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.743+0000 7fdd8b59e640 10 client.4322.objecter successfully canceled tick 2024-08-19T12:38:35.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.744+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3580148894 >> v1:172.21.15.104:6800/899887506 conn(0x7fdd6c040eb0 legacy=0x7fdd6c043370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.744+0000 7fdd8b59e640 10 monclient: shutdown 2024-08-19T12:38:35.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.744+0000 7fdd8b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:35.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.744+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3580148894 >> v1:172.21.15.26:6790/0 conn(0x7fdd8c0a17b0 legacy=0x7fdd8c136730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.744+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3580148894 shutdown_connections 2024-08-19T12:38:35.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.744+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3580148894 >> 172.21.15.26:0/3580148894 conn(0x7fdd8c0a53c0 msgr2=0x7fdd8c0a5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:35.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.744+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3580148894 shutdown_connections 2024-08-19T12:38:35.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.744+0000 7fdd8b59e640 1 -- 172.21.15.26:0/3580148894 wait complete. 2024-08-19T12:38:35.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.744+0000 7fdd8b59e640 1 librados: shutdown 2024-08-19T12:38:35.753 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.0 2024-08-19T12:38:35.756 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.2 2024-08-19T12:38:35.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.840+0000 7f936e027640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T12:38:35.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.840+0000 7f936e027640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T12:38:35.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.840+0000 7f936e027640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.840+0000 7f936e027640 1 -- 172.21.15.26:0/271140412 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f9348005370 con 0x7f9368073470 2024-08-19T12:38:35.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.841+0000 7f934ffff640 1 -- 172.21.15.26:0/271140412 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 956180462) 0x7f935801b480 con 0x7f9368073470 2024-08-19T12:38:35.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.841+0000 7f934ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T12:38:35.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.841+0000 7f934ffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:35.843 INFO:teuthology.orchestra.run.smithi026.stdout:51539607555 2024-08-19T12:38:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 10 librados: watch_flush enter 2024-08-19T12:38:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 10 librados: watch_flush exit 2024-08-19T12:38:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 20 client.4409.objecter shutdown clearing up homeless session... 2024-08-19T12:38:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 10 client.4409.objecter successfully canceled tick 2024-08-19T12:38:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 1 -- 172.21.15.26:0/271140412 >> v1:172.21.15.104:6800/899887506 conn(0x7f9354041560 legacy=0x7f9354043a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 10 monclient: shutdown 2024-08-19T12:38:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 1 -- 172.21.15.26:0/271140412 >> v1:172.21.15.26:6793/0 conn(0x7f9368073470 legacy=0x7f93680a2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 1 -- 172.21.15.26:0/271140412 shutdown_connections 2024-08-19T12:38:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 1 -- 172.21.15.26:0/271140412 >> 172.21.15.26:0/271140412 conn(0x7f93680abff0 msgr2=0x7f93681338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 1 -- 172.21.15.26:0/271140412 shutdown_connections 2024-08-19T12:38:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 1 -- 172.21.15.26:0/271140412 wait complete. 2024-08-19T12:38:35.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.843+0000 7f936e027640 1 librados: shutdown 2024-08-19T12:38:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.847+0000 7f6bde022640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T12:38:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.847+0000 7f6bde022640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T12:38:35.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.847+0000 7f6bde022640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.847+0000 7f6bde022640 1 -- 172.21.15.26:0/151073713 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6bd00a7260 con 0x7f6bd00201f0 2024-08-19T12:38:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.848+0000 7f6bcdffb640 1 -- 172.21.15.26:0/151073713 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 956180462) 0x7f6bd4017520 con 0x7f6bd00201f0 2024-08-19T12:38:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.848+0000 7f6bcdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T12:38:35.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.848+0000 7f6bcdffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:35.849 INFO:teuthology.orchestra.run.smithi026.stdout:51539607555 2024-08-19T12:38:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 10 librados: watch_flush enter 2024-08-19T12:38:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 10 librados: watch_flush exit 2024-08-19T12:38:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 20 client.4418.objecter shutdown clearing up homeless session... 2024-08-19T12:38:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 10 client.4418.objecter successfully canceled tick 2024-08-19T12:38:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 1 -- 172.21.15.26:0/151073713 >> v1:172.21.15.104:6800/899887506 conn(0x7f6bbc045520 legacy=0x7f6bbc0479e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 10 monclient: shutdown 2024-08-19T12:38:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 1 -- 172.21.15.26:0/151073713 >> v1:172.21.15.26:6793/0 conn(0x7f6bd00201f0 legacy=0x7f6bd00a3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 1 -- 172.21.15.26:0/151073713 shutdown_connections 2024-08-19T12:38:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 1 -- 172.21.15.26:0/151073713 >> 172.21.15.26:0/151073713 conn(0x7f6bd0007970 msgr2=0x7f6bd0022de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 1 -- 172.21.15.26:0/151073713 shutdown_connections 2024-08-19T12:38:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 1 -- 172.21.15.26:0/151073713 wait complete. 2024-08-19T12:38:35.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:35.850+0000 7f6bde022640 1 librados: shutdown 2024-08-19T12:38:35.855 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.4 2024-08-19T12:38:35.861 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.1 2024-08-19T12:38:36.603 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-19T12:38:36.661 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-19T12:38:36.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.715+0000 7f1fec986640 10 monclient: get_monmap_and_config 2024-08-19T12:38:36.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.715+0000 7f1fec986640 10 monclient: build_initial_monmap 2024-08-19T12:38:36.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.715+0000 7f1fec986640 10 monclient: monmap: 2024-08-19T12:38:36.716 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:36.716682+0000 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:36.716682+0000 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:36.717 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:36.718 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.716+0000 7f1fec986640 1 Processor -- start 2024-08-19T12:38:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.716+0000 7f1fec986640 1 -- start start 2024-08-19T12:38:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.716+0000 7f1fec986640 10 monclient: init 2024-08-19T12:38:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fec986640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fec986640 10 monclient: _add_conns ranks=[1,2,6,0,8,7,5,3,4] 2024-08-19T12:38:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fec986640 10 monclient(hunting): picked mon.noname-f con 0x7f1fe8073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fec986640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe80a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fec986640 10 monclient(hunting): picked mon.noname-d con 0x7f1fe813f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fec986640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe809dae0 con 0x7f1fe8073470 2024-08-19T12:38:36.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fec986640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe8136910 con 0x7f1fe80a57a0 2024-08-19T12:38:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fec986640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe809dd50 con 0x7f1fe813f2e0 2024-08-19T12:38:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fec986640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fec986640 10 monclient(hunting): authenticate will time out at 710.804267s 2024-08-19T12:38:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fe6d9d640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f1fe813f2e0 0x7f1fe81416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58598/0 (socket says 172.21.15.26:58598) 2024-08-19T12:38:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fe5d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1fe80a57a0 0x7f1fe80a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50346/0 (socket says 172.21.15.26:50346) 2024-08-19T12:38:36.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.717+0000 7f1fe5d9b640 1 -- 172.21.15.26:0/3131462277 learned_addr learned my addr 172.21.15.26:0/3131462277 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe6d9d640 1 -- 172.21.15.26:0/3131462277 >> v1:172.21.15.26:6792/0 conn(0x7f1fe813f2e0 legacy=0x7f1fe81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe5d9b640 1 -- 172.21.15.26:0/3131462277 >> v1:172.21.15.26:6790/0 conn(0x7f1fe80a57a0 legacy=0x7f1fe80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe659c640 1 -- 172.21.15.26:0/3131462277 >> v1:172.21.15.104:6789/0 conn(0x7f1fe8073470 legacy=0x7f1fe8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2721546900 0 0) 0x7f1fe809dd50 con 0x7f1fe813f2e0 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 10 monclient(hunting): my global_id is 4416 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fd4003520 con 0x7f1fe813f2e0 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 162723894 0 0) 0x7f1fe8136910 con 0x7f1fe80a57a0 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 10 monclient(hunting): my global_id is 4340 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fe809dd50 con 0x7f1fe80a57a0 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.719+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 647218833 0 0) 0x7f1fe809dae0 con 0x7f1fe8073470 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient(hunting): my global_id is 4321 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fe8136910 con 0x7f1fe8073470 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1545870205 0 0) 0x7f1fd4003520 con 0x7f1fe813f2e0 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 >> v1:172.21.15.26:6790/0 conn(0x7f1fe80a57a0 legacy=0x7f1fe80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 >> v1:172.21.15.104:6789/0 conn(0x7f1fe8073470 legacy=0x7f1fe8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient: _finish_hunting 0 2024-08-19T12:38:36.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 1 monclient: found mon.noname-d 2024-08-19T12:38:36.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:36.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:36.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe809e030 con 0x7f1fe813f2e0 2024-08-19T12:38:36.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient: _finish_auth 0 2024-08-19T12:38:36.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient: _check_auth_tickets 2024-08-19T12:38:36.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:36.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1fe0001e60 con 0x7f1fe813f2e0 2024-08-19T12:38:36.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:36.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:36.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient: dump: 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.727 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fec986640 5 monclient: authenticate success, global_id 4416 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fec986640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1fe0004b60 con 0x7f1fe813f2e0 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 1 -- 172.21.15.26:0/3131462277 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1fe00050d0 con 0x7f1fe813f2e0 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fec986640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fec986640 4 set_mon_vals no callback set 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fe559a640 10 monclient: dump: 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.728 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fec986640 10 monclient: shutdown 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fec986640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.720+0000 7f1fec986640 1 -- 172.21.15.26:0/3131462277 >> v1:172.21.15.26:6792/0 conn(0x7f1fe813f2e0 legacy=0x7f1fe81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.721+0000 7f1fec986640 1 -- 172.21.15.26:0/3131462277 shutdown_connections 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.721+0000 7f1fec986640 1 -- 172.21.15.26:0/3131462277 >> 172.21.15.26:0/3131462277 conn(0x7f1fe80abff0 msgr2=0x7f1fe80ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.721+0000 7f1fec986640 1 -- 172.21.15.26:0/3131462277 shutdown_connections 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.721+0000 7f1fec986640 1 -- 172.21.15.26:0/3131462277 wait complete. 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.721+0000 7f1fec986640 10 monclient: build_initial_monmap 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.721+0000 7f1fec986640 10 monclient: monmap: 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:36.730 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 1 librados: starting msgr at 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 1 librados: starting objecter 2024-08-19T12:38:36.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 1 Processor -- start 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 1 -- start start 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 1 librados: setting wanted keys 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 1 librados: calling monclient init 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 10 monclient: init 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 10 monclient: _add_conns ranks=[6,0,7,3,4,8,5,1,2] 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 10 monclient(hunting): picked mon.noname-d con 0x7f1fe8073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe80a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 10 monclient(hunting): picked mon.noname-i con 0x7f1fe813f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe8135350 con 0x7f1fe80a57a0 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe8136f30 con 0x7f1fe8073470 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe80a5450 con 0x7f1fe813f2e0 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:36.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fec986640 10 monclient(hunting): authenticate will time out at 710.809842s 2024-08-19T12:38:36.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fe5d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1fe80a57a0 0x7f1fe80a28f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58462/0 (socket says 172.21.15.26:58462) 2024-08-19T12:38:36.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fe659c640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f1fe8073470 0x7f1fe80a21e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58606/0 (socket says 172.21.15.26:58606) 2024-08-19T12:38:36.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.722+0000 7f1fe5d9b640 1 -- 172.21.15.26:0/3194084336 learned_addr learned my addr 172.21.15.26:0/3194084336 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:36.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fe5d9b640 1 -- 172.21.15.26:0/3194084336 >> v1:172.21.15.26:6789/0 conn(0x7f1fe80a57a0 legacy=0x7f1fe80a28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fe659c640 1 -- 172.21.15.26:0/3194084336 >> v1:172.21.15.26:6792/0 conn(0x7f1fe8073470 legacy=0x7f1fe80a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 client.?.objecter ms_handle_connect 0x7f1fe80a57a0 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 client.?.objecter ms_handle_connect 0x7f1fe8073470 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 client.?.objecter ms_handle_connect 0x7f1fe813f2e0 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fe6d9d640 1 -- 172.21.15.26:0/3194084336 >> v1:172.21.15.104:6792/0 conn(0x7f1fe813f2e0 legacy=0x7f1fe81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1894599401 0 0) 0x7f1fe809dd50 con 0x7f1fe80a57a0 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): my global_id is 4338 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fc8004eb0 con 0x7f1fe80a57a0 2024-08-19T12:38:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 468011466 0 0) 0x7f1fe8136f30 con 0x7f1fe8073470 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): my global_id is 4425 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fe809dd50 con 0x7f1fe8073470 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3427673904 0 0) 0x7f1fe80a5450 con 0x7f1fe813f2e0 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): my global_id is 4318 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fe8136f30 con 0x7f1fe813f2e0 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1092798823 0 0) 0x7f1fc8004eb0 con 0x7f1fe80a57a0 2024-08-19T12:38:36.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fe80a5450 con 0x7f1fe80a57a0 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2861041076 0 0) 0x7f1fe809dd50 con 0x7f1fe8073470 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fc8004eb0 con 0x7f1fe8073470 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1fd80047e0 con 0x7f1fe80a57a0 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): dump: 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.739 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1fd0003610 con 0x7f1fe8073470 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): dump: 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.740 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.741 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 546079097 0 0) 0x7f1fe8136f30 con 0x7f1fe813f2e0 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fe809dd50 con 0x7f1fe813f2e0 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1fe00016a0 con 0x7f1fe813f2e0 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient(hunting): dump: 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.742 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 388453230 0 0) 0x7f1fe80a5450 con 0x7f1fe80a57a0 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 >> v1:172.21.15.104:6792/0 conn(0x7f1fe813f2e0 legacy=0x7f1fe81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 >> v1:172.21.15.26:6792/0 conn(0x7f1fe8073470 legacy=0x7f1fe80a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:36.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 monclient: found mon.a 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe80a44f0 con 0x7f1fe80a57a0 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fec986640 5 monclient: authenticate success, global_id 4338 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fec986640 10 monclient: _renew_subs 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fec986640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.725+0000 7f1fec986640 1 -- 172.21.15.26:0/3194084336 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe80a3000 con 0x7f1fe80a57a0 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fec986640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fec986640 10 monclient: _renew_subs 2024-08-19T12:38:36.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fec986640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fec986640 1 -- 172.21.15.26:0/3194084336 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1fe81d7870 con 0x7f1fe80a57a0 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fec986640 1 librados: init done 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1fd8004de0 con 0x7f1fe80a57a0 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fdeffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1fd80052d0 con 0x7f1fe80a57a0 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fdeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fe4d99640 4 set_mon_vals no callback set 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fdeffd640 10 monclient: dump: 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.745 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fec986640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:36.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fec986640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fec986640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.726+0000 7f1fec986640 1 -- 172.21.15.26:0/3194084336 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fcc005080 con 0x7f1fe80a57a0 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.727+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f1fd8011730 con 0x7f1fe80a57a0 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.727+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f1fd804db10 con 0x7f1fe80a57a0 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.727+0000 7f1fdeffd640 10 client.4338.objecter ms_dispatch 0x7f1fe8000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.727+0000 7f1fdeffd640 3 client.4338.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.727+0000 7f1fdeffd640 3 client.4338.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.727+0000 7f1fdeffd640 20 client.4338.objecter dump_active .. 0 homeless 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.727+0000 7f1fe659c640 10 monclient: get_auth_request con 0x7f1fc8041ad0 auth_method 0 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.729+0000 7f1fdeffd640 10 client.4338.objecter ms_handle_connect 0x7f1fc8041ad0 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.731+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f1fe8135350 con 0x7f1fe80a57a0 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.731+0000 7f1fdeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:36.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.731+0000 7f1fdeffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:36.754 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-19T12:38:36.756 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-19T12:38:36.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.767+0000 7f130797f640 10 monclient: get_monmap_and_config 2024-08-19T12:38:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.767+0000 7f130797f640 10 monclient: build_initial_monmap 2024-08-19T12:38:36.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.767+0000 7f130797f640 10 monclient: monmap: 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:36.768708+0000 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:36.768708+0000 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:36.824 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:36.825 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:36.825 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:36.825 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.768+0000 7f130797f640 1 Processor -- start 2024-08-19T12:38:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.768+0000 7f130797f640 1 -- start start 2024-08-19T12:38:36.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.768+0000 7f130797f640 10 monclient: init 2024-08-19T12:38:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.769+0000 7f130797f640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.769+0000 7f130797f640 10 monclient: _add_conns ranks=[1,3,8,5,4,7,6,0,2] 2024-08-19T12:38:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.769+0000 7f130797f640 10 monclient(hunting): picked mon.noname-f con 0x7f1300073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.769+0000 7f130797f640 10 monclient(hunting): picked mon.noname-g con 0x7f13000a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.769+0000 7f130797f640 10 monclient(hunting): picked mon.noname-e con 0x7f130013f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.769+0000 7f130797f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130009dae0 con 0x7f1300073470 2024-08-19T12:38:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.769+0000 7f130797f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1300136910 con 0x7f13000a57a0 2024-08-19T12:38:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.769+0000 7f130797f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130009dd50 con 0x7f130013f2e0 2024-08-19T12:38:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.769+0000 7f130797f640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.769+0000 7f130797f640 10 monclient(hunting): authenticate will time out at 710.856398s 2024-08-19T12:38:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.769+0000 7f1305f1c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f130013f2e0 0x7f13001416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34030/0 (socket says 172.21.15.26:34030) 2024-08-19T12:38:36.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.769+0000 7f1305f1c640 1 -- 172.21.15.26:0/2670375727 learned_addr learned my addr 172.21.15.26:0/2670375727 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f1305f1c640 1 -- 172.21.15.26:0/2670375727 >> v1:172.21.15.26:6793/0 conn(0x7f130013f2e0 legacy=0x7f13001416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f1304f1a640 1 -- 172.21.15.26:0/2670375727 >> v1:172.21.15.104:6790/0 conn(0x7f13000a57a0 legacy=0x7f13000a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f130571b640 1 -- 172.21.15.26:0/2670375727 >> v1:172.21.15.104:6789/0 conn(0x7f1300073470 legacy=0x7f1300136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4098494656 0 0) 0x7f130009dd50 con 0x7f130013f2e0 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 10 monclient(hunting): my global_id is 4427 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12ec003520 con 0x7f130013f2e0 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3974528502 0 0) 0x7f1300136910 con 0x7f13000a57a0 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 10 monclient(hunting): my global_id is 4422 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f130009dd50 con 0x7f13000a57a0 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2717066214 0 0) 0x7f130009dae0 con 0x7f1300073470 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 10 monclient(hunting): my global_id is 4330 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.771+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1300136910 con 0x7f1300073470 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2680504160 0 0) 0x7f12ec003520 con 0x7f130013f2e0 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 >> v1:172.21.15.104:6790/0 conn(0x7f13000a57a0 legacy=0x7f13000a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 >> v1:172.21.15.104:6789/0 conn(0x7f1300073470 legacy=0x7f1300136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 1 monclient: found mon.noname-e 2024-08-19T12:38:36.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130009e030 con 0x7f130013f2e0 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 10 monclient: _finish_auth 0 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f12f8001e60 con 0x7f130013f2e0 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f130797f640 5 monclient: authenticate success, global_id 4427 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f130797f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 10 monclient: dump: 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.838 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f130797f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12f8004c20 con 0x7f130013f2e0 2024-08-19T12:38:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f130797f640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 1 -- 172.21.15.26:0/2670375727 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f12f8005190 con 0x7f130013f2e0 2024-08-19T12:38:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f130797f640 4 set_mon_vals no callback set 2024-08-19T12:38:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.772+0000 7f12f7fff640 10 monclient: dump: 2024-08-19T12:38:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.839 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.840 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.773+0000 7f130797f640 10 monclient: shutdown 2024-08-19T12:38:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.773+0000 7f130797f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.773+0000 7f130797f640 1 -- 172.21.15.26:0/2670375727 >> v1:172.21.15.26:6793/0 conn(0x7f130013f2e0 legacy=0x7f13001416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.773+0000 7f130797f640 1 -- 172.21.15.26:0/2670375727 shutdown_connections 2024-08-19T12:38:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.773+0000 7f130797f640 1 -- 172.21.15.26:0/2670375727 >> 172.21.15.26:0/2670375727 conn(0x7f13000abff0 msgr2=0x7f13000ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.773+0000 7f130797f640 1 -- 172.21.15.26:0/2670375727 shutdown_connections 2024-08-19T12:38:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.773+0000 7f130797f640 1 -- 172.21.15.26:0/2670375727 wait complete. 2024-08-19T12:38:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.773+0000 7f130797f640 10 monclient: build_initial_monmap 2024-08-19T12:38:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.773+0000 7f130797f640 10 monclient: monmap: 2024-08-19T12:38:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:36.841 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:36.842 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 1 librados: starting msgr at 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 1 librados: starting objecter 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 1 Processor -- start 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 1 -- start start 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 1 librados: setting wanted keys 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 1 librados: calling monclient init 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 10 monclient: init 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 10 monclient: _add_conns ranks=[4,5,8,3,1,6,7,0,2] 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 10 monclient(hunting): picked mon.noname-c con 0x7f1300073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 10 monclient(hunting): picked mon.noname-h con 0x7f13000a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:36.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 10 monclient(hunting): picked mon.noname-e con 0x7f130013f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13000a26a0 con 0x7f1300073470 2024-08-19T12:38:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13000a5430 con 0x7f13000a57a0 2024-08-19T12:38:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13000a0be0 con 0x7f130013f2e0 2024-08-19T12:38:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130797f640 10 monclient(hunting): authenticate will time out at 710.861834s 2024-08-19T12:38:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130571b640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f1300073470 0x7f130012f5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57910/0 (socket says 172.21.15.26:57910) 2024-08-19T12:38:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.774+0000 7f130571b640 1 -- 172.21.15.26:0/3151564670 learned_addr learned my addr 172.21.15.26:0/3151564670 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f1305f1c640 1 -- 172.21.15.26:0/3151564670 >> v1:172.21.15.26:6793/0 conn(0x7f130013f2e0 legacy=0x7f13001d6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 client.?.objecter ms_handle_connect 0x7f130013f2e0 2024-08-19T12:38:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f130571b640 1 -- 172.21.15.26:0/3151564670 >> v1:172.21.15.26:6791/0 conn(0x7f1300073470 legacy=0x7f130012f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 client.?.objecter ms_handle_connect 0x7f1300073470 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 client.?.objecter ms_handle_connect 0x7f13000a57a0 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f1304f1a640 1 -- 172.21.15.26:0/3151564670 >> v1:172.21.15.104:6791/0 conn(0x7f13000a57a0 legacy=0x7f13001d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4049764163 0 0) 0x7f13000a0be0 con 0x7f130013f2e0 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): my global_id is 4436 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12e4004eb0 con 0x7f130013f2e0 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1994222795 0 0) 0x7f13000a26a0 con 0x7f1300073470 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): my global_id is 4387 2024-08-19T12:38:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13000a0be0 con 0x7f1300073470 2024-08-19T12:38:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2460044684 0 0) 0x7f13000a5430 con 0x7f13000a57a0 2024-08-19T12:38:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): my global_id is 4343 2024-08-19T12:38:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13000a26a0 con 0x7f13000a57a0 2024-08-19T12:38:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1922603442 0 0) 0x7f13000a0be0 con 0x7f1300073470 2024-08-19T12:38:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13000a5430 con 0x7f1300073470 2024-08-19T12:38:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2621709373 0 0) 0x7f12e4004eb0 con 0x7f130013f2e0 2024-08-19T12:38:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13000a0be0 con 0x7f130013f2e0 2024-08-19T12:38:36.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f12f80020b0 con 0x7f130013f2e0 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.847 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.848 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1106646571 0 0) 0x7f13000a26a0 con 0x7f13000a57a0 2024-08-19T12:38:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12e4004eb0 con 0x7f13000a57a0 2024-08-19T12:38:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f12f0003610 con 0x7f13000a57a0 2024-08-19T12:38:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:36.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.849 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3531863202 0 0) 0x7f13000a0be0 con 0x7f130013f2e0 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 >> v1:172.21.15.104:6791/0 conn(0x7f13000a57a0 legacy=0x7f13001d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 >> v1:172.21.15.26:6791/0 conn(0x7f1300073470 legacy=0x7f130012f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 monclient: found mon.e 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13000a2f40 con 0x7f130013f2e0 2024-08-19T12:38:36.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f12f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f130797f640 5 monclient: authenticate success, global_id 4436 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f130797f640 10 monclient: _renew_subs 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f130797f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.777+0000 7f130797f640 1 -- 172.21.15.26:0/3151564670 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f13000a2370 con 0x7f130013f2e0 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.778+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12f8001ec0 con 0x7f130013f2e0 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.778+0000 7f12f5ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.778+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f12f8005700 con 0x7f130013f2e0 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.778+0000 7f12f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.778+0000 7f12f5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.778+0000 7f12f5ffb640 10 monclient: dump: 2024-08-19T12:38:36.851 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.852 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.778+0000 7f130797f640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.778+0000 7f130797f640 10 monclient: _renew_subs 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.778+0000 7f130797f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.778+0000 7f130797f640 1 -- 172.21.15.26:0/3151564670 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f13000a02c0 con 0x7f130013f2e0 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.778+0000 7f130797f640 1 librados: init done 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.778+0000 7f12f77fe640 4 set_mon_vals no callback set 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.779+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f12f8011b60 con 0x7f130013f2e0 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.779+0000 7f130797f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.779+0000 7f130797f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.779+0000 7f130797f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.779+0000 7f130797f640 1 -- 172.21.15.26:0/3151564670 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1300142780 con 0x7f130013f2e0 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.779+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f12f804db90 con 0x7f130013f2e0 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.779+0000 7f12f5ffb640 10 client.4436.objecter ms_dispatch 0x7f1300000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:36.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.779+0000 7f12f5ffb640 3 client.4436.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.779+0000 7f12f5ffb640 3 client.4436.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.779+0000 7f12f5ffb640 20 client.4436.objecter dump_active .. 0 homeless 2024-08-19T12:38:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.779+0000 7f130571b640 10 monclient: get_auth_request con 0x7f12e4041940 auth_method 0 2024-08-19T12:38:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.782+0000 7f12f5ffb640 10 client.4436.objecter ms_handle_connect 0x7f12e4041940 2024-08-19T12:38:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.783+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f12f8017040 con 0x7f130013f2e0 2024-08-19T12:38:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.784+0000 7f12f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.784+0000 7f12f5ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:36.856 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-19T12:38:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.856+0000 7f7330c60640 10 monclient: get_monmap_and_config 2024-08-19T12:38:36.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.856+0000 7f7330c60640 10 monclient: build_initial_monmap 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.856+0000 7f7330c60640 10 monclient: monmap: 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:36.857696+0000 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:36.857696+0000 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:36.858 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:36.859 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:36.859 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.857+0000 7f7330c60640 1 Processor -- start 2024-08-19T12:38:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.857+0000 7f7330c60640 1 -- start start 2024-08-19T12:38:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.857+0000 7f7330c60640 10 monclient: init 2024-08-19T12:38:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f7330c60640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f7330c60640 10 monclient: _add_conns ranks=[2,8,6,3,4,0,5,1,7] 2024-08-19T12:38:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f7330c60640 10 monclient(hunting): picked mon.noname-b con 0x7f732c073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f7330c60640 10 monclient(hunting): picked mon.noname-e con 0x7f732c0a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f7330c60640 10 monclient(hunting): picked mon.noname-d con 0x7f732c13f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f7330c60640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f732c09dae0 con 0x7f732c073470 2024-08-19T12:38:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f7330c60640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f732c136910 con 0x7f732c13f2e0 2024-08-19T12:38:36.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f7330c60640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f732c09dd50 con 0x7f732c0a57a0 2024-08-19T12:38:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f7330c60640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f7330c60640 10 monclient(hunting): authenticate will time out at 710.945394s 2024-08-19T12:38:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f732ad9d640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f732c13f2e0 0x7f732c1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58610/0 (socket says 172.21.15.26:58610) 2024-08-19T12:38:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f7329d9b640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f732c0a57a0 0x7f732c0a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34048/0 (socket says 172.21.15.26:34048) 2024-08-19T12:38:36.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.858+0000 7f732ad9d640 1 -- 172.21.15.26:0/310602828 learned_addr learned my addr 172.21.15.26:0/310602828 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:36.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.860+0000 7f7329d9b640 1 -- 172.21.15.26:0/310602828 >> v1:172.21.15.26:6793/0 conn(0x7f732c0a57a0 legacy=0x7f732c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.860+0000 7f732ad9d640 1 -- 172.21.15.26:0/310602828 >> v1:172.21.15.26:6792/0 conn(0x7f732c13f2e0 legacy=0x7f732c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.860+0000 7f732a59c640 1 -- 172.21.15.26:0/310602828 >> v1:172.21.15.26:6790/0 conn(0x7f732c073470 legacy=0x7f732c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.860+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 811271428 0 0) 0x7f732c136910 con 0x7f732c13f2e0 2024-08-19T12:38:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.860+0000 7f732959a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.860+0000 7f732959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.860+0000 7f732959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.860+0000 7f732959a640 10 monclient(hunting): my global_id is 4434 2024-08-19T12:38:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7314003520 con 0x7f732c13f2e0 2024-08-19T12:38:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3817705859 0 0) 0x7f732c09dd50 con 0x7f732c0a57a0 2024-08-19T12:38:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient(hunting): my global_id is 4445 2024-08-19T12:38:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f732c136910 con 0x7f732c0a57a0 2024-08-19T12:38:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3598798061 0 0) 0x7f732c09dae0 con 0x7f732c073470 2024-08-19T12:38:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient(hunting): my global_id is 4349 2024-08-19T12:38:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f732c09dd50 con 0x7f732c073470 2024-08-19T12:38:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1217675063 0 0) 0x7f7314003520 con 0x7f732c13f2e0 2024-08-19T12:38:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 >> v1:172.21.15.26:6793/0 conn(0x7f732c0a57a0 legacy=0x7f732c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 >> v1:172.21.15.26:6790/0 conn(0x7f732c073470 legacy=0x7f732c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient: _finish_hunting 0 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 monclient: found mon.noname-d 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732c09e030 con 0x7f732c13f2e0 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient: _finish_auth 0 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient: _check_auth_tickets 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7324001e60 con 0x7f732c13f2e0 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient: dump: 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.866 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.867 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f7330c60640 5 monclient: authenticate success, global_id 4434 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7324004c20 con 0x7f732c13f2e0 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f7330c60640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f7330c60640 4 set_mon_vals no callback set 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 1 -- 172.21.15.26:0/310602828 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f73240051d0 con 0x7f732c13f2e0 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.861+0000 7f732959a640 10 monclient: dump: 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.868 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.862+0000 7f7330c60640 10 monclient: shutdown 2024-08-19T12:38:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.862+0000 7f7330c60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.862+0000 7f7330c60640 1 -- 172.21.15.26:0/310602828 >> v1:172.21.15.26:6792/0 conn(0x7f732c13f2e0 legacy=0x7f732c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.862+0000 7f7330c60640 1 -- 172.21.15.26:0/310602828 shutdown_connections 2024-08-19T12:38:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.862+0000 7f7330c60640 1 -- 172.21.15.26:0/310602828 >> 172.21.15.26:0/310602828 conn(0x7f732c0abff0 msgr2=0x7f732c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.863+0000 7f7330c60640 1 -- 172.21.15.26:0/310602828 shutdown_connections 2024-08-19T12:38:36.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.863+0000 7f7330c60640 1 -- 172.21.15.26:0/310602828 wait complete. 2024-08-19T12:38:36.870 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-19T12:38:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.864+0000 7f7330c60640 10 monclient: build_initial_monmap 2024-08-19T12:38:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.864+0000 7f7330c60640 10 monclient: monmap: 2024-08-19T12:38:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:36.879 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.864+0000 7f7330c60640 1 librados: starting msgr at 2024-08-19T12:38:36.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.864+0000 7f7330c60640 1 librados: starting objecter 2024-08-19T12:38:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.864+0000 7f7330c60640 1 Processor -- start 2024-08-19T12:38:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.864+0000 7f7330c60640 1 -- start start 2024-08-19T12:38:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.864+0000 7f7330c60640 1 librados: setting wanted keys 2024-08-19T12:38:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.864+0000 7f7330c60640 1 librados: calling monclient init 2024-08-19T12:38:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.864+0000 7f7330c60640 10 monclient: init 2024-08-19T12:38:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.865+0000 7f7330c60640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.865+0000 7f7330c60640 10 monclient: _add_conns ranks=[2,0,4,8,3,1,5,7,6] 2024-08-19T12:38:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.865+0000 7f7330c60640 10 monclient(hunting): picked mon.noname-b con 0x7f732c073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.865+0000 7f7330c60640 10 monclient(hunting): picked mon.noname-a con 0x7f732c0a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.865+0000 7f732a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f732c073470 0x7f732c132a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50376/0 (socket says 172.21.15.26:50376) 2024-08-19T12:38:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.866+0000 7f7329d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f732c0a57a0 0x7f732c1d2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58478/0 (socket says 172.21.15.26:58478) 2024-08-19T12:38:36.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.866+0000 7f732ad9d640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f732c13f2e0 0x7f732c1d61d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57918/0 (socket says 172.21.15.26:57918) 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.866+0000 7f7330c60640 10 monclient(hunting): picked mon.noname-c con 0x7f732c13f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.866+0000 7f732a59c640 1 -- 172.21.15.26:0/2368157577 learned_addr learned my addr 172.21.15.26:0/2368157577 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.866+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f732c131610 con 0x7f732c0a57a0 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.866+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f732c130030 con 0x7f732c073470 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.866+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f732c12f2b0 con 0x7f732c13f2e0 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.866+0000 7f7330c60640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.866+0000 7f7330c60640 10 monclient(hunting): authenticate will time out at 710.953816s 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.868+0000 7f732a59c640 1 -- 172.21.15.26:0/2368157577 >> v1:172.21.15.26:6790/0 conn(0x7f732c073470 legacy=0x7f732c132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.868+0000 7f731affd640 10 client.?.objecter ms_handle_connect 0x7f732c073470 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.868+0000 7f731affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.868+0000 7f7329d9b640 1 -- 172.21.15.26:0/2368157577 >> v1:172.21.15.26:6789/0 conn(0x7f732c0a57a0 legacy=0x7f732c1d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.868+0000 7f731affd640 10 client.?.objecter ms_handle_connect 0x7f732c0a57a0 2024-08-19T12:38:36.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.868+0000 7f731affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.868+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3159260178 0 0) 0x7f732c130030 con 0x7f732c073470 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.868+0000 7f731affd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.868+0000 7f731affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.868+0000 7f731affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.868+0000 7f731affd640 10 monclient(hunting): my global_id is 4358 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.868+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f730c004eb0 con 0x7f732c073470 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f732ad9d640 1 -- 172.21.15.26:0/2368157577 >> v1:172.21.15.26:6791/0 conn(0x7f732c13f2e0 legacy=0x7f732c1d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 client.?.objecter ms_handle_connect 0x7f732c13f2e0 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3602351755 0 0) 0x7f732c131610 con 0x7f732c0a57a0 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient(hunting): my global_id is 4347 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f732c130030 con 0x7f732c0a57a0 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3246372019 0 0) 0x7f732c12f2b0 con 0x7f732c13f2e0 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient(hunting): my global_id is 4396 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f732c131610 con 0x7f732c13f2e0 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1999202939 0 0) 0x7f730c004eb0 con 0x7f732c073470 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f732c12f2b0 con 0x7f732c073470 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f731c003610 con 0x7f732c073470 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient(hunting): dump: 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.884 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1741705509 0 0) 0x7f732c130030 con 0x7f732c0a57a0 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f730c004eb0 con 0x7f732c0a57a0 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1176868357 0 0) 0x7f732c12f2b0 con 0x7f732c073470 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 >> v1:172.21.15.26:6791/0 conn(0x7f732c13f2e0 legacy=0x7f732c1d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 >> v1:172.21.15.26:6789/0 conn(0x7f732c0a57a0 legacy=0x7f732c1d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 monclient: found mon.b 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732c131330 con 0x7f732c073470 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient: _finish_auth 0 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:36.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f731affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:36.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f7330c60640 5 monclient: authenticate success, global_id 4358 2024-08-19T12:38:36.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f7330c60640 10 monclient: _renew_subs 2024-08-19T12:38:36.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f7330c60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:36.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f732c133150 con 0x7f732c073470 2024-08-19T12:38:36.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f7330c60640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:36.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f7330c60640 10 monclient: _renew_subs 2024-08-19T12:38:36.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f7330c60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:36.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f732c1cd900 con 0x7f732c073470 2024-08-19T12:38:36.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.869+0000 7f7330c60640 1 librados: init done 2024-08-19T12:38:36.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.870+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f731c003ac0 con 0x7f732c073470 2024-08-19T12:38:36.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.870+0000 7f731affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:36.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.870+0000 7f7328d99640 4 set_mon_vals no callback set 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.870+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f731c003ff0 con 0x7f732c073470 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.870+0000 7f731affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.870+0000 7f731affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.870+0000 7f731affd640 10 monclient: dump: 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.871+0000 7f7330c60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.871+0000 7f7330c60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.871+0000 7f7330c60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.871+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7310005080 con 0x7f732c073470 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.871+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f731c004190 con 0x7f732c073470 2024-08-19T12:38:36.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.872+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f731c04da80 con 0x7f732c073470 2024-08-19T12:38:36.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.872+0000 7f731affd640 10 client.4358.objecter ms_dispatch 0x7f732c000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:36.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.872+0000 7f731affd640 3 client.4358.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:36.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.872+0000 7f731affd640 3 client.4358.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:36.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.872+0000 7f731affd640 20 client.4358.objecter dump_active .. 0 homeless 2024-08-19T12:38:36.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.872+0000 7f7329d9b640 10 monclient: get_auth_request con 0x7f730c049c20 auth_method 0 2024-08-19T12:38:36.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.874+0000 7f731affd640 10 client.4358.objecter ms_handle_connect 0x7f730c049c20 2024-08-19T12:38:36.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.880+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f731c0174c0 con 0x7f732c073470 2024-08-19T12:38:36.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.881+0000 7f731affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:36.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.881+0000 7f731affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:36.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.937+0000 7f1fec986640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T12:38:36.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.938+0000 7f1fec986640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T12:38:36.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.938+0000 7f1fec986640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:36.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.938+0000 7f1fec986640 1 -- 172.21.15.26:0/3194084336 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f1fcc005370 con 0x7f1fe80a57a0 2024-08-19T12:38:36.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.938+0000 7f1fdeffd640 1 -- 172.21.15.26:0/3194084336 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 532354816) 0x7f1fd8017610 con 0x7f1fe80a57a0 2024-08-19T12:38:36.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.938+0000 7f1fdeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-19T12:38:36.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.938+0000 7f1fdeffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:36.940 INFO:teuthology.orchestra.run.smithi026.stdout:51539607557 2024-08-19T12:38:36.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 10 librados: watch_flush enter 2024-08-19T12:38:36.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 10 librados: watch_flush exit 2024-08-19T12:38:36.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 20 client.4338.objecter shutdown clearing up homeless session... 2024-08-19T12:38:36.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 10 client.4338.objecter successfully canceled tick 2024-08-19T12:38:36.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 1 -- 172.21.15.26:0/3194084336 >> v1:172.21.15.104:6800/899887506 conn(0x7f1fc8041ad0 legacy=0x7f1fc8043f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 10 monclient: shutdown 2024-08-19T12:38:36.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:36.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 1 -- 172.21.15.26:0/3194084336 >> v1:172.21.15.26:6789/0 conn(0x7f1fe80a57a0 legacy=0x7f1fe80a28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 1 -- 172.21.15.26:0/3194084336 shutdown_connections 2024-08-19T12:38:36.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 1 -- 172.21.15.26:0/3194084336 >> 172.21.15.26:0/3194084336 conn(0x7f1fe80abff0 msgr2=0x7f1fe81349a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:36.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 1 -- 172.21.15.26:0/3194084336 shutdown_connections 2024-08-19T12:38:36.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 1 -- 172.21.15.26:0/3194084336 wait complete. 2024-08-19T12:38:36.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.941+0000 7f1fec986640 1 librados: shutdown 2024-08-19T12:38:36.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.948+0000 7f87fb59e640 10 monclient: get_monmap_and_config 2024-08-19T12:38:36.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.948+0000 7f87fb59e640 10 monclient: build_initial_monmap 2024-08-19T12:38:36.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.948+0000 7f87fb59e640 10 monclient: monmap: 2024-08-19T12:38:36.949 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:36.949738+0000 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:36.949738+0000 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:36.950 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:36.951 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.949+0000 7f87fb59e640 1 Processor -- start 2024-08-19T12:38:36.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.949+0000 7f87fb59e640 1 -- start start 2024-08-19T12:38:36.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.949+0000 7f87fb59e640 10 monclient: init 2024-08-19T12:38:36.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.950+0000 7f87fb59e640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:36.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.950+0000 7f87fb59e640 10 monclient: _add_conns ranks=[7,8,1,6,5,2,4,0,3] 2024-08-19T12:38:36.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.950+0000 7f87fb59e640 10 monclient(hunting): picked mon.noname-i con 0x7f87fc073340 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:36.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.950+0000 7f87fb59e640 10 monclient(hunting): picked mon.noname-e con 0x7f87fc1361c0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:36.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.950+0000 7f87fb59e640 10 monclient(hunting): picked mon.noname-f con 0x7f87fc13f8e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:36.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.950+0000 7f87fb59e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87fc136610 con 0x7f87fc13f8e0 2024-08-19T12:38:36.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.950+0000 7f87f9d9b640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f87fc1361c0 0x7f87fc13e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34062/0 (socket says 172.21.15.26:34062) 2024-08-19T12:38:36.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.950+0000 7f87f9d9b640 1 -- 172.21.15.26:0/829654304 learned_addr learned my addr 172.21.15.26:0/829654304 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:36.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.951+0000 7f87fb59e640 1 -- 172.21.15.26:0/829654304 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87fc135d80 con 0x7f87fc073340 2024-08-19T12:38:36.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.951+0000 7f87fb59e640 1 -- 172.21.15.26:0/829654304 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87fc09e210 con 0x7f87fc1361c0 2024-08-19T12:38:36.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.951+0000 7f87fb59e640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:36.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.951+0000 7f87fb59e640 10 monclient(hunting): authenticate will time out at 711.038214s 2024-08-19T12:38:36.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f9d9b640 1 -- 172.21.15.26:0/829654304 >> v1:172.21.15.26:6793/0 conn(0x7f87fc1361c0 legacy=0x7f87fc13e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87fa59c640 1 -- 172.21.15.26:0/829654304 >> v1:172.21.15.104:6792/0 conn(0x7f87fc073340 legacy=0x7f87fc0a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1477248693 0 0) 0x7f87fc09e210 con 0x7f87fc1361c0 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 10 monclient(hunting): my global_id is 4454 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87e4003520 con 0x7f87fc1361c0 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 458415999 0 0) 0x7f87fc135d80 con 0x7f87fc073340 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 10 monclient(hunting): my global_id is 4327 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87fc09e210 con 0x7f87fc073340 2024-08-19T12:38:36.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87fad9d640 1 -- 172.21.15.26:0/829654304 >> v1:172.21.15.104:6789/0 conn(0x7f87fc13f8e0 legacy=0x7f87fc141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.956 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.3 2024-08-19T12:38:36.956 DEBUG:teuthology.parallel:result is None 2024-08-19T12:38:36.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3262961120 0 0) 0x7f87fc136610 con 0x7f87fc13f8e0 2024-08-19T12:38:36.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 10 monclient(hunting): my global_id is 4339 2024-08-19T12:38:36.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87e4004a70 con 0x7f87fc13f8e0 2024-08-19T12:38:36.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2096766895 0 0) 0x7f87fc09e210 con 0x7f87fc073340 2024-08-19T12:38:36.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.953+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 >> v1:172.21.15.26:6793/0 conn(0x7f87fc1361c0 legacy=0x7f87fc13e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 >> v1:172.21.15.104:6789/0 conn(0x7f87fc13f8e0 legacy=0x7f87fc141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 10 monclient: _finish_hunting 0 2024-08-19T12:38:36.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 1 monclient: found mon.noname-i 2024-08-19T12:38:36.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:36.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:38:36.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87fc09dc50 con 0x7f87fc073340 2024-08-19T12:38:36.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 10 monclient: _finish_auth 0 2024-08-19T12:38:36.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 10 monclient: _check_auth_tickets 2024-08-19T12:38:36.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:36.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f87ec001ad0 con 0x7f87fc073340 2024-08-19T12:38:36.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:36.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 10 monclient: dump: 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87fb59e640 5 monclient: authenticate success, global_id 4327 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87fb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f87ec001d20 con 0x7f87fc073340 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 1 -- 172.21.15.26:0/829654304 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f87ec003950 con 0x7f87fc073340 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87f959a640 10 monclient: dump: 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.968 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.969 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87fb59e640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87fb59e640 4 set_mon_vals no callback set 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.954+0000 7f87fb59e640 10 monclient: shutdown 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.955+0000 7f87fb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.955+0000 7f87fb59e640 1 -- 172.21.15.26:0/829654304 >> v1:172.21.15.104:6792/0 conn(0x7f87fc073340 legacy=0x7f87fc0a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.955+0000 7f87fb59e640 1 -- 172.21.15.26:0/829654304 shutdown_connections 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.955+0000 7f87fb59e640 1 -- 172.21.15.26:0/829654304 >> 172.21.15.26:0/829654304 conn(0x7f87fc0abe30 msgr2=0x7f87fc0ac240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.955+0000 7f87fb59e640 1 -- 172.21.15.26:0/829654304 shutdown_connections 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.955+0000 7f87fb59e640 1 -- 172.21.15.26:0/829654304 wait complete. 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.956+0000 7f87fb59e640 10 monclient: build_initial_monmap 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.956+0000 7f87fb59e640 10 monclient: monmap: 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:36.970 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:36.971 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.956+0000 7f87fb59e640 1 librados: starting msgr at 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.956+0000 7f87fb59e640 1 librados: starting objecter 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.956+0000 7f87fb59e640 1 Processor -- start 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.956+0000 7f87fb59e640 1 -- start start 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.956+0000 7f87fb59e640 1 librados: setting wanted keys 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.956+0000 7f87fb59e640 1 librados: calling monclient init 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.956+0000 7f87fb59e640 10 monclient: init 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.957+0000 7f87fb59e640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.957+0000 7f87fb59e640 10 monclient: _add_conns ranks=[1,0,6,8,4,2,7,5,3] 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.957+0000 7f87fb59e640 10 monclient(hunting): picked mon.noname-f con 0x7f87fc073340 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.957+0000 7f87fb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f87fc1361c0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.957+0000 7f87fb59e640 10 monclient(hunting): picked mon.noname-d con 0x7f87fc13f8e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:36.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.957+0000 7f87fb59e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87fc0a4f60 con 0x7f87fc1361c0 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.957+0000 7f87fb59e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87fc13e810 con 0x7f87fc073340 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.957+0000 7f87fb59e640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87fc0a56d0 con 0x7f87fc13f8e0 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.957+0000 7f87fb59e640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.957+0000 7f87fb59e640 10 monclient(hunting): authenticate will time out at 711.044489s 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.957+0000 7f87f9d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f87fc1361c0 0x7f87fc1cec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58488/0 (socket says 172.21.15.26:58488) 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.957+0000 7f87f9d9b640 1 -- 172.21.15.26:0/2947974289 learned_addr learned my addr 172.21.15.26:0/2947974289 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.959+0000 7f87f9d9b640 1 -- 172.21.15.26:0/2947974289 >> v1:172.21.15.26:6789/0 conn(0x7f87fc1361c0 legacy=0x7f87fc1cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.959+0000 7f87eaffd640 10 client.?.objecter ms_handle_connect 0x7f87fc1361c0 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.959+0000 7f87eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.959+0000 7f87eaffd640 10 client.?.objecter ms_handle_connect 0x7f87fc13f8e0 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.959+0000 7f87eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.959+0000 7f87fad9d640 1 -- 172.21.15.26:0/2947974289 >> v1:172.21.15.26:6792/0 conn(0x7f87fc13f8e0 legacy=0x7f87fc1d2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87fa59c640 1 -- 172.21.15.26:0/2947974289 >> v1:172.21.15.104:6789/0 conn(0x7f87fc073340 legacy=0x7f87fc1c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 client.?.objecter ms_handle_connect 0x7f87fc073340 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366952412 0 0) 0x7f87fc0a4f60 con 0x7f87fc1361c0 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): my global_id is 4356 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87dc004eb0 con 0x7f87fc1361c0 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 960584995 0 0) 0x7f87fc0a56d0 con 0x7f87fc13f8e0 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): my global_id is 4443 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87fc0a4f60 con 0x7f87fc13f8e0 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1057265914 0 0) 0x7f87fc13e810 con 0x7f87fc073340 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): my global_id is 4348 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87fc0a56d0 con 0x7f87fc073340 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1150478244 0 0) 0x7f87dc004eb0 con 0x7f87fc1361c0 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87fc13e810 con 0x7f87fc1361c0 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f87f0001d30 con 0x7f87fc1361c0 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): dump: 2024-08-19T12:38:36.975 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.976 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2878935162 0 0) 0x7f87fc0a4f60 con 0x7f87fc13f8e0 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87dc004eb0 con 0x7f87fc13f8e0 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f87f4003990 con 0x7f87fc13f8e0 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.960+0000 7f87eaffd640 10 monclient(hunting): dump: 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.977 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1813670681 0 0) 0x7f87fc0a56d0 con 0x7f87fc073340 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87fc0a4f60 con 0x7f87fc073340 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2514353530 0 0) 0x7f87fc13e810 con 0x7f87fc1361c0 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 >> v1:172.21.15.26:6792/0 conn(0x7f87fc13f8e0 legacy=0x7f87fc1d2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 >> v1:172.21.15.104:6789/0 conn(0x7f87fc073340 legacy=0x7f87fc1c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 1 monclient: found mon.a 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87fc0a2c00 con 0x7f87fc1361c0 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87fb59e640 5 monclient: authenticate success, global_id 4356 2024-08-19T12:38:36.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87fb59e640 10 monclient: _renew_subs 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87fb59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87fb59e640 1 -- 172.21.15.26:0/2947974289 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f87fc0a4c80 con 0x7f87fc1361c0 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f87f0004a90 con 0x7f87fc1361c0 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f87f0004fc0 con 0x7f87fc1361c0 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87eaffd640 10 monclient: dump: 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:36.980 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87f8d99640 4 set_mon_vals no callback set 2024-08-19T12:38:36.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87fb59e640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87fb59e640 10 monclient: _renew_subs 2024-08-19T12:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87fb59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87fb59e640 1 -- 172.21.15.26:0/2947974289 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f87fc0a2170 con 0x7f87fc1361c0 2024-08-19T12:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.961+0000 7f87fb59e640 1 librados: init done 2024-08-19T12:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.962+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f87f0011420 con 0x7f87fc1361c0 2024-08-19T12:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.962+0000 7f87fb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.963+0000 7f87fb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.963+0000 7f87fb59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.963+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 1475869027 0 0) 0x7f87f004d920 con 0x7f87fc1361c0 2024-08-19T12:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.963+0000 7f87eaffd640 10 client.4356.objecter ms_dispatch 0x7f87fc000b90 osd_map(19..19 src has 1..19) 2024-08-19T12:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.963+0000 7f87eaffd640 3 client.4356.objecter handle_osd_map got epochs [19,19] > 0 2024-08-19T12:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.963+0000 7f87eaffd640 3 client.4356.objecter handle_osd_map decoding full epoch 19 2024-08-19T12:38:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.963+0000 7f87fb59e640 1 -- 172.21.15.26:0/2947974289 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87fc074280 con 0x7f87fc1361c0 2024-08-19T12:38:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.963+0000 7f87eaffd640 20 client.4356.objecter dump_active .. 0 homeless 2024-08-19T12:38:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.963+0000 7f87fa59c640 10 monclient: get_auth_request con 0x7f87dc041930 auth_method 0 2024-08-19T12:38:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.965+0000 7f87eaffd640 10 client.4356.objecter ms_handle_connect 0x7f87dc041930 2024-08-19T12:38:36.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.968+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f87fc074280 con 0x7f87fc1361c0 2024-08-19T12:38:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.968+0000 7f87eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:36.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:36.968+0000 7f87eaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.002+0000 7f130797f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T12:38:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.002+0000 7f130797f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T12:38:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.002+0000 7f130797f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:37.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.002+0000 7f130797f640 1 -- 172.21.15.26:0/3151564670 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f13000a2a70 con 0x7f130013f2e0 2024-08-19T12:38:37.004 INFO:teuthology.orchestra.run.smithi026.stdout:51539607556 2024-08-19T12:38:37.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.003+0000 7f12f5ffb640 1 -- 172.21.15.26:0/3151564670 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 203001719) 0x7f12f801ff80 con 0x7f130013f2e0 2024-08-19T12:38:37.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.003+0000 7f12f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-19T12:38:37.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.003+0000 7f12f5ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.006+0000 7f130797f640 10 librados: watch_flush enter 2024-08-19T12:38:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.006+0000 7f130797f640 10 librados: watch_flush exit 2024-08-19T12:38:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.006+0000 7f130797f640 20 client.4436.objecter shutdown clearing up homeless session... 2024-08-19T12:38:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.006+0000 7f130797f640 10 client.4436.objecter successfully canceled tick 2024-08-19T12:38:37.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.006+0000 7f130797f640 1 -- 172.21.15.26:0/3151564670 >> v1:172.21.15.104:6800/899887506 conn(0x7f12e4041940 legacy=0x7f12e4043e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.006+0000 7f130797f640 10 monclient: shutdown 2024-08-19T12:38:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.006+0000 7f130797f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.006+0000 7f130797f640 1 -- 172.21.15.26:0/3151564670 >> v1:172.21.15.26:6793/0 conn(0x7f130013f2e0 legacy=0x7f13001d6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.006+0000 7f130797f640 1 -- 172.21.15.26:0/3151564670 shutdown_connections 2024-08-19T12:38:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.006+0000 7f130797f640 1 -- 172.21.15.26:0/3151564670 >> 172.21.15.26:0/3151564670 conn(0x7f13000abff0 msgr2=0x7f1300134fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.006+0000 7f130797f640 1 -- 172.21.15.26:0/3151564670 shutdown_connections 2024-08-19T12:38:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.007+0000 7f130797f640 1 -- 172.21.15.26:0/3151564670 wait complete. 2024-08-19T12:38:37.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.007+0000 7f130797f640 1 librados: shutdown 2024-08-19T12:38:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.008+0000 7fc1825a0640 10 monclient: get_monmap_and_config 2024-08-19T12:38:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.008+0000 7fc1825a0640 10 monclient: build_initial_monmap 2024-08-19T12:38:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.008+0000 7fc1825a0640 10 monclient: monmap: 2024-08-19T12:38:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:37.009455+0000 2024-08-19T12:38:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:37.009455+0000 2024-08-19T12:38:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.009 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:37.010 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:37.010 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.009+0000 7fc1825a0640 1 Processor -- start 2024-08-19T12:38:37.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.009+0000 7fc1825a0640 1 -- start start 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.009+0000 7fc1825a0640 10 monclient: init 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.009+0000 7fc1825a0640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.009+0000 7fc1825a0640 10 monclient: _add_conns ranks=[1,0,6,2,5,7,3,4,8] 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.010+0000 7fc1825a0640 10 monclient(hunting): picked mon.noname-f con 0x7fc17c0a77f0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.010+0000 7fc1825a0640 10 monclient(hunting): picked mon.noname-a con 0x7fc17c0773c0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.010+0000 7fc1825a0640 10 monclient(hunting): picked mon.noname-d con 0x7fc17c0a82e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.010+0000 7fc1825a0640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c072e00 con 0x7fc17c0773c0 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.010+0000 7fc181d9f640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fc17c0a82e0 0x7fc17c13fa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58640/0 (socket says 172.21.15.26:58640) 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.010+0000 7fc181d9f640 1 -- 172.21.15.26:0/1110608516 learned_addr learned my addr 172.21.15.26:0/1110608516 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.010+0000 7fc1825a0640 1 -- 172.21.15.26:0/1110608516 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c076ff0 con 0x7fc17c0a77f0 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.010+0000 7fc1825a0640 1 -- 172.21.15.26:0/1110608516 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c0729e0 con 0x7fc17c0a82e0 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.010+0000 7fc1825a0640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:37.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.010+0000 7fc1825a0640 10 monclient(hunting): authenticate will time out at 711.097400s 2024-08-19T12:38:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc180d9d640 1 -- 172.21.15.26:0/1110608516 >> v1:172.21.15.26:6789/0 conn(0x7fc17c0773c0 legacy=0x7fc17c13c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc181d9f640 1 -- 172.21.15.26:0/1110608516 >> v1:172.21.15.26:6792/0 conn(0x7fc17c0a82e0 legacy=0x7fc17c13fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc18159e640 1 -- 172.21.15.26:0/1110608516 >> v1:172.21.15.104:6789/0 conn(0x7fc17c0a77f0 legacy=0x7fc17c0a7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc173fff640 1 -- 172.21.15.26:0/1110608516 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 112186667 0 0) 0x7fc16c0014a0 con 0x7fc17c0773c0 2024-08-19T12:38:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc173fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc173fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc173fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc173fff640 10 monclient(hunting): my global_id is 4365 2024-08-19T12:38:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc173fff640 1 -- 172.21.15.26:0/1110608516 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc168003520 con 0x7fc17c0773c0 2024-08-19T12:38:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc173fff640 1 -- 172.21.15.26:0/1110608516 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 92097211 0 0) 0x7fc17c0729e0 con 0x7fc17c0a82e0 2024-08-19T12:38:37.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc173fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc173fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc173fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc173fff640 10 monclient(hunting): my global_id is 4452 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.012+0000 7fc173fff640 1 -- 172.21.15.26:0/1110608516 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc16c0014a0 con 0x7fc17c0a82e0 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 1 -- 172.21.15.26:0/1110608516 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3163929508 0 0) 0x7fc168003520 con 0x7fc17c0773c0 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 1 -- 172.21.15.26:0/1110608516 >> v1:172.21.15.26:6792/0 conn(0x7fc17c0a82e0 legacy=0x7fc17c13fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 1 -- 172.21.15.26:0/1110608516 >> v1:172.21.15.104:6789/0 conn(0x7fc17c0a77f0 legacy=0x7fc17c0a7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 1 monclient: found mon.noname-a 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 1 -- 172.21.15.26:0/1110608516 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c076d70 con 0x7fc17c0773c0 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 10 monclient: _finish_auth 0 2024-08-19T12:38:37.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 1 -- 172.21.15.26:0/1110608516 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc16c003610 con 0x7fc17c0773c0 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc1825a0640 5 monclient: authenticate success, global_id 4365 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc1825a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 10 monclient: dump: 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.016 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc1825a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 1 -- 172.21.15.26:0/1110608516 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc16c003830 con 0x7fc17c0773c0 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 1 -- 172.21.15.26:0/1110608516 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc16c003d60 con 0x7fc17c0773c0 2024-08-19T12:38:37.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc1825a0640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc1825a0640 4 set_mon_vals no callback set 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.013+0000 7fc173fff640 10 monclient: dump: 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.018 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.019 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.019 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.014+0000 7fc1825a0640 10 monclient: shutdown 2024-08-19T12:38:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.014+0000 7fc1825a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:37.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.014+0000 7fc1825a0640 1 -- 172.21.15.26:0/1110608516 >> v1:172.21.15.26:6789/0 conn(0x7fc17c0773c0 legacy=0x7fc17c13c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.014+0000 7fc1825a0640 1 -- 172.21.15.26:0/1110608516 shutdown_connections 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.014+0000 7fc1825a0640 1 -- 172.21.15.26:0/1110608516 >> 172.21.15.26:0/1110608516 conn(0x7fc17c0688d0 msgr2=0x7fc17c068ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.014+0000 7fc1825a0640 1 -- 172.21.15.26:0/1110608516 shutdown_connections 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.014+0000 7fc1825a0640 1 -- 172.21.15.26:0/1110608516 wait complete. 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.014+0000 7fc1825a0640 10 monclient: build_initial_monmap 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.014+0000 7fc1825a0640 10 monclient: monmap: 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.022 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 1 librados: starting msgr at 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 1 librados: starting objecter 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 1 Processor -- start 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 1 -- start start 2024-08-19T12:38:37.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 1 librados: setting wanted keys 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 1 librados: calling monclient init 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 10 monclient: init 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 10 monclient: _add_conns ranks=[6,2,8,5,0,1,3,4,7] 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 10 monclient(hunting): picked mon.noname-d con 0x7fc17c0773c0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 10 monclient(hunting): picked mon.noname-b con 0x7fc17c0a77f0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 10 monclient(hunting): picked mon.noname-e con 0x7fc17c0a82e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c120e20 con 0x7fc17c0a77f0 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c121100 con 0x7fc17c0773c0 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c1ceef0 con 0x7fc17c0a82e0 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc180d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc17c0a77f0 0x7fc17c1ce7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50382/0 (socket says 172.21.15.26:50382) 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc180d9d640 1 -- 172.21.15.26:0/4265446290 learned_addr learned my addr 172.21.15.26:0/4265446290 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:37.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.015+0000 7fc1825a0640 10 monclient(hunting): authenticate will time out at 711.102844s 2024-08-19T12:38:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc180d9d640 1 -- 172.21.15.26:0/4265446290 >> v1:172.21.15.26:6790/0 conn(0x7fc17c0a77f0 legacy=0x7fc17c1ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 client.?.objecter ms_handle_connect 0x7fc17c0a77f0 2024-08-19T12:38:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7f338e37f640 10 monclient: get_monmap_and_config 2024-08-19T12:38:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7f338e37f640 10 monclient: build_initial_monmap 2024-08-19T12:38:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7f338e37f640 10 monclient: monmap: 2024-08-19T12:38:37.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:37.019126+0000 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:37.019126+0000 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:37.026 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:37.027 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:37.027 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2353982391 0 0) 0x7fc17c120e20 con 0x7fc17c0a77f0 2024-08-19T12:38:37.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 monclient(hunting): my global_id is 4367 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc160004ee0 con 0x7fc17c0a77f0 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 client.?.objecter ms_handle_connect 0x7fc17c0a82e0 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc181d9f640 1 -- 172.21.15.26:0/4265446290 >> v1:172.21.15.26:6793/0 conn(0x7fc17c0a82e0 legacy=0x7fc17c1d63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 client.?.objecter ms_handle_connect 0x7fc17c0773c0 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc18159e640 1 -- 172.21.15.26:0/4265446290 >> v1:172.21.15.26:6792/0 conn(0x7fc17c0773c0 legacy=0x7fc17c1ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3978458687 0 0) 0x7fc17c1ceef0 con 0x7fc17c0a82e0 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 monclient(hunting): my global_id is 4463 2024-08-19T12:38:37.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc17c120e20 con 0x7fc17c0a82e0 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2468486132 0 0) 0x7fc160004ee0 con 0x7fc17c0a77f0 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc17c1ceef0 con 0x7fc17c0a77f0 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc16c004410 con 0x7fc17c0a77f0 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.032 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.018+0000 7fc171ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2863133240 0 0) 0x7fc17c1ceef0 con 0x7fc17c0a77f0 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 >> v1:172.21.15.26:6793/0 conn(0x7fc17c0a82e0 legacy=0x7fc17c1d63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 >> v1:172.21.15.26:6792/0 conn(0x7fc17c0773c0 legacy=0x7fc17c1ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:37.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 1 monclient: found mon.b 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c120870 con 0x7fc17c0a77f0 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc1825a0640 5 monclient: authenticate success, global_id 4367 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc1825a0640 10 monclient: _renew_subs 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc1825a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc1825a0640 1 -- 172.21.15.26:0/4265446290 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc17c1213e0 con 0x7fc17c0a77f0 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc1825a0640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc1825a0640 10 monclient: _renew_subs 2024-08-19T12:38:37.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc1825a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc1825a0640 1 -- 172.21.15.26:0/4265446290 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc17c1403f0 con 0x7fc17c0a77f0 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc1825a0640 1 librados: init done 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc16c003610 con 0x7fc17c0a77f0 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc16c003c90 con 0x7fc17c0a77f0 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc171ffb640 10 monclient: dump: 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.035 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.036 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.019+0000 7fc1737fe640 4 set_mon_vals no callback set 2024-08-19T12:38:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.020+0000 7f338e37f640 1 Processor -- start 2024-08-19T12:38:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.020+0000 7f338e37f640 1 -- start start 2024-08-19T12:38:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.020+0000 7f338e37f640 10 monclient: init 2024-08-19T12:38:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.020+0000 7fc1825a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.020+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fc16c011dc0 con 0x7fc17c0a77f0 2024-08-19T12:38:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.020+0000 7fc1825a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.020+0000 7fc1825a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:37.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.020+0000 7fc1825a0640 1 -- 172.21.15.26:0/4265446290 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc17c0008d0 con 0x7fc17c0a77f0 2024-08-19T12:38:37.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7f338e37f640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:37.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7f338e37f640 10 monclient: _add_conns ranks=[3,6,7,0,2,1,4,5,8] 2024-08-19T12:38:37.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7f338e37f640 10 monclient(hunting): picked mon.noname-g con 0x7f3388135a30 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:37.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7fc16c04dcd0 con 0x7fc17c0a77f0 2024-08-19T12:38:37.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7fc171ffb640 10 client.4367.objecter ms_dispatch 0x7fc17c000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:37.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7fc171ffb640 3 client.4367.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:37.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7fc171ffb640 3 client.4367.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:37.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7f338e37f640 10 monclient(hunting): picked mon.noname-d con 0x7f3388136520 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:37.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7f338e37f640 10 monclient(hunting): picked mon.noname-i con 0x7f338813c390 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:37.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7f338e37f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f338806e110 con 0x7f3388135a30 2024-08-19T12:38:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7fc18159e640 10 monclient: get_auth_request con 0x7fc160040fe0 auth_method 0 2024-08-19T12:38:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7f338e37f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33880a1a70 con 0x7f3388136520 2024-08-19T12:38:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7f338e37f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33880a8050 con 0x7f338813c390 2024-08-19T12:38:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7f338e37f640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7f338e37f640 10 monclient(hunting): authenticate will time out at 711.108321s 2024-08-19T12:38:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7f33877fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f3388136520 0x7f338813ac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:58654/0 (socket says 172.21.15.26:58654) 2024-08-19T12:38:37.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7f33877fe640 1 -- 172.21.15.26:0/348485771 learned_addr learned my addr 172.21.15.26:0/348485771 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:37.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.021+0000 7fc171ffb640 20 client.4367.objecter dump_active .. 0 homeless 2024-08-19T12:38:37.042 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.5 2024-08-19T12:38:37.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3387fff640 1 -- 172.21.15.26:0/348485771 >> v1:172.21.15.104:6790/0 conn(0x7f3388135a30 legacy=0x7f3388135e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f33877fe640 1 -- 172.21.15.26:0/348485771 >> v1:172.21.15.26:6792/0 conn(0x7f3388136520 legacy=0x7f338813ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.042 DEBUG:teuthology.parallel:result is None 2024-08-19T12:38:37.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7fc171ffb640 10 client.4367.objecter ms_handle_connect 0x7fc160040fe0 2024-08-19T12:38:37.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f338c91c640 1 -- 172.21.15.26:0/348485771 >> v1:172.21.15.104:6792/0 conn(0x7f338813c390 legacy=0x7f338813e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 624973923 0 0) 0x7f338806e110 con 0x7f3388135a30 2024-08-19T12:38:37.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3386ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3386ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3386ffd640 10 monclient(hunting): my global_id is 4431 2024-08-19T12:38:37.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f336c003520 con 0x7f3388135a30 2024-08-19T12:38:37.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 673874773 0 0) 0x7f33880a1a70 con 0x7f3388136520 2024-08-19T12:38:37.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3386ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3386ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3386ffd640 10 monclient(hunting): my global_id is 4470 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.023+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f338806e110 con 0x7f3388136520 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 299944094 0 0) 0x7f33880a8050 con 0x7f338813c390 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient(hunting): my global_id is 4336 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33880a1a70 con 0x7f338813c390 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3624699708 0 0) 0x7f336c003520 con 0x7f3388135a30 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 >> v1:172.21.15.104:6792/0 conn(0x7f338813c390 legacy=0x7f338813e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 >> v1:172.21.15.26:6792/0 conn(0x7f3388136520 legacy=0x7f338813ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:37.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 1 monclient: found mon.noname-g 2024-08-19T12:38:37.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:37.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:38:37.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f338809fb60 con 0x7f3388135a30 2024-08-19T12:38:37.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:37.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:37.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:37.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3380001eb0 con 0x7f3388135a30 2024-08-19T12:38:37.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:37.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:38:37.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient: dump: 2024-08-19T12:38:37.048 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.048 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.049 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f338e37f640 5 monclient: authenticate success, global_id 4431 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f338e37f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3380003990 con 0x7f3388135a30 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 1 -- 172.21.15.26:0/348485771 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3380003f00 con 0x7f3388135a30 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f338e37f640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f338e37f640 4 set_mon_vals no callback set 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.024+0000 7f3386ffd640 10 monclient: dump: 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.050 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.025+0000 7f338e37f640 10 monclient: shutdown 2024-08-19T12:38:37.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.025+0000 7f338e37f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.025+0000 7f338e37f640 1 -- 172.21.15.26:0/348485771 >> v1:172.21.15.104:6790/0 conn(0x7f3388135a30 legacy=0x7f3388135e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.025+0000 7f338e37f640 1 -- 172.21.15.26:0/348485771 shutdown_connections 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.025+0000 7f338e37f640 1 -- 172.21.15.26:0/348485771 >> 172.21.15.26:0/348485771 conn(0x7f33880a4d70 msgr2=0x7f33880a5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.025+0000 7f338e37f640 1 -- 172.21.15.26:0/348485771 shutdown_connections 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.025+0000 7f338e37f640 1 -- 172.21.15.26:0/348485771 wait complete. 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.025+0000 7f338e37f640 10 monclient: build_initial_monmap 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.025+0000 7f338e37f640 10 monclient: monmap: 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:37.052 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.025+0000 7f338e37f640 1 librados: starting msgr at 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.025+0000 7f338e37f640 1 librados: starting objecter 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 1 Processor -- start 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 1 -- start start 2024-08-19T12:38:37.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 1 librados: setting wanted keys 2024-08-19T12:38:37.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 1 librados: calling monclient init 2024-08-19T12:38:37.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 10 monclient: init 2024-08-19T12:38:37.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:37.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 10 monclient: _add_conns ranks=[7,1,8,5,2,6,3,4,0] 2024-08-19T12:38:37.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 10 monclient(hunting): picked mon.noname-i con 0x7f3388135a30 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:37.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 10 monclient(hunting): picked mon.noname-f con 0x7f3388136520 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:37.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 10 monclient(hunting): picked mon.noname-e con 0x7f338813c390 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:37.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f338813a500 con 0x7f3388136520 2024-08-19T12:38:37.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3388135130 con 0x7f3388135a30 2024-08-19T12:38:37.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3388074350 con 0x7f338813c390 2024-08-19T12:38:37.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:37.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7f338e37f640 10 monclient(hunting): authenticate will time out at 711.113876s 2024-08-19T12:38:37.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fc17c072e00 con 0x7fc17c0a77f0 2024-08-19T12:38:37.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7fc171ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:37.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.026+0000 7fc171ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:37.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.027+0000 7f338c91c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f338813c390 0x7f33881d6800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34086/0 (socket says 172.21.15.26:34086) 2024-08-19T12:38:37.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.027+0000 7f338c91c640 1 -- 172.21.15.26:0/4209825151 learned_addr learned my addr 172.21.15.26:0/4209825151 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:37.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 client.?.objecter ms_handle_connect 0x7f338813c390 2024-08-19T12:38:37.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:37.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f338c91c640 1 -- 172.21.15.26:0/4209825151 >> v1:172.21.15.26:6793/0 conn(0x7f338813c390 legacy=0x7f33881d6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 client.?.objecter ms_handle_connect 0x7f3388135a30 2024-08-19T12:38:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3387fff640 1 -- 172.21.15.26:0/4209825151 >> v1:172.21.15.104:6792/0 conn(0x7f3388135a30 legacy=0x7f3388134a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3148025873 0 0) 0x7f33880a1a70 con 0x7f338813c390 2024-08-19T12:38:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient(hunting): my global_id is 4472 2024-08-19T12:38:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3370004eb0 con 0x7f338813c390 2024-08-19T12:38:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 543344866 0 0) 0x7f3388135130 con 0x7f3388135a30 2024-08-19T12:38:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient(hunting): my global_id is 4345 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33880a1a70 con 0x7f3388135a30 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4239950568 0 0) 0x7f3370004eb0 con 0x7f338813c390 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3388135130 con 0x7f338813c390 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3378003610 con 0x7f338813c390 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.063 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:37.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 824588258 0 0) 0x7f3388135130 con 0x7f338813c390 2024-08-19T12:38:37.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f33877fe640 1 -- 172.21.15.26:0/4209825151 >> v1:172.21.15.104:6789/0 conn(0x7f3388136520 legacy=0x7f33881d30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 >> v1:172.21.15.104:6792/0 conn(0x7f3388135a30 legacy=0x7f3388134a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 >> v1:172.21.15.104:6789/0 conn(0x7f3388136520 legacy=0x7f33881d30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:38:37.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 1 monclient: found mon.e 2024-08-19T12:38:37.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:37.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:37.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f338813b8c0 con 0x7f338813c390 2024-08-19T12:38:37.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient: _finish_auth 0 2024-08-19T12:38:37.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:38:37.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 client.?.objecter ms_handle_connect 0x7f3388136520 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f3384ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f338e37f640 5 monclient: authenticate success, global_id 4472 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f338e37f640 10 monclient: _renew_subs 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f338e37f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f338e37f640 1 -- 172.21.15.26:0/4209825151 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3388073f70 con 0x7f338813c390 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f338e37f640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f338e37f640 10 monclient: _renew_subs 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.029+0000 7f338e37f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.030+0000 7f338e37f640 1 -- 172.21.15.26:0/4209825151 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f3388139560 con 0x7f338813c390 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.030+0000 7f338e37f640 1 librados: init done 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.030+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3378003b10 con 0x7f338813c390 2024-08-19T12:38:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.030+0000 7f3384ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.030+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3378004080 con 0x7f338813c390 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.030+0000 7f3384ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.030+0000 7f3384ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.030+0000 7f3384ff9640 10 monclient: dump: 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.067 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.030+0000 7f33867fc640 4 set_mon_vals no callback set 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.030+0000 7f338e37f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.031+0000 7f338e37f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.031+0000 7f338e37f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.031+0000 7f338e37f640 1 -- 172.21.15.26:0/4209825151 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f338813b6a0 con 0x7f338813c390 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.032+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3378004220 con 0x7f338813c390 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.032+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f337804daf0 con 0x7f338813c390 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.032+0000 7f3384ff9640 10 client.4472.objecter ms_dispatch 0x7f3388000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:37.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.032+0000 7f3384ff9640 3 client.4472.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:37.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.032+0000 7f3384ff9640 3 client.4472.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:37.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.033+0000 7f3384ff9640 20 client.4472.objecter dump_active .. 0 homeless 2024-08-19T12:38:37.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.033+0000 7f3387fff640 10 monclient: get_auth_request con 0x7f3370040fd0 auth_method 0 2024-08-19T12:38:37.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.035+0000 7f3384ff9640 10 client.4472.objecter ms_handle_connect 0x7f3370040fd0 2024-08-19T12:38:37.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.040+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f3388074350 con 0x7f338813c390 2024-08-19T12:38:37.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.040+0000 7f3384ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:37.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.040+0000 7f3384ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:37.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.139+0000 7f7330c60640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T12:38:37.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.139+0000 7f7330c60640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T12:38:37.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.139+0000 7f7330c60640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:37.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.139+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f7310005370 con 0x7f732c073470 2024-08-19T12:38:37.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.139+0000 7f731affd640 1 -- 172.21.15.26:0/2368157577 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 1902013079) 0x7f731c01b3c0 con 0x7f732c073470 2024-08-19T12:38:37.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.139+0000 7f731affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-19T12:38:37.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.139+0000 7f731affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:37.141 INFO:teuthology.orchestra.run.smithi026.stdout:47244640262 2024-08-19T12:38:37.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.141+0000 7f7330c60640 10 librados: watch_flush enter 2024-08-19T12:38:37.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.141+0000 7f7330c60640 10 librados: watch_flush exit 2024-08-19T12:38:37.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.141+0000 7f7330c60640 20 client.4358.objecter shutdown clearing up homeless session... 2024-08-19T12:38:37.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.141+0000 7f7330c60640 10 client.4358.objecter successfully canceled tick 2024-08-19T12:38:37.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.142+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 >> v1:172.21.15.104:6800/899887506 conn(0x7f730c049c20 legacy=0x7f730c04c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.142+0000 7f7330c60640 10 monclient: shutdown 2024-08-19T12:38:37.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.142+0000 7f7330c60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:37.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.142+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 >> v1:172.21.15.26:6790/0 conn(0x7f732c073470 legacy=0x7f732c132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.142+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 shutdown_connections 2024-08-19T12:38:37.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.142+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 >> 172.21.15.26:0/2368157577 conn(0x7f732c0abff0 msgr2=0x7f732c133df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:37.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.142+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 shutdown_connections 2024-08-19T12:38:37.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.142+0000 7f7330c60640 1 -- 172.21.15.26:0/2368157577 wait complete. 2024-08-19T12:38:37.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.142+0000 7f7330c60640 1 librados: shutdown 2024-08-19T12:38:37.153 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640262 for osd.0 2024-08-19T12:38:37.153 DEBUG:teuthology.parallel:result is None 2024-08-19T12:38:37.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.177+0000 7f87fb59e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T12:38:37.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.177+0000 7f87fb59e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T12:38:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.177+0000 7f87fb59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.177+0000 7f87fb59e640 1 -- 172.21.15.26:0/2947974289 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f87fc0a24b0 con 0x7f87fc1361c0 2024-08-19T12:38:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.178+0000 7f87eaffd640 1 -- 172.21.15.26:0/2947974289 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 203001719) 0x7f87f0017360 con 0x7f87fc1361c0 2024-08-19T12:38:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.178+0000 7f87eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-19T12:38:37.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.178+0000 7f87eaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:37.179 INFO:teuthology.orchestra.run.smithi026.stdout:51539607556 2024-08-19T12:38:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 10 librados: watch_flush enter 2024-08-19T12:38:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 10 librados: watch_flush exit 2024-08-19T12:38:37.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 20 client.4356.objecter shutdown clearing up homeless session... 2024-08-19T12:38:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 10 client.4356.objecter successfully canceled tick 2024-08-19T12:38:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 1 -- 172.21.15.26:0/2947974289 >> v1:172.21.15.104:6800/899887506 conn(0x7f87dc041930 legacy=0x7f87dc043df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 10 monclient: shutdown 2024-08-19T12:38:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 1 -- 172.21.15.26:0/2947974289 >> v1:172.21.15.26:6789/0 conn(0x7f87fc1361c0 legacy=0x7f87fc1cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 1 -- 172.21.15.26:0/2947974289 shutdown_connections 2024-08-19T12:38:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 1 -- 172.21.15.26:0/2947974289 >> 172.21.15.26:0/2947974289 conn(0x7f87fc0abe30 msgr2=0x7f87fc134430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 1 -- 172.21.15.26:0/2947974289 shutdown_connections 2024-08-19T12:38:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 1 -- 172.21.15.26:0/2947974289 wait complete. 2024-08-19T12:38:37.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.180+0000 7f87fb59e640 1 librados: shutdown 2024-08-19T12:38:37.192 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.2 2024-08-19T12:38:37.192 DEBUG:teuthology.parallel:result is None 2024-08-19T12:38:37.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.218+0000 7fc1825a0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T12:38:37.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.218+0000 7fc1825a0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T12:38:37.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.218+0000 7fc1825a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:37.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.218+0000 7fc1825a0640 1 -- 172.21.15.26:0/4265446290 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc17c06d760 con 0x7fc17c0a77f0 2024-08-19T12:38:37.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.218+0000 7fc171ffb640 1 -- 172.21.15.26:0/4265446290 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 203001719) 0x7fc16c017b80 con 0x7fc17c0a77f0 2024-08-19T12:38:37.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.218+0000 7fc171ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-19T12:38:37.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.218+0000 7fc171ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:37.219 INFO:teuthology.orchestra.run.smithi026.stdout:51539607556 2024-08-19T12:38:37.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 10 librados: watch_flush enter 2024-08-19T12:38:37.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 10 librados: watch_flush exit 2024-08-19T12:38:37.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 20 client.4367.objecter shutdown clearing up homeless session... 2024-08-19T12:38:37.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 10 client.4367.objecter successfully canceled tick 2024-08-19T12:38:37.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 1 -- 172.21.15.26:0/4265446290 >> v1:172.21.15.104:6800/899887506 conn(0x7fc160040fe0 legacy=0x7fc1600434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 10 monclient: shutdown 2024-08-19T12:38:37.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:37.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 1 -- 172.21.15.26:0/4265446290 >> v1:172.21.15.26:6790/0 conn(0x7fc17c0a77f0 legacy=0x7fc17c1ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 1 -- 172.21.15.26:0/4265446290 shutdown_connections 2024-08-19T12:38:37.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 1 -- 172.21.15.26:0/4265446290 >> 172.21.15.26:0/4265446290 conn(0x7fc17c0688d0 msgr2=0x7fc17c0a4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:37.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 1 -- 172.21.15.26:0/4265446290 shutdown_connections 2024-08-19T12:38:37.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 1 -- 172.21.15.26:0/4265446290 wait complete. 2024-08-19T12:38:37.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.220+0000 7fc1825a0640 1 librados: shutdown 2024-08-19T12:38:37.231 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.1 2024-08-19T12:38:37.232 DEBUG:teuthology.parallel:result is None 2024-08-19T12:38:37.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.232+0000 7f338e37f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T12:38:37.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.232+0000 7f338e37f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T12:38:37.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.232+0000 7f338e37f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:37.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.232+0000 7f338e37f640 1 -- 172.21.15.26:0/4209825151 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f338813ac90 con 0x7f338813c390 2024-08-19T12:38:37.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.232+0000 7f3384ff9640 1 -- 172.21.15.26:0/4209825151 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 203001719) 0x7f3378017610 con 0x7f338813c390 2024-08-19T12:38:37.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.232+0000 7f3384ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-19T12:38:37.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.232+0000 7f3384ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:37.234 INFO:teuthology.orchestra.run.smithi026.stdout:51539607556 2024-08-19T12:38:37.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.234+0000 7f338e37f640 10 librados: watch_flush enter 2024-08-19T12:38:37.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.234+0000 7f338e37f640 10 librados: watch_flush exit 2024-08-19T12:38:37.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.234+0000 7f338e37f640 20 client.4472.objecter shutdown clearing up homeless session... 2024-08-19T12:38:37.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.234+0000 7f338e37f640 10 client.4472.objecter successfully canceled tick 2024-08-19T12:38:37.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.235+0000 7f338e37f640 1 -- 172.21.15.26:0/4209825151 >> v1:172.21.15.104:6800/899887506 conn(0x7f3370040fd0 legacy=0x7f3370043490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.235+0000 7f338e37f640 10 monclient: shutdown 2024-08-19T12:38:37.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.235+0000 7f338e37f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:37.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.235+0000 7f338e37f640 1 -- 172.21.15.26:0/4209825151 >> v1:172.21.15.26:6793/0 conn(0x7f338813c390 legacy=0x7f33881d6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.235+0000 7f338e37f640 1 -- 172.21.15.26:0/4209825151 shutdown_connections 2024-08-19T12:38:37.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.235+0000 7f338e37f640 1 -- 172.21.15.26:0/4209825151 >> 172.21.15.26:0/4209825151 conn(0x7f33880a4d70 msgr2=0x7f338809fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:37.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.235+0000 7f338e37f640 1 -- 172.21.15.26:0/4209825151 shutdown_connections 2024-08-19T12:38:37.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.235+0000 7f338e37f640 1 -- 172.21.15.26:0/4209825151 wait complete. 2024-08-19T12:38:37.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.235+0000 7f338e37f640 1 librados: shutdown 2024-08-19T12:38:37.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.4 2024-08-19T12:38:37.246 DEBUG:teuthology.parallel:result is None 2024-08-19T12:38:37.246 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-19T12:38:37.246 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T12:38:37.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.383+0000 7f1cf2667640 10 monclient: get_monmap_and_config 2024-08-19T12:38:37.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.383+0000 7f1cf2667640 10 monclient: build_initial_monmap 2024-08-19T12:38:37.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.383+0000 7f1cf2667640 10 monclient: monmap: 2024-08-19T12:38:37.384 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:37.384 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:37.384530+0000 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:37.384530+0000 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:37.385 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 1 Processor -- start 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 1 -- start start 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 10 monclient: init 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 10 monclient: _add_conns ranks=[7,4,6,3,8,0,1,2,5] 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 10 monclient(hunting): picked mon.noname-i con 0x7f1cec06e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 10 monclient(hunting): picked mon.noname-c con 0x7f1cec0a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 10 monclient(hunting): picked mon.noname-d con 0x7f1cec0a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cec136790 con 0x7f1cec0a5920 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cec09ddf0 con 0x7f1cec0a5d50 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cec09e0d0 con 0x7f1cec06e7b0 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.384+0000 7f1cf2667640 10 monclient(hunting): authenticate will time out at 711.471961s 2024-08-19T12:38:37.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.385+0000 7f1ceb7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f1cec0a5920 0x7f1cec13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57930/0 (socket says 172.21.15.26:57930) 2024-08-19T12:38:37.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.385+0000 7f1ceb7fe640 1 -- 172.21.15.26:0/924754147 learned_addr learned my addr 172.21.15.26:0/924754147 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:37.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceb7fe640 1 -- 172.21.15.26:0/924754147 >> v1:172.21.15.26:6791/0 conn(0x7f1cec0a5920 legacy=0x7f1cec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1cf0c04640 1 -- 172.21.15.26:0/924754147 >> v1:172.21.15.26:6792/0 conn(0x7f1cec0a5d50 legacy=0x7f1cec141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 1 -- 172.21.15.26:0/924754147 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1050915430 0 0) 0x7f1cec136790 con 0x7f1cec0a5920 2024-08-19T12:38:37.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient(hunting): my global_id is 4405 2024-08-19T12:38:37.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 1 -- 172.21.15.26:0/924754147 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cd4003520 con 0x7f1cec0a5920 2024-08-19T12:38:37.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 1 -- 172.21.15.26:0/924754147 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3391964175 0 0) 0x7f1ce4001820 con 0x7f1cec0a5d50 2024-08-19T12:38:37.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient(hunting): my global_id is 4479 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1cebfff640 1 -- 172.21.15.26:0/924754147 >> v1:172.21.15.104:6792/0 conn(0x7f1cec06e7b0 legacy=0x7f1cec136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 1 -- 172.21.15.26:0/924754147 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cec136790 con 0x7f1cec0a5d50 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 1 -- 172.21.15.26:0/924754147 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2150581566 0 0) 0x7f1cd4003520 con 0x7f1cec0a5920 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 1 -- 172.21.15.26:0/924754147 >> v1:172.21.15.104:6792/0 conn(0x7f1cec06e7b0 legacy=0x7f1cec136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 1 -- 172.21.15.26:0/924754147 >> v1:172.21.15.26:6792/0 conn(0x7f1cec0a5d50 legacy=0x7f1cec141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 1 monclient: found mon.noname-c 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 1 -- 172.21.15.26:0/924754147 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cec09e3b0 con 0x7f1cec0a5920 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:37.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1cf2667640 5 monclient: authenticate success, global_id 4405 2024-08-19T12:38:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 1 -- 172.21.15.26:0/924754147 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1ce0001ae0 con 0x7f1cec0a5920 2024-08-19T12:38:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1cf2667640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient: dump: 2024-08-19T12:38:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1cf2667640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:37.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 1 -- 172.21.15.26:0/924754147 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ce00048a0 con 0x7f1cec0a5920 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1cf2667640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 1 -- 172.21.15.26:0/924754147 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1ce0004e10 con 0x7f1cec0a5920 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1cf2667640 4 set_mon_vals no callback set 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.387+0000 7f1ceaffd640 10 monclient: dump: 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.403 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.388+0000 7f1cf2667640 10 monclient: shutdown 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.388+0000 7f1cf2667640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.388+0000 7f1cf2667640 1 -- 172.21.15.26:0/924754147 >> v1:172.21.15.26:6791/0 conn(0x7f1cec0a5920 legacy=0x7f1cec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.388+0000 7f1cf2667640 1 -- 172.21.15.26:0/924754147 shutdown_connections 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.388+0000 7f1cf2667640 1 -- 172.21.15.26:0/924754147 >> 172.21.15.26:0/924754147 conn(0x7f1cec0a8740 msgr2=0x7f1cec0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.388+0000 7f1cf2667640 1 -- 172.21.15.26:0/924754147 shutdown_connections 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.388+0000 7f1cf2667640 1 -- 172.21.15.26:0/924754147 wait complete. 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.389+0000 7f1cf2667640 10 monclient: build_initial_monmap 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.389+0000 7f1cf2667640 10 monclient: monmap: 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.389+0000 7f1cf2667640 1 librados: starting msgr at 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.389+0000 7f1cf2667640 1 librados: starting objecter 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.389+0000 7f1cf2667640 1 Processor -- start 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.389+0000 7f1cf2667640 1 -- start start 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.389+0000 7f1cf2667640 1 librados: setting wanted keys 2024-08-19T12:38:37.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.389+0000 7f1cf2667640 1 librados: calling monclient init 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.389+0000 7f1cf2667640 10 monclient: init 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.389+0000 7f1cf2667640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.389+0000 7f1cf2667640 10 monclient: _add_conns ranks=[4,2,7,5,0,1,8,6,3] 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.389+0000 7f1cf2667640 10 monclient(hunting): picked mon.noname-c con 0x7f1cec06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.390+0000 7f1cf2667640 10 monclient(hunting): picked mon.noname-b con 0x7f1cec0a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.390+0000 7f1cf2667640 10 monclient(hunting): picked mon.noname-i con 0x7f1cec0a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.390+0000 7f1cf2667640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cec13b2c0 con 0x7f1cec0a5920 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.390+0000 7f1cf2667640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cec1345c0 con 0x7f1cec06e7b0 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.390+0000 7f1cf2667640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cec138f60 con 0x7f1cec0a5d50 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.390+0000 7f1cf2667640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.390+0000 7f1cf2667640 10 monclient(hunting): authenticate will time out at 711.477102s 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.390+0000 7f1cebfff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f1cec06e7b0 0x7f1cec1ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57940/0 (socket says 172.21.15.26:57940) 2024-08-19T12:38:37.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.390+0000 7f1cebfff640 1 -- 172.21.15.26:0/1372402906 learned_addr learned my addr 172.21.15.26:0/1372402906 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.390+0000 7f1ceb7fe640 1 --1- 172.21.15.26:0/1372402906 >> v1:172.21.15.26:6790/0 conn(0x7f1cec0a5920 0x7f1cec1d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50398/0 (socket says 172.21.15.26:50398) 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1cebfff640 1 -- 172.21.15.26:0/1372402906 >> v1:172.21.15.26:6791/0 conn(0x7f1cec06e7b0 legacy=0x7f1cec1ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ceb7fe640 1 -- 172.21.15.26:0/1372402906 >> v1:172.21.15.26:6790/0 conn(0x7f1cec0a5920 legacy=0x7f1cec1d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 client.?.objecter ms_handle_connect 0x7f1cec06e7b0 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 client.?.objecter ms_handle_connect 0x7f1cec0a5920 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2952640522 0 0) 0x7f1cec1345c0 con 0x7f1cec06e7b0 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): my global_id is 4414 2024-08-19T12:38:37.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cd8004eb0 con 0x7f1cec06e7b0 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2628041276 0 0) 0x7f1cec13b2c0 con 0x7f1cec0a5920 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): my global_id is 4376 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cec1345c0 con 0x7f1cec0a5920 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1cf0c04640 1 -- 172.21.15.26:0/1372402906 >> v1:172.21.15.104:6792/0 conn(0x7f1cec0a5d50 legacy=0x7f1cec1d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 client.?.objecter ms_handle_connect 0x7f1cec0a5d50 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 508573204 0 0) 0x7f1cd8004eb0 con 0x7f1cec06e7b0 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cec13b2c0 con 0x7f1cec06e7b0 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1cdc003610 con 0x7f1cec06e7b0 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:37.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.410 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.411 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.411 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.411 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.411 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:37.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1857470243 0 0) 0x7f1cec1345c0 con 0x7f1cec0a5920 2024-08-19T12:38:37.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1cd8004eb0 con 0x7f1cec0a5920 2024-08-19T12:38:37.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1ce0004d90 con 0x7f1cec0a5920 2024-08-19T12:38:37.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1547356442 0 0) 0x7f1cec138f60 con 0x7f1cec0a5d50 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient(hunting): my global_id is 4363 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cec1345c0 con 0x7f1cec0a5d50 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1323538400 0 0) 0x7f1cec13b2c0 con 0x7f1cec06e7b0 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 >> v1:172.21.15.104:6792/0 conn(0x7f1cec0a5d50 legacy=0x7f1cec1d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 >> v1:172.21.15.26:6790/0 conn(0x7f1cec0a5920 legacy=0x7f1cec1d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:38:37.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 monclient: found mon.c 2024-08-19T12:38:37.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:37.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:37.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cec13aa20 con 0x7f1cec06e7b0 2024-08-19T12:38:37.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient: _finish_auth 0 2024-08-19T12:38:37.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:38:37.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1ce8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:37.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1cf2667640 5 monclient: authenticate success, global_id 4414 2024-08-19T12:38:37.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1cf2667640 10 monclient: _renew_subs 2024-08-19T12:38:37.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1cf2667640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:37.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.392+0000 7f1cf2667640 1 -- 172.21.15.26:0/1372402906 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1cec13ad00 con 0x7f1cec06e7b0 2024-08-19T12:38:37.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1cf2667640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:37.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1cf2667640 10 monclient: _renew_subs 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1cf2667640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1cf2667640 1 -- 172.21.15.26:0/1372402906 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f1cec139ac0 con 0x7f1cec06e7b0 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1cf2667640 1 librados: init done 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1cdc003b90 con 0x7f1cec06e7b0 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1ce8ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1cdc0040c0 con 0x7f1cec06e7b0 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1ce8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1cea7fc640 4 set_mon_vals no callback set 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1ce8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1ce8ff9640 10 monclient: dump: 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.415 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f1cdc004260 con 0x7f1cec06e7b0 2024-08-19T12:38:37.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1cf2667640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:37.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1cf2667640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:37.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1cf2667640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:37.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.393+0000 7f1cf2667640 1 -- 172.21.15.26:0/1372402906 --> v1:172.21.15.26:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ccc005080 con 0x7f1cec06e7b0 2024-08-19T12:38:37.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.394+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f1cdc04da90 con 0x7f1cec06e7b0 2024-08-19T12:38:37.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.394+0000 7f1ce8ff9640 10 client.4414.objecter ms_dispatch 0x7f1cec000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:37.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.394+0000 7f1ce8ff9640 3 client.4414.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:37.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.394+0000 7f1ce8ff9640 3 client.4414.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:37.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.394+0000 7f1ce8ff9640 20 client.4414.objecter dump_active .. 0 homeless 2024-08-19T12:38:37.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.394+0000 7f1ceb7fe640 10 monclient: get_auth_request con 0x7f1cd80415c0 auth_method 0 2024-08-19T12:38:37.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.396+0000 7f1ce8ff9640 10 client.4414.objecter ms_handle_connect 0x7f1cd80415c0 2024-08-19T12:38:37.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.399+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mon.4 v1:172.21.15.26:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f1cdc017650 con 0x7f1cec06e7b0 2024-08-19T12:38:37.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.399+0000 7f1ce8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:37.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.399+0000 7f1ce8ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:37.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.577+0000 7f1cf2667640 1 -- 172.21.15.26:0/1372402906 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1ccc000b90 con 0x7f1cd80415c0 2024-08-19T12:38:37.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.578+0000 7f1ce8ff9640 1 -- 172.21.15.26:0/1372402906 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34175 (unknown 2965378022 0 1573968203) 0x7f1ccc000b90 con 0x7f1cd80415c0 2024-08-19T12:38:37.580 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:38:37.580 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-08-19T12:38:37.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.580+0000 7f1cf2667640 10 librados: watch_flush enter 2024-08-19T12:38:37.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.580+0000 7f1cf2667640 10 librados: watch_flush exit 2024-08-19T12:38:37.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.580+0000 7f1cf2667640 20 client.4414.objecter shutdown clearing up homeless session... 2024-08-19T12:38:37.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.580+0000 7f1cf2667640 10 client.4414.objecter successfully canceled tick 2024-08-19T12:38:37.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.580+0000 7f1cf2667640 1 -- 172.21.15.26:0/1372402906 >> v1:172.21.15.104:6800/899887506 conn(0x7f1cd80415c0 legacy=0x7f1cd8043a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.580+0000 7f1cf2667640 10 monclient: shutdown 2024-08-19T12:38:37.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.580+0000 7f1cf2667640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:37.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.580+0000 7f1cf2667640 1 -- 172.21.15.26:0/1372402906 >> v1:172.21.15.26:6791/0 conn(0x7f1cec06e7b0 legacy=0x7f1cec1ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.580+0000 7f1cf2667640 1 -- 172.21.15.26:0/1372402906 shutdown_connections 2024-08-19T12:38:37.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.580+0000 7f1cf2667640 1 -- 172.21.15.26:0/1372402906 >> 172.21.15.26:0/1372402906 conn(0x7f1cec0a8740 msgr2=0x7f1cec0a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:37.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.580+0000 7f1cf2667640 1 -- 172.21.15.26:0/1372402906 shutdown_connections 2024-08-19T12:38:37.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.581+0000 7f1cf2667640 1 -- 172.21.15.26:0/1372402906 wait complete. 2024-08-19T12:38:37.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.581+0000 7f1cf2667640 1 librados: shutdown 2024-08-19T12:38:37.592 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-08-19T12:38:37.095372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":171600,"kb_used_data":1072,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566059440,"statfs":{"total":579820584960,"available":579644866560,"internally_reserved":0,"allocated":1097728,"data_stored":1368562,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9566,"internal_metadata":174447266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.148909"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.958765+0000","last_change":"2024-08-19T12:38:31.959032+0000","last_active":"2024-08-19T12:38:31.958765+0000","last_peered":"2024-08-19T12:38:31.958765+0000","last_clean":"2024-08-19T12:38:31.958765+0000","last_became_active":"2024-08-19T12:38:29.948183+0000","last_became_peered":"2024-08-19T12:38:29.948183+0000","last_unstale":"2024-08-19T12:38:31.958765+0000","last_undegraded":"2024-08-19T12:38:31.958765+0000","last_fullsized":"2024-08-19T12:38:31.958765+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:51:55.545957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052740100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.963149+0000","last_change":"2024-08-19T12:38:31.963379+0000","last_active":"2024-08-19T12:38:31.963149+0000","last_peered":"2024-08-19T12:38:31.963149+0000","last_clean":"2024-08-19T12:38:31.963149+0000","last_became_active":"2024-08-19T12:38:29.950467+0000","last_became_peered":"2024-08-19T12:38:29.950467+0000","last_unstale":"2024-08-19T12:38:31.963149+0000","last_undegraded":"2024-08-19T12:38:31.963149+0000","last_fullsized":"2024-08-19T12:38:31.963149+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:40:07.574639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049350399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.958711+0000","last_change":"2024-08-19T12:38:31.960297+0000","last_active":"2024-08-19T12:38:31.958711+0000","last_peered":"2024-08-19T12:38:31.958711+0000","last_clean":"2024-08-19T12:38:31.958711+0000","last_became_active":"2024-08-19T12:38:29.949846+0000","last_became_peered":"2024-08-19T12:38:29.949846+0000","last_unstale":"2024-08-19T12:38:31.958711+0000","last_undegraded":"2024-08-19T12:38:31.958711+0000","last_fullsized":"2024-08-19T12:38:31.958711+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:34:31.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001822192,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.963092+0000","last_change":"2024-08-19T12:38:31.963352+0000","last_active":"2024-08-19T12:38:31.963092+0000","last_peered":"2024-08-19T12:38:31.963092+0000","last_clean":"2024-08-19T12:38:31.963092+0000","last_became_active":"2024-08-19T12:38:29.951233+0000","last_became_peered":"2024-08-19T12:38:29.951233+0000","last_unstale":"2024-08-19T12:38:31.963092+0000","last_undegraded":"2024-08-19T12:38:31.963092+0000","last_fullsized":"2024-08-19T12:38:31.963092+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:45:51.806389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048735899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.962598+0000","last_change":"2024-08-19T12:38:31.959929+0000","last_active":"2024-08-19T12:38:31.962598+0000","last_peered":"2024-08-19T12:38:31.962598+0000","last_clean":"2024-08-19T12:38:31.962598+0000","last_became_active":"2024-08-19T12:38:29.951392+0000","last_became_peered":"2024-08-19T12:38:29.951392+0000","last_unstale":"2024-08-19T12:38:31.962598+0000","last_undegraded":"2024-08-19T12:38:31.962598+0000","last_fullsized":"2024-08-19T12:38:31.962598+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:47:44.756214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056602099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.957194+0000","last_change":"2024-08-19T12:38:31.957412+0000","last_active":"2024-08-19T12:38:31.957194+0000","last_peered":"2024-08-19T12:38:31.957194+0000","last_clean":"2024-08-19T12:38:31.957194+0000","last_became_active":"2024-08-19T12:38:29.952459+0000","last_became_peered":"2024-08-19T12:38:29.952459+0000","last_unstale":"2024-08-19T12:38:31.957194+0000","last_undegraded":"2024-08-19T12:38:31.957194+0000","last_fullsized":"2024-08-19T12:38:31.957194+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:47:27.910005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039863799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.959093+0000","last_change":"2024-08-19T12:38:31.960408+0000","last_active":"2024-08-19T12:38:31.959093+0000","last_peered":"2024-08-19T12:38:31.959093+0000","last_clean":"2024-08-19T12:38:31.959093+0000","last_became_active":"2024-08-19T12:38:29.951331+0000","last_became_peered":"2024-08-19T12:38:29.951331+0000","last_unstale":"2024-08-19T12:38:31.959093+0000","last_undegraded":"2024-08-19T12:38:31.959093+0000","last_fullsized":"2024-08-19T12:38:31.959093+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:55:28.246342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018789830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.959938+0000","last_change":"2024-08-19T12:38:31.959938+0000","last_active":"2024-08-19T12:38:31.959938+0000","last_peered":"2024-08-19T12:38:31.959938+0000","last_clean":"2024-08-19T12:38:31.959938+0000","last_became_active":"2024-08-19T12:38:29.952471+0000","last_became_peered":"2024-08-19T12:38:29.952471+0000","last_unstale":"2024-08-19T12:38:31.959938+0000","last_undegraded":"2024-08-19T12:38:31.959938+0000","last_fullsized":"2024-08-19T12:38:31.959938+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:48:18.927151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057881800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.958676+0000","last_change":"2024-08-19T12:38:22.728884+0000","last_active":"2024-08-19T12:38:31.958676+0000","last_peered":"2024-08-19T12:38:31.958676+0000","last_clean":"2024-08-19T12:38:31.958676+0000","last_became_active":"2024-08-19T12:38:22.728595+0000","last_became_peered":"2024-08-19T12:38:22.728595+0000","last_unstale":"2024-08-19T12:38:31.958676+0000","last_undegraded":"2024-08-19T12:38:31.958676+0000","last_fullsized":"2024-08-19T12:38:31.958676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_clean_scrub_stamp":"2024-08-19T12:38:21.711006+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:42:50.800603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":51797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":580706,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":167936,"data_stored":51797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":217088,"data_stored":580706,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T12:38:37.593 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T12:38:37.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.690+0000 7fa7c723f640 10 monclient: get_monmap_and_config 2024-08-19T12:38:37.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.690+0000 7fa7c723f640 10 monclient: build_initial_monmap 2024-08-19T12:38:37.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.690+0000 7fa7c723f640 10 monclient: monmap: 2024-08-19T12:38:37.691 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:37.691 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.691 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:37.691229+0000 2024-08-19T12:38:37.691 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:37.691229+0000 2024-08-19T12:38:37.691 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:37.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.692 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:37.692 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:37.692 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:37.692 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:37.692 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:37.692 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:37.692 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:37.692 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:37.692 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:37.692 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 1 Processor -- start 2024-08-19T12:38:37.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 1 -- start start 2024-08-19T12:38:37.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 10 monclient: init 2024-08-19T12:38:37.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:37.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 10 monclient: _add_conns ranks=[2,7,3,6,1,8,4,0,5] 2024-08-19T12:38:37.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 10 monclient(hunting): picked mon.noname-b con 0x7fa7c006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:37.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 10 monclient(hunting): picked mon.noname-i con 0x7fa7c00a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:37.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 10 monclient(hunting): picked mon.noname-g con 0x7fa7c00a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:37.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c0136790 con 0x7fa7c006e7b0 2024-08-19T12:38:37.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c009ddf0 con 0x7fa7c00a5d50 2024-08-19T12:38:37.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c009e0d0 con 0x7fa7c00a5920 2024-08-19T12:38:37.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:37.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c723f640 10 monclient(hunting): authenticate will time out at 711.778659s 2024-08-19T12:38:37.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c4fdb640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa7c006e7b0 0x7fa7c0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50400/0 (socket says 172.21.15.26:50400) 2024-08-19T12:38:37.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.691+0000 7fa7c4fdb640 1 -- 172.21.15.26:0/1678753799 learned_addr learned my addr 172.21.15.26:0/1678753799 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:37.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.693+0000 7fa7c4fdb640 1 -- 172.21.15.26:0/1678753799 >> v1:172.21.15.26:6790/0 conn(0x7fa7c006e7b0 legacy=0x7fa7c0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 1 -- 172.21.15.26:0/1678753799 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2237774038 0 0) 0x7fa7c0136790 con 0x7fa7c006e7b0 2024-08-19T12:38:37.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient(hunting): my global_id is 4385 2024-08-19T12:38:37.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 1 -- 172.21.15.26:0/1678753799 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7a8003520 con 0x7fa7c006e7b0 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7c57dc640 1 -- 172.21.15.26:0/1678753799 >> v1:172.21.15.104:6790/0 conn(0x7fa7c00a5d50 legacy=0x7fa7c0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b7fff640 1 -- 172.21.15.26:0/1678753799 >> v1:172.21.15.104:6792/0 conn(0x7fa7c00a5920 legacy=0x7fa7c013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 1 -- 172.21.15.26:0/1678753799 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1451806128 0 0) 0x7fa7a8003520 con 0x7fa7c006e7b0 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 1 -- 172.21.15.26:0/1678753799 >> v1:172.21.15.104:6792/0 conn(0x7fa7c00a5920 legacy=0x7fa7c013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 1 -- 172.21.15.26:0/1678753799 >> v1:172.21.15.104:6790/0 conn(0x7fa7c00a5d50 legacy=0x7fa7c0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 1 monclient: found mon.noname-b 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 1 -- 172.21.15.26:0/1678753799 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c009e3b0 con 0x7fa7c006e7b0 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:37.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 1 -- 172.21.15.26:0/1678753799 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7bc001ae0 con 0x7fa7c006e7b0 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7c723f640 5 monclient: authenticate success, global_id 4385 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7c723f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient: dump: 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.700 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7c723f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 1 -- 172.21.15.26:0/1678753799 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7bc0048a0 con 0x7fa7c006e7b0 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 1 -- 172.21.15.26:0/1678753799 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7bc004e10 con 0x7fa7c006e7b0 2024-08-19T12:38:37.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7c723f640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7c723f640 4 set_mon_vals no callback set 2024-08-19T12:38:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7b77fe640 10 monclient: dump: 2024-08-19T12:38:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.702 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7c723f640 10 monclient: shutdown 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7c723f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.694+0000 7fa7c723f640 1 -- 172.21.15.26:0/1678753799 >> v1:172.21.15.26:6790/0 conn(0x7fa7c006e7b0 legacy=0x7fa7c0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.695+0000 7fa7c723f640 1 -- 172.21.15.26:0/1678753799 shutdown_connections 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.695+0000 7fa7c723f640 1 -- 172.21.15.26:0/1678753799 >> 172.21.15.26:0/1678753799 conn(0x7fa7c00a8740 msgr2=0x7fa7c00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:37.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.695+0000 7fa7c723f640 1 -- 172.21.15.26:0/1678753799 shutdown_connections 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.695+0000 7fa7c723f640 1 -- 172.21.15.26:0/1678753799 wait complete. 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.695+0000 7fa7c723f640 10 monclient: build_initial_monmap 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.695+0000 7fa7c723f640 10 monclient: monmap: 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:37.704 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.695+0000 7fa7c723f640 1 librados: starting msgr at 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.695+0000 7fa7c723f640 1 librados: starting objecter 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.695+0000 7fa7c723f640 1 Processor -- start 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 1 -- start start 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 1 librados: setting wanted keys 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 1 librados: calling monclient init 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 10 monclient: init 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 10 monclient: _add_conns ranks=[7,8,4,6,0,5,3,2,1] 2024-08-19T12:38:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 10 monclient(hunting): picked mon.noname-i con 0x7fa7c006e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 10 monclient(hunting): picked mon.noname-e con 0x7fa7c00a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 10 monclient(hunting): picked mon.noname-c con 0x7fa7c00a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c013b3c0 con 0x7fa7c00a5d50 2024-08-19T12:38:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c0139060 con 0x7fa7c006e7b0 2024-08-19T12:38:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c013ab20 con 0x7fa7c00a5920 2024-08-19T12:38:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c723f640 10 monclient(hunting): authenticate will time out at 711.783616s 2024-08-19T12:38:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7b7fff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fa7c00a5920 0x7fa7c01d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34088/0 (socket says 172.21.15.26:34088) 2024-08-19T12:38:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7b7fff640 1 -- 172.21.15.26:0/814074869 learned_addr learned my addr 172.21.15.26:0/814074869 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:37.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.696+0000 7fa7c57dc640 1 --1- 172.21.15.26:0/814074869 >> v1:172.21.15.26:6791/0 conn(0x7fa7c00a5d50 0x7fa7c01d6bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57948/0 (socket says 172.21.15.26:57948) 2024-08-19T12:38:37.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.698+0000 7fa7b7fff640 1 -- 172.21.15.26:0/814074869 >> v1:172.21.15.26:6793/0 conn(0x7fa7c00a5920 legacy=0x7fa7c01d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.698+0000 7fa7b57fa640 10 client.?.objecter ms_handle_connect 0x7fa7c00a5920 2024-08-19T12:38:37.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.698+0000 7fa7b57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:37.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.698+0000 7fa7c57dc640 1 -- 172.21.15.26:0/814074869 >> v1:172.21.15.26:6791/0 conn(0x7fa7c00a5d50 legacy=0x7fa7c01d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.698+0000 7fa7b57fa640 10 client.?.objecter ms_handle_connect 0x7fa7c00a5d50 2024-08-19T12:38:37.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.698+0000 7fa7b57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:37.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.698+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3264921489 0 0) 0x7fa7c013ab20 con 0x7fa7c00a5920 2024-08-19T12:38:37.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): my global_id is 4481 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7ac004eb0 con 0x7fa7c00a5920 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4196735145 0 0) 0x7fa7c013b3c0 con 0x7fa7c00a5d50 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): my global_id is 4423 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7c013ab20 con 0x7fa7c00a5d50 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7c4fdb640 1 -- 172.21.15.26:0/814074869 >> v1:172.21.15.104:6792/0 conn(0x7fa7c006e7b0 legacy=0x7fa7c01ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 client.?.objecter ms_handle_connect 0x7fa7c006e7b0 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:37.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3259094741 0 0) 0x7fa7ac004eb0 con 0x7fa7c00a5920 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7c013b3c0 con 0x7fa7c00a5920 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7b0003610 con 0x7fa7c00a5920 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): dump: 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.712 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3906819926 0 0) 0x7fa7c013ab20 con 0x7fa7c00a5d50 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7ac004eb0 con 0x7fa7c00a5d50 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7b8002030 con 0x7fa7c00a5d50 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:37.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): dump: 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.714 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4136636972 0 0) 0x7fa7c0139060 con 0x7fa7c006e7b0 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient(hunting): my global_id is 4381 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7c013ab20 con 0x7fa7c006e7b0 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2739546514 0 0) 0x7fa7c013b3c0 con 0x7fa7c00a5920 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 >> v1:172.21.15.104:6792/0 conn(0x7fa7c006e7b0 legacy=0x7fa7c01ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 >> v1:172.21.15.26:6791/0 conn(0x7fa7c00a5d50 legacy=0x7fa7c01d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient: _finish_hunting 0 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 monclient: found mon.e 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c013ae00 con 0x7fa7c00a5920 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient: _finish_auth 0 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient: _check_auth_tickets 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7c723f640 5 monclient: authenticate success, global_id 4481 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7c723f640 10 monclient: _renew_subs 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7c723f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7c723f640 1 -- 172.21.15.26:0/814074869 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c013a7f0 con 0x7fa7c00a5920 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7c723f640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7c723f640 10 monclient: _renew_subs 2024-08-19T12:38:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7c723f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7c723f640 1 -- 172.21.15.26:0/814074869 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fa7c0138650 con 0x7fa7c00a5920 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7c723f640 1 librados: init done 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7b0003bc0 con 0x7fa7c00a5920 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7b00040b0 con 0x7fa7c00a5920 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b6ffd640 4 set_mon_vals no callback set 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.699+0000 7fa7b57fa640 10 monclient: dump: 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.717 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:37.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.700+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa7b0004250 con 0x7fa7c00a5920 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.700+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7fa7b004dac0 con 0x7fa7c00a5920 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.700+0000 7fa7b57fa640 10 client.4481.objecter ms_dispatch 0x7fa7c0000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.700+0000 7fa7b57fa640 3 client.4481.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.700+0000 7fa7b57fa640 3 client.4481.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.700+0000 7fa7c723f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.700+0000 7fa7c723f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.700+0000 7fa7c723f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.700+0000 7fa7c723f640 1 -- 172.21.15.26:0/814074869 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a0005080 con 0x7fa7c00a5920 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.700+0000 7fa7b57fa640 20 client.4481.objecter dump_active .. 0 homeless 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.700+0000 7fa7c4fdb640 10 monclient: get_auth_request con 0x7fa7ac0416b0 auth_method 0 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.703+0000 7fa7b57fa640 10 client.4481.objecter ms_handle_connect 0x7fa7ac0416b0 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.705+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fa7b00175e0 con 0x7fa7c00a5920 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.705+0000 7fa7b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.705+0000 7fa7b57fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:37.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.885+0000 7fa7c723f640 1 -- 172.21.15.26:0/814074869 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa7a0000b90 con 0x7fa7ac0416b0 2024-08-19T12:38:37.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.886+0000 7fa7b57fa640 1 -- 172.21.15.26:0/814074869 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34175 (unknown 2965378022 0 1573968203) 0x7fa7a0000b90 con 0x7fa7ac0416b0 2024-08-19T12:38:37.887 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:38:37.887 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-08-19T12:38:37.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 10 librados: watch_flush enter 2024-08-19T12:38:37.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 10 librados: watch_flush exit 2024-08-19T12:38:37.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 20 client.4481.objecter shutdown clearing up homeless session... 2024-08-19T12:38:37.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 10 client.4481.objecter successfully canceled tick 2024-08-19T12:38:37.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 1 -- 172.21.15.26:0/814074869 >> v1:172.21.15.104:6800/899887506 conn(0x7fa7ac0416b0 legacy=0x7fa7ac043b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 10 monclient: shutdown 2024-08-19T12:38:37.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:37.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 1 -- 172.21.15.26:0/814074869 >> v1:172.21.15.26:6793/0 conn(0x7fa7c00a5920 legacy=0x7fa7c01d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:37.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 1 -- 172.21.15.26:0/814074869 shutdown_connections 2024-08-19T12:38:37.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 1 -- 172.21.15.26:0/814074869 >> 172.21.15.26:0/814074869 conn(0x7fa7c00a8740 msgr2=0x7fa7c00ab380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:37.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 1 -- 172.21.15.26:0/814074869 shutdown_connections 2024-08-19T12:38:37.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 1 -- 172.21.15.26:0/814074869 wait complete. 2024-08-19T12:38:37.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:37.888+0000 7fa7c723f640 1 librados: shutdown 2024-08-19T12:38:37.899 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-08-19T12:38:37.095372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":171600,"kb_used_data":1072,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566059440,"statfs":{"total":579820584960,"available":579644866560,"internally_reserved":0,"allocated":1097728,"data_stored":1368562,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9566,"internal_metadata":174447266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.148909"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.958765+0000","last_change":"2024-08-19T12:38:31.959032+0000","last_active":"2024-08-19T12:38:31.958765+0000","last_peered":"2024-08-19T12:38:31.958765+0000","last_clean":"2024-08-19T12:38:31.958765+0000","last_became_active":"2024-08-19T12:38:29.948183+0000","last_became_peered":"2024-08-19T12:38:29.948183+0000","last_unstale":"2024-08-19T12:38:31.958765+0000","last_undegraded":"2024-08-19T12:38:31.958765+0000","last_fullsized":"2024-08-19T12:38:31.958765+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:51:55.545957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052740100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.963149+0000","last_change":"2024-08-19T12:38:31.963379+0000","last_active":"2024-08-19T12:38:31.963149+0000","last_peered":"2024-08-19T12:38:31.963149+0000","last_clean":"2024-08-19T12:38:31.963149+0000","last_became_active":"2024-08-19T12:38:29.950467+0000","last_became_peered":"2024-08-19T12:38:29.950467+0000","last_unstale":"2024-08-19T12:38:31.963149+0000","last_undegraded":"2024-08-19T12:38:31.963149+0000","last_fullsized":"2024-08-19T12:38:31.963149+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:40:07.574639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049350399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.958711+0000","last_change":"2024-08-19T12:38:31.960297+0000","last_active":"2024-08-19T12:38:31.958711+0000","last_peered":"2024-08-19T12:38:31.958711+0000","last_clean":"2024-08-19T12:38:31.958711+0000","last_became_active":"2024-08-19T12:38:29.949846+0000","last_became_peered":"2024-08-19T12:38:29.949846+0000","last_unstale":"2024-08-19T12:38:31.958711+0000","last_undegraded":"2024-08-19T12:38:31.958711+0000","last_fullsized":"2024-08-19T12:38:31.958711+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:34:31.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001822192,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.963092+0000","last_change":"2024-08-19T12:38:31.963352+0000","last_active":"2024-08-19T12:38:31.963092+0000","last_peered":"2024-08-19T12:38:31.963092+0000","last_clean":"2024-08-19T12:38:31.963092+0000","last_became_active":"2024-08-19T12:38:29.951233+0000","last_became_peered":"2024-08-19T12:38:29.951233+0000","last_unstale":"2024-08-19T12:38:31.963092+0000","last_undegraded":"2024-08-19T12:38:31.963092+0000","last_fullsized":"2024-08-19T12:38:31.963092+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:45:51.806389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048735899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.962598+0000","last_change":"2024-08-19T12:38:31.959929+0000","last_active":"2024-08-19T12:38:31.962598+0000","last_peered":"2024-08-19T12:38:31.962598+0000","last_clean":"2024-08-19T12:38:31.962598+0000","last_became_active":"2024-08-19T12:38:29.951392+0000","last_became_peered":"2024-08-19T12:38:29.951392+0000","last_unstale":"2024-08-19T12:38:31.962598+0000","last_undegraded":"2024-08-19T12:38:31.962598+0000","last_fullsized":"2024-08-19T12:38:31.962598+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:47:44.756214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056602099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.957194+0000","last_change":"2024-08-19T12:38:31.957412+0000","last_active":"2024-08-19T12:38:31.957194+0000","last_peered":"2024-08-19T12:38:31.957194+0000","last_clean":"2024-08-19T12:38:31.957194+0000","last_became_active":"2024-08-19T12:38:29.952459+0000","last_became_peered":"2024-08-19T12:38:29.952459+0000","last_unstale":"2024-08-19T12:38:31.957194+0000","last_undegraded":"2024-08-19T12:38:31.957194+0000","last_fullsized":"2024-08-19T12:38:31.957194+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:47:27.910005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039863799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.959093+0000","last_change":"2024-08-19T12:38:31.960408+0000","last_active":"2024-08-19T12:38:31.959093+0000","last_peered":"2024-08-19T12:38:31.959093+0000","last_clean":"2024-08-19T12:38:31.959093+0000","last_became_active":"2024-08-19T12:38:29.951331+0000","last_became_peered":"2024-08-19T12:38:29.951331+0000","last_unstale":"2024-08-19T12:38:31.959093+0000","last_undegraded":"2024-08-19T12:38:31.959093+0000","last_fullsized":"2024-08-19T12:38:31.959093+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:55:28.246342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018789830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.959938+0000","last_change":"2024-08-19T12:38:31.959938+0000","last_active":"2024-08-19T12:38:31.959938+0000","last_peered":"2024-08-19T12:38:31.959938+0000","last_clean":"2024-08-19T12:38:31.959938+0000","last_became_active":"2024-08-19T12:38:29.952471+0000","last_became_peered":"2024-08-19T12:38:29.952471+0000","last_unstale":"2024-08-19T12:38:31.959938+0000","last_undegraded":"2024-08-19T12:38:31.959938+0000","last_fullsized":"2024-08-19T12:38:31.959938+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:48:18.927151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057881800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.958676+0000","last_change":"2024-08-19T12:38:22.728884+0000","last_active":"2024-08-19T12:38:31.958676+0000","last_peered":"2024-08-19T12:38:31.958676+0000","last_clean":"2024-08-19T12:38:31.958676+0000","last_became_active":"2024-08-19T12:38:22.728595+0000","last_became_peered":"2024-08-19T12:38:22.728595+0000","last_unstale":"2024-08-19T12:38:31.958676+0000","last_undegraded":"2024-08-19T12:38:31.958676+0000","last_fullsized":"2024-08-19T12:38:31.958676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_clean_scrub_stamp":"2024-08-19T12:38:21.711006+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:42:50.800603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":51797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":580706,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":167936,"data_stored":51797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":217088,"data_stored":580706,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T12:38:37.900 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-19T12:38:37.900 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-19T12:38:37.901 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-08-19T12:38:37.901 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-08-19T12:38:38.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.000+0000 7fb233860640 10 monclient: get_monmap_and_config 2024-08-19T12:38:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.000+0000 7fb233860640 10 monclient: build_initial_monmap 2024-08-19T12:38:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.000+0000 7fb233860640 10 monclient: monmap: 2024-08-19T12:38:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:38.001462+0000 2024-08-19T12:38:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:38.001462+0000 2024-08-19T12:38:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:38.002 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:38.003 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 1 Processor -- start 2024-08-19T12:38:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 1 -- start start 2024-08-19T12:38:38.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 10 monclient: init 2024-08-19T12:38:38.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 10 monclient: _add_conns ranks=[2,8,3,5,0,1,7,6,4] 2024-08-19T12:38:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 10 monclient(hunting): picked mon.noname-b con 0x7fb22c073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 10 monclient(hunting): picked mon.noname-e con 0x7fb22c0a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 10 monclient(hunting): picked mon.noname-g con 0x7fb22c13f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22c09dae0 con 0x7fb22c073470 2024-08-19T12:38:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22c136910 con 0x7fb22c13f2e0 2024-08-19T12:38:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22c09dd50 con 0x7fb22c0a57a0 2024-08-19T12:38:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.001+0000 7fb233860640 10 monclient(hunting): authenticate will time out at 712.088965s 2024-08-19T12:38:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.002+0000 7fb230dfb640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fb22c0a57a0 0x7fb22c0a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34100/0 (socket says 172.21.15.26:34100) 2024-08-19T12:38:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.002+0000 7fb230dfb640 1 -- 172.21.15.26:0/2381603088 learned_addr learned my addr 172.21.15.26:0/2381603088 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:38.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.002+0000 7fb2315fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb22c073470 0x7fb22c136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50416/0 (socket says 172.21.15.26:50416) 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb230dfb640 1 -- 172.21.15.26:0/2381603088 >> v1:172.21.15.26:6793/0 conn(0x7fb22c0a57a0 legacy=0x7fb22c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb2315fc640 1 -- 172.21.15.26:0/2381603088 >> v1:172.21.15.26:6790/0 conn(0x7fb22c073470 legacy=0x7fb22c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb231dfd640 1 -- 172.21.15.26:0/2381603088 >> v1:172.21.15.104:6790/0 conn(0x7fb22c13f2e0 legacy=0x7fb22c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 1 -- 172.21.15.26:0/2381603088 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2596737785 0 0) 0x7fb22c09dd50 con 0x7fb22c0a57a0 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient(hunting): my global_id is 4490 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 1 -- 172.21.15.26:0/2381603088 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb218003520 con 0x7fb22c0a57a0 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 1 -- 172.21.15.26:0/2381603088 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3080990380 0 0) 0x7fb22c09dae0 con 0x7fb22c073470 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient(hunting): my global_id is 4394 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 1 -- 172.21.15.26:0/2381603088 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb22c09dd50 con 0x7fb22c073470 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 1 -- 172.21.15.26:0/2381603088 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1928750510 0 0) 0x7fb22c09dd50 con 0x7fb22c073470 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 1 -- 172.21.15.26:0/2381603088 >> v1:172.21.15.26:6793/0 conn(0x7fb22c0a57a0 legacy=0x7fb22c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 1 -- 172.21.15.26:0/2381603088 >> v1:172.21.15.104:6790/0 conn(0x7fb22c13f2e0 legacy=0x7fb22c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 1 monclient: found mon.noname-b 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 1 -- 172.21.15.26:0/2381603088 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22c09e030 con 0x7fb22c073470 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient: _finish_auth 0 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:38.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 1 -- 172.21.15.26:0/2381603088 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb2280019f0 con 0x7fb22c073470 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient: dump: 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.013 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb233860640 5 monclient: authenticate success, global_id 4394 2024-08-19T12:38:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb233860640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 1 -- 172.21.15.26:0/2381603088 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb228001cc0 con 0x7fb22c073470 2024-08-19T12:38:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 1 -- 172.21.15.26:0/2381603088 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb228004b20 con 0x7fb22c073470 2024-08-19T12:38:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:38.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb223fff640 10 monclient: dump: 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.014 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb233860640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.004+0000 7fb233860640 4 set_mon_vals no callback set 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.005+0000 7fb233860640 10 monclient: shutdown 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.005+0000 7fb233860640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.005+0000 7fb233860640 1 -- 172.21.15.26:0/2381603088 >> v1:172.21.15.26:6790/0 conn(0x7fb22c073470 legacy=0x7fb22c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.005+0000 7fb233860640 1 -- 172.21.15.26:0/2381603088 shutdown_connections 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.005+0000 7fb233860640 1 -- 172.21.15.26:0/2381603088 >> 172.21.15.26:0/2381603088 conn(0x7fb22c0abff0 msgr2=0x7fb22c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.005+0000 7fb233860640 1 -- 172.21.15.26:0/2381603088 shutdown_connections 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.005+0000 7fb233860640 1 -- 172.21.15.26:0/2381603088 wait complete. 2024-08-19T12:38:38.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.006+0000 7fb233860640 10 monclient: build_initial_monmap 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.006+0000 7fb233860640 10 monclient: monmap: 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.006+0000 7fb233860640 1 librados: starting msgr at 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.006+0000 7fb233860640 1 librados: starting objecter 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.006+0000 7fb233860640 1 Processor -- start 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.006+0000 7fb233860640 1 -- start start 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.006+0000 7fb233860640 1 librados: setting wanted keys 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.006+0000 7fb233860640 1 librados: calling monclient init 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.006+0000 7fb233860640 10 monclient: init 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.007+0000 7fb233860640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.007+0000 7fb233860640 10 monclient: _add_conns ranks=[7,5,0,1,3,2,8,4,6] 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.007+0000 7fb233860640 10 monclient(hunting): picked mon.noname-i con 0x7fb22c073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.007+0000 7fb233860640 10 monclient(hunting): picked mon.noname-h con 0x7fb22c0a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.007+0000 7fb233860640 10 monclient(hunting): picked mon.noname-a con 0x7fb22c13f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.007+0000 7fb233860640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22c1375f0 con 0x7fb22c13f2e0 2024-08-19T12:38:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.007+0000 7fb233860640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22c142b60 con 0x7fb22c0a57a0 2024-08-19T12:38:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.007+0000 7fb233860640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22c136c50 con 0x7fb22c073470 2024-08-19T12:38:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.007+0000 7fb233860640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.007+0000 7fb233860640 10 monclient(hunting): authenticate will time out at 712.094237s 2024-08-19T12:38:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.007+0000 7fb231dfd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb22c13f2e0 0x7fb22c1d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58502/0 (socket says 172.21.15.26:58502) 2024-08-19T12:38:38.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.007+0000 7fb231dfd640 1 -- 172.21.15.26:0/2871760150 learned_addr learned my addr 172.21.15.26:0/2871760150 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:38.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb231dfd640 1 -- 172.21.15.26:0/2871760150 >> v1:172.21.15.26:6789/0 conn(0x7fb22c13f2e0 legacy=0x7fb22c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 client.?.objecter ms_handle_connect 0x7fb22c13f2e0 2024-08-19T12:38:38.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:38.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 client.?.objecter ms_handle_connect 0x7fb22c073470 2024-08-19T12:38:38.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 client.?.objecter ms_handle_connect 0x7fb22c0a57a0 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb2315fc640 1 -- 172.21.15.26:0/2871760150 >> v1:172.21.15.104:6792/0 conn(0x7fb22c073470 legacy=0x7fb22c0a1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb230dfb640 1 -- 172.21.15.26:0/2871760150 >> v1:172.21.15.104:6791/0 conn(0x7fb22c0a57a0 legacy=0x7fb22c0a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3267311123 0 0) 0x7fb22c136910 con 0x7fb22c13f2e0 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): my global_id is 4374 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb210004eb0 con 0x7fb22c13f2e0 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2155877871 0 0) 0x7fb22c136c50 con 0x7fb22c073470 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): my global_id is 4390 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb22c136910 con 0x7fb22c073470 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3744964002 0 0) 0x7fb21c0015e0 con 0x7fb22c0a57a0 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): my global_id is 4352 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb22c136c50 con 0x7fb22c0a57a0 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1691891515 0 0) 0x7fb210004eb0 con 0x7fb22c13f2e0 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb21c0015e0 con 0x7fb22c13f2e0 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb224003990 con 0x7fb22c13f2e0 2024-08-19T12:38:38.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.025 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.009+0000 7fb221ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 522694830 0 0) 0x7fb22c136c50 con 0x7fb22c0a57a0 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb210004eb0 con 0x7fb22c0a57a0 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2618584530 0 0) 0x7fb22c136910 con 0x7fb22c073470 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb22c136c50 con 0x7fb22c073470 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb21c001320 con 0x7fb22c0a57a0 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:38.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.027 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.028 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb228004b20 con 0x7fb22c073470 2024-08-19T12:38:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.028 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2209319015 0 0) 0x7fb21c0015e0 con 0x7fb22c13f2e0 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 >> v1:172.21.15.104:6792/0 conn(0x7fb22c073470 legacy=0x7fb22c0a1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 >> v1:172.21.15.104:6791/0 conn(0x7fb22c0a57a0 legacy=0x7fb22c0a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 monclient: found mon.a 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22c135270 con 0x7fb22c13f2e0 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb233860640 5 monclient: authenticate success, global_id 4374 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb233860640 10 monclient: _renew_subs 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb233860640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb233860640 1 -- 172.21.15.26:0/2871760150 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb22c0a2df0 con 0x7fb22c13f2e0 2024-08-19T12:38:38.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb224003e90 con 0x7fb22c13f2e0 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb224004400 con 0x7fb22c13f2e0 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb221ffb640 10 monclient: dump: 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.031 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb2237fe640 4 set_mon_vals no callback set 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb233860640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb233860640 10 monclient: _renew_subs 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb233860640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb233860640 1 -- 172.21.15.26:0/2871760150 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb22c13e8b0 con 0x7fb22c13f2e0 2024-08-19T12:38:38.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.010+0000 7fb233860640 1 librados: init done 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.011+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fb2240046a0 con 0x7fb22c13f2e0 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.011+0000 7fb233860640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.011+0000 7fb233860640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.011+0000 7fb233860640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.011+0000 7fb233860640 1 -- 172.21.15.26:0/2871760150 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb22c13eb90 con 0x7fb22c13f2e0 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.011+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7fb22404dcf0 con 0x7fb22c13f2e0 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.011+0000 7fb221ffb640 10 client.4374.objecter ms_dispatch 0x7fb22c000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.011+0000 7fb221ffb640 3 client.4374.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.011+0000 7fb221ffb640 3 client.4374.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.011+0000 7fb221ffb640 20 client.4374.objecter dump_active .. 0 homeless 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.011+0000 7fb2315fc640 10 monclient: get_auth_request con 0x7fb210041b80 auth_method 0 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.013+0000 7fb221ffb640 10 client.4374.objecter ms_handle_connect 0x7fb210041b80 2024-08-19T12:38:38.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.015+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fb22c1375f0 con 0x7fb22c13f2e0 2024-08-19T12:38:38.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.015+0000 7fb221ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:38.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.015+0000 7fb221ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:38.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.237+0000 7fb233860640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-08-19T12:38:38.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.237+0000 7fb233860640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-08-19T12:38:38.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.237+0000 7fb233860640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:38.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.237+0000 7fb233860640 1 -- 172.21.15.26:0/2871760150 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb22c06c840 con 0x7fb22c13f2e0 2024-08-19T12:38:38.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.238+0000 7fb221ffb640 1 -- 172.21.15.26:0/2871760150 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7fb2240171c0 con 0x7fb22c13f2e0 2024-08-19T12:38:38.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.238+0000 7fb221ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-08-19T12:38:38.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.238+0000 7fb221ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:38.239 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:38:38.239 INFO:teuthology.orchestra.run.smithi026.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-19T12:38:38.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.239+0000 7fb233860640 10 librados: watch_flush enter 2024-08-19T12:38:38.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.239+0000 7fb233860640 10 librados: watch_flush exit 2024-08-19T12:38:38.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.239+0000 7fb233860640 20 client.4374.objecter shutdown clearing up homeless session... 2024-08-19T12:38:38.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.239+0000 7fb233860640 10 client.4374.objecter successfully canceled tick 2024-08-19T12:38:38.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.240+0000 7fb233860640 1 -- 172.21.15.26:0/2871760150 >> v1:172.21.15.104:6800/899887506 conn(0x7fb210041b80 legacy=0x7fb210044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.240+0000 7fb233860640 10 monclient: shutdown 2024-08-19T12:38:38.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.240+0000 7fb233860640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:38.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.240+0000 7fb233860640 1 -- 172.21.15.26:0/2871760150 >> v1:172.21.15.26:6789/0 conn(0x7fb22c13f2e0 legacy=0x7fb22c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.240+0000 7fb233860640 1 -- 172.21.15.26:0/2871760150 shutdown_connections 2024-08-19T12:38:38.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.240+0000 7fb233860640 1 -- 172.21.15.26:0/2871760150 >> 172.21.15.26:0/2871760150 conn(0x7fb22c0abff0 msgr2=0x7fb22c1338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:38.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.240+0000 7fb233860640 1 -- 172.21.15.26:0/2871760150 shutdown_connections 2024-08-19T12:38:38.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.240+0000 7fb233860640 1 -- 172.21.15.26:0/2871760150 wait complete. 2024-08-19T12:38:38.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.240+0000 7fb233860640 1 librados: shutdown 2024-08-19T12:38:38.251 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-08-19T12:38:38.251 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-08-19T12:38:38.267 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-08-19T12:38:38.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-19T12:38:38.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.365+0000 7f7c2ec15640 10 monclient: get_monmap_and_config 2024-08-19T12:38:38.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.365+0000 7f7c2ec15640 10 monclient: build_initial_monmap 2024-08-19T12:38:38.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.365+0000 7f7c2ec15640 10 monclient: monmap: 2024-08-19T12:38:38.366 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:38.366 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.366 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:38.366066+0000 2024-08-19T12:38:38.366 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:38.366066+0000 2024-08-19T12:38:38.366 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:38.367 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.367 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:38.367 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:38.367 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:38.367 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:38.367 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:38.367 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:38.367 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:38.367 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:38.367 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:38.367 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.365+0000 7f7c2ec15640 1 Processor -- start 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2ec15640 1 -- start start 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2ec15640 10 monclient: init 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2ec15640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2ec15640 10 monclient: _add_conns ranks=[2,0,6,7,3,1,8,5,4] 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2ec15640 10 monclient(hunting): picked mon.noname-b con 0x7f7c28073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2ec15640 10 monclient(hunting): picked mon.noname-a con 0x7f7c280a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2ec15640 10 monclient(hunting): picked mon.noname-d con 0x7f7c2813f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2ec15640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c2809dae0 con 0x7f7c280a57a0 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2ec15640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c28136910 con 0x7f7c28073470 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2ec15640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c2809dd50 con 0x7f7c2813f2e0 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2ec15640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2ec15640 10 monclient(hunting): authenticate will time out at 712.453631s 2024-08-19T12:38:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c27fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7c280a57a0 0x7f7c280a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58514/0 (socket says 172.21.15.26:58514) 2024-08-19T12:38:38.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c2c9b1640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7c28073470 0x7f7c28136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50430/0 (socket says 172.21.15.26:50430) 2024-08-19T12:38:38.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.366+0000 7f7c27fff640 1 -- 172.21.15.26:0/3661149388 learned_addr learned my addr 172.21.15.26:0/3661149388 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:38.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.368+0000 7f7c27fff640 1 -- 172.21.15.26:0/3661149388 >> v1:172.21.15.26:6789/0 conn(0x7f7c280a57a0 legacy=0x7f7c280a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.368+0000 7f7c2c9b1640 1 -- 172.21.15.26:0/3661149388 >> v1:172.21.15.26:6790/0 conn(0x7f7c28073470 legacy=0x7f7c28136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.368+0000 7f7c2d1b2640 1 -- 172.21.15.26:0/3661149388 >> v1:172.21.15.26:6792/0 conn(0x7f7c2813f2e0 legacy=0x7f7c281416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2174472206 0 0) 0x7f7c28136910 con 0x7f7c28073470 2024-08-19T12:38:38.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient(hunting): my global_id is 4403 2024-08-19T12:38:38.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c18003520 con 0x7f7c28073470 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2504371471 0 0) 0x7f7c2809dae0 con 0x7f7c280a57a0 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient(hunting): my global_id is 4383 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c28136910 con 0x7f7c280a57a0 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1583200997 0 0) 0x7f7c2809dd50 con 0x7f7c2813f2e0 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient(hunting): my global_id is 4488 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c2809dae0 con 0x7f7c2813f2e0 2024-08-19T12:38:38.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1749640295 0 0) 0x7f7c28136910 con 0x7f7c280a57a0 2024-08-19T12:38:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 >> v1:172.21.15.26:6792/0 conn(0x7f7c2813f2e0 legacy=0x7f7c281416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 >> v1:172.21.15.26:6790/0 conn(0x7f7c28073470 legacy=0x7f7c28136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 monclient: found mon.noname-a 2024-08-19T12:38:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:38:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2809e030 con 0x7f7c280a57a0 2024-08-19T12:38:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7c14001a40 con 0x7f7c280a57a0 2024-08-19T12:38:38.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c2ec15640 5 monclient: authenticate success, global_id 4383 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c2ec15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient: dump: 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.380 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c2ec15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c14001d10 con 0x7f7c280a57a0 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c2ec15640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 1 -- 172.21.15.26:0/3661149388 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7c14004be0 con 0x7f7c280a57a0 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c2ec15640 4 set_mon_vals no callback set 2024-08-19T12:38:38.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.369+0000 7f7c277fe640 10 monclient: dump: 2024-08-19T12:38:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.382 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.370+0000 7f7c2ec15640 10 monclient: shutdown 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.370+0000 7f7c2ec15640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.370+0000 7f7c2ec15640 1 -- 172.21.15.26:0/3661149388 >> v1:172.21.15.26:6789/0 conn(0x7f7c280a57a0 legacy=0x7f7c280a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.370+0000 7f7c2ec15640 1 -- 172.21.15.26:0/3661149388 shutdown_connections 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.370+0000 7f7c2ec15640 1 -- 172.21.15.26:0/3661149388 >> 172.21.15.26:0/3661149388 conn(0x7f7c280abff0 msgr2=0x7f7c280ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.370+0000 7f7c2ec15640 1 -- 172.21.15.26:0/3661149388 shutdown_connections 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.370+0000 7f7c2ec15640 1 -- 172.21.15.26:0/3661149388 wait complete. 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.370+0000 7f7c2ec15640 10 monclient: build_initial_monmap 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.370+0000 7f7c2ec15640 10 monclient: monmap: 2024-08-19T12:38:38.383 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:38.384 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.370+0000 7f7c2ec15640 1 librados: starting msgr at 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.370+0000 7f7c2ec15640 1 librados: starting objecter 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 1 Processor -- start 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 1 -- start start 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 1 librados: setting wanted keys 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 1 librados: calling monclient init 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 10 monclient: init 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 10 monclient: _add_conns ranks=[3,1,2,4,8,7,0,6,5] 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 10 monclient(hunting): picked mon.noname-g con 0x7f7c28073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 10 monclient(hunting): picked mon.noname-f con 0x7f7c280a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:38.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 10 monclient(hunting): picked mon.noname-b con 0x7f7c2813f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:38.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c28130fe0 con 0x7f7c280a57a0 2024-08-19T12:38:38.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c28142ba0 con 0x7f7c2813f2e0 2024-08-19T12:38:38.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c2812f520 con 0x7f7c28073470 2024-08-19T12:38:38.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:38.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2ec15640 10 monclient(hunting): authenticate will time out at 712.458701s 2024-08-19T12:38:38.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2d1b2640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7c2813f2e0 0x7f7c2812ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:50438/0 (socket says 172.21.15.26:50438) 2024-08-19T12:38:38.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.371+0000 7f7c2d1b2640 1 -- 172.21.15.26:0/1287872444 learned_addr learned my addr 172.21.15.26:0/1287872444 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:38.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.373+0000 7f7c2d1b2640 1 -- 172.21.15.26:0/1287872444 >> v1:172.21.15.26:6790/0 conn(0x7f7c2813f2e0 legacy=0x7f7c2812ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.373+0000 7f7c257fa640 10 client.?.objecter ms_handle_connect 0x7f7c2813f2e0 2024-08-19T12:38:38.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.373+0000 7f7c257fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:38.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2327702816 0 0) 0x7f7c28142ba0 con 0x7f7c2813f2e0 2024-08-19T12:38:38.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient(hunting): my global_id is 4412 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c0c004eb0 con 0x7f7c2813f2e0 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c2c9b1640 1 -- 172.21.15.26:0/1287872444 >> v1:172.21.15.104:6790/0 conn(0x7f7c28073470 legacy=0x7f7c2812dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 client.?.objecter ms_handle_connect 0x7f7c28073470 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 client.?.objecter ms_handle_connect 0x7f7c280a57a0 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c27fff640 1 -- 172.21.15.26:0/1287872444 >> v1:172.21.15.104:6789/0 conn(0x7f7c280a57a0 legacy=0x7f7c2812e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3152973961 0 0) 0x7f7c0c004eb0 con 0x7f7c2813f2e0 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c28142ba0 con 0x7f7c2813f2e0 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7c20002080 con 0x7f7c2813f2e0 2024-08-19T12:38:38.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient(hunting): dump: 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.388 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3623283871 0 0) 0x7f7c28142ba0 con 0x7f7c2813f2e0 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 >> v1:172.21.15.104:6790/0 conn(0x7f7c28073470 legacy=0x7f7c2812dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 >> v1:172.21.15.104:6789/0 conn(0x7f7c280a57a0 legacy=0x7f7c2812e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient: _finish_hunting 0 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 1 monclient: found mon.b 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:38.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c28131880 con 0x7f7c2813f2e0 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient: _finish_auth 0 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient: _check_auth_tickets 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c2ec15640 5 monclient: authenticate success, global_id 4412 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c2ec15640 10 monclient: _renew_subs 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c2ec15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c2ec15640 1 -- 172.21.15.26:0/1287872444 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7c28132f90 con 0x7f7c2813f2e0 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c2ec15640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c2ec15640 10 monclient: _renew_subs 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c2ec15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c2ec15640 1 -- 172.21.15.26:0/1287872444 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7c281d7160 con 0x7f7c2813f2e0 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c2ec15640 1 librados: init done 2024-08-19T12:38:38.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c20003db0 con 0x7f7c2813f2e0 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7c200042a0 con 0x7f7c2813f2e0 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c26ffd640 4 set_mon_vals no callback set 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.374+0000 7f7c257fa640 10 monclient: dump: 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.391 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.375+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f7c20004440 con 0x7f7c2813f2e0 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.375+0000 7f7c2ec15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.375+0000 7f7c2ec15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.375+0000 7f7c2ec15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.375+0000 7f7c2ec15640 1 -- 172.21.15.26:0/1287872444 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c08005080 con 0x7f7c2813f2e0 2024-08-19T12:38:38.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.375+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f7c2004dd20 con 0x7f7c2813f2e0 2024-08-19T12:38:38.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.375+0000 7f7c257fa640 10 client.4412.objecter ms_dispatch 0x7f7c28000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:38.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.375+0000 7f7c257fa640 3 client.4412.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:38.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.375+0000 7f7c257fa640 3 client.4412.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:38.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.375+0000 7f7c257fa640 20 client.4412.objecter dump_active .. 0 homeless 2024-08-19T12:38:38.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.375+0000 7f7c2c9b1640 10 monclient: get_auth_request con 0x7f7c0c040fd0 auth_method 0 2024-08-19T12:38:38.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.378+0000 7f7c257fa640 10 client.4412.objecter ms_handle_connect 0x7f7c0c040fd0 2024-08-19T12:38:38.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.380+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f7c20017190 con 0x7f7c2813f2e0 2024-08-19T12:38:38.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.380+0000 7f7c257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:38.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.380+0000 7f7c257fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:38.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.564+0000 7f7c2ec15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:38.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.564+0000 7f7c2ec15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:38.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.564+0000 7f7c2ec15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:38.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.564+0000 7f7c2ec15640 1 -- 172.21.15.26:0/1287872444 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c08005370 con 0x7f7c2813f2e0 2024-08-19T12:38:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.565+0000 7f7c257fa640 1 -- 172.21.15.26:0/1287872444 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+10910 (unknown 3413328019 0 1739965073) 0x7f7c2001b680 con 0x7f7c2813f2e0 2024-08-19T12:38:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.565+0000 7f7c257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:38:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.565+0000 7f7c257fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:38:38.568 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":20,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","created":"2024-08-19T12:38:08.215243+0000","modified":"2024-08-19T12:38:36.968971+0000","last_up_change":"2024-08-19T12:38:20.704140+0000","last_in_change":"2024-08-19T12:38:12.064517+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T12:38:21.116424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-19T12:38:27.276571+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae69b71a-73f9-47e2-b81e-92e9cdd7df0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6800","nonce":3531533167}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6801","nonce":3531533167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6803","nonce":3531533167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6802","nonce":3531533167}]},"public_addr":"172.21.15.26:6800/3531533167","cluster_addr":"172.21.15.26:6801/3531533167","heartbeat_back_addr":"172.21.15.26:6803/3531533167","heartbeat_front_addr":"172.21.15.26:6802/3531533167","state":["exists","up"]},{"osd":1,"uuid":"fc6dcabb-211c-484a-b215-2ef5fe1cbd21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6804","nonce":2594673920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6805","nonce":2594673920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6807","nonce":2594673920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6806","nonce":2594673920}]},"public_addr":"172.21.15.26:6804/2594673920","cluster_addr":"172.21.15.26:6805/2594673920","heartbeat_back_addr":"172.21.15.26:6807/2594673920","heartbeat_front_addr":"172.21.15.26:6806/2594673920","state":["exists","up"]},{"osd":2,"uuid":"88a4da36-5efe-4f64-84c3-28413eba4011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6808","nonce":1443966639}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6809","nonce":1443966639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6811","nonce":1443966639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6810","nonce":1443966639}]},"public_addr":"172.21.15.26:6808/1443966639","cluster_addr":"172.21.15.26:6809/1443966639","heartbeat_back_addr":"172.21.15.26:6811/1443966639","heartbeat_front_addr":"172.21.15.26:6810/1443966639","state":["exists","up"]},{"osd":3,"uuid":"98c08a65-4fb2-4e5f-a9bc-e8899aed1103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6809","nonce":847838248}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6810","nonce":847838248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6812","nonce":847838248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6811","nonce":847838248}]},"public_addr":"172.21.15.104:6809/847838248","cluster_addr":"172.21.15.104:6810/847838248","heartbeat_back_addr":"172.21.15.104:6812/847838248","heartbeat_front_addr":"172.21.15.104:6811/847838248","state":["exists","up"]},{"osd":4,"uuid":"a4c308b6-5e55-4328-b7b0-1db11fd51d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6805","nonce":1352976736}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6806","nonce":1352976736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6808","nonce":1352976736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6807","nonce":1352976736}]},"public_addr":"172.21.15.104:6805/1352976736","cluster_addr":"172.21.15.104:6806/1352976736","heartbeat_back_addr":"172.21.15.104:6808/1352976736","heartbeat_front_addr":"172.21.15.104:6807/1352976736","state":["exists","up"]},{"osd":5,"uuid":"99117114-07ce-4b9b-8df6-39f5fb1b610b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6801","nonce":3713183117}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6802","nonce":3713183117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6804","nonce":3713183117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6803","nonce":3713183117}]},"public_addr":"172.21.15.104:6801/3713183117","cluster_addr":"172.21.15.104:6802/3713183117","heartbeat_back_addr":"172.21.15.104:6804/3713183117","heartbeat_front_addr":"172.21.15.104:6803/3713183117","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.063585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.566627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.758447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.327856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.657329+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.299879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T12:38:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 10 librados: watch_flush enter 2024-08-19T12:38:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 10 librados: watch_flush exit 2024-08-19T12:38:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 20 client.4412.objecter shutdown clearing up homeless session... 2024-08-19T12:38:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 10 client.4412.objecter successfully canceled tick 2024-08-19T12:38:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 1 -- 172.21.15.26:0/1287872444 >> v1:172.21.15.104:6800/899887506 conn(0x7f7c0c040fd0 legacy=0x7f7c0c043490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 10 monclient: shutdown 2024-08-19T12:38:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 1 -- 172.21.15.26:0/1287872444 >> v1:172.21.15.26:6790/0 conn(0x7f7c2813f2e0 legacy=0x7f7c2812ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 1 -- 172.21.15.26:0/1287872444 shutdown_connections 2024-08-19T12:38:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 1 -- 172.21.15.26:0/1287872444 >> 172.21.15.26:0/1287872444 conn(0x7f7c280abff0 msgr2=0x7f7c280a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 1 -- 172.21.15.26:0/1287872444 shutdown_connections 2024-08-19T12:38:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 1 -- 172.21.15.26:0/1287872444 wait complete. 2024-08-19T12:38:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.567+0000 7f7c2ec15640 1 librados: shutdown 2024-08-19T12:38:38.579 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-19T12:38:21.116424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-08-19T12:38:27.276571+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-19T12:38:38.579 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-08-19T12:38:38.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.681+0000 7f20e09c9640 10 monclient: get_monmap_and_config 2024-08-19T12:38:38.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.681+0000 7f20e09c9640 10 monclient: build_initial_monmap 2024-08-19T12:38:38.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.681+0000 7f20e09c9640 10 monclient: monmap: 2024-08-19T12:38:38.682 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:38.682 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.682 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:38.681970+0000 2024-08-19T12:38:38.682 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:38.681970+0000 2024-08-19T12:38:38.682 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:38.682 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.682 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:38.682 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:38.682 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:38.683 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:38.683 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:38.683 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:38.683 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:38.683 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:38.683 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:38.683 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.681+0000 7f20e09c9640 1 Processor -- start 2024-08-19T12:38:38.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.681+0000 7f20e09c9640 1 -- start start 2024-08-19T12:38:38.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.681+0000 7f20e09c9640 10 monclient: init 2024-08-19T12:38:38.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.682+0000 7f20e09c9640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:38.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.682+0000 7f20e09c9640 10 monclient: _add_conns ranks=[3,5,6,2,8,1,7,0,4] 2024-08-19T12:38:38.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.682+0000 7f20e09c9640 10 monclient(hunting): picked mon.noname-g con 0x7f20dc073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:38.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.682+0000 7f20e09c9640 10 monclient(hunting): picked mon.noname-h con 0x7f20dc0a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:38.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.682+0000 7f20e09c9640 10 monclient(hunting): picked mon.noname-d con 0x7f20dc13f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:38.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.682+0000 7f20e09c9640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20dc09dae0 con 0x7f20dc073470 2024-08-19T12:38:38.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.682+0000 7f20e09c9640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20dc136910 con 0x7f20dc0a57a0 2024-08-19T12:38:38.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.682+0000 7f20e09c9640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20dc09dd50 con 0x7f20dc13f2e0 2024-08-19T12:38:38.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.682+0000 7f20e09c9640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:38.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.682+0000 7f20e09c9640 10 monclient(hunting): authenticate will time out at 712.769427s 2024-08-19T12:38:38.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.682+0000 7f20dad9d640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f20dc13f2e0 0x7f20dc1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:40884/0 (socket says 172.21.15.26:40884) 2024-08-19T12:38:38.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.682+0000 7f20dad9d640 1 -- 172.21.15.26:0/3097247854 learned_addr learned my addr 172.21.15.26:0/3097247854 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:38.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.684+0000 7f20da59c640 1 -- 172.21.15.26:0/3097247854 >> v1:172.21.15.104:6790/0 conn(0x7f20dc073470 legacy=0x7f20dc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.684+0000 7f20dad9d640 1 -- 172.21.15.26:0/3097247854 >> v1:172.21.15.26:6792/0 conn(0x7f20dc13f2e0 legacy=0x7f20dc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.684+0000 7f20d9d9b640 1 -- 172.21.15.26:0/3097247854 >> v1:172.21.15.104:6791/0 conn(0x7f20dc0a57a0 legacy=0x7f20dc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 855888947 0 0) 0x7f20dc09dae0 con 0x7f20dc073470 2024-08-19T12:38:38.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient(hunting): my global_id is 4467 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20c8003520 con 0x7f20dc073470 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2662741706 0 0) 0x7f20dc09dd50 con 0x7f20dc13f2e0 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient(hunting): my global_id is 4497 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20dc09dae0 con 0x7f20dc13f2e0 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700884893 0 0) 0x7f20dc136910 con 0x7f20dc0a57a0 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient(hunting): my global_id is 4361 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20dc09dd50 con 0x7f20dc0a57a0 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3978107365 0 0) 0x7f20dc09dae0 con 0x7f20dc13f2e0 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 >> v1:172.21.15.104:6791/0 conn(0x7f20dc0a57a0 legacy=0x7f20dc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 >> v1:172.21.15.104:6790/0 conn(0x7f20dc073470 legacy=0x7f20dc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient: _finish_hunting 0 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 monclient: found mon.noname-d 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20dc09e030 con 0x7f20dc13f2e0 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient: _finish_auth 0 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient: _check_auth_tickets 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:38.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20e09c9640 5 monclient: authenticate success, global_id 4497 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20e09c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20cc0019f0 con 0x7f20dc13f2e0 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient: dump: 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.693 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.694 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.694 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.694 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.694 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.694 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.694 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.694 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.694 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20e09c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20cc0048a0 con 0x7f20dc13f2e0 2024-08-19T12:38:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 1 -- 172.21.15.26:0/3097247854 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20cc001d40 con 0x7f20dc13f2e0 2024-08-19T12:38:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20d959a640 10 monclient: dump: 2024-08-19T12:38:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.696 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20e09c9640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:38.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.685+0000 7f20e09c9640 4 set_mon_vals no callback set 2024-08-19T12:38:38.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.686+0000 7f20e09c9640 10 monclient: shutdown 2024-08-19T12:38:38.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.686+0000 7f20e09c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:38.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.686+0000 7f20e09c9640 1 -- 172.21.15.26:0/3097247854 >> v1:172.21.15.26:6792/0 conn(0x7f20dc13f2e0 legacy=0x7f20dc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.686+0000 7f20e09c9640 1 -- 172.21.15.26:0/3097247854 shutdown_connections 2024-08-19T12:38:38.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.686+0000 7f20e09c9640 1 -- 172.21.15.26:0/3097247854 >> 172.21.15.26:0/3097247854 conn(0x7f20dc0abff0 msgr2=0x7f20dc0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:38.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.686+0000 7f20e09c9640 1 -- 172.21.15.26:0/3097247854 shutdown_connections 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.686+0000 7f20e09c9640 1 -- 172.21.15.26:0/3097247854 wait complete. 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.686+0000 7f20e09c9640 10 monclient: build_initial_monmap 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.686+0000 7f20e09c9640 10 monclient: monmap: 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:38.698 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:38.699 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:38.699 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:38.699 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:38.699 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:38.699 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:38.699 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 1 librados: starting msgr at 2024-08-19T12:38:38.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 1 librados: starting objecter 2024-08-19T12:38:38.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 1 Processor -- start 2024-08-19T12:38:38.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 1 -- start start 2024-08-19T12:38:38.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 1 librados: setting wanted keys 2024-08-19T12:38:38.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 1 librados: calling monclient init 2024-08-19T12:38:38.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 10 monclient: init 2024-08-19T12:38:38.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:38.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 10 monclient: _add_conns ranks=[8,1,7,0,3,6,2,4,5] 2024-08-19T12:38:38.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 10 monclient(hunting): picked mon.noname-e con 0x7f20dc073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:38.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 10 monclient(hunting): picked mon.noname-f con 0x7f20dc0a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:38.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 10 monclient(hunting): picked mon.noname-i con 0x7f20dc13f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:38.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20dc136c50 con 0x7f20dc0a57a0 2024-08-19T12:38:38.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20dc136f30 con 0x7f20dc13f2e0 2024-08-19T12:38:38.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20dc0a4630 con 0x7f20dc073470 2024-08-19T12:38:38.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:38.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20e09c9640 10 monclient(hunting): authenticate will time out at 712.774800s 2024-08-19T12:38:38.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20da59c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f20dc073470 0x7f20dc0a2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40070/0 (socket says 172.21.15.26:40070) 2024-08-19T12:38:38.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.687+0000 7f20da59c640 1 -- 172.21.15.26:0/2374930735 learned_addr learned my addr 172.21.15.26:0/2374930735 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:38.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.689+0000 7f20da59c640 1 -- 172.21.15.26:0/2374930735 >> v1:172.21.15.26:6793/0 conn(0x7f20dc073470 legacy=0x7f20dc0a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.689+0000 7f20d2ffd640 10 client.?.objecter ms_handle_connect 0x7f20dc073470 2024-08-19T12:38:38.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.689+0000 7f20d2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:38.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1843091104 0 0) 0x7f20dc0a4630 con 0x7f20dc073470 2024-08-19T12:38:38.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): my global_id is 4499 2024-08-19T12:38:38.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20dad9d640 1 -- 172.21.15.26:0/2374930735 >> v1:172.21.15.104:6792/0 conn(0x7f20dc13f2e0 legacy=0x7f20dc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20bc004eb0 con 0x7f20dc073470 2024-08-19T12:38:38.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d9d9b640 1 -- 172.21.15.26:0/2374930735 >> v1:172.21.15.104:6789/0 conn(0x7f20dc0a57a0 legacy=0x7f20dc0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:38.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 client.?.objecter ms_handle_connect 0x7f20dc13f2e0 2024-08-19T12:38:38.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:38.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 client.?.objecter ms_handle_connect 0x7f20dc0a57a0 2024-08-19T12:38:38.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:38.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2724509064 0 0) 0x7f20dc136c50 con 0x7f20dc0a57a0 2024-08-19T12:38:38.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): my global_id is 4375 2024-08-19T12:38:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20dc0a4630 con 0x7f20dc0a57a0 2024-08-19T12:38:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2546871515 0 0) 0x7f20dc136f30 con 0x7f20dc13f2e0 2024-08-19T12:38:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): my global_id is 4399 2024-08-19T12:38:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20dc136c50 con 0x7f20dc13f2e0 2024-08-19T12:38:38.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1858888804 0 0) 0x7f20bc004eb0 con 0x7f20dc073470 2024-08-19T12:38:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20dc136f30 con 0x7f20dc073470 2024-08-19T12:38:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20d4003cd0 con 0x7f20dc073470 2024-08-19T12:38:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.707 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1563817341 0 0) 0x7f20dc136f30 con 0x7f20dc073470 2024-08-19T12:38:38.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 >> v1:172.21.15.104:6792/0 conn(0x7f20dc13f2e0 legacy=0x7f20dc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 >> v1:172.21.15.104:6789/0 conn(0x7f20dc0a57a0 legacy=0x7f20dc0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 monclient: found mon.e 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20dc0aa3c0 con 0x7f20dc073470 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20e09c9640 5 monclient: authenticate success, global_id 4499 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20e09c9640 10 monclient: _renew_subs 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20e09c9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:38.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20e09c9640 1 -- 172.21.15.26:0/2374930735 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f20dc13edb0 con 0x7f20dc073470 2024-08-19T12:38:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20e09c9640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20e09c9640 10 monclient: _renew_subs 2024-08-19T12:38:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20e09c9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20e09c9640 1 -- 172.21.15.26:0/2374930735 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f20dc0a9f00 con 0x7f20dc073470 2024-08-19T12:38:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20e09c9640 1 librados: init done 2024-08-19T12:38:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20d4001f10 con 0x7f20dc073470 2024-08-19T12:38:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20d4004460 con 0x7f20dc073470 2024-08-19T12:38:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:38.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d8d99640 4 set_mon_vals no callback set 2024-08-19T12:38:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.690+0000 7f20d2ffd640 10 monclient: dump: 2024-08-19T12:38:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:38.711 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:38.712 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.691+0000 7f20e09c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:38.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.691+0000 7f20e09c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.691+0000 7f20e09c9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.691+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f20d40046b0 con 0x7f20dc073470 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.691+0000 7f20e09c9640 1 -- 172.21.15.26:0/2374930735 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c0005080 con 0x7f20dc073470 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.691+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f20d404df00 con 0x7f20dc073470 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.691+0000 7f20d2ffd640 10 client.4499.objecter ms_dispatch 0x7f20dc000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.691+0000 7f20d2ffd640 3 client.4499.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.691+0000 7f20d2ffd640 3 client.4499.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.691+0000 7f20d2ffd640 20 client.4499.objecter dump_active .. 0 homeless 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.692+0000 7f20d9d9b640 10 monclient: get_auth_request con 0x7f20bc041220 auth_method 0 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.694+0000 7f20d2ffd640 10 client.4499.objecter ms_handle_connect 0x7f20bc041220 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.696+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f20d4017a20 con 0x7f20dc073470 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.696+0000 7f20d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:38.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.696+0000 7f20d2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.877+0000 7f20e09c9640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-08-19T12:38:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.877+0000 7f20e09c9640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-08-19T12:38:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.877+0000 7f20e09c9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.877+0000 7f20e09c9640 1 -- 172.21.15.26:0/2374930735 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f20c0005c40 con 0x7f20dc073470 2024-08-19T12:38:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.878+0000 7f20d2ffd640 1 -- 172.21.15.26:0/2374930735 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (unknown 3471955896 0 2170607528) 0x7f20d4017320 con 0x7f20dc073470 2024-08-19T12:38:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.878+0000 7f20d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-08-19T12:38:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.878+0000 7f20d2ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:38.879 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-08-19T12:38:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 10 librados: watch_flush enter 2024-08-19T12:38:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 10 librados: watch_flush exit 2024-08-19T12:38:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 20 client.4499.objecter shutdown clearing up homeless session... 2024-08-19T12:38:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 10 client.4499.objecter successfully canceled tick 2024-08-19T12:38:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 1 -- 172.21.15.26:0/2374930735 >> v1:172.21.15.104:6800/899887506 conn(0x7f20bc041220 legacy=0x7f20bc0436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 10 monclient: shutdown 2024-08-19T12:38:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 1 -- 172.21.15.26:0/2374930735 >> v1:172.21.15.26:6793/0 conn(0x7f20dc073470 legacy=0x7f20dc0a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 1 -- 172.21.15.26:0/2374930735 shutdown_connections 2024-08-19T12:38:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 1 -- 172.21.15.26:0/2374930735 >> 172.21.15.26:0/2374930735 conn(0x7f20dc0abff0 msgr2=0x7f20dc134240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 1 -- 172.21.15.26:0/2374930735 shutdown_connections 2024-08-19T12:38:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 1 -- 172.21.15.26:0/2374930735 wait complete. 2024-08-19T12:38:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.880+0000 7f20e09c9640 1 librados: shutdown 2024-08-19T12:38:38.892 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-08-19T12:38:38.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.992+0000 7ff609308640 10 monclient: get_monmap_and_config 2024-08-19T12:38:38.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.992+0000 7ff609308640 10 monclient: build_initial_monmap 2024-08-19T12:38:38.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.992+0000 7ff609308640 10 monclient: monmap: 2024-08-19T12:38:38.994 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:38.994 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:38.994 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:38.993397+0000 2024-08-19T12:38:38.994 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:38.993397+0000 2024-08-19T12:38:38.994 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:38.994 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:38.994 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:38.994 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:38.994 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:38.995 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:38.995 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:38.995 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:38.995 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:38.995 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:38.995 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:38.995 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:38.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.993+0000 7ff609308640 1 Processor -- start 2024-08-19T12:38:38.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.993+0000 7ff609308640 1 -- start start 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.993+0000 7ff609308640 10 monclient: init 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.994+0000 7ff609308640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.994+0000 7ff609308640 10 monclient: _add_conns ranks=[5,2,3,1,8,0,4,7,6] 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.994+0000 7ff609308640 10 monclient(hunting): picked mon.noname-h con 0x7ff60406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.994+0000 7ff609308640 10 monclient(hunting): picked mon.noname-b con 0x7ff6040a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.994+0000 7ff609308640 10 monclient(hunting): picked mon.noname-g con 0x7ff6040a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.994+0000 7ff609308640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff604136790 con 0x7ff6040a5920 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.994+0000 7ff609308640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff60409ddf0 con 0x7ff6040a5d50 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.994+0000 7ff6027fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff6040a5920 0x7ff60413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57006/0 (socket says 172.21.15.26:57006) 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.994+0000 7ff6027fc640 1 -- 172.21.15.26:0/489251806 learned_addr learned my addr 172.21.15.26:0/489251806 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.994+0000 7ff609308640 1 -- 172.21.15.26:0/489251806 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff60409e0d0 con 0x7ff60406e7b0 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.994+0000 7ff609308640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:38.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.994+0000 7ff609308640 10 monclient(hunting): authenticate will time out at 713.081382s 2024-08-19T12:38:39.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff6027fc640 1 -- 172.21.15.26:0/489251806 >> v1:172.21.15.26:6790/0 conn(0x7ff6040a5920 legacy=0x7ff60413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff602ffd640 1 -- 172.21.15.26:0/489251806 >> v1:172.21.15.104:6791/0 conn(0x7ff60406e7b0 legacy=0x7ff604136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2482183242 0 0) 0x7ff604136790 con 0x7ff6040a5920 2024-08-19T12:38:39.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient(hunting): my global_id is 4421 2024-08-19T12:38:39.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff6037fe640 1 -- 172.21.15.26:0/489251806 >> v1:172.21.15.104:6790/0 conn(0x7ff6040a5d50 legacy=0x7ff604141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5f0003520 con 0x7ff6040a5920 2024-08-19T12:38:39.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434758039 0 0) 0x7ff60409e0d0 con 0x7ff60406e7b0 2024-08-19T12:38:39.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient(hunting): my global_id is 4370 2024-08-19T12:38:39.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff604136790 con 0x7ff60406e7b0 2024-08-19T12:38:39.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2703952358 0 0) 0x7ff5ec0013d0 con 0x7ff6040a5d50 2024-08-19T12:38:39.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient(hunting): my global_id is 4476 2024-08-19T12:38:39.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff60409e0d0 con 0x7ff6040a5d50 2024-08-19T12:38:39.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1326706250 0 0) 0x7ff5f0003520 con 0x7ff6040a5920 2024-08-19T12:38:39.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 >> v1:172.21.15.104:6791/0 conn(0x7ff60406e7b0 legacy=0x7ff604136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 >> v1:172.21.15.104:6790/0 conn(0x7ff6040a5d50 legacy=0x7ff604141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:39.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 1 monclient: found mon.noname-b 2024-08-19T12:38:39.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:39.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:39.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60409e3b0 con 0x7ff6040a5920 2024-08-19T12:38:39.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:39.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:39.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:39.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff5f4001e60 con 0x7ff6040a5920 2024-08-19T12:38:39.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:39.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:39.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff601ffb640 10 monclient: dump: 2024-08-19T12:38:39.006 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.006 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.006 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.006 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.006 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.006 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.007 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.007 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.007 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.007 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.007 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.007 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.007 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.007 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.008 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.008 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff609308640 5 monclient: authenticate success, global_id 4421 2024-08-19T12:38:39.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.996+0000 7ff609308640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:39.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff5f4004c20 con 0x7ff6040a5920 2024-08-19T12:38:39.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff601ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:39.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff601ffb640 1 -- 172.21.15.26:0/489251806 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff5f4005190 con 0x7ff6040a5920 2024-08-19T12:38:39.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff601ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff601ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff601ffb640 10 monclient: dump: 2024-08-19T12:38:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.009 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.010 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.010 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.010 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.010 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.010 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.010 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.010 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.010 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff609308640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff609308640 4 set_mon_vals no callback set 2024-08-19T12:38:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff609308640 10 monclient: shutdown 2024-08-19T12:38:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff609308640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff609308640 1 -- 172.21.15.26:0/489251806 >> v1:172.21.15.26:6790/0 conn(0x7ff6040a5920 legacy=0x7ff60413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff609308640 1 -- 172.21.15.26:0/489251806 shutdown_connections 2024-08-19T12:38:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff609308640 1 -- 172.21.15.26:0/489251806 >> 172.21.15.26:0/489251806 conn(0x7ff6040a8740 msgr2=0x7ff6040a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff609308640 1 -- 172.21.15.26:0/489251806 shutdown_connections 2024-08-19T12:38:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.997+0000 7ff609308640 1 -- 172.21.15.26:0/489251806 wait complete. 2024-08-19T12:38:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.998+0000 7ff609308640 10 monclient: build_initial_monmap 2024-08-19T12:38:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.998+0000 7ff609308640 10 monclient: monmap: 2024-08-19T12:38:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:39.012 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:39.013 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:39.014 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.998+0000 7ff609308640 1 librados: starting msgr at 2024-08-19T12:38:39.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.998+0000 7ff609308640 1 librados: starting objecter 2024-08-19T12:38:39.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.998+0000 7ff609308640 1 Processor -- start 2024-08-19T12:38:39.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.998+0000 7ff609308640 1 -- start start 2024-08-19T12:38:39.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.998+0000 7ff609308640 1 librados: setting wanted keys 2024-08-19T12:38:39.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.998+0000 7ff609308640 1 librados: calling monclient init 2024-08-19T12:38:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.998+0000 7ff609308640 10 monclient: init 2024-08-19T12:38:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.999+0000 7ff609308640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.999+0000 7ff609308640 10 monclient: _add_conns ranks=[8,0,6,4,7,2,5,1,3] 2024-08-19T12:38:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.999+0000 7ff609308640 10 monclient(hunting): picked mon.noname-e con 0x7ff60406e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.999+0000 7ff609308640 10 monclient(hunting): picked mon.noname-a con 0x7ff6040a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.999+0000 7ff609308640 10 monclient(hunting): picked mon.noname-d con 0x7ff6040a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.999+0000 7ff609308640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff604134ee0 con 0x7ff6040a5920 2024-08-19T12:38:39.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.999+0000 7ff609308640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff604134b20 con 0x7ff6040a5d50 2024-08-19T12:38:39.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.999+0000 7ff609308640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6040a3350 con 0x7ff60406e7b0 2024-08-19T12:38:39.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.999+0000 7ff609308640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:39.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.999+0000 7ff609308640 10 monclient(hunting): authenticate will time out at 713.086440s 2024-08-19T12:38:39.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.999+0000 7ff602ffd640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7ff60406e7b0 0x7ff6040a5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40082/0 (socket says 172.21.15.26:40082) 2024-08-19T12:38:39.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:38.999+0000 7ff602ffd640 1 -- 172.21.15.26:0/2138724592 learned_addr learned my addr 172.21.15.26:0/2138724592 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:39.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff6037fe640 1 -- 172.21.15.26:0/2138724592 >> v1:172.21.15.26:6792/0 conn(0x7ff6040a5d50 legacy=0x7ff6041d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 client.?.objecter ms_handle_connect 0x7ff6040a5d50 2024-08-19T12:38:39.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:39.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff6027fc640 1 -- 172.21.15.26:0/2138724592 >> v1:172.21.15.26:6789/0 conn(0x7ff6040a5920 legacy=0x7ff6041d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff602ffd640 1 -- 172.21.15.26:0/2138724592 >> v1:172.21.15.26:6793/0 conn(0x7ff60406e7b0 legacy=0x7ff6040a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 client.?.objecter ms_handle_connect 0x7ff6040a5920 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 client.?.objecter ms_handle_connect 0x7ff60406e7b0 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 524492270 0 0) 0x7ff604134b20 con 0x7ff6040a5d50 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 monclient(hunting): my global_id is 4506 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5e4004eb0 con 0x7ff6040a5d50 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862108973 0 0) 0x7ff604134ee0 con 0x7ff6040a5920 2024-08-19T12:38:39.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 monclient(hunting): my global_id is 4392 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff604134b20 con 0x7ff6040a5920 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4256472502 0 0) 0x7ff6040a3350 con 0x7ff60406e7b0 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 10 monclient(hunting): my global_id is 4508 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.001+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff604134ee0 con 0x7ff60406e7b0 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 91301207 0 0) 0x7ff5e4004eb0 con 0x7ff6040a5d50 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6040a3350 con 0x7ff6040a5d50 2024-08-19T12:38:39.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3299429947 0 0) 0x7ff604134b20 con 0x7ff6040a5920 2024-08-19T12:38:39.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5e4004eb0 con 0x7ff6040a5920 2024-08-19T12:38:39.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff5ec001450 con 0x7ff6040a5d50 2024-08-19T12:38:39.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:39.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:39.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient(hunting): dump: 2024-08-19T12:38:39.023 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.023 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.023 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.024 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.024 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.024 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.024 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.024 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.024 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.024 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.024 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.024 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.024 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.025 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:39.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2220404798 0 0) 0x7ff604134ee0 con 0x7ff60406e7b0 2024-08-19T12:38:39.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff604134b20 con 0x7ff60406e7b0 2024-08-19T12:38:39.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff5f4005110 con 0x7ff6040a5920 2024-08-19T12:38:39.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:39.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:39.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient(hunting): dump: 2024-08-19T12:38:39.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.026 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff5fc003610 con 0x7ff60406e7b0 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient(hunting): dump: 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.027 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 490216902 0 0) 0x7ff6040a3350 con 0x7ff6040a5d50 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 >> v1:172.21.15.26:6793/0 conn(0x7ff60406e7b0 legacy=0x7ff6040a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 >> v1:172.21.15.26:6789/0 conn(0x7ff6040a5920 legacy=0x7ff6041d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 monclient: found mon.d 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:39.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6040a3070 con 0x7ff6040a5d50 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff609308640 5 monclient: authenticate success, global_id 4506 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff609308640 10 monclient: _renew_subs 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff609308640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff609308640 1 -- 172.21.15.26:0/2138724592 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff604132dc0 con 0x7ff6040a5d50 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff609308640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff609308640 10 monclient: _renew_subs 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff609308640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff609308640 1 -- 172.21.15.26:0/2138724592 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6040a2110 con 0x7ff6040a5d50 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff609308640 1 librados: init done 2024-08-19T12:38:39.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff5ec0038c0 con 0x7ff6040a5d50 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff5ec003df0 con 0x7ff6040a5d50 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff6017fa640 4 set_mon_vals no callback set 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.002+0000 7ff5fb7fe640 10 monclient: dump: 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.030 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.003+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff5ec003f90 con 0x7ff6040a5d50 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.003+0000 7ff609308640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.003+0000 7ff609308640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.003+0000 7ff609308640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:39.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.003+0000 7ff609308640 1 -- 172.21.15.26:0/2138724592 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e8005080 con 0x7ff6040a5d50 2024-08-19T12:38:39.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.003+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7ff5ec04d770 con 0x7ff6040a5d50 2024-08-19T12:38:39.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.003+0000 7ff5fb7fe640 10 client.4506.objecter ms_dispatch 0x7ff604000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:39.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.003+0000 7ff5fb7fe640 3 client.4506.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:39.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.003+0000 7ff5fb7fe640 3 client.4506.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:39.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.003+0000 7ff5fb7fe640 20 client.4506.objecter dump_active .. 0 homeless 2024-08-19T12:38:39.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.003+0000 7ff602ffd640 10 monclient: get_auth_request con 0x7ff5e4041a30 auth_method 0 2024-08-19T12:38:39.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.006+0000 7ff5fb7fe640 10 client.4506.objecter ms_handle_connect 0x7ff5e4041a30 2024-08-19T12:38:39.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.008+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7ff60409ddf0 con 0x7ff6040a5d50 2024-08-19T12:38:39.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.008+0000 7ff5fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:39.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.008+0000 7ff5fb7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:39.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.194+0000 7ff609308640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-08-19T12:38:39.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.194+0000 7ff609308640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-08-19T12:38:39.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.194+0000 7ff609308640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:39.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.194+0000 7ff609308640 1 -- 172.21.15.26:0/2138724592 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7ff5e8005c40 con 0x7ff6040a5d50 2024-08-19T12:38:39.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.194+0000 7ff5fb7fe640 1 -- 172.21.15.26:0/2138724592 <== mon.6 v1:172.21.15.26:6792/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) ==== 92+0+10 (unknown 3412311620 0 265718887) 0x7ff5ec01b0d0 con 0x7ff6040a5d50 2024-08-19T12:38:39.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.194+0000 7ff5fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-08-19T12:38:39.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.194+0000 7ff5fb7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:39.196 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 8 2024-08-19T12:38:39.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.196+0000 7ff609308640 10 librados: watch_flush enter 2024-08-19T12:38:39.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.196+0000 7ff609308640 10 librados: watch_flush exit 2024-08-19T12:38:39.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.196+0000 7ff609308640 20 client.4506.objecter shutdown clearing up homeless session... 2024-08-19T12:38:39.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.196+0000 7ff609308640 10 client.4506.objecter successfully canceled tick 2024-08-19T12:38:39.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.196+0000 7ff609308640 1 -- 172.21.15.26:0/2138724592 >> v1:172.21.15.104:6800/899887506 conn(0x7ff5e4041a30 legacy=0x7ff5e4043ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.196+0000 7ff609308640 10 monclient: shutdown 2024-08-19T12:38:39.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.196+0000 7ff609308640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:39.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.196+0000 7ff609308640 1 -- 172.21.15.26:0/2138724592 >> v1:172.21.15.26:6792/0 conn(0x7ff6040a5d50 legacy=0x7ff6041d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.197+0000 7ff609308640 1 -- 172.21.15.26:0/2138724592 shutdown_connections 2024-08-19T12:38:39.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.197+0000 7ff609308640 1 -- 172.21.15.26:0/2138724592 >> 172.21.15.26:0/2138724592 conn(0x7ff6040a8740 msgr2=0x7ff6040aa900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:39.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.197+0000 7ff609308640 1 -- 172.21.15.26:0/2138724592 shutdown_connections 2024-08-19T12:38:39.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.197+0000 7ff609308640 1 -- 172.21.15.26:0/2138724592 wait complete. 2024-08-19T12:38:39.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.197+0000 7ff609308640 1 librados: shutdown 2024-08-19T12:38:39.208 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-08-19T12:38:39.208 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T12:38:39.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.305+0000 7fdf7046e640 10 monclient: get_monmap_and_config 2024-08-19T12:38:39.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.305+0000 7fdf7046e640 10 monclient: build_initial_monmap 2024-08-19T12:38:39.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.305+0000 7fdf7046e640 10 monclient: monmap: 2024-08-19T12:38:39.307 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:39.307 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.307 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:39.306682+0000 2024-08-19T12:38:39.307 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:39.306682+0000 2024-08-19T12:38:39.307 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:39.307 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.307 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:39.307 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:39.307 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:39.307 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:39.307 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:39.308 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:39.308 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:39.308 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:39.308 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:39.308 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.306+0000 7fdf7046e640 1 Processor -- start 2024-08-19T12:38:39.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.306+0000 7fdf7046e640 1 -- start start 2024-08-19T12:38:39.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.306+0000 7fdf7046e640 10 monclient: init 2024-08-19T12:38:39.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.307+0000 7fdf7046e640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:39.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.307+0000 7fdf7046e640 10 monclient: _add_conns ranks=[5,0,8,3,6,7,1,4,2] 2024-08-19T12:38:39.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.307+0000 7fdf7046e640 10 monclient(hunting): picked mon.noname-h con 0x7fdf6806e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:39.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.307+0000 7fdf7046e640 10 monclient(hunting): picked mon.noname-a con 0x7fdf680a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:39.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.307+0000 7fdf7046e640 10 monclient(hunting): picked mon.noname-e con 0x7fdf680a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:39.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.307+0000 7fdf7046e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf68136790 con 0x7fdf680a5920 2024-08-19T12:38:39.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.307+0000 7fdf7046e640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf6809ddf0 con 0x7fdf6806e7b0 2024-08-19T12:38:39.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.307+0000 7fdf7046e640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf6809e0d0 con 0x7fdf680a5d50 2024-08-19T12:38:39.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.307+0000 7fdf7046e640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:39.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.307+0000 7fdf7046e640 10 monclient(hunting): authenticate will time out at 713.394168s 2024-08-19T12:38:39.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.307+0000 7fdf6da09640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fdf680a5920 0x7fdf6813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51364/0 (socket says 172.21.15.26:51364) 2024-08-19T12:38:39.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.307+0000 7fdf6da09640 1 -- 172.21.15.26:0/1660139335 learned_addr learned my addr 172.21.15.26:0/1660139335 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:39.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6da09640 1 -- 172.21.15.26:0/1660139335 >> v1:172.21.15.26:6789/0 conn(0x7fdf680a5920 legacy=0x7fdf6813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6ea0b640 1 -- 172.21.15.26:0/1660139335 >> v1:172.21.15.26:6793/0 conn(0x7fdf680a5d50 legacy=0x7fdf68141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6e20a640 1 -- 172.21.15.26:0/1660139335 >> v1:172.21.15.104:6791/0 conn(0x7fdf6806e7b0 legacy=0x7fdf68136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3983758736 0 0) 0x7fdf6809e0d0 con 0x7fdf680a5d50 2024-08-19T12:38:39.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient(hunting): my global_id is 4517 2024-08-19T12:38:39.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf54003520 con 0x7fdf680a5d50 2024-08-19T12:38:39.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 353603777 0 0) 0x7fdf68136790 con 0x7fdf680a5920 2024-08-19T12:38:39.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient(hunting): my global_id is 4401 2024-08-19T12:38:39.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf6809e0d0 con 0x7fdf680a5920 2024-08-19T12:38:39.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 256954851 0 0) 0x7fdf640014a0 con 0x7fdf6806e7b0 2024-08-19T12:38:39.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient(hunting): my global_id is 4379 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf68136790 con 0x7fdf6806e7b0 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1140525797 0 0) 0x7fdf54003520 con 0x7fdf680a5d50 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 >> v1:172.21.15.104:6791/0 conn(0x7fdf6806e7b0 legacy=0x7fdf68136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 >> v1:172.21.15.26:6789/0 conn(0x7fdf680a5920 legacy=0x7fdf6813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient: _finish_hunting 0 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 1 monclient: found mon.noname-e 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6809e3b0 con 0x7fdf680a5d50 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient: _finish_auth 0 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient: _check_auth_tickets 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:39.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf7046e640 5 monclient: authenticate success, global_id 4517 2024-08-19T12:38:39.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdf60001de0 con 0x7fdf680a5d50 2024-08-19T12:38:39.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf7046e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:39.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:39.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:39.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf6d208640 10 monclient: dump: 2024-08-19T12:38:39.317 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.317 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.317 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.317 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.317 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.318 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.318 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.318 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.318 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.318 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.318 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.318 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.318 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.318 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.318 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.319 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.309+0000 7fdf7046e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:39.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf600020b0 con 0x7fdf680a5d50 2024-08-19T12:38:39.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf6d208640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:39.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf6d208640 1 -- 172.21.15.26:0/1660139335 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdf60003cd0 con 0x7fdf680a5d50 2024-08-19T12:38:39.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf6d208640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:39.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf6d208640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:39.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf6d208640 10 monclient: dump: 2024-08-19T12:38:39.319 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.319 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf7046e640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf7046e640 4 set_mon_vals no callback set 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf7046e640 10 monclient: shutdown 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf7046e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf7046e640 1 -- 172.21.15.26:0/1660139335 >> v1:172.21.15.26:6793/0 conn(0x7fdf680a5d50 legacy=0x7fdf68141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf7046e640 1 -- 172.21.15.26:0/1660139335 shutdown_connections 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf7046e640 1 -- 172.21.15.26:0/1660139335 >> 172.21.15.26:0/1660139335 conn(0x7fdf680a8740 msgr2=0x7fdf680a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf7046e640 1 -- 172.21.15.26:0/1660139335 shutdown_connections 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.310+0000 7fdf7046e640 1 -- 172.21.15.26:0/1660139335 wait complete. 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.311+0000 7fdf7046e640 10 monclient: build_initial_monmap 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.311+0000 7fdf7046e640 10 monclient: monmap: 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:39.321 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.311+0000 7fdf7046e640 1 librados: starting msgr at 2024-08-19T12:38:39.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.311+0000 7fdf7046e640 1 librados: starting objecter 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.311+0000 7fdf7046e640 1 Processor -- start 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.311+0000 7fdf7046e640 1 -- start start 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.311+0000 7fdf7046e640 1 librados: setting wanted keys 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.311+0000 7fdf7046e640 1 librados: calling monclient init 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.311+0000 7fdf7046e640 10 monclient: init 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.312+0000 7fdf7046e640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.312+0000 7fdf7046e640 10 monclient: _add_conns ranks=[1,8,2,6,3,4,0,5,7] 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.312+0000 7fdf7046e640 10 monclient(hunting): picked mon.noname-f con 0x7fdf6806e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.312+0000 7fdf7046e640 10 monclient(hunting): picked mon.noname-e con 0x7fdf680a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.312+0000 7fdf7046e640 10 monclient(hunting): picked mon.noname-b con 0x7fdf680a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.312+0000 7fdf7046e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf680a2c40 con 0x7fdf6806e7b0 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.312+0000 7fdf7046e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf680a2f20 con 0x7fdf680a5d50 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.312+0000 7fdf7046e640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf680a1aa0 con 0x7fdf680a5920 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.312+0000 7fdf7046e640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.312+0000 7fdf7046e640 10 monclient(hunting): authenticate will time out at 713.399104s 2024-08-19T12:38:39.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.312+0000 7fdf6da09640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fdf680a5920 0x7fdf680a01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40100/0 (socket says 172.21.15.26:40100) 2024-08-19T12:38:39.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.312+0000 7fdf6da09640 1 -- 172.21.15.26:0/1240582568 learned_addr learned my addr 172.21.15.26:0/1240582568 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:39.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf6da09640 1 -- 172.21.15.26:0/1240582568 >> v1:172.21.15.26:6793/0 conn(0x7fdf680a5920 legacy=0x7fdf680a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 client.?.objecter ms_handle_connect 0x7fdf680a5920 2024-08-19T12:38:39.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:39.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 client.?.objecter ms_handle_connect 0x7fdf680a5d50 2024-08-19T12:38:39.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf6ea0b640 1 -- 172.21.15.26:0/1240582568 >> v1:172.21.15.26:6790/0 conn(0x7fdf680a5d50 legacy=0x7fdf680a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:39.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf6e20a640 1 -- 172.21.15.26:0/1240582568 >> v1:172.21.15.104:6789/0 conn(0x7fdf6806e7b0 legacy=0x7fdf6809fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2259297857 0 0) 0x7fdf680a1aa0 con 0x7fdf680a5920 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): my global_id is 4526 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf4c004eb0 con 0x7fdf680a5920 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 client.?.objecter ms_handle_connect 0x7fdf6806e7b0 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3307285264 0 0) 0x7fdf680a2f20 con 0x7fdf680a5d50 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): my global_id is 4430 2024-08-19T12:38:39.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf680a1aa0 con 0x7fdf680a5d50 2024-08-19T12:38:39.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1365478329 0 0) 0x7fdf680a2c40 con 0x7fdf6806e7b0 2024-08-19T12:38:39.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): my global_id is 4384 2024-08-19T12:38:39.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf680a2f20 con 0x7fdf6806e7b0 2024-08-19T12:38:39.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3022237760 0 0) 0x7fdf4c004eb0 con 0x7fdf680a5920 2024-08-19T12:38:39.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf680a2c40 con 0x7fdf680a5920 2024-08-19T12:38:39.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdf580047e0 con 0x7fdf680a5920 2024-08-19T12:38:39.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:39.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:39.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): dump: 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.331 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1167156944 0 0) 0x7fdf680a1aa0 con 0x7fdf680a5d50 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf4c004eb0 con 0x7fdf680a5d50 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdf60003cd0 con 0x7fdf680a5d50 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): dump: 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.332 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1514390114 0 0) 0x7fdf680a2f20 con 0x7fdf6806e7b0 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf680a1aa0 con 0x7fdf6806e7b0 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdf64003950 con 0x7fdf6806e7b0 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:39.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient(hunting): dump: 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.334 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 660203769 0 0) 0x7fdf680a2c40 con 0x7fdf680a5920 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 >> v1:172.21.15.26:6790/0 conn(0x7fdf680a5d50 legacy=0x7fdf680a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 >> v1:172.21.15.104:6789/0 conn(0x7fdf6806e7b0 legacy=0x7fdf6809fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.314+0000 7fdf5effd640 1 monclient: found mon.e 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf680a26d0 con 0x7fdf680a5920 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 10 monclient: _finish_auth 0 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:39.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 5 monclient: authenticate success, global_id 4526 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 10 monclient: _renew_subs 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 1 -- 172.21.15.26:0/1240582568 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdf680a2960 con 0x7fdf680a5920 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 10 monclient: _renew_subs 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 1 -- 172.21.15.26:0/1240582568 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fdf681d7480 con 0x7fdf680a5920 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 1 librados: init done 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf58004cb0 con 0x7fdf680a5920 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdf580051a0 con 0x7fdf680a5920 2024-08-19T12:38:39.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf6ca07640 4 set_mon_vals no callback set 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 10 monclient: dump: 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.337 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fdf58005440 con 0x7fdf680a5920 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.315+0000 7fdf7046e640 1 -- 172.21.15.26:0/1240582568 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf50005080 con 0x7fdf680a5920 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.316+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7fdf5804dae0 con 0x7fdf680a5920 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.316+0000 7fdf5effd640 10 client.4526.objecter ms_dispatch 0x7fdf68000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:39.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.316+0000 7fdf5effd640 3 client.4526.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:39.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.316+0000 7fdf5effd640 3 client.4526.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:39.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.316+0000 7fdf5effd640 20 client.4526.objecter dump_active .. 0 homeless 2024-08-19T12:38:39.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.316+0000 7fdf6e20a640 10 monclient: get_auth_request con 0x7fdf4c0419a0 auth_method 0 2024-08-19T12:38:39.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.318+0000 7fdf5effd640 10 client.4526.objecter ms_handle_connect 0x7fdf4c0419a0 2024-08-19T12:38:39.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.320+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fdf58017660 con 0x7fdf680a5920 2024-08-19T12:38:39.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.320+0000 7fdf5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:39.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.320+0000 7fdf5effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:39.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.500+0000 7fdf7046e640 1 -- 172.21.15.26:0/1240582568 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdf50000b90 con 0x7fdf4c0419a0 2024-08-19T12:38:39.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.502+0000 7fdf5effd640 1 -- 172.21.15.26:0/1240582568 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34123 (unknown 2965378022 0 2719398077) 0x7fdf50000b90 con 0x7fdf4c0419a0 2024-08-19T12:38:39.503 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:38:39.504 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-08-19T12:38:39.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 10 librados: watch_flush enter 2024-08-19T12:38:39.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 10 librados: watch_flush exit 2024-08-19T12:38:39.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 20 client.4526.objecter shutdown clearing up homeless session... 2024-08-19T12:38:39.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 10 client.4526.objecter successfully canceled tick 2024-08-19T12:38:39.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 1 -- 172.21.15.26:0/1240582568 >> v1:172.21.15.104:6800/899887506 conn(0x7fdf4c0419a0 legacy=0x7fdf4c043e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 10 monclient: shutdown 2024-08-19T12:38:39.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:39.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 1 -- 172.21.15.26:0/1240582568 >> v1:172.21.15.26:6793/0 conn(0x7fdf680a5920 legacy=0x7fdf680a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 1 -- 172.21.15.26:0/1240582568 shutdown_connections 2024-08-19T12:38:39.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 1 -- 172.21.15.26:0/1240582568 >> 172.21.15.26:0/1240582568 conn(0x7fdf680a8740 msgr2=0x7fdf68135150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:39.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 1 -- 172.21.15.26:0/1240582568 shutdown_connections 2024-08-19T12:38:39.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 1 -- 172.21.15.26:0/1240582568 wait complete. 2024-08-19T12:38:39.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.504+0000 7fdf7046e640 1 librados: shutdown 2024-08-19T12:38:39.515 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-08-19T12:38:39.095665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":171600,"kb_used_data":1080,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566059440,"statfs":{"total":579820584960,"available":579644866560,"internally_reserved":0,"allocated":1105920,"data_stored":1372324,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9566,"internal_metadata":174447266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001289"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.958765+0000","last_change":"2024-08-19T12:38:31.959032+0000","last_active":"2024-08-19T12:38:31.958765+0000","last_peered":"2024-08-19T12:38:31.958765+0000","last_clean":"2024-08-19T12:38:31.958765+0000","last_became_active":"2024-08-19T12:38:29.948183+0000","last_became_peered":"2024-08-19T12:38:29.948183+0000","last_unstale":"2024-08-19T12:38:31.958765+0000","last_undegraded":"2024-08-19T12:38:31.958765+0000","last_fullsized":"2024-08-19T12:38:31.958765+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:51:55.545957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052740100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.963149+0000","last_change":"2024-08-19T12:38:31.963379+0000","last_active":"2024-08-19T12:38:31.963149+0000","last_peered":"2024-08-19T12:38:31.963149+0000","last_clean":"2024-08-19T12:38:31.963149+0000","last_became_active":"2024-08-19T12:38:29.950467+0000","last_became_peered":"2024-08-19T12:38:29.950467+0000","last_unstale":"2024-08-19T12:38:31.963149+0000","last_undegraded":"2024-08-19T12:38:31.963149+0000","last_fullsized":"2024-08-19T12:38:31.963149+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:40:07.574639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049350399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.958711+0000","last_change":"2024-08-19T12:38:31.960297+0000","last_active":"2024-08-19T12:38:31.958711+0000","last_peered":"2024-08-19T12:38:31.958711+0000","last_clean":"2024-08-19T12:38:31.958711+0000","last_became_active":"2024-08-19T12:38:29.949846+0000","last_became_peered":"2024-08-19T12:38:29.949846+0000","last_unstale":"2024-08-19T12:38:31.958711+0000","last_undegraded":"2024-08-19T12:38:31.958711+0000","last_fullsized":"2024-08-19T12:38:31.958711+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:34:31.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001822192,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.963092+0000","last_change":"2024-08-19T12:38:31.963352+0000","last_active":"2024-08-19T12:38:31.963092+0000","last_peered":"2024-08-19T12:38:31.963092+0000","last_clean":"2024-08-19T12:38:31.963092+0000","last_became_active":"2024-08-19T12:38:29.951233+0000","last_became_peered":"2024-08-19T12:38:29.951233+0000","last_unstale":"2024-08-19T12:38:31.963092+0000","last_undegraded":"2024-08-19T12:38:31.963092+0000","last_fullsized":"2024-08-19T12:38:31.963092+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:45:51.806389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048735899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-19T12:38:36.974119+0000","last_change":"2024-08-19T12:38:31.959929+0000","last_active":"2024-08-19T12:38:36.974119+0000","last_peered":"2024-08-19T12:38:36.974119+0000","last_clean":"2024-08-19T12:38:36.974119+0000","last_became_active":"2024-08-19T12:38:29.951392+0000","last_became_peered":"2024-08-19T12:38:29.951392+0000","last_unstale":"2024-08-19T12:38:36.974119+0000","last_undegraded":"2024-08-19T12:38:36.974119+0000","last_fullsized":"2024-08-19T12:38:36.974119+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:47:44.756214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056602099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.957194+0000","last_change":"2024-08-19T12:38:31.957412+0000","last_active":"2024-08-19T12:38:31.957194+0000","last_peered":"2024-08-19T12:38:31.957194+0000","last_clean":"2024-08-19T12:38:31.957194+0000","last_became_active":"2024-08-19T12:38:29.952459+0000","last_became_peered":"2024-08-19T12:38:29.952459+0000","last_unstale":"2024-08-19T12:38:31.957194+0000","last_undegraded":"2024-08-19T12:38:31.957194+0000","last_fullsized":"2024-08-19T12:38:31.957194+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:47:27.910005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039863799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.959093+0000","last_change":"2024-08-19T12:38:31.960408+0000","last_active":"2024-08-19T12:38:31.959093+0000","last_peered":"2024-08-19T12:38:31.959093+0000","last_clean":"2024-08-19T12:38:31.959093+0000","last_became_active":"2024-08-19T12:38:29.951331+0000","last_became_peered":"2024-08-19T12:38:29.951331+0000","last_unstale":"2024-08-19T12:38:31.959093+0000","last_undegraded":"2024-08-19T12:38:31.959093+0000","last_fullsized":"2024-08-19T12:38:31.959093+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:55:28.246342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018789830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":23,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-19T12:38:36.973995+0000","last_change":"2024-08-19T12:38:31.959938+0000","last_active":"2024-08-19T12:38:36.973995+0000","last_peered":"2024-08-19T12:38:36.973995+0000","last_clean":"2024-08-19T12:38:36.973995+0000","last_became_active":"2024-08-19T12:38:29.952471+0000","last_became_peered":"2024-08-19T12:38:29.952471+0000","last_unstale":"2024-08-19T12:38:36.973995+0000","last_undegraded":"2024-08-19T12:38:36.973995+0000","last_fullsized":"2024-08-19T12:38:36.973995+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:48:18.927151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057881800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.958676+0000","last_change":"2024-08-19T12:38:22.728884+0000","last_active":"2024-08-19T12:38:31.958676+0000","last_peered":"2024-08-19T12:38:31.958676+0000","last_clean":"2024-08-19T12:38:31.958676+0000","last_became_active":"2024-08-19T12:38:22.728595+0000","last_became_peered":"2024-08-19T12:38:22.728595+0000","last_unstale":"2024-08-19T12:38:31.958676+0000","last_undegraded":"2024-08-19T12:38:31.958676+0000","last_fullsized":"2024-08-19T12:38:31.958676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_clean_scrub_stamp":"2024-08-19T12:38:21.711006+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:42:50.800603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":55559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":580706,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":167936,"data_stored":51797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":217088,"data_stored":580706,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T12:38:39.516 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T12:38:39.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.614+0000 7fbed3188640 10 monclient: get_monmap_and_config 2024-08-19T12:38:39.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.614+0000 7fbed3188640 10 monclient: build_initial_monmap 2024-08-19T12:38:39.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.614+0000 7fbed3188640 10 monclient: monmap: 2024-08-19T12:38:39.615 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:39.616 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.616 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:39.615451+0000 2024-08-19T12:38:39.616 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:39.615451+0000 2024-08-19T12:38:39.616 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:39.616 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.616 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:39.616 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:39.616 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:39.616 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:39.616 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:39.617 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:39.617 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:39.617 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:39.617 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:39.617 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 1 Processor -- start 2024-08-19T12:38:39.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 1 -- start start 2024-08-19T12:38:39.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 10 monclient: init 2024-08-19T12:38:39.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:39.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 10 monclient: _add_conns ranks=[1,0,2,6,4,5,8,3,7] 2024-08-19T12:38:39.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 10 monclient(hunting): picked mon.noname-f con 0x7fbecc06e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:39.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 10 monclient(hunting): picked mon.noname-a con 0x7fbecc0a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:39.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 10 monclient(hunting): picked mon.noname-b con 0x7fbecc0a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:39.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbecc136790 con 0x7fbecc0a5920 2024-08-19T12:38:39.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbecc09ddf0 con 0x7fbecc06e7b0 2024-08-19T12:38:39.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbecc09e0d0 con 0x7fbecc0a5d50 2024-08-19T12:38:39.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:39.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed3188640 10 monclient(hunting): authenticate will time out at 713.702872s 2024-08-19T12:38:39.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbec3fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fbecc0a5920 0x7fbecc13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51368/0 (socket says 172.21.15.26:51368) 2024-08-19T12:38:39.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.615+0000 7fbed1725640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbecc0a5d50 0x7fbecc141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57034/0 (socket says 172.21.15.26:57034) 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.616+0000 7fbec3fff640 1 -- 172.21.15.26:0/3979493 learned_addr learned my addr 172.21.15.26:0/3979493 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec3fff640 1 -- 172.21.15.26:0/3979493 >> v1:172.21.15.26:6789/0 conn(0x7fbecc0a5920 legacy=0x7fbecc13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbed1725640 1 -- 172.21.15.26:0/3979493 >> v1:172.21.15.26:6790/0 conn(0x7fbecc0a5d50 legacy=0x7fbecc141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 1 -- 172.21.15.26:0/3979493 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2753846908 0 0) 0x7fbecc09e0d0 con 0x7fbecc0a5d50 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient(hunting): my global_id is 4439 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 1 -- 172.21.15.26:0/3979493 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeb4003520 con 0x7fbecc0a5d50 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 1 -- 172.21.15.26:0/3979493 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1673011918 0 0) 0x7fbecc136790 con 0x7fbecc0a5920 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient(hunting): my global_id is 4410 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 1 -- 172.21.15.26:0/3979493 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbecc09e0d0 con 0x7fbecc0a5920 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbed0f24640 1 -- 172.21.15.26:0/3979493 >> v1:172.21.15.104:6789/0 conn(0x7fbecc06e7b0 legacy=0x7fbecc136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 1 -- 172.21.15.26:0/3979493 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3603529805 0 0) 0x7fbeb4003520 con 0x7fbecc0a5d50 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 1 -- 172.21.15.26:0/3979493 >> v1:172.21.15.104:6789/0 conn(0x7fbecc06e7b0 legacy=0x7fbecc136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 1 -- 172.21.15.26:0/3979493 >> v1:172.21.15.26:6789/0 conn(0x7fbecc0a5920 legacy=0x7fbecc13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 1 monclient: found mon.noname-b 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 1 -- 172.21.15.26:0/3979493 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc09e3b0 con 0x7fbecc0a5d50 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:39.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:39.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 1 -- 172.21.15.26:0/3979493 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbec4001d90 con 0x7fbecc0a5d50 2024-08-19T12:38:39.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbed3188640 5 monclient: authenticate success, global_id 4439 2024-08-19T12:38:39.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbed3188640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:39.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:39.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:39.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient: dump: 2024-08-19T12:38:39.633 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.633 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.633 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.633 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.633 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.633 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.634 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.634 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.634 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.634 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.634 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.634 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.634 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.634 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.634 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbed3188640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 1 -- 172.21.15.26:0/3979493 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbec40020a0 con 0x7fbecc0a5d50 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 1 -- 172.21.15.26:0/3979493 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbec4004ea0 con 0x7fbecc0a5d50 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbed3188640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbed3188640 4 set_mon_vals no callback set 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.618+0000 7fbec37fe640 10 monclient: dump: 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.635 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.619+0000 7fbed3188640 10 monclient: shutdown 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.619+0000 7fbed3188640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.619+0000 7fbed3188640 1 -- 172.21.15.26:0/3979493 >> v1:172.21.15.26:6790/0 conn(0x7fbecc0a5d50 legacy=0x7fbecc141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.619+0000 7fbed3188640 1 -- 172.21.15.26:0/3979493 shutdown_connections 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.619+0000 7fbed3188640 1 -- 172.21.15.26:0/3979493 >> 172.21.15.26:0/3979493 conn(0x7fbecc0a8740 msgr2=0x7fbecc0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.619+0000 7fbed3188640 1 -- 172.21.15.26:0/3979493 shutdown_connections 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.619+0000 7fbed3188640 1 -- 172.21.15.26:0/3979493 wait complete. 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.619+0000 7fbed3188640 10 monclient: build_initial_monmap 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.619+0000 7fbed3188640 10 monclient: monmap: 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:39.637 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 1 librados: starting msgr at 2024-08-19T12:38:39.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 1 librados: starting objecter 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 1 Processor -- start 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 1 -- start start 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 1 librados: setting wanted keys 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 1 librados: calling monclient init 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 10 monclient: init 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 10 monclient: _add_conns ranks=[8,3,5,0,4,2,6,1,7] 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 10 monclient(hunting): picked mon.noname-e con 0x7fbecc06e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 10 monclient(hunting): picked mon.noname-g con 0x7fbecc0a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 10 monclient(hunting): picked mon.noname-h con 0x7fbecc0a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbecc13b380 con 0x7fbecc0a5920 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbecc138440 con 0x7fbecc0a5d50 2024-08-19T12:38:39.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbecc13b0a0 con 0x7fbecc06e7b0 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed3188640 10 monclient(hunting): authenticate will time out at 713.707825s 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed0f24640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fbecc06e7b0 0x7fbecc137620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40106/0 (socket says 172.21.15.26:40106) 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.620+0000 7fbed0f24640 1 -- 172.21.15.26:0/2781651979 learned_addr learned my addr 172.21.15.26:0/2781651979 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbed0f24640 1 -- 172.21.15.26:0/2781651979 >> v1:172.21.15.26:6793/0 conn(0x7fbecc06e7b0 legacy=0x7fbecc137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 client.?.objecter ms_handle_connect 0x7fbecc06e7b0 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 client.?.objecter ms_handle_connect 0x7fbecc0a5920 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec3fff640 1 -- 172.21.15.26:0/2781651979 >> v1:172.21.15.104:6790/0 conn(0x7fbecc0a5920 legacy=0x7fbecc137d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 799002515 0 0) 0x7fbecc13b0a0 con 0x7fbecc06e7b0 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): my global_id is 4535 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeb8004eb0 con 0x7fbecc06e7b0 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 client.?.objecter ms_handle_connect 0x7fbecc0a5d50 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbed1725640 1 -- 172.21.15.26:0/2781651979 >> v1:172.21.15.104:6791/0 conn(0x7fbecc0a5d50 legacy=0x7fbecc1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3856118141 0 0) 0x7fbecc13b380 con 0x7fbecc0a5920 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): my global_id is 4485 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbecc13b0a0 con 0x7fbecc0a5920 2024-08-19T12:38:39.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 181121428 0 0) 0x7fbecc138440 con 0x7fbecc0a5d50 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): my global_id is 4388 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbecc13b380 con 0x7fbecc0a5d50 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 332486159 0 0) 0x7fbeb8004eb0 con 0x7fbecc06e7b0 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbecc138440 con 0x7fbecc06e7b0 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbec8003610 con 0x7fbecc06e7b0 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): dump: 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.642 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.643 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.643 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.643 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.643 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.643 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.643 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.643 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.643 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.643 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.644 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.644 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.644 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.644 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.644 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:39.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3464186049 0 0) 0x7fbecc13b0a0 con 0x7fbecc0a5920 2024-08-19T12:38:39.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbeb8004eb0 con 0x7fbecc0a5920 2024-08-19T12:38:39.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbebc001320 con 0x7fbecc0a5920 2024-08-19T12:38:39.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient(hunting): dump: 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 37289641 0 0) 0x7fbecc138440 con 0x7fbecc06e7b0 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 >> v1:172.21.15.104:6791/0 conn(0x7fbecc0a5d50 legacy=0x7fbecc1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 >> v1:172.21.15.104:6790/0 conn(0x7fbecc0a5920 legacy=0x7fbecc137d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient: _finish_hunting 0 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 monclient: found mon.e 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:39.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc142af0 con 0x7fbecc06e7b0 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient: _finish_auth 0 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient: _check_auth_tickets 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbed3188640 5 monclient: authenticate success, global_id 4535 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbed3188640 10 monclient: _renew_subs 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbed3188640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbed3188640 1 -- 172.21.15.26:0/2781651979 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbecc13b7a0 con 0x7fbecc06e7b0 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbed3188640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbed3188640 10 monclient: _renew_subs 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbed3188640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbed3188640 1 -- 172.21.15.26:0/2781651979 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fbecc1d79b0 con 0x7fbecc06e7b0 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbed3188640 1 librados: init done 2024-08-19T12:38:39.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbec8003b90 con 0x7fbecc06e7b0 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbec80040c0 con 0x7fbecc06e7b0 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec2ffd640 4 set_mon_vals no callback set 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.623+0000 7fbec17fa640 10 monclient: dump: 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.648 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.624+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fbec8004260 con 0x7fbecc06e7b0 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.624+0000 7fbed3188640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.624+0000 7fbed3188640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.624+0000 7fbed3188640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.624+0000 7fbed3188640 1 -- 172.21.15.26:0/2781651979 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeac005080 con 0x7fbecc06e7b0 2024-08-19T12:38:39.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.625+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7fbec804da90 con 0x7fbecc06e7b0 2024-08-19T12:38:39.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.625+0000 7fbec17fa640 10 client.4535.objecter ms_dispatch 0x7fbecc000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:39.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.625+0000 7fbec17fa640 3 client.4535.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:39.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.625+0000 7fbec17fa640 3 client.4535.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:39.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.625+0000 7fbec17fa640 20 client.4535.objecter dump_active .. 0 homeless 2024-08-19T12:38:39.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.625+0000 7fbec3fff640 10 monclient: get_auth_request con 0x7fbeb8041800 auth_method 0 2024-08-19T12:38:39.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.627+0000 7fbec17fa640 10 client.4535.objecter ms_handle_connect 0x7fbeb8041800 2024-08-19T12:38:39.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.629+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fbecc09ddf0 con 0x7fbecc06e7b0 2024-08-19T12:38:39.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.629+0000 7fbec17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:39.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.629+0000 7fbec17fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:39.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.810+0000 7fbed3188640 1 -- 172.21.15.26:0/2781651979 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbeac000b90 con 0x7fbeb8041800 2024-08-19T12:38:39.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.811+0000 7fbec17fa640 1 -- 172.21.15.26:0/2781651979 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34123 (unknown 2965378022 0 2719398077) 0x7fbeac000b90 con 0x7fbeb8041800 2024-08-19T12:38:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:38:39.813 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.813+0000 7fbed3188640 10 librados: watch_flush enter 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.813+0000 7fbed3188640 10 librados: watch_flush exit 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.813+0000 7fbed3188640 20 client.4535.objecter shutdown clearing up homeless session... 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.813+0000 7fbed3188640 10 client.4535.objecter successfully canceled tick 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.813+0000 7fbed3188640 1 -- 172.21.15.26:0/2781651979 >> v1:172.21.15.104:6800/899887506 conn(0x7fbeb8041800 legacy=0x7fbeb8043cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.813+0000 7fbed3188640 10 monclient: shutdown 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.813+0000 7fbed3188640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.813+0000 7fbed3188640 1 -- 172.21.15.26:0/2781651979 >> v1:172.21.15.26:6793/0 conn(0x7fbecc06e7b0 legacy=0x7fbecc137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.813+0000 7fbed3188640 1 -- 172.21.15.26:0/2781651979 shutdown_connections 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.813+0000 7fbed3188640 1 -- 172.21.15.26:0/2781651979 >> 172.21.15.26:0/2781651979 conn(0x7fbecc0a8740 msgr2=0x7fbecc0aaf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.813+0000 7fbed3188640 1 -- 172.21.15.26:0/2781651979 shutdown_connections 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.813+0000 7fbed3188640 1 -- 172.21.15.26:0/2781651979 wait complete. 2024-08-19T12:38:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.814+0000 7fbed3188640 1 librados: shutdown 2024-08-19T12:38:39.825 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-08-19T12:38:39.095665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":171600,"kb_used_data":1080,"kb_used_omap":9,"kb_used_meta":170358,"kb_avail":566059440,"statfs":{"total":579820584960,"available":579644866560,"internally_reserved":0,"allocated":1105920,"data_stored":1372324,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9566,"internal_metadata":174447266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001289"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.958765+0000","last_change":"2024-08-19T12:38:31.959032+0000","last_active":"2024-08-19T12:38:31.958765+0000","last_peered":"2024-08-19T12:38:31.958765+0000","last_clean":"2024-08-19T12:38:31.958765+0000","last_became_active":"2024-08-19T12:38:29.948183+0000","last_became_peered":"2024-08-19T12:38:29.948183+0000","last_unstale":"2024-08-19T12:38:31.958765+0000","last_undegraded":"2024-08-19T12:38:31.958765+0000","last_fullsized":"2024-08-19T12:38:31.958765+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:51:55.545957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052740100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.963149+0000","last_change":"2024-08-19T12:38:31.963379+0000","last_active":"2024-08-19T12:38:31.963149+0000","last_peered":"2024-08-19T12:38:31.963149+0000","last_clean":"2024-08-19T12:38:31.963149+0000","last_became_active":"2024-08-19T12:38:29.950467+0000","last_became_peered":"2024-08-19T12:38:29.950467+0000","last_unstale":"2024-08-19T12:38:31.963149+0000","last_undegraded":"2024-08-19T12:38:31.963149+0000","last_fullsized":"2024-08-19T12:38:31.963149+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:40:07.574639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049350399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.958711+0000","last_change":"2024-08-19T12:38:31.960297+0000","last_active":"2024-08-19T12:38:31.958711+0000","last_peered":"2024-08-19T12:38:31.958711+0000","last_clean":"2024-08-19T12:38:31.958711+0000","last_became_active":"2024-08-19T12:38:29.949846+0000","last_became_peered":"2024-08-19T12:38:29.949846+0000","last_unstale":"2024-08-19T12:38:31.958711+0000","last_undegraded":"2024-08-19T12:38:31.958711+0000","last_fullsized":"2024-08-19T12:38:31.958711+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:34:31.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001822192,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.963092+0000","last_change":"2024-08-19T12:38:31.963352+0000","last_active":"2024-08-19T12:38:31.963092+0000","last_peered":"2024-08-19T12:38:31.963092+0000","last_clean":"2024-08-19T12:38:31.963092+0000","last_became_active":"2024-08-19T12:38:29.951233+0000","last_became_peered":"2024-08-19T12:38:29.951233+0000","last_unstale":"2024-08-19T12:38:31.963092+0000","last_undegraded":"2024-08-19T12:38:31.963092+0000","last_fullsized":"2024-08-19T12:38:31.963092+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:45:51.806389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048735899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-19T12:38:36.974119+0000","last_change":"2024-08-19T12:38:31.959929+0000","last_active":"2024-08-19T12:38:36.974119+0000","last_peered":"2024-08-19T12:38:36.974119+0000","last_clean":"2024-08-19T12:38:36.974119+0000","last_became_active":"2024-08-19T12:38:29.951392+0000","last_became_peered":"2024-08-19T12:38:29.951392+0000","last_unstale":"2024-08-19T12:38:36.974119+0000","last_undegraded":"2024-08-19T12:38:36.974119+0000","last_fullsized":"2024-08-19T12:38:36.974119+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:47:44.756214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056602099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.957194+0000","last_change":"2024-08-19T12:38:31.957412+0000","last_active":"2024-08-19T12:38:31.957194+0000","last_peered":"2024-08-19T12:38:31.957194+0000","last_clean":"2024-08-19T12:38:31.957194+0000","last_became_active":"2024-08-19T12:38:29.952459+0000","last_became_peered":"2024-08-19T12:38:29.952459+0000","last_unstale":"2024-08-19T12:38:31.957194+0000","last_undegraded":"2024-08-19T12:38:31.957194+0000","last_fullsized":"2024-08-19T12:38:31.957194+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:47:27.910005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039863799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.959093+0000","last_change":"2024-08-19T12:38:31.960408+0000","last_active":"2024-08-19T12:38:31.959093+0000","last_peered":"2024-08-19T12:38:31.959093+0000","last_clean":"2024-08-19T12:38:31.959093+0000","last_became_active":"2024-08-19T12:38:29.951331+0000","last_became_peered":"2024-08-19T12:38:29.951331+0000","last_unstale":"2024-08-19T12:38:31.959093+0000","last_undegraded":"2024-08-19T12:38:31.959093+0000","last_fullsized":"2024-08-19T12:38:31.959093+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:55:28.246342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018789830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":23,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-08-19T12:38:36.973995+0000","last_change":"2024-08-19T12:38:31.959938+0000","last_active":"2024-08-19T12:38:36.973995+0000","last_peered":"2024-08-19T12:38:36.973995+0000","last_clean":"2024-08-19T12:38:36.973995+0000","last_became_active":"2024-08-19T12:38:29.952471+0000","last_became_peered":"2024-08-19T12:38:29.952471+0000","last_unstale":"2024-08-19T12:38:36.973995+0000","last_undegraded":"2024-08-19T12:38:36.973995+0000","last_fullsized":"2024-08-19T12:38:36.973995+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:48:18.927151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057881800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-08-19T12:38:31.958676+0000","last_change":"2024-08-19T12:38:22.728884+0000","last_active":"2024-08-19T12:38:31.958676+0000","last_peered":"2024-08-19T12:38:31.958676+0000","last_clean":"2024-08-19T12:38:31.958676+0000","last_became_active":"2024-08-19T12:38:22.728595+0000","last_became_peered":"2024-08-19T12:38:22.728595+0000","last_unstale":"2024-08-19T12:38:31.958676+0000","last_undegraded":"2024-08-19T12:38:31.958676+0000","last_fullsized":"2024-08-19T12:38:31.958676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_clean_scrub_stamp":"2024-08-19T12:38:21.711006+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:42:50.800603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":55559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":580706,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":163840,"data_stored":51778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":167936,"data_stored":51797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":217088,"data_stored":580706,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T12:38:39.826 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-08-19T12:38:39.826 DEBUG:tasks.mon_thrash:Yielding 2024-08-19T12:38:39.826 INFO:teuthology.run_tasks:Running task workunit... 2024-08-19T12:38:39.843 INFO:tasks.workunit:Pulling workunits from ref 964adda23da9f7b057cb34fdc0b82aeb7ed65324 2024-08-19T12:38:39.844 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-19T12:38:39.844 DEBUG:teuthology.orchestra.run.smithi104:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-19T12:38:39.844 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-08-19T12:38:39.844 INFO:tasks.mon_thrash.mon_thrasher:seed: 1724071119, revive delay: 90.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-08-19T12:38:39.845 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:38:39.845 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:38:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T12:38:39.864 INFO:teuthology.orchestra.run.smithi104.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-19T12:38:39.865 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-19T12:38:39.923 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-19T12:38:39.923 DEBUG:teuthology.orchestra.run.smithi104:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-19T12:38:39.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.941+0000 7fe046838640 10 monclient: get_monmap_and_config 2024-08-19T12:38:39.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.941+0000 7fe046838640 10 monclient: build_initial_monmap 2024-08-19T12:38:39.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.942+0000 7fe046838640 10 monclient: monmap: 2024-08-19T12:38:39.943 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:39.943 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.943 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:39.942825+0000 2024-08-19T12:38:39.943 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:39.942825+0000 2024-08-19T12:38:39.943 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:39.943 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.943 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:39.943 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:39.944 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:39.944 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:39.944 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:39.944 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:39.944 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:39.944 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:39.944 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:39.944 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.942+0000 7fe046838640 1 Processor -- start 2024-08-19T12:38:39.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.942+0000 7fe046838640 1 -- start start 2024-08-19T12:38:39.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.942+0000 7fe046838640 10 monclient: init 2024-08-19T12:38:39.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.943+0000 7fe046838640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:39.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.943+0000 7fe046838640 10 monclient: _add_conns ranks=[4,1,3,5,6,0,2,7,8] 2024-08-19T12:38:39.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.943+0000 7fe046838640 10 monclient(hunting): picked mon.noname-c con 0x7fe04006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.943+0000 7fe046838640 10 monclient(hunting): picked mon.noname-f con 0x7fe0400a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.943+0000 7fe046838640 10 monclient(hunting): picked mon.noname-g con 0x7fe0400a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.943+0000 7fe046838640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe040136790 con 0x7fe0400a5920 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.943+0000 7fe046838640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe04009ddf0 con 0x7fe0400a5d50 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.943+0000 7fe046838640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe04009e0d0 con 0x7fe04006e7b0 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.943+0000 7fe046838640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.943+0000 7fe046838640 10 monclient(hunting): authenticate will time out at 714.030443s 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.943+0000 7fe03ffff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fe04006e7b0 0x7fe040136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:43260/0 (socket says 172.21.15.26:43260) 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.943+0000 7fe03ffff640 1 -- 172.21.15.26:0/2302749255 learned_addr learned my addr 172.21.15.26:0/2302749255 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.945+0000 7fe03ffff640 1 -- 172.21.15.26:0/2302749255 >> v1:172.21.15.26:6791/0 conn(0x7fe04006e7b0 legacy=0x7fe040136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.945+0000 7fe03effd640 1 -- 172.21.15.26:0/2302749255 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 128865203 0 0) 0x7fe04009e0d0 con 0x7fe04006e7b0 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.945+0000 7fe03effd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.945+0000 7fe03effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.945+0000 7fe03effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.945+0000 7fe03effd640 10 monclient(hunting): my global_id is 4432 2024-08-19T12:38:39.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.945+0000 7fe03effd640 1 -- 172.21.15.26:0/2302749255 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe028003520 con 0x7fe04006e7b0 2024-08-19T12:38:39.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.945+0000 7fe03f7fe640 1 -- 172.21.15.26:0/2302749255 >> v1:172.21.15.104:6789/0 conn(0x7fe0400a5920 legacy=0x7fe04013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.945+0000 7fe044dd5640 1 -- 172.21.15.26:0/2302749255 >> v1:172.21.15.104:6790/0 conn(0x7fe0400a5d50 legacy=0x7fe040141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.945+0000 7fe03effd640 1 -- 172.21.15.26:0/2302749255 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3786602735 0 0) 0x7fe028003520 con 0x7fe04006e7b0 2024-08-19T12:38:39.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 1 -- 172.21.15.26:0/2302749255 >> v1:172.21.15.104:6790/0 conn(0x7fe0400a5d50 legacy=0x7fe040141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 1 -- 172.21.15.26:0/2302749255 >> v1:172.21.15.104:6789/0 conn(0x7fe0400a5920 legacy=0x7fe04013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:39.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 1 monclient: found mon.noname-c 2024-08-19T12:38:39.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:39.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 1 -- 172.21.15.26:0/2302749255 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe04009e3b0 con 0x7fe04006e7b0 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 10 monclient: _finish_auth 0 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 1 -- 172.21.15.26:0/2302749255 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe030001ae0 con 0x7fe04006e7b0 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 10 monclient: dump: 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.951 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe046838640 5 monclient: authenticate success, global_id 4432 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe046838640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:39.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 1 -- 172.21.15.26:0/2302749255 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe0300048a0 con 0x7fe04006e7b0 2024-08-19T12:38:39.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:39.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 1 -- 172.21.15.26:0/2302749255 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe030004dd0 con 0x7fe04006e7b0 2024-08-19T12:38:39.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:39.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:39.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe03effd640 10 monclient: dump: 2024-08-19T12:38:39.953 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.953 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.953 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.954 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.954 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.954 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.954 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.954 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.954 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.954 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.954 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.954 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.954 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.955 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.955 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.955 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe046838640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:39.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe046838640 4 set_mon_vals no callback set 2024-08-19T12:38:39.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe046838640 10 monclient: shutdown 2024-08-19T12:38:39.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe046838640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:39.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe046838640 1 -- 172.21.15.26:0/2302749255 >> v1:172.21.15.26:6791/0 conn(0x7fe04006e7b0 legacy=0x7fe040136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe046838640 1 -- 172.21.15.26:0/2302749255 shutdown_connections 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe046838640 1 -- 172.21.15.26:0/2302749255 >> 172.21.15.26:0/2302749255 conn(0x7fe0400a8740 msgr2=0x7fe0400a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe046838640 1 -- 172.21.15.26:0/2302749255 shutdown_connections 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.946+0000 7fe046838640 1 -- 172.21.15.26:0/2302749255 wait complete. 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.947+0000 7fe046838640 10 monclient: build_initial_monmap 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.947+0000 7fe046838640 10 monclient: monmap: 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:39.956 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.947+0000 7fe046838640 1 librados: starting msgr at 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.947+0000 7fe046838640 1 librados: starting objecter 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.947+0000 7fe046838640 1 Processor -- start 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.947+0000 7fe046838640 1 -- start start 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.947+0000 7fe046838640 1 librados: setting wanted keys 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.947+0000 7fe046838640 1 librados: calling monclient init 2024-08-19T12:38:39.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.947+0000 7fe046838640 10 monclient: init 2024-08-19T12:38:39.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.948+0000 7fe046838640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:39.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.948+0000 7fe046838640 10 monclient: _add_conns ranks=[7,3,4,0,8,5,6,1,2] 2024-08-19T12:38:39.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.948+0000 7fe046838640 10 monclient(hunting): picked mon.noname-i con 0x7fe04006e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:39.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.948+0000 7fe046838640 10 monclient(hunting): picked mon.noname-g con 0x7fe0400a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.948+0000 7fe046838640 10 monclient(hunting): picked mon.noname-c con 0x7fe0400a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.948+0000 7fe046838640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0401346f0 con 0x7fe0400a5920 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.948+0000 7fe046838640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe040138e90 con 0x7fe0400a5d50 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.948+0000 7fe046838640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0401381c0 con 0x7fe04006e7b0 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.948+0000 7fe046838640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.948+0000 7fe046838640 10 monclient(hunting): authenticate will time out at 714.035366s 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.948+0000 7fe044dd5640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fe0400a5d50 0x7fe0401d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:43272/0 (socket says 172.21.15.26:43272) 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.948+0000 7fe044dd5640 1 -- 172.21.15.26:0/3059236263 learned_addr learned my addr 172.21.15.26:0/3059236263 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe044dd5640 1 -- 172.21.15.26:0/3059236263 >> v1:172.21.15.26:6791/0 conn(0x7fe0400a5d50 legacy=0x7fe0401d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 client.?.objecter ms_handle_connect 0x7fe0400a5d50 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03f7fe640 1 -- 172.21.15.26:0/3059236263 >> v1:172.21.15.104:6790/0 conn(0x7fe0400a5920 legacy=0x7fe040137ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2201225628 0 0) 0x7fe040138e90 con 0x7fe0400a5d50 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): my global_id is 4441 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe02c004eb0 con 0x7fe0400a5d50 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 client.?.objecter ms_handle_connect 0x7fe0400a5920 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 client.?.objecter ms_handle_connect 0x7fe04006e7b0 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03ffff640 1 -- 172.21.15.26:0/3059236263 >> v1:172.21.15.104:6792/0 conn(0x7fe04006e7b0 legacy=0x7fe0401373a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1394700752 0 0) 0x7fe0401346f0 con 0x7fe0400a5920 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): my global_id is 4503 2024-08-19T12:38:39.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe040138e90 con 0x7fe0400a5920 2024-08-19T12:38:39.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2389156344 0 0) 0x7fe02c004eb0 con 0x7fe0400a5d50 2024-08-19T12:38:39.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0401346f0 con 0x7fe0400a5d50 2024-08-19T12:38:39.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe038003990 con 0x7fe0400a5d50 2024-08-19T12:38:39.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:39.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:39.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): dump: 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.965 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4115760727 0 0) 0x7fe0401381c0 con 0x7fe04006e7b0 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 10 monclient(hunting): my global_id is 4408 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.950+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe02c004eb0 con 0x7fe04006e7b0 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4164954789 0 0) 0x7fe0401346f0 con 0x7fe0400a5d50 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 >> v1:172.21.15.104:6792/0 conn(0x7fe04006e7b0 legacy=0x7fe0401373a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 >> v1:172.21.15.104:6790/0 conn(0x7fe0400a5920 legacy=0x7fe040137ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:39.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 10 monclient: _finish_hunting 0 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 1 monclient: found mon.c 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe04013b1f0 con 0x7fe0400a5d50 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 10 monclient: _finish_auth 0 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 10 monclient: _check_auth_tickets 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe046838640 5 monclient: authenticate success, global_id 4441 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe046838640 10 monclient: _renew_subs 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe046838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe046838640 1 -- 172.21.15.26:0/3059236263 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe04013a950 con 0x7fe0400a5d50 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe046838640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe046838640 10 monclient: _renew_subs 2024-08-19T12:38:39.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe046838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe046838640 1 -- 172.21.15.26:0/3059236263 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fe04013b610 con 0x7fe0400a5d50 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe046838640 1 librados: init done 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe038003e60 con 0x7fe0400a5d50 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe0380043d0 con 0x7fe0400a5d50 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03e7fc640 4 set_mon_vals no callback set 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.951+0000 7fe03cff9640 10 monclient: dump: 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.968 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.952+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe0380045f0 con 0x7fe0400a5d50 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.952+0000 7fe046838640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:39.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.952+0000 7fe046838640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.952+0000 7fe046838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.952+0000 7fe046838640 1 -- 172.21.15.26:0/3059236263 --> v1:172.21.15.26:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe020005080 con 0x7fe0400a5d50 2024-08-19T12:38:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.952+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7fe03804ddb0 con 0x7fe0400a5d50 2024-08-19T12:38:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.952+0000 7fe03cff9640 10 client.4441.objecter ms_dispatch 0x7fe040000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.952+0000 7fe03cff9640 3 client.4441.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.952+0000 7fe03cff9640 3 client.4441.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.952+0000 7fe03cff9640 20 client.4441.objecter dump_active .. 0 homeless 2024-08-19T12:38:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.952+0000 7fe03ffff640 10 monclient: get_auth_request con 0x7fe02c041040 auth_method 0 2024-08-19T12:38:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.954+0000 7fe03cff9640 10 client.4441.objecter ms_handle_connect 0x7fe02c041040 2024-08-19T12:38:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.956+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 <== mon.4 v1:172.21.15.26:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fe04009ddf0 con 0x7fe0400a5d50 2024-08-19T12:38:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.956+0000 7fe03cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:39.956+0000 7fe03cff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:39.988 INFO:tasks.workunit:timeout=3h 2024-08-19T12:38:39.988 INFO:tasks.workunit:cleanup=True 2024-08-19T12:38:39.989 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 964adda23da9f7b057cb34fdc0b82aeb7ed65324 2024-08-19T12:38:40.048 INFO:tasks.workunit.client.0.smithi104.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-19T12:38:40.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.223+0000 7fe046838640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:38:40.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.223+0000 7fe046838640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:38:40.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.223+0000 7fe046838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:40.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.223+0000 7fe046838640 1 -- 172.21.15.26:0/3059236263 --> v1:172.21.15.26:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe020005370 con 0x7fe0400a5d50 2024-08-19T12:38:40.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.223+0000 7fe03cff9640 1 -- 172.21.15.26:0/3059236263 <== mon.4 v1:172.21.15.26:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 3865730208) 0x7fe038017240 con 0x7fe0400a5d50 2024-08-19T12:38:40.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.223+0000 7fe03cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:38:40.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.223+0000 7fe03cff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:40.225 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":4,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":32,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:38:40.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 10 librados: watch_flush enter 2024-08-19T12:38:40.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 10 librados: watch_flush exit 2024-08-19T12:38:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 20 client.4441.objecter shutdown clearing up homeless session... 2024-08-19T12:38:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 10 client.4441.objecter successfully canceled tick 2024-08-19T12:38:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 1 -- 172.21.15.26:0/3059236263 >> v1:172.21.15.104:6800/899887506 conn(0x7fe02c041040 legacy=0x7fe02c043500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 10 monclient: shutdown 2024-08-19T12:38:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 1 -- 172.21.15.26:0/3059236263 >> v1:172.21.15.26:6791/0 conn(0x7fe0400a5d50 legacy=0x7fe0401d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 1 -- 172.21.15.26:0/3059236263 shutdown_connections 2024-08-19T12:38:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 1 -- 172.21.15.26:0/3059236263 >> 172.21.15.26:0/3059236263 conn(0x7fe0400a8740 msgr2=0x7fe04006dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 1 -- 172.21.15.26:0/3059236263 shutdown_connections 2024-08-19T12:38:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 1 -- 172.21.15.26:0/3059236263 wait complete. 2024-08-19T12:38:40.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.226+0000 7fe046838640 1 librados: shutdown 2024-08-19T12:38:40.238 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:38:40.238 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T12:38:40.238 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:38:40.238 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:38:40.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.337+0000 7f5687bba640 10 monclient: get_monmap_and_config 2024-08-19T12:38:40.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.337+0000 7f5687bba640 10 monclient: build_initial_monmap 2024-08-19T12:38:40.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.337+0000 7f5687bba640 10 monclient: monmap: 2024-08-19T12:38:40.338 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:40.338 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:40.337954+0000 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:40.337954+0000 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:40.340 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.337+0000 7f5687bba640 1 Processor -- start 2024-08-19T12:38:40.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.337+0000 7f5687bba640 1 -- start start 2024-08-19T12:38:40.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.337+0000 7f5687bba640 10 monclient: init 2024-08-19T12:38:40.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.338+0000 7f5687bba640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:40.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.338+0000 7f5687bba640 10 monclient: _add_conns ranks=[1,2,7,6,8,3,5,4,0] 2024-08-19T12:38:40.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.338+0000 7f5687bba640 10 monclient(hunting): picked mon.noname-f con 0x7f5680073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:40.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.338+0000 7f5687bba640 10 monclient(hunting): picked mon.noname-b con 0x7f56800a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:40.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.338+0000 7f5687bba640 10 monclient(hunting): picked mon.noname-i con 0x7f568013f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:40.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.338+0000 7f5687bba640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f568009dae0 con 0x7f5680073470 2024-08-19T12:38:40.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.338+0000 7f5687bba640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5680136910 con 0x7f56800a57a0 2024-08-19T12:38:40.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.338+0000 7f5687bba640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f568009dd50 con 0x7f568013f2e0 2024-08-19T12:38:40.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.338+0000 7f5687bba640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:40.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.338+0000 7f5687bba640 10 monclient(hunting): authenticate will time out at 714.425385s 2024-08-19T12:38:40.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.338+0000 7f5685155640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f56800a57a0 0x7f56800a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57040/0 (socket says 172.21.15.26:57040) 2024-08-19T12:38:40.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.338+0000 7f5685155640 1 -- 172.21.15.26:0/1900634119 learned_addr learned my addr 172.21.15.26:0/1900634119 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:40.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.340+0000 7f5685155640 1 -- 172.21.15.26:0/1900634119 >> v1:172.21.15.26:6790/0 conn(0x7f56800a57a0 legacy=0x7f56800a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.340+0000 7f5686157640 1 -- 172.21.15.26:0/1900634119 >> v1:172.21.15.104:6792/0 conn(0x7f568013f2e0 legacy=0x7f56801416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.340+0000 7f5685956640 1 -- 172.21.15.26:0/1900634119 >> v1:172.21.15.104:6789/0 conn(0x7f5680073470 legacy=0x7f5680136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1331011403 0 0) 0x7f5680136910 con 0x7f56800a57a0 2024-08-19T12:38:40.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient(hunting): my global_id is 4448 2024-08-19T12:38:40.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5668003520 con 0x7f56800a57a0 2024-08-19T12:38:40.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3660091360 0 0) 0x7f568009dae0 con 0x7f5680073470 2024-08-19T12:38:40.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient(hunting): my global_id is 4411 2024-08-19T12:38:40.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5680136910 con 0x7f5680073470 2024-08-19T12:38:40.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895766276 0 0) 0x7f568009dd50 con 0x7f568013f2e0 2024-08-19T12:38:40.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient(hunting): my global_id is 4417 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f568009dae0 con 0x7f568013f2e0 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3302821298 0 0) 0x7f5668003520 con 0x7f56800a57a0 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 >> v1:172.21.15.104:6792/0 conn(0x7f568013f2e0 legacy=0x7f56801416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 >> v1:172.21.15.104:6789/0 conn(0x7f5680073470 legacy=0x7f5680136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient: _finish_hunting 0 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 monclient: found mon.noname-b 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568009e030 con 0x7f56800a57a0 2024-08-19T12:38:40.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient: _finish_auth 0 2024-08-19T12:38:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient: _check_auth_tickets 2024-08-19T12:38:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5674001ae0 con 0x7f56800a57a0 2024-08-19T12:38:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5687bba640 5 monclient: authenticate success, global_id 4448 2024-08-19T12:38:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5687bba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient: dump: 2024-08-19T12:38:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.348 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.349 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.350 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5687bba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f56740048a0 con 0x7f56800a57a0 2024-08-19T12:38:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 1 -- 172.21.15.26:0/1900634119 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5674004e10 con 0x7f56800a57a0 2024-08-19T12:38:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5684954640 10 monclient: dump: 2024-08-19T12:38:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.350 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5687bba640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5687bba640 4 set_mon_vals no callback set 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5687bba640 10 monclient: shutdown 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5687bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.341+0000 7f5687bba640 1 -- 172.21.15.26:0/1900634119 >> v1:172.21.15.26:6790/0 conn(0x7f56800a57a0 legacy=0x7f56800a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.342+0000 7f5687bba640 1 -- 172.21.15.26:0/1900634119 shutdown_connections 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.342+0000 7f5687bba640 1 -- 172.21.15.26:0/1900634119 >> 172.21.15.26:0/1900634119 conn(0x7f56800abff0 msgr2=0x7f56800ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.342+0000 7f5687bba640 1 -- 172.21.15.26:0/1900634119 shutdown_connections 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.342+0000 7f5687bba640 1 -- 172.21.15.26:0/1900634119 wait complete. 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.342+0000 7f5687bba640 10 monclient: build_initial_monmap 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.342+0000 7f5687bba640 10 monclient: monmap: 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:40.352 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.342+0000 7f5687bba640 1 librados: starting msgr at 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.342+0000 7f5687bba640 1 librados: starting objecter 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.342+0000 7f5687bba640 1 Processor -- start 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 1 -- start start 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 1 librados: setting wanted keys 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 1 librados: calling monclient init 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 10 monclient: init 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 10 monclient: _add_conns ranks=[2,4,7,1,5,3,8,6,0] 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 10 monclient(hunting): picked mon.noname-b con 0x7f5680073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 10 monclient(hunting): picked mon.noname-c con 0x7f56800a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 10 monclient(hunting): picked mon.noname-i con 0x7f568013f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56801371d0 con 0x7f5680073470 2024-08-19T12:38:40.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56801353b0 con 0x7f56800a57a0 2024-08-19T12:38:40.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5680136c50 con 0x7f568013f2e0 2024-08-19T12:38:40.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:40.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5687bba640 10 monclient(hunting): authenticate will time out at 714.430466s 2024-08-19T12:38:40.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5685956640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5680073470 0x7f56800a2050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57050/0 (socket says 172.21.15.26:57050) 2024-08-19T12:38:40.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.343+0000 7f5685956640 1 -- 172.21.15.26:0/656732504 learned_addr learned my addr 172.21.15.26:0/656732504 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:40.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f5685155640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.26:6791/0 conn(0x7f56800a57a0 legacy=0x7f56800a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f5685956640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.26:6790/0 conn(0x7f5680073470 legacy=0x7f56800a2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 client.?.objecter ms_handle_connect 0x7f56800a57a0 2024-08-19T12:38:40.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:40.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 client.?.objecter ms_handle_connect 0x7f5680073470 2024-08-19T12:38:40.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:40.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 client.?.objecter ms_handle_connect 0x7f568013f2e0 2024-08-19T12:38:40.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:40.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f5686157640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.104:6792/0 conn(0x7f568013f2e0 legacy=0x7f56801d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3835817903 0 0) 0x7f56801371d0 con 0x7f5680073470 2024-08-19T12:38:40.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 monclient(hunting): my global_id is 4457 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f566c004eb0 con 0x7f5680073470 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3621856855 0 0) 0x7f5680136c50 con 0x7f568013f2e0 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 monclient(hunting): my global_id is 4426 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56801371d0 con 0x7f568013f2e0 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1151920181 0 0) 0x7f56801353b0 con 0x7f56800a57a0 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 10 monclient(hunting): my global_id is 4450 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.345+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5680136c50 con 0x7f56800a57a0 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4262226948 0 0) 0x7f56801371d0 con 0x7f568013f2e0 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56801353b0 con 0x7f568013f2e0 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 417565481 0 0) 0x7f566c004eb0 con 0x7f5680073470 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56801371d0 con 0x7f5680073470 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5678001c30 con 0x7f568013f2e0 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient(hunting): dump: 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.365 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5670003990 con 0x7f5680073470 2024-08-19T12:38:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient(hunting): dump: 2024-08-19T12:38:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3576856579 0 0) 0x7f5680136c50 con 0x7f56800a57a0 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f566c004eb0 con 0x7f56800a57a0 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5674004d90 con 0x7f56800a57a0 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient(hunting): dump: 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.369 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2720202554 0 0) 0x7f56801353b0 con 0x7f568013f2e0 2024-08-19T12:38:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.26:6791/0 conn(0x7f56800a57a0 legacy=0x7f56800a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.26:6790/0 conn(0x7f5680073470 legacy=0x7f56800a2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 monclient: found mon.i 2024-08-19T12:38:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568013eac0 con 0x7f568013f2e0 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f5687bba640 5 monclient: authenticate success, global_id 4426 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f5687bba640 10 monclient: _renew_subs 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f5687bba640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f5687bba640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f56800a2e70 con 0x7f568013f2e0 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f5687bba640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f5687bba640 10 monclient: _renew_subs 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f5687bba640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f5687bba640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f56800a4610 con 0x7f568013f2e0 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f56780039e0 con 0x7f568013f2e0 2024-08-19T12:38:40.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f5687bba640 1 librados: init done 2024-08-19T12:38:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5678003f50 con 0x7f568013f2e0 2024-08-19T12:38:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567e7fc640 10 monclient: dump: 2024-08-19T12:38:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.346+0000 7f567ffff640 4 set_mon_vals no callback set 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.347+0000 7f5687bba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.347+0000 7f5687bba640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f568006c840 con 0x7f5680142df0 2024-08-19T12:38:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.347+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f56780040f0 con 0x7f568013f2e0 2024-08-19T12:38:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.347+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f567804d8b0 con 0x7f568013f2e0 2024-08-19T12:38:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.347+0000 7f567e7fc640 10 client.4426.objecter ms_dispatch 0x7f5680000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.347+0000 7f567e7fc640 3 client.4426.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.347+0000 7f567e7fc640 3 client.4426.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.347+0000 7f567e7fc640 20 client.4426.objecter dump_active .. 0 homeless 2024-08-19T12:38:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.347+0000 7f5685155640 10 monclient: get_auth_request con 0x7f566c041920 auth_method 0 2024-08-19T12:38:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f5685956640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.26:6789/0 conn(0x7f5680142df0 legacy=0x7f56800aa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 10 client.4426.objecter ms_handle_connect 0x7f5680142df0 2024-08-19T12:38:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 10 client.4426.objecter resend_mon_ops 2024-08-19T12:38:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366813814 0 0) 0x7f568006c840 con 0x7f5680142df0 2024-08-19T12:38:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 10 monclient(hunting): my global_id is 4419 2024-08-19T12:38:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56801353b0 con 0x7f5680142df0 2024-08-19T12:38:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2882255292 0 0) 0x7f56801353b0 con 0x7f5680142df0 2024-08-19T12:38:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f568006c840 con 0x7f5680142df0 2024-08-19T12:38:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5670004550 con 0x7f5680142df0 2024-08-19T12:38:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.349+0000 7f567e7fc640 10 monclient: dump: 2024-08-19T12:38:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.350+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 605371073 0 0) 0x7f568006c840 con 0x7f5680142df0 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.350+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f56800a5570 con 0x7f5680142df0 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.350+0000 7f567e7fc640 10 client.4426.objecter ms_handle_connect 0x7f566c041920 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.350+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f56700047b0 con 0x7f5680142df0 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.350+0000 7f567e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.350+0000 7f567e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.350+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.26:6789/0 conn(0x7f5680142df0 legacy=0x7f56800aa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.350+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.26:6789/0 conn(0x7f5680142df0 legacy=0x7f56800aa000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.356+0000 7f5687bba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.356+0000 7f5687bba640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f568002e5b0 con 0x7f568002e1d0 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.358+0000 7f5685956640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.26:6789/0 conn(0x7f568002e1d0 legacy=0x7f56800aa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.358+0000 7f567e7fc640 10 client.4426.objecter ms_handle_connect 0x7f568002e1d0 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.358+0000 7f567e7fc640 10 client.4426.objecter resend_mon_ops 2024-08-19T12:38:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.358+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883902680 0 0) 0x7f568002e5b0 con 0x7f568002e1d0 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.358+0000 7f567e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.358+0000 7f567e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.358+0000 7f567e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.358+0000 7f567e7fc640 10 monclient(hunting): my global_id is 4428 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.358+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f568006c840 con 0x7f568002e1d0 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3542005090 0 0) 0x7f568006c840 con 0x7f568002e1d0 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f568002e5b0 con 0x7f568002e1d0 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5670006070 con 0x7f568002e1d0 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 10 monclient: dump: 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.378 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1396206797 0 0) 0x7f568002e5b0 con 0x7f568002e1d0 2024-08-19T12:38:40.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f56800aae20 con 0x7f568002e1d0 2024-08-19T12:38:40.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 3795194312) 0x7f56700083b0 con 0x7f568002e1d0 2024-08-19T12:38:40.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:40.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:40.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.26:6789/0 conn(0x7f568002e1d0 legacy=0x7f56800aa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f567e7fc640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.26:6789/0 conn(0x7f568002e1d0 legacy=0x7f56800aa710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:40.379 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:40.379 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:40.379 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:40.379 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 4, 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 33401, 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:40.380 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 32, 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:40.381 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:40.382 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:40.383 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:40.383 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:40.383 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.383 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:40.383 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:40.383 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:40.383 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:40.383 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:40.383 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:40.383 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:40.383 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:40.384 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:40.385 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:40.386 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.386 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.386 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.386 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:40.386 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.386 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.386 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.386 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:40.386 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:40.387 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.387 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.387 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.387 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:40.387 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:40.387 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.387 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.389 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.390 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.390 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:40.390 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:40.390 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.390 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.390 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.390 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:40.390 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.390 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:40.392 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.393 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.394 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.397 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:40.398 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:40.399 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:38:40.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.399 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.399 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:40.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.399 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:40.399 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:40.399 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:40.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.399 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.399 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:40.400 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:40.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f5687bba640 10 librados: watch_flush enter 2024-08-19T12:38:40.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f5687bba640 10 librados: watch_flush exit 2024-08-19T12:38:40.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f5687bba640 20 client.4426.objecter shutdown clearing up homeless session... 2024-08-19T12:38:40.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f5687bba640 10 client.4426.objecter successfully canceled tick 2024-08-19T12:38:40.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f5687bba640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.104:6800/899887506 conn(0x7f566c041920 legacy=0x7f566c043de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f5687bba640 10 monclient: shutdown 2024-08-19T12:38:40.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f5687bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:40.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f5687bba640 1 -- 172.21.15.26:0/656732504 >> v1:172.21.15.104:6792/0 conn(0x7f568013f2e0 legacy=0x7f56801d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f5686157640 1 -- 172.21.15.26:0/656732504 reap_dead start 2024-08-19T12:38:40.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.359+0000 7f5686157640 1 -- 172.21.15.26:0/656732504 reap_dead start 2024-08-19T12:38:40.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.360+0000 7f5687bba640 1 -- 172.21.15.26:0/656732504 shutdown_connections 2024-08-19T12:38:40.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.360+0000 7f5687bba640 1 -- 172.21.15.26:0/656732504 >> 172.21.15.26:0/656732504 conn(0x7f56800abff0 msgr2=0x7f5680134980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:40.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.360+0000 7f5687bba640 1 -- 172.21.15.26:0/656732504 shutdown_connections 2024-08-19T12:38:40.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.360+0000 7f5687bba640 1 -- 172.21.15.26:0/656732504 wait complete. 2024-08-19T12:38:40.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.360+0000 7f5687bba640 1 librados: shutdown 2024-08-19T12:38:40.402 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:38:40.402 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:38:40.403 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:38:40.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.501+0000 7f132804f640 10 monclient: get_monmap_and_config 2024-08-19T12:38:40.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.501+0000 7f132804f640 10 monclient: build_initial_monmap 2024-08-19T12:38:40.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.501+0000 7f132804f640 10 monclient: monmap: 2024-08-19T12:38:40.502 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:40.502 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:40.502414+0000 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:40.502414+0000 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:40.503 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 1 Processor -- start 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 1 -- start start 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 10 monclient: init 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 10 monclient: _add_conns ranks=[2,3,4,1,0,8,5,6,7] 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 10 monclient(hunting): picked mon.noname-b con 0x7f132006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 10 monclient(hunting): picked mon.noname-g con 0x7f13200a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 10 monclient(hunting): picked mon.noname-c con 0x7f13200a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1320136790 con 0x7f132006e7b0 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132009ddf0 con 0x7f13200a5920 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132009e0d0 con 0x7f13200a5d50 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f132804f640 10 monclient(hunting): authenticate will time out at 714.589882s 2024-08-19T12:38:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f1325deb640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f132006e7b0 0x7f1320136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57062/0 (socket says 172.21.15.26:57062) 2024-08-19T12:38:40.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.503+0000 7f1325deb640 1 -- 172.21.15.26:0/3334175729 learned_addr learned my addr 172.21.15.26:0/3334175729 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:40.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.502+0000 7f13265ec640 1 --1- 172.21.15.26:0/3334175729 >> v1:172.21.15.26:6791/0 conn(0x7f13200a5d50 0x7f1320141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:43294/0 (socket says 172.21.15.26:43294) 2024-08-19T12:38:40.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f13265ec640 1 -- 172.21.15.26:0/3334175729 >> v1:172.21.15.26:6791/0 conn(0x7f13200a5d50 legacy=0x7f1320141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1325deb640 1 -- 172.21.15.26:0/3334175729 >> v1:172.21.15.26:6790/0 conn(0x7f132006e7b0 legacy=0x7f1320136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f13255ea640 1 -- 172.21.15.26:0/3334175729 >> v1:172.21.15.104:6790/0 conn(0x7f13200a5920 legacy=0x7f132013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3229843352 0 0) 0x7f132009e0d0 con 0x7f13200a5d50 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient(hunting): my global_id is 4459 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f130c003520 con 0x7f13200a5d50 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4216703285 0 0) 0x7f1320136790 con 0x7f132006e7b0 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient(hunting): my global_id is 4466 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f132009e0d0 con 0x7f132006e7b0 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862352098 0 0) 0x7f13100014a0 con 0x7f13200a5920 2024-08-19T12:38:40.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient(hunting): my global_id is 4512 2024-08-19T12:38:40.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1320136790 con 0x7f13200a5920 2024-08-19T12:38:40.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 161696578 0 0) 0x7f130c003520 con 0x7f13200a5d50 2024-08-19T12:38:40.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 >> v1:172.21.15.104:6790/0 conn(0x7f13200a5920 legacy=0x7f132013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 >> v1:172.21.15.26:6790/0 conn(0x7f132006e7b0 legacy=0x7f1320136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient: _finish_hunting 0 2024-08-19T12:38:40.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 monclient: found mon.noname-c 2024-08-19T12:38:40.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:40.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:38:40.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132009e3b0 con 0x7f13200a5d50 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient: _finish_auth 0 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient: _check_auth_tickets 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1318001f20 con 0x7f13200a5d50 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f132804f640 5 monclient: authenticate success, global_id 4459 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f132804f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient: dump: 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.512 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f132804f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:40.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1318004b60 con 0x7f13200a5d50 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 1 -- 172.21.15.26:0/3334175729 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f13180050d0 con 0x7f13200a5d50 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f132804f640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f132804f640 4 set_mon_vals no callback set 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.505+0000 7f1324de9640 10 monclient: dump: 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.514 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.515 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.506+0000 7f132804f640 10 monclient: shutdown 2024-08-19T12:38:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.506+0000 7f132804f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.506+0000 7f132804f640 1 -- 172.21.15.26:0/3334175729 >> v1:172.21.15.26:6791/0 conn(0x7f13200a5d50 legacy=0x7f1320141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.506+0000 7f132804f640 1 -- 172.21.15.26:0/3334175729 shutdown_connections 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.506+0000 7f132804f640 1 -- 172.21.15.26:0/3334175729 >> 172.21.15.26:0/3334175729 conn(0x7f13200a8740 msgr2=0x7f13200a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.506+0000 7f132804f640 1 -- 172.21.15.26:0/3334175729 shutdown_connections 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.506+0000 7f132804f640 1 -- 172.21.15.26:0/3334175729 wait complete. 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.506+0000 7f132804f640 10 monclient: build_initial_monmap 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 10 monclient: monmap: 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.516 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 1 librados: starting msgr at 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 1 librados: starting objecter 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 1 Processor -- start 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 1 -- start start 2024-08-19T12:38:40.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 1 librados: setting wanted keys 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 1 librados: calling monclient init 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 10 monclient: init 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 10 monclient: _add_conns ranks=[4,5,3,8,1,7,6,2,0] 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 10 monclient(hunting): picked mon.noname-c con 0x7f132006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 10 monclient(hunting): picked mon.noname-h con 0x7f13200a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 10 monclient(hunting): picked mon.noname-g con 0x7f13200a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132013bd40 con 0x7f13200a5d50 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1320139000 con 0x7f132006e7b0 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1320139b60 con 0x7f13200a5920 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f132804f640 10 monclient(hunting): authenticate will time out at 714.594866s 2024-08-19T12:38:40.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f1325deb640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f132006e7b0 0x7f13201ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:43298/0 (socket says 172.21.15.26:43298) 2024-08-19T12:38:40.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.507+0000 7f1325deb640 1 -- 172.21.15.26:0/1404983129 learned_addr learned my addr 172.21.15.26:0/1404983129 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:40.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f1325deb640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.26:6791/0 conn(0x7f132006e7b0 legacy=0x7f13201ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 client.?.objecter ms_handle_connect 0x7f132006e7b0 2024-08-19T12:38:40.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:40.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 client.?.objecter ms_handle_connect 0x7f13200a5d50 2024-08-19T12:38:40.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:40.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13265ec640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.104:6790/0 conn(0x7f13200a5d50 legacy=0x7f13201d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 client.?.objecter ms_handle_connect 0x7f13200a5920 2024-08-19T12:38:40.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13255ea640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.104:6791/0 conn(0x7f13200a5920 legacy=0x7f13201d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:40.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315599734 0 0) 0x7f1320139000 con 0x7f132006e7b0 2024-08-19T12:38:40.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): my global_id is 4468 2024-08-19T12:38:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1304004eb0 con 0x7f132006e7b0 2024-08-19T12:38:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 705354695 0 0) 0x7f132009e3b0 con 0x7f13200a5d50 2024-08-19T12:38:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): my global_id is 4521 2024-08-19T12:38:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1320139000 con 0x7f13200a5d50 2024-08-19T12:38:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700779692 0 0) 0x7f1320139b60 con 0x7f13200a5920 2024-08-19T12:38:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): my global_id is 4397 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f132009e3b0 con 0x7f13200a5920 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1077365080 0 0) 0x7f1304004eb0 con 0x7f132006e7b0 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1320139b60 con 0x7f132006e7b0 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f131c001a20 con 0x7f132006e7b0 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): dump: 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2267707619 0 0) 0x7f1320139000 con 0x7f13200a5d50 2024-08-19T12:38:40.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1304004eb0 con 0x7f13200a5d50 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1318001ee0 con 0x7f13200a5d50 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient(hunting): dump: 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.529 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3342666235 0 0) 0x7f132009e3b0 con 0x7f13200a5920 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1320139000 con 0x7f13200a5920 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1975910605 0 0) 0x7f1320139b60 con 0x7f132006e7b0 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.104:6791/0 conn(0x7f13200a5920 legacy=0x7f13201d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.104:6790/0 conn(0x7f13200a5d50 legacy=0x7f13201d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 monclient: found mon.c 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132013b360 con 0x7f132006e7b0 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f132804f640 5 monclient: authenticate success, global_id 4468 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f132804f640 10 monclient: _renew_subs 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f132804f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f132804f640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f132013aac0 con 0x7f132006e7b0 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f132804f640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f132804f640 10 monclient: _renew_subs 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f132804f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:40.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f132804f640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f132013a790 con 0x7f132006e7b0 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f131c004dd0 con 0x7f132006e7b0 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f132804f640 1 librados: init done 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f131c0052c0 con 0x7f132006e7b0 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f13167fc640 10 monclient: dump: 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.532 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.510+0000 7f1317fff640 4 set_mon_vals no callback set 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.511+0000 7f132804f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:40.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.511+0000 7f132804f640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1320133c50 con 0x7f13201cebc0 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.511+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f131c011720 con 0x7f132006e7b0 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.511+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f131c04dac0 con 0x7f132006e7b0 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.511+0000 7f13167fc640 10 client.4468.objecter ms_dispatch 0x7f1320000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.511+0000 7f13167fc640 3 client.4468.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.511+0000 7f13167fc640 3 client.4468.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.512+0000 7f13167fc640 20 client.4468.objecter dump_active .. 0 homeless 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.512+0000 7f13265ec640 10 monclient: get_auth_request con 0x7f1304041790 auth_method 0 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.513+0000 7f13255ea640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.26:6790/0 conn(0x7f13201cebc0 legacy=0x7f13200a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.513+0000 7f13167fc640 10 client.4468.objecter ms_handle_connect 0x7f13201cebc0 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.513+0000 7f13167fc640 10 client.4468.objecter resend_mon_ops 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2728481873 0 0) 0x7f1320139000 con 0x7f13201cebc0 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 10 monclient(hunting): my global_id is 4475 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1320139b60 con 0x7f13201cebc0 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1879493386 0 0) 0x7f1320139b60 con 0x7f13201cebc0 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1320139000 con 0x7f13201cebc0 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 10 client.4468.objecter ms_handle_connect 0x7f1304041790 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1310003e30 con 0x7f13201cebc0 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 10 monclient: dump: 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.535 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3010989591 0 0) 0x7f1320139000 con 0x7f13201cebc0 2024-08-19T12:38:40.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13200008d0 con 0x7f13201cebc0 2024-08-19T12:38:40.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1310003e30 con 0x7f13201cebc0 2024-08-19T12:38:40.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:40.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:40.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.26:6790/0 conn(0x7f13201cebc0 legacy=0x7f13200a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.514+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.26:6790/0 conn(0x7f13201cebc0 legacy=0x7f13200a18d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:40.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.520+0000 7f132804f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:40.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.520+0000 7f132804f640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f13201cefa0 con 0x7f132002e1d0 2024-08-19T12:38:40.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13255ea640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.26:6790/0 conn(0x7f132002e1d0 legacy=0x7f13201385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 10 client.4468.objecter ms_handle_connect 0x7f132002e1d0 2024-08-19T12:38:40.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 10 client.4468.objecter resend_mon_ops 2024-08-19T12:38:40.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3351493145 0 0) 0x7f13201cefa0 con 0x7f132002e1d0 2024-08-19T12:38:40.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 10 monclient(hunting): my global_id is 4484 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1320139000 con 0x7f132002e1d0 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2277477093 0 0) 0x7f1320139000 con 0x7f132002e1d0 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13201cefa0 con 0x7f132002e1d0 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1310004440 con 0x7f132002e1d0 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 10 monclient: dump: 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.538 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.539 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1368983022 0 0) 0x7f13201cefa0 con 0x7f132002e1d0 2024-08-19T12:38:40.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f132002e5b0 con 0x7f132002e1d0 2024-08-19T12:38:40.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 2557279503) 0x7f1310007d80 con 0x7f132002e1d0 2024-08-19T12:38:40.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:40.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:40.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.26:6790/0 conn(0x7f132002e1d0 legacy=0x7f13201385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.523+0000 7f13167fc640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.26:6790/0 conn(0x7f132002e1d0 legacy=0x7f13201385f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:40.540 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:40.540 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 4, 2024-08-19T12:38:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 33578, 2024-08-19T12:38:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:40.541 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:38:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 32, 2024-08-19T12:38:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:40.542 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:40.543 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:40.544 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:40.545 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:40.546 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.547 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.548 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.549 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:40.550 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:40.551 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:40.552 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.553 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.554 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:40.560 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.561 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:40.561 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:40.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 10 librados: watch_flush enter 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 10 librados: watch_flush exit 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 20 client.4468.objecter shutdown clearing up homeless session... 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 10 client.4468.objecter successfully canceled tick 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.104:6800/899887506 conn(0x7f1304041790 legacy=0x7f1304043c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 10 monclient: shutdown 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 1 -- 172.21.15.26:0/1404983129 >> v1:172.21.15.26:6791/0 conn(0x7f132006e7b0 legacy=0x7f13201ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f13265ec640 1 -- 172.21.15.26:0/1404983129 reap_dead start 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f13265ec640 1 -- 172.21.15.26:0/1404983129 reap_dead start 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 1 -- 172.21.15.26:0/1404983129 shutdown_connections 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 1 -- 172.21.15.26:0/1404983129 >> 172.21.15.26:0/1404983129 conn(0x7f13200a8740 msgr2=0x7f13200aa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 1 -- 172.21.15.26:0/1404983129 shutdown_connections 2024-08-19T12:38:40.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 1 -- 172.21.15.26:0/1404983129 wait complete. 2024-08-19T12:38:40.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.524+0000 7f132804f640 1 librados: shutdown 2024-08-19T12:38:40.564 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:38:40.564 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:38:40.564 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:38:40.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.663+0000 7f4c4f7cf640 10 monclient: get_monmap_and_config 2024-08-19T12:38:40.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.663+0000 7f4c4f7cf640 10 monclient: build_initial_monmap 2024-08-19T12:38:40.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.663+0000 7f4c4f7cf640 10 monclient: monmap: 2024-08-19T12:38:40.664 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:40.664 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.664 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:40.664012+0000 2024-08-19T12:38:40.664 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:40.664012+0000 2024-08-19T12:38:40.664 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:40.664 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.664 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:40.664 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:40.664 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:40.665 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:40.665 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:40.665 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:40.665 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:40.665 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:40.665 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:40.665 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.663+0000 7f4c4f7cf640 1 Processor -- start 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.663+0000 7f4c4f7cf640 1 -- start start 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.663+0000 7f4c4f7cf640 10 monclient: init 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4f7cf640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4f7cf640 10 monclient: _add_conns ranks=[2,8,0,7,5,4,3,6,1] 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4f7cf640 10 monclient(hunting): picked mon.noname-b con 0x7f4c48073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4f7cf640 10 monclient(hunting): picked mon.noname-e con 0x7f4c480a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4f7cf640 10 monclient(hunting): picked mon.noname-a con 0x7f4c4813f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4f7cf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4809dae0 con 0x7f4c4813f2e0 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4f7cf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c48136910 con 0x7f4c48073470 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4f7cf640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4809dd50 con 0x7f4c480a57a0 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4f7cf640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:40.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4f7cf640 10 monclient(hunting): authenticate will time out at 714.751492s 2024-08-19T12:38:40.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4cd6a640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f4c480a57a0 0x7f4c480a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40118/0 (socket says 172.21.15.26:40118) 2024-08-19T12:38:40.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4cd6a640 1 -- 172.21.15.26:0/4220946358 learned_addr learned my addr 172.21.15.26:0/4220946358 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:40.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.664+0000 7f4c4d56b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4c48073470 0x7f4c48136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57080/0 (socket says 172.21.15.26:57080) 2024-08-19T12:38:40.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c4cd6a640 1 -- 172.21.15.26:0/4220946358 >> v1:172.21.15.26:6793/0 conn(0x7f4c480a57a0 legacy=0x7f4c480a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c4dd6c640 1 -- 172.21.15.26:0/4220946358 >> v1:172.21.15.26:6789/0 conn(0x7f4c4813f2e0 legacy=0x7f4c481416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c4d56b640 1 -- 172.21.15.26:0/4220946358 >> v1:172.21.15.26:6790/0 conn(0x7f4c48073470 legacy=0x7f4c48136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1168857958 0 0) 0x7f4c4809dd50 con 0x7f4c480a57a0 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c3ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c3ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c3ffff640 10 monclient(hunting): my global_id is 4544 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c34003520 con 0x7f4c480a57a0 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 948161467 0 0) 0x7f4c4809dae0 con 0x7f4c4813f2e0 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c3ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c3ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.666+0000 7f4c3ffff640 10 monclient(hunting): my global_id is 4437 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4809dd50 con 0x7f4c4813f2e0 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2719056231 0 0) 0x7f4c48136910 con 0x7f4c48073470 2024-08-19T12:38:40.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient(hunting): my global_id is 4493 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4809dae0 con 0x7f4c48073470 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1540578977 0 0) 0x7f4c34003520 con 0x7f4c480a57a0 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 >> v1:172.21.15.26:6790/0 conn(0x7f4c48073470 legacy=0x7f4c48136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 >> v1:172.21.15.26:6789/0 conn(0x7f4c4813f2e0 legacy=0x7f4c481416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 1 monclient: found mon.noname-e 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4809e030 con 0x7f4c480a57a0 2024-08-19T12:38:40.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient: _finish_auth 0 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c38001ae0 con 0x7f4c480a57a0 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient: dump: 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.677 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.678 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.678 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.678 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.678 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.678 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.678 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.678 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.678 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.678 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c4f7cf640 5 monclient: authenticate success, global_id 4544 2024-08-19T12:38:40.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c4f7cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:40.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4c380048a0 con 0x7f4c480a57a0 2024-08-19T12:38:40.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:40.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 1 -- 172.21.15.26:0/4220946358 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c38004d90 con 0x7f4c480a57a0 2024-08-19T12:38:40.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:40.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c3ffff640 10 monclient: dump: 2024-08-19T12:38:40.679 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.679 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.679 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.680 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c4f7cf640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c4f7cf640 4 set_mon_vals no callback set 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c4f7cf640 10 monclient: shutdown 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c4f7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/4220946358 >> v1:172.21.15.26:6793/0 conn(0x7f4c480a57a0 legacy=0x7f4c480a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/4220946358 shutdown_connections 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.667+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/4220946358 >> 172.21.15.26:0/4220946358 conn(0x7f4c480abff0 msgr2=0x7f4c480ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.668+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/4220946358 shutdown_connections 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.668+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/4220946358 wait complete. 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.668+0000 7f4c4f7cf640 10 monclient: build_initial_monmap 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.668+0000 7f4c4f7cf640 10 monclient: monmap: 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:40.682 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.668+0000 7f4c4f7cf640 1 librados: starting msgr at 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.668+0000 7f4c4f7cf640 1 librados: starting objecter 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.668+0000 7f4c4f7cf640 1 Processor -- start 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.668+0000 7f4c4f7cf640 1 -- start start 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.668+0000 7f4c4f7cf640 1 librados: setting wanted keys 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.668+0000 7f4c4f7cf640 1 librados: calling monclient init 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.668+0000 7f4c4f7cf640 10 monclient: init 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.669+0000 7f4c4f7cf640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.669+0000 7f4c4f7cf640 10 monclient: _add_conns ranks=[2,1,7,6,0,5,8,4,3] 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.669+0000 7f4c4f7cf640 10 monclient(hunting): picked mon.noname-b con 0x7f4c48073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:40.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.669+0000 7f4c4f7cf640 10 monclient(hunting): picked mon.noname-f con 0x7f4c480a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:40.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.669+0000 7f4c4f7cf640 10 monclient(hunting): picked mon.noname-i con 0x7f4c4813f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:40.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.669+0000 7f4c4f7cf640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4813f070 con 0x7f4c480a57a0 2024-08-19T12:38:40.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.669+0000 7f4c4f7cf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4812f3d0 con 0x7f4c48073470 2024-08-19T12:38:40.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.669+0000 7f4c4f7cf640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4812e8d0 con 0x7f4c4813f2e0 2024-08-19T12:38:40.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.669+0000 7f4c4f7cf640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:40.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.669+0000 7f4c4f7cf640 10 monclient(hunting): authenticate will time out at 714.756387s 2024-08-19T12:38:40.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.669+0000 7f4c4d56b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4c48073470 0x7f4c4812dab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57090/0 (socket says 172.21.15.26:57090) 2024-08-19T12:38:40.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.669+0000 7f4c4d56b640 1 -- 172.21.15.26:0/1650069993 learned_addr learned my addr 172.21.15.26:0/1650069993 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:40.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c4d56b640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6790/0 conn(0x7f4c48073470 legacy=0x7f4c4812dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c3dffb640 10 client.?.objecter ms_handle_connect 0x7f4c48073470 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c3dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c3dffb640 10 client.?.objecter ms_handle_connect 0x7f4c480a57a0 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c3dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c4cd6a640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.104:6789/0 conn(0x7f4c480a57a0 legacy=0x7f4c4812e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c3dffb640 10 client.?.objecter ms_handle_connect 0x7f4c4813f2e0 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c3dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c4dd6c640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.104:6792/0 conn(0x7f4c4813f2e0 legacy=0x7f4c481d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1399717483 0 0) 0x7f4c4812f3d0 con 0x7f4c48073470 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c3dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c3dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.671+0000 7f4c3dffb640 10 monclient(hunting): my global_id is 4502 2024-08-19T12:38:40.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c2c004eb0 con 0x7f4c48073470 2024-08-19T12:38:40.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3067606633 0 0) 0x7f4c4813f070 con 0x7f4c480a57a0 2024-08-19T12:38:40.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): my global_id is 4420 2024-08-19T12:38:40.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4812f3d0 con 0x7f4c480a57a0 2024-08-19T12:38:40.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3078578454 0 0) 0x7f4c4812e8d0 con 0x7f4c4813f2e0 2024-08-19T12:38:40.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): my global_id is 4435 2024-08-19T12:38:40.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4813f070 con 0x7f4c4813f2e0 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 366691855 0 0) 0x7f4c2c004eb0 con 0x7f4c48073470 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c4812e8d0 con 0x7f4c48073470 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c440047e0 con 0x7f4c48073470 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): dump: 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.697 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 213078297 0 0) 0x7f4c4812f3d0 con 0x7f4c480a57a0 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c2c004eb0 con 0x7f4c480a57a0 2024-08-19T12:38:40.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c38001a60 con 0x7f4c480a57a0 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): dump: 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.699 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.700 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.700 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.700 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.700 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.700 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.700 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3092828569 0 0) 0x7f4c4813f070 con 0x7f4c4813f2e0 2024-08-19T12:38:40.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c4812f3d0 con 0x7f4c4813f2e0 2024-08-19T12:38:40.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c40002010 con 0x7f4c4813f2e0 2024-08-19T12:38:40.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:40.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient(hunting): dump: 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.701 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.702 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2169027106 0 0) 0x7f4c4812e8d0 con 0x7f4c48073470 2024-08-19T12:38:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.104:6792/0 conn(0x7f4c4813f2e0 legacy=0x7f4c481d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.104:6789/0 conn(0x7f4c480a57a0 legacy=0x7f4c4812e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 monclient: found mon.b 2024-08-19T12:38:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:40.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c48132370 con 0x7f4c48073470 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c4f7cf640 5 monclient: authenticate success, global_id 4502 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c4f7cf640 10 monclient: _renew_subs 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c4f7cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4c48130e90 con 0x7f4c48073470 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4c44004c30 con 0x7f4c48073470 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c440051e0 con 0x7f4c48073470 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3dffb640 10 monclient: dump: 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.704 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c3f7fe640 4 set_mon_vals no callback set 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c4f7cf640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c4f7cf640 10 monclient: _renew_subs 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c4f7cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4c48130110 con 0x7f4c48073470 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.672+0000 7f4c4f7cf640 1 librados: init done 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f4c440116c0 con 0x7f4c48073470 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c4f7cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f4c4404db40 con 0x7f4c48073470 2024-08-19T12:38:40.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c3dffb640 10 client.4502.objecter ms_dispatch 0x7f4c48000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c3dffb640 3 client.4502.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c3dffb640 3 client.4502.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4c48131690 con 0x7f4c48142df0 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c3dffb640 20 client.4502.objecter dump_active .. 0 homeless 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c4dd6c640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6791/0 conn(0x7f4c48142df0 legacy=0x7f4c48134790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c4dd6c640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6791/0 conn(0x7f4c48142df0 legacy=0x7f4c48134790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c4dd6c640 1 --1- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6791/0 conn(0x7f4c48142df0 0x7f4c48134790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify read banner and identify addresses failed 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c4dd6c640 1 --1- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6791/0 conn(0x7f4c48142df0 0x7f4c48134790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6791/0 conn(0x7f4c48142df0 legacy=0x7f4c48134790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6791/0 conn(0x7f4c48142df0 legacy=0x7f4c48134790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4c4812e8d0 con 0x7f4c2c047dd0 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.673+0000 7f4c4cd6a640 10 monclient: get_auth_request con 0x7f4c2c041890 auth_method 0 2024-08-19T12:38:40.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 10 client.4502.objecter ms_handle_connect 0x7f4c2c047dd0 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c4dd6c640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6791/0 conn(0x7f4c2c047dd0 legacy=0x7f4c2c04a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 10 client.4502.objecter resend_mon_ops 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 10 client.4502.objecter ms_handle_connect 0x7f4c2c041890 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1380521111 0 0) 0x7f4c4812e8d0 con 0x7f4c2c047dd0 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 10 monclient(hunting): my global_id is 4477 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4813f070 con 0x7f4c2c047dd0 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 497254197 0 0) 0x7f4c4813f070 con 0x7f4c2c047dd0 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c4812e8d0 con 0x7f4c2c047dd0 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c40003fe0 con 0x7f4c2c047dd0 2024-08-19T12:38:40.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 10 monclient: dump: 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.708 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3458593532 0 0) 0x7f4c4812e8d0 con 0x7f4c2c047dd0 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c2c04ca70 con 0x7f4c2c047dd0 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4c40005b40 con 0x7f4c2c047dd0 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6791/0 conn(0x7f4c2c047dd0 legacy=0x7f4c2c04a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.676+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6791/0 conn(0x7f4c2c047dd0 legacy=0x7f4c2c04a1f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:40.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.682+0000 7f4c4f7cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.682+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4c48133270 con 0x7f4c48132e90 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c4dd6c640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6791/0 conn(0x7f4c48132e90 legacy=0x7f4c48134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 10 client.4502.objecter ms_handle_connect 0x7f4c48132e90 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 10 client.4502.objecter resend_mon_ops 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3858131225 0 0) 0x7f4c48133270 con 0x7f4c48132e90 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 10 monclient(hunting): my global_id is 4486 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4812e8d0 con 0x7f4c48132e90 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3866990340 0 0) 0x7f4c4812e8d0 con 0x7f4c48132e90 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c48133270 con 0x7f4c48132e90 2024-08-19T12:38:40.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c40005cf0 con 0x7f4c48132e90 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 10 monclient: dump: 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.711 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1305523204 0 0) 0x7f4c48133270 con 0x7f4c48132e90 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.685+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4c4806c840 con 0x7f4c48132e90 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 251218847) 0x7f4c40007d80 con 0x7f4c48132e90 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c3dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c3dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6791/0 conn(0x7f4c48132e90 legacy=0x7f4c48134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c3dffb640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6791/0 conn(0x7f4c48132e90 legacy=0x7f4c48134ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:40.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4dd6c640 1 -- 172.21.15.26:0/1650069993 reap_dead start 2024-08-19T12:38:40.713 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:40.713 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:40.713 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:40.713 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:40.713 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 4, 2024-08-19T12:38:40.713 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 33715, 2024-08-19T12:38:40.713 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:40.714 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:40.714 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:40.714 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:40.714 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:40.714 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:40.714 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:40.714 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:40.714 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:38:40.714 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:40.714 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.714 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 32, 2024-08-19T12:38:40.714 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:40.715 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:40.715 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:40.715 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:40.715 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:40.715 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:40.715 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:40.715 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:40.715 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:40.715 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:40.715 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:40.715 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:40.716 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:40.717 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:40.718 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:40.719 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.720 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.721 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.721 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:40.721 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:40.721 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.721 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:40.721 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.721 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.722 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:40.726 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.727 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.729 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.730 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.731 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:40.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.733 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:40.733 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:40.733 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:40.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.733 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.733 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:40.733 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 10 librados: watch_flush enter 2024-08-19T12:38:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 10 librados: watch_flush exit 2024-08-19T12:38:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 20 client.4502.objecter shutdown clearing up homeless session... 2024-08-19T12:38:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 10 client.4502.objecter successfully canceled tick 2024-08-19T12:38:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.104:6800/899887506 conn(0x7f4c2c041890 legacy=0x7f4c2c043d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 10 monclient: shutdown 2024-08-19T12:38:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/1650069993 >> v1:172.21.15.26:6790/0 conn(0x7f4c48073470 legacy=0x7f4c4812dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/1650069993 shutdown_connections 2024-08-19T12:38:40.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/1650069993 >> 172.21.15.26:0/1650069993 conn(0x7f4c480abff0 msgr2=0x7f4c480a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:40.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/1650069993 shutdown_connections 2024-08-19T12:38:40.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 1 -- 172.21.15.26:0/1650069993 wait complete. 2024-08-19T12:38:40.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.686+0000 7f4c4f7cf640 1 librados: shutdown 2024-08-19T12:38:40.736 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:38:40.736 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:38:40.736 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:38:40.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.836+0000 7f669b638640 10 monclient: get_monmap_and_config 2024-08-19T12:38:40.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.836+0000 7f669b638640 10 monclient: build_initial_monmap 2024-08-19T12:38:40.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.836+0000 7f669b638640 10 monclient: monmap: 2024-08-19T12:38:40.837 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:40.837508+0000 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:40.837508+0000 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:40.838 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:40.839 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:40.839 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.837+0000 7f669b638640 1 Processor -- start 2024-08-19T12:38:40.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.837+0000 7f669b638640 1 -- start start 2024-08-19T12:38:40.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.837+0000 7f669b638640 10 monclient: init 2024-08-19T12:38:40.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.837+0000 7f669b638640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:40.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.837+0000 7f669b638640 10 monclient: _add_conns ranks=[7,2,3,0,6,8,4,1,5] 2024-08-19T12:38:40.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.838+0000 7f669b638640 10 monclient(hunting): picked mon.noname-i con 0x7f669406e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:40.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.838+0000 7f669b638640 10 monclient(hunting): picked mon.noname-b con 0x7f66940a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:40.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.838+0000 7f669b638640 10 monclient(hunting): picked mon.noname-g con 0x7f66940a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:40.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.838+0000 7f669b638640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6694136790 con 0x7f66940a5920 2024-08-19T12:38:40.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.838+0000 7f669b638640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f669409ddf0 con 0x7f66940a5d50 2024-08-19T12:38:40.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.838+0000 7f669b638640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f669409e0d0 con 0x7f669406e7b0 2024-08-19T12:38:40.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.838+0000 7f669b638640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:40.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.838+0000 7f669b638640 10 monclient(hunting): authenticate will time out at 714.925086s 2024-08-19T12:38:40.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.838+0000 7f6698bd3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f66940a5920 0x7f669413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57096/0 (socket says 172.21.15.26:57096) 2024-08-19T12:38:40.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.838+0000 7f6698bd3640 1 -- 172.21.15.26:0/1464901073 learned_addr learned my addr 172.21.15.26:0/1464901073 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:40.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6698bd3640 1 -- 172.21.15.26:0/1464901073 >> v1:172.21.15.26:6790/0 conn(0x7f66940a5920 legacy=0x7f669413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f66993d4640 1 -- 172.21.15.26:0/1464901073 >> v1:172.21.15.104:6792/0 conn(0x7f669406e7b0 legacy=0x7f6694136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6699bd5640 1 -- 172.21.15.26:0/1464901073 >> v1:172.21.15.104:6790/0 conn(0x7f66940a5d50 legacy=0x7f6694141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1076914140 0 0) 0x7f6694136790 con 0x7f66940a5920 2024-08-19T12:38:40.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient(hunting): my global_id is 4511 2024-08-19T12:38:40.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f667c003520 con 0x7f66940a5920 2024-08-19T12:38:40.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3140294059 0 0) 0x7f669409e0d0 con 0x7f669406e7b0 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient(hunting): my global_id is 4444 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6694136790 con 0x7f669406e7b0 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3348814729 0 0) 0x7f6684001820 con 0x7f66940a5d50 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient(hunting): my global_id is 4530 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f669409e0d0 con 0x7f66940a5d50 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2410858983 0 0) 0x7f667c003520 con 0x7f66940a5920 2024-08-19T12:38:40.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 >> v1:172.21.15.104:6792/0 conn(0x7f669406e7b0 legacy=0x7f6694136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 >> v1:172.21.15.104:6790/0 conn(0x7f66940a5d50 legacy=0x7f6694141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 monclient: found mon.noname-b 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f669409e3b0 con 0x7f66940a5920 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient: _finish_auth 0 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6688001ae0 con 0x7f66940a5920 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient: dump: 2024-08-19T12:38:40.851 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f669b638640 5 monclient: authenticate success, global_id 4511 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f669b638640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f66880048a0 con 0x7f66940a5920 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 1 -- 172.21.15.26:0/1464901073 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6688004e10 con 0x7f66940a5920 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f669b638640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f669b638640 4 set_mon_vals no callback set 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.840+0000 7f6693fff640 10 monclient: dump: 2024-08-19T12:38:40.853 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.855 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.855 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.841+0000 7f669b638640 10 monclient: shutdown 2024-08-19T12:38:40.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.841+0000 7f669b638640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:40.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.841+0000 7f669b638640 1 -- 172.21.15.26:0/1464901073 >> v1:172.21.15.26:6790/0 conn(0x7f66940a5920 legacy=0x7f669413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.841+0000 7f669b638640 1 -- 172.21.15.26:0/1464901073 shutdown_connections 2024-08-19T12:38:40.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.841+0000 7f669b638640 1 -- 172.21.15.26:0/1464901073 >> 172.21.15.26:0/1464901073 conn(0x7f66940a8740 msgr2=0x7f66940a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:40.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.841+0000 7f669b638640 1 -- 172.21.15.26:0/1464901073 shutdown_connections 2024-08-19T12:38:40.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.841+0000 7f669b638640 1 -- 172.21.15.26:0/1464901073 wait complete. 2024-08-19T12:38:40.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 10 monclient: build_initial_monmap 2024-08-19T12:38:40.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 10 monclient: monmap: 2024-08-19T12:38:40.855 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:40.857 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:40.857 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:40.857 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 1 librados: starting msgr at 2024-08-19T12:38:40.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 1 librados: starting objecter 2024-08-19T12:38:40.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 1 Processor -- start 2024-08-19T12:38:40.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 1 -- start start 2024-08-19T12:38:40.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 1 librados: setting wanted keys 2024-08-19T12:38:40.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 1 librados: calling monclient init 2024-08-19T12:38:40.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 10 monclient: init 2024-08-19T12:38:40.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:40.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 10 monclient: _add_conns ranks=[7,8,2,5,1,4,3,6,0] 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 10 monclient(hunting): picked mon.noname-i con 0x7f669406e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 10 monclient(hunting): picked mon.noname-e con 0x7f66940a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 10 monclient(hunting): picked mon.noname-b con 0x7f66940a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6694134e80 con 0x7f66940a5d50 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6694126e80 con 0x7f669406e7b0 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6694134a50 con 0x7f66940a5920 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f669b638640 10 monclient(hunting): authenticate will time out at 714.929878s 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f6699bd5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f66940a5d50 0x7f66941d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57100/0 (socket says 172.21.15.26:57100) 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.842+0000 7f6699bd5640 1 -- 172.21.15.26:0/1457707528 learned_addr learned my addr 172.21.15.26:0/1457707528 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6699bd5640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6790/0 conn(0x7f66940a5d50 legacy=0x7f66941d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 client.?.objecter ms_handle_connect 0x7f66940a5d50 2024-08-19T12:38:40.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:40.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 client.?.objecter ms_handle_connect 0x7f66940a5920 2024-08-19T12:38:40.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6698bd3640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6793/0 conn(0x7f66940a5920 legacy=0x7f66941285e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:40.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 client.?.objecter ms_handle_connect 0x7f669406e7b0 2024-08-19T12:38:40.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f66993d4640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.104:6792/0 conn(0x7f669406e7b0 legacy=0x7f6694132ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:40.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 836479700 0 0) 0x7f669409e0d0 con 0x7f66940a5d50 2024-08-19T12:38:40.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): my global_id is 4520 2024-08-19T12:38:40.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6680004eb0 con 0x7f66940a5d50 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4030428846 0 0) 0x7f6694126e80 con 0x7f669406e7b0 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): my global_id is 4453 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f669409e0d0 con 0x7f669406e7b0 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1911599299 0 0) 0x7f6694134a50 con 0x7f66940a5920 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): my global_id is 4553 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6694126e80 con 0x7f66940a5920 2024-08-19T12:38:40.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1612575406 0 0) 0x7f6680004eb0 con 0x7f66940a5d50 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6694134a50 con 0x7f66940a5d50 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6684003990 con 0x7f66940a5d50 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.862 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.862 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.862 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.862 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.862 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.862 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.862 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.862 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:40.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 606651496 0 0) 0x7f669409e0d0 con 0x7f669406e7b0 2024-08-19T12:38:40.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6680004eb0 con 0x7f669406e7b0 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f668c003610 con 0x7f669406e7b0 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.863 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2217392696 0 0) 0x7f6694126e80 con 0x7f66940a5920 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f669409e0d0 con 0x7f66940a5920 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6688004d90 con 0x7f66940a5920 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.864 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.865 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 88583073 0 0) 0x7f6694134a50 con 0x7f66940a5d50 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6793/0 conn(0x7f66940a5920 legacy=0x7f66941285e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.104:6792/0 conn(0x7f669406e7b0 legacy=0x7f6694132ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 monclient: found mon.b 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66941265e0 con 0x7f66940a5d50 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:40.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f669b638640 5 monclient: authenticate success, global_id 4520 2024-08-19T12:38:40.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f669b638640 10 monclient: _renew_subs 2024-08-19T12:38:40.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f669b638640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:40.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f669b638640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6694125680 con 0x7f66940a5d50 2024-08-19T12:38:40.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6684003e90 con 0x7f66940a5d50 2024-08-19T12:38:40.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:40.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6684004400 con 0x7f66940a5d50 2024-08-19T12:38:40.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:40.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f6691ffb640 10 monclient: dump: 2024-08-19T12:38:40.867 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.867 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.867 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.846+0000 7f669b638640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:40.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.846+0000 7f669b638640 10 monclient: _renew_subs 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.846+0000 7f669b638640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.846+0000 7f669b638640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f66941d79b0 con 0x7f66940a5d50 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.845+0000 7f66937fe640 4 set_mon_vals no callback set 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.846+0000 7f669b638640 1 librados: init done 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.846+0000 7f669b638640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.846+0000 7f669b638640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f669406bfa0 con 0x7f66940a1a10 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.846+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f66840045a0 con 0x7f66940a5d50 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.847+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f668404dcf0 con 0x7f66940a5d50 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.847+0000 7f6691ffb640 10 client.4520.objecter ms_dispatch 0x7f6694000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.847+0000 7f6691ffb640 3 client.4520.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.847+0000 7f6691ffb640 3 client.4520.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.847+0000 7f6691ffb640 20 client.4520.objecter dump_active .. 0 homeless 2024-08-19T12:38:40.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.847+0000 7f6698bd3640 10 monclient: get_auth_request con 0x7f6680041900 auth_method 0 2024-08-19T12:38:40.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 10 client.4520.objecter ms_handle_connect 0x7f66940a1a10 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 10 client.4520.objecter resend_mon_ops 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.848+0000 7f66993d4640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f66940a1a10 legacy=0x7f66940a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344002227 0 0) 0x7f6694136790 con 0x7f66940a1a10 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 10 monclient(hunting): my global_id is 4515 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6694134a50 con 0x7f66940a1a10 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2000545853 0 0) 0x7f6694134a50 con 0x7f66940a1a10 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6694136790 con 0x7f66940a1a10 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f668c0040d0 con 0x7f66940a1a10 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 10 monclient: dump: 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.874 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f66993d4640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f66940a1a10 legacy=0x7f66940a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f66993d4640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f66940a1a10 legacy=0x7f66940a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f66993d4640 1 --1- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f66940a1a10 0x7f66940a1df0 :-1 s=OPENED pgs=84 cs=1 l=1).handle_message read tag failed 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f66993d4640 1 --1- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f66940a1a10 0x7f66940a1df0 :-1 s=OPENED pgs=84 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f66940a1a10 legacy=0x7f66940a1df0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f66940a1a10 legacy=0x7f66940a1df0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6694134a50 con 0x7f6680049300 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.849+0000 7f6691ffb640 10 client.4520.objecter ms_handle_connect 0x7f6680041900 2024-08-19T12:38:40.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f66993d4640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f6680049300 legacy=0x7f6680047dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 10 client.4520.objecter ms_handle_connect 0x7f6680049300 2024-08-19T12:38:40.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 10 client.4520.objecter resend_mon_ops 2024-08-19T12:38:40.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1490781983 0 0) 0x7f6694134a50 con 0x7f6680049300 2024-08-19T12:38:40.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 10 monclient(hunting): my global_id is 4524 2024-08-19T12:38:40.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6680004eb0 con 0x7f6680049300 2024-08-19T12:38:40.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3327829987 0 0) 0x7f6680004eb0 con 0x7f6680049300 2024-08-19T12:38:40.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6694134a50 con 0x7f6680049300 2024-08-19T12:38:40.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f668c0043d0 con 0x7f6680049300 2024-08-19T12:38:40.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:40.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.851+0000 7f6691ffb640 10 monclient: dump: 2024-08-19T12:38:40.878 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.878 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.878 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.878 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.878 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.878 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.878 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.852+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2458911237 0 0) 0x7f6694134a50 con 0x7f6680049300 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.852+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6680048f40 con 0x7f6680049300 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.852+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f668c007bc0 con 0x7f6680049300 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.852+0000 7f6691ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:40.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.852+0000 7f6691ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.852+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f6680049300 legacy=0x7f6680047dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.852+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f6680049300 legacy=0x7f6680047dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.858+0000 7f669b638640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.858+0000 7f669b638640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f66940a28e0 con 0x7f66940a2500 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.860+0000 7f66993d4640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f66940a2500 legacy=0x7f66941dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.860+0000 7f6691ffb640 10 client.4520.objecter ms_handle_connect 0x7f66940a2500 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.860+0000 7f6691ffb640 10 client.4520.objecter resend_mon_ops 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 479598124 0 0) 0x7f66940a28e0 con 0x7f66940a2500 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 10 monclient(hunting): my global_id is 4533 2024-08-19T12:38:40.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6694134a50 con 0x7f66940a2500 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4028294213 0 0) 0x7f6694134a50 con 0x7f66940a2500 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66940a28e0 con 0x7f66940a2500 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f668c0062a0 con 0x7f66940a2500 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 10 monclient: dump: 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:40.881 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2533957931 0 0) 0x7f66940a28e0 con 0x7f66940a2500 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6694143140 con 0x7f66940a2500 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7710 (unknown 0 0 2294093493) 0x7f668c006570 con 0x7f66940a2500 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:40.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:40.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f66940a2500 legacy=0x7f66941dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6691ffb640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6792/0 conn(0x7f66940a2500 legacy=0x7f66941dbba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:40.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.861+0000 7f6699bd5640 1 -- 172.21.15.26:0/1457707528 reap_dead start 2024-08-19T12:38:40.883 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:40.883 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:40.883 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:40.883 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:40.883 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 4, 2024-08-19T12:38:40.883 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 33902, 2024-08-19T12:38:40.883 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:40.883 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 32, 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:40.884 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:40.885 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:40.886 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:40.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:40.886 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:40.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:40.886 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:40.886 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:40.886 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:40.886 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:40.886 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:40.886 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:40.886 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:40.887 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:40.888 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:40.890 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:40.890 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:40.890 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.890 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.890 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.890 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:40.891 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.892 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.892 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.892 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:40.892 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:40.892 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:40.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:40.893 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.894 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.895 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:40.897 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:40.898 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:40.898 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.898 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:40.898 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:40.898 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.900 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.901 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:40.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.903 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:40.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:40.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:40.903 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:40.904 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:40.904 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:40.904 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:40.904 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:40.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:40.904 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:40.904 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 10 librados: watch_flush enter 2024-08-19T12:38:40.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 10 librados: watch_flush exit 2024-08-19T12:38:40.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 20 client.4520.objecter shutdown clearing up homeless session... 2024-08-19T12:38:40.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 10 client.4520.objecter successfully canceled tick 2024-08-19T12:38:40.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.104:6800/899887506 conn(0x7f6680041900 legacy=0x7f6680043dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 10 monclient: shutdown 2024-08-19T12:38:40.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:40.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 1 -- 172.21.15.26:0/1457707528 >> v1:172.21.15.26:6790/0 conn(0x7f66940a5d50 legacy=0x7f66941d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:40.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 1 -- 172.21.15.26:0/1457707528 shutdown_connections 2024-08-19T12:38:40.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 1 -- 172.21.15.26:0/1457707528 >> 172.21.15.26:0/1457707528 conn(0x7f66940a8740 msgr2=0x7f66940aa870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:40.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 1 -- 172.21.15.26:0/1457707528 shutdown_connections 2024-08-19T12:38:40.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 1 -- 172.21.15.26:0/1457707528 wait complete. 2024-08-19T12:38:40.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:40.862+0000 7f669b638640 1 librados: shutdown 2024-08-19T12:38:40.907 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:38:40.907 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:38:40.907 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:38:41.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.008+0000 7f19927b1640 10 monclient: get_monmap_and_config 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.008+0000 7f19927b1640 10 monclient: build_initial_monmap 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.008+0000 7f19927b1640 10 monclient: monmap: 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:41.009016+0000 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:41.009016+0000 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:41.009 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:41.010 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:41.010 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:41.010 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:41.010 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:41.010 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.008+0000 7f19927b1640 1 Processor -- start 2024-08-19T12:38:41.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.008+0000 7f19927b1640 1 -- start start 2024-08-19T12:38:41.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.008+0000 7f19927b1640 10 monclient: init 2024-08-19T12:38:41.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.009+0000 7f19927b1640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:41.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.009+0000 7f19927b1640 10 monclient: _add_conns ranks=[3,5,7,6,0,4,8,1,2] 2024-08-19T12:38:41.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.009+0000 7f19927b1640 10 monclient(hunting): picked mon.noname-g con 0x7f198c073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:41.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.009+0000 7f19927b1640 10 monclient(hunting): picked mon.noname-h con 0x7f198c0a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:41.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.009+0000 7f19927b1640 10 monclient(hunting): picked mon.noname-i con 0x7f198c13f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:41.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.009+0000 7f19927b1640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198c09dae0 con 0x7f198c073470 2024-08-19T12:38:41.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.009+0000 7f19927b1640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198c136910 con 0x7f198c0a57a0 2024-08-19T12:38:41.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.009+0000 7f19927b1640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198c09dd50 con 0x7f198c13f2e0 2024-08-19T12:38:41.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.009+0000 7f19927b1640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:41.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.009+0000 7f19927b1640 10 monclient(hunting): authenticate will time out at 715.096510s 2024-08-19T12:38:41.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.009+0000 7f198bfff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f198c073470 0x7f198c136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:50330/0 (socket says 172.21.15.26:50330) 2024-08-19T12:38:41.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.009+0000 7f198bfff640 1 -- 172.21.15.26:0/3150235772 learned_addr learned my addr 172.21.15.26:0/3150235772 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:41.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.011+0000 7f198bfff640 1 -- 172.21.15.26:0/3150235772 >> v1:172.21.15.104:6790/0 conn(0x7f198c073470 legacy=0x7f198c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f1990d4e640 1 -- 172.21.15.26:0/3150235772 >> v1:172.21.15.104:6792/0 conn(0x7f198c13f2e0 legacy=0x7f198c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198b7fe640 1 -- 172.21.15.26:0/3150235772 >> v1:172.21.15.104:6791/0 conn(0x7f198c0a57a0 legacy=0x7f198c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346315640 0 0) 0x7f198c09dae0 con 0x7f198c073470 2024-08-19T12:38:41.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient(hunting): my global_id is 4539 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1974003520 con 0x7f198c073470 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1572319943 0 0) 0x7f198c09dd50 con 0x7f198c13f2e0 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient(hunting): my global_id is 4462 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f198c09dae0 con 0x7f198c13f2e0 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 765042753 0 0) 0x7f198c136910 con 0x7f198c0a57a0 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient(hunting): my global_id is 4406 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f198c09dd50 con 0x7f198c0a57a0 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1327972297 0 0) 0x7f198c09dae0 con 0x7f198c13f2e0 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 >> v1:172.21.15.104:6791/0 conn(0x7f198c0a57a0 legacy=0x7f198c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 >> v1:172.21.15.104:6790/0 conn(0x7f198c073470 legacy=0x7f198c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 monclient: found mon.noname-i 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198c09e030 con 0x7f198c13f2e0 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient: _finish_auth 0 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:41.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1984001e90 con 0x7f198c13f2e0 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f19927b1640 5 monclient: authenticate success, global_id 4462 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f19927b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient: dump: 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.019 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f19927b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1984003990 con 0x7f198c13f2e0 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:41.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f19927b1640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f19927b1640 4 set_mon_vals no callback set 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 1 -- 172.21.15.26:0/3150235772 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1984003f40 con 0x7f198c13f2e0 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.012+0000 7f198affd640 10 monclient: dump: 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.013+0000 7f19927b1640 10 monclient: shutdown 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.013+0000 7f19927b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.013+0000 7f19927b1640 1 -- 172.21.15.26:0/3150235772 >> v1:172.21.15.104:6792/0 conn(0x7f198c13f2e0 legacy=0x7f198c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.013+0000 7f19927b1640 1 -- 172.21.15.26:0/3150235772 shutdown_connections 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.013+0000 7f19927b1640 1 -- 172.21.15.26:0/3150235772 >> 172.21.15.26:0/3150235772 conn(0x7f198c0abff0 msgr2=0x7f198c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.013+0000 7f19927b1640 1 -- 172.21.15.26:0/3150235772 shutdown_connections 2024-08-19T12:38:41.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.013+0000 7f19927b1640 1 -- 172.21.15.26:0/3150235772 wait complete. 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.013+0000 7f19927b1640 10 monclient: build_initial_monmap 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.013+0000 7f19927b1640 10 monclient: monmap: 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:41.023 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.013+0000 7f19927b1640 1 librados: starting msgr at 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.013+0000 7f19927b1640 1 librados: starting objecter 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.013+0000 7f19927b1640 1 Processor -- start 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 1 -- start start 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 1 librados: setting wanted keys 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 1 librados: calling monclient init 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 10 monclient: init 2024-08-19T12:38:41.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 10 monclient: _add_conns ranks=[4,2,6,3,0,7,1,8,5] 2024-08-19T12:38:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 10 monclient(hunting): picked mon.noname-c con 0x7f198c073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 10 monclient(hunting): picked mon.noname-b con 0x7f198c0a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 10 monclient(hunting): picked mon.noname-d con 0x7f198c13f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198c131610 con 0x7f198c0a57a0 2024-08-19T12:38:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198c12f5b0 con 0x7f198c073470 2024-08-19T12:38:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198c0aa100 con 0x7f198c13f2e0 2024-08-19T12:38:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f19927b1640 10 monclient(hunting): authenticate will time out at 715.101634s 2024-08-19T12:38:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f198b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f198c0a57a0 0x7f198c12e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57108/0 (socket says 172.21.15.26:57108) 2024-08-19T12:38:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f198bfff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f198c073470 0x7f198c12dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:43352/0 (socket says 172.21.15.26:43352) 2024-08-19T12:38:41.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.014+0000 7f198b7fe640 1 -- 172.21.15.26:0/2535578381 learned_addr learned my addr 172.21.15.26:0/2535578381 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:41.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.016+0000 7f198bfff640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.26:6791/0 conn(0x7f198c073470 legacy=0x7f198c12dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.016+0000 7f198b7fe640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.26:6790/0 conn(0x7f198c0a57a0 legacy=0x7f198c12e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.016+0000 7f1988ff9640 10 client.?.objecter ms_handle_connect 0x7f198c073470 2024-08-19T12:38:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.016+0000 7f1988ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.016+0000 7f1990d4e640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.26:6792/0 conn(0x7f198c13f2e0 legacy=0x7f198c1d61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.016+0000 7f1988ff9640 10 client.?.objecter ms_handle_connect 0x7f198c0a57a0 2024-08-19T12:38:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.016+0000 7f1988ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.016+0000 7f1988ff9640 10 client.?.objecter ms_handle_connect 0x7f198c13f2e0 2024-08-19T12:38:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.016+0000 7f1988ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1393428051 0 0) 0x7f198c12f5b0 con 0x7f198c073470 2024-08-19T12:38:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:41.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): my global_id is 4495 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1978004eb0 con 0x7f198c073470 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2757826382 0 0) 0x7f198c0aa100 con 0x7f198c13f2e0 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): my global_id is 4542 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f198c12f5b0 con 0x7f198c13f2e0 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651637073 0 0) 0x7f198c131610 con 0x7f198c0a57a0 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): my global_id is 4529 2024-08-19T12:38:41.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f198c0aa100 con 0x7f198c0a57a0 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1630111227 0 0) 0x7f198c0aa100 con 0x7f198c0a57a0 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f198c131610 con 0x7f198c0a57a0 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2998310080 0 0) 0x7f1978004eb0 con 0x7f198c073470 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f198c0aa100 con 0x7f198c073470 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1980001cf0 con 0x7f198c0a57a0 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.034 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 94905609 0 0) 0x7f198c12f5b0 con 0x7f198c13f2e0 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1978004eb0 con 0x7f198c13f2e0 2024-08-19T12:38:41.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f197c0019e0 con 0x7f198c073470 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1984003e80 con 0x7f198c13f2e0 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:41.037 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2574925197 0 0) 0x7f198c131610 con 0x7f198c0a57a0 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.26:6792/0 conn(0x7f198c13f2e0 legacy=0x7f198c1d61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.26:6791/0 conn(0x7f198c073470 legacy=0x7f198c12dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 monclient: found mon.b 2024-08-19T12:38:41.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198c12eaf0 con 0x7f198c0a57a0 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient: _finish_auth 0 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f19927b1640 5 monclient: authenticate success, global_id 4529 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f19927b1640 10 monclient: _renew_subs 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f19927b1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f19927b1640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f198c131330 con 0x7f198c0a57a0 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f19927b1640 10 client.4529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f19927b1640 10 monclient: _renew_subs 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f19927b1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f19927b1640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f198c1302e0 con 0x7f198c0a57a0 2024-08-19T12:38:41.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f19927b1640 1 librados: init done 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f19800038c0 con 0x7f198c0a57a0 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1980003e30 con 0x7f198c0a57a0 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f198a7fc640 4 set_mon_vals no callback set 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.017+0000 7f1988ff9640 10 monclient: dump: 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:41.040 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.018+0000 7f19927b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.018+0000 7f19927b1640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f196c0065c0 con 0x7f196c002860 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.018+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f1980003fd0 con 0x7f198c0a57a0 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.018+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f198004d820 con 0x7f198c0a57a0 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.018+0000 7f1988ff9640 10 client.4529.objecter ms_dispatch 0x7f198c000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:41.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.018+0000 7f1988ff9640 3 client.4529.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.018+0000 7f1988ff9640 3 client.4529.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.019+0000 7f1988ff9640 20 client.4529.objecter dump_active .. 0 homeless 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.019+0000 7f1990d4e640 10 monclient: get_auth_request con 0x7f1978041a70 auth_method 0 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.020+0000 7f1988ff9640 10 client.4529.objecter ms_handle_connect 0x7f196c002860 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.020+0000 7f1988ff9640 10 client.4529.objecter resend_mon_ops 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.020+0000 7f198bfff640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.26:6793/0 conn(0x7f196c002860 legacy=0x7f196c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.020+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1867415504 0 0) 0x7f196c0065c0 con 0x7f196c002860 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.020+0000 7f1988ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.020+0000 7f1988ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.020+0000 7f1988ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.020+0000 7f1988ff9640 10 monclient(hunting): my global_id is 4562 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.020+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f198c131610 con 0x7f196c002860 2024-08-19T12:38:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1489688529 0 0) 0x7f198c131610 con 0x7f196c002860 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f196c0065c0 con 0x7f196c002860 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f197c005300 con 0x7f196c002860 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 10 monclient: dump: 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:41.043 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3672244725 0 0) 0x7f196c0065c0 con 0x7f196c002860 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f196c00a160 con 0x7f196c002860 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 10 client.4529.objecter ms_handle_connect 0x7f1978041a70 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f197c005430 con 0x7f196c002860 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.26:6793/0 conn(0x7f196c002860 legacy=0x7f196c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.021+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.26:6793/0 conn(0x7f196c002860 legacy=0x7f196c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.027+0000 7f19927b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.027+0000 7f19927b1640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f196c009e70 con 0x7f196c009130 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f198bfff640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.26:6793/0 conn(0x7f196c009130 legacy=0x7f196c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 10 client.4529.objecter ms_handle_connect 0x7f196c009130 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 10 client.4529.objecter resend_mon_ops 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2677050009 0 0) 0x7f196c009e70 con 0x7f196c009130 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 10 monclient(hunting): my global_id is 4571 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f196c0065c0 con 0x7f196c009130 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 450553825 0 0) 0x7f196c0065c0 con 0x7f196c009130 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f196c009e70 con 0x7f196c009130 2024-08-19T12:38:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f197c005c30 con 0x7f196c009130 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 10 monclient: dump: 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:41.046 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4135218817 0 0) 0x7f196c009e70 con 0x7f196c009130 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.030+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f196c00c3f0 con 0x7f196c009130 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 3814819674) 0x7f197c006040 con 0x7f196c009130 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f1988ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f1988ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.26:6793/0 conn(0x7f196c009130 legacy=0x7f196c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f1988ff9640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.26:6793/0 conn(0x7f196c009130 legacy=0x7f196c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:41.048 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:41.048 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:41.048 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:41.048 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:41.048 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 4, 2024-08-19T12:38:41.048 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 34047, 2024-08-19T12:38:41.048 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:41.048 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:41.048 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:41.048 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:41.048 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 32, 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:41.049 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:41.050 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:41.051 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:41.052 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:41.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:41.054 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:41.055 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.056 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:41.057 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:41.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:41.059 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:41.060 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.062 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:41.063 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:41.064 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:41.065 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:41.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:41.066 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:41.066 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:41.066 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:41.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:41.066 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:41.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:41.066 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:41.066 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 10 librados: watch_flush enter 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 10 librados: watch_flush exit 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 20 client.4529.objecter shutdown clearing up homeless session... 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 10 client.4529.objecter successfully canceled tick 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.104:6800/899887506 conn(0x7f1978041a70 legacy=0x7f1978043f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 10 monclient: shutdown 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 1 -- 172.21.15.26:0/2535578381 >> v1:172.21.15.26:6790/0 conn(0x7f198c0a57a0 legacy=0x7f198c12e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f1990d4e640 1 -- 172.21.15.26:0/2535578381 reap_dead start 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f1990d4e640 1 -- 172.21.15.26:0/2535578381 reap_dead start 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 1 -- 172.21.15.26:0/2535578381 shutdown_connections 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 1 -- 172.21.15.26:0/2535578381 >> 172.21.15.26:0/2535578381 conn(0x7f198c0abff0 msgr2=0x7f198c0a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:41.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 1 -- 172.21.15.26:0/2535578381 shutdown_connections 2024-08-19T12:38:41.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 1 -- 172.21.15.26:0/2535578381 wait complete. 2024-08-19T12:38:41.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.031+0000 7f19927b1640 1 librados: shutdown 2024-08-19T12:38:41.069 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:38:41.069 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:38:41.069 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:38:41.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.168+0000 7f580ccee640 10 monclient: get_monmap_and_config 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.169+0000 7f580ccee640 10 monclient: build_initial_monmap 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.169+0000 7f580ccee640 10 monclient: monmap: 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:41.169862+0000 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:41.169862+0000 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:41.170 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:41.171 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:41.171 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:41.171 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:41.171 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:41.171 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.169+0000 7f580ccee640 1 Processor -- start 2024-08-19T12:38:41.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.169+0000 7f580ccee640 1 -- start start 2024-08-19T12:38:41.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.169+0000 7f580ccee640 10 monclient: init 2024-08-19T12:38:41.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f580ccee640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:41.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f580ccee640 10 monclient: _add_conns ranks=[0,5,2,3,8,1,7,6,4] 2024-08-19T12:38:41.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f580ccee640 10 monclient(hunting): picked mon.noname-a con 0x7f580806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:41.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f580ccee640 10 monclient(hunting): picked mon.noname-h con 0x7f58080a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:41.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f580ccee640 10 monclient(hunting): picked mon.noname-b con 0x7f58080a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:41.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f580ccee640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5808136790 con 0x7f580806e7b0 2024-08-19T12:38:41.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f580ccee640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f580809ddf0 con 0x7f58080a5d50 2024-08-19T12:38:41.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f580ccee640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f580809e0d0 con 0x7f58080a5920 2024-08-19T12:38:41.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f580ccee640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:41.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f580ccee640 10 monclient(hunting): authenticate will time out at 715.257378s 2024-08-19T12:38:41.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f580659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f580806e7b0 0x7f5808136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51412/0 (socket says 172.21.15.26:51412) 2024-08-19T12:38:41.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f5806d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f58080a5d50 0x7f5808141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57120/0 (socket says 172.21.15.26:57120) 2024-08-19T12:38:41.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.170+0000 7f580659c640 1 -- 172.21.15.26:0/997738068 learned_addr learned my addr 172.21.15.26:0/997738068 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.172+0000 7f5806d9d640 1 -- 172.21.15.26:0/997738068 >> v1:172.21.15.26:6790/0 conn(0x7f58080a5d50 legacy=0x7f5808141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.172+0000 7f580659c640 1 -- 172.21.15.26:0/997738068 >> v1:172.21.15.26:6789/0 conn(0x7f580806e7b0 legacy=0x7f5808136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.172+0000 7f5805d9b640 1 -- 172.21.15.26:0/997738068 >> v1:172.21.15.104:6791/0 conn(0x7f58080a5920 legacy=0x7f580813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.172+0000 7f580559a640 1 -- 172.21.15.26:0/997738068 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2741149065 0 0) 0x7f5808136790 con 0x7f580806e7b0 2024-08-19T12:38:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.172+0000 7f580559a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.172+0000 7f580559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.172+0000 7f580559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.172+0000 7f580559a640 10 monclient(hunting): my global_id is 4446 2024-08-19T12:38:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.172+0000 7f580559a640 1 -- 172.21.15.26:0/997738068 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57f4003520 con 0x7f580806e7b0 2024-08-19T12:38:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 1 -- 172.21.15.26:0/997738068 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3085194720 0 0) 0x7f57f4003520 con 0x7f580806e7b0 2024-08-19T12:38:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 1 -- 172.21.15.26:0/997738068 >> v1:172.21.15.104:6791/0 conn(0x7f58080a5920 legacy=0x7f580813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 1 -- 172.21.15.26:0/997738068 >> v1:172.21.15.26:6790/0 conn(0x7f58080a5d50 legacy=0x7f5808141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 10 monclient: _finish_hunting 0 2024-08-19T12:38:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 1 monclient: found mon.noname-a 2024-08-19T12:38:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:38:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 1 -- 172.21.15.26:0/997738068 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580809e3b0 con 0x7f580806e7b0 2024-08-19T12:38:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 10 monclient: _finish_auth 0 2024-08-19T12:38:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 10 monclient: _check_auth_tickets 2024-08-19T12:38:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 1 -- 172.21.15.26:0/997738068 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f57f8001a10 con 0x7f580806e7b0 2024-08-19T12:38:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580ccee640 5 monclient: authenticate success, global_id 4446 2024-08-19T12:38:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580ccee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:41.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 10 monclient: dump: 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:41.179 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580ccee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 1 -- 172.21.15.26:0/997738068 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f57f8001d20 con 0x7f580806e7b0 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 1 -- 172.21.15.26:0/997738068 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f57f8004be0 con 0x7f580806e7b0 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580559a640 10 monclient: dump: 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.180 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580ccee640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:41.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580ccee640 4 set_mon_vals no callback set 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580ccee640 10 monclient: shutdown 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580ccee640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580ccee640 1 -- 172.21.15.26:0/997738068 >> v1:172.21.15.26:6789/0 conn(0x7f580806e7b0 legacy=0x7f5808136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580ccee640 1 -- 172.21.15.26:0/997738068 shutdown_connections 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580ccee640 1 -- 172.21.15.26:0/997738068 >> 172.21.15.26:0/997738068 conn(0x7f58080a8740 msgr2=0x7f58080a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580ccee640 1 -- 172.21.15.26:0/997738068 shutdown_connections 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.173+0000 7f580ccee640 1 -- 172.21.15.26:0/997738068 wait complete. 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.174+0000 7f580ccee640 10 monclient: build_initial_monmap 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.174+0000 7f580ccee640 10 monclient: monmap: 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:41.182 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.174+0000 7f580ccee640 1 librados: starting msgr at 2024-08-19T12:38:41.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.174+0000 7f580ccee640 1 librados: starting objecter 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.174+0000 7f580ccee640 1 Processor -- start 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 1 -- start start 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 1 librados: setting wanted keys 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 1 librados: calling monclient init 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 10 monclient: init 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 10 monclient: _add_conns ranks=[1,5,0,2,4,3,8,7,6] 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 10 monclient(hunting): picked mon.noname-f con 0x7f580806e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 10 monclient(hunting): picked mon.noname-h con 0x7f58080a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 10 monclient(hunting): picked mon.noname-a con 0x7f58080a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58081349d0 con 0x7f58080a5d50 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58080a2e00 con 0x7f580806e7b0 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58080a1bc0 con 0x7f58080a5920 2024-08-19T12:38:41.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:41.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f580ccee640 10 monclient(hunting): authenticate will time out at 715.262459s 2024-08-19T12:38:41.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f5806d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f58080a5d50 0x7f58081d69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51416/0 (socket says 172.21.15.26:51416) 2024-08-19T12:38:41.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.175+0000 7f5806d9d640 1 -- 172.21.15.26:0/479621046 learned_addr learned my addr 172.21.15.26:0/479621046 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:41.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f5806d9d640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.26:6789/0 conn(0x7f58080a5d50 legacy=0x7f58081d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f57feffd640 10 client.?.objecter ms_handle_connect 0x7f58080a5d50 2024-08-19T12:38:41.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f57feffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:41.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f57feffd640 10 client.?.objecter ms_handle_connect 0x7f580806e7b0 2024-08-19T12:38:41.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f57feffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:41.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f580659c640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.104:6789/0 conn(0x7f580806e7b0 legacy=0x7f58080a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 186465133 0 0) 0x7f58081349d0 con 0x7f58080a5d50 2024-08-19T12:38:41.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f57feffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:41.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f57feffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:41.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f57feffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f57feffd640 10 monclient(hunting): my global_id is 4455 2024-08-19T12:38:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57e8004eb0 con 0x7f58080a5d50 2024-08-19T12:38:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f57feffd640 10 client.?.objecter ms_handle_connect 0x7f58080a5920 2024-08-19T12:38:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f57feffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.177+0000 7f5805d9b640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.104:6791/0 conn(0x7f58080a5920 legacy=0x7f58081d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2511729060 0 0) 0x7f57e8004eb0 con 0x7f58080a5d50 2024-08-19T12:38:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58081349d0 con 0x7f58080a5d50 2024-08-19T12:38:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5800002080 con 0x7f58080a5d50 2024-08-19T12:38:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:38:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 10 monclient(hunting): dump: 2024-08-19T12:38:41.189 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:41.190 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.190 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.190 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.190 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:41.190 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.190 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:41.190 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:41.190 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:41.190 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:41.190 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:41.190 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:41.190 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:41.191 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:41.191 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:41.191 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:41.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2033720200 0 0) 0x7f58081349d0 con 0x7f58080a5d50 2024-08-19T12:38:41.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.104:6791/0 conn(0x7f58080a5920 legacy=0x7f58081d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.104:6789/0 conn(0x7f580806e7b0 legacy=0x7f58080a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:41.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:41.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 1 monclient: found mon.a 2024-08-19T12:38:41.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:41.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:41.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58080a2b20 con 0x7f58080a5d50 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f580ccee640 5 monclient: authenticate success, global_id 4455 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f580ccee640 10 monclient: _renew_subs 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f580ccee640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f580ccee640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f58080a2810 con 0x7f58080a5d50 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f580ccee640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f580ccee640 10 monclient: _renew_subs 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f580ccee640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f580ccee640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f58080a51e0 con 0x7f58080a5d50 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f580ccee640 1 librados: init done 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5800004f80 con 0x7f58080a5d50 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:41.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5800005470 con 0x7f58080a5d50 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f5804d99640 4 set_mon_vals no callback set 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.178+0000 7f57feffd640 10 monclient: dump: 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:41.193 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.179+0000 7f580ccee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.179+0000 7f580ccee640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f57ec0065c0 con 0x7f57ec002860 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.179+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5800005610 con 0x7f58080a5d50 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.179+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f580004dd10 con 0x7f58080a5d50 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.179+0000 7f57feffd640 10 client.4455.objecter ms_dispatch 0x7f5808000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.179+0000 7f57feffd640 3 client.4455.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:41.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.179+0000 7f57feffd640 3 client.4455.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:41.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.179+0000 7f57feffd640 20 client.4455.objecter dump_active .. 0 homeless 2024-08-19T12:38:41.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.179+0000 7f5805d9b640 10 monclient: get_auth_request con 0x7f57e8040f80 auth_method 0 2024-08-19T12:38:41.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.181+0000 7f580659c640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.104:6789/0 conn(0x7f57ec002860 legacy=0x7f57ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:41.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.181+0000 7f57feffd640 10 client.4455.objecter ms_handle_connect 0x7f57ec002860 2024-08-19T12:38:41.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.181+0000 7f57feffd640 10 client.4455.objecter resend_mon_ops 2024-08-19T12:38:41.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.181+0000 7f57feffd640 10 client.4455.objecter ms_handle_connect 0x7f57e8040f80 2024-08-19T12:38:41.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.502+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 760387184 0 0) 0x7f57ec0065c0 con 0x7f57ec002860 2024-08-19T12:38:41.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.502+0000 7f57feffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:41.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.502+0000 7f57feffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:41.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.502+0000 7f57feffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:41.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.502+0000 7f57feffd640 10 monclient(hunting): my global_id is 4429 2024-08-19T12:38:41.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.502+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58081349d0 con 0x7f57ec002860 2024-08-19T12:38:42.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.999+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3803372593 0 0) 0x7f58081349d0 con 0x7f57ec002860 2024-08-19T12:38:42.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.999+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57ec0065c0 con 0x7f57ec002860 2024-08-19T12:38:42.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.999+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f57f8004b20 con 0x7f57ec002860 2024-08-19T12:38:42.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.999+0000 7f57feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.999+0000 7f57feffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:42.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:41.999+0000 7f57feffd640 10 monclient: dump: 2024-08-19T12:38:42.001 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.001 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.001 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.001 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.001 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.000+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3318307730 0 0) 0x7f57ec0065c0 con 0x7f57ec002860 2024-08-19T12:38:42.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.000+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f57ec00a160 con 0x7f57ec002860 2024-08-19T12:38:42.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.000+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f57f80014b0 con 0x7f57ec002860 2024-08-19T12:38:42.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.001+0000 7f57feffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:42.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.001+0000 7f57feffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:42.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.001+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.104:6789/0 conn(0x7f57ec002860 legacy=0x7f57ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.001+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.104:6789/0 conn(0x7f57ec002860 legacy=0x7f57ec004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:42.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.008+0000 7f580ccee640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:42.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.008+0000 7f580ccee640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f57ec009e70 con 0x7f57ec009130 2024-08-19T12:38:42.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.010+0000 7f57feffd640 10 client.4455.objecter ms_handle_connect 0x7f57ec009130 2024-08-19T12:38:42.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.010+0000 7f57feffd640 10 client.4455.objecter resend_mon_ops 2024-08-19T12:38:42.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.010+0000 7f580659c640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.104:6789/0 conn(0x7f57ec009130 legacy=0x7f57ec0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.124+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 993023392 0 0) 0x7f57ec009e70 con 0x7f57ec009130 2024-08-19T12:38:42.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.124+0000 7f57feffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.124+0000 7f57feffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.124+0000 7f57feffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.124+0000 7f57feffd640 10 monclient(hunting): my global_id is 4438 2024-08-19T12:38:42.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.125+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57ec0065c0 con 0x7f57ec009130 2024-08-19T12:38:42.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.125+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1772065611 0 0) 0x7f57ec0065c0 con 0x7f57ec009130 2024-08-19T12:38:42.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.125+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57ec009e70 con 0x7f57ec009130 2024-08-19T12:38:42.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.125+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f57f8001950 con 0x7f57ec009130 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.125+0000 7f57feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.125+0000 7f57feffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.125+0000 7f57feffd640 10 monclient: dump: 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.128 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.129 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.125+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2915251546 0 0) 0x7f57ec009e70 con 0x7f57ec009130 2024-08-19T12:38:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.125+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f57ec00c3f0 con 0x7f57ec009130 2024-08-19T12:38:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.126+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7710 (unknown 0 0 2982864045) 0x7f57f80053b0 con 0x7f57ec009130 2024-08-19T12:38:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.126+0000 7f57feffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.126+0000 7f57feffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.126+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.104:6789/0 conn(0x7f57ec009130 legacy=0x7f57ec0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.126+0000 7f57feffd640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.104:6789/0 conn(0x7f57ec009130 legacy=0x7f57ec0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:42.129 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 4, 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 35129, 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:38:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 33, 2024-08-19T12:38:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:42.150 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:42.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.126+0000 7f580ccee640 10 librados: watch_flush enter 2024-08-19T12:38:42.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f580ccee640 10 librados: watch_flush exit 2024-08-19T12:38:42.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f580ccee640 20 client.4455.objecter shutdown clearing up homeless session... 2024-08-19T12:38:42.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f580ccee640 10 client.4455.objecter successfully canceled tick 2024-08-19T12:38:42.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f580ccee640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.104:6800/899887506 conn(0x7f57e8040f80 legacy=0x7f57e8043440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f580ccee640 10 monclient: shutdown 2024-08-19T12:38:42.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f580ccee640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:42.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f580ccee640 1 -- 172.21.15.26:0/479621046 >> v1:172.21.15.26:6789/0 conn(0x7f58080a5d50 legacy=0x7f58081d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f5806d9d640 1 -- 172.21.15.26:0/479621046 reap_dead start 2024-08-19T12:38:42.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f5806d9d640 1 -- 172.21.15.26:0/479621046 reap_dead start 2024-08-19T12:38:42.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f580ccee640 1 -- 172.21.15.26:0/479621046 shutdown_connections 2024-08-19T12:38:42.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f580ccee640 1 -- 172.21.15.26:0/479621046 >> 172.21.15.26:0/479621046 conn(0x7f58080a8740 msgr2=0x7f58080aa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:42.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f580ccee640 1 -- 172.21.15.26:0/479621046 shutdown_connections 2024-08-19T12:38:42.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f580ccee640 1 -- 172.21.15.26:0/479621046 wait complete. 2024-08-19T12:38:42.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.127+0000 7f580ccee640 1 librados: shutdown 2024-08-19T12:38:42.153 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:38:42.153 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:38:42.153 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:38:42.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.258+0000 7f3cd8c84640 10 monclient: get_monmap_and_config 2024-08-19T12:38:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.258+0000 7f3cd8c84640 10 monclient: build_initial_monmap 2024-08-19T12:38:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.258+0000 7f3cd8c84640 10 monclient: monmap: 2024-08-19T12:38:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:42.259745+0000 2024-08-19T12:38:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:42.259745+0000 2024-08-19T12:38:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:42.261 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:42.261 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.259+0000 7f3cd8c84640 1 Processor -- start 2024-08-19T12:38:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.259+0000 7f3cd8c84640 1 -- start start 2024-08-19T12:38:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.259+0000 7f3cd8c84640 10 monclient: init 2024-08-19T12:38:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.260+0000 7f3cd8c84640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.260+0000 7f3cd8c84640 10 monclient: _add_conns ranks=[2,1,5,6,3,7,8,0,4] 2024-08-19T12:38:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.260+0000 7f3cd8c84640 10 monclient(hunting): picked mon.noname-b con 0x7f3cd406e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:42.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.260+0000 7f3cd8c84640 10 monclient(hunting): picked mon.noname-f con 0x7f3cd40a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.260+0000 7f3cd8c84640 10 monclient(hunting): picked mon.noname-h con 0x7f3cd40a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.260+0000 7f3cd8c84640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd4136790 con 0x7f3cd40a5920 2024-08-19T12:38:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.260+0000 7f3cd8c84640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd409ddf0 con 0x7f3cd406e7b0 2024-08-19T12:38:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.260+0000 7f3cd8c84640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd409e0d0 con 0x7f3cd40a5d50 2024-08-19T12:38:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.260+0000 7f3cd8c84640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.260+0000 7f3cd8c84640 10 monclient(hunting): authenticate will time out at 716.347212s 2024-08-19T12:38:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.260+0000 7f3cd259c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3cd406e7b0 0x7f3cd4136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57130/0 (socket says 172.21.15.26:57130) 2024-08-19T12:38:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.260+0000 7f3cd259c640 1 -- 172.21.15.26:0/1821314668 learned_addr learned my addr 172.21.15.26:0/1821314668 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd259c640 1 -- 172.21.15.26:0/1821314668 >> v1:172.21.15.26:6790/0 conn(0x7f3cd406e7b0 legacy=0x7f3cd4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 1 -- 172.21.15.26:0/1821314668 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1794342409 0 0) 0x7f3cc4001450 con 0x7f3cd406e7b0 2024-08-19T12:38:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient(hunting): my global_id is 4538 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 1 -- 172.21.15.26:0/1821314668 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cc0003520 con 0x7f3cd406e7b0 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cc9d9b640 1 -- 172.21.15.26:0/1821314668 >> v1:172.21.15.104:6789/0 conn(0x7f3cd40a5920 legacy=0x7f3cd413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd2d9d640 1 -- 172.21.15.26:0/1821314668 >> v1:172.21.15.104:6791/0 conn(0x7f3cd40a5d50 legacy=0x7f3cd4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 1 -- 172.21.15.26:0/1821314668 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 943191427 0 0) 0x7f3cd4136790 con 0x7f3cd40a5920 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient(hunting): my global_id is 4447 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 1 -- 172.21.15.26:0/1821314668 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cc4001450 con 0x7f3cd40a5920 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 1 -- 172.21.15.26:0/1821314668 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 926784516 0 0) 0x7f3cc0003520 con 0x7f3cd406e7b0 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 1 -- 172.21.15.26:0/1821314668 >> v1:172.21.15.104:6791/0 conn(0x7f3cd40a5d50 legacy=0x7f3cd4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 1 -- 172.21.15.26:0/1821314668 >> v1:172.21.15.104:6789/0 conn(0x7f3cd40a5920 legacy=0x7f3cd413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 1 monclient: found mon.noname-b 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 1 -- 172.21.15.26:0/1821314668 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd409e3b0 con 0x7f3cd406e7b0 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient: _finish_auth 0 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 1 -- 172.21.15.26:0/1821314668 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3cc4001cb0 con 0x7f3cd406e7b0 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd8c84640 5 monclient: authenticate success, global_id 4538 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd8c84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:42.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd1d9b640 10 monclient: dump: 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.269 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.262+0000 7f3cd8c84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd1d9b640 1 -- 172.21.15.26:0/1821314668 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3cc4004950 con 0x7f3cd406e7b0 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd1d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd1d9b640 1 -- 172.21.15.26:0/1821314668 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3cc4004f00 con 0x7f3cd406e7b0 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd8c84640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd8c84640 4 set_mon_vals no callback set 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd1d9b640 10 monclient: dump: 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.270 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.271 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd8c84640 10 monclient: shutdown 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd8c84640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd8c84640 1 -- 172.21.15.26:0/1821314668 >> v1:172.21.15.26:6790/0 conn(0x7f3cd406e7b0 legacy=0x7f3cd4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd8c84640 1 -- 172.21.15.26:0/1821314668 shutdown_connections 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd8c84640 1 -- 172.21.15.26:0/1821314668 >> 172.21.15.26:0/1821314668 conn(0x7f3cd40a8740 msgr2=0x7f3cd40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd8c84640 1 -- 172.21.15.26:0/1821314668 shutdown_connections 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.263+0000 7f3cd8c84640 1 -- 172.21.15.26:0/1821314668 wait complete. 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.264+0000 7f3cd8c84640 10 monclient: build_initial_monmap 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.264+0000 7f3cd8c84640 10 monclient: monmap: 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:42.272 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.264+0000 7f3cd8c84640 1 librados: starting msgr at 2024-08-19T12:38:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.264+0000 7f3cd8c84640 1 librados: starting objecter 2024-08-19T12:38:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.264+0000 7f3cd8c84640 1 Processor -- start 2024-08-19T12:38:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.264+0000 7f3cd8c84640 1 -- start start 2024-08-19T12:38:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.264+0000 7f3cd8c84640 1 librados: setting wanted keys 2024-08-19T12:38:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.264+0000 7f3cd8c84640 1 librados: calling monclient init 2024-08-19T12:38:42.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.264+0000 7f3cd8c84640 10 monclient: init 2024-08-19T12:38:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.265+0000 7f3cd8c84640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.265+0000 7f3cd8c84640 10 monclient: _add_conns ranks=[3,7,2,5,4,6,1,8,0] 2024-08-19T12:38:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.265+0000 7f3cd8c84640 10 monclient(hunting): picked mon.noname-g con 0x7f3cd406e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.265+0000 7f3cd8c84640 10 monclient(hunting): picked mon.noname-i con 0x7f3cd40a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.265+0000 7f3cd8c84640 10 monclient(hunting): picked mon.noname-b con 0x7f3cd40a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.265+0000 7f3cd8c84640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd40a0c90 con 0x7f3cd40a5d50 2024-08-19T12:38:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.265+0000 7f3cd8c84640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd409e930 con 0x7f3cd406e7b0 2024-08-19T12:38:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.265+0000 7f3cd8c84640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd40a03f0 con 0x7f3cd40a5920 2024-08-19T12:38:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.265+0000 7f3cd8c84640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:42.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.265+0000 7f3cd8c84640 10 monclient(hunting): authenticate will time out at 716.352259s 2024-08-19T12:38:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.265+0000 7f3cd2d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3cd40a5d50 0x7f3cd41d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57134/0 (socket says 172.21.15.26:57134) 2024-08-19T12:38:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.265+0000 7f3cd2d9d640 1 -- 172.21.15.26:0/2193588691 learned_addr learned my addr 172.21.15.26:0/2193588691 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3cd2d9d640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.26:6790/0 conn(0x7f3cd40a5d50 legacy=0x7f3cd41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 client.?.objecter ms_handle_connect 0x7f3cd40a5d50 2024-08-19T12:38:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649514553 0 0) 0x7f3cd40a0c90 con 0x7f3cd40a5d50 2024-08-19T12:38:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3cc9d9b640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.104:6792/0 conn(0x7f3cd40a5920 legacy=0x7f3cd4132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): my global_id is 4547 2024-08-19T12:38:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3cd259c640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.104:6790/0 conn(0x7f3cd406e7b0 legacy=0x7f3cd4131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cb4004eb0 con 0x7f3cd40a5d50 2024-08-19T12:38:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 client.?.objecter ms_handle_connect 0x7f3cd40a5920 2024-08-19T12:38:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 client.?.objecter ms_handle_connect 0x7f3cd406e7b0 2024-08-19T12:38:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3751808736 0 0) 0x7f3cb4004eb0 con 0x7f3cd40a5d50 2024-08-19T12:38:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cd40a0c90 con 0x7f3cd40a5d50 2024-08-19T12:38:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3ccc0020a0 con 0x7f3cd40a5d50 2024-08-19T12:38:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:42.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): dump: 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.283 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.284 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 264527498 0 0) 0x7f3cd409e930 con 0x7f3cd406e7b0 2024-08-19T12:38:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): my global_id is 4548 2024-08-19T12:38:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cb4004eb0 con 0x7f3cd406e7b0 2024-08-19T12:38:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2807859710 0 0) 0x7f3cd40a03f0 con 0x7f3cd40a5920 2024-08-19T12:38:42.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 10 monclient(hunting): my global_id is 4471 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.267+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cd409e930 con 0x7f3cd40a5920 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3399141174 0 0) 0x7f3cd40a0c90 con 0x7f3cd40a5d50 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.104:6792/0 conn(0x7f3cd40a5920 legacy=0x7f3cd4132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.104:6790/0 conn(0x7f3cd406e7b0 legacy=0x7f3cd4131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 1 monclient: found mon.b 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd4132b70 con 0x7f3cd40a5d50 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:42.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3cd8c84640 5 monclient: authenticate success, global_id 4547 2024-08-19T12:38:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3cd8c84640 10 monclient: _renew_subs 2024-08-19T12:38:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3cd8c84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3cd8c84640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd40a5340 con 0x7f3cd40a5d50 2024-08-19T12:38:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3cd8c84640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3cd8c84640 10 monclient: _renew_subs 2024-08-19T12:38:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3cd8c84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3cd8c84640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3cd409f490 con 0x7f3cd40a5d50 2024-08-19T12:38:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3cd8c84640 1 librados: init done 2024-08-19T12:38:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ccc003e10 con 0x7f3cd40a5d50 2024-08-19T12:38:42.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3ccc004380 con 0x7f3cd40a5d50 2024-08-19T12:38:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3cd159a640 4 set_mon_vals no callback set 2024-08-19T12:38:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.268+0000 7f3ccb7fe640 10 monclient: dump: 2024-08-19T12:38:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.287 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.269+0000 7f3cd8c84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.269+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3ccc0045a0 con 0x7f3cd40a5d50 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.269+0000 7f3cd8c84640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3cb80065c0 con 0x7f3cb8002860 2024-08-19T12:38:42.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.269+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f3ccc04dde0 con 0x7f3cd40a5d50 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.269+0000 7f3ccb7fe640 10 client.4547.objecter ms_dispatch 0x7f3cd4000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.269+0000 7f3ccb7fe640 3 client.4547.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.269+0000 7f3ccb7fe640 3 client.4547.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.269+0000 7f3ccb7fe640 20 client.4547.objecter dump_active .. 0 homeless 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.269+0000 7f3cc9d9b640 10 monclient: get_auth_request con 0x7f3cb4041170 auth_method 0 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3cd259c640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.104:6790/0 conn(0x7f3cb8002860 legacy=0x7f3cb8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 10 client.4547.objecter ms_handle_connect 0x7f3cb8002860 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 10 client.4547.objecter resend_mon_ops 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3678460948 0 0) 0x7f3cb80065c0 con 0x7f3cb8002860 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 10 monclient(hunting): my global_id is 4557 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cd40a0c90 con 0x7f3cb8002860 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4121169252 0 0) 0x7f3cd40a0c90 con 0x7f3cb8002860 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cb80065c0 con 0x7f3cb8002860 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 10 client.4547.objecter ms_handle_connect 0x7f3cb4041170 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3cc4001ce0 con 0x7f3cb8002860 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.271+0000 7f3ccb7fe640 10 monclient: dump: 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.290 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.291 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.272+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1543742018 0 0) 0x7f3cb80065c0 con 0x7f3cb8002860 2024-08-19T12:38:42.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.272+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3cb800a160 con 0x7f3cb8002860 2024-08-19T12:38:42.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.272+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3cc4004bb0 con 0x7f3cb8002860 2024-08-19T12:38:42.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.272+0000 7f3ccb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:42.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.272+0000 7f3ccb7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:42.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.272+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.104:6790/0 conn(0x7f3cb8002860 legacy=0x7f3cb8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.272+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.104:6790/0 conn(0x7f3cb8002860 legacy=0x7f3cb8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:42.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.278+0000 7f3cd8c84640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:42.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.278+0000 7f3cd8c84640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3cb8009e50 con 0x7f3cb8009110 2024-08-19T12:38:42.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3cd259c640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.104:6790/0 conn(0x7f3cb8009110 legacy=0x7f3cb80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 10 client.4547.objecter ms_handle_connect 0x7f3cb8009110 2024-08-19T12:38:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 10 client.4547.objecter resend_mon_ops 2024-08-19T12:38:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2400115439 0 0) 0x7f3cb8009e50 con 0x7f3cb8009110 2024-08-19T12:38:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 10 monclient(hunting): my global_id is 4566 2024-08-19T12:38:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cb80065c0 con 0x7f3cb8009110 2024-08-19T12:38:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1762259143 0 0) 0x7f3cb80065c0 con 0x7f3cb8009110 2024-08-19T12:38:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cb8009e50 con 0x7f3cb8009110 2024-08-19T12:38:42.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3cc40051c0 con 0x7f3cb8009110 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 10 monclient: dump: 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.295 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2741453867 0 0) 0x7f3cb8009e50 con 0x7f3cb8009110 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.281+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3cb800c3a0 con 0x7f3cb8009110 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7710 (unknown 0 0 885142445) 0x7f3cc4008670 con 0x7f3cb8009110 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3ccb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3ccb7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.104:6790/0 conn(0x7f3cb8009110 legacy=0x7f3cb80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3ccb7fe640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.104:6790/0 conn(0x7f3cb8009110 legacy=0x7f3cb80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:42.297 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 4, 2024-08-19T12:38:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 35294, 2024-08-19T12:38:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:38:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 34, 2024-08-19T12:38:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:42.318 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:42.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 10 librados: watch_flush enter 2024-08-19T12:38:42.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 10 librados: watch_flush exit 2024-08-19T12:38:42.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 20 client.4547.objecter shutdown clearing up homeless session... 2024-08-19T12:38:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 10 client.4547.objecter successfully canceled tick 2024-08-19T12:38:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.104:6800/899887506 conn(0x7f3cb4041170 legacy=0x7f3cb4043630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 10 monclient: shutdown 2024-08-19T12:38:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 1 -- 172.21.15.26:0/2193588691 >> v1:172.21.15.26:6790/0 conn(0x7f3cd40a5d50 legacy=0x7f3cd41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd2d9d640 1 -- 172.21.15.26:0/2193588691 reap_dead start 2024-08-19T12:38:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd2d9d640 1 -- 172.21.15.26:0/2193588691 reap_dead start 2024-08-19T12:38:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 1 -- 172.21.15.26:0/2193588691 shutdown_connections 2024-08-19T12:38:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 1 -- 172.21.15.26:0/2193588691 >> 172.21.15.26:0/2193588691 conn(0x7f3cd40a8740 msgr2=0x7f3cd406d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 1 -- 172.21.15.26:0/2193588691 shutdown_connections 2024-08-19T12:38:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 1 -- 172.21.15.26:0/2193588691 wait complete. 2024-08-19T12:38:42.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.282+0000 7f3cd8c84640 1 librados: shutdown 2024-08-19T12:38:42.322 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:38:42.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:38:42.322 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:38:42.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.421+0000 7f5d1e0b3640 10 monclient: get_monmap_and_config 2024-08-19T12:38:42.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.421+0000 7f5d1e0b3640 10 monclient: build_initial_monmap 2024-08-19T12:38:42.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.421+0000 7f5d1e0b3640 10 monclient: monmap: 2024-08-19T12:38:42.422 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:42.422 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.422 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:42.422070+0000 2024-08-19T12:38:42.422 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:42.422070+0000 2024-08-19T12:38:42.422 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:42.423 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.423 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:42.423 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:42.423 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:42.423 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:42.423 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:42.423 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:42.423 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:42.423 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:42.423 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:42.423 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.421+0000 7f5d1e0b3640 1 Processor -- start 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.421+0000 7f5d1e0b3640 1 -- start start 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d1e0b3640 10 monclient: init 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d1e0b3640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d1e0b3640 10 monclient: _add_conns ranks=[0,5,6,8,7,4,1,2,3] 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d1e0b3640 10 monclient(hunting): picked mon.noname-a con 0x7f5d1806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d1e0b3640 10 monclient(hunting): picked mon.noname-h con 0x7f5d180a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d1e0b3640 10 monclient(hunting): picked mon.noname-d con 0x7f5d180a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d1e0b3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d18136790 con 0x7f5d1806e7b0 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d1e0b3640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1809ddf0 con 0x7f5d180a5920 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d1e0b3640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1809e0d0 con 0x7f5d180a5d50 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d1e0b3640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d1e0b3640 10 monclient(hunting): authenticate will time out at 716.509501s 2024-08-19T12:38:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d177fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5d1806e7b0 0x7f5d18136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51420/0 (socket says 172.21.15.26:51420) 2024-08-19T12:38:42.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d17fff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f5d180a5d50 0x7f5d18141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:40942/0 (socket says 172.21.15.26:40942) 2024-08-19T12:38:42.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.422+0000 7f5d177fe640 1 -- 172.21.15.26:0/3529305947 learned_addr learned my addr 172.21.15.26:0/3529305947 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:42.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.424+0000 7f5d177fe640 1 -- 172.21.15.26:0/3529305947 >> v1:172.21.15.26:6789/0 conn(0x7f5d1806e7b0 legacy=0x7f5d18136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.424+0000 7f5d17fff640 1 -- 172.21.15.26:0/3529305947 >> v1:172.21.15.26:6792/0 conn(0x7f5d180a5d50 legacy=0x7f5d18141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.424+0000 7f5d167fc640 1 -- 172.21.15.26:0/3529305947 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3734895216 0 0) 0x7f5d18136790 con 0x7f5d1806e7b0 2024-08-19T12:38:42.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.424+0000 7f5d167fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.424+0000 7f5d167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.424+0000 7f5d167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.424+0000 7f5d167fc640 10 monclient(hunting): my global_id is 4464 2024-08-19T12:38:42.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 1 -- 172.21.15.26:0/3529305947 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d00003520 con 0x7f5d1806e7b0 2024-08-19T12:38:42.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 1 -- 172.21.15.26:0/3529305947 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4094496694 0 0) 0x7f5d1809e0d0 con 0x7f5d180a5d50 2024-08-19T12:38:42.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient(hunting): my global_id is 4551 2024-08-19T12:38:42.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 1 -- 172.21.15.26:0/3529305947 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d18136790 con 0x7f5d180a5d50 2024-08-19T12:38:42.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d16ffd640 1 -- 172.21.15.26:0/3529305947 >> v1:172.21.15.104:6791/0 conn(0x7f5d180a5920 legacy=0x7f5d1813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 1 -- 172.21.15.26:0/3529305947 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3474525561 0 0) 0x7f5d18136790 con 0x7f5d180a5d50 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 1 -- 172.21.15.26:0/3529305947 >> v1:172.21.15.104:6791/0 conn(0x7f5d180a5920 legacy=0x7f5d1813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 1 -- 172.21.15.26:0/3529305947 >> v1:172.21.15.26:6789/0 conn(0x7f5d1806e7b0 legacy=0x7f5d18136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 1 monclient: found mon.noname-d 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 1 -- 172.21.15.26:0/3529305947 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1809e3b0 con 0x7f5d180a5d50 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 1 -- 172.21.15.26:0/3529305947 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5d10001e40 con 0x7f5d180a5d50 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d1e0b3640 5 monclient: authenticate success, global_id 4551 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d1e0b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:42.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient: dump: 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.432 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d1e0b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 1 -- 172.21.15.26:0/3529305947 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d10004b60 con 0x7f5d180a5d50 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 1 -- 172.21.15.26:0/3529305947 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5d100050d0 con 0x7f5d180a5d50 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d167fc640 10 monclient: dump: 2024-08-19T12:38:42.433 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.434 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d1e0b3640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d1e0b3640 4 set_mon_vals no callback set 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d1e0b3640 10 monclient: shutdown 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d1e0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/3529305947 >> v1:172.21.15.26:6792/0 conn(0x7f5d180a5d50 legacy=0x7f5d18141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/3529305947 shutdown_connections 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.425+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/3529305947 >> 172.21.15.26:0/3529305947 conn(0x7f5d180a8740 msgr2=0x7f5d180a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.426+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/3529305947 shutdown_connections 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.426+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/3529305947 wait complete. 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.426+0000 7f5d1e0b3640 10 monclient: build_initial_monmap 2024-08-19T12:38:42.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.426+0000 7f5d1e0b3640 10 monclient: monmap: 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:42.436 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:42.437 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:42.437 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:42.437 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.426+0000 7f5d1e0b3640 1 librados: starting msgr at 2024-08-19T12:38:42.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.426+0000 7f5d1e0b3640 1 librados: starting objecter 2024-08-19T12:38:42.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.426+0000 7f5d1e0b3640 1 Processor -- start 2024-08-19T12:38:42.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 1 -- start start 2024-08-19T12:38:42.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 1 librados: setting wanted keys 2024-08-19T12:38:42.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 1 librados: calling monclient init 2024-08-19T12:38:42.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 10 monclient: init 2024-08-19T12:38:42.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:42.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 10 monclient: _add_conns ranks=[8,2,3,5,4,1,6,0,7] 2024-08-19T12:38:42.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 10 monclient(hunting): picked mon.noname-e con 0x7f5d1806e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:42.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 10 monclient(hunting): picked mon.noname-b con 0x7f5d180a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:42.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 10 monclient(hunting): picked mon.noname-g con 0x7f5d180a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:42.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d181cf060 con 0x7f5d180a5920 2024-08-19T12:38:42.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d18142ec0 con 0x7f5d180a5d50 2024-08-19T12:38:42.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1813b2e0 con 0x7f5d1806e7b0 2024-08-19T12:38:42.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:42.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d1e0b3640 10 monclient(hunting): authenticate will time out at 716.514551s 2024-08-19T12:38:42.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d177fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f5d1806e7b0 0x7f5d181ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40156/0 (socket says 172.21.15.26:40156) 2024-08-19T12:38:42.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.427+0000 7f5d177fe640 1 -- 172.21.15.26:0/2020708884 learned_addr learned my addr 172.21.15.26:0/2020708884 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:42.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5d177fe640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.26:6793/0 conn(0x7f5d1806e7b0 legacy=0x7f5d181ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 10 client.?.objecter ms_handle_connect 0x7f5d1806e7b0 2024-08-19T12:38:42.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:42.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 10 client.?.objecter ms_handle_connect 0x7f5d180a5920 2024-08-19T12:38:42.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:42.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5d16ffd640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.26:6790/0 conn(0x7f5d180a5920 legacy=0x7f5d181ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1422193243 0 0) 0x7f5d181cf060 con 0x7f5d180a5920 2024-08-19T12:38:42.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 10 monclient(hunting): my global_id is 4556 2024-08-19T12:38:42.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d04004eb0 con 0x7f5d180a5920 2024-08-19T12:38:42.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3344232799 0 0) 0x7f5d1813b2e0 con 0x7f5d1806e7b0 2024-08-19T12:38:42.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 10 monclient(hunting): my global_id is 4580 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.429+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d181cf060 con 0x7f5d1806e7b0 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5d17fff640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.104:6790/0 conn(0x7f5d180a5d50 legacy=0x7f5d181d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 client.?.objecter ms_handle_connect 0x7f5d180a5d50 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 800101032 0 0) 0x7f5d181cf060 con 0x7f5d1806e7b0 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d1813b2e0 con 0x7f5d1806e7b0 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5d080019d0 con 0x7f5d1806e7b0 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:42.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient(hunting): dump: 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.446 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3055559279 0 0) 0x7f5d04004eb0 con 0x7f5d180a5920 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d181cf060 con 0x7f5d180a5920 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 835989745 0 0) 0x7f5d18142ec0 con 0x7f5d180a5d50 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient(hunting): my global_id is 4575 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d04004eb0 con 0x7f5d180a5d50 2024-08-19T12:38:42.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5d0c003610 con 0x7f5d180a5920 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient(hunting): dump: 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.448 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.449 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.449 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.449 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.449 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.449 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.449 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2692839859 0 0) 0x7f5d1813b2e0 con 0x7f5d1806e7b0 2024-08-19T12:38:42.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.104:6790/0 conn(0x7f5d180a5d50 legacy=0x7f5d181d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.26:6790/0 conn(0x7f5d180a5920 legacy=0x7f5d181ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:42.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 monclient: found mon.e 2024-08-19T12:38:42.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d18142a50 con 0x7f5d1806e7b0 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient: _finish_auth 0 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5d1e0b3640 5 monclient: authenticate success, global_id 4580 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5d1e0b3640 10 monclient: _renew_subs 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5d1e0b3640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5d1813aa40 con 0x7f5d1806e7b0 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5d1e0b3640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5d1e0b3640 10 monclient: _renew_subs 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5d1e0b3640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:42.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f5d1813b700 con 0x7f5d1806e7b0 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5d1e0b3640 1 librados: init done 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d08004dd0 con 0x7f5d1806e7b0 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5d080052c0 con 0x7f5d1806e7b0 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5d15ffb640 4 set_mon_vals no callback set 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.430+0000 7f5cfffff640 10 monclient: dump: 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.451 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.431+0000 7f5d1e0b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.431+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5cf80065c0 con 0x7f5cf8002860 2024-08-19T12:38:42.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.431+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5d08011720 con 0x7f5d1806e7b0 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.431+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f5d0804db30 con 0x7f5d1806e7b0 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.431+0000 7f5cfffff640 10 client.4580.objecter ms_dispatch 0x7f5d18000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.431+0000 7f5cfffff640 3 client.4580.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.431+0000 7f5cfffff640 3 client.4580.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.432+0000 7f5cfffff640 20 client.4580.objecter dump_active .. 0 homeless 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.432+0000 7f5d17fff640 10 monclient: get_auth_request con 0x7f5d04041790 auth_method 0 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.433+0000 7f5cfffff640 10 client.4580.objecter ms_handle_connect 0x7f5cf8002860 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.433+0000 7f5cfffff640 10 client.4580.objecter resend_mon_ops 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.433+0000 7f5d16ffd640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.104:6791/0 conn(0x7f5cf8002860 legacy=0x7f5cf8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 358029972 0 0) 0x7f5cf80065c0 con 0x7f5cf8002860 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 10 monclient(hunting): my global_id is 4433 2024-08-19T12:38:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d1813b2e0 con 0x7f5cf8002860 2024-08-19T12:38:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 594241897 0 0) 0x7f5d1813b2e0 con 0x7f5cf8002860 2024-08-19T12:38:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5cf80065c0 con 0x7f5cf8002860 2024-08-19T12:38:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5d0c003d60 con 0x7f5cf8002860 2024-08-19T12:38:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 10 monclient: dump: 2024-08-19T12:38:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.454 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 10 client.4580.objecter ms_handle_connect 0x7f5d04041790 2024-08-19T12:38:42.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 13929899 0 0) 0x7f5cf80065c0 con 0x7f5cf8002860 2024-08-19T12:38:42.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5cf800a160 con 0x7f5cf8002860 2024-08-19T12:38:42.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5d0c003fe0 con 0x7f5cf8002860 2024-08-19T12:38:42.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:42.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:42.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.104:6791/0 conn(0x7f5cf8002860 legacy=0x7f5cf8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.434+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.104:6791/0 conn(0x7f5cf8002860 legacy=0x7f5cf8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:42.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.440+0000 7f5d1e0b3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:42.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.441+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5cf8009e70 con 0x7f5cf8009130 2024-08-19T12:38:42.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.443+0000 7f5d16ffd640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.104:6791/0 conn(0x7f5cf8009130 legacy=0x7f5cf80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.443+0000 7f5cfffff640 10 client.4580.objecter ms_handle_connect 0x7f5cf8009130 2024-08-19T12:38:42.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.443+0000 7f5cfffff640 10 client.4580.objecter resend_mon_ops 2024-08-19T12:38:42.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.443+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4111796048 0 0) 0x7f5cf8009e70 con 0x7f5cf8009130 2024-08-19T12:38:42.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 10 monclient(hunting): my global_id is 4442 2024-08-19T12:38:42.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cf80065c0 con 0x7f5cf8009130 2024-08-19T12:38:42.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2655945278 0 0) 0x7f5cf80065c0 con 0x7f5cf8009130 2024-08-19T12:38:42.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5cf8009e70 con 0x7f5cf8009130 2024-08-19T12:38:42.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5d0c005b00 con 0x7f5cf8009130 2024-08-19T12:38:42.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 10 monclient: dump: 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.460 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.460 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.460 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.460 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1051072967 0 0) 0x7f5cf8009e70 con 0x7f5cf8009130 2024-08-19T12:38:42.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5cf800c410 con 0x7f5cf8009130 2024-08-19T12:38:42.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7710 (unknown 0 0 1082675523) 0x7f5d0c006f10 con 0x7f5cf8009130 2024-08-19T12:38:42.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:42.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:42.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.104:6791/0 conn(0x7f5cf8009130 legacy=0x7f5cf80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.444+0000 7f5cfffff640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.104:6791/0 conn(0x7f5cf8009130 legacy=0x7f5cf80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:42.460 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 4, 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 35467, 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:42.461 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 34, 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:42.462 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:42.463 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:42.464 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:42.465 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:42.466 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.468 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.469 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:42.470 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.471 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.472 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.473 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.475 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.476 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.477 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:42.478 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:42.479 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:38:42.480 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.480 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.480 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:42.480 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:42.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 10 librados: watch_flush enter 2024-08-19T12:38:42.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 10 librados: watch_flush exit 2024-08-19T12:38:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 20 client.4580.objecter shutdown clearing up homeless session... 2024-08-19T12:38:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 10 client.4580.objecter successfully canceled tick 2024-08-19T12:38:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.104:6800/899887506 conn(0x7f5d04041790 legacy=0x7f5d04043c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 10 monclient: shutdown 2024-08-19T12:38:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/2020708884 >> v1:172.21.15.26:6793/0 conn(0x7f5d1806e7b0 legacy=0x7f5d181ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d17fff640 1 -- 172.21.15.26:0/2020708884 reap_dead start 2024-08-19T12:38:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d17fff640 1 -- 172.21.15.26:0/2020708884 reap_dead start 2024-08-19T12:38:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/2020708884 shutdown_connections 2024-08-19T12:38:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/2020708884 >> 172.21.15.26:0/2020708884 conn(0x7f5d180a8740 msgr2=0x7f5d180aaf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/2020708884 shutdown_connections 2024-08-19T12:38:42.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 1 -- 172.21.15.26:0/2020708884 wait complete. 2024-08-19T12:38:42.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.445+0000 7f5d1e0b3640 1 librados: shutdown 2024-08-19T12:38:42.483 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:38:42.483 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:38:42.483 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:38:42.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.583+0000 7f6012060640 10 monclient: get_monmap_and_config 2024-08-19T12:38:42.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.583+0000 7f6012060640 10 monclient: build_initial_monmap 2024-08-19T12:38:42.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.583+0000 7f6012060640 10 monclient: monmap: 2024-08-19T12:38:42.584 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:42.584 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:42.584240+0000 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:42.584240+0000 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:42.585 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:42.586 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 1 Processor -- start 2024-08-19T12:38:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 1 -- start start 2024-08-19T12:38:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 10 monclient: init 2024-08-19T12:38:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 10 monclient: _add_conns ranks=[0,7,6,4,5,2,8,1,3] 2024-08-19T12:38:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 10 monclient(hunting): picked mon.noname-a con 0x7f600c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 10 monclient(hunting): picked mon.noname-i con 0x7f600c0a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 10 monclient(hunting): picked mon.noname-d con 0x7f600c0a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:42.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f600c136790 con 0x7f600c06e7b0 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f600c09ddf0 con 0x7f600c0a5d50 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f600c09e0d0 con 0x7f600c0a5920 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f6012060640 10 monclient(hunting): authenticate will time out at 716.671713s 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f600b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f600c06e7b0 0x7f600c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:51426/0 (socket says 172.21.15.26:51426) 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f600bfff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f600c0a5d50 0x7f600c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:40946/0 (socket says 172.21.15.26:40946) 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.584+0000 7f600b7fe640 1 -- 172.21.15.26:0/2522822236 learned_addr learned my addr 172.21.15.26:0/2522822236 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.586+0000 7f600bfff640 1 -- 172.21.15.26:0/2522822236 >> v1:172.21.15.26:6792/0 conn(0x7f600c0a5d50 legacy=0x7f600c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600b7fe640 1 -- 172.21.15.26:0/2522822236 >> v1:172.21.15.26:6789/0 conn(0x7f600c06e7b0 legacy=0x7f600c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 1 -- 172.21.15.26:0/2522822236 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2163578694 0 0) 0x7f60040017d0 con 0x7f600c0a5d50 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient(hunting): my global_id is 4560 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 1 -- 172.21.15.26:0/2522822236 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ff8003520 con 0x7f600c0a5d50 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 1 -- 172.21.15.26:0/2522822236 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 920770763 0 0) 0x7f600c136790 con 0x7f600c06e7b0 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient(hunting): my global_id is 4473 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 1 -- 172.21.15.26:0/2522822236 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60040017d0 con 0x7f600c06e7b0 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f6002ffd640 1 -- 172.21.15.26:0/2522822236 >> v1:172.21.15.104:6792/0 conn(0x7f600c0a5920 legacy=0x7f600c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 1 -- 172.21.15.26:0/2522822236 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1599675046 0 0) 0x7f5ff8003520 con 0x7f600c0a5d50 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 1 -- 172.21.15.26:0/2522822236 >> v1:172.21.15.104:6792/0 conn(0x7f600c0a5920 legacy=0x7f600c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 1 -- 172.21.15.26:0/2522822236 >> v1:172.21.15.26:6789/0 conn(0x7f600c06e7b0 legacy=0x7f600c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 1 monclient: found mon.noname-d 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 1 -- 172.21.15.26:0/2522822236 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600c09e3b0 con 0x7f600c0a5d50 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient: _finish_auth 0 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 1 -- 172.21.15.26:0/2522822236 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6004002030 con 0x7f600c0a5d50 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f6012060640 5 monclient: authenticate success, global_id 4560 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f6012060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient: dump: 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f6012060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600bfff640 1 --1- 172.21.15.26:0/2522822236 >> v1:172.21.15.26:6792/0 conn(0x7f600c0a5d50 0x7f600c141a20 :-1 s=OPENED pgs=89 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f6004005180 mon_map magic: 0 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 1 -- 172.21.15.26:0/2522822236 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6004004c10 con 0x7f600c0a5d50 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f600affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f6012060640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.587+0000 7f6012060640 4 set_mon_vals no callback set 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f6012060640 10 monclient: shutdown 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f6012060640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f6012060640 1 -- 172.21.15.26:0/2522822236 >> v1:172.21.15.26:6792/0 conn(0x7f600c0a5d50 legacy=0x7f600c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f600affd640 1 -- 172.21.15.26:0/2522822236 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6004005180 con 0x7f600c0a5d50 2024-08-19T12:38:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f600affd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:38:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f6012060640 1 -- 172.21.15.26:0/2522822236 shutdown_connections 2024-08-19T12:38:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f6012060640 1 -- 172.21.15.26:0/2522822236 >> 172.21.15.26:0/2522822236 conn(0x7f600c0a8740 msgr2=0x7f600c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f6012060640 1 -- 172.21.15.26:0/2522822236 shutdown_connections 2024-08-19T12:38:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f6012060640 1 -- 172.21.15.26:0/2522822236 wait complete. 2024-08-19T12:38:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f6012060640 10 monclient: build_initial_monmap 2024-08-19T12:38:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f6012060640 10 monclient: monmap: 2024-08-19T12:38:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f6012060640 1 librados: starting msgr at 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.588+0000 7f6012060640 1 librados: starting objecter 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 1 Processor -- start 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 1 -- start start 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 1 librados: setting wanted keys 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 1 librados: calling monclient init 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 10 monclient: init 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 10 monclient: _add_conns ranks=[4,3,2,8,5,7,0,6,1] 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 10 monclient(hunting): picked mon.noname-c con 0x7f600c06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 10 monclient(hunting): picked mon.noname-g con 0x7f600c0a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:42.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 10 monclient(hunting): picked mon.noname-b con 0x7f600c0a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:42.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f600c138fa0 con 0x7f600c0a5d50 2024-08-19T12:38:42.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f600c13b300 con 0x7f600c0a5920 2024-08-19T12:38:42.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f600c13aa60 con 0x7f600c06e7b0 2024-08-19T12:38:42.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:42.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f6012060640 10 monclient(hunting): authenticate will time out at 716.676819s 2024-08-19T12:38:42.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f600b7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f600c06e7b0 0x7f600c137870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:43354/0 (socket says 172.21.15.26:43354) 2024-08-19T12:38:42.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.589+0000 7f600b7fe640 1 -- 172.21.15.26:0/4117844861 learned_addr learned my addr 172.21.15.26:0/4117844861 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f600b7fe640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.26:6791/0 conn(0x7f600c06e7b0 legacy=0x7f600c137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 client.?.objecter ms_handle_connect 0x7f600c06e7b0 2024-08-19T12:38:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f600bfff640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.26:6790/0 conn(0x7f600c0a5d50 legacy=0x7f600c1d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 client.?.objecter ms_handle_connect 0x7f600c0a5d50 2024-08-19T12:38:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 815796705 0 0) 0x7f600c13aa60 con 0x7f600c06e7b0 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): my global_id is 4504 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fec004eb0 con 0x7f600c06e7b0 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3680808509 0 0) 0x7f600c138fa0 con 0x7f600c0a5d50 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): my global_id is 4565 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f600c13aa60 con 0x7f600c0a5d50 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 client.?.objecter ms_handle_connect 0x7f600c0a5920 2024-08-19T12:38:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6002ffd640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.104:6790/0 conn(0x7f600c0a5920 legacy=0x7f600c137f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 546816015 0 0) 0x7f600c13b300 con 0x7f600c0a5920 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): my global_id is 4584 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f600c138fa0 con 0x7f600c0a5920 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2082998489 0 0) 0x7f5fec004eb0 con 0x7f600c06e7b0 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f600c13b300 con 0x7f600c06e7b0 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3756445535 0 0) 0x7f600c13aa60 con 0x7f600c0a5d50 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fec004eb0 con 0x7f600c0a5d50 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5ffc0047e0 con 0x7f600c06e7b0 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6004005140 con 0x7f600c0a5d50 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1708166549 0 0) 0x7f600c13b300 con 0x7f600c06e7b0 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.104:6790/0 conn(0x7f600c0a5920 legacy=0x7f600c137f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.26:6790/0 conn(0x7f600c0a5d50 legacy=0x7f600c1d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 monclient: found mon.c 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600c13a710 con 0x7f600c06e7b0 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient: _finish_auth 0 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6012060640 5 monclient: authenticate success, global_id 4504 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6012060640 10 monclient: _renew_subs 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6012060640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6012060640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f600c139b00 con 0x7f600c06e7b0 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6012060640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6012060640 10 monclient: _renew_subs 2024-08-19T12:38:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6012060640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6012060640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f600c138870 con 0x7f600c06e7b0 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6012060640 1 librados: init done 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ffc004cb0 con 0x7f600c06e7b0 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5ffc0051e0 con 0x7f600c06e7b0 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f600a7fc640 4 set_mon_vals no callback set 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.592+0000 7f6008ff9640 10 monclient: dump: 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.593+0000 7f6012060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.593+0000 7f6012060640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5ff00065c0 con 0x7f5ff0002860 2024-08-19T12:38:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.593+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5ffc005380 con 0x7f600c06e7b0 2024-08-19T12:38:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.594+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f5ffc04db90 con 0x7f600c06e7b0 2024-08-19T12:38:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.594+0000 7f6008ff9640 10 client.4504.objecter ms_dispatch 0x7f600c000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.594+0000 7f6008ff9640 3 client.4504.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.594+0000 7f6008ff9640 3 client.4504.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.594+0000 7f6008ff9640 20 client.4504.objecter dump_active .. 0 homeless 2024-08-19T12:38:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.594+0000 7f600bfff640 10 monclient: get_auth_request con 0x7f5fec0414b0 auth_method 0 2024-08-19T12:38:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 10 client.4504.objecter ms_handle_connect 0x7f5ff0002860 2024-08-19T12:38:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6002ffd640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.104:6792/0 conn(0x7f5ff0002860 legacy=0x7f5ff0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 10 client.4504.objecter resend_mon_ops 2024-08-19T12:38:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2911656511 0 0) 0x7f5ff00065c0 con 0x7f5ff0002860 2024-08-19T12:38:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 10 monclient(hunting): my global_id is 4489 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f600c13b300 con 0x7f5ff0002860 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 10 client.4504.objecter ms_handle_connect 0x7f5fec0414b0 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 233965220 0 0) 0x7f600c13b300 con 0x7f5ff0002860 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ff00065c0 con 0x7f5ff0002860 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5ff4003c10 con 0x7f5ff0002860 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 10 monclient: dump: 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2703778727 0 0) 0x7f5ff00065c0 con 0x7f5ff0002860 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.596+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ff000a160 con 0x7f5ff0002860 2024-08-19T12:38:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.597+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5ff4003f80 con 0x7f5ff0002860 2024-08-19T12:38:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.597+0000 7f6008ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.597+0000 7f6008ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.597+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.104:6792/0 conn(0x7f5ff0002860 legacy=0x7f5ff0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.597+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.104:6792/0 conn(0x7f5ff0002860 legacy=0x7f5ff0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.603+0000 7f6012060640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.603+0000 7f6012060640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5ff0009e50 con 0x7f5ff0009110 2024-08-19T12:38:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.605+0000 7f6002ffd640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.104:6792/0 conn(0x7f5ff0009110 legacy=0x7f5ff00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.605+0000 7f6008ff9640 10 client.4504.objecter ms_handle_connect 0x7f5ff0009110 2024-08-19T12:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.605+0000 7f6008ff9640 10 client.4504.objecter resend_mon_ops 2024-08-19T12:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 537327839 0 0) 0x7f5ff0009e50 con 0x7f5ff0009110 2024-08-19T12:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 10 monclient(hunting): my global_id is 4498 2024-08-19T12:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ff00065c0 con 0x7f5ff0009110 2024-08-19T12:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3654838979 0 0) 0x7f5ff00065c0 con 0x7f5ff0009110 2024-08-19T12:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ff0009e50 con 0x7f5ff0009110 2024-08-19T12:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5ff4004460 con 0x7f5ff0009110 2024-08-19T12:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 10 monclient: dump: 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.621 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4169340105 0 0) 0x7f5ff0009e50 con 0x7f5ff0009110 2024-08-19T12:38:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5ff000c3f0 con 0x7f5ff0009110 2024-08-19T12:38:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7710 (unknown 0 0 222814993) 0x7f5ff4005c70 con 0x7f5ff0009110 2024-08-19T12:38:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.104:6792/0 conn(0x7f5ff0009110 legacy=0x7f5ff00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.606+0000 7f6008ff9640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.104:6792/0 conn(0x7f5ff0009110 legacy=0x7f5ff00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:42.621 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:42.622 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:42.622 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:42.622 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:42.622 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 4, 2024-08-19T12:38:42.622 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 35619, 2024-08-19T12:38:42.622 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:42.622 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:42.622 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:42.622 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:42.622 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:42.622 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:42.622 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:42.623 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:42.623 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:38:42.623 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:42.623 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.623 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 34, 2024-08-19T12:38:42.623 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:42.623 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:42.623 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:42.623 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:42.623 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:42.623 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:42.623 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:42.624 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:42.626 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.627 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.629 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.630 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.631 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.633 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:42.634 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.635 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.637 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.638 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:42.639 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.640 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:42.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:42.642 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:42.642 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:42.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 10 librados: watch_flush enter 2024-08-19T12:38:42.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 10 librados: watch_flush exit 2024-08-19T12:38:42.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 20 client.4504.objecter shutdown clearing up homeless session... 2024-08-19T12:38:42.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 10 client.4504.objecter successfully canceled tick 2024-08-19T12:38:42.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.104:6800/899887506 conn(0x7f5fec0414b0 legacy=0x7f5fec043970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 10 monclient: shutdown 2024-08-19T12:38:42.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:42.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 1 -- 172.21.15.26:0/4117844861 >> v1:172.21.15.26:6791/0 conn(0x7f600c06e7b0 legacy=0x7f600c137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f600bfff640 1 -- 172.21.15.26:0/4117844861 reap_dead start 2024-08-19T12:38:42.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f600bfff640 1 -- 172.21.15.26:0/4117844861 reap_dead start 2024-08-19T12:38:42.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 1 -- 172.21.15.26:0/4117844861 shutdown_connections 2024-08-19T12:38:42.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 1 -- 172.21.15.26:0/4117844861 >> 172.21.15.26:0/4117844861 conn(0x7f600c0a8740 msgr2=0x7f600c06e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:42.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 1 -- 172.21.15.26:0/4117844861 shutdown_connections 2024-08-19T12:38:42.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 1 -- 172.21.15.26:0/4117844861 wait complete. 2024-08-19T12:38:42.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.607+0000 7f6012060640 1 librados: shutdown 2024-08-19T12:38:42.645 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:38:42.645 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['i'] 2024-08-19T12:38:42.645 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-19T12:38:42.646 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-08-19T12:38:42.646 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-08-19T12:38:42.646 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-08-19T12:38:42.646 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:38:42.659 INFO:tasks.ceph.mon.i:Stopped 2024-08-19T12:38:42.659 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-08-19T12:38:42.660 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:38:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.757+0000 7f7aa4e07640 10 monclient: get_monmap_and_config 2024-08-19T12:38:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.757+0000 7f7aa4e07640 10 monclient: build_initial_monmap 2024-08-19T12:38:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.757+0000 7f7aa4e07640 10 monclient: monmap: 2024-08-19T12:38:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:42.757919+0000 2024-08-19T12:38:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:42.757919+0000 2024-08-19T12:38:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:42.759 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.757+0000 7f7aa4e07640 1 Processor -- start 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.757+0000 7f7aa4e07640 1 -- start start 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.757+0000 7f7aa4e07640 10 monclient: init 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.758+0000 7f7aa4e07640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.758+0000 7f7aa4e07640 10 monclient: _add_conns ranks=[1,5,2,7,8,6,3,0,4] 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.758+0000 7f7aa4e07640 10 monclient(hunting): picked mon.noname-f con 0x7f7aa0073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.758+0000 7f7aa4e07640 10 monclient(hunting): picked mon.noname-h con 0x7f7aa00a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.758+0000 7f7aa4e07640 10 monclient(hunting): picked mon.noname-b con 0x7f7aa013f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.758+0000 7f7aa4e07640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aa009dae0 con 0x7f7aa0073470 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.758+0000 7f7aa4e07640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aa0136910 con 0x7f7aa013f2e0 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.758+0000 7f7aa4e07640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aa009dd50 con 0x7f7aa00a57a0 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.758+0000 7f7aa4e07640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.758+0000 7f7aa4e07640 10 monclient(hunting): authenticate will time out at 716.845325s 2024-08-19T12:38:42.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.758+0000 7f7a9ed9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7aa013f2e0 0x7f7aa01416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:57160/0 (socket says 172.21.15.26:57160) 2024-08-19T12:38:42.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.758+0000 7f7a9ed9d640 1 -- 172.21.15.26:0/2592927465 learned_addr learned my addr 172.21.15.26:0/2592927465 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:42.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.760+0000 7f7a9ed9d640 1 -- 172.21.15.26:0/2592927465 >> v1:172.21.15.26:6790/0 conn(0x7f7aa013f2e0 legacy=0x7f7aa01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.760+0000 7f7a9dd9b640 1 -- 172.21.15.26:0/2592927465 >> v1:172.21.15.104:6791/0 conn(0x7f7aa00a57a0 legacy=0x7f7aa00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.760+0000 7f7a9e59c640 1 -- 172.21.15.26:0/2592927465 >> v1:172.21.15.104:6789/0 conn(0x7f7aa0073470 legacy=0x7f7aa0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.760+0000 7f7a9d59a640 1 -- 172.21.15.26:0/2592927465 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 931697811 0 0) 0x7f7aa0136910 con 0x7f7aa013f2e0 2024-08-19T12:38:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.760+0000 7f7a9d59a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.760+0000 7f7a9d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.760+0000 7f7a9d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.760+0000 7f7a9d59a640 10 monclient(hunting): my global_id is 4574 2024-08-19T12:38:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.760+0000 7f7a9d59a640 1 -- 172.21.15.26:0/2592927465 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a8c003520 con 0x7f7aa013f2e0 2024-08-19T12:38:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 1 -- 172.21.15.26:0/2592927465 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3675217191 0 0) 0x7f7aa009dd50 con 0x7f7aa00a57a0 2024-08-19T12:38:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient(hunting): my global_id is 4451 2024-08-19T12:38:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 1 -- 172.21.15.26:0/2592927465 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7aa0136910 con 0x7f7aa00a57a0 2024-08-19T12:38:42.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 1 -- 172.21.15.26:0/2592927465 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1923363945 0 0) 0x7f7a8c003520 con 0x7f7aa013f2e0 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 1 -- 172.21.15.26:0/2592927465 >> v1:172.21.15.104:6791/0 conn(0x7f7aa00a57a0 legacy=0x7f7aa00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 1 -- 172.21.15.26:0/2592927465 >> v1:172.21.15.104:6789/0 conn(0x7f7aa0073470 legacy=0x7f7aa0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient: _finish_hunting 0 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 1 monclient: found mon.noname-b 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 1 -- 172.21.15.26:0/2592927465 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa009e030 con 0x7f7aa013f2e0 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient: _finish_auth 0 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient: _check_auth_tickets 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7aa4e07640 5 monclient: authenticate success, global_id 4574 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 1 -- 172.21.15.26:0/2592927465 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7a98001e60 con 0x7f7aa013f2e0 2024-08-19T12:38:42.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7aa4e07640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient: dump: 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.768 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7aa4e07640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 1 -- 172.21.15.26:0/2592927465 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7a98004c20 con 0x7f7aa013f2e0 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 1 -- 172.21.15.26:0/2592927465 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7a98005190 con 0x7f7aa013f2e0 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7a9d59a640 10 monclient: dump: 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.769 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.770 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7aa4e07640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7aa4e07640 4 set_mon_vals no callback set 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7aa4e07640 10 monclient: shutdown 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7aa4e07640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2592927465 >> v1:172.21.15.26:6790/0 conn(0x7f7aa013f2e0 legacy=0x7f7aa01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2592927465 shutdown_connections 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2592927465 >> 172.21.15.26:0/2592927465 conn(0x7f7aa00abff0 msgr2=0x7f7aa00ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2592927465 shutdown_connections 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.761+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2592927465 wait complete. 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.762+0000 7f7aa4e07640 10 monclient: build_initial_monmap 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.762+0000 7f7aa4e07640 10 monclient: monmap: 2024-08-19T12:38:42.771 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:42.772 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.762+0000 7f7aa4e07640 1 librados: starting msgr at 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.762+0000 7f7aa4e07640 1 librados: starting objecter 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.762+0000 7f7aa4e07640 1 Processor -- start 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.762+0000 7f7aa4e07640 1 -- start start 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.762+0000 7f7aa4e07640 1 librados: setting wanted keys 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.762+0000 7f7aa4e07640 1 librados: calling monclient init 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.762+0000 7f7aa4e07640 10 monclient: init 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7aa4e07640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7aa4e07640 10 monclient: _add_conns ranks=[1,4,7,0,3,2,6,8,5] 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7aa4e07640 10 monclient(hunting): picked mon.noname-f con 0x7f7aa0073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7aa4e07640 10 monclient(hunting): picked mon.noname-c con 0x7f7aa00a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7aa4e07640 10 monclient(hunting): picked mon.noname-i con 0x7f7aa013f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:42.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7aa4e07640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aa0136e40 con 0x7f7aa0073470 2024-08-19T12:38:42.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7aa4e07640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aa006d8f0 con 0x7f7aa00a57a0 2024-08-19T12:38:42.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7aa4e07640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aa00a39d0 con 0x7f7aa013f2e0 2024-08-19T12:38:42.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7aa4e07640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:42.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7aa4e07640 10 monclient(hunting): authenticate will time out at 716.850377s 2024-08-19T12:38:42.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7a9dd9b640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f7aa00a57a0 0x7f7aa00a27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:43368/0 (socket says 172.21.15.26:43368) 2024-08-19T12:38:42.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7a9dd9b640 1 -- 172.21.15.26:0/2857094903 learned_addr learned my addr 172.21.15.26:0/2857094903 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:42.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.763+0000 7f7a9ed9d640 1 -- 172.21.15.26:0/2857094903 >> v1:172.21.15.104:6792/0 conn(0x7f7aa013f2e0 legacy=0x7f7aa01d6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:38:42.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a9dd9b640 1 -- 172.21.15.26:0/2857094903 >> v1:172.21.15.26:6791/0 conn(0x7f7aa00a57a0 legacy=0x7f7aa00a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 10 client.?.objecter ms_handle_connect 0x7f7aa00a57a0 2024-08-19T12:38:42.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:42.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a9e59c640 1 -- 172.21.15.26:0/2857094903 >> v1:172.21.15.104:6789/0 conn(0x7f7aa0073470 legacy=0x7f7aa00a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:42.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2856031463 0 0) 0x7f7aa006d8f0 con 0x7f7aa00a57a0 2024-08-19T12:38:42.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 10 monclient(hunting): my global_id is 4513 2024-08-19T12:38:42.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a80004eb0 con 0x7f7aa00a57a0 2024-08-19T12:38:42.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 10 client.?.objecter ms_handle_connect 0x7f7aa0073470 2024-08-19T12:38:42.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:42.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1032855628 0 0) 0x7f7a90001630 con 0x7f7aa0073470 2024-08-19T12:38:42.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:42.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 10 monclient(hunting): my global_id is 4465 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7aa006d8f0 con 0x7f7aa0073470 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.765+0000 7f7a9dd9b640 1 --1- 172.21.15.26:0/2857094903 >> v1:172.21.15.26:6791/0 conn(0x7f7aa00a57a0 0x7f7aa00a27c0 :-1 s=OPENED pgs=102 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f7a80004eb0 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1778121498 0 0) 0x7f7a80004eb0 con 0x7f7aa00a57a0 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a90001630 con 0x7f7aa00a57a0 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7a88003610 con 0x7f7aa00a57a0 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.780 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:42.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 435994668 0 0) 0x7f7aa006d8f0 con 0x7f7aa0073470 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a80004eb0 con 0x7f7aa0073470 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7a90003950 con 0x7f7aa0073470 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.782 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3830716306 0 0) 0x7f7a80004eb0 con 0x7f7aa0073470 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 >> v1:172.21.15.104:6792/0 conn(0x7f7aa013f2e0 legacy=0x7f7aa01d6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 >> v1:172.21.15.26:6791/0 conn(0x7f7aa00a57a0 legacy=0x7f7aa00a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:42.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 monclient: found mon.f 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa00a2ed0 con 0x7f7aa0073470 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7aa4e07640 5 monclient: authenticate success, global_id 4465 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1171631270 0 0) 0x7f7a90001630 con 0x7f7aa00a57a0 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7aa4e07640 10 monclient: _renew_subs 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7aa4e07640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2857094903 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa00a47d0 con 0x7f7aa0073470 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7aa4e07640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7aa4e07640 10 monclient: _renew_subs 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7aa4e07640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2857094903 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7aa0137a00 con 0x7f7aa0073470 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7aa4e07640 1 librados: init done 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7a90003c20 con 0x7f7aa0073470 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7a90004210 con 0x7f7aa0073470 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a9cd99640 4 set_mon_vals no callback set 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.766+0000 7f7a96ffd640 10 monclient: dump: 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.785 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:42.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.767+0000 7f7aa4e07640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.767+0000 7f7aa4e07640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.767+0000 7f7aa4e07640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.767+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2857094903 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a84005080 con 0x7f7aa0073470 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.767+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f7a90011a80 con 0x7f7aa0073470 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.767+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f7a9004dd20 con 0x7f7aa0073470 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.767+0000 7f7a96ffd640 10 client.4465.objecter ms_dispatch 0x7f7aa0000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.767+0000 7f7a96ffd640 3 client.4465.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.767+0000 7f7a96ffd640 3 client.4465.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.767+0000 7f7a96ffd640 20 client.4465.objecter dump_active .. 0 homeless 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.768+0000 7f7a9dd9b640 10 monclient: get_auth_request con 0x7f7a80041310 auth_method 0 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.770+0000 7f7a96ffd640 10 client.4465.objecter ms_handle_connect 0x7f7a80041310 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.772+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.1 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f7aa0136e40 con 0x7f7aa0073470 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.772+0000 7f7a96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:42.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:42.772+0000 7f7a96ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:43.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.030+0000 7f7aa4e07640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:38:43.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.030+0000 7f7aa4e07640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:38:43.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.030+0000 7f7aa4e07640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:38:43.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.030+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2857094903 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7a84005370 con 0x7f7aa0073470 2024-08-19T12:38:43.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.031+0000 7f7a96ffd640 1 -- 172.21.15.26:0/2857094903 <== mon.1 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 2108680076) 0x7f7a90017840 con 0x7f7aa0073470 2024-08-19T12:38:43.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.031+0000 7f7a96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:38:43.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.031+0000 7f7a96ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:43.032 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":4,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":34,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:38:43.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.032+0000 7f7aa4e07640 10 librados: watch_flush enter 2024-08-19T12:38:43.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.033+0000 7f7aa4e07640 10 librados: watch_flush exit 2024-08-19T12:38:43.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.033+0000 7f7aa4e07640 20 client.4465.objecter shutdown clearing up homeless session... 2024-08-19T12:38:43.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.033+0000 7f7aa4e07640 10 client.4465.objecter successfully canceled tick 2024-08-19T12:38:43.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.033+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2857094903 >> v1:172.21.15.104:6800/899887506 conn(0x7f7a80041310 legacy=0x7f7a800437d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:43.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.033+0000 7f7aa4e07640 10 monclient: shutdown 2024-08-19T12:38:43.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.033+0000 7f7aa4e07640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:43.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.033+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2857094903 >> v1:172.21.15.104:6789/0 conn(0x7f7aa0073470 legacy=0x7f7aa00a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:43.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.033+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2857094903 shutdown_connections 2024-08-19T12:38:43.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.033+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2857094903 >> 172.21.15.26:0/2857094903 conn(0x7f7aa00abff0 msgr2=0x7f7aa0135430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:43.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.033+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2857094903 shutdown_connections 2024-08-19T12:38:43.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.033+0000 7f7aa4e07640 1 -- 172.21.15.26:0/2857094903 wait complete. 2024-08-19T12:38:43.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:43.033+0000 7f7aa4e07640 1 librados: shutdown 2024-08-19T12:38:46.045 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:38:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.153+0000 7ff002372640 10 monclient: get_monmap_and_config 2024-08-19T12:38:46.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.153+0000 7ff002372640 10 monclient: build_initial_monmap 2024-08-19T12:38:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.153+0000 7ff002372640 10 monclient: monmap: 2024-08-19T12:38:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:46.154672+0000 2024-08-19T12:38:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:46.154672+0000 2024-08-19T12:38:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:46.155 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:46.156 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:46.156 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.154+0000 7ff002372640 1 Processor -- start 2024-08-19T12:38:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.154+0000 7ff002372640 1 -- start start 2024-08-19T12:38:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.154+0000 7ff002372640 10 monclient: init 2024-08-19T12:38:46.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7ff002372640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7ff002372640 10 monclient: _add_conns ranks=[7,6,8,3,1,0,4,5,2] 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7ff002372640 10 monclient(hunting): picked mon.noname-i con 0x7feffc06e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7ff002372640 10 monclient(hunting): picked mon.noname-d con 0x7feffc0a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7ff002372640 10 monclient(hunting): picked mon.noname-e con 0x7feffc0a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7ff002372640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc136790 con 0x7feffc0a5920 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7ff002372640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc09ddf0 con 0x7feffc06e7b0 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7ff002372640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc09e0d0 con 0x7feffc0a5d50 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7ff002372640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7ff002372640 10 monclient(hunting): authenticate will time out at 720.242209s 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7feffbfff640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7feffc06e7b0 legacy=0x7feffc136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7feffb7fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7feffc0a5920 0x7feffc13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:40956/0 (socket says 172.21.15.26:40956) 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7feffb7fe640 1 -- 172.21.15.26:0/4254773014 learned_addr learned my addr 172.21.15.26:0/4254773014 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:46.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.155+0000 7ff00090f640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7feffc0a5d50 0x7feffc141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40162/0 (socket says 172.21.15.26:40162) 2024-08-19T12:38:46.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.157+0000 7feffb7fe640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.26:6792/0 conn(0x7feffc0a5920 legacy=0x7feffc13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:46.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.157+0000 7ff00090f640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.26:6793/0 conn(0x7feffc0a5d50 legacy=0x7feffc141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:46.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.356+0000 7feffbfff640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6792/0 conn(0x7feffc06e7b0 legacy=0x7feffc136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:38:46.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:46.757+0000 7feffbfff640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6792/0 conn(0x7feffc06e7b0 legacy=0x7feffc136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:38:47.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:47.558+0000 7feffbfff640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6792/0 conn(0x7feffc06e7b0 legacy=0x7feffc136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:38:49.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 10 monclient(hunting): tick 2024-08-19T12:38:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:38:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 1 monclient(hunting): continuing hunt 2024-08-19T12:38:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:38:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.26:6793/0 conn(0x7feffc0a5d50 legacy=0x7feffc141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6792/0 conn(0x7feffc06e7b0 legacy=0x7feffc136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:38:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.26:6792/0 conn(0x7feffc0a5920 legacy=0x7feffc13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 10 monclient: _add_conns ranks=[0,5,1,3,2,4] 2024-08-19T12:38:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fefe80038c0 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 10 monclient(hunting): picked mon.noname-h con 0x7fefe8007530 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 10 monclient(hunting): picked mon.noname-f con 0x7fefe800b2a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc09e3b0 con 0x7fefe80038c0 2024-08-19T12:38:49.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe8010040 con 0x7fefe800b2a0 2024-08-19T12:38:49.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe80112e0 con 0x7fefe8007530 2024-08-19T12:38:49.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.155+0000 7feff9ffb640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:49.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.157+0000 7feffbfff640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.26:6789/0 conn(0x7fefe80038c0 legacy=0x7fefe8005cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:49.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.157+0000 7ff00090f640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6789/0 conn(0x7fefe800b2a0 legacy=0x7fefe800d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:49.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:49.157+0000 7feffb7fe640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6791/0 conn(0x7fefe8007530 legacy=0x7fefe80099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:52.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 10 monclient(hunting): tick 2024-08-19T12:38:52.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:38:52.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 1 monclient(hunting): continuing hunt 2024-08-19T12:38:52.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:38:52.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6791/0 conn(0x7fefe8007530 legacy=0x7fefe80099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:52.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6789/0 conn(0x7fefe800b2a0 legacy=0x7fefe800d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:52.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.26:6789/0 conn(0x7fefe80038c0 legacy=0x7fefe8005cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:52.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 10 monclient: _add_conns ranks=[4,2,3] 2024-08-19T12:38:52.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7ff00090f640 1 -- 172.21.15.26:0/4254773014 reap_dead start 2024-08-19T12:38:52.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fefe8010510 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:52.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fefe800f010 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:52.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 10 monclient(hunting): picked mon.noname-g con 0x7fefe80179a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:52.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe8012640 con 0x7fefe800f010 2024-08-19T12:38:52.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe80064c0 con 0x7fefe80179a0 2024-08-19T12:38:52.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe8006690 con 0x7fefe8010510 2024-08-19T12:38:52.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7ff00090f640 1 -- 172.21.15.26:0/4254773014 reap_dead start 2024-08-19T12:38:52.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.156+0000 7feff9ffb640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:52.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.158+0000 7feffb7fe640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.26:6790/0 conn(0x7fefe800f010 legacy=0x7fefe800f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:52.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.158+0000 7feffbfff640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.26:6791/0 conn(0x7fefe8010510 legacy=0x7fefe8010950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:52.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:52.158+0000 7ff00090f640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6790/0 conn(0x7fefe80179a0 legacy=0x7fefe8019d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:55.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 10 monclient(hunting): tick 2024-08-19T12:38:55.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:38:55.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 1 monclient(hunting): continuing hunt 2024-08-19T12:38:55.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:38:55.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.26:6791/0 conn(0x7fefe8010510 legacy=0x7fefe8010950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:55.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6790/0 conn(0x7fefe80179a0 legacy=0x7fefe8019d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:55.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.26:6790/0 conn(0x7fefe800f010 legacy=0x7fefe800f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:55.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 10 monclient: _add_conns ranks=[1,0,5,6,4,8,7,2,3] 2024-08-19T12:38:55.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 10 monclient(hunting): picked mon.noname-f con 0x7fefe8006c90 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:55.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fefe8007130 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:55.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 10 monclient(hunting): picked mon.noname-h con 0x7fefe800e100 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:55.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe80068b0 con 0x7fefe8007130 2024-08-19T12:38:55.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe800ed40 con 0x7fefe8006c90 2024-08-19T12:38:55.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 1 -- 172.21.15.26:0/4254773014 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe8005d60 con 0x7fefe800e100 2024-08-19T12:38:55.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.156+0000 7feff9ffb640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:55.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.158+0000 7feffb7fe640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.26:6789/0 conn(0x7fefe8007130 legacy=0x7fefe800d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:55.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.158+0000 7ff00090f640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6791/0 conn(0x7fefe800e100 legacy=0x7fefe800e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:55.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:55.158+0000 7feffbfff640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6789/0 conn(0x7fefe8006c90 legacy=0x7fefe80099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.137+0000 7feffaffd640 1 -- 172.21.15.26:0/4254773014 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2558628210 0 0) 0x7fefe800ed40 con 0x7fefe8006c90 2024-08-19T12:38:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.137+0000 7feffaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.137+0000 7feffaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.137+0000 7feffaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.137+0000 7feffaffd640 10 monclient(hunting): my global_id is 4474 2024-08-19T12:38:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.137+0000 7feffaffd640 1 -- 172.21.15.26:0/4254773014 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefe4003520 con 0x7fefe8006c90 2024-08-19T12:38:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.137+0000 7feffaffd640 1 -- 172.21.15.26:0/4254773014 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 319454549 0 0) 0x7fefe4003520 con 0x7fefe8006c90 2024-08-19T12:38:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.137+0000 7feffaffd640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6791/0 conn(0x7fefe800e100 legacy=0x7fefe800e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.26:6789/0 conn(0x7fefe8007130 legacy=0x7fefe800d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7ff00090f640 1 -- 172.21.15.26:0/4254773014 reap_dead start 2024-08-19T12:38:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 1 monclient: found mon.noname-f 2024-08-19T12:38:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:38:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 1 -- 172.21.15.26:0/4254773014 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe8005fd0 con 0x7fefe8006c90 2024-08-19T12:38:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 1 -- 172.21.15.26:0/4254773014 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fefec004640 con 0x7fefe8006c90 2024-08-19T12:38:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:38:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 10 monclient: dump: 2024-08-19T12:38:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7ff002372640 5 monclient: authenticate success, global_id 4474 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7ff002372640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 1 -- 172.21.15.26:0/4254773014 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fefec0048e0 con 0x7fefe8006c90 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 1 -- 172.21.15.26:0/4254773014 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fefec004dd0 con 0x7fefe8006c90 2024-08-19T12:38:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7feffaffd640 10 monclient: dump: 2024-08-19T12:38:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7ff002372640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.138+0000 7ff002372640 4 set_mon_vals no callback set 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.139+0000 7ff002372640 10 monclient: shutdown 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.139+0000 7ff002372640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.139+0000 7ff002372640 1 -- 172.21.15.26:0/4254773014 >> v1:172.21.15.104:6789/0 conn(0x7fefe8006c90 legacy=0x7fefe80099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.139+0000 7ff002372640 1 -- 172.21.15.26:0/4254773014 shutdown_connections 2024-08-19T12:38:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.139+0000 7ff002372640 1 -- 172.21.15.26:0/4254773014 >> 172.21.15.26:0/4254773014 conn(0x7feffc0a8740 msgr2=0x7feffc0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.139+0000 7ff002372640 1 -- 172.21.15.26:0/4254773014 shutdown_connections 2024-08-19T12:38:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.139+0000 7ff002372640 1 -- 172.21.15.26:0/4254773014 wait complete. 2024-08-19T12:38:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.140+0000 7ff002372640 10 monclient: build_initial_monmap 2024-08-19T12:38:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.140+0000 7ff002372640 10 monclient: monmap: 2024-08-19T12:38:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:57.152 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.140+0000 7ff002372640 1 librados: starting msgr at 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.140+0000 7ff002372640 1 librados: starting objecter 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.140+0000 7ff002372640 1 Processor -- start 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.140+0000 7ff002372640 1 -- start start 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.140+0000 7ff002372640 1 librados: setting wanted keys 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.140+0000 7ff002372640 1 librados: calling monclient init 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.140+0000 7ff002372640 10 monclient: init 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.141+0000 7ff002372640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.141+0000 7ff002372640 10 monclient: _add_conns ranks=[0,6,2,4,8,3,5,1,7] 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.141+0000 7ff002372640 10 monclient(hunting): picked mon.noname-a con 0x7fefe8006c90 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.141+0000 7ff002372640 10 monclient(hunting): picked mon.noname-d con 0x7feffc09f7e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.141+0000 7ff002372640 10 monclient(hunting): picked mon.noname-b con 0x7feffc1cea70 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.141+0000 7ff002372640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc135210 con 0x7fefe8006c90 2024-08-19T12:38:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.141+0000 7ff002372640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc0a18f0 con 0x7feffc1cea70 2024-08-19T12:38:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.141+0000 7ff002372640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc1cee70 con 0x7feffc09f7e0 2024-08-19T12:38:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.141+0000 7feffb7fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7feffc09f7e0 0x7feffc1d3590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:55898/0 (socket says 172.21.15.26:55898) 2024-08-19T12:38:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.141+0000 7feffb7fe640 1 -- 172.21.15.26:0/2191544574 learned_addr learned my addr 172.21.15.26:0/2191544574 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.141+0000 7ff002372640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.141+0000 7ff002372640 10 monclient(hunting): authenticate will time out at 731.228915s 2024-08-19T12:38:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.143+0000 7feffb7fe640 1 -- 172.21.15.26:0/2191544574 >> v1:172.21.15.26:6792/0 conn(0x7feffc09f7e0 legacy=0x7feffc1d3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 client.?.objecter ms_handle_connect 0x7feffc09f7e0 2024-08-19T12:38:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 client.?.objecter ms_handle_connect 0x7fefe8006c90 2024-08-19T12:38:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 client.?.objecter ms_handle_connect 0x7feffc1cea70 2024-08-19T12:38:57.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feffbfff640 1 -- 172.21.15.26:0/2191544574 >> v1:172.21.15.26:6789/0 conn(0x7fefe8006c90 legacy=0x7feffc1ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7ff00090f640 1 -- 172.21.15.26:0/2191544574 >> v1:172.21.15.26:6790/0 conn(0x7feffc1cea70 legacy=0x7feffc1d6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242711075 0 0) 0x7feffc135210 con 0x7fefe8006c90 2024-08-19T12:38:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 monclient(hunting): my global_id is 4491 2024-08-19T12:38:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefe800ed40 con 0x7fefe8006c90 2024-08-19T12:38:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1574859995 0 0) 0x7fefe800ed40 con 0x7fefe8006c90 2024-08-19T12:38:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feffc135210 con 0x7fefe8006c90 2024-08-19T12:38:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fefec004dd0 con 0x7fefe8006c90 2024-08-19T12:38:57.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 monclient(hunting): dump: 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.156 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 617837167 0 0) 0x7feffc135210 con 0x7fefe8006c90 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 >> v1:172.21.15.26:6792/0 conn(0x7feffc09f7e0 legacy=0x7feffc1d3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 >> v1:172.21.15.26:6790/0 conn(0x7feffc1cea70 legacy=0x7feffc1d6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 1 monclient: found mon.a 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc0a5690 con 0x7fefe8006c90 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 monclient: _finish_auth 0 2024-08-19T12:38:57.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7feff8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7ff002372640 5 monclient: authenticate success, global_id 4491 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7ff002372640 10 monclient: _renew_subs 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7ff002372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.144+0000 7ff002372640 1 -- 172.21.15.26:0/2191544574 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feffc0a2bc0 con 0x7fefe8006c90 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.145+0000 7ff002372640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.145+0000 7ff002372640 10 monclient: _renew_subs 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.145+0000 7ff002372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.145+0000 7ff002372640 1 -- 172.21.15.26:0/2191544574 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7feffc1d83e0 con 0x7fefe8006c90 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.145+0000 7ff002372640 1 librados: init done 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.145+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fefec005220 con 0x7fefe8006c90 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.145+0000 7feff8ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:57.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.145+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fefec005710 con 0x7fefe8006c90 2024-08-19T12:38:57.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.145+0000 7feff8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.145+0000 7feffa7fc640 4 set_mon_vals no callback set 2024-08-19T12:38:57.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.145+0000 7feff8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:57.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.145+0000 7feff8ff9640 10 monclient: dump: 2024-08-19T12:38:57.159 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.159 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.159 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.159 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.159 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.159 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.159 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.159 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.146+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fefec0047a0 con 0x7fefe8006c90 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.146+0000 7ff002372640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.146+0000 7ff002372640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.146+0000 7ff002372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:57.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.146+0000 7ff002372640 1 -- 172.21.15.26:0/2191544574 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefdc005080 con 0x7fefe8006c90 2024-08-19T12:38:57.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.146+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7fefec04db50 con 0x7fefe8006c90 2024-08-19T12:38:57.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.146+0000 7feff8ff9640 10 client.4491.objecter ms_dispatch 0x7feffc000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:57.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.146+0000 7feff8ff9640 3 client.4491.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:57.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.146+0000 7feff8ff9640 3 client.4491.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:57.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.146+0000 7feff8ff9640 20 client.4491.objecter dump_active .. 0 homeless 2024-08-19T12:38:57.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.146+0000 7feffb7fe640 10 monclient: get_auth_request con 0x7fefe80442d0 auth_method 0 2024-08-19T12:38:57.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.149+0000 7feff8ff9640 10 client.4491.objecter ms_handle_connect 0x7fefe80442d0 2024-08-19T12:38:57.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.151+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fefec017730 con 0x7fefe8006c90 2024-08-19T12:38:57.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.151+0000 7feff8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:57.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.151+0000 7feff8ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:57.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.409+0000 7ff002372640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:38:57.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.409+0000 7ff002372640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:38:57.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.409+0000 7ff002372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:38:57.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.409+0000 7ff002372640 1 -- 172.21.15.26:0/2191544574 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fefdc005370 con 0x7fefe8006c90 2024-08-19T12:38:57.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.409+0000 7feff8ff9640 1 -- 172.21.15.26:0/2191544574 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2722 (unknown 254746657 0 531612152) 0x7fefec017030 con 0x7fefe8006c90 2024-08-19T12:38:57.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.410+0000 7feff8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:38:57.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.410+0000 7feff8ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:57.411 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":8,"quorum":[0,1,2,3,4,5,6,8],"quorum_names":["a","f","b","g","c","h","d","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:38:57.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.411+0000 7ff002372640 10 librados: watch_flush enter 2024-08-19T12:38:57.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.411+0000 7ff002372640 10 librados: watch_flush exit 2024-08-19T12:38:57.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.411+0000 7ff002372640 20 client.4491.objecter shutdown clearing up homeless session... 2024-08-19T12:38:57.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.411+0000 7ff002372640 10 client.4491.objecter successfully canceled tick 2024-08-19T12:38:57.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.412+0000 7ff002372640 1 -- 172.21.15.26:0/2191544574 >> v1:172.21.15.104:6800/899887506 conn(0x7fefe80442d0 legacy=0x7fefe8046790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.412+0000 7ff002372640 10 monclient: shutdown 2024-08-19T12:38:57.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.412+0000 7ff002372640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:57.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.412+0000 7ff002372640 1 -- 172.21.15.26:0/2191544574 >> v1:172.21.15.26:6789/0 conn(0x7fefe8006c90 legacy=0x7feffc1ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.412+0000 7ff002372640 1 -- 172.21.15.26:0/2191544574 shutdown_connections 2024-08-19T12:38:57.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.412+0000 7ff002372640 1 -- 172.21.15.26:0/2191544574 >> 172.21.15.26:0/2191544574 conn(0x7feffc0a8740 msgr2=0x7feffc132c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:57.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.412+0000 7ff002372640 1 -- 172.21.15.26:0/2191544574 shutdown_connections 2024-08-19T12:38:57.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.412+0000 7ff002372640 1 -- 172.21.15.26:0/2191544574 wait complete. 2024-08-19T12:38:57.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.412+0000 7ff002372640 1 librados: shutdown 2024-08-19T12:38:57.424 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-08-19T12:38:57.424 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-08-19T12:38:57.424 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:38:57.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.522+0000 7f681d155640 10 monclient: get_monmap_and_config 2024-08-19T12:38:57.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.522+0000 7f681d155640 10 monclient: build_initial_monmap 2024-08-19T12:38:57.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.522+0000 7f681d155640 10 monclient: monmap: 2024-08-19T12:38:57.524 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:57.524 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.524 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:57.523480+0000 2024-08-19T12:38:57.524 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:57.523480+0000 2024-08-19T12:38:57.524 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:57.524 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.524 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:57.524 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:57.524 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:57.525 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 1 Processor -- start 2024-08-19T12:38:57.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 1 -- start start 2024-08-19T12:38:57.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 10 monclient: init 2024-08-19T12:38:57.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:57.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 10 monclient: _add_conns ranks=[7,8,2,5,3,1,6,4,0] 2024-08-19T12:38:57.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 10 monclient(hunting): picked mon.noname-i con 0x7f681806e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:57.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 10 monclient(hunting): picked mon.noname-e con 0x7f68180a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:57.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 10 monclient(hunting): picked mon.noname-b con 0x7f68180a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:57.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6818136790 con 0x7f68180a5d50 2024-08-19T12:38:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f681809ddf0 con 0x7f681806e7b0 2024-08-19T12:38:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f681809e0d0 con 0x7f68180a5920 2024-08-19T12:38:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681d155640 10 monclient(hunting): authenticate will time out at 731.610749s 2024-08-19T12:38:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f6816d9d640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f681806e7b0 legacy=0x7f6818136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:38:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681659c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f68180a5920 0x7f681813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:54194/0 (socket says 172.21.15.26:54194) 2024-08-19T12:38:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.523+0000 7f681659c640 1 -- 172.21.15.26:0/1758286607 learned_addr learned my addr 172.21.15.26:0/1758286607 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f681659c640 1 -- 172.21.15.26:0/1758286607 >> v1:172.21.15.26:6793/0 conn(0x7f68180a5920 legacy=0x7f681813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f681759e640 1 -- 172.21.15.26:0/1758286607 >> v1:172.21.15.26:6790/0 conn(0x7f68180a5d50 legacy=0x7f6818141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 1 -- 172.21.15.26:0/1758286607 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839745499 0 0) 0x7f681809e0d0 con 0x7f68180a5920 2024-08-19T12:38:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient(hunting): my global_id is 4589 2024-08-19T12:38:57.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 1 -- 172.21.15.26:0/1758286607 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6800003520 con 0x7f68180a5920 2024-08-19T12:38:57.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 1 -- 172.21.15.26:0/1758286607 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 950881098 0 0) 0x7f6800003520 con 0x7f68180a5920 2024-08-19T12:38:57.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 1 -- 172.21.15.26:0/1758286607 >> v1:172.21.15.104:6792/0 conn(0x7f681806e7b0 legacy=0x7f6818136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:38:57.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 1 -- 172.21.15.26:0/1758286607 >> v1:172.21.15.26:6790/0 conn(0x7f68180a5d50 legacy=0x7f6818141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:38:57.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 1 monclient: found mon.noname-e 2024-08-19T12:38:57.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:57.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:38:57.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 1 -- 172.21.15.26:0/1758286607 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681809e3b0 con 0x7f68180a5920 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient: _finish_auth 0 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 1 -- 172.21.15.26:0/1758286607 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f680c001ae0 con 0x7f68180a5920 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient: dump: 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.535 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f681d155640 5 monclient: authenticate success, global_id 4589 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f681d155640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:57.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 1 -- 172.21.15.26:0/1758286607 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f680c0048a0 con 0x7f68180a5920 2024-08-19T12:38:57.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:57.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 1 -- 172.21.15.26:0/1758286607 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f680c004e10 con 0x7f68180a5920 2024-08-19T12:38:57.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f681d155640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:57.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f681d155640 4 set_mon_vals no callback set 2024-08-19T12:38:57.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:57.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.526+0000 7f6815d9b640 10 monclient: dump: 2024-08-19T12:38:57.537 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.537 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.537 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.537 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.538 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.538 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.538 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.538 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.538 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.538 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.538 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.538 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.538 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.527+0000 7f681d155640 10 monclient: shutdown 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.527+0000 7f681d155640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.527+0000 7f681d155640 1 -- 172.21.15.26:0/1758286607 >> v1:172.21.15.26:6793/0 conn(0x7f68180a5920 legacy=0x7f681813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.527+0000 7f681d155640 1 -- 172.21.15.26:0/1758286607 shutdown_connections 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.527+0000 7f681d155640 1 -- 172.21.15.26:0/1758286607 >> 172.21.15.26:0/1758286607 conn(0x7f68180a8740 msgr2=0x7f68180a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.527+0000 7f681d155640 1 -- 172.21.15.26:0/1758286607 shutdown_connections 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.527+0000 7f681d155640 1 -- 172.21.15.26:0/1758286607 wait complete. 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.527+0000 7f681d155640 10 monclient: build_initial_monmap 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.527+0000 7f681d155640 10 monclient: monmap: 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:57.539 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:57.540 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.527+0000 7f681d155640 1 librados: starting msgr at 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.527+0000 7f681d155640 1 librados: starting objecter 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 1 Processor -- start 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 1 -- start start 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 1 librados: setting wanted keys 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 1 librados: calling monclient init 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 10 monclient: init 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 10 monclient: _add_conns ranks=[8,4,7,1,0,6,3,5,2] 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 10 monclient(hunting): picked mon.noname-e con 0x7f681806e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 10 monclient(hunting): picked mon.noname-c con 0x7f68180a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 10 monclient(hunting): picked mon.noname-i con 0x7f68180a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:57.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68180a0ed0 con 0x7f68180a5920 2024-08-19T12:38:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f681809f9a0 con 0x7f68180a5d50 2024-08-19T12:38:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68180a0630 con 0x7f681806e7b0 2024-08-19T12:38:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681d155640 10 monclient(hunting): authenticate will time out at 731.615782s 2024-08-19T12:38:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681759e640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f68180a5d50 legacy=0x7f68181d6290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:38:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681659c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f68180a5920 0x7f68181328f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:38904/0 (socket says 172.21.15.26:38904) 2024-08-19T12:38:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f681659c640 1 -- 172.21.15.26:0/1320642875 learned_addr learned my addr 172.21.15.26:0/1320642875 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:57.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.528+0000 7f6816d9d640 1 --1- 172.21.15.26:0/1320642875 >> v1:172.21.15.26:6793/0 conn(0x7f681806e7b0 0x7f68181321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:54210/0 (socket says 172.21.15.26:54210) 2024-08-19T12:38:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.530+0000 7f6816d9d640 1 -- 172.21.15.26:0/1320642875 >> v1:172.21.15.26:6793/0 conn(0x7f681806e7b0 legacy=0x7f68181321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 client.?.objecter ms_handle_connect 0x7f681806e7b0 2024-08-19T12:38:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 client.?.objecter ms_handle_connect 0x7f68180a5920 2024-08-19T12:38:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f681659c640 1 -- 172.21.15.26:0/1320642875 >> v1:172.21.15.26:6791/0 conn(0x7f68180a5920 legacy=0x7f68181328f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 330278543 0 0) 0x7f68180a0630 con 0x7f681806e7b0 2024-08-19T12:38:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): my global_id is 4598 2024-08-19T12:38:57.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6804004eb0 con 0x7f681806e7b0 2024-08-19T12:38:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4284515859 0 0) 0x7f68180a0ed0 con 0x7f68180a5920 2024-08-19T12:38:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): my global_id is 4522 2024-08-19T12:38:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68180a0630 con 0x7f68180a5920 2024-08-19T12:38:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3616303895 0 0) 0x7f6804004eb0 con 0x7f681806e7b0 2024-08-19T12:38:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68180a0ed0 con 0x7f681806e7b0 2024-08-19T12:38:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2173684751 0 0) 0x7f68180a0630 con 0x7f68180a5920 2024-08-19T12:38:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6804004eb0 con 0x7f68180a5920 2024-08-19T12:38:57.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6808004590 con 0x7f681806e7b0 2024-08-19T12:38:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:38:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): dump: 2024-08-19T12:38:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.551 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f680c004d90 con 0x7f68180a5920 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient(hunting): dump: 2024-08-19T12:38:57.552 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.553 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.554 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1955433919 0 0) 0x7f68180a0ed0 con 0x7f681806e7b0 2024-08-19T12:38:57.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 >> v1:172.21.15.104:6792/0 conn(0x7f68180a5d50 legacy=0x7f68181d6290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:38:57.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 >> v1:172.21.15.26:6791/0 conn(0x7f68180a5920 legacy=0x7f68181328f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:57.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 monclient: found mon.e 2024-08-19T12:38:57.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:57.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:57.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681809f6d0 con 0x7f681806e7b0 2024-08-19T12:38:57.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f681d155640 5 monclient: authenticate success, global_id 4598 2024-08-19T12:38:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f681d155640 10 monclient: _renew_subs 2024-08-19T12:38:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f681d155640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f681d155640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f68180a55d0 con 0x7f681806e7b0 2024-08-19T12:38:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f681d155640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f681d155640 10 monclient: _renew_subs 2024-08-19T12:38:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f681d155640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:38:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f681d155640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f68181331e0 con 0x7f681806e7b0 2024-08-19T12:38:57.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f681d155640 1 librados: init done 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6808004830 con 0x7f681806e7b0 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6808004da0 con 0x7f681806e7b0 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f681559a640 4 set_mon_vals no callback set 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.531+0000 7f67ff7fe640 10 monclient: dump: 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.556 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.557 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.532+0000 7f681d155640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.532+0000 7f681d155640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f67f80065c0 con 0x7f67f8002860 2024-08-19T12:38:57.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.532+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6808004f40 con 0x7f681806e7b0 2024-08-19T12:38:57.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.533+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f680804da40 con 0x7f681806e7b0 2024-08-19T12:38:57.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.533+0000 7f67ff7fe640 10 client.4598.objecter ms_dispatch 0x7f6818000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:57.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.533+0000 7f67ff7fe640 3 client.4598.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:57.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.533+0000 7f67ff7fe640 3 client.4598.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:57.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.533+0000 7f67ff7fe640 20 client.4598.objecter dump_active .. 0 homeless 2024-08-19T12:38:57.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.533+0000 7f681759e640 10 monclient: get_auth_request con 0x7f68040413c0 auth_method 0 2024-08-19T12:38:57.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.534+0000 7f681659c640 1 -- 172.21.15.26:0/1320642875 >> v1:172.21.15.26:6789/0 conn(0x7f67f8002860 legacy=0x7f67f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.534+0000 7f67ff7fe640 10 client.4598.objecter ms_handle_connect 0x7f67f8002860 2024-08-19T12:38:57.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.534+0000 7f67ff7fe640 10 client.4598.objecter resend_mon_ops 2024-08-19T12:38:57.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.534+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1459835092 0 0) 0x7f67f80065c0 con 0x7f67f8002860 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.534+0000 7f67ff7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.534+0000 7f67ff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.534+0000 7f67ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.534+0000 7f67ff7fe640 10 monclient(hunting): my global_id is 4500 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.534+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68180a0ed0 con 0x7f67f8002860 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 73528938 0 0) 0x7f68180a0ed0 con 0x7f67f8002860 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67f80065c0 con 0x7f67f8002860 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f680c0048a0 con 0x7f67f8002860 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 10 monclient: dump: 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.559 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.560 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3715233077 0 0) 0x7f67f80065c0 con 0x7f67f8002860 2024-08-19T12:38:57.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67f800a160 con 0x7f67f8002860 2024-08-19T12:38:57.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 10 client.4598.objecter ms_handle_connect 0x7f68040413c0 2024-08-19T12:38:57.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f680c005740 con 0x7f67f8002860 2024-08-19T12:38:57.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:57.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:57.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 >> v1:172.21.15.26:6789/0 conn(0x7f67f8002860 legacy=0x7f67f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.535+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 >> v1:172.21.15.26:6789/0 conn(0x7f67f8002860 legacy=0x7f67f8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:57.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.541+0000 7f681d155640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:57.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.541+0000 7f681d155640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f67f8009e50 con 0x7f67f8009110 2024-08-19T12:38:57.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f681659c640 1 -- 172.21.15.26:0/1320642875 >> v1:172.21.15.26:6789/0 conn(0x7f67f8009110 legacy=0x7f67f80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 10 client.4598.objecter ms_handle_connect 0x7f67f8009110 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 10 client.4598.objecter resend_mon_ops 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1030193319 0 0) 0x7f67f8009e50 con 0x7f67f8009110 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 10 monclient(hunting): my global_id is 4509 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67f80065c0 con 0x7f67f8009110 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2666683767 0 0) 0x7f67f80065c0 con 0x7f67f8009110 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67f8009e50 con 0x7f67f8009110 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f680c005e60 con 0x7f67f8009110 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 10 monclient: dump: 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.564 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.564 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.564 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.564 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 627349857 0 0) 0x7f67f8009e50 con 0x7f67f8009110 2024-08-19T12:38:57.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.544+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f67f800c3a0 con 0x7f67f8009110 2024-08-19T12:38:57.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7700 (unknown 0 0 2685286340) 0x7f680c00ae20 con 0x7f67f8009110 2024-08-19T12:38:57.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f67ff7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:57.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f67ff7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:57.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 >> v1:172.21.15.26:6789/0 conn(0x7f67f8009110 legacy=0x7f67f80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f67ff7fe640 1 -- 172.21.15.26:0/1320642875 >> v1:172.21.15.26:6789/0 conn(0x7f67f8009110 legacy=0x7f67f80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 8, 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 50586, 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:57.565 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:57.566 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:57.566 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:57.566 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.566 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:38:57.566 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:57.566 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:57.566 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:57.566 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:57.566 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:57.566 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:57.566 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:57.566 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:57.567 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:57.568 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:57.569 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:57.569 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:57.569 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:57.569 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:57.569 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:57.569 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:57.569 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:57.569 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:57.569 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:57.569 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:57.569 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:57.569 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:57.570 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:57.570 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:57.570 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:57.570 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:57.570 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:57.570 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:57.570 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:57.570 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:57.570 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:57.570 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:57.570 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:57.571 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:57.571 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.571 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:57.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.571 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:57.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.571 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:57.571 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:57.571 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.572 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.573 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:57.573 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:57.573 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.573 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:57.573 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.574 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:57.574 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:57.574 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.574 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.574 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.574 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:57.574 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.575 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.576 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:57.579 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:57.580 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.581 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.581 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.581 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:57.581 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:57.581 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.581 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.581 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.582 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:57.582 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:57.582 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.582 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.582 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.582 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:57.582 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.582 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.582 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.582 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:57.583 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:57.583 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.583 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.583 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.583 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:57.583 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:57.583 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.584 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.584 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.584 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:57.584 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.584 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.584 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.584 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:57.584 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:57.584 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.584 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.585 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.586 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:57.587 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:57.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 10 librados: watch_flush enter 2024-08-19T12:38:57.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 10 librados: watch_flush exit 2024-08-19T12:38:57.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 20 client.4598.objecter shutdown clearing up homeless session... 2024-08-19T12:38:57.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 10 client.4598.objecter successfully canceled tick 2024-08-19T12:38:57.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 1 -- 172.21.15.26:0/1320642875 >> v1:172.21.15.104:6800/899887506 conn(0x7f68040413c0 legacy=0x7f6804043880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 10 monclient: shutdown 2024-08-19T12:38:57.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:57.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 1 -- 172.21.15.26:0/1320642875 >> v1:172.21.15.26:6793/0 conn(0x7f681806e7b0 legacy=0x7f68181321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681759e640 1 -- 172.21.15.26:0/1320642875 reap_dead start 2024-08-19T12:38:57.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681759e640 1 -- 172.21.15.26:0/1320642875 reap_dead start 2024-08-19T12:38:57.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 1 -- 172.21.15.26:0/1320642875 shutdown_connections 2024-08-19T12:38:57.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 1 -- 172.21.15.26:0/1320642875 >> 172.21.15.26:0/1320642875 conn(0x7f68180a8740 msgr2=0x7f681812c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:57.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 1 -- 172.21.15.26:0/1320642875 shutdown_connections 2024-08-19T12:38:57.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 1 -- 172.21.15.26:0/1320642875 wait complete. 2024-08-19T12:38:57.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.545+0000 7f681d155640 1 librados: shutdown 2024-08-19T12:38:57.590 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:38:57.590 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-08-19T12:38:57.590 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:38:57.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.689+0000 7fe04b015640 10 monclient: get_monmap_and_config 2024-08-19T12:38:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.689+0000 7fe04b015640 10 monclient: build_initial_monmap 2024-08-19T12:38:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.689+0000 7fe04b015640 10 monclient: monmap: 2024-08-19T12:38:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:57.689975+0000 2024-08-19T12:38:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:57.689975+0000 2024-08-19T12:38:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:57.690 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:57.691 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.689+0000 7fe04b015640 1 Processor -- start 2024-08-19T12:38:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.689+0000 7fe04b015640 1 -- start start 2024-08-19T12:38:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.689+0000 7fe04b015640 10 monclient: init 2024-08-19T12:38:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.690+0000 7fe04b015640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.690+0000 7fe04b015640 10 monclient: _add_conns ranks=[6,3,5,2,4,8,0,1,7] 2024-08-19T12:38:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.690+0000 7fe04b015640 10 monclient(hunting): picked mon.noname-d con 0x7fe044073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.690+0000 7fe04b015640 10 monclient(hunting): picked mon.noname-g con 0x7fe0440a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.690+0000 7fe04b015640 10 monclient(hunting): picked mon.noname-h con 0x7fe04413f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:57.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.690+0000 7fe04b015640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe04409dae0 con 0x7fe0440a57a0 2024-08-19T12:38:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.690+0000 7fe04b015640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe044136910 con 0x7fe04413f2e0 2024-08-19T12:38:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.690+0000 7fe04b015640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe04409dd50 con 0x7fe044073470 2024-08-19T12:38:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.690+0000 7fe04b015640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.690+0000 7fe04b015640 10 monclient(hunting): authenticate will time out at 731.777400s 2024-08-19T12:38:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.690+0000 7fe048db1640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fe044073470 0x7fe044136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:55902/0 (socket says 172.21.15.26:55902) 2024-08-19T12:38:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.690+0000 7fe048db1640 1 -- 172.21.15.26:0/1523221101 learned_addr learned my addr 172.21.15.26:0/1523221101 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe048db1640 1 -- 172.21.15.26:0/1523221101 >> v1:172.21.15.26:6792/0 conn(0x7fe044073470 legacy=0x7fe044136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe043fff640 1 -- 172.21.15.26:0/1523221101 >> v1:172.21.15.104:6790/0 conn(0x7fe0440a57a0 legacy=0x7fe0440a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0495b2640 1 -- 172.21.15.26:0/1523221101 >> v1:172.21.15.104:6791/0 conn(0x7fe04413f2e0 legacy=0x7fe0441416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1681269313 0 0) 0x7fe04409dd50 con 0x7fe044073470 2024-08-19T12:38:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 10 monclient(hunting): my global_id is 4569 2024-08-19T12:38:57.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe030003520 con 0x7fe044073470 2024-08-19T12:38:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879659860 0 0) 0x7fe04409dae0 con 0x7fe0440a57a0 2024-08-19T12:38:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 10 monclient(hunting): my global_id is 4593 2024-08-19T12:38:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe04409dd50 con 0x7fe0440a57a0 2024-08-19T12:38:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1354351847 0 0) 0x7fe044136910 con 0x7fe04413f2e0 2024-08-19T12:38:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 10 monclient(hunting): my global_id is 4460 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.692+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe04409dae0 con 0x7fe04413f2e0 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3279745554 0 0) 0x7fe030003520 con 0x7fe044073470 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 >> v1:172.21.15.104:6791/0 conn(0x7fe04413f2e0 legacy=0x7fe0441416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 >> v1:172.21.15.104:6790/0 conn(0x7fe0440a57a0 legacy=0x7fe0440a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 1 monclient: found mon.noname-d 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe04409e030 con 0x7fe044073470 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:57.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe034001e60 con 0x7fe044073470 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe04b015640 5 monclient: authenticate success, global_id 4569 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe04b015640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 10 monclient: dump: 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.700 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe04b015640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe034004c20 con 0x7fe044073470 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 1 -- 172.21.15.26:0/1523221101 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe034005190 con 0x7fe044073470 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe0437fe640 10 monclient: dump: 2024-08-19T12:38:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe04b015640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe04b015640 4 set_mon_vals no callback set 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe04b015640 10 monclient: shutdown 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe04b015640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe04b015640 1 -- 172.21.15.26:0/1523221101 >> v1:172.21.15.26:6792/0 conn(0x7fe044073470 legacy=0x7fe044136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe04b015640 1 -- 172.21.15.26:0/1523221101 shutdown_connections 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe04b015640 1 -- 172.21.15.26:0/1523221101 >> 172.21.15.26:0/1523221101 conn(0x7fe0440abff0 msgr2=0x7fe0440ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe04b015640 1 -- 172.21.15.26:0/1523221101 shutdown_connections 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.693+0000 7fe04b015640 1 -- 172.21.15.26:0/1523221101 wait complete. 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.694+0000 7fe04b015640 10 monclient: build_initial_monmap 2024-08-19T12:38:57.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.694+0000 7fe04b015640 10 monclient: monmap: 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:57.704 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.694+0000 7fe04b015640 1 librados: starting msgr at 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.694+0000 7fe04b015640 1 librados: starting objecter 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.694+0000 7fe04b015640 1 Processor -- start 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.694+0000 7fe04b015640 1 -- start start 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.694+0000 7fe04b015640 1 librados: setting wanted keys 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.694+0000 7fe04b015640 1 librados: calling monclient init 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.694+0000 7fe04b015640 10 monclient: init 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe04b015640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe04b015640 10 monclient: _add_conns ranks=[5,6,7,1,4,3,0,2,8] 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe04b015640 10 monclient(hunting): picked mon.noname-h con 0x7fe044073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe04b015640 10 monclient(hunting): picked mon.noname-d con 0x7fe0440a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:57.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe04b015640 10 monclient(hunting): picked mon.noname-i con 0x7fe04413f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:57.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe04b015640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe044136e00 con 0x7fe044073470 2024-08-19T12:38:57.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe04b015640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe04413f0b0 con 0x7fe0440a57a0 2024-08-19T12:38:57.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe04b015640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe044136b20 con 0x7fe04413f2e0 2024-08-19T12:38:57.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe04b015640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:57.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe04b015640 10 monclient(hunting): authenticate will time out at 731.782382s 2024-08-19T12:38:57.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe0495b2640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7fe04413f2e0 legacy=0x7fe0440a2c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:38:57.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe048db1640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7fe044073470 0x7fe0440a1e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:44380/0 (socket says 172.21.15.26:44380) 2024-08-19T12:38:57.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe043fff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fe0440a57a0 0x7fe0440a2520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:55912/0 (socket says 172.21.15.26:55912) 2024-08-19T12:38:57.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.695+0000 7fe048db1640 1 -- 172.21.15.26:0/3186263813 learned_addr learned my addr 172.21.15.26:0/3186263813 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:57.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe043fff640 1 -- 172.21.15.26:0/3186263813 >> v1:172.21.15.26:6792/0 conn(0x7fe0440a57a0 legacy=0x7fe0440a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe048db1640 1 -- 172.21.15.26:0/3186263813 >> v1:172.21.15.104:6791/0 conn(0x7fe044073470 legacy=0x7fe0440a1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 10 client.?.objecter ms_handle_connect 0x7fe0440a57a0 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 10 client.?.objecter ms_handle_connect 0x7fe044073470 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3186626351 0 0) 0x7fe04413f0b0 con 0x7fe0440a57a0 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 10 monclient(hunting): my global_id is 4578 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe024004eb0 con 0x7fe0440a57a0 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2232303524 0 0) 0x7fe044136e00 con 0x7fe044073470 2024-08-19T12:38:57.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 10 monclient(hunting): my global_id is 4469 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe04413f0b0 con 0x7fe044073470 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.697+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2674512361 0 0) 0x7fe024004eb0 con 0x7fe0440a57a0 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe044136e00 con 0x7fe0440a57a0 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe02c003610 con 0x7fe0440a57a0 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient(hunting): dump: 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.713 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2988349148 0 0) 0x7fe04413f0b0 con 0x7fe044073470 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe024004eb0 con 0x7fe044073470 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe034005110 con 0x7fe044073470 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient(hunting): dump: 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.715 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1751807351 0 0) 0x7fe044136e00 con 0x7fe0440a57a0 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 >> v1:172.21.15.104:6792/0 conn(0x7fe04413f2e0 legacy=0x7fe0440a2c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 >> v1:172.21.15.104:6791/0 conn(0x7fe044073470 legacy=0x7fe0440a1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient: _finish_hunting 0 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 1 monclient: found mon.d 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0441da190 con 0x7fe0440a57a0 2024-08-19T12:38:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient: _finish_auth 0 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient: _check_auth_tickets 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe04b015640 5 monclient: authenticate success, global_id 4578 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe04b015640 10 monclient: _renew_subs 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe04b015640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe04b015640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe0441d7160 con 0x7fe0440a57a0 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe04b015640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe04b015640 10 monclient: _renew_subs 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe04b015640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe04b015640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe0441d75d0 con 0x7fe0440a57a0 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe04b015640 1 librados: init done 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe02c003b60 con 0x7fe0440a57a0 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe02c004090 con 0x7fe0440a57a0 2024-08-19T12:38:57.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe042ffd640 4 set_mon_vals no callback set 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe0417fa640 10 monclient: dump: 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.718 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.719 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.698+0000 7fe04b015640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.699+0000 7fe04b015640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe0280065c0 con 0x7fe028002860 2024-08-19T12:38:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.699+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe02c004230 con 0x7fe0440a57a0 2024-08-19T12:38:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.699+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7fe02c04dac0 con 0x7fe0440a57a0 2024-08-19T12:38:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.699+0000 7fe0417fa640 10 client.4578.objecter ms_dispatch 0x7fe044000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.699+0000 7fe0417fa640 3 client.4578.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:57.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.699+0000 7fe0417fa640 3 client.4578.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.699+0000 7fe0417fa640 20 client.4578.objecter dump_active .. 0 homeless 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.699+0000 7fe0495b2640 10 monclient: get_auth_request con 0x7fe024041540 auth_method 0 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 10 client.4578.objecter ms_handle_connect 0x7fe028002860 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe048db1640 1 -- 172.21.15.26:0/3186263813 >> v1:172.21.15.26:6790/0 conn(0x7fe028002860 legacy=0x7fe028004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 10 client.4578.objecter resend_mon_ops 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2333868928 0 0) 0x7fe0280065c0 con 0x7fe028002860 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 10 monclient(hunting): my global_id is 4583 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe044136e00 con 0x7fe028002860 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4032719719 0 0) 0x7fe044136e00 con 0x7fe028002860 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0280065c0 con 0x7fe028002860 2024-08-19T12:38:57.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe0340020b0 con 0x7fe028002860 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 10 monclient: dump: 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.721 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2855473638 0 0) 0x7fe0280065c0 con 0x7fe028002860 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe02800a160 con 0x7fe028002860 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.701+0000 7fe0417fa640 10 client.4578.objecter ms_handle_connect 0x7fe024041540 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.702+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe034005810 con 0x7fe028002860 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.702+0000 7fe0417fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.702+0000 7fe0417fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.702+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 >> v1:172.21.15.26:6790/0 conn(0x7fe028002860 legacy=0x7fe028004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.702+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 >> v1:172.21.15.26:6790/0 conn(0x7fe028002860 legacy=0x7fe028004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:57.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.708+0000 7fe04b015640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.708+0000 7fe04b015640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe028009e50 con 0x7fe028009110 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe048db1640 1 -- 172.21.15.26:0/3186263813 >> v1:172.21.15.26:6790/0 conn(0x7fe028009110 legacy=0x7fe0280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 10 client.4578.objecter ms_handle_connect 0x7fe028009110 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 10 client.4578.objecter resend_mon_ops 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 210694275 0 0) 0x7fe028009e50 con 0x7fe028009110 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 10 monclient(hunting): my global_id is 4592 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0280065c0 con 0x7fe028009110 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2857673964 0 0) 0x7fe0280065c0 con 0x7fe028009110 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe028009e50 con 0x7fe028009110 2024-08-19T12:38:57.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe034005ef0 con 0x7fe028009110 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 10 monclient: dump: 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.724 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.725 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4090724099 0 0) 0x7fe028009e50 con 0x7fe028009110 2024-08-19T12:38:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.710+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe02800c3f0 con 0x7fe028009110 2024-08-19T12:38:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7531 (unknown 0 0 3288792840) 0x7fe0340081f0 con 0x7fe028009110 2024-08-19T12:38:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe0417fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe0417fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 >> v1:172.21.15.26:6790/0 conn(0x7fe028009110 legacy=0x7fe0280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe0417fa640 1 -- 172.21.15.26:0/3186263813 >> v1:172.21.15.26:6790/0 conn(0x7fe028009110 legacy=0x7fe0280095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:57.725 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:57.725 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 8, 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 50766, 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:38:57.726 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.727 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:57.728 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:57.729 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:57.730 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.731 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:57.733 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.734 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:57.736 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:57.739 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.740 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:57.743 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:57.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.744 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.744 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:57.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.744 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:57.744 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:57.744 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:57.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.744 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.744 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:57.744 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 10 librados: watch_flush enter 2024-08-19T12:38:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 10 librados: watch_flush exit 2024-08-19T12:38:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 20 client.4578.objecter shutdown clearing up homeless session... 2024-08-19T12:38:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 10 client.4578.objecter successfully canceled tick 2024-08-19T12:38:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 1 -- 172.21.15.26:0/3186263813 >> v1:172.21.15.104:6800/899887506 conn(0x7fe024041540 legacy=0x7fe024043a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 10 monclient: shutdown 2024-08-19T12:38:57.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 1 -- 172.21.15.26:0/3186263813 >> v1:172.21.15.26:6792/0 conn(0x7fe0440a57a0 legacy=0x7fe0440a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe0495b2640 1 -- 172.21.15.26:0/3186263813 reap_dead start 2024-08-19T12:38:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe0495b2640 1 -- 172.21.15.26:0/3186263813 reap_dead start 2024-08-19T12:38:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 1 -- 172.21.15.26:0/3186263813 shutdown_connections 2024-08-19T12:38:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 1 -- 172.21.15.26:0/3186263813 >> 172.21.15.26:0/3186263813 conn(0x7fe0440abff0 msgr2=0x7fe0441338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 1 -- 172.21.15.26:0/3186263813 shutdown_connections 2024-08-19T12:38:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 1 -- 172.21.15.26:0/3186263813 wait complete. 2024-08-19T12:38:57.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.711+0000 7fe04b015640 1 librados: shutdown 2024-08-19T12:38:57.748 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:38:57.748 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-08-19T12:38:57.748 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:38:57.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.846+0000 7fd6c4955640 10 monclient: get_monmap_and_config 2024-08-19T12:38:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.846+0000 7fd6c4955640 10 monclient: build_initial_monmap 2024-08-19T12:38:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.846+0000 7fd6c4955640 10 monclient: monmap: 2024-08-19T12:38:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:57.847766+0000 2024-08-19T12:38:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:57.847766+0000 2024-08-19T12:38:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:57.849 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:57.849 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.847+0000 7fd6c4955640 1 Processor -- start 2024-08-19T12:38:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.847+0000 7fd6c4955640 1 -- start start 2024-08-19T12:38:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.847+0000 7fd6c4955640 10 monclient: init 2024-08-19T12:38:57.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.848+0000 7fd6c4955640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.848+0000 7fd6c4955640 10 monclient: _add_conns ranks=[5,1,6,8,0,4,2,3,7] 2024-08-19T12:38:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.848+0000 7fd6c4955640 10 monclient(hunting): picked mon.noname-h con 0x7fd6c0073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.848+0000 7fd6c4955640 10 monclient(hunting): picked mon.noname-f con 0x7fd6c00a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.848+0000 7fd6c4955640 10 monclient(hunting): picked mon.noname-d con 0x7fd6c013f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.848+0000 7fd6c4955640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c009dae0 con 0x7fd6c00a57a0 2024-08-19T12:38:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.848+0000 7fd6c4955640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c0136910 con 0x7fd6c0073470 2024-08-19T12:38:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.848+0000 7fd6c4955640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c009dd50 con 0x7fd6c013f2e0 2024-08-19T12:38:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.848+0000 7fd6c4955640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.848+0000 7fd6c4955640 10 monclient(hunting): authenticate will time out at 731.935337s 2024-08-19T12:38:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.848+0000 7fd6bed9d640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fd6c013f2e0 0x7fd6c01416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:55926/0 (socket says 172.21.15.26:55926) 2024-08-19T12:38:57.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.848+0000 7fd6bed9d640 1 -- 172.21.15.26:0/1076034786 learned_addr learned my addr 172.21.15.26:0/1076034786 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:57.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bed9d640 1 -- 172.21.15.26:0/1076034786 >> v1:172.21.15.26:6792/0 conn(0x7fd6c013f2e0 legacy=0x7fd6c01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bdd9b640 1 -- 172.21.15.26:0/1076034786 >> v1:172.21.15.104:6789/0 conn(0x7fd6c00a57a0 legacy=0x7fd6c00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6be59c640 1 -- 172.21.15.26:0/1076034786 >> v1:172.21.15.104:6791/0 conn(0x7fd6c0073470 legacy=0x7fd6c0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3617270742 0 0) 0x7fd6c009dd50 con 0x7fd6c013f2e0 2024-08-19T12:38:57.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 10 monclient(hunting): my global_id is 4587 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6ac003520 con 0x7fd6c013f2e0 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4187577634 0 0) 0x7fd6c009dae0 con 0x7fd6c00a57a0 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 10 monclient(hunting): my global_id is 4483 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6c009dd50 con 0x7fd6c00a57a0 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2580345658 0 0) 0x7fd6c0136910 con 0x7fd6c0073470 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 10 monclient(hunting): my global_id is 4478 2024-08-19T12:38:57.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.850+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6c009dae0 con 0x7fd6c0073470 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2473192126 0 0) 0x7fd6ac003520 con 0x7fd6c013f2e0 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 >> v1:172.21.15.104:6791/0 conn(0x7fd6c0073470 legacy=0x7fd6c0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 >> v1:172.21.15.104:6789/0 conn(0x7fd6c00a57a0 legacy=0x7fd6c00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 10 monclient: _finish_hunting 0 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 1 monclient: found mon.noname-d 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c009e030 con 0x7fd6c013f2e0 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 10 monclient: _finish_auth 0 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 10 monclient: _check_auth_tickets 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd6b8001e60 con 0x7fd6c013f2e0 2024-08-19T12:38:57.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6c4955640 5 monclient: authenticate success, global_id 4587 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6c4955640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 10 monclient: dump: 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.858 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.859 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6c4955640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd6b8004c20 con 0x7fd6c013f2e0 2024-08-19T12:38:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6c4955640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 1 -- 172.21.15.26:0/1076034786 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd6b8005190 con 0x7fd6c013f2e0 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6c4955640 4 set_mon_vals no callback set 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6bd59a640 10 monclient: dump: 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.860 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6c4955640 10 monclient: shutdown 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6c4955640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6c4955640 1 -- 172.21.15.26:0/1076034786 >> v1:172.21.15.26:6792/0 conn(0x7fd6c013f2e0 legacy=0x7fd6c01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6c4955640 1 -- 172.21.15.26:0/1076034786 shutdown_connections 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6c4955640 1 -- 172.21.15.26:0/1076034786 >> 172.21.15.26:0/1076034786 conn(0x7fd6c00abff0 msgr2=0x7fd6c00ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6c4955640 1 -- 172.21.15.26:0/1076034786 shutdown_connections 2024-08-19T12:38:57.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.851+0000 7fd6c4955640 1 -- 172.21.15.26:0/1076034786 wait complete. 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.852+0000 7fd6c4955640 10 monclient: build_initial_monmap 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.852+0000 7fd6c4955640 10 monclient: monmap: 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:57.862 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.852+0000 7fd6c4955640 1 librados: starting msgr at 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.852+0000 7fd6c4955640 1 librados: starting objecter 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.852+0000 7fd6c4955640 1 Processor -- start 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.852+0000 7fd6c4955640 1 -- start start 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.852+0000 7fd6c4955640 1 librados: setting wanted keys 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.852+0000 7fd6c4955640 1 librados: calling monclient init 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.852+0000 7fd6c4955640 10 monclient: init 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.853+0000 7fd6c4955640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:57.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.853+0000 7fd6c4955640 10 monclient: _add_conns ranks=[1,4,3,6,8,7,0,2,5] 2024-08-19T12:38:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.853+0000 7fd6c4955640 10 monclient(hunting): picked mon.noname-f con 0x7fd6c0073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.853+0000 7fd6c4955640 10 monclient(hunting): picked mon.noname-c con 0x7fd6c00a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.853+0000 7fd6c4955640 10 monclient(hunting): picked mon.noname-g con 0x7fd6c013f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.853+0000 7fd6c4955640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c0142b10 con 0x7fd6c0073470 2024-08-19T12:38:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.853+0000 7fd6c4955640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c012f790 con 0x7fd6c013f2e0 2024-08-19T12:38:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.853+0000 7fd6c4955640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c012ec50 con 0x7fd6c00a57a0 2024-08-19T12:38:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.853+0000 7fd6c4955640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.853+0000 7fd6c4955640 10 monclient(hunting): authenticate will time out at 731.940437s 2024-08-19T12:38:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.853+0000 7fd6bdd9b640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fd6c00a57a0 0x7fd6c012e540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:38906/0 (socket says 172.21.15.26:38906) 2024-08-19T12:38:57.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.853+0000 7fd6bdd9b640 1 -- 172.21.15.26:0/3414754818 learned_addr learned my addr 172.21.15.26:0/3414754818 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6bdd9b640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.26:6791/0 conn(0x7fd6c00a57a0 legacy=0x7fd6c012e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 client.?.objecter ms_handle_connect 0x7fd6c00a57a0 2024-08-19T12:38:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 client.?.objecter ms_handle_connect 0x7fd6c0073470 2024-08-19T12:38:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 client.?.objecter ms_handle_connect 0x7fd6c013f2e0 2024-08-19T12:38:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6be59c640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.104:6789/0 conn(0x7fd6c0073470 legacy=0x7fd6c012de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6bed9d640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.104:6790/0 conn(0x7fd6c013f2e0 legacy=0x7fd6c01d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4284468052 0 0) 0x7fd6c012ec50 con 0x7fd6c00a57a0 2024-08-19T12:38:57.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 monclient(hunting): my global_id is 4531 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6a0004ee0 con 0x7fd6c00a57a0 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1987618363 0 0) 0x7fd6c0142b10 con 0x7fd6c0073470 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 monclient(hunting): my global_id is 4492 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6c012ec50 con 0x7fd6c0073470 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2529774542 0 0) 0x7fd6c012f790 con 0x7fd6c013f2e0 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 10 monclient(hunting): my global_id is 4602 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.855+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6c0142b10 con 0x7fd6c013f2e0 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3311666850 0 0) 0x7fd6a0004ee0 con 0x7fd6c00a57a0 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6c012f790 con 0x7fd6c00a57a0 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd6a8003950 con 0x7fd6c00a57a0 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3472929821 0 0) 0x7fd6c012ec50 con 0x7fd6c0073470 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6a0004ee0 con 0x7fd6c0073470 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd6b0001350 con 0x7fd6c0073470 2024-08-19T12:38:57.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.875 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1120757890 0 0) 0x7fd6c0142b10 con 0x7fd6c013f2e0 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6c012ec50 con 0x7fd6c013f2e0 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd6b80020b0 con 0x7fd6c013f2e0 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient(hunting): dump: 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.876 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3540084315 0 0) 0x7fd6c012f790 con 0x7fd6c00a57a0 2024-08-19T12:38:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.104:6790/0 conn(0x7fd6c013f2e0 legacy=0x7fd6c01d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.104:6789/0 conn(0x7fd6c0073470 legacy=0x7fd6c012de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:38:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 monclient: found mon.c 2024-08-19T12:38:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c0131810 con 0x7fd6c00a57a0 2024-08-19T12:38:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient: _finish_auth 0 2024-08-19T12:38:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:38:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6c4955640 5 monclient: authenticate success, global_id 4531 2024-08-19T12:38:57.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6c4955640 10 monclient: _renew_subs 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6c4955640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6c4955640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c0131af0 con 0x7fd6c00a57a0 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6c4955640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6c4955640 10 monclient: _renew_subs 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6c4955640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6c4955640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fd6c01302f0 con 0x7fd6c00a57a0 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6c4955640 1 librados: init done 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd6a8001bd0 con 0x7fd6c00a57a0 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd6a8004160 con 0x7fd6c00a57a0 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6bcd99640 4 set_mon_vals no callback set 2024-08-19T12:38:57.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.856+0000 7fd6b6ffd640 10 monclient: dump: 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.857+0000 7fd6c4955640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.857+0000 7fd6c4955640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd6a40065c0 con 0x7fd6a4002860 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.857+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd6a8011a80 con 0x7fd6c00a57a0 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.857+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7fd6a804dba0 con 0x7fd6c00a57a0 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.857+0000 7fd6b6ffd640 10 client.4531.objecter ms_dispatch 0x7fd6c0000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.857+0000 7fd6b6ffd640 3 client.4531.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.857+0000 7fd6b6ffd640 3 client.4531.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.857+0000 7fd6b6ffd640 20 client.4531.objecter dump_active .. 0 homeless 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.857+0000 7fd6bed9d640 10 monclient: get_auth_request con 0x7fd6a0041b30 auth_method 0 2024-08-19T12:38:57.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 10 client.4531.objecter ms_handle_connect 0x7fd6a4002860 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 10 client.4531.objecter resend_mon_ops 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6be59c640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.26:6791/0 conn(0x7fd6a4002860 legacy=0x7fd6a4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2841242964 0 0) 0x7fd6a40065c0 con 0x7fd6a4002860 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 10 monclient(hunting): my global_id is 4540 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6c012f790 con 0x7fd6a4002860 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3833071909 0 0) 0x7fd6c012f790 con 0x7fd6a4002860 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6a40065c0 con 0x7fd6a4002860 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd6b0003df0 con 0x7fd6a4002860 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 10 monclient: dump: 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.883 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3925335072 0 0) 0x7fd6a40065c0 con 0x7fd6a4002860 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6a400a160 con 0x7fd6a4002860 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.859+0000 7fd6b6ffd640 10 client.4531.objecter ms_handle_connect 0x7fd6a0041b30 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.860+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd6b00040e0 con 0x7fd6a4002860 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.860+0000 7fd6b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.860+0000 7fd6b6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.860+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.26:6791/0 conn(0x7fd6a4002860 legacy=0x7fd6a4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.860+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.26:6791/0 conn(0x7fd6a4002860 legacy=0x7fd6a4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.866+0000 7fd6c4955640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.866+0000 7fd6c4955640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd6a4009e50 con 0x7fd6a4009110 2024-08-19T12:38:57.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6be59c640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.26:6791/0 conn(0x7fd6a4009110 legacy=0x7fd6a40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 10 client.4531.objecter ms_handle_connect 0x7fd6a4009110 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 10 client.4531.objecter resend_mon_ops 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4025046278 0 0) 0x7fd6a4009e50 con 0x7fd6a4009110 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 10 monclient(hunting): my global_id is 4549 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6a40065c0 con 0x7fd6a4009110 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 187836145 0 0) 0x7fd6a40065c0 con 0x7fd6a4009110 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6a4009e50 con 0x7fd6a4009110 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd6b0005b00 con 0x7fd6a4009110 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:57.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 10 monclient: dump: 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:57.886 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:57.887 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3001316753 0 0) 0x7fd6a4009e50 con 0x7fd6a4009110 2024-08-19T12:38:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.868+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6a400c3a0 con 0x7fd6a4009110 2024-08-19T12:38:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7698 (unknown 0 0 3223801872) 0x7fd6b0005b80 con 0x7fd6a4009110 2024-08-19T12:38:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6b6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.26:6791/0 conn(0x7fd6a4009110 legacy=0x7fd6a40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6b6ffd640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.26:6791/0 conn(0x7fd6a4009110 legacy=0x7fd6a40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:57.887 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 8, 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 50899, 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:57.888 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:57.889 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:57.890 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:57.891 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.893 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:57.894 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:57.895 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:57.897 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.898 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:57.901 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:57.903 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:57.904 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:57.905 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.906 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.907 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:57.908 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:57.909 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:57.910 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:57.910 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 10 librados: watch_flush enter 2024-08-19T12:38:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 10 librados: watch_flush exit 2024-08-19T12:38:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 20 client.4531.objecter shutdown clearing up homeless session... 2024-08-19T12:38:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 10 client.4531.objecter successfully canceled tick 2024-08-19T12:38:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.104:6800/899887506 conn(0x7fd6a0041b30 legacy=0x7fd6a0043ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 10 monclient: shutdown 2024-08-19T12:38:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 1 -- 172.21.15.26:0/3414754818 >> v1:172.21.15.26:6791/0 conn(0x7fd6c00a57a0 legacy=0x7fd6c012e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6bed9d640 1 -- 172.21.15.26:0/3414754818 reap_dead start 2024-08-19T12:38:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6bed9d640 1 -- 172.21.15.26:0/3414754818 reap_dead start 2024-08-19T12:38:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 1 -- 172.21.15.26:0/3414754818 shutdown_connections 2024-08-19T12:38:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 1 -- 172.21.15.26:0/3414754818 >> 172.21.15.26:0/3414754818 conn(0x7fd6c00abff0 msgr2=0x7fd6c00a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:57.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 1 -- 172.21.15.26:0/3414754818 shutdown_connections 2024-08-19T12:38:57.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 1 -- 172.21.15.26:0/3414754818 wait complete. 2024-08-19T12:38:57.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:57.869+0000 7fd6c4955640 1 librados: shutdown 2024-08-19T12:38:57.913 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:38:57.913 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon d 2024-08-19T12:38:57.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:38:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.013+0000 7f3dd10c7640 10 monclient: get_monmap_and_config 2024-08-19T12:38:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.013+0000 7f3dd10c7640 10 monclient: build_initial_monmap 2024-08-19T12:38:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.013+0000 7f3dd10c7640 10 monclient: monmap: 2024-08-19T12:38:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:58.013962+0000 2024-08-19T12:38:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:58.013962+0000 2024-08-19T12:38:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:58.014 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:58.015 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:58.015 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.013+0000 7f3dd10c7640 1 Processor -- start 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.013+0000 7f3dd10c7640 1 -- start start 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.013+0000 7f3dd10c7640 10 monclient: init 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dd10c7640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dd10c7640 10 monclient: _add_conns ranks=[7,4,6,2,8,5,0,3,1] 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dd10c7640 10 monclient(hunting): picked mon.noname-i con 0x7f3dcc073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dd10c7640 10 monclient(hunting): picked mon.noname-c con 0x7f3dcc0a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dd10c7640 10 monclient(hunting): picked mon.noname-d con 0x7f3dcc13f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dd10c7640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc09dae0 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dd10c7640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc136910 con 0x7f3dcc13f2e0 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dd10c7640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc09dd50 con 0x7f3dcc073470 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dd10c7640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dd10c7640 10 monclient(hunting): authenticate will time out at 732.101500s 2024-08-19T12:38:58.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dcad9d640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f3dcc073470 legacy=0x7f3dcc136200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:38:58.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dc3fff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f3dcc0a57a0 0x7f3dcc0a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:38926/0 (socket says 172.21.15.26:38926) 2024-08-19T12:38:58.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.014+0000 7f3dc3fff640 1 -- 172.21.15.26:0/2301325658 learned_addr learned my addr 172.21.15.26:0/2301325658 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dcb59e640 1 -- 172.21.15.26:0/2301325658 >> v1:172.21.15.26:6792/0 conn(0x7f3dcc13f2e0 legacy=0x7f3dcc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dc3fff640 1 -- 172.21.15.26:0/2301325658 >> v1:172.21.15.26:6791/0 conn(0x7f3dcc0a57a0 legacy=0x7f3dcc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dca59c640 1 -- 172.21.15.26:0/2301325658 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2808025383 0 0) 0x7f3dcc136910 con 0x7f3dcc13f2e0 2024-08-19T12:38:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dca59c640 10 monclient(hunting): _negotiate 2024-08-19T12:38:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dca59c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dca59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dca59c640 10 monclient(hunting): my global_id is 4596 2024-08-19T12:38:58.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dca59c640 1 -- 172.21.15.26:0/2301325658 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3db8003520 con 0x7f3dcc13f2e0 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dca59c640 1 -- 172.21.15.26:0/2301325658 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 393432303 0 0) 0x7f3dcc09dae0 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dca59c640 10 monclient(hunting): _negotiate 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dca59c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dca59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dca59c640 10 monclient(hunting): my global_id is 4558 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.016+0000 7f3dca59c640 1 -- 172.21.15.26:0/2301325658 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dcc136910 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 1 -- 172.21.15.26:0/2301325658 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2275663281 0 0) 0x7f3dcc136910 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 1 -- 172.21.15.26:0/2301325658 >> v1:172.21.15.104:6792/0 conn(0x7f3dcc073470 legacy=0x7f3dcc136200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 1 -- 172.21.15.26:0/2301325658 >> v1:172.21.15.26:6792/0 conn(0x7f3dcc13f2e0 legacy=0x7f3dcc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 10 monclient: _finish_hunting 0 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 1 monclient: found mon.noname-c 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:38:58.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 1 -- 172.21.15.26:0/2301325658 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc09e030 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 10 monclient: _finish_auth 0 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 10 monclient: _check_auth_tickets 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 1 -- 172.21.15.26:0/2301325658 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3db4001ac0 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dd10c7640 5 monclient: authenticate success, global_id 4558 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dd10c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 10 monclient: dump: 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.023 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dd10c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 1 -- 172.21.15.26:0/2301325658 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3db40048a0 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 1 -- 172.21.15.26:0/2301325658 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3db4004d90 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dca59c640 10 monclient: dump: 2024-08-19T12:38:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.025 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dd10c7640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dd10c7640 4 set_mon_vals no callback set 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dd10c7640 10 monclient: shutdown 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dd10c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dd10c7640 1 -- 172.21.15.26:0/2301325658 >> v1:172.21.15.26:6791/0 conn(0x7f3dcc0a57a0 legacy=0x7f3dcc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dd10c7640 1 -- 172.21.15.26:0/2301325658 shutdown_connections 2024-08-19T12:38:58.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dd10c7640 1 -- 172.21.15.26:0/2301325658 >> 172.21.15.26:0/2301325658 conn(0x7f3dcc0abff0 msgr2=0x7f3dcc0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dd10c7640 1 -- 172.21.15.26:0/2301325658 shutdown_connections 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.017+0000 7f3dd10c7640 1 -- 172.21.15.26:0/2301325658 wait complete. 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.018+0000 7f3dd10c7640 10 monclient: build_initial_monmap 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.018+0000 7f3dd10c7640 10 monclient: monmap: 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:58.027 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.018+0000 7f3dd10c7640 1 librados: starting msgr at 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.018+0000 7f3dd10c7640 1 librados: starting objecter 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.018+0000 7f3dd10c7640 1 Processor -- start 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 1 -- start start 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 1 librados: setting wanted keys 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 1 librados: calling monclient init 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 10 monclient: init 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 10 monclient: _add_conns ranks=[5,6,2,4,7,1,8,3,0] 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 10 monclient(hunting): picked mon.noname-h con 0x7f3dcc073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:58.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 10 monclient(hunting): picked mon.noname-d con 0x7f3dcc0a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:58.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 10 monclient(hunting): picked mon.noname-b con 0x7f3dcc13f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:58.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc0aa140 con 0x7f3dcc13f2e0 2024-08-19T12:38:58.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc13f0b0 con 0x7f3dcc073470 2024-08-19T12:38:58.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc142b60 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:58.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dd10c7640 10 monclient(hunting): authenticate will time out at 732.106517s 2024-08-19T12:38:58.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dc3fff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f3dcc0a57a0 0x7f3dcc12e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:55942/0 (socket says 172.21.15.26:55942) 2024-08-19T12:38:58.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.019+0000 7f3dc3fff640 1 -- 172.21.15.26:0/1573533485 learned_addr learned my addr 172.21.15.26:0/1573533485 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc3fff640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.26:6792/0 conn(0x7f3dcc0a57a0 legacy=0x7f3dcc12e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 client.?.objecter ms_handle_connect 0x7f3dcc0a57a0 2024-08-19T12:38:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dcb59e640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.26:6790/0 conn(0x7f3dcc13f2e0 legacy=0x7f3dcc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 client.?.objecter ms_handle_connect 0x7f3dcc13f2e0 2024-08-19T12:38:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dcad9d640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.104:6791/0 conn(0x7f3dcc073470 legacy=0x7f3dcc12dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 client.?.objecter ms_handle_connect 0x7f3dcc073470 2024-08-19T12:38:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3121662825 0 0) 0x7f3dcc142b60 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 monclient(hunting): my global_id is 4605 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dac004ee0 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1218948344 0 0) 0x7f3dcc0aa140 con 0x7f3dcc13f2e0 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 monclient(hunting): my global_id is 4601 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dcc142b60 con 0x7f3dcc13f2e0 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1992918811 0 0) 0x7f3dcc13f0b0 con 0x7f3dcc073470 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:58.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 10 monclient(hunting): my global_id is 4487 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.021+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dcc0aa140 con 0x7f3dcc073470 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 284773936 0 0) 0x7f3dac004ee0 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3dcc13f0b0 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3db4004d90 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient(hunting): dump: 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.036 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2606199148 0 0) 0x7f3dcc142b60 con 0x7f3dcc13f2e0 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3dac004ee0 con 0x7f3dcc13f2e0 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 420666067 0 0) 0x7f3dcc0aa140 con 0x7f3dcc073470 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3dcc142b60 con 0x7f3dcc073470 2024-08-19T12:38:58.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dc40016d0 con 0x7f3dcc13f2e0 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient(hunting): dump: 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.038 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dbc004450 con 0x7f3dcc073470 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient(hunting): dump: 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.039 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2372375702 0 0) 0x7f3dcc13f0b0 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.104:6791/0 conn(0x7f3dcc073470 legacy=0x7f3dcc12dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.26:6790/0 conn(0x7f3dcc13f2e0 legacy=0x7f3dcc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 monclient: found mon.d 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc12eb10 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dd10c7640 5 monclient: authenticate success, global_id 4605 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dd10c7640 10 monclient: _renew_subs 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dd10c7640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:58.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dd10c7640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3dcc131690 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dd10c7640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dd10c7640 10 monclient: _renew_subs 2024-08-19T12:38:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dd10c7640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dd10c7640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f3dcc132f60 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dd10c7640 1 librados: init done 2024-08-19T12:38:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3db4004820 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3db4001a60 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:58.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc9d9b640 4 set_mon_vals no callback set 2024-08-19T12:38:58.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:58.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.022+0000 7f3dc37fe640 10 monclient: dump: 2024-08-19T12:38:58.043 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.043 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.043 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.043 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.043 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.043 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.043 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.043 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.043 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.023+0000 7f3dd10c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.023+0000 7f3dd10c7640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3db00065c0 con 0x7f3db0002860 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.023+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3db4011980 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.023+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f3db404db30 con 0x7f3dcc0a57a0 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.023+0000 7f3dc37fe640 10 client.4605.objecter ms_dispatch 0x7f3dcc000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:58.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.023+0000 7f3dc37fe640 3 client.4605.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:58.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.023+0000 7f3dc37fe640 3 client.4605.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:58.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.023+0000 7f3dc37fe640 20 client.4605.objecter dump_active .. 0 homeless 2024-08-19T12:38:58.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.024+0000 7f3dcb59e640 10 monclient: get_auth_request con 0x7f3dac041aa0 auth_method 0 2024-08-19T12:38:58.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 10 client.4605.objecter ms_handle_connect 0x7f3db0002860 2024-08-19T12:38:58.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dcad9d640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.26:6792/0 conn(0x7f3db0002860 legacy=0x7f3db0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 10 client.4605.objecter resend_mon_ops 2024-08-19T12:38:58.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526352602 0 0) 0x7f3db00065c0 con 0x7f3db0002860 2024-08-19T12:38:58.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:58.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:58.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 10 monclient(hunting): my global_id is 4614 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dcc13f0b0 con 0x7f3db0002860 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 304804497 0 0) 0x7f3dcc13f0b0 con 0x7f3db0002860 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3db00065c0 con 0x7f3db0002860 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dbc004c30 con 0x7f3db0002860 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.025+0000 7f3dc37fe640 10 monclient: dump: 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.026+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1336632968 0 0) 0x7f3db00065c0 con 0x7f3db0002860 2024-08-19T12:38:58.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.026+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3db000a160 con 0x7f3db0002860 2024-08-19T12:38:58.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.026+0000 7f3dc37fe640 10 client.4605.objecter ms_handle_connect 0x7f3dac041aa0 2024-08-19T12:38:58.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.026+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3dbc0061d0 con 0x7f3db0002860 2024-08-19T12:38:58.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.026+0000 7f3dc37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:58.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.026+0000 7f3dc37fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:58.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.026+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.26:6792/0 conn(0x7f3db0002860 legacy=0x7f3db0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:58.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.026+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.26:6792/0 conn(0x7f3db0002860 legacy=0x7f3db0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:58.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.032+0000 7f3dd10c7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:58.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.032+0000 7f3dd10c7640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3db0009e70 con 0x7f3db0009130 2024-08-19T12:38:58.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.034+0000 7f3dcad9d640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.26:6792/0 conn(0x7f3db0009130 legacy=0x7f3db00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.034+0000 7f3dc37fe640 10 client.4605.objecter ms_handle_connect 0x7f3db0009130 2024-08-19T12:38:58.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.034+0000 7f3dc37fe640 10 client.4605.objecter resend_mon_ops 2024-08-19T12:38:58.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1785855525 0 0) 0x7f3db0009e70 con 0x7f3db0009130 2024-08-19T12:38:58.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:58.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:58.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:58.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 10 monclient(hunting): my global_id is 4623 2024-08-19T12:38:58.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3db00065c0 con 0x7f3db0009130 2024-08-19T12:38:58.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2466391402 0 0) 0x7f3db00065c0 con 0x7f3db0009130 2024-08-19T12:38:58.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3db0009e70 con 0x7f3db0009130 2024-08-19T12:38:58.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dbc006a50 con 0x7f3db0009130 2024-08-19T12:38:58.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:58.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 10 monclient: dump: 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.051 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.052 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.052 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.052 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.052 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1222585209 0 0) 0x7f3db0009e70 con 0x7f3db0009130 2024-08-19T12:38:58.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3db000c3f0 con 0x7f3db0009130 2024-08-19T12:38:58.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7698 (unknown 0 0 3825297568) 0x7f3dbc004c30 con 0x7f3db0009130 2024-08-19T12:38:58.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:58.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:58.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.26:6792/0 conn(0x7f3db0009130 legacy=0x7f3db00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:58.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dc37fe640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.26:6792/0 conn(0x7f3db0009130 legacy=0x7f3db00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 8, 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 51076, 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:58.053 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:58.054 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:58.055 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:58.056 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:58.057 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:58.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.059 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:58.059 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:58.059 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:58.059 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:58.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.059 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:58.059 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:58.059 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:58.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.059 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:58.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.059 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:58.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.062 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:58.063 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:58.064 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:58.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:58.067 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:58.068 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:58.069 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:58.070 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:58.070 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:58.070 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:58.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:58.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.070 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:58.070 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:58.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.070 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:58.070 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:58.070 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:58.071 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:58.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:58.072 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:58.072 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:58.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dd10c7640 10 librados: watch_flush enter 2024-08-19T12:38:58.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dd10c7640 10 librados: watch_flush exit 2024-08-19T12:38:58.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dd10c7640 20 client.4605.objecter shutdown clearing up homeless session... 2024-08-19T12:38:58.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.035+0000 7f3dd10c7640 10 client.4605.objecter successfully canceled tick 2024-08-19T12:38:58.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.036+0000 7f3dd10c7640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.104:6800/899887506 conn(0x7f3dac041aa0 legacy=0x7f3dac043f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:58.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.036+0000 7f3dd10c7640 10 monclient: shutdown 2024-08-19T12:38:58.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.036+0000 7f3dd10c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:58.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.036+0000 7f3dd10c7640 1 -- 172.21.15.26:0/1573533485 >> v1:172.21.15.26:6792/0 conn(0x7f3dcc0a57a0 legacy=0x7f3dcc12e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:58.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.036+0000 7f3dcb59e640 1 -- 172.21.15.26:0/1573533485 reap_dead start 2024-08-19T12:38:58.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.036+0000 7f3dcb59e640 1 -- 172.21.15.26:0/1573533485 reap_dead start 2024-08-19T12:38:58.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.036+0000 7f3dd10c7640 1 -- 172.21.15.26:0/1573533485 shutdown_connections 2024-08-19T12:38:58.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.036+0000 7f3dd10c7640 1 -- 172.21.15.26:0/1573533485 >> 172.21.15.26:0/1573533485 conn(0x7f3dcc0abff0 msgr2=0x7f3dcc0a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:58.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.036+0000 7f3dd10c7640 1 -- 172.21.15.26:0/1573533485 shutdown_connections 2024-08-19T12:38:58.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.036+0000 7f3dd10c7640 1 -- 172.21.15.26:0/1573533485 wait complete. 2024-08-19T12:38:58.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.036+0000 7f3dd10c7640 1 librados: shutdown 2024-08-19T12:38:58.074 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:38:58.075 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-08-19T12:38:58.075 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:38:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.174+0000 7f8ba1acc640 10 monclient: get_monmap_and_config 2024-08-19T12:38:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.174+0000 7f8ba1acc640 10 monclient: build_initial_monmap 2024-08-19T12:38:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.174+0000 7f8ba1acc640 10 monclient: monmap: 2024-08-19T12:38:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.175 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:58.175373+0000 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:58.175373+0000 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:58.176 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 1 Processor -- start 2024-08-19T12:38:58.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 1 -- start start 2024-08-19T12:38:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 10 monclient: init 2024-08-19T12:38:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 10 monclient: _add_conns ranks=[7,1,4,2,8,3,5,6,0] 2024-08-19T12:38:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 10 monclient(hunting): picked mon.noname-i con 0x7f8b9c06e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 10 monclient(hunting): picked mon.noname-f con 0x7f8b9c0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 10 monclient(hunting): picked mon.noname-c con 0x7f8b9c0a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b9c136790 con 0x7f8b9c0a5920 2024-08-19T12:38:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b9c09ddf0 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b9c09e0d0 con 0x7f8b9c06e7b0 2024-08-19T12:38:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8ba1acc640 10 monclient(hunting): authenticate will time out at 732.262802s 2024-08-19T12:38:58.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8b9b7fe640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f8b9c06e7b0 legacy=0x7f8b9c136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:38:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8b9bfff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f8b9c0a5d50 0x7f8b9c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:38938/0 (socket says 172.21.15.26:38938) 2024-08-19T12:38:58.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.175+0000 7f8b9bfff640 1 -- 172.21.15.26:0/448731050 learned_addr learned my addr 172.21.15.26:0/448731050 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9bfff640 1 -- 172.21.15.26:0/448731050 >> v1:172.21.15.26:6791/0 conn(0x7f8b9c0a5d50 legacy=0x7f8b9c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9affd640 1 -- 172.21.15.26:0/448731050 >> v1:172.21.15.104:6789/0 conn(0x7f8b9c0a5920 legacy=0x7f8b9c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 1 -- 172.21.15.26:0/448731050 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2654465876 0 0) 0x7f8b9c136790 con 0x7f8b9c0a5920 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient(hunting): my global_id is 4501 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 1 -- 172.21.15.26:0/448731050 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b84003520 con 0x7f8b9c0a5920 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 1 -- 172.21.15.26:0/448731050 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3050494956 0 0) 0x7f8b84003520 con 0x7f8b9c0a5920 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 1 -- 172.21.15.26:0/448731050 >> v1:172.21.15.104:6792/0 conn(0x7f8b9c06e7b0 legacy=0x7f8b9c136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 1 -- 172.21.15.26:0/448731050 >> v1:172.21.15.26:6791/0 conn(0x7f8b9c0a5d50 legacy=0x7f8b9c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:58.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 1 monclient: found mon.noname-f 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 1 -- 172.21.15.26:0/448731050 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9c09e3b0 con 0x7f8b9c0a5920 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 1 -- 172.21.15.26:0/448731050 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8b90001bf0 con 0x7f8b9c0a5920 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient: dump: 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.183 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8ba1acc640 5 monclient: authenticate success, global_id 4501 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8ba1acc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 1 -- 172.21.15.26:0/448731050 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b90003610 con 0x7f8b9c0a5920 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8ba1acc640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8ba1acc640 4 set_mon_vals no callback set 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 1 -- 172.21.15.26:0/448731050 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8b90003bc0 con 0x7f8b9c0a5920 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.178+0000 7f8b9a7fc640 10 monclient: dump: 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.185 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.179+0000 7f8ba1acc640 10 monclient: shutdown 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.179+0000 7f8ba1acc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:58.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.179+0000 7f8ba1acc640 1 -- 172.21.15.26:0/448731050 >> v1:172.21.15.104:6789/0 conn(0x7f8b9c0a5920 legacy=0x7f8b9c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.179+0000 7f8ba1acc640 1 -- 172.21.15.26:0/448731050 shutdown_connections 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.179+0000 7f8ba1acc640 1 -- 172.21.15.26:0/448731050 >> 172.21.15.26:0/448731050 conn(0x7f8b9c0a8740 msgr2=0x7f8b9c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.179+0000 7f8ba1acc640 1 -- 172.21.15.26:0/448731050 shutdown_connections 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.179+0000 7f8ba1acc640 1 -- 172.21.15.26:0/448731050 wait complete. 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.179+0000 7f8ba1acc640 10 monclient: build_initial_monmap 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.179+0000 7f8ba1acc640 10 monclient: monmap: 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:58.187 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.179+0000 7f8ba1acc640 1 librados: starting msgr at 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.179+0000 7f8ba1acc640 1 librados: starting objecter 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 1 Processor -- start 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 1 -- start start 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 1 librados: setting wanted keys 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 1 librados: calling monclient init 2024-08-19T12:38:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 10 monclient: init 2024-08-19T12:38:58.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:58.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 10 monclient: _add_conns ranks=[4,0,2,8,3,7,1,5,6] 2024-08-19T12:38:58.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 10 monclient(hunting): picked mon.noname-c con 0x7f8b9c06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:58.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 10 monclient(hunting): picked mon.noname-a con 0x7f8b9c0a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:58.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 10 monclient(hunting): picked mon.noname-b con 0x7f8b9c0a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:58.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b9c0a49d0 con 0x7f8b9c0a5920 2024-08-19T12:38:58.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b9c0a04d0 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b9c0a07b0 con 0x7f8b9c06e7b0 2024-08-19T12:38:58.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:58.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8ba1acc640 10 monclient(hunting): authenticate will time out at 732.267731s 2024-08-19T12:38:58.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8b9b7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f8b9c06e7b0 0x7f8b9c132220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:38954/0 (socket says 172.21.15.26:38954) 2024-08-19T12:38:58.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.180+0000 7f8b9b7fe640 1 -- 172.21.15.26:0/2320428318 learned_addr learned my addr 172.21.15.26:0/2320428318 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:58.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.182+0000 7f8b9b7fe640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.26:6791/0 conn(0x7f8b9c06e7b0 legacy=0x7f8b9c132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.182+0000 7f8b83fff640 10 client.?.objecter ms_handle_connect 0x7f8b9c06e7b0 2024-08-19T12:38:58.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.182+0000 7f8b83fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:58.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.182+0000 7f8b9affd640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.26:6789/0 conn(0x7f8b9c0a5920 legacy=0x7f8b9c132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 client.?.objecter ms_handle_connect 0x7f8b9c0a5920 2024-08-19T12:38:58.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b9bfff640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.26:6790/0 conn(0x7f8b9c0a5d50 legacy=0x7f8b9c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:58.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 client.?.objecter ms_handle_connect 0x7f8b9c0a5d50 2024-08-19T12:38:58.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:58.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2099484492 0 0) 0x7f8b9c0a04d0 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): my global_id is 4610 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b88004eb0 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1228268869 0 0) 0x7f8b9c0a49d0 con 0x7f8b9c0a5920 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): my global_id is 4518 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b9c0a04d0 con 0x7f8b9c0a5920 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 831736965 0 0) 0x7f8b88004eb0 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b9c0a49d0 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3758068580 0 0) 0x7f8b9c0a04d0 con 0x7f8b9c0a5920 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b88004eb0 con 0x7f8b9c0a5920 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8b94001f60 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): dump: 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.195 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8b90001bb0 con 0x7f8b9c0a5920 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient(hunting): dump: 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.196 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3737489707 0 0) 0x7f8b9c0a49d0 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.26:6791/0 conn(0x7f8b9c06e7b0 legacy=0x7f8b9c132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.26:6789/0 conn(0x7f8b9c0a5920 legacy=0x7f8b9c132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 monclient: found mon.b 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9c09ea70 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient: _finish_auth 0 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8ba1acc640 5 monclient: authenticate success, global_id 4610 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8ba1acc640 10 monclient: _renew_subs 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8ba1acc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8ba1acc640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8b9c0a53b0 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8ba1acc640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:58.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8ba1acc640 10 monclient: _renew_subs 2024-08-19T12:38:58.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8ba1acc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:58.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8ba1acc640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8b9c133220 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8ba1acc640 1 librados: init done 2024-08-19T12:38:58.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b94004e10 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:58.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8b94005300 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:58.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b99ffb640 4 set_mon_vals no callback set 2024-08-19T12:38:58.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:58.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.183+0000 7f8b83fff640 10 monclient: dump: 2024-08-19T12:38:58.199 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:58.199 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:58.200 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.184+0000 7f8ba1acc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.184+0000 7f8ba1acc640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8b7c0065c0 con 0x7f8b7c002860 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.184+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f8b94011760 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.184+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f8b9404dad0 con 0x7f8b9c0a5d50 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.184+0000 7f8b83fff640 10 client.4610.objecter ms_dispatch 0x7f8b9c000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.184+0000 7f8b83fff640 3 client.4610.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.184+0000 7f8b83fff640 3 client.4610.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.185+0000 7f8b83fff640 20 client.4610.objecter dump_active .. 0 homeless 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.185+0000 7f8b9affd640 10 monclient: get_auth_request con 0x7f8b88041430 auth_method 0 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.186+0000 7f8b9b7fe640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.26:6793/0 conn(0x7f8b7c002860 legacy=0x7f8b7c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.186+0000 7f8b83fff640 10 client.4610.objecter ms_handle_connect 0x7f8b7c002860 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.186+0000 7f8b83fff640 10 client.4610.objecter resend_mon_ops 2024-08-19T12:38:58.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.187+0000 7f8b83fff640 10 client.4610.objecter ms_handle_connect 0x7f8b88041430 2024-08-19T12:38:58.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.885+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1237074696 0 0) 0x7f8b7c0065c0 con 0x7f8b7c002860 2024-08-19T12:38:58.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.885+0000 7f8b83fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:58.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.885+0000 7f8b83fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:58.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.885+0000 7f8b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:58.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.885+0000 7f8b83fff640 10 monclient(hunting): my global_id is 4607 2024-08-19T12:38:58.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:58.885+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b9c0a49d0 con 0x7f8b7c002860 2024-08-19T12:38:59.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.013+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2940148000 0 0) 0x7f8b9c0a49d0 con 0x7f8b7c002860 2024-08-19T12:38:59.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.013+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b7c0065c0 con 0x7f8b7c002860 2024-08-19T12:38:59.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.013+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8b8c004370 con 0x7f8b7c002860 2024-08-19T12:38:59.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.013+0000 7f8b83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.013+0000 7f8b83fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:59.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.013+0000 7f8b83fff640 10 monclient: dump: 2024-08-19T12:38:59.015 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.015 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.015 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.015 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.015 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.015 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.016 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.013+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1715382125 0 0) 0x7f8b7c0065c0 con 0x7f8b7c002860 2024-08-19T12:38:59.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.013+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8b7c00a160 con 0x7f8b7c002860 2024-08-19T12:38:59.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.014+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8b8c0058b0 con 0x7f8b7c002860 2024-08-19T12:38:59.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.014+0000 7f8b83fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:59.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.014+0000 7f8b83fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:59.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.014+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.26:6793/0 conn(0x7f8b7c002860 legacy=0x7f8b7c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.014+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.26:6793/0 conn(0x7f8b7c002860 legacy=0x7f8b7c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:59.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.020+0000 7f8ba1acc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:59.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.020+0000 7f8ba1acc640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8b7c009e70 con 0x7f8b7c009130 2024-08-19T12:38:59.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b9b7fe640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.26:6793/0 conn(0x7f8b7c009130 legacy=0x7f8b7c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 10 client.4610.objecter ms_handle_connect 0x7f8b7c009130 2024-08-19T12:38:59.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 10 client.4610.objecter resend_mon_ops 2024-08-19T12:38:59.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2637305345 0 0) 0x7f8b7c009e70 con 0x7f8b7c009130 2024-08-19T12:38:59.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 10 monclient(hunting): my global_id is 4616 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b7c0065c0 con 0x7f8b7c009130 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 59570976 0 0) 0x7f8b7c0065c0 con 0x7f8b7c009130 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b7c009e70 con 0x7f8b7c009130 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8b8c004bf0 con 0x7f8b7c009130 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 10 monclient: dump: 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1567930174 0 0) 0x7f8b7c009e70 con 0x7f8b7c009130 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.023+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8b7c00c410 con 0x7f8b7c009130 2024-08-19T12:38:59.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7531 (unknown 0 0 3443291614) 0x7f8b8c005b60 con 0x7f8b7c009130 2024-08-19T12:38:59.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8b83fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:59.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8b83fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:59.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.26:6793/0 conn(0x7f8b7c009130 legacy=0x7f8b7c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8b83fff640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.26:6793/0 conn(0x7f8b7c009130 legacy=0x7f8b7c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:59.028 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:59.028 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:59.028 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:59.028 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:59.028 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 8, 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 52041, 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:38:59.029 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:59.030 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:59.031 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:59.032 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:59.033 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.034 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:59.035 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.036 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.037 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.038 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:59.039 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:59.041 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.045 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:59.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.047 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:59.047 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:59.047 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:59.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.047 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.047 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:59.047 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8ba1acc640 10 librados: watch_flush enter 2024-08-19T12:38:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8ba1acc640 10 librados: watch_flush exit 2024-08-19T12:38:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8ba1acc640 20 client.4610.objecter shutdown clearing up homeless session... 2024-08-19T12:38:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8ba1acc640 10 client.4610.objecter successfully canceled tick 2024-08-19T12:38:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8ba1acc640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.104:6800/899887506 conn(0x7f8b88041430 legacy=0x7f8b880438f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8ba1acc640 10 monclient: shutdown 2024-08-19T12:38:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8ba1acc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8ba1acc640 1 -- 172.21.15.26:0/2320428318 >> v1:172.21.15.26:6790/0 conn(0x7f8b9c0a5d50 legacy=0x7f8b9c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8b9bfff640 1 -- 172.21.15.26:0/2320428318 reap_dead start 2024-08-19T12:38:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8b9bfff640 1 -- 172.21.15.26:0/2320428318 reap_dead start 2024-08-19T12:38:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8ba1acc640 1 -- 172.21.15.26:0/2320428318 shutdown_connections 2024-08-19T12:38:59.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8ba1acc640 1 -- 172.21.15.26:0/2320428318 >> 172.21.15.26:0/2320428318 conn(0x7f8b9c0a8740 msgr2=0x7f8b9c06d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:59.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.024+0000 7f8ba1acc640 1 -- 172.21.15.26:0/2320428318 shutdown_connections 2024-08-19T12:38:59.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.025+0000 7f8ba1acc640 1 -- 172.21.15.26:0/2320428318 wait complete. 2024-08-19T12:38:59.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.025+0000 7f8ba1acc640 1 librados: shutdown 2024-08-19T12:38:59.050 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:38:59.050 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-08-19T12:38:59.050 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:38:59.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.155+0000 7fcdeb6de640 10 monclient: get_monmap_and_config 2024-08-19T12:38:59.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.155+0000 7fcdeb6de640 10 monclient: build_initial_monmap 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.156+0000 7fcdeb6de640 10 monclient: monmap: 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:59.156819+0000 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:59.156819+0000 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:59.157 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:59.158 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:59.158 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:59.158 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:59.158 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.156+0000 7fcdeb6de640 1 Processor -- start 2024-08-19T12:38:59.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.156+0000 7fcdeb6de640 1 -- start start 2024-08-19T12:38:59.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.156+0000 7fcdeb6de640 10 monclient: init 2024-08-19T12:38:59.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcdeb6de640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:59.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcdeb6de640 10 monclient: _add_conns ranks=[2,3,7,5,1,0,6,4,8] 2024-08-19T12:38:59.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcdeb6de640 10 monclient(hunting): picked mon.noname-b con 0x7fcde406e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:59.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcdeb6de640 10 monclient(hunting): picked mon.noname-g con 0x7fcde40a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:59.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcdeb6de640 10 monclient(hunting): picked mon.noname-i con 0x7fcde40a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:59.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcdeb6de640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcde4136790 con 0x7fcde406e7b0 2024-08-19T12:38:59.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcdeb6de640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcde409ddf0 con 0x7fcde40a5920 2024-08-19T12:38:59.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcdeb6de640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcde409e0d0 con 0x7fcde40a5d50 2024-08-19T12:38:59.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcdeb6de640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:59.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcdeb6de640 10 monclient(hunting): authenticate will time out at 733.244391s 2024-08-19T12:38:59.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcde9c7b640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7fcde40a5d50 legacy=0x7fcde4141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:38:59.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcde947a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fcde406e7b0 0x7fcde4136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59346/0 (socket says 172.21.15.26:59346) 2024-08-19T12:38:59.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.157+0000 7fcde947a640 1 -- 172.21.15.26:0/3608985556 learned_addr learned my addr 172.21.15.26:0/3608985556 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde947a640 1 -- 172.21.15.26:0/3608985556 >> v1:172.21.15.26:6790/0 conn(0x7fcde406e7b0 legacy=0x7fcde4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde8c79640 1 -- 172.21.15.26:0/3608985556 >> v1:172.21.15.104:6790/0 conn(0x7fcde40a5920 legacy=0x7fcde413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde3fff640 1 -- 172.21.15.26:0/3608985556 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3709221686 0 0) 0x7fcde4136790 con 0x7fcde406e7b0 2024-08-19T12:38:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde3fff640 10 monclient(hunting): my global_id is 4619 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde3fff640 1 -- 172.21.15.26:0/3608985556 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdcc003520 con 0x7fcde406e7b0 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde3fff640 1 -- 172.21.15.26:0/3608985556 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3578984965 0 0) 0x7fcdd8001820 con 0x7fcde40a5920 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde3fff640 10 monclient(hunting): my global_id is 4611 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.159+0000 7fcde3fff640 1 -- 172.21.15.26:0/3608985556 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcde4136790 con 0x7fcde40a5920 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 1 -- 172.21.15.26:0/3608985556 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3177234306 0 0) 0x7fcdcc003520 con 0x7fcde406e7b0 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 1 -- 172.21.15.26:0/3608985556 >> v1:172.21.15.104:6792/0 conn(0x7fcde40a5d50 legacy=0x7fcde4141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 1 -- 172.21.15.26:0/3608985556 >> v1:172.21.15.104:6790/0 conn(0x7fcde40a5920 legacy=0x7fcde413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 10 monclient: _finish_hunting 0 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 1 monclient: found mon.noname-b 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:59.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 1 -- 172.21.15.26:0/3608985556 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde409e3b0 con 0x7fcde406e7b0 2024-08-19T12:38:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 10 monclient: _finish_auth 0 2024-08-19T12:38:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 10 monclient: _check_auth_tickets 2024-08-19T12:38:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 1 -- 172.21.15.26:0/3608985556 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcdd4001ae0 con 0x7fcde406e7b0 2024-08-19T12:38:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 10 monclient: dump: 2024-08-19T12:38:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.166 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 1 -- 172.21.15.26:0/3608985556 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcdd8002060 con 0x7fcde40a5920 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcdeb6de640 5 monclient: authenticate success, global_id 4619 2024-08-19T12:38:59.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcdeb6de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:38:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 1 -- 172.21.15.26:0/3608985556 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcdd40048a0 con 0x7fcde406e7b0 2024-08-19T12:38:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 1 -- 172.21.15.26:0/3608985556 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcdd4004e10 con 0x7fcde406e7b0 2024-08-19T12:38:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcde3fff640 10 monclient: dump: 2024-08-19T12:38:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.168 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcdeb6de640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcdeb6de640 4 set_mon_vals no callback set 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcdeb6de640 10 monclient: shutdown 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcdeb6de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3608985556 >> v1:172.21.15.26:6790/0 conn(0x7fcde406e7b0 legacy=0x7fcde4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3608985556 shutdown_connections 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3608985556 >> 172.21.15.26:0/3608985556 conn(0x7fcde40a8740 msgr2=0x7fcde40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3608985556 shutdown_connections 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.160+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3608985556 wait complete. 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.161+0000 7fcdeb6de640 10 monclient: build_initial_monmap 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.161+0000 7fcdeb6de640 10 monclient: monmap: 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:59.170 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.161+0000 7fcdeb6de640 1 librados: starting msgr at 2024-08-19T12:38:59.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.161+0000 7fcdeb6de640 1 librados: starting objecter 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.161+0000 7fcdeb6de640 1 Processor -- start 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.161+0000 7fcdeb6de640 1 -- start start 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.161+0000 7fcdeb6de640 1 librados: setting wanted keys 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.161+0000 7fcdeb6de640 1 librados: calling monclient init 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.161+0000 7fcdeb6de640 10 monclient: init 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.162+0000 7fcdeb6de640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.162+0000 7fcdeb6de640 10 monclient: _add_conns ranks=[2,5,3,7,8,6,1,0,4] 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.162+0000 7fcdeb6de640 10 monclient(hunting): picked mon.noname-b con 0x7fcde406e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.162+0000 7fcdeb6de640 10 monclient(hunting): picked mon.noname-h con 0x7fcde40a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.162+0000 7fcdeb6de640 10 monclient(hunting): picked mon.noname-g con 0x7fcde40a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.162+0000 7fcdeb6de640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcde409ee00 con 0x7fcde406e7b0 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.162+0000 7fcdeb6de640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcde409e980 con 0x7fcde40a5d50 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.162+0000 7fcdeb6de640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcde4132ce0 con 0x7fcde40a5920 2024-08-19T12:38:59.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.162+0000 7fcdeb6de640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:59.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.162+0000 7fcdeb6de640 10 monclient(hunting): authenticate will time out at 733.249270s 2024-08-19T12:38:59.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.162+0000 7fcde947a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fcde406e7b0 0x7fcde4131ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59362/0 (socket says 172.21.15.26:59362) 2024-08-19T12:38:59.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.162+0000 7fcde947a640 1 -- 172.21.15.26:0/3900216520 learned_addr learned my addr 172.21.15.26:0/3900216520 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:59.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde947a640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.26:6790/0 conn(0x7fcde406e7b0 legacy=0x7fcde4131ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 client.?.objecter ms_handle_connect 0x7fcde406e7b0 2024-08-19T12:38:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 client.?.objecter ms_handle_connect 0x7fcde40a5d50 2024-08-19T12:38:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde9c7b640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.104:6790/0 conn(0x7fcde40a5d50 legacy=0x7fcde41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2220424347 0 0) 0x7fcde409ee00 con 0x7fcde406e7b0 2024-08-19T12:38:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): my global_id is 4628 2024-08-19T12:38:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde8c79640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.104:6791/0 conn(0x7fcde40a5920 legacy=0x7fcde41325d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdd0004eb0 con 0x7fcde406e7b0 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 client.?.objecter ms_handle_connect 0x7fcde40a5920 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1286665769 0 0) 0x7fcde409e980 con 0x7fcde40a5d50 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): my global_id is 4620 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcde409ee00 con 0x7fcde40a5d50 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3515646333 0 0) 0x7fcde4132ce0 con 0x7fcde40a5920 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): my global_id is 4496 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcde409e980 con 0x7fcde40a5920 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4013123062 0 0) 0x7fcdd0004eb0 con 0x7fcde406e7b0 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcde4132ce0 con 0x7fcde406e7b0 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcdd4004d90 con 0x7fcde406e7b0 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.180 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.164+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 729399956 0 0) 0x7fcde409ee00 con 0x7fcde40a5d50 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcdd0004eb0 con 0x7fcde40a5d50 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcddc0044c0 con 0x7fcde40a5d50 2024-08-19T12:38:59.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 10 monclient(hunting): dump: 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.182 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3720512514 0 0) 0x7fcde4132ce0 con 0x7fcde406e7b0 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.104:6791/0 conn(0x7fcde40a5920 legacy=0x7fcde41325d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.104:6790/0 conn(0x7fcde40a5d50 legacy=0x7fcde41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 1 monclient: found mon.b 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde40a0700 con 0x7fcde406e7b0 2024-08-19T12:38:59.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcdeb6de640 5 monclient: authenticate success, global_id 4628 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcdeb6de640 10 monclient: _renew_subs 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcdeb6de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcde40a09e0 con 0x7fcde406e7b0 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcdd4001d30 con 0x7fcde406e7b0 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcdd4001b00 con 0x7fcde406e7b0 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 10 monclient: dump: 2024-08-19T12:38:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.185 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcdeb6de640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:59.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcdeb6de640 10 monclient: _renew_subs 2024-08-19T12:38:59.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcdeb6de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:38:59.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcde40a54a0 con 0x7fcde406e7b0 2024-08-19T12:38:59.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcdeb6de640 1 librados: init done 2024-08-19T12:38:59.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde37fe640 4 set_mon_vals no callback set 2024-08-19T12:38:59.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.165+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fcdd4011a00 con 0x7fcde406e7b0 2024-08-19T12:38:59.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.166+0000 7fcdeb6de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:59.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.166+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcde4134cd0 con 0x7fcde412da10 2024-08-19T12:38:59.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.166+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7fcdd404da20 con 0x7fcde406e7b0 2024-08-19T12:38:59.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.166+0000 7fcde1ffb640 10 client.4628.objecter ms_dispatch 0x7fcde4000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:59.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.166+0000 7fcde1ffb640 3 client.4628.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.166+0000 7fcde1ffb640 3 client.4628.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.166+0000 7fcde1ffb640 20 client.4628.objecter dump_active .. 0 homeless 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.166+0000 7fcde9c7b640 10 monclient: get_auth_request con 0x7fcdd00416a0 auth_method 0 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde8c79640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.104:6789/0 conn(0x7fcde412da10 legacy=0x7fcde412ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 10 client.4628.objecter ms_handle_connect 0x7fcde412da10 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 10 client.4628.objecter resend_mon_ops 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1333710846 0 0) 0x7fcde4134cd0 con 0x7fcde412da10 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 10 monclient(hunting): my global_id is 4510 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcde4132ce0 con 0x7fcde412da10 2024-08-19T12:38:59.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 10 client.4628.objecter ms_handle_connect 0x7fcdd00416a0 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2451907857 0 0) 0x7fcde4132ce0 con 0x7fcde412da10 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcde4134cd0 con 0x7fcde412da10 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcdd8003de0 con 0x7fcde412da10 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 10 monclient: dump: 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.188 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3575415737 0 0) 0x7fcde4134cd0 con 0x7fcde412da10 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.168+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcde41347a0 con 0x7fcde412da10 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.169+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcdd80018b0 con 0x7fcde412da10 2024-08-19T12:38:59.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.169+0000 7fcde1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:59.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.169+0000 7fcde1ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:59.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.169+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.104:6789/0 conn(0x7fcde412da10 legacy=0x7fcde412ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.169+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.104:6789/0 conn(0x7fcde412da10 legacy=0x7fcde412ddf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:59.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.175+0000 7fcdeb6de640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:59.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.175+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcde4068df0 con 0x7fcde412e500 2024-08-19T12:38:59.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.177+0000 7fcde8c79640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.104:6789/0 conn(0x7fcde412e500 legacy=0x7fcde41db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.177+0000 7fcde1ffb640 10 client.4628.objecter ms_handle_connect 0x7fcde412e500 2024-08-19T12:38:59.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.177+0000 7fcde1ffb640 10 client.4628.objecter resend_mon_ops 2024-08-19T12:38:59.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1693488383 0 0) 0x7fcde4068df0 con 0x7fcde412e500 2024-08-19T12:38:59.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 10 monclient(hunting): my global_id is 4519 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcde4134cd0 con 0x7fcde412e500 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3254351520 0 0) 0x7fcde4134cd0 con 0x7fcde412e500 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcde4068df0 con 0x7fcde412e500 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcdd8004630 con 0x7fcde412e500 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 10 monclient: dump: 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.192 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3271141916 0 0) 0x7fcde4068df0 con 0x7fcde412e500 2024-08-19T12:38:59.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcde412e8e0 con 0x7fcde412e500 2024-08-19T12:38:59.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7698 (unknown 0 0 899281086) 0x7fcdd8005b40 con 0x7fcde412e500 2024-08-19T12:38:59.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:59.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:59.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.104:6789/0 conn(0x7fcde412e500 legacy=0x7fcde41db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcde1ffb640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.104:6789/0 conn(0x7fcde412e500 legacy=0x7fcde41db9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:59.194 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:59.194 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:59.194 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:59.194 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 8, 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 52182, 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:38:59.195 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:59.196 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:59.197 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:59.198 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:59.199 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.200 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:59.201 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:59.202 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.202 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.202 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.202 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:59.202 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.202 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.202 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:59.202 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:59.202 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.205 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.206 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:59.208 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.209 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:59.211 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.212 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.213 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.214 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:59.214 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:59.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.178+0000 7fcdeb6de640 10 librados: watch_flush enter 2024-08-19T12:38:59.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcdeb6de640 10 librados: watch_flush exit 2024-08-19T12:38:59.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcdeb6de640 20 client.4628.objecter shutdown clearing up homeless session... 2024-08-19T12:38:59.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcdeb6de640 10 client.4628.objecter successfully canceled tick 2024-08-19T12:38:59.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.104:6800/899887506 conn(0x7fcdd00416a0 legacy=0x7fcdd0043b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcdeb6de640 10 monclient: shutdown 2024-08-19T12:38:59.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcdeb6de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:59.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3900216520 >> v1:172.21.15.26:6790/0 conn(0x7fcde406e7b0 legacy=0x7fcde4131ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcde9c7b640 1 -- 172.21.15.26:0/3900216520 reap_dead start 2024-08-19T12:38:59.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcde9c7b640 1 -- 172.21.15.26:0/3900216520 reap_dead start 2024-08-19T12:38:59.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3900216520 shutdown_connections 2024-08-19T12:38:59.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3900216520 >> 172.21.15.26:0/3900216520 conn(0x7fcde40a8740 msgr2=0x7fcde406d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:59.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3900216520 shutdown_connections 2024-08-19T12:38:59.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcdeb6de640 1 -- 172.21.15.26:0/3900216520 wait complete. 2024-08-19T12:38:59.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.179+0000 7fcdeb6de640 1 librados: shutdown 2024-08-19T12:38:59.216 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:38:59.217 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-08-19T12:38:59.217 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:38:59.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.317+0000 7f1c4bbaa640 10 monclient: get_monmap_and_config 2024-08-19T12:38:59.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.317+0000 7f1c4bbaa640 10 monclient: build_initial_monmap 2024-08-19T12:38:59.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.317+0000 7f1c4bbaa640 10 monclient: monmap: 2024-08-19T12:38:59.318 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:59.318 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.318 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:59.318128+0000 2024-08-19T12:38:59.318 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:59.318128+0000 2024-08-19T12:38:59.318 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:59.323 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.323 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:59.323 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:59.323 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:59.323 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:59.323 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:59.323 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:59.323 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:59.323 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:59.323 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:59.324 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.317+0000 7f1c4bbaa640 1 Processor -- start 2024-08-19T12:38:59.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4bbaa640 1 -- start start 2024-08-19T12:38:59.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4bbaa640 10 monclient: init 2024-08-19T12:38:59.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4bbaa640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:59.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4bbaa640 10 monclient: _add_conns ranks=[4,2,6,3,1,0,8,7,5] 2024-08-19T12:38:59.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4bbaa640 10 monclient(hunting): picked mon.noname-c con 0x7f1c44073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:59.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4bbaa640 10 monclient(hunting): picked mon.noname-b con 0x7f1c440a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4bbaa640 10 monclient(hunting): picked mon.noname-d con 0x7f1c4413f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4bbaa640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4409dae0 con 0x7f1c440a57a0 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4bbaa640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c44136910 con 0x7f1c44073470 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4bbaa640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4409dd50 con 0x7f1c4413f2e0 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4bbaa640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4bbaa640 10 monclient(hunting): authenticate will time out at 733.405574s 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4a147640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f1c4413f2e0 0x7f1c441416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34320/0 (socket says 172.21.15.26:34320) 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c4a147640 1 -- 172.21.15.26:0/2803153620 learned_addr learned my addr 172.21.15.26:0/2803153620 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.318+0000 7f1c49145640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1c440a57a0 0x7f1c440a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59372/0 (socket says 172.21.15.26:59372) 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.320+0000 7f1c49946640 1 -- 172.21.15.26:0/2803153620 >> v1:172.21.15.26:6791/0 conn(0x7f1c44073470 legacy=0x7f1c44136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.320+0000 7f1c49145640 1 -- 172.21.15.26:0/2803153620 >> v1:172.21.15.26:6790/0 conn(0x7f1c440a57a0 legacy=0x7f1c440a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.320+0000 7f1c4a147640 1 -- 172.21.15.26:0/2803153620 >> v1:172.21.15.26:6792/0 conn(0x7f1c4413f2e0 legacy=0x7f1c441416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 987710525 0 0) 0x7f1c4409dae0 con 0x7f1c440a57a0 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient(hunting): my global_id is 4637 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c2c003520 con 0x7f1c440a57a0 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3558415001 0 0) 0x7f1c4409dd50 con 0x7f1c4413f2e0 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient(hunting): my global_id is 4632 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c4409dae0 con 0x7f1c4413f2e0 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 676738159 0 0) 0x7f1c44136910 con 0x7f1c44073470 2024-08-19T12:38:59.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient(hunting): my global_id is 4567 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c4409dd50 con 0x7f1c44073470 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 218689053 0 0) 0x7f1c2c003520 con 0x7f1c440a57a0 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 >> v1:172.21.15.26:6792/0 conn(0x7f1c4413f2e0 legacy=0x7f1c441416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 >> v1:172.21.15.26:6791/0 conn(0x7f1c44073470 legacy=0x7f1c44136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient: _finish_hunting 0 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 monclient: found mon.noname-b 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4409e030 con 0x7f1c440a57a0 2024-08-19T12:38:59.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient: _finish_auth 0 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient: _check_auth_tickets 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1c38001ae0 con 0x7f1c440a57a0 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c4bbaa640 5 monclient: authenticate success, global_id 4637 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient: dump: 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.337 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c4bbaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c380047e0 con 0x7f1c440a57a0 2024-08-19T12:38:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 1 -- 172.21.15.26:0/2803153620 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1c38004d50 con 0x7f1c440a57a0 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c4bbaa640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c4bbaa640 4 set_mon_vals no callback set 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c48944640 10 monclient: dump: 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.339 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c4bbaa640 10 monclient: shutdown 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c4bbaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/2803153620 >> v1:172.21.15.26:6790/0 conn(0x7f1c440a57a0 legacy=0x7f1c440a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/2803153620 shutdown_connections 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.321+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/2803153620 >> 172.21.15.26:0/2803153620 conn(0x7f1c440abff0 msgr2=0x7f1c440ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:59.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.322+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/2803153620 shutdown_connections 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.322+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/2803153620 wait complete. 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.322+0000 7f1c4bbaa640 10 monclient: build_initial_monmap 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.322+0000 7f1c4bbaa640 10 monclient: monmap: 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:59.341 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.322+0000 7f1c4bbaa640 1 librados: starting msgr at 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.322+0000 7f1c4bbaa640 1 librados: starting objecter 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.322+0000 7f1c4bbaa640 1 Processor -- start 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 1 -- start start 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 1 librados: setting wanted keys 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 1 librados: calling monclient init 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 10 monclient: init 2024-08-19T12:38:59.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 10 monclient: _add_conns ranks=[0,7,6,5,4,8,3,1,2] 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 10 monclient(hunting): picked mon.noname-a con 0x7f1c44073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 10 monclient(hunting): picked mon.noname-i con 0x7f1c440a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 10 monclient(hunting): picked mon.noname-d con 0x7f1c4413f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c440a3810 con 0x7f1c44073470 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c440a9e80 con 0x7f1c4413f2e0 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4413edb0 con 0x7f1c440a57a0 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4bbaa640 10 monclient(hunting): authenticate will time out at 733.410515s 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c49145640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f1c440a57a0 legacy=0x7f1c440a25c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c49946640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1c44073470 0x7f1c440a1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39950/0 (socket says 172.21.15.26:39950) 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c49946640 1 -- 172.21.15.26:0/4011915866 learned_addr learned my addr 172.21.15.26:0/4011915866 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.323+0000 7f1c4a147640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f1c4413f2e0 0x7f1c440a2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34334/0 (socket says 172.21.15.26:34334) 2024-08-19T12:38:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c4a147640 1 -- 172.21.15.26:0/4011915866 >> v1:172.21.15.26:6792/0 conn(0x7f1c4413f2e0 legacy=0x7f1c440a2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 10 client.?.objecter ms_handle_connect 0x7f1c4413f2e0 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 10 client.?.objecter ms_handle_connect 0x7f1c44073470 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c49946640 1 -- 172.21.15.26:0/4011915866 >> v1:172.21.15.26:6789/0 conn(0x7f1c44073470 legacy=0x7f1c440a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1859540100 0 0) 0x7f1c440a9e80 con 0x7f1c4413f2e0 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 10 monclient(hunting): my global_id is 4641 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c30004eb0 con 0x7f1c4413f2e0 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2671134986 0 0) 0x7f1c440a3810 con 0x7f1c44073470 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 10 monclient(hunting): my global_id is 4527 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.325+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c440a9e80 con 0x7f1c44073470 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2049848654 0 0) 0x7f1c30004eb0 con 0x7f1c4413f2e0 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c440a3810 con 0x7f1c4413f2e0 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1c34002030 con 0x7f1c4413f2e0 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient(hunting): dump: 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.345 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:59.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3173362755 0 0) 0x7f1c440a9e80 con 0x7f1c44073470 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c30004eb0 con 0x7f1c44073470 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1c3c003610 con 0x7f1c44073470 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient(hunting): dump: 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.347 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.348 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.348 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.348 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.348 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.348 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.348 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.348 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.348 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3631357410 0 0) 0x7f1c440a3810 con 0x7f1c4413f2e0 2024-08-19T12:38:59.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 >> v1:172.21.15.104:6792/0 conn(0x7f1c440a57a0 legacy=0x7f1c440a25c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:38:59.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 >> v1:172.21.15.26:6789/0 conn(0x7f1c44073470 legacy=0x7f1c440a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient: _finish_hunting 0 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 monclient: found mon.d 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c441373e0 con 0x7f1c4413f2e0 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient: _finish_auth 0 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient: _check_auth_tickets 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c4bbaa640 5 monclient: authenticate success, global_id 4641 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c4bbaa640 10 monclient: _renew_subs 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c4bbaa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1c440a4370 con 0x7f1c4413f2e0 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c4bbaa640 10 client.4641.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c4bbaa640 10 monclient: _renew_subs 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c4bbaa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:38:59.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f1c441d7160 con 0x7f1c4413f2e0 2024-08-19T12:38:59.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c4bbaa640 1 librados: init done 2024-08-19T12:38:59.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c34004e10 con 0x7f1c4413f2e0 2024-08-19T12:38:59.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:59.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1c34005340 con 0x7f1c4413f2e0 2024-08-19T12:38:59.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c437fe640 4 set_mon_vals no callback set 2024-08-19T12:38:59.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:38:59.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.326+0000 7f1c427fc640 10 monclient: dump: 2024-08-19T12:38:59.350 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.350 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.350 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.350 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.327+0000 7f1c4bbaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.327+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1c240065c0 con 0x7f1c24002860 2024-08-19T12:38:59.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.327+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f1c340055e0 con 0x7f1c4413f2e0 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.327+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f1c3404dc90 con 0x7f1c4413f2e0 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.327+0000 7f1c427fc640 10 client.4641.objecter ms_dispatch 0x7f1c44000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.327+0000 7f1c427fc640 3 client.4641.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.327+0000 7f1c427fc640 3 client.4641.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.327+0000 7f1c427fc640 20 client.4641.objecter dump_active .. 0 homeless 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.327+0000 7f1c49145640 10 monclient: get_auth_request con 0x7f1c300414f0 auth_method 0 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.329+0000 7f1c49946640 1 -- 172.21.15.26:0/4011915866 >> v1:172.21.15.104:6790/0 conn(0x7f1c24002860 legacy=0x7f1c24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.329+0000 7f1c427fc640 10 client.4641.objecter ms_handle_connect 0x7f1c24002860 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.329+0000 7f1c427fc640 10 client.4641.objecter resend_mon_ops 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.329+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 407907431 0 0) 0x7f1c240065c0 con 0x7f1c24002860 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.329+0000 7f1c427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.329+0000 7f1c427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.329+0000 7f1c427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.329+0000 7f1c427fc640 10 monclient(hunting): my global_id is 4629 2024-08-19T12:38:59.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.329+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c440a3810 con 0x7f1c24002860 2024-08-19T12:38:59.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2691351883 0 0) 0x7f1c440a3810 con 0x7f1c24002860 2024-08-19T12:38:59.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c240065c0 con 0x7f1c24002860 2024-08-19T12:38:59.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 10 client.4641.objecter ms_handle_connect 0x7f1c300414f0 2024-08-19T12:38:59.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1c3c003e10 con 0x7f1c24002860 2024-08-19T12:38:59.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:59.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 10 monclient: dump: 2024-08-19T12:38:59.353 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.354 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.355 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.355 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3165905548 0 0) 0x7f1c240065c0 con 0x7f1c24002860 2024-08-19T12:38:59.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c2400a160 con 0x7f1c24002860 2024-08-19T12:38:59.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1c3c004080 con 0x7f1c24002860 2024-08-19T12:38:59.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:59.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:59.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 >> v1:172.21.15.104:6790/0 conn(0x7f1c24002860 legacy=0x7f1c24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.330+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 >> v1:172.21.15.104:6790/0 conn(0x7f1c24002860 legacy=0x7f1c24004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:59.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.336+0000 7f1c4bbaa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:59.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.336+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1c24009e50 con 0x7f1c24009110 2024-08-19T12:38:59.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c49946640 1 -- 172.21.15.26:0/4011915866 >> v1:172.21.15.104:6790/0 conn(0x7f1c24009110 legacy=0x7f1c240095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 10 client.4641.objecter ms_handle_connect 0x7f1c24009110 2024-08-19T12:38:59.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 10 client.4641.objecter resend_mon_ops 2024-08-19T12:38:59.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4189363103 0 0) 0x7f1c24009e50 con 0x7f1c24009110 2024-08-19T12:38:59.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 10 monclient(hunting): my global_id is 4638 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c240065c0 con 0x7f1c24009110 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1115598721 0 0) 0x7f1c240065c0 con 0x7f1c24009110 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c24009e50 con 0x7f1c24009110 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1c3c005b00 con 0x7f1c24009110 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 10 monclient: dump: 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.358 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2206686368 0 0) 0x7f1c24009e50 con 0x7f1c24009110 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.339+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1c2400c3f0 con 0x7f1c24009110 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7698 (unknown 0 0 3952767301) 0x7f1c3c007e20 con 0x7f1c24009110 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:59.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c427fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:59.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 >> v1:172.21.15.104:6790/0 conn(0x7f1c24009110 legacy=0x7f1c240095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c427fc640 1 -- 172.21.15.26:0/4011915866 >> v1:172.21.15.104:6790/0 conn(0x7f1c24009110 legacy=0x7f1c240095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:59.360 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:59.360 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:59.360 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:59.360 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:59.360 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 8, 2024-08-19T12:38:59.360 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 52352, 2024-08-19T12:38:59.360 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:59.361 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:59.362 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:59.364 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:59.365 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.366 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:59.367 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.368 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:59.371 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.372 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.374 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:59.374 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:59.374 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.374 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.374 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.374 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:59.374 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:59.374 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.374 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.375 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:59.376 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.377 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.378 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:59.378 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:59.378 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.378 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.378 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.378 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.378 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:59.378 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:59.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.379 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:38:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.380 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:59.380 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 10 librados: watch_flush enter 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 10 librados: watch_flush exit 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 20 client.4641.objecter shutdown clearing up homeless session... 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 10 client.4641.objecter successfully canceled tick 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/4011915866 >> v1:172.21.15.104:6800/899887506 conn(0x7f1c300414f0 legacy=0x7f1c300439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 10 monclient: shutdown 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/4011915866 >> v1:172.21.15.26:6792/0 conn(0x7f1c4413f2e0 legacy=0x7f1c440a2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4a147640 1 -- 172.21.15.26:0/4011915866 reap_dead start 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4a147640 1 -- 172.21.15.26:0/4011915866 reap_dead start 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/4011915866 shutdown_connections 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/4011915866 >> 172.21.15.26:0/4011915866 conn(0x7f1c440abff0 msgr2=0x7f1c441338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:59.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/4011915866 shutdown_connections 2024-08-19T12:38:59.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 1 -- 172.21.15.26:0/4011915866 wait complete. 2024-08-19T12:38:59.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.340+0000 7f1c4bbaa640 1 librados: shutdown 2024-08-19T12:38:59.383 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:38:59.383 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-08-19T12:38:59.383 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:38:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.482+0000 7ffbb1308640 10 monclient: get_monmap_and_config 2024-08-19T12:38:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.482+0000 7ffbb1308640 10 monclient: build_initial_monmap 2024-08-19T12:38:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.482+0000 7ffbb1308640 10 monclient: monmap: 2024-08-19T12:38:59.483 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:38:59.483529+0000 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:38:59.483529+0000 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:59.484 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:59.485 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:59.485 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 1 Processor -- start 2024-08-19T12:38:59.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 1 -- start start 2024-08-19T12:38:59.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 10 monclient: init 2024-08-19T12:38:59.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:59.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 10 monclient: _add_conns ranks=[4,2,8,0,6,7,5,1,3] 2024-08-19T12:38:59.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 10 monclient(hunting): picked mon.noname-c con 0x7ffbac06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:38:59.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 10 monclient(hunting): picked mon.noname-b con 0x7ffbac0a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:38:59.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 10 monclient(hunting): picked mon.noname-e con 0x7ffbac0a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:38:59.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbac136790 con 0x7ffbac0a5920 2024-08-19T12:38:59.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbac09ddf0 con 0x7ffbac06e7b0 2024-08-19T12:38:59.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbac09e0d0 con 0x7ffbac0a5d50 2024-08-19T12:38:59.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:59.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbb1308640 10 monclient(hunting): authenticate will time out at 733.570978s 2024-08-19T12:38:59.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbaaffd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7ffbac06e7b0 0x7ffbac136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:33506/0 (socket says 172.21.15.26:33506) 2024-08-19T12:38:59.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.483+0000 7ffbaaffd640 1 -- 172.21.15.26:0/3718132158 learned_addr learned my addr 172.21.15.26:0/3718132158 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:59.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffbaaffd640 1 -- 172.21.15.26:0/3718132158 >> v1:172.21.15.26:6791/0 conn(0x7ffbac06e7b0 legacy=0x7ffbac136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffbaa7fc640 1 -- 172.21.15.26:0/3718132158 >> v1:172.21.15.26:6790/0 conn(0x7ffbac0a5920 legacy=0x7ffbac13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffbab7fe640 1 -- 172.21.15.26:0/3718132158 >> v1:172.21.15.26:6793/0 conn(0x7ffbac0a5d50 legacy=0x7ffbac141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 833321072 0 0) 0x7ffb9c001450 con 0x7ffbac06e7b0 2024-08-19T12:38:59.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient(hunting): my global_id is 4576 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb94003520 con 0x7ffbac06e7b0 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3096895489 0 0) 0x7ffbac136790 con 0x7ffbac0a5920 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient(hunting): my global_id is 4646 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb9c001450 con 0x7ffbac0a5920 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3840001667 0 0) 0x7ffbac09e0d0 con 0x7ffbac0a5d50 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient(hunting): my global_id is 4625 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbac136790 con 0x7ffbac0a5d50 2024-08-19T12:38:59.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 58490145 0 0) 0x7ffb94003520 con 0x7ffbac06e7b0 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 >> v1:172.21.15.26:6793/0 conn(0x7ffbac0a5d50 legacy=0x7ffbac141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 >> v1:172.21.15.26:6790/0 conn(0x7ffbac0a5920 legacy=0x7ffbac13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 monclient: found mon.noname-c 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbac09e3b0 con 0x7ffbac06e7b0 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient: _finish_auth 0 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffbb1308640 5 monclient: authenticate success, global_id 4576 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffb9c001cb0 con 0x7ffbac06e7b0 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffbb1308640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:59.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient: dump: 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.498 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffbb1308640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffb9c004950 con 0x7ffbac06e7b0 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 1 -- 172.21.15.26:0/3718132158 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffb9c004ec0 con 0x7ffbac06e7b0 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffbb1308640 10 monclient: get_monmap_and_config success 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffbb1308640 4 set_mon_vals no callback set 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.486+0000 7ffba9ffb640 10 monclient: dump: 2024-08-19T12:38:59.499 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.500 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.487+0000 7ffbb1308640 10 monclient: shutdown 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.487+0000 7ffbb1308640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.487+0000 7ffbb1308640 1 -- 172.21.15.26:0/3718132158 >> v1:172.21.15.26:6791/0 conn(0x7ffbac06e7b0 legacy=0x7ffbac136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.487+0000 7ffbb1308640 1 -- 172.21.15.26:0/3718132158 shutdown_connections 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.487+0000 7ffbb1308640 1 -- 172.21.15.26:0/3718132158 >> 172.21.15.26:0/3718132158 conn(0x7ffbac0a8740 msgr2=0x7ffbac0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.487+0000 7ffbb1308640 1 -- 172.21.15.26:0/3718132158 shutdown_connections 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.487+0000 7ffbb1308640 1 -- 172.21.15.26:0/3718132158 wait complete. 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.487+0000 7ffbb1308640 10 monclient: build_initial_monmap 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.487+0000 7ffbb1308640 10 monclient: monmap: 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:38:59.501 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 1 librados: starting msgr at 2024-08-19T12:38:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 1 librados: starting objecter 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 1 Processor -- start 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 1 -- start start 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 1 librados: setting wanted keys 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 1 librados: calling monclient init 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 10 monclient: init 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 10 monclient: _reopen_session rank -1 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 10 monclient: _add_conns ranks=[3,5,1,8,7,2,6,0,4] 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 10 monclient(hunting): picked mon.noname-g con 0x7ffbac06e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 10 monclient(hunting): picked mon.noname-h con 0x7ffbac0a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 10 monclient(hunting): picked mon.noname-f con 0x7ffbac0a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbac141da0 con 0x7ffbac0a5d50 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbac1349b0 con 0x7ffbac06e7b0 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbac1345d0 con 0x7ffbac0a5920 2024-08-19T12:38:59.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 10 monclient(hunting): _renew_subs 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.488+0000 7ffbb1308640 10 monclient(hunting): authenticate will time out at 733.575911s 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.489+0000 7ffbaaffd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7ffbac06e7b0 0x7ffbac137430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:39242/0 (socket says 172.21.15.26:39242) 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.489+0000 7ffbaaffd640 1 -- 172.21.15.26:0/1627363613 learned_addr learned my addr 172.21.15.26:0/1627363613 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffbaaffd640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6790/0 conn(0x7ffbac06e7b0 legacy=0x7ffbac137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 client.?.objecter ms_handle_connect 0x7ffbac06e7b0 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 client.?.objecter ms_handle_connect 0x7ffbac0a5920 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffbaa7fc640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6791/0 conn(0x7ffbac0a5920 legacy=0x7ffbac137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 client.?.objecter ms_handle_connect 0x7ffbac0a5d50 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffbab7fe640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6789/0 conn(0x7ffbac0a5d50 legacy=0x7ffbac1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 141326225 0 0) 0x7ffbac09e3b0 con 0x7ffbac06e7b0 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): my global_id is 4647 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb98004eb0 con 0x7ffbac06e7b0 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2645654885 0 0) 0x7ffba0001500 con 0x7ffbac0a5920 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): my global_id is 4505 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbac09e3b0 con 0x7ffbac0a5920 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3773623466 0 0) 0x7ffbac141da0 con 0x7ffbac0a5d50 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): my global_id is 4528 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffba0001500 con 0x7ffbac0a5d50 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2439996283 0 0) 0x7ffbac09e3b0 con 0x7ffbac0a5920 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffbac141da0 con 0x7ffbac0a5920 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffba0003950 con 0x7ffbac0a5920 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): dump: 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.506 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4238941888 0 0) 0x7ffba0001500 con 0x7ffbac0a5d50 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffbac09e3b0 con 0x7ffbac0a5d50 2024-08-19T12:38:59.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2487056232 0 0) 0x7ffb98004eb0 con 0x7ffbac06e7b0 2024-08-19T12:38:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffba0001500 con 0x7ffbac06e7b0 2024-08-19T12:38:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffba4002030 con 0x7ffbac0a5d50 2024-08-19T12:38:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:38:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): dump: 2024-08-19T12:38:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.508 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffb9c004e40 con 0x7ffbac06e7b0 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient(hunting): dump: 2024-08-19T12:38:59.509 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.510 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 961485399 0 0) 0x7ffbac141da0 con 0x7ffbac0a5920 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6790/0 conn(0x7ffbac06e7b0 legacy=0x7ffbac137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6789/0 conn(0x7ffbac0a5d50 legacy=0x7ffbac1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient: _finish_hunting 0 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 monclient: found mon.h 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbac138250 con 0x7ffbac0a5920 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient: _finish_auth 0 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 10 monclient: _check_auth_tickets 2024-08-19T12:38:59.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffb937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffbb1308640 5 monclient: authenticate success, global_id 4505 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffbb1308640 10 monclient: _renew_subs 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffbb1308640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffbb1308640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffbac13ae30 con 0x7ffbac0a5920 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffbb1308640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffbb1308640 10 monclient: _renew_subs 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffbb1308640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffbb1308640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7ffbac139b80 con 0x7ffbac0a5920 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.491+0000 7ffbb1308640 1 librados: init done 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.492+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffba0003da0 con 0x7ffbac0a5920 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.492+0000 7ffb937fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.492+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffba0004310 con 0x7ffbac0a5920 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.492+0000 7ffb937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.492+0000 7ffb937fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.492+0000 7ffba97fa640 4 set_mon_vals no callback set 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.492+0000 7ffb937fe640 10 monclient: dump: 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.513 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.492+0000 7ffbb1308640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.492+0000 7ffbb1308640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ffb8c0065c0 con 0x7ffb8c002860 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.492+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ffba0011a80 con 0x7ffbac0a5920 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.493+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7ffba004dc00 con 0x7ffbac0a5920 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.493+0000 7ffb937fe640 10 client.4505.objecter ms_dispatch 0x7ffbac000b90 osd_map(20..20 src has 1..20) 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.493+0000 7ffb937fe640 3 client.4505.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.493+0000 7ffb937fe640 3 client.4505.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.493+0000 7ffb937fe640 20 client.4505.objecter dump_active .. 0 homeless 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.493+0000 7ffbab7fe640 10 monclient: get_auth_request con 0x7ffb980419e0 auth_method 0 2024-08-19T12:38:59.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.494+0000 7ffbaaffd640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6791/0 conn(0x7ffb8c002860 legacy=0x7ffb8c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.494+0000 7ffb937fe640 10 client.4505.objecter ms_handle_connect 0x7ffb8c002860 2024-08-19T12:38:59.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.494+0000 7ffb937fe640 10 client.4505.objecter resend_mon_ops 2024-08-19T12:38:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 395528634 0 0) 0x7ffb8c0065c0 con 0x7ffb8c002860 2024-08-19T12:38:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 10 monclient(hunting): my global_id is 4514 2024-08-19T12:38:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbac141da0 con 0x7ffb8c002860 2024-08-19T12:38:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3091954470 0 0) 0x7ffbac141da0 con 0x7ffb8c002860 2024-08-19T12:38:59.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb8c0065c0 con 0x7ffb8c002860 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffb9c004950 con 0x7ffb8c002860 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 10 monclient: dump: 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.518 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 736216862 0 0) 0x7ffb8c0065c0 con 0x7ffb8c002860 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffb8c00a160 con 0x7ffb8c002860 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 10 client.4505.objecter ms_handle_connect 0x7ffb980419e0 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ffb9c005540 con 0x7ffb8c002860 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:38:59.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6791/0 conn(0x7ffb8c002860 legacy=0x7ffb8c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.495+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6791/0 conn(0x7ffb8c002860 legacy=0x7ffb8c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.501+0000 7ffbb1308640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.502+0000 7ffbb1308640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ffb8c009e50 con 0x7ffb8c009110 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffbaaffd640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6791/0 conn(0x7ffb8c009110 legacy=0x7ffb8c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 10 client.4505.objecter ms_handle_connect 0x7ffb8c009110 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 10 client.4505.objecter resend_mon_ops 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1720395586 0 0) 0x7ffb8c009e50 con 0x7ffb8c009110 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 10 monclient(hunting): _negotiate 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 10 monclient(hunting): my global_id is 4523 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb8c0065c0 con 0x7ffb8c009110 2024-08-19T12:38:59.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 542549488 0 0) 0x7ffb8c0065c0 con 0x7ffb8c009110 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb8c009e50 con 0x7ffb8c009110 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffb9c005d20 con 0x7ffb8c009110 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 10 monclient: dump: 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:38:59.521 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:38:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:38:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:38:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:38:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:38:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:38:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:38:59.522 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:38:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2922465140 0 0) 0x7ffb8c009e50 con 0x7ffb8c009110 2024-08-19T12:38:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.504+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffb8c00c3f0 con 0x7ffb8c009110 2024-08-19T12:38:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7698 (unknown 0 0 78550145) 0x7ffb9c007150 con 0x7ffb8c009110 2024-08-19T12:38:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffb937fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:38:59.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffb937fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:38:59.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6791/0 conn(0x7ffb8c009110 legacy=0x7ffb8c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffb937fe640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6791/0 conn(0x7ffb8c009110 legacy=0x7ffb8c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:38:59.526 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:38:59.526 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:59.526 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 8, 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 52527, 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:38:59.527 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:59.528 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:59.529 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:59.530 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.530 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:38:59.530 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:38:59.530 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:38:59.530 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:38:59.530 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:38:59.530 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:38:59.530 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:59.530 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:38:59.530 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:38:59.530 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:38:59.531 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.533 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:38:59.534 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.535 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.536 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:38:59.538 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:38:59.542 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:38:59.543 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.544 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:38:59.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:38:59.546 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:38:59.546 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:38:59.546 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:38:59.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:38:59.546 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:38:59.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:38:59.546 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:38:59.546 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:38:59.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 10 librados: watch_flush enter 2024-08-19T12:38:59.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 10 librados: watch_flush exit 2024-08-19T12:38:59.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 20 client.4505.objecter shutdown clearing up homeless session... 2024-08-19T12:38:59.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 10 client.4505.objecter successfully canceled tick 2024-08-19T12:38:59.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6800/899887506 conn(0x7ffb980419e0 legacy=0x7ffb98043ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 10 monclient: shutdown 2024-08-19T12:38:59.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:38:59.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 1 -- 172.21.15.26:0/1627363613 >> v1:172.21.15.104:6791/0 conn(0x7ffbac0a5920 legacy=0x7ffbac137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:38:59.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbab7fe640 1 -- 172.21.15.26:0/1627363613 reap_dead start 2024-08-19T12:38:59.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbab7fe640 1 -- 172.21.15.26:0/1627363613 reap_dead start 2024-08-19T12:38:59.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 1 -- 172.21.15.26:0/1627363613 shutdown_connections 2024-08-19T12:38:59.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 1 -- 172.21.15.26:0/1627363613 >> 172.21.15.26:0/1627363613 conn(0x7ffbac0a8740 msgr2=0x7ffbac0aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:38:59.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 1 -- 172.21.15.26:0/1627363613 shutdown_connections 2024-08-19T12:38:59.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 1 -- 172.21.15.26:0/1627363613 wait complete. 2024-08-19T12:38:59.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:38:59.505+0000 7ffbb1308640 1 librados: shutdown 2024-08-19T12:38:59.549 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:38:59.549 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T12:39:34.681 INFO:tasks.workunit.client.0.smithi104.stderr:Note: switching to '964adda23da9f7b057cb34fdc0b82aeb7ed65324'. 2024-08-19T12:39:34.681 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-08-19T12:39:34.681 INFO:tasks.workunit.client.0.smithi104.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr:state without impacting any branches by switching back to a branch. 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr: git switch -c 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr:Or undo this operation with: 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr: git switch - 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-08-19T12:39:34.682 INFO:tasks.workunit.client.0.smithi104.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-19T12:39:34.683 INFO:tasks.workunit.client.0.smithi104.stderr: 2024-08-19T12:39:34.683 INFO:tasks.workunit.client.0.smithi104.stderr:HEAD is now at 964adda23da Merge pull request #59290 from anthonyeleven/mountpoint 2024-08-19T12:39:34.683 DEBUG:teuthology.orchestra.run.smithi104:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-19T12:39:34.741 INFO:tasks.workunit.client.0.smithi104.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-19T12:39:34.743 INFO:tasks.workunit.client.0.smithi104.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-19T12:39:34.743 INFO:tasks.workunit.client.0.smithi104.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-19T12:39:35.269 INFO:tasks.workunit.client.0.smithi104.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-19T12:39:35.322 INFO:tasks.workunit.client.0.smithi104.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-19T12:39:35.360 INFO:tasks.workunit.client.0.smithi104.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-19T12:39:35.362 INFO:tasks.workunit.client.0.smithi104.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-19T12:39:35.362 INFO:tasks.workunit.client.0.smithi104.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-19T12:39:35.400 INFO:tasks.workunit.client.0.smithi104.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-19T12:39:35.403 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-08-19T12:39:35.403 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-19T12:39:35.419 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-08-19T12:39:35.420 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-08-19T12:39:35.420 DEBUG:teuthology.orchestra.run.smithi104:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-08-19T12:39:35.479 INFO:tasks.workunit.client.0.smithi104.stderr:+ parallel=1 2024-08-19T12:39:35.479 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' '' = --serial ']' 2024-08-19T12:39:35.479 INFO:tasks.workunit.client.0.smithi104.stderr:+ crimson=0 2024-08-19T12:39:35.479 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' '' = --crimson ']' 2024-08-19T12:39:35.479 INFO:tasks.workunit.client.0.smithi104.stderr:+ color= 2024-08-19T12:39:35.479 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -t 1 ']' 2024-08-19T12:39:35.479 INFO:tasks.workunit.client.0.smithi104.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-08-19T12:39:35.480 INFO:tasks.workunit.client.0.smithi104.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-19T12:39:35.480 INFO:tasks.workunit.client.0.smithi104.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-19T12:39:35.481 INFO:tasks.workunit.client.0.smithi104.stderr:+ declare -A pids 2024-08-19T12:39:35.481 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.481 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.481 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_aio 2024-08-19T12:39:35.481 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_aio' 2024-08-19T12:39:35.481 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_aio 2024-08-19T12:39:35.481 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.482 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_aio 2024-08-19T12:39:35.482 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24430 2024-08-19T12:39:35.483 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_aio on pid 24430' 2024-08-19T12:39:35.483 INFO:tasks.workunit.client.0.smithi104.stdout:test api_aio on pid 24430 2024-08-19T12:39:35.486 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24430 2024-08-19T12:39:35.486 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.486 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.487 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-08-19T12:39:35.487 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_aio_pp 2024-08-19T12:39:35.487 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_aio_pp' 2024-08-19T12:39:35.487 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_aio_pp 2024-08-19T12:39:35.487 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.487 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.487 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.487 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.487 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.488 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.488 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.488 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.488 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.488 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.488 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.488 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.488 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.488 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.488 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.488 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_aio_pp 2024-08-19T12:39:35.489 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24437 2024-08-19T12:39:35.490 INFO:tasks.workunit.client.0.smithi104.stdout:test api_aio_pp on pid 24437 2024-08-19T12:39:35.496 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_aio_pp on pid 24437' 2024-08-19T12:39:35.496 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24437 2024-08-19T12:39:35.496 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.496 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.497 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-08-19T12:39:35.497 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_io 2024-08-19T12:39:35.497 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_io' 2024-08-19T12:39:35.497 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_io 2024-08-19T12:39:35.497 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.498 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.498 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.498 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.498 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.498 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.498 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.498 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.498 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.498 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.498 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.498 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.499 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.499 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.499 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.499 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.499 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.499 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.499 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.499 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.499 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.499 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.500 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.500 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.500 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.500 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.500 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.500 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.500 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.500 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.500 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.500 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.500 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.500 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.501 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.501 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.501 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.501 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.501 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.501 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.501 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.501 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.501 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.501 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.501 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.502 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.502 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_io 2024-08-19T12:39:35.502 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24446 2024-08-19T12:39:35.502 INFO:tasks.workunit.client.0.smithi104.stdout:test api_io on pid 24446 2024-08-19T12:39:35.510 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_io on pid 24446' 2024-08-19T12:39:35.510 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24446 2024-08-19T12:39:35.510 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.510 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.510 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-08-19T12:39:35.511 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_io_pp 2024-08-19T12:39:35.511 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_io_pp' 2024-08-19T12:39:35.511 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_io_pp 2024-08-19T12:39:35.511 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.511 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.511 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.511 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.512 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.512 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.512 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.512 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.512 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.512 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.512 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.512 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.512 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.512 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.512 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.513 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.513 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.513 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.513 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.513 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.513 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.513 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.514 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.514 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.514 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.514 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.514 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.514 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.514 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.514 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.515 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.515 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.515 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.515 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.515 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.515 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.515 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.515 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.515 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.515 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.515 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.515 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.516 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.517 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_io_pp 2024-08-19T12:39:35.517 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24458 2024-08-19T12:39:35.517 INFO:tasks.workunit.client.0.smithi104.stdout:test api_io_pp on pid 24458 2024-08-19T12:39:35.524 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_io_pp on pid 24458' 2024-08-19T12:39:35.524 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24458 2024-08-19T12:39:35.524 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.524 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.524 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-08-19T12:39:35.524 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_asio 2024-08-19T12:39:35.524 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_asio' 2024-08-19T12:39:35.524 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.524 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.524 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.524 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_asio 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.525 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.526 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.527 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.528 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.529 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_asio 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24472 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-08-19T12:39:35.530 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_asio on pid 24472' 2024-08-19T12:39:35.531 INFO:tasks.workunit.client.0.smithi104.stdout:test api_asio on pid 24472 2024-08-19T12:39:35.537 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24472 2024-08-19T12:39:35.538 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.538 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.538 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_list 2024-08-19T12:39:35.538 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_list' 2024-08-19T12:39:35.538 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.538 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.538 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.538 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.538 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.538 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.539 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.539 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_list 2024-08-19T12:39:35.539 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.539 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.539 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.539 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.539 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.539 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.539 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.539 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.540 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.540 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.540 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.540 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.540 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.540 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.540 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.540 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.540 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.540 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.540 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.541 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24430/exe 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.542 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.543 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.543 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.543 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.543 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.543 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.543 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.543 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.543 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.543 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.543 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.544 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.545 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_list 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.546 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.547 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24488 2024-08-19T12:39:35.547 INFO:tasks.workunit.client.0.smithi104.stdout:test api_list on pid 24488 2024-08-19T12:39:35.555 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_list on pid 24488' 2024-08-19T12:39:35.555 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.555 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24488 2024-08-19T12:39:35.555 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_lock 2024-08-19T12:39:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.557 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-08-19T12:39:35.557 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.557 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_lock' 2024-08-19T12:39:35.557 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_aio.log 2024-08-19T12:39:35.557 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_aio: /' 2024-08-19T12:39:35.557 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_lock 2024-08-19T12:39:35.557 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-08-19T12:39:35.557 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.557 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.557 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.557 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.557 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.558 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.559 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.560 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.561 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.562 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.562 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.562 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.562 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.562 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.562 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.562 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.562 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.562 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.563 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.564 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.565 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24437/exe 2024-08-19T12:39:35.565 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.565 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.565 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.565 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.565 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.565 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.565 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.565 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_lock 2024-08-19T12:39:35.565 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24512 2024-08-19T12:39:35.565 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_lock on pid 24512' 2024-08-19T12:39:35.565 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-08-19T12:39:35.566 INFO:tasks.workunit.client.0.smithi104.stdout:test api_lock on pid 24512 2024-08-19T12:39:35.574 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24512 2024-08-19T12:39:35.574 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.575 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.575 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.575 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.575 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.575 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.575 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.575 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.575 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.575 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_lock_pp 2024-08-19T12:39:35.575 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.575 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.575 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_lock_pp' 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_lock_pp 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.576 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.577 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_aio_pp: /' 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.578 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.579 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.579 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.579 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.579 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.579 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.579 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.579 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.579 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.579 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.579 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.579 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.579 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.581 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.581 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.581 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.581 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.581 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.581 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.581 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.581 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.582 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24446/exe 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.583 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.584 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.584 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.584 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.584 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.584 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.584 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.584 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.584 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.584 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.585 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.585 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_lock_pp 2024-08-19T12:39:35.585 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.585 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.585 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24528 2024-08-19T12:39:35.585 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_lock_pp on pid 24528' 2024-08-19T12:39:35.585 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.585 INFO:tasks.workunit.client.0.smithi104.stdout:test api_lock_pp on pid 24528 2024-08-19T12:39:35.595 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24528 2024-08-19T12:39:35.595 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.595 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.595 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.595 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.595 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.595 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.595 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-08-19T12:39:35.595 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.595 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.595 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_misc 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_misc' 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.596 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.597 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.598 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.598 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.598 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.598 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.598 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_misc 2024-08-19T12:39:35.598 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.598 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.598 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.598 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.598 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.598 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.598 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.599 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.600 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.600 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.600 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.600 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.600 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.600 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.600 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.600 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.600 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.600 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.600 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.600 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_io.log 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.601 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.602 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.603 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_io: /' 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_misc 2024-08-19T12:39:35.604 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24549 2024-08-19T12:39:35.605 INFO:tasks.workunit.client.0.smithi104.stdout:test api_misc on pid 24549 2024-08-19T12:39:35.618 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_misc on pid 24549' 2024-08-19T12:39:35.618 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24549 2024-08-19T12:39:35.618 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.618 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.618 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.618 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_misc_pp 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_misc_pp' 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24458/exe 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.619 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.620 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.620 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.620 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-08-19T12:39:35.620 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.620 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.620 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.620 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.620 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.620 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.620 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.620 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.620 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.621 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.622 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24472/exe 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_io_pp: /' 2024-08-19T12:39:35.623 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.624 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.625 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.626 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.627 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_asio: /' 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.628 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_misc_pp 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.629 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_asio.log 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.630 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_misc_pp 2024-08-19T12:39:35.631 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.631 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.631 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24575 2024-08-19T12:39:35.631 INFO:tasks.workunit.client.0.smithi104.stdout:test api_misc_pp on pid 24575 2024-08-19T12:39:35.635 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_misc_pp on pid 24575' 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24575 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_tier_pp 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_tier_pp' 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.636 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.637 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.638 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_tier_pp 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_tier_pp 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24588 2024-08-19T12:39:35.639 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_tier_pp on pid 24588' 2024-08-19T12:39:35.640 INFO:tasks.workunit.client.0.smithi104.stdout:test api_tier_pp on pid 24588 2024-08-19T12:39:35.646 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24588 2024-08-19T12:39:35.646 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.646 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.646 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.646 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.646 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_pool 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.647 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.648 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_pool' 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24488/exe 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.649 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.650 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_pool 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.651 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_pool 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24601 2024-08-19T12:39:35.652 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_pool on pid 24601' 2024-08-19T12:39:35.653 INFO:tasks.workunit.client.0.smithi104.stdout:test api_pool on pid 24601 2024-08-19T12:39:35.669 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24601 2024-08-19T12:39:35.669 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.669 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.669 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-08-19T12:39:35.669 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-08-19T12:39:35.669 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.669 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.669 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.669 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.669 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.669 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_list.log 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.670 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.671 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_list: /' 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.672 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.673 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.674 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.675 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24528/exe 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_snapshots 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_snapshots' 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.676 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_snapshots 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.677 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.678 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.678 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.678 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.678 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.678 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.678 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.678 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.678 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.678 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.678 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.678 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.678 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.679 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.680 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.680 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.680 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.680 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24512/exe 2024-08-19T12:39:35.680 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_lock_pp: /' 2024-08-19T12:39:35.680 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.680 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.680 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.680 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.680 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.680 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.680 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.681 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_snapshots 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.682 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.683 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.683 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.683 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24656 2024-08-19T12:39:35.683 INFO:tasks.workunit.client.0.smithi104.stdout:test api_snapshots on pid 24656 2024-08-19T12:39:35.700 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_snapshots on pid 24656' 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24656 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_snapshots_pp 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.701 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_lock: /' 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.702 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.703 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_lock.log 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.704 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.705 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.706 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.706 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.706 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.706 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-08-19T12:39:35.706 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.706 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.706 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.706 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.706 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.706 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.706 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.706 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.707 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.708 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.708 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.708 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.708 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_snapshots_pp' 2024-08-19T12:39:35.708 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.708 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.708 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.708 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.708 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.708 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.708 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.708 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.709 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.710 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.711 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.711 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.711 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.711 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.711 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.711 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.711 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.711 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.711 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.711 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-08-19T12:39:35.711 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.711 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.712 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.713 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.713 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.713 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.713 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.713 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.713 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.713 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.713 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.713 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.713 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.713 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.713 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.714 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.714 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.714 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.714 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.714 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.714 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.714 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.714 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.714 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.714 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.714 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.714 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.715 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.715 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.715 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.715 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.715 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.715 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.715 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.715 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24575/exe 2024-08-19T12:39:35.715 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.715 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.715 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_snapshots_pp 2024-08-19T12:39:35.715 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.716 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.716 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.716 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.716 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.716 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.716 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.716 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.716 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.716 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.716 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.716 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.716 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.717 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.717 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.717 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.717 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.717 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.717 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.717 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.717 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.717 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.717 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.717 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.717 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_snapshots_pp 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.718 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.719 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.719 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.719 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.719 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.719 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.719 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24725 2024-08-19T12:39:35.719 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.719 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_snapshots_pp on pid 24725' 2024-08-19T12:39:35.719 INFO:tasks.workunit.client.0.smithi104.stdout:test api_snapshots_pp on pid 24725 2024-08-19T12:39:35.725 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24725 2024-08-19T12:39:35.725 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.726 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.726 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.726 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.726 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.726 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.726 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.726 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.726 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.726 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.726 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.726 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.726 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.727 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.728 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24588/exe 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.729 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_misc_pp: /' 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24549/exe 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_stat 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24601/exe 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_stat' 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.730 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.731 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.731 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.731 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.731 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.731 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.731 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.731 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.731 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.731 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.731 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.731 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [----------] Global test environment set-up. 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [----------] 12 tests from AsioRados 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.570+0000 7f328153a780 10 monclient: get_monmap_and_config 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.570+0000 7f328153a780 10 monclient: build_initial_monmap 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.570+0000 7f328153a780 10 monclient: monmap: 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: epoch 0 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: last_changed 0.000000 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: created 0.000000 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: min_mon_release 0 (unknown) 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: election_strategy: 1 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:39:35.732 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.570+0000 7f328153a780 1 Processor -- start 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.570+0000 7f328153a780 1 -- start start 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.570+0000 7f328153a780 10 monclient: init 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f328153a780 10 monclient: _reopen_session rank -1 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f328153a780 10 monclient: _add_conns ranks=[5,7,8,1,6,0,3,2,4] 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f328153a780 10 monclient(hunting): picked mon.noname-h con 0x56032da3e470 addr v1:172.21.15.104:6791/0 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f328153a780 10 monclient(hunting): picked mon.noname-i con 0x56032da3e850 addr v1:172.21.15.104:6792/0 2024-08-19T12:39:35.733 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f328153a780 10 monclient(hunting): picked mon.noname-e con 0x56032db08bb0 addr v1:172.21.15.26:6793/0 2024-08-19T12:39:35.734 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f328153a780 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x56032db069b0 con 0x56032da3e470 2024-08-19T12:39:35.734 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f328153a780 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x56032da42580 con 0x56032da3e850 2024-08-19T12:39:35.734 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f328153a780 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x56032db03970 con 0x56032db08bb0 2024-08-19T12:39:35.734 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f328153a780 10 monclient(hunting): _renew_subs 2024-08-19T12:39:35.734 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f328153a780 10 monclient(hunting): authenticate will time out at 739.600469s 2024-08-19T12:39:35.734 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f327e828640 1 -- >> v1:172.21.15.104:6792/0 conn(0x56032da3e850 legacy=0x56032db062a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:39:35.734 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f327f029640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x56032da3e470 0x56032db05b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.104:52404/0 (socket says 172.21.15.104:52404) 2024-08-19T12:39:35.734 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.571+0000 7f327f029640 1 -- 172.21.15.104:0/1038722137 learned_addr learned my addr 172.21.15.104:0/1038722137 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:39:35.736 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.736 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.736 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.736 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.736 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.736 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.736 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.736 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_stat 2024-08-19T12:39:35.736 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.736 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.737 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.737 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.737 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.737 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.737 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.737 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.737 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.737 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.737 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327f029640 1 -- 172.21.15.104:0/1038722137 >> v1:172.21.15.104:6791/0 conn(0x56032da3e470 legacy=0x56032db05b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:39:35.737 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 1 -- 172.21.15.104:0/1038722137 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 653705493 0 0) 0x56032db069b0 con 0x56032da3e470 2024-08-19T12:39:35.737 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 10 monclient(hunting): _negotiate 2024-08-19T12:39:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:39:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:39:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 10 monclient(hunting): my global_id is 4568 2024-08-19T12:39:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 1 -- 172.21.15.104:0/1038722137 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f326c001e10 con 0x56032da3e470 2024-08-19T12:39:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 1 -- 172.21.15.104:0/1038722137 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3954460925 0 0) 0x7f326c001e10 con 0x56032da3e470 2024-08-19T12:39:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 1 -- 172.21.15.104:0/1038722137 >> v1:172.21.15.26:6793/0 conn(0x56032db08bb0 legacy=0x56032db97d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T12:39:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 1 -- 172.21.15.104:0/1038722137 >> v1:172.21.15.104:6792/0 conn(0x56032da3e850 legacy=0x56032db062a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:39:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 10 monclient: _finish_hunting 0 2024-08-19T12:39:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 1 monclient: found mon.noname-h 2024-08-19T12:39:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:39:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:39:35.738 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 1 -- 172.21.15.104:0/1038722137 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x56032da41d40 con 0x56032da3e470 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 10 monclient: _finish_auth 0 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 10 monclient: _check_auth_tickets 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 1 -- 172.21.15.104:0/1038722137 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3270001c00 con 0x56032da3e470 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f327d826640 10 monclient: dump: 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: epoch 1 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: min_mon_release 19 (squid) 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: election_strategy: 1 2024-08-19T12:39:35.739 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:39:35.740 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:39:35.740 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:39:35.740 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:39:35.740 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:39:35.740 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:39:35.740 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:39:35.740 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:39:35.742 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_pool: /' 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.743 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.744 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.744 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.744 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.744 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.744 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.744 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.744 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.744 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.744 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.744 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.744 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.744 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_tier_pp: /' 2024-08-19T12:39:35.745 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:39:35.745 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.745 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f328153a780 5 monclient: authenticate success, global_id 4568 2024-08-19T12:39:35.745 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.574+0000 7f328153a780 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:39:35.745 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.575+0000 7f327d826640 1 -- 172.21.15.104:0/1038722137 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3270001f40 con 0x56032da3e470 2024-08-19T12:39:35.745 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.575+0000 7f327d826640 10 monclient: handle_config config(0 keys) 2024-08-19T12:39:35.745 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.575+0000 7f327d826640 1 -- 172.21.15.104:0/1038722137 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f32700024b0 con 0x56032da3e470 2024-08-19T12:39:35.745 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.575+0000 7f327d826640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:39:35.745 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.575+0000 7f327d826640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:39:35.745 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.575+0000 7f327d826640 10 monclient: dump: 2024-08-19T12:39:35.745 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: epoch 1 2024-08-19T12:39:35.745 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: min_mon_release 19 (squid) 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: election_strategy: 1 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:39:35.746 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.575+0000 7f328153a780 10 monclient: get_monmap_and_config success 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.575+0000 7f328153a780 4 set_mon_vals no callback set 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.576+0000 7f328153a780 10 monclient: shutdown 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.576+0000 7f328153a780 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.576+0000 7f328153a780 1 -- 172.21.15.104:0/1038722137 >> v1:172.21.15.104:6791/0 conn(0x56032da3e470 legacy=0x56032db05b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.576+0000 7f328153a780 1 -- 172.21.15.104:0/1038722137 shutdown_connections 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.576+0000 7f328153a780 1 -- 172.21.15.104:0/1038722137 >> 172.21.15.104:0/1038722137 conn(0x56032da70520 msgr2=0x56032da410f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.576+0000 7f328153a780 1 -- 172.21.15.104:0/1038722137 shutdown_connections 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.576+0000 7f328153a780 1 -- 172.21.15.104:0/1038722137 wait complete. 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.576+0000 7f328153a780 10 monclient: build_initial_monmap 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.576+0000 7f328153a780 10 monclient: monmap: 2024-08-19T12:39:35.747 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: epoch 0 2024-08-19T12:39:35.748 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:39:35.748 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: last_changed 0.000000 2024-08-19T12:39:35.748 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: created 0.000000 2024-08-19T12:39:35.748 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: min_mon_release 0 (unknown) 2024-08-19T12:39:35.748 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: election_strategy: 1 2024-08-19T12:39:35.748 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:39:35.748 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:39:35.748 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:39:35.749 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-08-19T12:39:35.749 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-08-19T12:39:35.749 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.749 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.749 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.750 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.750 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.750 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_stat 2024-08-19T12:39:35.750 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.750 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.750 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.750 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24776 2024-08-19T12:39:35.750 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: test api_stat on pid 24776 2024-08-19T12:39:35.752 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_stat on pid 24776' 2024-08-19T12:39:35.752 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24776 2024-08-19T12:39:35.752 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.752 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.753 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_pool.log 2024-08-19T12:39:35.753 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.753 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.753 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.753 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.753 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.753 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.753 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.753 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.753 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.753 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.753 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.754 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.754 INFO:tasks.workunit.client.0.smithi104.stdout:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:39:35.754 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:39:35.754 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:39:35.754 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.577+0000 7f328153a780 1 librados: starting msgr at 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.577+0000 7f328153a780 1 librados: starting objecter 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.577+0000 7f328153a780 1 Processor -- start 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.577+0000 7f328153a780 1 -- start start 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.577+0000 7f328153a780 1 librados: setting wanted keys 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.577+0000 7f328153a780 1 librados: calling monclient init 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.577+0000 7f328153a780 10 monclient: init 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f328153a780 10 monclient: _reopen_session rank -1 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f328153a780 10 monclient: _add_conns ranks=[3,7,2,0,4,5,1,8,6] 2024-08-19T12:39:35.755 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f328153a780 10 monclient(hunting): picked mon.noname-g con 0x56032da3e850 addr v1:172.21.15.104:6790/0 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f328153a780 10 monclient(hunting): picked mon.noname-i con 0x56032db08bb0 addr v1:172.21.15.104:6792/0 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f328153a780 10 monclient(hunting): picked mon.noname-b con 0x56032da6e580 addr v1:172.21.15.26:6790/0 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f328153a780 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x56032da6da20 con 0x56032da6e580 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f328153a780 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x56032da6f4e0 con 0x56032da3e850 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f328153a780 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x56032da6f7c0 con 0x56032db08bb0 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f328153a780 10 monclient(hunting): _renew_subs 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f328153a780 10 monclient(hunting): authenticate will time out at 739.607559s 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f327e828640 1 -- >> v1:172.21.15.104:6792/0 conn(0x56032db08bb0 legacy=0x56032db9f6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f327f029640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x56032da3e850 0x56032db9efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:48324/0 (socket says 172.21.15.104:48324) 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.578+0000 7f327f029640 1 -- 172.21.15.104:0/1047516331 learned_addr learned my addr 172.21.15.104:0/1047516331 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327f029640 1 -- 172.21.15.104:0/1047516331 >> v1:172.21.15.104:6790/0 conn(0x56032da3e850 legacy=0x56032db9efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 client.?.objecter ms_handle_connect 0x56032da3e850 2024-08-19T12:39:35.756 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 client.?.objecter resend_mon_ops 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 >> v1:172.21.15.26:6790/0 conn(0x56032da6e580 legacy=0x56032dba7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 client.?.objecter ms_handle_connect 0x56032da6e580 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 client.?.objecter resend_mon_ops 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3785725191 0 0) 0x56032da6f4e0 con 0x56032da3e850 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): _negotiate 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): my global_id is 4683 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f326c001e10 con 0x56032da3e850 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3531351271 0 0) 0x56032da6da20 con 0x56032da6e580 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): _negotiate 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:39:35.757 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): my global_id is 4691 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x56032da6f4e0 con 0x56032da6e580 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3135639482 0 0) 0x56032da6f4e0 con 0x56032da6e580 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x56032da6da20 con 0x56032da6e580 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3274001ca0 con 0x56032da6e580 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): dump: 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: epoch 1 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: min_mon_release 19 (squid) 2024-08-19T12:39:35.758 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: election_strategy: 1 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3732554611 0 0) 0x7f326c001e10 con 0x56032da3e850 2024-08-19T12:39:35.759 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x56032da6f4e0 con 0x56032da3e850 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3270002240 con 0x56032da3e850 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.581+0000 7f327c824640 10 monclient(hunting): dump: 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: epoch 1 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: min_mon_release 19 (squid) 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: election_strategy: 1 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:39:35.760 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2297143136 0 0) 0x56032da6f4e0 con 0x56032da3e850 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 >> v1:172.21.15.104:6792/0 conn(0x56032db08bb0 legacy=0x56032db9f6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 >> v1:172.21.15.26:6790/0 conn(0x56032da6e580 legacy=0x56032dba7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f327c824640 10 monclient: _finish_hunting 0 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f327c824640 1 monclient: found mon.g 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f327c824640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:39:35.761 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f327c824640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:39:35.762 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x56032daf2d90 con 0x56032da3e850 2024-08-19T12:39:35.762 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f327c824640 10 monclient: _finish_auth 0 2024-08-19T12:39:35.762 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f327c824640 10 monclient: _check_auth_tickets 2024-08-19T12:39:35.762 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f327c824640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:39:35.762 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f328153a780 5 monclient: authenticate success, global_id 4683 2024-08-19T12:39:35.762 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f328153a780 10 monclient: _renew_subs 2024-08-19T12:39:35.762 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f328153a780 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:39:35.763 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-08-19T12:39:35.763 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.763 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.763 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.764 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.764 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.764 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.764 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.764 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.764 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.764 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x56032da6d420 con 0x56032da3e850 2024-08-19T12:39:35.764 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f328153a780 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:39:35.764 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f328153a780 10 monclient: _renew_subs 2024-08-19T12:39:35.764 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f328153a780 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) -- 0x56032dba8a20 con 0x56032da3e850 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f328153a780 1 librados: init done 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.582+0000 7f328153a780 10 librados: wait_for_osdmap waiting 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.3 v1:172.21.15.104:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f32700016b0 con 0x56032da3e850 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327c824640 10 monclient: handle_config config(0 keys) 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.3 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3270002ad0 con 0x56032da3e850 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327c824640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327c824640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327c824640 10 monclient: dump: 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: epoch 1 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:39:35.765 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: min_mon_release 19 (squid) 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: election_strategy: 1 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.766 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.3 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3270010370 con 0x56032da3e850 2024-08-19T12:39:35.767 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.3 v1:172.21.15.104:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 3235213014 0 0) 0x7f327004c360 con 0x56032da3e850 2024-08-19T12:39:35.767 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327c824640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_map(20..20 src has 1..20) 2024-08-19T12:39:35.767 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327c824640 3 client.4683.objecter handle_osd_map got epochs [20,20] > 0 2024-08-19T12:39:35.767 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327c824640 3 client.4683.objecter handle_osd_map decoding full epoch 20 2024-08-19T12:39:35.767 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327c824640 20 client.4683.objecter dump_active .. 0 homeless 2024-08-19T12:39:35.767 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.584+0000 7f327d025640 4 set_mon_vals no callback set 2024-08-19T12:39:35.767 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.585+0000 7f328153a780 10 librados: wait_for_osdmap done waiting 2024-08-19T12:39:35.767 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.585+0000 7f328153a780 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:39:35.780 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_stat_pp 2024-08-19T12:39:35.780 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_stat_pp' 2024-08-19T12:39:35.780 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_stat_pp 2024-08-19T12:39:35.780 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_misc: /' 2024-08-19T12:39:35.780 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_misc.log 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.781 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.782 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.782 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.782 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.782 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.782 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.782 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.782 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.782 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.782 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.782 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.782 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.783 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.785 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.785 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.785 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.785 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.785 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.785 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.785 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.785 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.785 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.786 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.787 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.788 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.789 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.790 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24656/exe 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.791 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.792 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.793 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.794 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.794 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-08-19T12:39:35.794 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_snapshots: /' 2024-08-19T12:39:35.794 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-08-19T12:39:35.794 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_stat_pp 2024-08-19T12:39:35.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.585+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6790/0 --test api_stat_pp on pid 24819 2024-08-19T12:39:35.804 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24819 2024-08-19T12:39:35.804 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_stat_pp on pid 24819' 2024-08-19T12:39:35.804 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24819 2024-08-19T12:39:35.804 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_watch_notify 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_watch_notify' 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_watch_notify 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.805 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24725/exe 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.806 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.807 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.808 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-08-19T12:39:35.809 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.810 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24776/exe 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.811 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.812 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.813 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.814 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-08-19T12:39:35.814 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_stat: /' 2024-08-19T12:39:35.814 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_stat.log 2024-08-19T12:39:35.814 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.814 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.814 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.814 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_watch_notify 2024-08-19T12:39:35.814 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24866 2024-08-19T12:39:35.814 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.814 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.814 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_watch_notify on pid 24866' 2024-08-19T12:39:35.814 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.815 INFO:tasks.workunit.client.0.smithi104.stdout:test api_watch_notify on pid 24866 2024-08-19T12:39:35.821 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.821 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24866 2024-08-19T12:39:35.821 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.821 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.821 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_watch_notify_pp 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_watch_notify_pp' 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.822 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.823 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.824 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.825 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_watch_notify_pp 2024-08-19T12:39:35.826 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.827 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_watch_notify_pp 2024-08-19T12:39:35.827 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24904 2024-08-19T12:39:35.827 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_watch_notify_pp on pid 24904' 2024-08-19T12:39:35.827 INFO:tasks.workunit.client.0.smithi104.stdout:test api_watch_notify_pp on pid 24904 2024-08-19T12:39:35.837 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24904 2024-08-19T12:39:35.837 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.837 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.837 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-08-19T12:39:35.837 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.837 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_cmd 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_cmd' 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.838 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_cmd 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.839 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.840 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.840 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.840 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.840 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.840 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.840 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.840 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.840 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.840 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.840 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.840 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.840 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.841 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.842 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.843 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24819/exe 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.844 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.845 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_cmd 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24963 2024-08-19T12:39:35.846 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_cmd on pid 24963' 2024-08-19T12:39:35.847 INFO:tasks.workunit.client.0.smithi104.stdout:test api_cmd on pid 24963 2024-08-19T12:39:35.851 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24963 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_cmd_pp 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_cmd_pp' 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_cmd_pp 2024-08-19T12:39:35.852 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_stat_pp: /' 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.853 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.854 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_cmd_pp 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.855 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24987 2024-08-19T12:39:35.856 INFO:tasks.workunit.client.0.smithi104.stdout:test api_cmd_pp on pid 24987 2024-08-19T12:39:35.866 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_cmd_pp on pid 24987' 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=24987 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.867 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24866/exe 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.868 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.869 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_service 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_service' 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.870 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.871 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_watch_notify: /' 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-08-19T12:39:35.872 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_service 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.873 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.874 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.875 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.876 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_service 2024-08-19T12:39:35.877 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.877 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.877 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25034 2024-08-19T12:39:35.877 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_service on pid 25034' 2024-08-19T12:39:35.877 INFO:tasks.workunit.client.0.smithi104.stdout:test api_service on pid 25034 2024-08-19T12:39:35.880 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25034 2024-08-19T12:39:35.880 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.880 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.880 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.880 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.880 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-08-19T12:39:35.880 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_service_pp 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.881 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_service_pp' 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_service_pp 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_service_pp 2024-08-19T12:39:35.882 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25052 2024-08-19T12:39:35.883 INFO:tasks.workunit.client.0.smithi104.stdout:test api_service_pp on pid 25052 2024-08-19T12:39:35.887 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_service_pp on pid 25052' 2024-08-19T12:39:35.887 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25052 2024-08-19T12:39:35.887 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.887 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.887 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.887 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.887 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_c_write_operations 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_c_write_operations' 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.888 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.889 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_c_write_operations 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-08-19T12:39:35.890 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.891 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.891 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.891 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.891 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.891 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24904/exe 2024-08-19T12:39:35.891 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_c_write_operations 2024-08-19T12:39:35.891 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25067 2024-08-19T12:39:35.891 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_c_write_operations on pid 25067' 2024-08-19T12:39:35.891 INFO:tasks.workunit.client.0.smithi104.stdout:test api_c_write_operations on pid 25067 2024-08-19T12:39:35.900 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25067 2024-08-19T12:39:35.900 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.901 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s api_c_read_operations 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.902 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' api_c_read_operations' 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.903 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo api_c_read_operations 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.904 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.905 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.906 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.906 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.906 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.906 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.906 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.906 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.906 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.906 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.906 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.906 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-08-19T12:39:35.906 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.906 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.907 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.908 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.909 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.909 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-08-19T12:39:35.909 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=api_c_read_operations 2024-08-19T12:39:35.909 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.909 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.909 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25093 2024-08-19T12:39:35.909 INFO:tasks.workunit.client.0.smithi104.stdout:test api_c_read_operations on pid 25093 2024-08-19T12:39:35.922 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test api_c_read_operations on pid 25093' 2024-08-19T12:39:35.922 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25093 2024-08-19T12:39:35.922 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.922 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.922 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.922 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.922 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.922 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.922 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24963/exe 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.923 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.924 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s list_parallel 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.925 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' list_parallel' 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_cmd: /' 2024-08-19T12:39:35.926 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_cmd.log 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo list_parallel 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.927 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.928 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.929 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.930 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.931 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.932 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=list_parallel 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25124 2024-08-19T12:39:35.933 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test list_parallel on pid 25124' 2024-08-19T12:39:35.934 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.934 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.934 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.934 INFO:tasks.workunit.client.0.smithi104.stdout:test list_parallel on pid 25124 2024-08-19T12:39:35.940 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.940 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.940 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.940 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.940 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.940 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25124 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.941 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.942 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.943 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s open_pools_parallel 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' open_pools_parallel' 2024-08-19T12:39:35.944 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/24987/exe 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo open_pools_parallel 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.945 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.946 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.947 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.947 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=open_pools_parallel 2024-08-19T12:39:35.947 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25142 2024-08-19T12:39:35.947 INFO:tasks.workunit.client.0.smithi104.stdout:test open_pools_parallel on pid 25142 2024-08-19T12:39:35.961 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test open_pools_parallel on pid 25142' 2024-08-19T12:39:35.961 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25142 2024-08-19T12:39:35.961 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-19T12:39:35.961 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.961 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.961 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.961 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.961 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.962 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.963 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.963 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.963 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.963 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.963 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.963 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.963 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.963 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.963 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.963 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.964 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s delete_pools_parallel 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.965 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' delete_pools_parallel' 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.966 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.967 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.968 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo delete_pools_parallel 2024-08-19T12:39:35.969 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25034/exe 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.970 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.971 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.972 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:35.973 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.974 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.975 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=delete_pools_parallel 2024-08-19T12:39:35.975 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25184 2024-08-19T12:39:35.975 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test delete_pools_parallel on pid 25184' 2024-08-19T12:39:35.975 INFO:tasks.workunit.client.0.smithi104.stdout:test delete_pools_parallel on pid 25184 2024-08-19T12:39:35.984 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25184 2024-08-19T12:39:35.984 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:35.984 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:35.984 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_service.log 2024-08-19T12:39:35.984 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.984 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.984 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.984 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.984 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.984 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s cls 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' cls' 2024-08-19T12:39:35.985 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_service: /' 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:35.986 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:35.987 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.988 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:35.988 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:35.988 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:35.988 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.988 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:35.988 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:35.988 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:35.988 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.988 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:35.988 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.988 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:35.988 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo cls 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25052/exe 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.989 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:35.990 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:35.990 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:35.990 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:35.990 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:35.990 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.990 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:35.990 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:35.990 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.990 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:35.990 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.991 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:35.991 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.991 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:35.991 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:35.991 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.991 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:35.991 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.991 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:35.991 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.991 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:35.991 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:35.991 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:35.992 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25067/exe 2024-08-19T12:39:35.993 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=cls 2024-08-19T12:39:35.993 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:35.993 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25209 2024-08-19T12:39:35.993 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test cls on pid 25209' 2024-08-19T12:39:35.993 INFO:tasks.workunit.client.0.smithi104.stdout:test cls on pid 25209 2024-08-19T12:39:36.004 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25209 2024-08-19T12:39:36.004 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:36.004 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.004 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.005 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.006 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.007 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s cmd 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' cmd' 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.008 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25093/exe 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_service_pp: /' 2024-08-19T12:39:36.009 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.010 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.011 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.012 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.013 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.014 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.014 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.014 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.014 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.014 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.014 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-08-19T12:39:36.014 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-08-19T12:39:36.014 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo cmd 2024-08-19T12:39:36.014 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:36.014 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=cmd 2024-08-19T12:39:36.014 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25251 2024-08-19T12:39:36.014 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test cmd on pid 25251' 2024-08-19T12:39:36.015 INFO:tasks.workunit.client.0.smithi104.stdout:test cmd on pid 25251 2024-08-19T12:39:36.031 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25251 2024-08-19T12:39:36.031 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:36.031 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.031 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.031 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.031 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.031 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.032 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.032 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.032 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.032 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.032 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.032 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.032 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.032 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.032 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.032 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.032 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.032 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.033 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.034 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.035 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.035 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.035 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.035 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.035 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.035 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.035 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.035 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.035 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.035 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.035 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.035 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.036 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25184/exe 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.040 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.041 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.042 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s handler_error 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' handler_error' 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo handler_error 2024-08-19T12:39:36.043 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.044 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.045 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.046 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.047 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.048 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.048 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.048 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25142/exe 2024-08-19T12:39:36.048 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=handler_error 2024-08-19T12:39:36.048 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25319 2024-08-19T12:39:36.048 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test handler_error on pid 25319' 2024-08-19T12:39:36.048 INFO:tasks.workunit.client.0.smithi104.stdout:test handler_error on pid 25319 2024-08-19T12:39:36.062 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25319 2024-08-19T12:39:36.062 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:36.062 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.063 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.063 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-08-19T12:39:36.063 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.063 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.063 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.063 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.063 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.063 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.063 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.063 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.063 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.063 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.064 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s io 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' io' 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.065 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.066 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.067 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.068 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25124/exe 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo io 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.069 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.070 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.071 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.072 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.073 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25209/exe 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.074 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.075 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.075 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=io 2024-08-19T12:39:36.075 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25366 2024-08-19T12:39:36.075 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test io on pid 25366' 2024-08-19T12:39:36.075 INFO:tasks.workunit.client.0.smithi104.stdout:test io on pid 25366 2024-08-19T12:39:36.081 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25366 2024-08-19T12:39:36.081 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:36.081 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.081 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-08-19T12:39:36.081 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s ec_io 2024-08-19T12:39:36.081 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' ec_io' 2024-08-19T12:39:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_cls 2024-08-19T12:39:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_cls.log 2024-08-19T12:39:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ cls: /' 2024-08-19T12:39:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_rados_list_parallel.log 2024-08-19T12:39:36.082 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ list_parallel: /' 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.083 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.084 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.085 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo ec_io 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=ec_io 2024-08-19T12:39:36.086 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25408 2024-08-19T12:39:36.087 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test ec_io on pid 25408' 2024-08-19T12:39:36.087 INFO:tasks.workunit.client.0.smithi104.stdout:test ec_io on pid 25408 2024-08-19T12:39:36.097 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25408 2024-08-19T12:39:36.097 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:36.097 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.097 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.097 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.097 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.097 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-08-19T12:39:36.097 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.098 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.099 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25251/exe 2024-08-19T12:39:36.099 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s list 2024-08-19T12:39:36.099 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.099 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.099 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.099 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.099 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.099 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.099 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.099 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.099 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.099 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.100 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.101 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.102 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' list' 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_cmd 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ cmd: /' 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.103 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_cmd.log 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo list 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.104 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.105 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=list 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25531 2024-08-19T12:39:36.106 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test list on pid 25531' 2024-08-19T12:39:36.107 INFO:tasks.workunit.client.0.smithi104.stdout:test list on pid 25531 2024-08-19T12:39:36.111 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25531 2024-08-19T12:39:36.111 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:36.111 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.111 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.111 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.111 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.111 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s ec_list 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' ec_list' 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25319/exe 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.112 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo ec_list 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.113 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.114 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.115 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.115 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.115 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.115 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.115 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_handler_error 2024-08-19T12:39:36.115 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ handler_error: /' 2024-08-19T12:39:36.115 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:36.115 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_handler_error.log 2024-08-19T12:39:36.115 INFO:tasks.workunit.client.0.smithi104.stdout:test ec_list on pid 25571 2024-08-19T12:39:36.127 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=ec_list 2024-08-19T12:39:36.127 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.127 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.127 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25571 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test ec_list on pid 25571' 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25571 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.128 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.129 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.130 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s misc 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' misc' 2024-08-19T12:39:36.131 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.132 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.133 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.133 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.133 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.133 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.133 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.133 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.133 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.133 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.133 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.133 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.133 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.133 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.134 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.135 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo misc 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.136 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.137 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.138 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25366/exe 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.139 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.140 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.140 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.140 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=misc 2024-08-19T12:39:36.140 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.140 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.140 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25637 2024-08-19T12:39:36.140 INFO:tasks.workunit.client.0.smithi104.stdout:test misc on pid 25637 2024-08-19T12:39:36.145 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test misc on pid 25637' 2024-08-19T12:39:36.145 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25637 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.146 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25408/exe 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.147 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_io.log 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_io 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.148 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ io: /' 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_ec_io 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_ec_io.log 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.149 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s pool 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' pool' 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ ec_io: /' 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo pool 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=pool 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25663 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.150 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test pool on pid 25663' 2024-08-19T12:39:36.151 INFO:tasks.workunit.client.0.smithi104.stdout:test pool on pid 25663 2024-08-19T12:39:36.154 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.154 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25663 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s read_operations 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.155 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' read_operations' 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.156 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo read_operations 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.157 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.158 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.158 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.158 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.158 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.158 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=read_operations 2024-08-19T12:39:36.158 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25672 2024-08-19T12:39:36.158 INFO:tasks.workunit.client.0.smithi104.stdout:test read_operations on pid 25672 2024-08-19T12:39:36.165 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test read_operations on pid 25672' 2024-08-19T12:39:36.165 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25672 2024-08-19T12:39:36.165 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:36.165 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.165 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.165 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.165 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.165 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.165 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.165 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.165 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.165 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.166 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s snapshots 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' snapshots' 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.167 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.168 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo snapshots 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.169 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=snapshots 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25708 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test snapshots on pid 25708' 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.170 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.171 INFO:tasks.workunit.client.0.smithi104.stdout:test snapshots on pid 25708 2024-08-19T12:39:36.178 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.179 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.179 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25708 2024-08-19T12:39:36.179 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:36.179 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.179 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s watch_notify 2024-08-19T12:39:36.179 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.179 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.179 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.179 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.179 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.179 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.179 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' watch_notify' 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.180 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.181 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.182 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.183 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.184 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo watch_notify 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.185 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.186 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.186 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=watch_notify 2024-08-19T12:39:36.186 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25736 2024-08-19T12:39:36.186 INFO:tasks.workunit.client.0.smithi104.stdout:test watch_notify on pid 25736 2024-08-19T12:39:36.193 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test watch_notify on pid 25736' 2024-08-19T12:39:36.193 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25736 2024-08-19T12:39:36.193 INFO:tasks.workunit.client.0.smithi104.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-19T12:39:36.193 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.193 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25531/exe 2024-08-19T12:39:36.194 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.194 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.194 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.194 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.194 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.194 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.194 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.194 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.194 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.194 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.194 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.194 INFO:tasks.workunit.client.0.smithi104.stderr:++ printf %25s write_operations 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:+ r=' write_operations' 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.195 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.196 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.196 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.196 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.196 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.196 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.196 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.196 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.196 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.196 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.196 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.196 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:++ awk '{print $1}' 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.197 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.198 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.198 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.198 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.198 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.198 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.198 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.198 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.198 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.198 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.198 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.198 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ list: /' 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.199 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.200 INFO:tasks.workunit.client.0.smithi104.stderr:++ echo write_operations 2024-08-19T12:39:36.201 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_list 2024-08-19T12:39:36.201 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.201 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.201 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.201 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.201 INFO:tasks.workunit.client.0.smithi104.stderr:+ ff=write_operations 2024-08-19T12:39:36.201 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.201 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.201 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.201 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.201 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.201 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25764 2024-08-19T12:39:36.202 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test write_operations on pid 25764' 2024-08-19T12:39:36.202 INFO:tasks.workunit.client.0.smithi104.stdout:test write_operations on pid 25764 2024-08-19T12:39:36.238 INFO:tasks.workunit.client.0.smithi104.stderr:+ pids[$f]=25764 2024-08-19T12:39:36.238 INFO:tasks.workunit.client.0.smithi104.stderr:+ ret=0 2024-08-19T12:39:36.238 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' 1 -eq 1 ']' 2024-08-19T12:39:36.238 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:39:36.238 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24588 2024-08-19T12:39:36.238 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24588 2024-08-19T12:39:36.238 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_list.log 2024-08-19T12:39:36.238 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-08-19T12:39:36.238 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.238 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.238 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.239 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25571/exe 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.240 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.241 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.242 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -f /etc/bashrc ']' 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:+ . /etc/bashrc 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.243 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -z '' ']' 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:++ BASHRCSOURCED=Y 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:++ shopt -q login_shell 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.244 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.245 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.245 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.245 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.245 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.245 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.245 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.245 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.245 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.245 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.245 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.245 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.245 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_ec_list.log 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_ec_list 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:+++ umask 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ ec_list: /' 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.246 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.247 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' 0022 -eq 0 ']' 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:++ SHELL=/bin/bash 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.248 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.249 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.250 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.251 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.252 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.253 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.254 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.255 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'grep=grep --color=auto' 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25663/exe 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorls.sh 2024-08-19T12:39:36.256 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' '!' -t 0 ']' 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:+++ return 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export S_COLORS=auto 2024-08-19T12:39:36.257 INFO:tasks.workunit.client.0.smithi104.stderr:+++ S_COLORS=auto 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25637/exe 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.258 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.259 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.260 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_pool 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.261 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.262 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_pool.log 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_misc 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_misc.log 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ misc: /' 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.263 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ pool: /' 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-19T12:39:36.264 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25672/exe 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.265 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-19T12:39:36.266 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-19T12:39:36.267 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/gawk.sh 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.268 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.269 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25764/exe 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/lang.sh 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/locale 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25708/exe 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.270 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ read_operations: /' 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.271 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_read_operations 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_write_operations.log 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_snapshots 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:++++ export LANG=en_US.UTF-8 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:++++ LANG=en_US.UTF-8 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-19T12:39:36.272 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_read_operations.log 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:+++ LANG=en_US.UTF-8 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:+++ unset LANG_backup config 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n '' ']' 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' dumb = linux ']' 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/less.sh 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:++ for i in /etc/profile.d/*.sh 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:++ '[' '' ']' 2024-08-19T12:39:36.273 INFO:tasks.workunit.client.0.smithi104.stderr:++ . /etc/profile.d/which2.sh 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_snapshots.log 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ snapshots: /' 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_write_operations 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:+++++ readlink /proc/25736/exe 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:++++ basename /usr/bin/bash 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ write_operations: /' 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_declare='declare -f' 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:+++ which_opt=-f 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export which_declare 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:+++ export -f which 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset i 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:++ unset -f pathmunge 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-19T12:39:36.274 INFO:tasks.workunit.client.0.smithi104.stderr:+ export PATH 2024-08-19T12:39:36.275 INFO:tasks.workunit.client.0.smithi104.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-19T12:39:36.275 INFO:tasks.workunit.client.0.smithi104.stderr:+ unset rc 2024-08-19T12:39:36.275 INFO:tasks.workunit.client.0.smithi104.stderr:+ ceph_test_neorados_watch_notify 2024-08-19T12:39:36.275 INFO:tasks.workunit.client.0.smithi104.stderr:+ sed 's/^/ watch_notify: /' 2024-08-19T12:39:36.275 INFO:tasks.workunit.client.0.smithi104.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-08-19T12:39:36.287 INFO:tasks.workunit.client.0.smithi104.stdout: mon_get_version(what=osdmap handle=1) -- 0x7f3278003510 con 0x56032da3e850 2024-08-19T12:39:36.287 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.586+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.3 v1:172.21.15.104:6790/0 9 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (unknown 592526333 0 0) 0x7f3270015610 con 0x56032da3e850 2024-08-19T12:39:36.287 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.586+0000 7f327c824640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-08-19T12:39:36.287 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.586+0000 7f327d826640 10 client.4683.objecter _get_latest_version latest 20, have it 2024-08-19T12:39:36.287 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.586+0000 7f328153a780 10 client.4683.objecter create_pool name=ceph_test_rados_api_asio 2024-08-19T12:39:36.287 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.586+0000 7f328153a780 10 client.4683.objecter pool_op_submit 1 2024-08-19T12:39:36.287 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.586+0000 7f328153a780 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:39:36.287 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.586+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6790/0 -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x56032d9f8b10 con 0x56032da3e850 2024-08-19T12:39:36.287 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.587+0000 7f327e828640 10 monclient: get_auth_request con 0x7f326c03f880 auth_method 0 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:35.590+0000 7f327c824640 10 client.4683.objecter ms_handle_connect 0x7f326c03f880 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.281+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.3 v1:172.21.15.104:6790/0 10 ==== pool_op_reply(tid 1 (0) Success v21) ==== 43+0+0 (unknown 203505783 0 0) 0x7f3270015f40 con 0x56032da3e850 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.281+0000 7f327c824640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 pool_op_reply(tid 1 (0) Success v21) 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.281+0000 7f327c824640 10 client.4683.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v21) 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.281+0000 7f327c824640 10 client.4683.objecter have request 1 at 0x56032da6f990 Op: create 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.281+0000 7f327c824640 20 client.4683.objecter waiting for client to reach epoch 21 before calling back 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.281+0000 7f327c824640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.281+0000 7f327c824640 10 monclient: _renew_subs 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.281+0000 7f327c824640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.281+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=21}) -- 0x56032da6f4e0 con 0x56032da3e850 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.281+0000 7f327c824640 10 client.4683.objecter done 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.285+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== mon.3 v1:172.21.15.104:6790/0 11 ==== osd_map(21..21 src has 1..21) ==== 13849+0+0 (unknown 1564336373 0 0) 0x7f3270002c70 con 0x56032da3e850 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.285+0000 7f327c824640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_map(21..21 src has 1..21) 2024-08-19T12:39:36.288 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.285+0000 7f327c824640 3 client.4683.objecter handle_osd_map got epochs [21,21] > 20 2024-08-19T12:39:36.289 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.285+0000 7f327c824640 3 client.4683.objecter handle_osd_map decoding incremental epoch 21 2024-08-19T12:39:36.289 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.285+0000 7f327c824640 20 client.4683.objecter dump_active .. 0 homeless 2024-08-19T12:39:36.289 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 10 librados: set snap read head -> 1 2024-08-19T12:39:36.289 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 10 librados: writefull oid=exist nspace= 2024-08-19T12:39:38.321 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dba9c00 2024-08-19T12:39:38.322 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 21 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T12:39:38.322 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.322 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.322 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.322 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 2 osd.4 2024-08-19T12:39:38.322 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.322 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 2 2024-08-19T12:39:38.322 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 15 client.4683.objecter _send_op 2 to 7.1 on osd.4 2024-08-19T12:39:38.322 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:2 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x56032dbae240 con 0x56032dbaa730 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.286+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.290+0000 7f327f82a640 10 monclient: get_auth_request con 0x56032dbaa730 auth_method 0 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:36.297+0000 7f327c824640 10 client.4683.objecter ms_handle_connect 0x56032dbaa730 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.273+0000 7f327c824640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 1 ==== osd_map(22..22 src has 1..22) ==== 7689+0+0 (unknown 265797478 0 0) 0x7f32740034f0 con 0x56032dbaa730 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.273+0000 7f327c824640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_map(22..22 src has 1..22) 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.273+0000 7f327c824640 3 client.4683.objecter handle_osd_map got epochs [22,22] > 21 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.273+0000 7f327c824640 3 client.4683.objecter handle_osd_map decoding incremental epoch 22 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.273+0000 7f327c824640 10 client.4683.objecter checking op 2 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.273+0000 7f327c824640 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.273+0000 7f327c824640 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.273+0000 7f327c824640 20 client.4683.objecter dump_active .. 0 homeless 2024-08-19T12:39:38.323 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.273+0000 7f327c824640 20 client.4683.objecter 2 7.d3a59201 osd.4 exist [writefull 0~5 in=5b] 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 2 ==== osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) ==== 149+0+0 (unknown 1672188149 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 2 ondisk uv 1 in 7.1 attempt 0 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 2 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f327f82a640 15 client.4683.objecter _finish_op 2 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 2 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f328153a780 10 librados: Objecter returned from writefull r=0 2024-08-19T12:39:38.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dba8bf0 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.320+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 3 osd.4 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 3 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 15 client.4683.objecter _send_op 3 to 7.1 on osd.4 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:3 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x56032dba90a0 con 0x56032dbaa730 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dba9520 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 20 client.4683.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-08-19T12:39:38.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 10 client.4683.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 4 osd.4 2024-08-19T12:39:38.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 4 2024-08-19T12:39:38.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 15 client.4683.objecter _send_op 4 to 7.1 on osd.4 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:4 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x56032dba99d0 con 0x56032dbaa730 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 5 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.321+0000 7f328153a780 5 client.4683.objecter 2 in flight 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (unknown 472631335 0 3741568377) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 3 ondisk uv 1 in 7.1 attempt 0 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 5 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 3 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 15 client.4683.objecter _finish_op 3 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 5 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 3 2024-08-19T12:39:38.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 2347777278 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 4 ondisk uv 0 in 7.1 attempt 0 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 10 client.4683.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 4 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 15 client.4683.objecter _finish_op 4 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 4 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.328 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dba9520 2024-08-19T12:39:38.329 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:39:38.329 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.329 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.329 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.329 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 5 osd.4 2024-08-19T12:39:38.329 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.329 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 5 2024-08-19T12:39:38.329 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 15 client.4683.objecter _send_op 5 to 7.1 on osd.4 2024-08-19T12:39:38.329 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:5 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x56032dba8bf0 con 0x56032dbaa730 2024-08-19T12:39:38.329 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.329 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.329 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbae5e0 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 20 client.4683.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 10 client.4683.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 6 osd.4 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 6 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 15 client.4683.objecter _send_op 6 to 7.1 on osd.4 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:6 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x56032dbaea90 con 0x56032dbaa730 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 5 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.322+0000 7f328153a780 5 client.4683.objecter 2 in flight 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (unknown 472631335 0 3741568377) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.330 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-08-19T12:39:38.331 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.331 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 5 ondisk uv 1 in 7.1 attempt 0 2024-08-19T12:39:38.331 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 5 2024-08-19T12:39:38.331 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 5 2024-08-19T12:39:38.331 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 15 client.4683.objecter _finish_op 5 2024-08-19T12:39:38.331 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 5 2024-08-19T12:39:38.331 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 5 2024-08-19T12:39:38.331 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.331 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 2347777278 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.331 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T12:39:38.331 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.331 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 6 ondisk uv 0 in 7.1 attempt 0 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 6 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 15 client.4683.objecter _finish_op 6 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 6 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbae5e0 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.332 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 7 osd.4 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 7 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 15 client.4683.objecter _send_op 7 to 7.1 on osd.4 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:7 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x56032dba9520 con 0x56032dbaa730 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbeee50 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.333 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 10 client.4683.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 8 osd.4 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 8 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 15 client.4683.objecter _send_op 8 to 7.1 on osd.4 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:8 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x56032dbef300 con 0x56032dbaa730 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 5 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 5 client.4683.objecter 2 in flight 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (unknown 472631335 0 3741568377) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 7 ondisk uv 1 in 7.1 attempt 0 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 5 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 7 2024-08-19T12:39:38.334 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 15 client.4683.objecter _finish_op 7 2024-08-19T12:39:38.335 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 5 2024-08-19T12:39:38.335 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 7 2024-08-19T12:39:38.335 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.335 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 2347777278 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.335 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T12:39:38.335 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.335 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 8 ondisk uv 0 in 7.1 attempt 0 2024-08-19T12:39:38.335 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T12:39:38.335 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (0 ms) 2024-08-19T12:39:38.346 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.346 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-08-19T12:39:38.346 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 8 2024-08-19T12:39:38.346 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 15 client.4683.objecter _finish_op 8 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 8 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 librados: queue_aio_write 0x56032d9b05e0 completion 0x56032d985f70 write_seq 1 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbae5e0 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 9 osd.4 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 9 2024-08-19T12:39:38.347 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 15 client.4683.objecter _send_op 9 to 7.1 on osd.4 2024-08-19T12:39:38.348 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:9 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x56032dbafe40 con 0x56032dbaa730 2024-08-19T12:39:38.348 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.348 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.348 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.323+0000 7f328153a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-08-19T12:39:38.348 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 9 ==== osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) ==== 149+0+0 (unknown 1662904861 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.348 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) 2024-08-19T12:39:38.348 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.348 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 9 ondisk uv 2 in 7.1 attempt 0 2024-08-19T12:39:38.348 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.348 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 9 2024-08-19T12:39:38.348 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f327f82a640 15 client.4683.objecter _finish_op 9 2024-08-19T12:39:38.348 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.349 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 9 2024-08-19T12:39:38.349 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.349 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f327f82a640 20 librados: complete_aio_write 0x56032d985f70 2024-08-19T12:39:38.349 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (22 ms) 2024-08-19T12:39:38.349 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-08-19T12:39:38.349 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-08-19T12:39:38.349 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 20 librados: queue_aio_write 0x56032d9b05e0 completion 0x56032dbb02a0 write_seq 2 2024-08-19T12:39:38.349 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbae5e0 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 10 osd.4 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 10 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 15 client.4683.objecter _send_op 10 to 7.1 on osd.4 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:10 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x56032dbafe40 con 0x56032dbaa730 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.345+0000 7f328153a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 10 ==== osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) ==== 149+0+0 (unknown 396186057 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.350 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) 2024-08-19T12:39:38.351 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.351 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 10 ondisk uv 3 in 7.1 attempt 0 2024-08-19T12:39:38.351 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.351 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 10 2024-08-19T12:39:38.351 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f327f82a640 15 client.4683.objecter _finish_op 10 2024-08-19T12:39:38.351 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.351 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (2 ms) 2024-08-19T12:39:38.351 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 10 2024-08-19T12:39:38.351 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-08-19T12:39:38.353 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.353 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f327f82a640 20 librados: complete_aio_write 0x56032dbb02a0 2024-08-19T12:39:38.353 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-08-19T12:39:38.353 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 20 librados: queue_aio_write 0x56032d9b05e0 completion 0x56032dbb0060 write_seq 3 2024-08-19T12:39:38.353 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbae5e0 2024-08-19T12:39:38.353 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T12:39:38.353 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.353 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.353 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.353 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 11 osd.4 2024-08-19T12:39:38.354 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.354 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 11 2024-08-19T12:39:38.354 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 15 client.4683.objecter _send_op 11 to 7.1 on osd.4 2024-08-19T12:39:38.354 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:11 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x56032dbd2380 con 0x56032dbaa730 2024-08-19T12:39:38.354 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.354 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.354 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.347+0000 7f328153a780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-08-19T12:39:38.354 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 11 ==== osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) ==== 149+0+0 (unknown 1576141844 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.354 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) 2024-08-19T12:39:38.354 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.354 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 11 ondisk uv 4 in 7.1 attempt 0 2024-08-19T12:39:38.354 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.355 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 11 2024-08-19T12:39:38.355 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 15 client.4683.objecter _finish_op 11 2024-08-19T12:39:38.355 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.355 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 11 2024-08-19T12:39:38.355 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.355 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 20 librados: complete_aio_write 0x56032dbb0060 2024-08-19T12:39:38.355 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (5 ms) 2024-08-19T12:39:38.355 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-08-19T12:39:38.356 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbae5e0 2024-08-19T12:39:38.356 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:39:38.356 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.356 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.356 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.356 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 12 osd.4 2024-08-19T12:39:38.356 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.356 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 12 2024-08-19T12:39:38.356 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 15 client.4683.objecter _send_op 12 to 7.1 on osd.4 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:12 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x56032dbaee30 con 0x56032dbaa730 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbaf2e0 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 20 client.4683.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 10 client.4683.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 13 osd.4 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 13 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 15 client.4683.objecter _send_op 13 to 7.1 on osd.4 2024-08-19T12:39:38.357 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:13 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x56032dbaf790 con 0x56032dbaa730 2024-08-19T12:39:38.358 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 5 2024-08-19T12:39:38.358 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 5 client.4683.objecter 2 in flight 2024-08-19T12:39:38.358 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (unknown 1853434582 0 3741568377) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.358 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-08-19T12:39:38.358 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.358 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 12 ondisk uv 4 in 7.1 attempt 0 2024-08-19T12:39:38.358 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 5 2024-08-19T12:39:38.358 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 12 2024-08-19T12:39:38.358 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 15 client.4683.objecter _finish_op 12 2024-08-19T12:39:38.358 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 5 2024-08-19T12:39:38.358 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 12 2024-08-19T12:39:38.358 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 3866624710 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 13 ondisk uv 0 in 7.1 attempt 0 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 10 client.4683.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 13 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 15 client.4683.objecter _finish_op 13 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 13 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (0 ms) 2024-08-19T12:39:38.359 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbae5e0 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.352+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 14 osd.4 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 14 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 15 client.4683.objecter _send_op 14 to 7.1 on osd.4 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:14 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x56032dbaee30 con 0x56032dbaa730 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbaf2e0 2024-08-19T12:39:38.360 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 10 client.4683.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 15 osd.4 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 15 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 15 client.4683.objecter _send_op 15 to 7.1 on osd.4 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:15 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x56032dbaf790 con 0x56032dbaa730 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 5 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 5 client.4683.objecter 2 in flight 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (unknown 1853434582 0 3741568377) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-08-19T12:39:38.361 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 14 ondisk uv 4 in 7.1 attempt 0 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 5 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 14 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 15 client.4683.objecter _finish_op 14 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 5 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 14 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 3866624710 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 15 ondisk uv 0 in 7.1 attempt 0 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 10 client.4683.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T12:39:38.362 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 15 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 15 client.4683.objecter _finish_op 15 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 15 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbae5e0 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 16 osd.4 2024-08-19T12:39:38.363 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 16 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 15 client.4683.objecter _send_op 16 to 7.1 on osd.4 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:16 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x56032dbd2380 con 0x56032dbaa730 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbaf2e0 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 10 client.4683.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 17 osd.4 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.364 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 17 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 15 client.4683.objecter _send_op 17 to 7.1 on osd.4 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:17 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x56032dbaf790 con 0x56032dbaa730 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 5 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.353+0000 7f328153a780 5 client.4683.objecter 2 in flight 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (unknown 1853434582 0 3741568377) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 16 ondisk uv 4 in 7.1 attempt 0 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 5 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 16 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 15 client.4683.objecter _finish_op 16 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 5 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 16 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.365 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 3866624710 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.366 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-08-19T12:39:38.366 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-08-19T12:39:38.366 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-19T12:39:38.366 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.366 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 17 ondisk uv 0 in 7.1 attempt 0 2024-08-19T12:39:38.366 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 10 client.4683.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-19T12:39:38.366 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.366 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 17 2024-08-19T12:39:38.366 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 15 client.4683.objecter _finish_op 17 2024-08-19T12:39:38.366 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.366 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 17 2024-08-19T12:39:38.366 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.367 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 20 librados: queue_aio_write 0x56032d9b05e0 completion 0x56032dbaef40 write_seq 4 2024-08-19T12:39:38.367 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbae5e0 2024-08-19T12:39:38.367 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T12:39:38.367 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.367 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.367 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 18 osd.4 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 18 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 15 client.4683.objecter _send_op 18 to 7.1 on osd.4 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:18 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x56032dbaf2e0 con 0x56032dbaa730 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.354+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 18 ==== osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) ==== 149+0+0 (unknown 3024423252 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 18 ondisk uv 5 in 7.1 attempt 0 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 18 2024-08-19T12:39:38.368 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f327f82a640 15 client.4683.objecter _finish_op 18 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 18 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (5 ms) 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f327f82a640 20 librados: complete_aio_write 0x56032dbaef40 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 20 librados: queue_aio_write 0x56032d9b05e0 completion 0x56032dbaf6c0 write_seq 5 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbae5e0 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 19 osd.4 2024-08-19T12:39:38.369 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 19 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 15 client.4683.objecter _send_op 19 to 7.1 on osd.4 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:19 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x56032dbaf2e0 con 0x56032dbaa730 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.359+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.361+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 19 ==== osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) ==== 149+0+0 (unknown 700761384 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 19 ondisk uv 6 in 7.1 attempt 0 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 19 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f327f82a640 15 client.4683.objecter _finish_op 19 2024-08-19T12:39:38.370 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.371 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 19 2024-08-19T12:39:38.371 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.371 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f327f82a640 20 librados: complete_aio_write 0x56032dbaf6c0 2024-08-19T12:39:38.371 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (3 ms) 2024-08-19T12:39:38.371 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-08-19T12:39:38.371 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 20 librados: queue_aio_write 0x56032d9b05e0 completion 0x56032dbaef40 write_seq 6 2024-08-19T12:39:38.371 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 10 client.4683.objecter _op_submit op 0x56032dbae5e0 2024-08-19T12:39:38.371 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 20 client.4683.objecter _calc_target epoch 22 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-08-19T12:39:38.371 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 20 client.4683.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-08-19T12:39:38.371 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 10 client.4683.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-08-19T12:39:38.371 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 20 client.4683.objecter _get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.371 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 10 client.4683.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 20 osd.4 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 20 client.4683.objecter get_session s=0x56032dbaa0b0 osd=4 3 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 15 client.4683.objecter _session_op_assign 4 20 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 15 client.4683.objecter _send_op 20 to 7.1 on osd.4 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 --> v1:172.21.15.104:6805/1352976736 -- osd_op(unknown.0.0:20 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x56032dbaf2e0 con 0x56032dbaa730 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.362+0000 7f328153a780 5 client.4683.objecter 1 in flight 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 <== osd.4 v1:172.21.15.104:6805/1352976736 20 ==== osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) ==== 149+0+0 (unknown 1564770044 0 0) 0x7f32740037f0 con 0x56032dbaa730 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f327f82a640 10 client.4683.objecter ms_dispatch 0x56032d9eb570 osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f327f82a640 10 client.4683.objecter in handle_osd_op_reply 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f327f82a640 7 client.4683.objecter handle_osd_op_reply 20 ondisk uv 7 in 7.1 attempt 0 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f327f82a640 10 client.4683.objecter op 0 rval 0 len 0 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f327f82a640 15 client.4683.objecter handle_osd_op_reply completed tid 20 2024-08-19T12:39:38.372 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f327f82a640 15 client.4683.objecter _finish_op 20 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f327f82a640 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 4 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f327f82a640 15 client.4683.objecter _session_op_remove 4 20 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f327f82a640 5 client.4683.objecter 0 in flight 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f327f82a640 20 librados: complete_aio_write 0x56032dbaef40 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2 ms) 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 10 librados: watch_flush enter 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 10 librados: watch_flush exit 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 10 client.4683.objecter close_session for osd.4 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 >> v1:172.21.15.104:6805/1352976736 conn(0x56032dbaa730 legacy=0x56032dbacb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 20 client.4683.objecter put_session s=0x56032dbaa0b0 osd=4 1 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 20 client.4683.objecter shutdown clearing up homeless session... 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 10 client.4683.objecter successfully canceled tick 2024-08-19T12:39:38.373 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 >> v1:172.21.15.104:6800/899887506 conn(0x7f326c03f880 legacy=0x7f326c041d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:39:38.374 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 10 monclient: shutdown 2024-08-19T12:39:38.374 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:39:38.374 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 >> v1:172.21.15.104:6790/0 conn(0x56032da3e850 legacy=0x56032db9efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:39:38.374 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f327f82a640 1 -- 172.21.15.104:0/1047516331 reap_dead start 2024-08-19T12:39:38.374 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 shutdown_connections 2024-08-19T12:39:38.374 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 >> 172.21.15.104:0/1047516331 conn(0x56032da70520 msgr2=0x56032db05980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:39:38.374 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 shutdown_connections 2024-08-19T12:39:38.374 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 1 -- 172.21.15.104:0/1047516331 wait complete. 2024-08-19T12:39:38.374 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.364+0000 7f328153a780 1 librados: shutdown 2024-08-19T12:39:38.375 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [----------] 12 tests from AsioRados (44 ms total) 2024-08-19T12:39:38.375 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: 2024-08-19T12:39:38.375 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [----------] Global test environment tear-down 2024-08-19T12:39:38.375 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (2795 ms total) 2024-08-19T12:39:38.375 INFO:tasks.workunit.client.0.smithi104.stdout: api_asio: [ PASSED ] 12 tests. 2024-08-19T12:39:38.652 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-08-19T12:39:38.652 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [----------] Global test environment set-up. 2024-08-19T12:39:38.652 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [----------] 7 tests from LibRadosList 2024-08-19T12:39:38.652 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-08-19T12:39:38.652 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ OK ] LibRadosList.ListObjects (347 ms) 2024-08-19T12:39:38.652 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-08-19T12:39:38.652 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (42 ms) 2024-08-19T12:39:38.652 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-08-19T12:39:38.652 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: myset foo1,foo2,foo3 2024-08-19T12:39:38.652 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo1 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo2 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo3 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: myset foo1,foo4,foo5 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo4 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo5 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo1 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: myset foo6,foo7 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo7 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo6 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: ns1:foo4 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: ns1:foo5 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: ns2:foo7 2024-08-19T12:39:38.653 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: ns2:foo6 2024-08-19T12:39:38.654 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: ns1:foo1 2024-08-19T12:39:38.654 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: :foo1 2024-08-19T12:39:38.654 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: :foo2 2024-08-19T12:39:38.654 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: :foo3 2024-08-19T12:39:38.654 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (91 ms) 2024-08-19T12:39:38.654 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-08-19T12:39:38.654 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 1 0 2024-08-19T12:39:38.654 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 10 0 2024-08-19T12:39:38.654 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 13 0 2024-08-19T12:39:38.654 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 7 0 2024-08-19T12:39:38.654 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 14 0 2024-08-19T12:39:38.654 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 0 0 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 15 0 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 11 0 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 5 0 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 8 0 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 6 0 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 3 0 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 4 0 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 12 0 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 9 0 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 2 0 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (71 ms) 2024-08-19T12:39:38.655 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: x cursor=MIN 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=1 cursor=9:02547ec2:::1:head 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=10 cursor=9:52ea6a34:::10:head 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=13 cursor=9:566253c9:::13:head 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=7 cursor=9:5c6b0b28:::7:head 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=14 cursor=9:62a1935d:::14:head 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=0 cursor=9:6cac518f:::0:head 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=15 cursor=9:863748b0:::15:head 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=11 cursor=9:89d3ae78:::11:head 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=5 cursor=9:b29083e3:::5:head 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=8 cursor=9:bd63b0f1:::8:head 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=6 cursor=9:c4fdafeb:::6:head 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=3 cursor=9:cfc208b3:::3:head 2024-08-19T12:39:38.656 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=4 cursor=9:d83876eb:::4:head 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=12 cursor=9:de5d7c5f:::12:head 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=9 cursor=9:e960b815:::9:head 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > oid=2 cursor=9:f905c69b:::2:head 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: FIRST> seek to MIN oid=1 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=1 cursor=9:02547ec2:::1:head 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:02547ec2:::1:head 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:02547ec2:::1:head -> 1 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=10 cursor=9:52ea6a34:::10:head 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:52ea6a34:::10:head 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=13 cursor=9:566253c9:::13:head 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:566253c9:::13:head 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:566253c9:::13:head -> 13 2024-08-19T12:39:38.657 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=7 cursor=9:5c6b0b28:::7:head 2024-08-19T12:39:38.658 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2024-08-19T12:39:38.658 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2024-08-19T12:39:38.658 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=14 cursor=9:62a1935d:::14:head 2024-08-19T12:39:38.722 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:62a1935d:::14:head 2024-08-19T12:39:38.722 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:62a1935d:::14:head -> 14 2024-08-19T12:39:38.722 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=0 cursor=9:6cac518f:::0:head 2024-08-19T12:39:38.722 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:6cac518f:::0:head 2024-08-19T12:39:38.723 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:6cac518f:::0:head -> 0 2024-08-19T12:39:38.723 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=15 cursor=9:863748b0:::15:head 2024-08-19T12:39:38.723 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:863748b0:::15:head 2024-08-19T12:39:38.723 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:863748b0:::15:head -> 15 2024-08-19T12:39:38.723 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=11 cursor=9:89d3ae78:::11:head 2024-08-19T12:39:38.723 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:89d3ae78:::11:head 2024-08-19T12:39:38.723 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2024-08-19T12:39:38.723 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=5 cursor=9:b29083e3:::5:head 2024-08-19T12:39:38.723 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:b29083e3:::5:head 2024-08-19T12:39:38.723 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:b29083e3:::5:head -> 5 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=8 cursor=9:bd63b0f1:::8:head 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=6 cursor=9:c4fdafeb:::6:head 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=3 cursor=9:cfc208b3:::3:head 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:cfc208b3:::3:head 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=4 cursor=9:d83876eb:::4:head 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:d83876eb:::4:head 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:d83876eb:::4:head -> 4 2024-08-19T12:39:38.724 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=12 cursor=9:de5d7c5f:::12:head 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=9 cursor=9:e960b815:::9:head 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:e960b815:::9:head 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:e960b815:::9:head -> 9 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : oid=2 cursor=9:f905c69b:::2:head 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:f905c69b:::2:head 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:f905c69b:::2:head -> 2 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:52ea6a34:::10:head 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:52ea6a34:::10:head expected=9:52ea6a34:::10:head 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=10 expected=10 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:cfc208b3:::3:head 2024-08-19T12:39:38.725 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:cfc208b3:::3:head expected=9:cfc208b3:::3:head 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=3 expected=3 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:c4fdafeb:::6:head expected=9:c4fdafeb:::6:head 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=6 expected=6 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:bd63b0f1:::8:head expected=9:bd63b0f1:::8:head 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=8 expected=8 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:b29083e3:::5:head 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:b29083e3:::5:head expected=9:b29083e3:::5:head 2024-08-19T12:39:38.726 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:b29083e3:::5:head -> 5 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=5 expected=5 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:89d3ae78:::11:head 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:89d3ae78:::11:head expected=9:89d3ae78:::11:head 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=11 expected=11 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:863748b0:::15:head 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:863748b0:::15:head expected=9:863748b0:::15:head 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:863748b0:::15:head -> 15 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=15 expected=15 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:6cac518f:::0:head 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:6cac518f:::0:head expected=9:6cac518f:::0:head 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:6cac518f:::0:head -> 0 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=0 expected=0 2024-08-19T12:39:38.727 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:62a1935d:::14:head 2024-08-19T12:39:39.125 INFO:tasks.workunit.client.0.smithi104.stdout: api cls: Running main() from gmock_main.cc 2024-08-19T12:39:39.125 INFO:tasks.workunit.client.0.smithi104.stdout: cls: [==========] Running 1 test from 1 test suite. 2024-08-19T12:39:39.125 INFO:tasks.workunit.client.0.smithi104.stdout: cls: [----------] Global test environment set-up. 2024-08-19T12:39:39.125 INFO:tasks.workunit.client.0.smithi104.stdout: cls: [----------] 1 test from NeoRadosCls 2024-08-19T12:39:39.125 INFO:tasks.workunit.client.0.smithi104.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-08-19T12:39:39.125 INFO:tasks.workunit.client.0.smithi104.stdout: cls: [ OK ] NeoRadosCls.DNE (3312 ms) 2024-08-19T12:39:39.125 INFO:tasks.workunit.client.0.smithi104.stdout: cls: [----------] 1 test from NeoRadosCls (3312 ms total) 2024-08-19T12:39:39.125 INFO:tasks.workunit.client.0.smithi104.stdout: cls: 2024-08-19T12:39:39.126 INFO:tasks.workunit.client.0.smithi104.stdout: cls: [----------] Global test environment tear-down 2024-08-19T12:39:39.126 INFO:tasks.workunit.client.0.smithi104.stdout: cls: [==========] 1 test from 1 test suite ran. (3312 ms total) 2024-08-19T12:39:39.126 INFO:tasks.workunit.client.0.smithi104.stdout: cls: [ PASSED ] 1 test. 2024-08-19T12:39:39.126 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-08-19T12:39:39.126 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-08-19T12:39:39.126 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-08-19T12:39:39.126 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-08-19T12:39:39.126 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-08-19T12:39:39.126 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (56 ms) 2024-08-19T12:39:39.126 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-08-19T12:39:39.126 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (64 ms) 2024-08-19T12:39:39.126 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-08-19T12:39:39.127 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (3261 ms) 2024-08-19T12:39:39.127 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (3381 ms total) 2024-08-19T12:39:39.127 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: 2024-08-19T12:39:39.127 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-08-19T12:39:39.127 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (3381 ms total) 2024-08-19T12:39:39.127 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-08-19T12:39:39.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-08-19T12:39:39.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-08-19T12:39:39.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-08-19T12:39:39.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-08-19T12:39:39.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-08-19T12:39:39.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-08-19T12:39:39.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-08-19T12:39:39.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (288 ms) 2024-08-19T12:39:39.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-08-19T12:39:39.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (27 ms) 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (20 ms) 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (25 ms) 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (5 ms) 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (5 ms) 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (16 ms) 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (9 ms) 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-08-19T12:39:39.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (14 ms) 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (9 ms) 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (4 ms) 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (8 ms) 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (34 ms) 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (15 ms) 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (16 ms) 2024-08-19T12:39:39.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-08-19T12:39:39.134 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-08-19T12:39:39.134 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (499 ms total) 2024-08-19T12:39:39.134 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: 2024-08-19T12:39:39.134 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-08-19T12:39:39.134 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (3369 ms total) 2024-08-19T12:39:39.134 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-08-19T12:39:39.135 INFO:tasks.workunit.client.0.smithi104.stdout: handler_error: Running main() from gmock_main.cc 2024-08-19T12:39:39.135 INFO:tasks.workunit.client.0.smithi104.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-08-19T12:39:39.135 INFO:tasks.workunit.client.0.smithi104.stdout: handler_error: [----------] Global test environment set-up. 2024-08-19T12:39:39.135 INFO:tasks.workunit.client.0.smithi104.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-08-19T12:39:39.135 INFO:tasks.workunit.client.0.smithi104.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-08-19T12:39:39.135 INFO:tasks.workunit.client.0.smithi104.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (3259 ms) 2024-08-19T12:39:39.135 INFO:tasks.workunit.client.0.smithi104.stdout: handler_error: [----------] 1 test from neocls_handler_error (3259 ms total) 2024-08-19T12:39:39.135 INFO:tasks.workunit.client.0.smithi104.stdout: handler_error: 2024-08-19T12:39:39.135 INFO:tasks.workunit.client.0.smithi104.stdout: handler_error: [----------] Global test environment tear-down 2024-08-19T12:39:39.135 INFO:tasks.workunit.client.0.smithi104.stdout: handler_error: [==========] 1 test from 1 test suite ran. (3259 ms total) 2024-08-19T12:39:39.135 INFO:tasks.workunit.client.0.smithi104.stdout: handler_error: [ PASSED ] 1 test. 2024-08-19T12:39:39.307 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: Running main() from gmock_main.cc 2024-08-19T12:39:39.307 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-08-19T12:39:39.307 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [----------] Global test environment set-up. 2024-08-19T12:39:39.307 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-08-19T12:39:39.307 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-08-19T12:39:39.307 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (77 ms) 2024-08-19T12:39:39.307 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-08-19T12:39:39.307 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (66 ms) 2024-08-19T12:39:39.307 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-08-19T12:39:39.307 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (3276 ms) 2024-08-19T12:39:39.307 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-08-19T12:39:39.307 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:37.236792+0000 mon.f [INF] from='client.4555 v1:172.21.15.104:0/2532692816' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi104-24561-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T12:39:39.308 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:37.401257+0000 mon.a [INF] from='client.4555 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi104-24561-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T12:39:39.308 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.019639+0000 mon.a [INF] from='client.4754 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi104-24954-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:39.308 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.019898+0000 mon.a [INF] from='client.4742 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi104-25216-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:39.308 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.020020+0000 mon.a [INF] from='client.4709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi104-24597-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:39.308 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.020134+0000 mon.a [INF] from='client.4715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi104-25015-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:39.308 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.020229+0000 mon.a [INF] from='client.4682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi104-24517-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:39.308 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.020328+0000 mon.a [INF] from='client.4793 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi104-25103-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:39.308 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.020427+0000 mon.a [INF] from='client.4697 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi104-24748-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:39.308 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.020535+0000 mon.a [INF] from='client.? v1:172.21.15.104:0/4084487544' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi104-24631-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:39.308 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.020622+0000 mon.a [INF] from='client.4609 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi104-24837-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:39.308 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.020716+0000 mon.a [INF] from='client.4781 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi104-25238-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:39.961 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.020780+0000 mon.a [INF] from='client.4766 ' entity='client.admin' cmd='[{"prefix": "osd pool appl list_parallel: process_1_[25377]: starting. 2024-08-19T12:39:39.961 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_1_[25377]: creating pool ceph_test_rados_list_parallel.smithi104-25349 2024-08-19T12:39:39.961 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_1_[25377]: created object 0... 2024-08-19T12:39:39.961 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_1_[25377]: created object 25... 2024-08-19T12:39:39.961 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_1_[25377]: created object 49... 2024-08-19T12:39:39.961 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_1_[25377]: finishing. 2024-08-19T12:39:39.962 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_1_[25377]: shutting down. 2024-08-19T12:39:39.962 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_2_[25378]: starting. 2024-08-19T12:39:39.962 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_2_[25378]: listing objects. 2024-08-19T12:39:39.962 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_2_[25378]: listed object 0... 2024-08-19T12:39:39.962 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_2_[25378]: listed object 25... 2024-08-19T12:39:39.962 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_2_[25378]: saw 50 objects 2024-08-19T12:39:39.962 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_2_[25378]: shutting down. 2024-08-19T12:39:39.962 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:39.962 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_3_[26067]: starting. 2024-08-19T12:39:39.962 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_3_[26067]: creating pool ceph_test_rados_list_parallel.smithi104-25349 2024-08-19T12:39:39.962 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_3_[26067]: created object 0... 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_3_[26067]: created object 25... 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_3_[26067]: created object 49... 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_3_[26067]: finishing. 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_3_[26067]: shutting down. 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_4_[26068]: starting. 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_4_[26068]: listing objects. 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_4_[26068]: listed object 0... 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_4_[26068]: listed object 25... 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_4_[26068]: saw 40 objects 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_4_[26068]: shutting down. 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:39.963 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_5_[26069]: starting. 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_5_[26069]: removed 25 objects... 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_5_[26069]: removed half of the objects 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_5_[26069]: removed 50 objects... 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_5_[26069]: removed 50 objects 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_5_[26069]: shutting down. 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_6_[26396]: starting. 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_6_[26396]: creating pool ceph_test_rados_list_parallel.smithi104-25349 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_6_[26396]: created object 0... 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_6_[26396]: created object 25... 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_6_[26396]: created object 49... 2024-08-19T12:39:39.964 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_6_[26396]: finishing. 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_6_[26396]: shutting down. 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_7_[26397]: starting. 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_7_[26397]: listing objects. 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_7_[26397]: listed object 0... 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_7_[26397]: listed object 25... 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_7_[26397]: listed object 50... 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_7_[26397]: saw 55 objects 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_7_[26397]: shutting down. 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:39.965 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_8_[26398]: starting. 2024-08-19T12:39:39.966 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_8_[26398]: added 25 objects... 2024-08-19T12:39:39.966 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_8_[26398]: added half of the objects 2024-08-19T12:39:39.966 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_8_[26398]: added 50 objects... 2024-08-19T12:39:39.966 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_8_[26398]: added 50 objects 2024-08-19T12:39:39.966 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_8_[26398]: shutting down. 2024-08-19T12:39:39.966 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:39.966 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.221 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: **********************ication enable","pool": "test-rados-api-smithi104-25143-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:40.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.020909+0000 mon.a [INF] from='client.4652 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi104-24757-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:40.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.020997+0000 mon.a [INF] from='client.4591 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi104-24815-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:40.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.021058+0000 mon.a [INF] from='client.4656 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi104-24494-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:40.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.021130+0000 mon.a [INF] from='client.4649 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi104-24859-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:40.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.021240+0000 mon.a [INF] from='client.4674 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi104-24537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:40.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.021308+0000 mon.a [INF] from='client.4701 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi104-24660-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:40.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.021384+0000 mon.a [INF] from='client.4746 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi104-24727-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:40.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.394582+0000 mon.a [WRN] Health check failed: 34 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-19T12:39:40.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:38.819575+0000 mon.b [INF] from='client.? v1:172.21.15.104:0/1049005196' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi104-24597-1","var":"pg_num","val":"11"} : dispatch 2024-08-19T12:39:40.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.016003+0000 mon.a [INF] from='client.4709 ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi104-24597-1","var":"pg_num","val":"11"} : dispatch 2024-08-19T12:39:40.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.087120+0000 mon.a [INF] from='client.4796 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi104-25648-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi104-25648-1"}]': finished 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.087218+0000 mon.a [INF] from='client.4898 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi104-25779-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi104-25779-1"}]': finished 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.087291+0000 mon.a [INF] from='client.4672 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi104-25087-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi104-25087-1"}]': finished 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.087375+0000 mon.a [INF] from='client.4555 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi104-24561-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.087433+0000 mon.a [INF] from='client.4709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi104-24597-1","var":"pg_num","val":"11"}]': finished 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.111236+0000 mon.e [INF] from='client.? v1:172.21.15.104:0/3727808758' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi104-24748-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.111606+0000 mon.a [INF] from='client.4895 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi104-24748-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.219320+0000 mon.d [INF] from='client.? v1:172.21.15.104:0/3663116331' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi104-24537-16"} : dispatch 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.220630+0000 mon.a [INF] from='client.4893 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi104-24537-16"} : dispatch 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.221536+0000 mon.d [INF] from='client.? v1:172.21.15.104:0/3663116331' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi104-24537-16"} : dispatch 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.222252+0000 mon.a [INF] from='client.4893 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi104-24537-16"} : dispatch 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.223044+0000 mon.d [INF] from='client.? v1:172.21.15.104:0/3663116331' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi104-24537-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.228826+0000 mon.a [INF] from='client.4893 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi104-24537-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T12:39:40.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.287450+0000 mon.d [INF] from='client.? v1:172.21.15.104:0/2494039843' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi104-24859-7"} : dispatch 2024-08-19T12:39:40.225 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.289909+0000 mon.a [INF] from='client.4929 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi104-24859-7"} : dispatch 2024-08-19T12:39:40.225 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.290538+0000 mon.d [INF] from='client.? v1:172.21.15.104:0/2494039843' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi104-24859-7"} : dispatch 2024-08-19T12:39:40.225 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.290906+0000 mon.a [INF] from='client.4929 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi104-24859-7"} : dispatch 2024-08-19T12:39:40.225 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.291427+0000 mon.d [INF] from='client.? v1:172.21.15.104:0/2494039843' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi104-24859-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T12:39:40.225 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.291856+0000 mon.a [INF] from='client.4929 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi104-24859-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T12:39:40.225 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.292552+0000 mon.c [INF] from='client.? v1:172.21.15.104:0/2426368545' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi104-24954-7"} : dispatch 2024-08-19T12:39:40.225 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.292937+0000 mon.a [INF] from='client.4882 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi104-24954-7"} : dispatch 2024-08-19T12:39:40.401 INFO:tasks.workunit.client.0.smithi104.stdout: ********* 2024-08-19T12:39:40.401 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_9_[26447]: starting. 2024-08-19T12:39:40.402 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_9_[26447]: creating pool ceph_test_rados_list_parallel.smithi104-25349 2024-08-19T12:39:40.402 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_9_[26447]: created object 0... 2024-08-19T12:39:40.402 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_9_[26447]: created object 25... 2024-08-19T12:39:40.402 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_9_[26447]: created object 49... 2024-08-19T12:39:40.402 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_9_[26447]: finishing. 2024-08-19T12:39:40.402 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_9_[26447]: shutting down. 2024-08-19T12:39:40.402 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.402 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.402 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_10_[26448]: starting. 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_10_[26448]: listing objects. 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_10_[26448]: listed object 0... 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_10_[26448]: listed object 25... 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_10_[26448]: listed object 50... 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_10_[26448]: listed object 75... 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_10_[26448]: saw 98 objects 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_10_[26448]: shutting down. 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_13_[26451]: starting. 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_13_[26451]: removed 25 objects... 2024-08-19T12:39:40.403 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_13_[26451]: removed half of the objects 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_13_[26451]: removed 50 objects... 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_13_[26451]: removed 50 objects 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_13_[26451]: shutting down. 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_11_[26449]: starting. 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_11_[26449]: added 25 objects... 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_11_[26449]: added half of the objects 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_11_[26449]: added 50 objects... 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_11_[26449]: added 50 objects 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_11_[26449]: shutting down. 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.404 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.405 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_12_[26450]: starting. 2024-08-19T12:39:40.405 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_12_[26450]: added 25 objects... 2024-08-19T12:39:40.405 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_12_[26450]: added half of the objects 2024-08-19T12:39:40.405 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_12_[26450]: added 50 objects... 2024-08-19T12:39:40.405 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_12_[26450]: added 50 objects 2024-08-19T12:39:40.405 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_12_[26450]: shutting down. 2024-08-19T12:39:40.405 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.405 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.405 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.405 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.405 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_14_[26547]: starting. 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_14_[26547]: creating pool ceph_test_rados_list_parallel.smithi104-25349 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_14_[26547]: created object 0... 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_14_[26547]: created object 25... 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_14_[26547]: created object 49... 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_14_[26547]: finishing. 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_14_[26547]: shutting down. 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_15_[26548]: starting. 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_15_[26548]: listing objects. 2024-08-19T12:39:40.406 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_15_[26548]: listed object 0... 2024-08-19T12:39:42.202 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_15_[26548]: listed object 25... 2024-08-19T12:39:42.202 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_15_[26548]: listed object 50... 2024-08-19T12:39:42.202 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_15_[26548]: listed object 75... 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_15_[26548]: listed object 100... 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_15_[26548]: listed object 125... 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_15_[26548]: saw 150 objects 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_15_[26548]: shutting down. 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_16_[26549]: starting. 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_16_[26549]: added 25 objects... 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_16_[26549]: added half of the objects 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_16_[26549]: added 50 objects... 2024-08-19T12:39:42.203 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_16_[26549]: added 50 objects 2024-08-19T12:39:42.204 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: process_16_[26549]: shutting down. 2024-08-19T12:39:42.204 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:42.204 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:42.204 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:42.204 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:42.204 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******************************* 2024-08-19T12:39:42.204 INFO:tasks.workunit.client.0.smithi104.stdout: list_parallel: ******* SUCCESS ********** 2024-08-19T12:39:42.208 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_1_[25380]: starting. 2024-08-19T12:39:42.208 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_1_[25380]: creating pool ceph_test_rados_open_pools_parallel.smithi104-25328 2024-08-19T12:39:42.208 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_1_[25380]: created object 0... 2024-08-19T12:39:42.208 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_1_[25380]: created object 25... 2024-08-19T12:39:42.208 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_1_[25380]: created object 49... 2024-08-19T12:39:42.209 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_1_[25380]: finishing. 2024-08-19T12:39:42.209 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_1_[25380]: shutting down. 2024-08-19T12:39:42.209 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_2_[25381]: starting. 2024-08-19T12:39:42.209 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_2_[25381]: rados_pool_create. 2024-08-19T12:39:42.209 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_2_[25381]: rados_ioctx_create. 2024-08-19T12:39:42.209 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_2_[25381]: shutting down. 2024-08-19T12:39:42.209 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: ******************************* 2024-08-19T12:39:42.209 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_3_[26120]: starting. 2024-08-19T12:39:42.209 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_3_[26120]: creating pool ceph_test_rados_open_pools_parallel.smithi104-25328 2024-08-19T12:39:42.209 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_3_[26120]: created object 0... 2024-08-19T12:39:42.209 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_3_[26120]: created object 25... 2024-08-19T12:39:42.209 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_3_[26120]: created object 49... 2024-08-19T12:39:42.210 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_3_[26120]: finishing. 2024-08-19T12:39:42.210 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_3_[26120]: shutting down. 2024-08-19T12:39:42.210 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: ******************************* 2024-08-19T12:39:42.210 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_4_[26121]: starting. 2024-08-19T12:39:42.210 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_4_[26121]: rados_pool_create. 2024-08-19T12:39:42.210 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_4_[26121]: rados_ioctx_create. 2024-08-19T12:39:42.210 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: process_4_[26121]: shutting down. 2024-08-19T12:39:42.210 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: ******************************* 2024-08-19T12:39:42.211 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: ******************************* 2024-08-19T12:39:42.211 INFO:tasks.workunit.client.0.smithi104.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-08-19T12:39:42.219 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_1_[25353]: starting. 2024-08-19T12:39:42.219 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_1_[25353]: creating pool ceph_test_rados_delete_pools_parallel.smithi104-25276 2024-08-19T12:39:42.219 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_1_[25353]: created object 0... 2024-08-19T12:39:42.219 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_1_[25353]: created object 25... 2024-08-19T12:39:42.219 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_1_[25353]: created object 49... 2024-08-19T12:39:42.219 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_1_[25353]: finishing. 2024-08-19T12:39:42.219 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_1_[25353]: shutting down. 2024-08-19T12:39:42.220 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_2_[25354]: starting. 2024-08-19T12:39:42.220 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_2_[25354]: deleting pool ceph_test_rados_delete_pools_parallel.smithi104-25276 2024-08-19T12:39:42.220 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_2_[25354]: shutting down. 2024-08-19T12:39:42.220 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: ******************************* 2024-08-19T12:39:42.220 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_3_[26116]: starting. 2024-08-19T12:39:42.220 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_3_[26116]: creating pool ceph_test_rados_delete_pools_parallel.smithi104-25276 2024-08-19T12:39:42.220 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_3_[26116]: created object 0... 2024-08-19T12:39:42.221 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_3_[26116]: created object 25... 2024-08-19T12:39:42.221 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_3_[26116]: created object 49... 2024-08-19T12:39:42.221 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_3_[26116]: finishing. 2024-08-19T12:39:42.221 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_3_[26116]: shutting down. 2024-08-19T12:39:42.221 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: ******************************* 2024-08-19T12:39:42.221 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_5_[26119]: starting. 2024-08-19T12:39:42.221 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_5_[26119]: listing objects. 2024-08-19T12:39:42.221 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_5_[26119]: listed object 0... 2024-08-19T12:39:42.221 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_5_[26119]: listed object 25... 2024-08-19T12:39:42.222 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_5_[26119]: saw 50 objects 2024-08-19T12:39:42.222 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_5_[26119]: shutting down. 2024-08-19T12:39:42.222 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: ******************************* 2024-08-19T12:39:42.222 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_4_[26117]: starting. 2024-08-19T12:39:42.222 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_4_[26117]: deleting pool ceph_test_rados_delete_pools_parallel.smithi104-25276 2024-08-19T12:39:42.222 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: process_4_[26117]: shutting down. 2024-08-19T12:39:42.222 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: ******************************* 2024-08-19T12:39:42.222 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: ******************************* 2024-08-19T12:39:42.222 INFO:tasks.workunit.client.0.smithi104.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-08-19T12:39:43.201 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: Running main() from gmock_main.cc 2024-08-19T12:39:43.201 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-08-19T12:39:43.201 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [----------] Global test environment set-up. 2024-08-19T12:39:43.201 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-08-19T12:39:43.201 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-08-19T12:39:43.201 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (2186 ms) 2024-08-19T12:39:43.201 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-08-19T12:39:43.201 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (2061 ms) 2024-08-19T12:39:43.201 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-08-19T12:39:43.202 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3083 ms) 2024-08-19T12:39:43.202 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [----------] 3 tests from NeoRadosCmd (7330 ms total) 2024-08-19T12:39:43.202 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: 2024-08-19T12:39:43.202 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [----------] Global test environment tear-down 2024-08-19T12:39:43.202 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [==========] 3 tests from 1 test suite ran. (7330 ms total) 2024-08-19T12:39:43.202 INFO:tasks.workunit.client.0.smithi104.stdout: cmd: [ PASSED ] 3 tests. 2024-08-19T12:39:43.269 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.299306+0000 mon.c [INF] from='client.? v1:172.21.15.104:0/2426368545' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi104-24954-7"} : dispatch 2024-08-19T12:39:43.269 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.302792+0000 mon.a [INF] from='client.4882 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi104-24954-7"} : dispatch 2024-08-19T12:39:43.269 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.303717+0000 mon.c [INF] from='client.? v1:172.21.15.104:0/2426368545' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi104-24954-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T12:39:43.269 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.306612+0000 mon.a [INF] from='client.4882 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi104-24954-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.306886+0000 mon.b [INF] from='client.4900 v1:172.21.15.104:0/2885672719' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.307848+0000 client.admin [INF] onexx 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:39.310548+0000 mon.a [INF] from='client.4900 ' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:40.092322+0000 mon.a [INF] from='client.4895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi104-24748-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:40.092425+0000 mon.a [INF] from='client.4893 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi104-24537-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:40.092513+0000 mon.a [INF] from='client.4929 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi104-24859-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:40.092605+0000 mon.a [INF] from='client.4882 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi104-24954-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:41.236016+0000 mon.a [INF] from='client.? v1:172.21.15.104:0/744868190' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi104-24660-10"} : dispatch 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:41.236905+0000 mon.a [INF] from='client.? v1:172.21.15.104:0/744868190' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi104-24660-10"} : dispatch 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:41.237290+0000 mon.a [INF] from='client.? v1:172.21.15.104:0/744868190' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi104-24660-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:41.263892+0000 mon.a [INF] from='client.4900 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:41.978271+0000 mon.a [WRN] Health check failed: 1/9 mons down, quorum a,f,b,g,c,h,d,e (MON_DOWN) 2024-08-19T12:39:43.270 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.164020+0000 mon.a [INF] from='client.4929 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi104-24859-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi104-24859-7"}]': finished 2024-08-19T12:39:43.271 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.164126+0000 mon.a [INF] from='client.4893 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi104-24537-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi104-24537-16"}]': finished 2024-08-19T12:39:43.271 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.164247+0000 mon.a [INF] from='client.4882 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi104-24954-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi104-24954-7"}]': finished 2024-08-19T12:39:43.271 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.164344+0000 mon.a [INF] from='client.4796 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi104-25648-1"}]': finished 2024-08-19T12:39:43.271 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.164412+0000 mon.a [INF] from='client.4672 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi104-25087-1"}]': finished 2024-08-19T12:39:43.271 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.164506+0000 mon.a [INF] from='client.4895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi104-24748-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:43.271 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.164565+0000 mon.a [INF] from='client.4898 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi104-25779-1"}]': finished 2024-08-19T12:39:43.272 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.164621+0000 mon.a [INF] from='client.5176 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi104-24631-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T12:39:43.272 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.164725+0000 mon.a [INF] from='client.? v1:172.21.15.104:0/744868190' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi104-24660-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T12:39:43.272 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.168763+0000 mon.b [INF] from='client.? v1:172.21.15.104:0/327507335' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi104-25779-1"} : dispatch 2024-08-19T12:39:43.272 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.170941+0000 mon.f [INF] from='client.? v1:172.21.15.104:0/3035443996' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi104-25087-1"} : dispatch 2024-08-19T12:39:43.272 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.171780+0000 mon.e [INF] from='client.? v1:172.21.15.104:0/3727808758' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi104-24748-4", "tierpool": "test-rados-api-smithi104-24748-4-cache"} : dispatch 2024-08-19T12:39:43.272 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.171802+0000 mon.f [INF] from='client.? v1:172.21.15.104:0/4137695674' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi104-24631-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi104-24631-10"} : dispatch 2024-08-19T12:39:43.272 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.178750+0000 mon.a [INF] from='client.? v1:172.21.15.104:0/744868190' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi104-24660-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi104-24660-10"} : dispatch 2024-08-19T12:39:43.272 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.179142+0000 mon.a [INF] from='client.4672 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi104-25087-1"} : dispatch 2024-08-19T12:39:43.797 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.179501+0000 mon.a [INF] from='client.4898 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi104-25779-1"} : dispatch 2024-08-19T12:39:43.798 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.179998+0000 mon.a [INF] from='client.4895 ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi104-24748-4", "tierpool": "test-rados-api-smithi104-24748-4-cache"} : dispatch 2024-08-19T12:39:43.798 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.180338+0000 mon.a [INF] from='client.5176 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi104-24631-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi104-24631-10"} : dispatch 2024-08-19T12:39:43.798 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.180505+0000 mon.h [INF] from='client.? v1:172.21.15.104:0/3379343529' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi104-24517-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T12:39:43.798 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.184800+0000 mon.a [INF] from='client.4910 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi104-24517-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T12:39:43.798 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.262157+0000 mon.d [INF] from='client.? v1:172.21.15.104:0/1007504900' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi104-25648-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T12:39:43.798 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:42.265189+0000 mon.a [INF] from='client.5217 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi104-25648-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T12:39:43.798 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.168376+0000 mon.a [INF] from='client.5032 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi104-24561-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi104-24561-24"}]': finished 2024-08-19T12:39:43.798 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.168475+0000 mon.a [INF] from='client.4672 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi104-25087-1"}]': finished 2024-08-19T12:39:43.798 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.168554+0000 mon.a [INF] from='client.4898 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi104-25779-1"}]': finished 2024-08-19T12:39:43.798 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.168632+0000 mon.a [INF] from='client.4895 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi104-24748-4", "tierpool": "test-rados-api-smithi104-24748-4-cache"}]': finished 2024-08-19T12:39:43.798 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.168711+0000 mon.a [INF] from='client.4910 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi104-24517-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-19T12:39:43.799 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.168899+0000 mon.a [INF] from='client.5217 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi104-25648-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-19T12:39:43.799 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.174529+0000 mon.e [INF] from='client.? v1:172.21.15.104:0/3727808758' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi104-24748-4", "overlaypool": "test-rados-api-smithi104-24748-4-cache"} : dispatch 2024-08-19T12:39:43.799 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.174619+0000 mon.d [INF] from='client.? v1:172.21.15.104:0/1007504900' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi104-25648-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi104-25648-2"} : dispatch 2024-08-19T12:39:44.501 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.185461+0000 mon.h [INF] from='client.? v1:172.21.15.104:0/3379343529' entity='client.admin' cmd={"prefix": "osd pool set api_io_pp: Running main() from gmock_main.cc 2024-08-19T12:39:44.501 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [==========] Running 40 tests from 2 test suites. 2024-08-19T12:39:44.501 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [----------] Global test environment set-up. 2024-08-19T12:39:44.501 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP 2024-08-19T12:39:44.501 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: seed 24561 2024-08-19T12:39:44.501 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-08-19T12:39:44.501 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-08-19T12:39:44.502 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-08-19T12:39:44.502 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (154 ms) 2024-08-19T12:39:44.502 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-08-19T12:39:44.502 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (41 ms) 2024-08-19T12:39:44.502 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-08-19T12:39:44.502 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (15 ms) 2024-08-19T12:39:44.502 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-08-19T12:39:44.502 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (426 ms) 2024-08-19T12:39:44.502 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-08-19T12:39:44.502 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (6 ms) 2024-08-19T12:39:44.502 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-08-19T12:39:44.502 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (1 ms) 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (7 ms) 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (1 ms) 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (5 ms) 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (7 ms) 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (8 ms) 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (6 ms) 2024-08-19T12:39:44.503 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (3 ms) 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (2 ms) 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (5 ms) 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (9 ms) 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (4 ms) 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CrcZeroWrite 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.CrcZeroWrite (8 ms) 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (4 ms) 2024-08-19T12:39:44.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-08-19T12:39:44.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (10 ms) 2024-08-19T12:39:44.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-08-19T12:39:44.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (5 ms) 2024-08-19T12:39:44.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP (727 ms total) 2024-08-19T12:39:44.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: 2024-08-19T12:39:44.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP 2024-08-19T12:39:44.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-08-19T12:39:44.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1303 ms) 2024-08-19T12:39:44.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-08-19T12:39:44.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (15 ms) 2024-08-19T12:39:44.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-08-19T12:39:44.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (8 ms) 2024-08-19T12:39:45.214 INFO:tasks.workunit.client.0.smithi104.stdout: list: Running main() from gmock_main.cc 2024-08-19T12:39:45.214 INFO:tasks.workunit.client.0.smithi104.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: [----------] Global test environment set-up. 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: [----------] 3 tests from NeoradosList 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: [ OK ] NeoradosList.ListObjects (3207 ms) 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3101 ms) 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: [ OK ] NeoradosList.ListObjectsMany (2987 ms) 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: [----------] 3 tests from NeoradosList (9295 ms total) 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: [----------] Global test environment tear-down 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: [==========] 3 tests from 1 test suite ran. (9295 ms total) 2024-08-19T12:39:45.215 INFO:tasks.workunit.client.0.smithi104.stdout: list: [ PASSED ] 3 tests. 2024-08-19T12:39:46.216 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: Running main() from gmock_main.cc 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: [----------] Global test environment set-up. 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: handle_notify cookie 93975581152480 notify_id 94489280512 notifier_gid 4868 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (3164 ms) 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: Trying... 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: handle_notify cookie 93975593230416 notify_id 107374182402 notifier_gid 4940 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: Waiting for 3.000000000s 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: Timed out. 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: Flushing... 2024-08-19T12:39:46.217 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: Flushed... 2024-08-19T12:39:46.218 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7089 ms) 2024-08-19T12:39:46.218 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (10253 ms total) 2024-08-19T12:39:46.218 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: 2024-08-19T12:39:46.218 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: [----------] Global test environment tear-down 2024-08-19T12:39:46.218 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (10253 ms total) 2024-08-19T12:39:46.218 INFO:tasks.workunit.client.0.smithi104.stdout: watch_notify: [ PASSED ] 2 tests. 2024-08-19T12:39:46.225 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-08-19T12:39:46.225 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-08-19T12:39:46.225 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-08-19T12:39:46.226 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-08-19T12:39:46.226 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: seed 24954 2024-08-19T12:39:46.226 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-08-19T12:39:46.226 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (291 ms) 2024-08-19T12:39:46.226 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-08-19T12:39:46.226 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (12 ms) 2024-08-19T12:39:46.226 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-08-19T12:39:46.226 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-08-19T12:39:46.226 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-08-19T12:39:46.226 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (2 ms) 2024-08-19T12:39:46.226 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-08-19T12:39:46.226 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (18 ms) 2024-08-19T12:39:46.227 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (324 ms total) 2024-08-19T12:39:46.227 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: 2024-08-19T12:39:46.227 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-08-19T12:39:46.227 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-08-19T12:39:46.227 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1093 ms) 2024-08-19T12:39:46.227 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-08-19T12:39:46.227 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-08-19T12:39:46.227 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-08-19T12:39:46.227 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (531 ms) 2024-08-19T12:39:46.227 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-08-19T12:39:46.228 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (17 ms) 2024-08-19T12:39:46.228 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1641 ms total) 2024-08-19T12:39:46.228 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: 2024-08-19T12:39:46.228 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-08-19T12:39:46.228 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (10533 ms total) 2024-08-19T12:39:46.228 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-08-19T12:39:46.239 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: Running main() from gmock_main.cc 2024-08-19T12:39:46.240 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-08-19T12:39:46.240 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [----------] Global test environment set-up. 2024-08-19T12:39:46.240 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-08-19T12:39:46.240 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-08-19T12:39:46.240 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (207 ms) 2024-08-19T12:39:46.240 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-08-19T12:39:46.240 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-08-19T12:39:46.240 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-08-19T12:39:46.240 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: no timeout :/ 2024-08-19T12:39:46.240 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: no timeout :/ 2024-08-19T12:39:46.240 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: no timeout :/ 2024-08-19T12:39:46.240 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: no timeout :/ 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: no timeout :/ 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (66 ms) 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (20 ms) 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.Checksum (6 ms) 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (8 ms) 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (8 ms) 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (7 ms) 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-08-19T12:39:46.241 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (6 ms) 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (3 ms) 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (4 ms) 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.RmXattr (15 ms) 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIo.XattrIter (16 ms) 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [----------] 14 tests from LibRadosIo (370 ms total) 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: 2024-08-19T12:39:46.242 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-08-19T12:39:46.243 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-08-19T12:39:46.243 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1086 ms) 2024-08-19T12:39:46.243 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-08-19T12:39:46.243 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (20 ms) 2024-08-19T12:39:46.243 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-08-19T12:39:46.243 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (18 ms) 2024-08-19T12:39:46.243 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-08-19T12:39:46.243 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (3 ms) 2024-08-19T12:39:46.243 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-08-19T12:39:46.243 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7 ms) 2024-08-19T12:39:46.243 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-08-19T12:39:46.243 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (3 ms) 2024-08-19T12:39:46.244 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-08-19T12:39:46.244 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-08-19T12:39:46.244 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-08-19T12:39:46.244 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-08-19T12:39:46.244 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-08-19T12:39:46.244 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (8 ms) 2024-08-19T12:39:46.244 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-08-19T12:39:46.244 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-08-19T12:39:46.244 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1155 ms total) 2024-08-19T12:39:46.244 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [--- api_stat: Running main() from gmock_main.cc 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [----------] Global test environment set-up. 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ OK ] LibRadosStat.Stat (253 ms) 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (26 ms) 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ OK ] LibRadosStat.StatNS (39 ms) 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-08-19T12:39:46.245 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-08-19T12:39:46.246 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (2 ms) 2024-08-19T12:39:46.246 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [----------] 5 tests from LibRadosStat (320 ms total) 2024-08-19T12:39:46.246 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: 2024-08-19T12:39:46.246 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-08-19T12:39:46.246 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-08-19T12:39:46.246 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1086 ms) 2024-08-19T12:39:46.246 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-08-19T12:39:46.246 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (24 ms) 2024-08-19T12:39:46.246 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-08-19T12:39:46.246 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (510 ms) 2024-08-19T12:39:46.246 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-08-19T12:39:46.247 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (6 ms) 2024-08-19T12:39:46.247 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1626 ms total) 2024-08-19T12:39:46.247 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: 2024-08-19T12:39:46.247 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [----------] Global test environment tear-down 2024-08-19T12:39:46.247 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (10593 ms total) 2024-08-19T12:39:46.247 INFO:tasks.workunit.client.0.smithi104.stdout: api_stat: [ PASSED ] 9 tests. 2024-08-19T12:39:46.247 INFO:tasks.workunit.client.0.smithi104.stdout:-------] Global test environment tear-down 2024-08-19T12:39:46.247 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [==========] 24 tests from 2 test suites ran. (10716 ms total) 2024-08-19T12:39:46.247 INFO:tasks.workunit.client.0.smithi104.stdout: api_io: [ PASSED ] 24 tests. 2024-08-19T12:39:46.803 INFO:tasks.workunit.client.0.smithi104.stdout:-quota", "pool": "PoolQuotaPP_smithi104-24517-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-08-19T12:39:46.803 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.191198+0000 mon.f [INF] from='client.? v1:172.21.15.104:0/3069904417' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi104-25216-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T12:39:46.804 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.192177+0000 mon.a [INF] from='client.4895 ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi104-24748-4", "overlaypool": "test-rados-api-smithi104-24748-4-cache"} : dispatch 2024-08-19T12:39:46.804 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.192476+0000 mon.a [INF] from='client.4910 ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi104-24517-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-08-19T12:39:46.804 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.192763+0000 mon.a [INF] from='client.5217 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi104-25648-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi104-25648-2"} : dispatch 2024-08-19T12:39:46.804 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.195505+0000 mon.a [INF] from='client.5221 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi104-25216-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T12:39:46.804 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.195947+0000 mon.e [INF] from='client.? v1:172.21.15.104:0/1451494997' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi104-24757-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T12:39:46.804 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.196601+0000 mon.a [INF] from='client.5219 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi104-24757-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T12:39:46.805 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.200528+0000 mon.f [INF] from='client.? v1:172.21.15.104:0/2513849613' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi104-24494-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T12:39:46.805 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.201176+0000 mon.a [INF] from='client.5230 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi104-24494-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-19T12:39:46.805 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.237682+0000 mon.g [INF] from='client.? v1:172.21.15.104:0/4253685607' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi104-25779-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T12:39:46.805 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.238092+0000 mon.a [INF] from='client.5331 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi104-25779-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-08-19T12:39:46.805 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.269476+0000 client.admin [INF] threexx 2024-08-19T12:39:46.805 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: got: 2024-08-19T12:39:43.269886+0000 mon.a [INF] from='client.4900 ' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-08-19T12:39:46.805 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7672 ms) 2024-08-19T12:39:46.805 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (11091 ms total) 2024-08-19T12:39:46.805 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: 2024-08-19T12:39:46.806 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [----------] Global test environment tear-down 2024-08-19T12:39:46.806 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (11091 ms total) 2024-08-19T12:39:46.806 INFO:tasks.workunit.client.0.smithi104.stdout: api_cmd: [ PASSED ] 4 tests. 2024-08-19T12:39:49.039 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: Running main() from gmock_main.cc 2024-08-19T12:39:49.039 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-08-19T12:39:49.039 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: [----------] Global test environment set-up. 2024-08-19T12:39:49.039 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-08-19T12:39:49.040 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-08-19T12:39:49.040 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (6284 ms) 2024-08-19T12:39:49.040 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-08-19T12:39:49.040 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6856 ms) 2024-08-19T12:39:49.040 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (13140 ms total) 2024-08-19T12:39:49.040 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: 2024-08-19T12:39:49.040 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: [----------] Global test environment tear-down 2024-08-19T12:39:49.040 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (13140 ms total) 2024-08-19T12:39:49.040 INFO:tasks.workunit.client.0.smithi104.stdout: ec_io: [ PASSED ] 2 tests. 2024-08-19T12:39:49.168 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: notify 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1095 ms) 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (11 ms) 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1106 ms total) 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-08-19T12:39:49.169 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: notify 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (9 ms) 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: notify 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3752 ms) 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (9 ms) 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (8 ms) 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: handle_notify cookie 94401107678416 notify_id 141733920771 notifier_gid 5139 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (5 ms) 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-08-19T12:39:49.170 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: handle_notify cookie 94401107678416 notify_id 141733920771 notifier_gid 5139 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: handle_notify cookie 94401107678416 notify_id 141733920772 notifier_gid 5139 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (10 ms) 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: handle_notify cookie 94401107680544 notify_id 141733920773 notifier_gid 5139 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (10 ms) 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: handle_notify cookie 94401107678416 notify_id 141733920773 notifier_gid 5139 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (18 ms) 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: handle_notify cookie 94401107678416 notify_id 141733920774 notifier_gid 5139 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (4 ms) 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: handle_notify cookie 94401107678416 notify_id 141733920772 notifier_gid 5139 2024-08-19T12:39:49.171 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (13 ms) 2024-08-19T12:39:49.172 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-08-19T12:39:49.172 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: trying... 2024-08-19T12:39:50.044 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: handle_notify cookie 94401107678416 notify_id 141733920774 notifie api_lock_pp: Running main() from gmock_main.cc 2024-08-19T12:39:50.044 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-08-19T12:39:50.044 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-08-19T12:39:50.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-08-19T12:39:50.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: seed 24631 2024-08-19T12:39:50.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-08-19T12:39:50.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (179 ms) 2024-08-19T12:39:50.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-08-19T12:39:50.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (71 ms) 2024-08-19T12:39:50.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-08-19T12:39:50.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1038 ms) 2024-08-19T12:39:50.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-08-19T12:39:50.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1020 ms) 2024-08-19T12:39:50.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (9 ms) 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4 ms) 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (5 ms) 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2329 ms total) 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1114 ms) 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-08-19T12:39:50.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (12 ms) 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1066 ms) 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1004 ms) 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (10 ms) 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (18 ms) 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (12 ms) 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (2 ms) 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3238 ms total) 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: 2024-08-19T12:39:50.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-08-19T12:39:50.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (14468 ms total) 2024-08-19T12:39:50.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-08-19T12:39:50.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: Running main() from gmock_main.cc 2024-08-19T12:39:50.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-08-19T12:39:50.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [----------] Global test environment set-up. 2024-08-19T12:39:50.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-08-19T12:39:50.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-08-19T12:39:50.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (211 ms) 2024-08-19T12:39:50.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-08-19T12:39:50.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLock.LockShared (4 ms) 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1101 ms) 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1019 ms) 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (10 ms) 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLock.Unlock (4 ms) 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (6 ms) 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-08-19T12:39:50.064 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [----------] 8 tests from LibRadosLock (2358 ms total) 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1103 ms) 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (9 ms) 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1061 ms) 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1008 ms) 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (10 ms) 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-08-19T12:39:50.065 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (13 ms) 2024-08-19T12:39:50.066 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-08-19T12:39:50.066 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (8 ms) 2024-08-19T12:39:50.066 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-08-19T12:39:50.066 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3 ms) 2024-08-19T12:39:50.066 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3215 ms total) 2024-08-19T12:39:50.066 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: 2024-08-19T12:39:50.066 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [----------] Global test environment tear-down 2024-08-19T12:39:50.066 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (14490 ms total) 2024-08-19T12:39:50.066 INFO:tasks.workunit.client.0.smithi104.stdout: api_lock: [ PASSED ] 16 tests. 2024-08-19T12:39:50.066 INFO:tasks.workunit.client.0.smithi104.stdout: pool: Running main() from gmock_main.cc 2024-08-19T12:39:50.066 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-08-19T12:39:50.066 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [----------] Global test environment set-up. 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ OK ] NeoRadosPools.PoolList (2119 ms) 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2059 ms) 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2076 ms) 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4028 ms) 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1782 ms) 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2056 ms) 2024-08-19T12:39:50.067 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [----------] 6 tests from NeoRadosPools (14120 ms total) 2024-08-19T12:39:50.068 INFO:tasks.workunit.client.0.smithi104.stdout: pool: 2024-08-19T12:39:50.068 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [----------] Global test environment tear-down 2024-08-19T12:39:50.068 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [==========] 6 tests from 1 test suite ran. (14121 ms total) 2024-08-19T12:39:50.068 INFO:tasks.workunit.client.0.smithi104.stdout: pool: [ PASSED ] 6 tests. 2024-08-19T12:39:55.054 INFO:tasks.workunit.client.0.smithi104.stdout:api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-08-19T12:39:55.054 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (10 ms) 2024-08-19T12:39:55.054 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-08-19T12:39:55.054 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (2 ms) 2024-08-19T12:39:55.054 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-08-19T12:39:55.054 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-08-19T12:39:55.054 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-08-19T12:39:55.054 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (14 ms) 2024-08-19T12:39:55.054 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-08-19T12:39:55.054 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (4 ms) 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (5 ms) 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (6 ms) 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (4 ms) 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (9 ms) 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CrcZeroWrite 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CrcZeroWrite (6502 ms) 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-08-19T12:39:55.055 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (1214 ms) 2024-08-19T12:39:55.056 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-08-19T12:39:55.056 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (14 ms) 2024-08-19T12:39:55.056 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-08-19T12:39:55.056 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (3 ms) 2024-08-19T12:39:55.056 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-08-19T12:39:55.056 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (18 ms) 2024-08-19T12:39:55.056 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP (9139 ms total) 2024-08-19T12:39:55.056 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: 2024-08-19T12:39:55.056 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [----------] Global test environment tear-down 2024-08-19T12:39:55.056 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [==========] 40 tests from 2 test suites ran. (19509 ms total) 2024-08-19T12:39:55.056 INFO:tasks.workunit.client.0.smithi104.stdout: api_io_pp: [ PASSED ] 40 tests. 2024-08-19T12:39:57.114 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-08-19T12:39:57.114 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-08-19T12:39:57.114 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify_test_cb 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (305 ms) 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94482302360960 err -107 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1011 ms) 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94482302362112 err -107 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1019 ms) 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-08-19T12:39:57.115 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify2_test_cb from 4715 notify_id 103079215104 cookie 94482302362112 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (10 ms) 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify2_test_cb from 4715 notify_id 103079215106 cookie 94482302362112 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (10 ms) 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify2_test_cb from 4715 notify_id 103079215104 cookie 94482302362112 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (5 ms) 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify2_test_cb from 4715 notify_id 103079215105 cookie 94482302362112 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify2_test_cb from 4715 notify_id 103079215105 cookie 94482302401536 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (6 ms) 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-08-19T12:39:57.116 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify2_test_cb from 4715 notify_id 103079215105 cookie 94482302362112 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify2_test_cb from 4715 notify_id 107374182403 cookie 94482302362112 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3006 ms) 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94482302390256 err -107 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify2_test_cb from 4715 notify_id 137438953473 cookie 94482302390256 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5011 ms) 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94482302393440 err -107 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1010 ms) 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11393 ms total) 2024-08-19T12:39:57.117 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: 2024-08-19T12:39:57.118 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-08-19T12:39:57.118 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-08-19T12:39:57.118 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: watch_notify_test_cb 2024-08-19T12:39:57.118 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1115 ms) 2024-08-19T12:39:57.118 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1115 ms total) 2024-08-19T12:39:57.118 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: 2024-08-19T12:39:57.118 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-08-19T12:39:57.118 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (21424 ms total) 2024-08-19T12:39:57.118 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-08-19T12:39:58.619 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: Running main() from gmock_main.cc 2024-08-19T12:39:58.619 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-08-19T12:39:58.619 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [----------] Global test environment set-up. 2024-08-19T12:39:58.620 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-08-19T12:39:58.620 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-08-19T12:39:58.620 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (7251 ms) 2024-08-19T12:39:58.620 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-08-19T12:39:58.620 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6874 ms) 2024-08-19T12:39:58.620 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-08-19T12:39:58.620 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (8558 ms) 2024-08-19T12:39:58.620 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [----------] 3 tests from NeoradosECList (22683 ms total) 2024-08-19T12:39:58.620 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: 2024-08-19T12:39:58.620 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [----------] Global test environment tear-down 2024-08-19T12:39:58.620 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (22683 ms total) 2024-08-19T12:39:58.620 INFO:tasks.workunit.client.0.smithi104.stdout: ec_list: [ PASSED ] 3 tests. 2024-08-19T12:39:58.629 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-08-19T12:39:58.629 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-08-19T12:39:58.629 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (3381 ms) 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3072 ms) 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3339 ms) 2024-08-19T12:39:58.630 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-08-19T12:39:58.631 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2484 ms) 2024-08-19T12:39:58.631 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-08-19T12:39:58.631 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3027 ms) 2024-08-19T12:39:58.631 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-08-19T12:39:58.631 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (2994 ms) 2024-08-19T12:39:58.631 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-08-19T12:39:58.631 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (4581 ms) 2024-08-19T12:39:58.631 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (22878 ms total) 2024-08-19T12:39:58.631 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: 2024-08-19T12:39:58.631 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-08-19T12:39:58.631 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (22878 ms total) 2024-08-19T12:39:58.631 INFO:tasks.workunit.client.0.smithi104.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-08-19T12:39:58.680 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: Running main() from gmock_main.cc 2024-08-19T12:39:58.680 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [----------] Global test environment set-up. 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (3173 ms) 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3074 ms) 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3015 ms) 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (2790 ms) 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3057 ms) 2024-08-19T12:39:58.681 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-08-19T12:39:58.682 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (2981 ms) 2024-08-19T12:39:58.682 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-08-19T12:39:58.682 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (4637 ms) 2024-08-19T12:39:58.682 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (22727 ms total) 2024-08-19T12:39:58.682 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: 2024-08-19T12:39:58.682 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [----------] Global test environment tear-down 2024-08-19T12:39:58.682 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (22727 ms total) 2024-08-19T12:39:58.682 INFO:tasks.workunit.client.0.smithi104.stdout: write_operations: [ PASSED ] 7 tests. 2024-08-19T12:40:03.098 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-08-19T12:40:03.098 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [----------] Global test environment set-up. 2024-08-19T12:40:03.099 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [----------] 4 tests from LibRadosService 2024-08-19T12:40:03.099 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-08-19T12:40:03.099 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5083 ms) 2024-08-19T12:40:03.099 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-08-19T12:40:03.099 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [ OK ] LibRadosService.RegisterLate (21 ms) 2024-08-19T12:40:03.099 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-08-19T12:40:03.099 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: cluster: 2024-08-19T12:40:03.099 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: id: 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:03.099 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: health: HEALTH_WARN 2024-08-19T12:40:03.099 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 10 pool(s) do not have an application enabled 2024-08-19T12:40:03.099 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: services: 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: mon: 9 daemons, quorum a,f,b,g,c,h,d,e (age 43s), out of quorum: i 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: mgr: x(active, since 87s) 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: osd: 6 osds: 6 up (since 80s), 6 in (since 88s) 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: data: 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: pools: 21 pools, 545 pgs 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: objects: 4 objects, 577 KiB 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: usage: 168 MiB used, 540 GiB / 540 GiB avail 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: pgs: 42.385% pgs unknown 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 55.963% pgs not active 2024-08-19T12:40:03.100 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 305 creating+peering 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 231 unknown 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 9 active+clean 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: cluster: 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: id: 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: health: HEALTH_WARN 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 1/9 mons down, quorum a,f,b,g,c,h,d,e 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 14 pool(s) do not have an application enabled 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: services: 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: mon: 9 daemons, quorum a,f,b,g,c,h,d,e (age 45s), out of quorum: i 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: mgr: x(active, since 89s) 2024-08-19T12:40:03.101 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: osd: 6 osds: 6 up (since 82s), 6 in (since 90s) 2024-08-19T12:40:03.102 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-08-19T12:40:03.102 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-08-19T12:40:03.102 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 2024-08-19T12:40:03.102 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: data: 2024-08-19T12:40:03.102 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: pools: 27 pools, 809 pgs 2024-08-19T12:40:03.102 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: objects: 16 objects, 578 KiB 2024-08-19T12:40:03.102 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: usage: 172 MiB used, 540 GiB / 540 GiB avail 2024-08-19T12:40:03.102 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: pgs: 66.996% pgs unknown 2024-08-19T12:40:03.102 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 24.969% pgs not active 2024-08-19T12:40:03.102 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 542 unknown 2024-08-19T12:40:03.102 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 202 creating+peering 2024-08-19T12:40:03.102 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 65 active+clean 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: io: 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: client: 0 B/s rd, 6.5 KiB/s wr, 172 op/s rd, 182 op/s wr 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2169 ms) 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [ RUN ] LibRadosService.Status 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [ OK ] LibRadosService.Status (20059 ms) 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [----------] 4 tests from LibRadosService (27332 ms total) 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [----------] Global test environment tear-down 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27332 ms total) 2024-08-19T12:40:03.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_service: [ PASSED ] 4 tests. 2024-08-19T12:40:04.130 INFO:tasks.workunit.client.0.smithi104.stdout:r_gid 5139 2024-08-19T12:40:04.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: timed out 2024-08-19T12:40:04.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: flushing 2024-08-19T12:40:04.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: flushed 2024-08-19T12:40:04.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3004 ms) 2024-08-19T12:40:04.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: trying... 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: handle_notify cookie 94401107678416 notify_id 154618822663 notifier_gid 5139 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: timed out 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: flushing 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: flushed 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3005 ms) 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: List watches 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: notify2 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: handle_notify cookie 94401107678416 notify_id 167503724552 notifier_gid 5139 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: notify2 done 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: watch_check 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: unwatch2 2024-08-19T12:40:04.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: flushing 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: done 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3011 ms) 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: List watches 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: notify2 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: handle_notify cookie 94401107673744 notify_id 176093659141 notifier_gid 5139 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: notify2 done 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: watch_check 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: unwatch2 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: flushing 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: done 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3012 ms) 2024-08-19T12:40:04.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (15870 ms total) 2024-08-19T12:40:04.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: 2024-08-19T12:40:04.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-08-19T12:40:04.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28379 ms total) 2024-08-19T12:40:04.133 INFO:tasks.workunit.client.0.smithi104.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-08-19T12:40:06.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-08-19T12:40:06.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [----------] Global test environment set-up. 2024-08-19T12:40:06.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-08-19T12:40:06.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-08-19T12:40:06.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5073 ms) 2024-08-19T12:40:06.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-08-19T12:40:06.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (39 ms) 2024-08-19T12:40:06.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-08-19T12:40:06.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20037 ms) 2024-08-19T12:40:06.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-08-19T12:40:06.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: attempt 0 of 20 2024-08-19T12:40:06.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5510 ms) 2024-08-19T12:40:06.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30659 ms total) 2024-08-19T12:40:06.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: 2024-08-19T12:40:06.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [----------] Global test environment tear-down 2024-08-19T12:40:06.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30659 ms total) 2024-08-19T12:40:06.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-08-19T12:40:09.107 INFO:tasks.workunit.client.0.smithi104.stdout: misc: Running main() from gmock_main.cc 2024-08-19T12:40:09.107 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-08-19T12:40:09.107 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [----------] Global test environment set-up. 2024-08-19T12:40:09.107 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-08-19T12:40:09.107 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-08-19T12:40:09.107 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ OK ] NeoRadosMisc.Version (2120 ms) 2024-08-19T12:40:09.107 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-08-19T12:40:09.107 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2059 ms) 2024-08-19T12:40:09.107 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-08-19T12:40:09.107 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ OK ] NeoRadosMisc.LongName (3084 ms) 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3024 ms) 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3653 ms) 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (2146 ms) 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ OK ] NeoRadosMisc.Exec (3031 ms) 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3581 ms) 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3098 ms) 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-08-19T12:40:09.108 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ OK ] NeoRadosMisc.BigObject (2335 ms) 2024-08-19T12:40:09.109 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-08-19T12:40:09.109 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2783 ms) 2024-08-19T12:40:09.109 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-08-19T12:40:09.109 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (2224 ms) 2024-08-19T12:40:09.109 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [----------] 12 tests from NeoRadosMisc (33138 ms total) 2024-08-19T12:40:09.109 INFO:tasks.workunit.client.0.smithi104.stdout: misc: 2024-08-19T12:40:09.109 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [----------] Global test environment tear-down 2024-08-19T12:40:09.109 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [==========] 12 tests from 1 test suite ran. (33139 ms total) 2024-08-19T12:40:09.109 INFO:tasks.workunit.client.0.smithi104.stdout: misc: [ PASSED ] 12 tests. 2024-08-19T12:40:14.061 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: Running main() from gmock_main.cc 2024-08-19T12:40:14.061 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-08-19T12:40:14.061 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [----------] Global test environment set-up. 2024-08-19T12:40:14.061 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-08-19T12:40:14.061 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-08-19T12:40:14.061 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ OK ] LibRadosPools.PoolList (3553 ms) 2024-08-19T12:40:14.061 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-08-19T12:40:14.062 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3043 ms) 2024-08-19T12:40:14.062 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-08-19T12:40:14.062 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3013 ms) 2024-08-19T12:40:14.062 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-08-19T12:40:14.062 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2802 ms) 2024-08-19T12:40:14.062 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-08-19T12:40:14.062 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3050 ms) 2024-08-19T12:40:14.062 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-08-19T12:40:14.062 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (4973 ms) 2024-08-19T12:40:14.062 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-08-19T12:40:14.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5709 ms) 2024-08-19T12:40:14.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-08-19T12:40:14.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4342 ms) 2024-08-19T12:40:14.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-08-19T12:40:14.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7969 ms) 2024-08-19T12:40:14.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [----------] 9 tests from LibRadosPools (38454 ms total) 2024-08-19T12:40:14.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: 2024-08-19T12:40:14.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [----------] Global test environment tear-down 2024-08-19T12:40:14.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (38454 ms total) 2024-08-19T12:40:14.063 INFO:tasks.workunit.client.0.smithi104.stdout: api_pool: [ PASSED ] 9 tests. 2024-08-19T12:40:15.128 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-08-19T12:40:15.128 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [----------] Global test environment set-up. 2024-08-19T12:40:15.128 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (1 ms total) 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: 2024-08-19T12:39:35.678+0000 7f3280a67780 -1 failed for service _ceph-mon._tcp 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: 2024-08-19T12:39:35.678+0000 7f3280a67780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (59 ms) 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5011 ms) 2024-08-19T12:40:15.129 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5070 ms total) 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: started 0x55c7f62c9920 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: started 0x55c7f60a1120 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: started 2 aios 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: waiting 0x55c7f62c9920 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: waiting 0x55c7f60a1120 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: done. 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5549 ms) 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5549 ms total) 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-08-19T12:40:15.130 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ OK ] LibRadosMisc.Exec (76 ms) 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (29 ms) 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (7 ms) 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4865 ms) 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-08-19T12:40:15.131 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-08-19T12:40:15.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (21852 ms) 2024-08-19T12:40:15.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [----------] 8 tests from LibRadosMisc (26830 ms total) 2024-08-19T12:40:15.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: 2024-08-19T12:40:15.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [----------] Global test environment tear-down 2024-08-19T12:40:15.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (39490 ms total) 2024-08-19T12:40:15.132 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc: [ PASSED ] 12 tests. 2024-08-19T12:40:18.076 INFO:tasks.workunit.client.0.smithi104.stdout: io: Running main() from gmock_main.cc 2024-08-19T12:40:18.077 INFO:tasks.workunit.client.0.smithi104.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-08-19T12:40:18.077 INFO:tasks.workunit.client.0.smithi104.stdout: io: [----------] Global test environment set-up. 2024-08-19T12:40:18.077 INFO:tasks.workunit.client.0.smithi104.stdout: io: [----------] 14 tests from NeoRadosIo 2024-08-19T12:40:18.077 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-08-19T12:40:18.077 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.Limits (3228 ms) 2024-08-19T12:40:18.077 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-08-19T12:40:18.077 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3072 ms) 2024-08-19T12:40:18.077 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-08-19T12:40:18.077 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.ReadOp (3015 ms) 2024-08-19T12:40:18.077 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-08-19T12:40:18.077 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.SparseRead (2795 ms) 2024-08-19T12:40:18.078 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-08-19T12:40:18.078 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3055 ms) 2024-08-19T12:40:18.078 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-08-19T12:40:18.078 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3002 ms) 2024-08-19T12:40:18.078 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-08-19T12:40:18.078 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3030 ms) 2024-08-19T12:40:18.078 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-08-19T12:40:18.078 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3689 ms) 2024-08-19T12:40:18.078 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-08-19T12:40:18.078 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2270 ms) 2024-08-19T12:40:18.079 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-08-19T12:40:18.079 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.Trunc (3030 ms) 2024-08-19T12:40:18.079 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-08-19T12:40:18.079 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.Remove (2981 ms) 2024-08-19T12:40:18.079 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-08-19T12:40:18.079 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2946 ms) 2024-08-19T12:40:18.079 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-08-19T12:40:18.079 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.RmXattr (3104 ms) 2024-08-19T12:40:18.079 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-08-19T12:40:18.079 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2969 ms) 2024-08-19T12:40:18.079 INFO:tasks.workunit.client.0.smithi104.stdout: io: [----------] 14 tests from NeoRadosIo (42186 ms total) 2024-08-19T12:40:18.080 INFO:tasks.workunit.client.0.smithi104.stdout: io: 2024-08-19T12:40:18.080 INFO:tasks.workunit.client.0.smithi104.stdout: io: [----------] Global test environment tear-down 2024-08-19T12:40:18.080 INFO:tasks.workunit.client.0.smithi104.stdout: io: [==========] 14 tests from 1 test suite ran. (42186 ms total) 2024-08-19T12:40:18.080 INFO:tasks.workunit.client.0.smithi104.stdout: io: [ PASSED ] 14 tests. 2024-08-19T12:40:21.104 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: Running main() from gmock_main.cc 2024-08-19T12:40:21.104 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-08-19T12:40:21.104 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [----------] Global test environment set-up. 2024-08-19T12:40:21.105 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-08-19T12:40:21.105 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-08-19T12:40:21.105 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (3181 ms) 2024-08-19T12:40:21.105 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-08-19T12:40:21.105 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3070 ms) 2024-08-19T12:40:21.105 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-08-19T12:40:21.105 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3013 ms) 2024-08-19T12:40:21.105 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-08-19T12:40:21.105 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (2793 ms) 2024-08-19T12:40:21.105 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-08-19T12:40:21.105 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3049 ms) 2024-08-19T12:40:21.105 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-08-19T12:40:21.106 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (2958 ms) 2024-08-19T12:40:21.106 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-08-19T12:40:21.106 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3082 ms) 2024-08-19T12:40:21.106 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-08-19T12:40:21.106 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3695 ms) 2024-08-19T12:40:21.106 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-08-19T12:40:21.106 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (5329 ms) 2024-08-19T12:40:21.106 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-08-19T12:40:21.106 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (2996 ms) 2024-08-19T12:40:21.106 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-08-19T12:40:21.106 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (2915 ms) 2024-08-19T12:40:21.107 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-08-19T12:40:21.107 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3084 ms) 2024-08-19T12:40:21.107 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-08-19T12:40:21.107 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2915 ms) 2024-08-19T12:40:21.107 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-08-19T12:40:21.107 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3073 ms) 2024-08-19T12:40:21.107 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (45153 ms total) 2024-08-19T12:40:21.107 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: 2024-08-19T12:40:21.107 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [----------] Global test environment tear-down 2024-08-19T12:40:21.107 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (45153 ms total) 2024-08-19T12:40:21.107 INFO:tasks.workunit.client.0.smithi104.stdout: read_operations: [ PASSED ] 14 tests. 2024-08-19T12:40:29.551 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-08-19T12:40:29.551 INFO:tasks.ceph.mon.i:Restarting daemon 2024-08-19T12:40:29.551 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-08-19T12:40:29.555 INFO:tasks.ceph.mon.i:Started 2024-08-19T12:40:29.555 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:40:29.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:40:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.665+0000 7f1a1db41640 10 monclient: get_monmap_and_config 2024-08-19T12:40:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.665+0000 7f1a1db41640 10 monclient: build_initial_monmap 2024-08-19T12:40:29.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.665+0000 7f1a1db41640 10 monclient: monmap: 2024-08-19T12:40:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:29.667578+0000 2024-08-19T12:40:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:29.667578+0000 2024-08-19T12:40:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:29.668 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:29.669 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:29.669 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.666+0000 7f1a1db41640 1 Processor -- start 2024-08-19T12:40:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.666+0000 7f1a1db41640 1 -- start start 2024-08-19T12:40:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.666+0000 7f1a1db41640 10 monclient: init 2024-08-19T12:40:29.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.667+0000 7f1a1db41640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.667+0000 7f1a1db41640 10 monclient: _add_conns ranks=[5,0,7,1,8,6,2,4,3] 2024-08-19T12:40:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.667+0000 7f1a1db41640 10 monclient(hunting): picked mon.noname-h con 0x7f1a1806e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.667+0000 7f1a1db41640 10 monclient(hunting): picked mon.noname-a con 0x7f1a180a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.667+0000 7f1a1db41640 10 monclient(hunting): picked mon.noname-i con 0x7f1a180a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.667+0000 7f1a1db41640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a18136790 con 0x7f1a180a5920 2024-08-19T12:40:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.667+0000 7f1a1db41640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a1809ddf0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.667+0000 7f1a1db41640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a1809e0d0 con 0x7f1a180a5d50 2024-08-19T12:40:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.667+0000 7f1a1db41640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.667+0000 7f1a1db41640 10 monclient(hunting): authenticate will time out at 823.755130s 2024-08-19T12:40:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.667+0000 7f1a16ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1a180a5920 0x7f1a1813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59786/0 (socket says 172.21.15.26:59786) 2024-08-19T12:40:29.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.667+0000 7f1a16ffd640 1 -- 172.21.15.26:0/3866877690 learned_addr learned my addr 172.21.15.26:0/3866877690 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:29.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a16ffd640 1 -- 172.21.15.26:0/3866877690 >> v1:172.21.15.26:6789/0 conn(0x7f1a180a5920 legacy=0x7f1a1813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:29.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a17fff640 1 -- 172.21.15.26:0/3866877690 >> v1:172.21.15.104:6792/0 conn(0x7f1a180a5d50 legacy=0x7f1a18141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:29.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a177fe640 1 -- 172.21.15.26:0/3866877690 >> v1:172.21.15.104:6791/0 conn(0x7f1a1806e7b0 legacy=0x7f1a18136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:29.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 1 -- 172.21.15.26:0/3866877690 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1833411911 0 0) 0x7f1a18136790 con 0x7f1a180a5920 2024-08-19T12:40:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient(hunting): my global_id is 26829 2024-08-19T12:40:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 1 -- 172.21.15.26:0/3866877690 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a00003520 con 0x7f1a180a5920 2024-08-19T12:40:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 1 -- 172.21.15.26:0/3866877690 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 401229862 0 0) 0x7f1a080014a0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient(hunting): my global_id is 25340 2024-08-19T12:40:29.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 1 -- 172.21.15.26:0/3866877690 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a18136790 con 0x7f1a1806e7b0 2024-08-19T12:40:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 1 -- 172.21.15.26:0/3866877690 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3192685438 0 0) 0x7f1a00003520 con 0x7f1a180a5920 2024-08-19T12:40:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 1 -- 172.21.15.26:0/3866877690 >> v1:172.21.15.104:6792/0 conn(0x7f1a180a5d50 legacy=0x7f1a18141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 1 -- 172.21.15.26:0/3866877690 >> v1:172.21.15.104:6791/0 conn(0x7f1a1806e7b0 legacy=0x7f1a18136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient: _finish_hunting 0 2024-08-19T12:40:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 1 monclient: found mon.noname-a 2024-08-19T12:40:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:40:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 1 -- 172.21.15.26:0/3866877690 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1809e3b0 con 0x7f1a180a5920 2024-08-19T12:40:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient: _finish_auth 0 2024-08-19T12:40:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient: _check_auth_tickets 2024-08-19T12:40:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:29.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 1 -- 172.21.15.26:0/3866877690 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1a0c001ae0 con 0x7f1a180a5920 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a1db41640 5 monclient: authenticate success, global_id 26829 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a1db41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a167fc640 10 monclient: dump: 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:29.678 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.669+0000 7f1a1db41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a167fc640 1 -- 172.21.15.26:0/3866877690 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a0c0048a0 con 0x7f1a180a5920 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a167fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a167fc640 1 -- 172.21.15.26:0/3866877690 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1a0c004d90 con 0x7f1a180a5920 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:29.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a167fc640 10 monclient: dump: 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:29.680 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:29.681 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a1db41640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a1db41640 4 set_mon_vals no callback set 2024-08-19T12:40:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a1db41640 10 monclient: shutdown 2024-08-19T12:40:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a1db41640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a1db41640 1 -- 172.21.15.26:0/3866877690 >> v1:172.21.15.26:6789/0 conn(0x7f1a180a5920 legacy=0x7f1a1813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a1db41640 1 -- 172.21.15.26:0/3866877690 shutdown_connections 2024-08-19T12:40:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a1db41640 1 -- 172.21.15.26:0/3866877690 >> 172.21.15.26:0/3866877690 conn(0x7f1a180a8740 msgr2=0x7f1a180a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:29.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a1db41640 1 -- 172.21.15.26:0/3866877690 shutdown_connections 2024-08-19T12:40:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.670+0000 7f1a1db41640 1 -- 172.21.15.26:0/3866877690 wait complete. 2024-08-19T12:40:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.671+0000 7f1a1db41640 10 monclient: build_initial_monmap 2024-08-19T12:40:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.671+0000 7f1a1db41640 10 monclient: monmap: 2024-08-19T12:40:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:29.682 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.671+0000 7f1a1db41640 1 librados: starting msgr at 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.671+0000 7f1a1db41640 1 librados: starting objecter 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.671+0000 7f1a1db41640 1 Processor -- start 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.671+0000 7f1a1db41640 1 -- start start 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.671+0000 7f1a1db41640 1 librados: setting wanted keys 2024-08-19T12:40:29.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.671+0000 7f1a1db41640 1 librados: calling monclient init 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.671+0000 7f1a1db41640 10 monclient: init 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.672+0000 7f1a1db41640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.672+0000 7f1a1db41640 10 monclient: _add_conns ranks=[8,7,6,4,5,0,2,3,1] 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.672+0000 7f1a1db41640 10 monclient(hunting): picked mon.noname-e con 0x7f1a1806e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.672+0000 7f1a1db41640 10 monclient(hunting): picked mon.noname-i con 0x7f1a180a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.672+0000 7f1a1db41640 10 monclient(hunting): picked mon.noname-d con 0x7f1a180a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.672+0000 7f1a1db41640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a18138ee0 con 0x7f1a180a5d50 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.672+0000 7f1a1db41640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a181429b0 con 0x7f1a180a5920 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.672+0000 7f1a1db41640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a18138210 con 0x7f1a1806e7b0 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.672+0000 7f1a1db41640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.672+0000 7f1a1db41640 10 monclient(hunting): authenticate will time out at 823.760289s 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.672+0000 7f1a177fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f1a1806e7b0 0x7f1a181373f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50170/0 (socket says 172.21.15.26:50170) 2024-08-19T12:40:29.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.672+0000 7f1a177fe640 1 -- 172.21.15.26:0/564275609 learned_addr learned my addr 172.21.15.26:0/564275609 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f1a177fe640 1 -- 172.21.15.26:0/564275609 >> v1:172.21.15.26:6793/0 conn(0x7f1a1806e7b0 legacy=0x7f1a181373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 client.?.objecter ms_handle_connect 0x7f1a1806e7b0 2024-08-19T12:40:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 client.?.objecter ms_handle_connect 0x7f1a180a5d50 2024-08-19T12:40:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f1a17fff640 1 -- 172.21.15.26:0/564275609 >> v1:172.21.15.26:6792/0 conn(0x7f1a180a5d50 legacy=0x7f1a181d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 client.?.objecter ms_handle_connect 0x7f1a180a5920 2024-08-19T12:40:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f1a16ffd640 1 -- 172.21.15.26:0/564275609 >> v1:172.21.15.104:6792/0 conn(0x7f1a180a5920 legacy=0x7f1a18137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056977069 0 0) 0x7f1a18138210 con 0x7f1a1806e7b0 2024-08-19T12:40:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 monclient(hunting): my global_id is 25154 2024-08-19T12:40:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a04004eb0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2261846624 0 0) 0x7f1a18138ee0 con 0x7f1a180a5d50 2024-08-19T12:40:29.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 monclient(hunting): my global_id is 25539 2024-08-19T12:40:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a18138210 con 0x7f1a180a5d50 2024-08-19T12:40:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 543133260 0 0) 0x7f1a04004eb0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a18138ee0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1a080014c0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:40:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.674+0000 7f19fffff640 10 monclient(hunting): dump: 2024-08-19T12:40:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:29.690 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:29.691 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 722105582 0 0) 0x7f1a18138210 con 0x7f1a180a5d50 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a04004eb0 con 0x7f1a180a5d50 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1a10002070 con 0x7f1a180a5d50 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 10 monclient(hunting): dump: 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:29.692 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 184446695 0 0) 0x7f1a18138ee0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 >> v1:172.21.15.104:6792/0 conn(0x7f1a180a5920 legacy=0x7f1a18137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 >> v1:172.21.15.26:6792/0 conn(0x7f1a180a5d50 legacy=0x7f1a181d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 10 monclient: _finish_hunting 0 2024-08-19T12:40:29.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 1 monclient: found mon.e 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1813b240 con 0x7f1a1806e7b0 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 10 monclient: _finish_auth 0 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 10 monclient: _check_auth_tickets 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f1a1db41640 5 monclient: authenticate success, global_id 25154 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f1a1db41640 10 monclient: _renew_subs 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f1a1db41640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f1a1db41640 1 -- 172.21.15.26:0/564275609 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1a1813a9a0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f1a1db41640 10 client.25154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f1a1db41640 10 monclient: _renew_subs 2024-08-19T12:40:29.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f1a1db41640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f1a1db41640 1 -- 172.21.15.26:0/564275609 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f1a1813b660 con 0x7f1a1806e7b0 2024-08-19T12:40:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f1a1db41640 1 librados: init done 2024-08-19T12:40:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a08003bb0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1a08004120 con 0x7f1a1806e7b0 2024-08-19T12:40:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f1a15ffb640 4 set_mon_vals no callback set 2024-08-19T12:40:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.675+0000 7f19fffff640 10 monclient: dump: 2024-08-19T12:40:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:29.695 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.676+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f1a080042c0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.676+0000 7f1a1db41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.676+0000 7f1a1db41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.676+0000 7f1a1db41640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.676+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(73..73 src has 1..73) ==== 7565+0+0 (unknown 3519168558 0 0) 0x7f1a0804ebb0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.676+0000 7f19fffff640 10 client.25154.objecter ms_dispatch 0x7f1a18000b90 osd_map(73..73 src has 1..73) 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.676+0000 7f19fffff640 3 client.25154.objecter handle_osd_map got epochs [73,73] > 0 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.676+0000 7f19fffff640 3 client.25154.objecter handle_osd_map decoding full epoch 73 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.676+0000 7f1a1db41640 1 -- 172.21.15.26:0/564275609 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19f8005080 con 0x7f1a1806e7b0 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.676+0000 7f19fffff640 20 client.25154.objecter dump_active .. 0 homeless 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.676+0000 7f1a16ffd640 10 monclient: get_auth_request con 0x7f1a04041580 auth_method 0 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.678+0000 7f19fffff640 10 client.25154.objecter ms_handle_connect 0x7f1a04041580 2024-08-19T12:40:29.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.681+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f1a1809ddf0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.681+0000 7f19fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:29.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.681+0000 7f19fffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.943+0000 7f1a1db41640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:40:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.943+0000 7f1a1db41640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:40:29.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.943+0000 7f1a1db41640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:29.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.943+0000 7f1a1db41640 1 -- 172.21.15.26:0/564275609 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f19f8005370 con 0x7f1a1806e7b0 2024-08-19T12:40:29.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.944+0000 7f19fffff640 1 -- 172.21.15.26:0/564275609 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 4252902809) 0x7f1a080176d0 con 0x7f1a1806e7b0 2024-08-19T12:40:29.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.944+0000 7f19fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:40:29.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.944+0000 7f19fffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:29.946 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":10,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:40:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.945+0000 7f1a1db41640 10 librados: watch_flush enter 2024-08-19T12:40:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.945+0000 7f1a1db41640 10 librados: watch_flush exit 2024-08-19T12:40:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.945+0000 7f1a1db41640 20 client.25154.objecter shutdown clearing up homeless session... 2024-08-19T12:40:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.945+0000 7f1a1db41640 10 client.25154.objecter successfully canceled tick 2024-08-19T12:40:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.945+0000 7f1a1db41640 1 -- 172.21.15.26:0/564275609 >> v1:172.21.15.104:6800/899887506 conn(0x7f1a04041580 legacy=0x7f1a04043a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.946+0000 7f1a1db41640 10 monclient: shutdown 2024-08-19T12:40:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.946+0000 7f1a1db41640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.946+0000 7f1a1db41640 1 -- 172.21.15.26:0/564275609 >> v1:172.21.15.26:6793/0 conn(0x7f1a1806e7b0 legacy=0x7f1a181373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.946+0000 7f1a1db41640 1 -- 172.21.15.26:0/564275609 shutdown_connections 2024-08-19T12:40:29.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.946+0000 7f1a1db41640 1 -- 172.21.15.26:0/564275609 >> 172.21.15.26:0/564275609 conn(0x7f1a180a8740 msgr2=0x7f1a181350d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.946+0000 7f1a1db41640 1 -- 172.21.15.26:0/564275609 shutdown_connections 2024-08-19T12:40:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.946+0000 7f1a1db41640 1 -- 172.21.15.26:0/564275609 wait complete. 2024-08-19T12:40:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:29.946+0000 7f1a1db41640 1 librados: shutdown 2024-08-19T12:40:29.958 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:40:29.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:40:29.959 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:40:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.071+0000 7f22ef02f640 10 monclient: get_monmap_and_config 2024-08-19T12:40:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.071+0000 7f22ef02f640 10 monclient: build_initial_monmap 2024-08-19T12:40:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.071+0000 7f22ef02f640 10 monclient: monmap: 2024-08-19T12:40:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:30.073326+0000 2024-08-19T12:40:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:30.073326+0000 2024-08-19T12:40:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:30.075 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.072+0000 7f22ef02f640 1 Processor -- start 2024-08-19T12:40:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.072+0000 7f22ef02f640 1 -- start start 2024-08-19T12:40:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.072+0000 7f22ef02f640 10 monclient: init 2024-08-19T12:40:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.072+0000 7f22ef02f640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.072+0000 7f22ef02f640 10 monclient: _add_conns ranks=[1,4,8,6,2,0,5,7,3] 2024-08-19T12:40:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.073+0000 7f22ef02f640 10 monclient(hunting): picked mon.noname-f con 0x7f22e809e9e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.073+0000 7f22ef02f640 10 monclient(hunting): picked mon.noname-c con 0x7f22e81371d0 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.073+0000 7f22ef02f640 10 monclient(hunting): picked mon.noname-e con 0x7f22e813d700 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.073+0000 7f22ef02f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e80a1e80 con 0x7f22e809e9e0 2024-08-19T12:40:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.073+0000 7f22ef02f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e81375d0 con 0x7f22e81371d0 2024-08-19T12:40:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.073+0000 7f22ef02f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e80a1c40 con 0x7f22e813d700 2024-08-19T12:40:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.073+0000 7f22ef02f640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.073+0000 7f22ef02f640 10 monclient(hunting): authenticate will time out at 824.161142s 2024-08-19T12:40:30.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.073+0000 7f22ee82e640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f22e813d700 0x7f22e813faf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50176/0 (socket says 172.21.15.26:50176) 2024-08-19T12:40:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.073+0000 7f22ee82e640 1 -- 172.21.15.26:0/2428104953 learned_addr learned my addr 172.21.15.26:0/2428104953 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.075+0000 7f22ee82e640 1 -- 172.21.15.26:0/2428104953 >> v1:172.21.15.26:6793/0 conn(0x7f22e813d700 legacy=0x7f22e813faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.075+0000 7f22ee02d640 1 -- 172.21.15.26:0/2428104953 >> v1:172.21.15.104:6789/0 conn(0x7f22e809e9e0 legacy=0x7f22e809edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.075+0000 7f22ed82c640 1 -- 172.21.15.26:0/2428104953 >> v1:172.21.15.26:6791/0 conn(0x7f22e81371d0 legacy=0x7f22e813bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4100573193 0 0) 0x7f22e80a1e80 con 0x7f22e809e9e0 2024-08-19T12:40:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient(hunting): my global_id is 25921 2024-08-19T12:40:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22d4003520 con 0x7f22e809e9e0 2024-08-19T12:40:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3749334208 0 0) 0x7f22e80a1c40 con 0x7f22e813d700 2024-08-19T12:40:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient(hunting): my global_id is 25163 2024-08-19T12:40:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22d40048e0 con 0x7f22e813d700 2024-08-19T12:40:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1360158626 0 0) 0x7f22e81375d0 con 0x7f22e81371d0 2024-08-19T12:40:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient(hunting): my global_id is 25546 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22e80a1c40 con 0x7f22e81371d0 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 658329149 0 0) 0x7f22d4003520 con 0x7f22e809e9e0 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 >> v1:172.21.15.26:6793/0 conn(0x7f22e813d700 legacy=0x7f22e813faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 >> v1:172.21.15.26:6791/0 conn(0x7f22e81371d0 legacy=0x7f22e813bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 10 monclient: _finish_hunting 0 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 1 monclient: found mon.noname-f 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.077+0000 7f22ed02b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e8068dd0 con 0x7f22e809e9e0 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 10 monclient: _finish_auth 0 2024-08-19T12:40:30.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 10 monclient: _check_auth_tickets 2024-08-19T12:40:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ef02f640 5 monclient: authenticate success, global_id 25921 2024-08-19T12:40:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ef02f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f22e4001b10 con 0x7f22e809e9e0 2024-08-19T12:40:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:40:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 10 monclient: dump: 2024-08-19T12:40:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.084 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ef02f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:30.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f22e4003610 con 0x7f22e809e9e0 2024-08-19T12:40:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 1 -- 172.21.15.26:0/2428104953 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f22e4003b00 con 0x7f22e809e9e0 2024-08-19T12:40:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ed02b640 10 monclient: dump: 2024-08-19T12:40:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.086 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ef02f640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.078+0000 7f22ef02f640 4 set_mon_vals no callback set 2024-08-19T12:40:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.079+0000 7f22ef02f640 10 monclient: shutdown 2024-08-19T12:40:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.079+0000 7f22ef02f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.079+0000 7f22ef02f640 1 -- 172.21.15.26:0/2428104953 >> v1:172.21.15.104:6789/0 conn(0x7f22e809e9e0 legacy=0x7f22e809edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.079+0000 7f22ef02f640 1 -- 172.21.15.26:0/2428104953 shutdown_connections 2024-08-19T12:40:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.079+0000 7f22ef02f640 1 -- 172.21.15.26:0/2428104953 >> 172.21.15.26:0/2428104953 conn(0x7f22e80a5030 msgr2=0x7f22e80a5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.079+0000 7f22ef02f640 1 -- 172.21.15.26:0/2428104953 shutdown_connections 2024-08-19T12:40:30.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.079+0000 7f22ef02f640 1 -- 172.21.15.26:0/2428104953 wait complete. 2024-08-19T12:40:30.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.080+0000 7f22ef02f640 10 monclient: build_initial_monmap 2024-08-19T12:40:30.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.080+0000 7f22ef02f640 10 monclient: monmap: 2024-08-19T12:40:30.096 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:30.096 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:30.096 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:30.096 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:30.096 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:30.096 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.096 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:30.097 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:30.097 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:30.097 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:30.097 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:30.097 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:30.097 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:30.097 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:30.097 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:30.097 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.080+0000 7f22ef02f640 1 librados: starting msgr at 2024-08-19T12:40:30.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.080+0000 7f22ef02f640 1 librados: starting objecter 2024-08-19T12:40:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.080+0000 7f22ef02f640 1 Processor -- start 2024-08-19T12:40:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.080+0000 7f22ef02f640 1 -- start start 2024-08-19T12:40:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.080+0000 7f22ef02f640 1 librados: setting wanted keys 2024-08-19T12:40:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.080+0000 7f22ef02f640 1 librados: calling monclient init 2024-08-19T12:40:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.080+0000 7f22ef02f640 10 monclient: init 2024-08-19T12:40:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.081+0000 7f22ef02f640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.081+0000 7f22ef02f640 10 monclient: _add_conns ranks=[4,1,6,7,3,5,8,2,0] 2024-08-19T12:40:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.081+0000 7f22ef02f640 10 monclient(hunting): picked mon.noname-c con 0x7f22e809e9e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.081+0000 7f22ef02f640 10 monclient(hunting): picked mon.noname-f con 0x7f22e81371d0 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:30.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.081+0000 7f22ef02f640 10 monclient(hunting): picked mon.noname-d con 0x7f22e813d700 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.081+0000 7f22ef02f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e80a7440 con 0x7f22e81371d0 2024-08-19T12:40:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.081+0000 7f22ef02f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e80a18d0 con 0x7f22e809e9e0 2024-08-19T12:40:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.081+0000 7f22ee02d640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f22e809e9e0 0x7f22e81ce650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:39078/0 (socket says 172.21.15.26:39078) 2024-08-19T12:40:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.081+0000 7f22ee02d640 1 -- 172.21.15.26:0/526640000 learned_addr learned my addr 172.21.15.26:0/526640000 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.081+0000 7f22ef02f640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e80a1030 con 0x7f22e813d700 2024-08-19T12:40:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.081+0000 7f22ef02f640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.081+0000 7f22ef02f640 10 monclient(hunting): authenticate will time out at 824.169634s 2024-08-19T12:40:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22ee02d640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.26:6791/0 conn(0x7f22e809e9e0 legacy=0x7f22e81ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22ee82e640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.26:6792/0 conn(0x7f22e813d700 legacy=0x7f22e81d6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 client.?.objecter ms_handle_connect 0x7f22e809e9e0 2024-08-19T12:40:30.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 client.?.objecter ms_handle_connect 0x7f22e813d700 2024-08-19T12:40:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 client.?.objecter ms_handle_connect 0x7f22e81371d0 2024-08-19T12:40:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22ed82c640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.104:6789/0 conn(0x7f22e81371d0 legacy=0x7f22e81d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1902892835 0 0) 0x7f22e80a1030 con 0x7f22e813d700 2024-08-19T12:40:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 monclient(hunting): my global_id is 25548 2024-08-19T12:40:30.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22cc004eb0 con 0x7f22e813d700 2024-08-19T12:40:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1112297663 0 0) 0x7f22e80a18d0 con 0x7f22e809e9e0 2024-08-19T12:40:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 monclient(hunting): my global_id is 25555 2024-08-19T12:40:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22e80a1030 con 0x7f22e809e9e0 2024-08-19T12:40:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3668766307 0 0) 0x7f22e80a7440 con 0x7f22e81371d0 2024-08-19T12:40:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 10 monclient(hunting): my global_id is 25930 2024-08-19T12:40:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.083+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22e80a18d0 con 0x7f22e81371d0 2024-08-19T12:40:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3506162820 0 0) 0x7f22cc004eb0 con 0x7f22e813d700 2024-08-19T12:40:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22e80a7440 con 0x7f22e813d700 2024-08-19T12:40:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2359051329 0 0) 0x7f22e80a18d0 con 0x7f22e81371d0 2024-08-19T12:40:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22cc004eb0 con 0x7f22e81371d0 2024-08-19T12:40:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f22d8003610 con 0x7f22e81371d0 2024-08-19T12:40:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:40:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient(hunting): dump: 2024-08-19T12:40:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.103 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.103 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.103 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.103 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.103 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.103 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.103 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.103 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.103 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.103 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.103 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.103 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f22e0002060 con 0x7f22e813d700 2024-08-19T12:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient(hunting): dump: 2024-08-19T12:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.104 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 324741444 0 0) 0x7f22e80a1030 con 0x7f22e809e9e0 2024-08-19T12:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22e80a18d0 con 0x7f22e809e9e0 2024-08-19T12:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f22e4001a90 con 0x7f22e809e9e0 2024-08-19T12:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient(hunting): dump: 2024-08-19T12:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.106 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2330670047 0 0) 0x7f22e80a7440 con 0x7f22e813d700 2024-08-19T12:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.26:6791/0 conn(0x7f22e809e9e0 legacy=0x7f22e81ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.104:6789/0 conn(0x7f22e81371d0 legacy=0x7f22e81d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient: _finish_hunting 0 2024-08-19T12:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 monclient: found mon.d 2024-08-19T12:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e80a00d0 con 0x7f22e813d700 2024-08-19T12:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient: _finish_auth 0 2024-08-19T12:40:30.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient: _check_auth_tickets 2024-08-19T12:40:30.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:30.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22ef02f640 5 monclient: authenticate success, global_id 25548 2024-08-19T12:40:30.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22ef02f640 10 monclient: _renew_subs 2024-08-19T12:40:30.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22ef02f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:30.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22ef02f640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22e80a0360 con 0x7f22e813d700 2024-08-19T12:40:30.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22ef02f640 10 client.25548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:30.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22ef02f640 10 monclient: _renew_subs 2024-08-19T12:40:30.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22ef02f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:30.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22ef02f640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f22e81cef40 con 0x7f22e813d700 2024-08-19T12:40:30.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22ef02f640 1 librados: init done 2024-08-19T12:40:30.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f22e0004e80 con 0x7f22e813d700 2024-08-19T12:40:30.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:30.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f22e00053b0 con 0x7f22e813d700 2024-08-19T12:40:30.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:30.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22deffd640 10 monclient: dump: 2024-08-19T12:40:30.110 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.110 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.111 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.111 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.111 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.111 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.111 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.111 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.111 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.111 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.111 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.111 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.084+0000 7f22ec82a640 4 set_mon_vals no callback set 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.085+0000 7f22ef02f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.085+0000 7f22ef02f640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f22e809dfa0 con 0x7f22e813c1c0 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.085+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f22e0005650 con 0x7f22e813d700 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.085+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(74..74 src has 1..74) ==== 7453+0+0 (unknown 2159442305 0 0) 0x7f22e004ec50 con 0x7f22e813d700 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.085+0000 7f22deffd640 10 client.25548.objecter ms_dispatch 0x7f22e8000b90 osd_map(74..74 src has 1..74) 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.085+0000 7f22deffd640 3 client.25548.objecter handle_osd_map got epochs [74,74] > 0 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.085+0000 7f22deffd640 3 client.25548.objecter handle_osd_map decoding full epoch 74 2024-08-19T12:40:30.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.085+0000 7f22deffd640 20 client.25548.objecter dump_active .. 0 homeless 2024-08-19T12:40:30.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.085+0000 7f22ed82c640 10 monclient: get_auth_request con 0x7f22cc041870 auth_method 0 2024-08-19T12:40:30.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22ee02d640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.26:6789/0 conn(0x7f22e813c1c0 legacy=0x7f22e813eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 10 client.25548.objecter ms_handle_connect 0x7f22e813c1c0 2024-08-19T12:40:30.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 10 client.25548.objecter resend_mon_ops 2024-08-19T12:40:30.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2106309994 0 0) 0x7f22e80a18d0 con 0x7f22e813c1c0 2024-08-19T12:40:30.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 10 monclient(hunting): my global_id is 26838 2024-08-19T12:40:30.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22e80a7440 con 0x7f22e813c1c0 2024-08-19T12:40:30.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3429739643 0 0) 0x7f22e80a7440 con 0x7f22e813c1c0 2024-08-19T12:40:30.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22e80a18d0 con 0x7f22e813c1c0 2024-08-19T12:40:30.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f22e4004100 con 0x7f22e813c1c0 2024-08-19T12:40:30.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:30.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 10 monclient: dump: 2024-08-19T12:40:30.114 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.114 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.114 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.114 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.114 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.114 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.115 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.115 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.115 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.115 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.115 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.115 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.115 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.115 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.115 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.115 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1450805242 0 0) 0x7f22e80a18d0 con 0x7f22e813c1c0 2024-08-19T12:40:30.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.087+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22e809f4a0 con 0x7f22e813c1c0 2024-08-19T12:40:30.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.088+0000 7f22deffd640 10 client.25548.objecter ms_handle_connect 0x7f22cc041870 2024-08-19T12:40:30.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.088+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f22e4004100 con 0x7f22e813c1c0 2024-08-19T12:40:30.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.088+0000 7f22deffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:30.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.088+0000 7f22deffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:30.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.088+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.26:6789/0 conn(0x7f22e813c1c0 legacy=0x7f22e813eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.088+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.26:6789/0 conn(0x7f22e813c1c0 legacy=0x7f22e813eaf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:30.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.094+0000 7f22ef02f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:30.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.094+0000 7f22ef02f640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f22e813c5a0 con 0x7f22e802e1d0 2024-08-19T12:40:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.096+0000 7f22ee02d640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.26:6789/0 conn(0x7f22e802e1d0 legacy=0x7f22e813f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.096+0000 7f22deffd640 10 client.25548.objecter ms_handle_connect 0x7f22e802e1d0 2024-08-19T12:40:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.096+0000 7f22deffd640 10 client.25548.objecter resend_mon_ops 2024-08-19T12:40:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.096+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3638198877 0 0) 0x7f22e813c5a0 con 0x7f22e802e1d0 2024-08-19T12:40:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.096+0000 7f22deffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.096+0000 7f22deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.096+0000 7f22deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.096+0000 7f22deffd640 10 monclient(hunting): my global_id is 26847 2024-08-19T12:40:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.096+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22e80a18d0 con 0x7f22e802e1d0 2024-08-19T12:40:30.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3554110837 0 0) 0x7f22e80a18d0 con 0x7f22e802e1d0 2024-08-19T12:40:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22e813c5a0 con 0x7f22e802e1d0 2024-08-19T12:40:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f22e4005b00 con 0x7f22e802e1d0 2024-08-19T12:40:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 10 monclient: dump: 2024-08-19T12:40:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.119 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3998198971 0 0) 0x7f22e813c5a0 con 0x7f22e802e1d0 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f22e809e550 con 0x7f22e802e1d0 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7546 (unknown 0 0 4122245595) 0x7f22e4007d90 con 0x7f22e802e1d0 2024-08-19T12:40:30.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:30.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:30.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.26:6789/0 conn(0x7f22e802e1d0 legacy=0x7f22e813f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22deffd640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.26:6789/0 conn(0x7f22e802e1d0 legacy=0x7f22e813f200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:30.121 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:40:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:30.121 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 143139, 2024-08-19T12:40:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.122 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:30.124 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:30.125 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:30.126 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.127 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:30.128 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.130 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:30.131 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.134 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.134 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:30.134 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.134 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.134 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.134 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:30.134 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:30.134 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.134 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.135 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.136 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:30.136 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:30.136 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.136 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.136 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.136 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:30.136 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:30.136 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.136 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.138 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:30.138 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:30.138 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.138 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.138 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:30.138 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.139 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:30.139 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:30.139 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.139 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.139 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.140 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:30.140 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:30.140 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.140 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:30.140 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.140 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.141 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:30.141 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:30.141 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.141 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.141 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.141 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:30.141 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:30.142 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.142 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.142 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.142 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:30.142 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.142 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.142 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:30.142 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:30.142 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.143 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.143 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.143 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.143 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:30.143 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:30.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.143 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:30.143 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:30.143 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:30.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.144 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:30.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.144 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:30.144 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:30.144 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:30.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.144 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.144 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:30.145 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:30.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ef02f640 10 librados: watch_flush enter 2024-08-19T12:40:30.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ef02f640 10 librados: watch_flush exit 2024-08-19T12:40:30.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ef02f640 20 client.25548.objecter shutdown clearing up homeless session... 2024-08-19T12:40:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ef02f640 10 client.25548.objecter successfully canceled tick 2024-08-19T12:40:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ef02f640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.104:6800/899887506 conn(0x7f22cc041870 legacy=0x7f22cc043d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ef02f640 10 monclient: shutdown 2024-08-19T12:40:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ef02f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ef02f640 1 -- 172.21.15.26:0/526640000 >> v1:172.21.15.26:6792/0 conn(0x7f22e813d700 legacy=0x7f22e81d6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ee82e640 1 -- 172.21.15.26:0/526640000 reap_dead start 2024-08-19T12:40:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ee82e640 1 -- 172.21.15.26:0/526640000 reap_dead start 2024-08-19T12:40:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ef02f640 1 -- 172.21.15.26:0/526640000 shutdown_connections 2024-08-19T12:40:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ef02f640 1 -- 172.21.15.26:0/526640000 >> 172.21.15.26:0/526640000 conn(0x7f22e80a5030 msgr2=0x7f22e80a6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.097+0000 7f22ef02f640 1 -- 172.21.15.26:0/526640000 shutdown_connections 2024-08-19T12:40:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.098+0000 7f22ef02f640 1 -- 172.21.15.26:0/526640000 wait complete. 2024-08-19T12:40:30.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.098+0000 7f22ef02f640 1 librados: shutdown 2024-08-19T12:40:30.149 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:30.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:40:30.150 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:40:30.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.247+0000 7f0af9b5c640 10 monclient: get_monmap_and_config 2024-08-19T12:40:30.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.247+0000 7f0af9b5c640 10 monclient: build_initial_monmap 2024-08-19T12:40:30.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.247+0000 7f0af9b5c640 10 monclient: monmap: 2024-08-19T12:40:30.250 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:30.250 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.250 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:30.249706+0000 2024-08-19T12:40:30.250 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:30.249706+0000 2024-08-19T12:40:30.250 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:30.250 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.250 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:30.250 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:30.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.248+0000 7f0af9b5c640 1 Processor -- start 2024-08-19T12:40:30.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.248+0000 7f0af9b5c640 1 -- start start 2024-08-19T12:40:30.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.248+0000 7f0af9b5c640 10 monclient: init 2024-08-19T12:40:30.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.249+0000 7f0af9b5c640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:30.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.249+0000 7f0af9b5c640 10 monclient: _add_conns ranks=[0,3,4,7,5,6,2,1,8] 2024-08-19T12:40:30.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.249+0000 7f0af9b5c640 10 monclient(hunting): picked mon.noname-a con 0x7f0af406e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:30.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.249+0000 7f0af9b5c640 10 monclient(hunting): picked mon.noname-g con 0x7f0af40a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:30.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.249+0000 7f0af9b5c640 10 monclient(hunting): picked mon.noname-c con 0x7f0af40a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:30.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.249+0000 7f0af9b5c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0af4136790 con 0x7f0af406e7b0 2024-08-19T12:40:30.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.249+0000 7f0af9b5c640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0af409ddf0 con 0x7f0af40a5920 2024-08-19T12:40:30.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.249+0000 7f0af9b5c640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0af409e0d0 con 0x7f0af40a5d50 2024-08-19T12:40:30.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.249+0000 7f0af9b5c640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:30.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.249+0000 7f0af9b5c640 10 monclient(hunting): authenticate will time out at 824.337171s 2024-08-19T12:40:30.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.249+0000 7f0af3fff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f0af40a5d50 0x7f0af4141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:39080/0 (socket says 172.21.15.26:39080) 2024-08-19T12:40:30.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.249+0000 7f0af3fff640 1 -- 172.21.15.26:0/3482847198 learned_addr learned my addr 172.21.15.26:0/3482847198 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:30.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af3fff640 1 -- 172.21.15.26:0/3482847198 >> v1:172.21.15.26:6791/0 conn(0x7f0af40a5d50 legacy=0x7f0af4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af37fe640 1 -- 172.21.15.26:0/3482847198 >> v1:172.21.15.26:6789/0 conn(0x7f0af406e7b0 legacy=0x7f0af4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af2ffd640 1 -- 172.21.15.26:0/3482847198 >> v1:172.21.15.104:6790/0 conn(0x7f0af40a5920 legacy=0x7f0af413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1776419929 0 0) 0x7f0af409e0d0 con 0x7f0af40a5d50 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient(hunting): my global_id is 25564 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0adc003520 con 0x7f0af40a5d50 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845413640 0 0) 0x7f0af4136790 con 0x7f0af406e7b0 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient(hunting): my global_id is 26856 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0af409e0d0 con 0x7f0af406e7b0 2024-08-19T12:40:30.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2267215498 0 0) 0x7f0ae80014a0 con 0x7f0af40a5920 2024-08-19T12:40:30.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient(hunting): my global_id is 25446 2024-08-19T12:40:30.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0af4136790 con 0x7f0af40a5920 2024-08-19T12:40:30.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 684542703 0 0) 0x7f0af409e0d0 con 0x7f0af406e7b0 2024-08-19T12:40:30.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 >> v1:172.21.15.26:6791/0 conn(0x7f0af40a5d50 legacy=0x7f0af4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 >> v1:172.21.15.104:6790/0 conn(0x7f0af40a5920 legacy=0x7f0af413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient: _finish_hunting 0 2024-08-19T12:40:30.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 1 monclient: found mon.noname-a 2024-08-19T12:40:30.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:30.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:40:30.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af409e3b0 con 0x7f0af406e7b0 2024-08-19T12:40:30.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient: _finish_auth 0 2024-08-19T12:40:30.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient: _check_auth_tickets 2024-08-19T12:40:30.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:30.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af9b5c640 5 monclient: authenticate success, global_id 26856 2024-08-19T12:40:30.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0ae4001a80 con 0x7f0af406e7b0 2024-08-19T12:40:30.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af9b5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:30.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:40:30.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af27fc640 10 monclient: dump: 2024-08-19T12:40:30.265 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.266 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.266 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.266 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.266 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.266 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.266 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.266 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.266 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.266 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.266 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.266 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.267 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.267 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.267 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.267 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.251+0000 7f0af9b5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:30.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ae4001c20 con 0x7f0af406e7b0 2024-08-19T12:40:30.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:30.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af27fc640 1 -- 172.21.15.26:0/3482847198 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0ae4003950 con 0x7f0af406e7b0 2024-08-19T12:40:30.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:30.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af27fc640 10 monclient: dump: 2024-08-19T12:40:30.268 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.268 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.268 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.268 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.268 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.268 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.268 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.268 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.269 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.269 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.269 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.269 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.269 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.269 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.269 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.269 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af9b5c640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:30.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af9b5c640 4 set_mon_vals no callback set 2024-08-19T12:40:30.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af9b5c640 10 monclient: shutdown 2024-08-19T12:40:30.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af9b5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:30.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3482847198 >> v1:172.21.15.26:6789/0 conn(0x7f0af406e7b0 legacy=0x7f0af4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3482847198 shutdown_connections 2024-08-19T12:40:30.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3482847198 >> 172.21.15.26:0/3482847198 conn(0x7f0af40a8740 msgr2=0x7f0af40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:30.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3482847198 shutdown_connections 2024-08-19T12:40:30.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.252+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3482847198 wait complete. 2024-08-19T12:40:30.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.253+0000 7f0af9b5c640 10 monclient: build_initial_monmap 2024-08-19T12:40:30.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.253+0000 7f0af9b5c640 10 monclient: monmap: 2024-08-19T12:40:30.270 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:30.270 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:30.271 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:30.271 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:30.271 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:30.271 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.271 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:30.271 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:30.271 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:30.271 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:30.271 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:30.271 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:30.271 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:30.272 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:30.272 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:30.272 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.253+0000 7f0af9b5c640 1 librados: starting msgr at 2024-08-19T12:40:30.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.253+0000 7f0af9b5c640 1 librados: starting objecter 2024-08-19T12:40:30.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.253+0000 7f0af9b5c640 1 Processor -- start 2024-08-19T12:40:30.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.253+0000 7f0af9b5c640 1 -- start start 2024-08-19T12:40:30.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.253+0000 7f0af9b5c640 1 librados: setting wanted keys 2024-08-19T12:40:30.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.253+0000 7f0af9b5c640 1 librados: calling monclient init 2024-08-19T12:40:30.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.253+0000 7f0af9b5c640 10 monclient: init 2024-08-19T12:40:30.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.254+0000 7f0af9b5c640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:30.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.254+0000 7f0af9b5c640 10 monclient: _add_conns ranks=[1,6,7,2,5,3,8,4,0] 2024-08-19T12:40:30.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.254+0000 7f0af9b5c640 10 monclient(hunting): picked mon.noname-f con 0x7f0af406e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:30.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.254+0000 7f0af9b5c640 10 monclient(hunting): picked mon.noname-d con 0x7f0af40a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:30.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.254+0000 7f0af9b5c640 10 monclient(hunting): picked mon.noname-i con 0x7f0af40a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:30.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.254+0000 7f0af9b5c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0af40a2c40 con 0x7f0af406e7b0 2024-08-19T12:40:30.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.254+0000 7f0af9b5c640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0af40a2f20 con 0x7f0af40a5920 2024-08-19T12:40:30.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.254+0000 7f0af9b5c640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0af40a1aa0 con 0x7f0af40a5d50 2024-08-19T12:40:30.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.254+0000 7f0af9b5c640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:30.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.254+0000 7f0af9b5c640 10 monclient(hunting): authenticate will time out at 824.342190s 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.254+0000 7f0af2ffd640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f0af40a5920 0x7f0af40a01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39042/0 (socket says 172.21.15.26:39042) 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.254+0000 7f0af2ffd640 1 -- 172.21.15.26:0/3580342196 learned_addr learned my addr 172.21.15.26:0/3580342196 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0af2ffd640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.26:6792/0 conn(0x7f0af40a5920 legacy=0x7f0af40a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 client.?.objecter ms_handle_connect 0x7f0af40a5920 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 client.?.objecter ms_handle_connect 0x7f0af40a5d50 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0af3fff640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.104:6792/0 conn(0x7f0af40a5d50 legacy=0x7f0af40a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1133908957 0 0) 0x7f0af40a2f20 con 0x7f0af40a5920 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): my global_id is 25557 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0af37fe640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.104:6789/0 conn(0x7f0af406e7b0 legacy=0x7f0af409fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ae0004eb0 con 0x7f0af40a5920 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 client.?.objecter ms_handle_connect 0x7f0af406e7b0 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1688297220 0 0) 0x7f0af40a2c40 con 0x7f0af406e7b0 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): my global_id is 25939 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0af40a2f20 con 0x7f0af406e7b0 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 731688512 0 0) 0x7f0ae0004eb0 con 0x7f0af40a5920 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0af40a2c40 con 0x7f0af40a5920 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0ae8003950 con 0x7f0af40a5920 2024-08-19T12:40:30.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): dump: 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.278 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4217813419 0 0) 0x7f0af40a2f20 con 0x7f0af406e7b0 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ae0004eb0 con 0x7f0af406e7b0 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0ae4001940 con 0x7f0af406e7b0 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.256+0000 7f0adbfff640 10 monclient(hunting): dump: 2024-08-19T12:40:30.279 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.280 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3277071427 0 0) 0x7f0af40a2c40 con 0x7f0af40a5920 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.104:6792/0 conn(0x7f0af40a5d50 legacy=0x7f0af40a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.104:6789/0 conn(0x7f0af406e7b0 legacy=0x7f0af409fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 10 monclient: _finish_hunting 0 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 1 monclient: found mon.d 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af40a26d0 con 0x7f0af40a5920 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 10 monclient: _finish_auth 0 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 10 monclient: _check_auth_tickets 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:30.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0af9b5c640 5 monclient: authenticate success, global_id 25557 2024-08-19T12:40:30.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0af9b5c640 10 monclient: _renew_subs 2024-08-19T12:40:30.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0af9b5c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:30.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0af40a2960 con 0x7f0af40a5920 2024-08-19T12:40:30.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0af9b5c640 10 client.25557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:30.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0af9b5c640 10 monclient: _renew_subs 2024-08-19T12:40:30.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0af9b5c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:30.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f0af41d7480 con 0x7f0af40a5920 2024-08-19T12:40:30.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0af9b5c640 1 librados: init done 2024-08-19T12:40:30.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ae8001ce0 con 0x7f0af40a5920 2024-08-19T12:40:30.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:30.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0ae8004160 con 0x7f0af40a5920 2024-08-19T12:40:30.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0af1ffb640 4 set_mon_vals no callback set 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.257+0000 7f0adbfff640 10 monclient: dump: 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.283 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.258+0000 7f0af9b5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.258+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0ad40065c0 con 0x7f0ad4002860 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.258+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f0ae8011a80 con 0x7f0af40a5920 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.258+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(74..74 src has 1..74) ==== 7453+0+0 (unknown 2159442305 0 0) 0x7f0ae804ebd0 con 0x7f0af40a5920 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.258+0000 7f0adbfff640 10 client.25557.objecter ms_dispatch 0x7f0af4000b90 osd_map(74..74 src has 1..74) 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.258+0000 7f0adbfff640 3 client.25557.objecter handle_osd_map got epochs [74,74] > 0 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.258+0000 7f0adbfff640 3 client.25557.objecter handle_osd_map decoding full epoch 74 2024-08-19T12:40:30.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.258+0000 7f0af3fff640 10 monclient: get_auth_request con 0x7f0ae0041580 auth_method 0 2024-08-19T12:40:30.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.258+0000 7f0adbfff640 20 client.25557.objecter dump_active .. 0 homeless 2024-08-19T12:40:30.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.260+0000 7f0af37fe640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.26:6790/0 conn(0x7f0ad4002860 legacy=0x7f0ad4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.260+0000 7f0adbfff640 10 client.25557.objecter ms_handle_connect 0x7f0ad4002860 2024-08-19T12:40:30.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.260+0000 7f0adbfff640 10 client.25557.objecter resend_mon_ops 2024-08-19T12:40:30.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.261+0000 7f0adbfff640 10 client.25557.objecter ms_handle_connect 0x7f0ae0041580 2024-08-19T12:40:30.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992074132 0 0) 0x7f0ad40065c0 con 0x7f0ad4002860 2024-08-19T12:40:30.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 10 monclient(hunting): my global_id is 25319 2024-08-19T12:40:30.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0af40a2c40 con 0x7f0ad4002860 2024-08-19T12:40:30.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1818814290 0 0) 0x7f0af40a2c40 con 0x7f0ad4002860 2024-08-19T12:40:30.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ad40065c0 con 0x7f0ad4002860 2024-08-19T12:40:30.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0ae4004000 con 0x7f0ad4002860 2024-08-19T12:40:30.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 10 monclient: dump: 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.674 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.675 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.675 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.675 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.675 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 687148697 0 0) 0x7f0ad40065c0 con 0x7f0ad4002860 2024-08-19T12:40:30.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.670+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ad400a160 con 0x7f0ad4002860 2024-08-19T12:40:30.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.671+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0ae4001c10 con 0x7f0ad4002860 2024-08-19T12:40:30.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.671+0000 7f0adbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:30.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.671+0000 7f0adbfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:30.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.671+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.26:6790/0 conn(0x7f0ad4002860 legacy=0x7f0ad4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.671+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.26:6790/0 conn(0x7f0ad4002860 legacy=0x7f0ad4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:30.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.678+0000 7f0af9b5c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:30.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.678+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0ad4009e70 con 0x7f0ad4009130 2024-08-19T12:40:30.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.680+0000 7f0adbfff640 10 client.25557.objecter ms_handle_connect 0x7f0ad4009130 2024-08-19T12:40:30.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.680+0000 7f0adbfff640 10 client.25557.objecter resend_mon_ops 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.680+0000 7f0af37fe640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.26:6790/0 conn(0x7f0ad4009130 legacy=0x7f0ad40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.680+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1374170680 0 0) 0x7f0ad4009e70 con 0x7f0ad4009130 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.680+0000 7f0adbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.680+0000 7f0adbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.680+0000 7f0adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.680+0000 7f0adbfff640 10 monclient(hunting): my global_id is 25328 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.680+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ad40065c0 con 0x7f0ad4009130 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4224506172 0 0) 0x7f0ad40065c0 con 0x7f0ad4009130 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ad4009e70 con 0x7f0ad4009130 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0ae40043c0 con 0x7f0ad4009130 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:30.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 10 monclient: dump: 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.686 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 190472796 0 0) 0x7f0ad4009e70 con 0x7f0ad4009130 2024-08-19T12:40:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ad400c3f0 con 0x7f0ad4009130 2024-08-19T12:40:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 962594072) 0x7f0ae4007ba0 con 0x7f0ad4009130 2024-08-19T12:40:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.26:6790/0 conn(0x7f0ad4009130 legacy=0x7f0ad40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0adbfff640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.26:6790/0 conn(0x7f0ad4009130 legacy=0x7f0ad40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:30.686 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 143736, 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:30.687 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:30.688 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:30.689 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:30.689 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:30.689 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:30.689 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:30.689 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:30.689 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.689 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:30.689 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:30.689 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:30.689 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:30.689 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:30.690 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:30.690 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:30.690 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:30.690 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:30.690 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:30.690 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:30.690 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:30.690 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:30.690 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.690 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:30.691 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:30.691 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:30.691 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:30.691 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:30.691 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:30.691 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:30.691 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:30.691 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:30.691 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:30.691 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:30.691 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:30.692 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:30.692 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:30.692 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:30.692 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:30.692 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:30.692 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:30.692 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:30.692 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:30.692 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:30.692 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:30.692 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:30.692 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:30.693 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:30.693 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:30.693 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:30.693 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:30.693 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.693 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:30.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.693 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:30.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.693 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:30.693 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:30.693 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.694 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.694 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.694 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:30.694 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.694 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.694 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.694 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:30.694 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:30.694 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.698 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.699 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:30.700 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.701 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.703 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.704 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.705 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.706 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.707 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:30.708 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:30.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0af9b5c640 10 librados: watch_flush enter 2024-08-19T12:40:30.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.681+0000 7f0af9b5c640 10 librados: watch_flush exit 2024-08-19T12:40:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af9b5c640 20 client.25557.objecter shutdown clearing up homeless session... 2024-08-19T12:40:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af9b5c640 10 client.25557.objecter successfully canceled tick 2024-08-19T12:40:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.104:6800/899887506 conn(0x7f0ae0041580 legacy=0x7f0ae0043a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af9b5c640 10 monclient: shutdown 2024-08-19T12:40:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af9b5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3580342196 >> v1:172.21.15.26:6792/0 conn(0x7f0af40a5920 legacy=0x7f0af40a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af3fff640 1 -- 172.21.15.26:0/3580342196 reap_dead start 2024-08-19T12:40:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af3fff640 1 -- 172.21.15.26:0/3580342196 reap_dead start 2024-08-19T12:40:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3580342196 shutdown_connections 2024-08-19T12:40:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3580342196 >> 172.21.15.26:0/3580342196 conn(0x7f0af40a8740 msgr2=0x7f0af4135150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3580342196 shutdown_connections 2024-08-19T12:40:30.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af9b5c640 1 -- 172.21.15.26:0/3580342196 wait complete. 2024-08-19T12:40:30.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.682+0000 7f0af9b5c640 1 librados: shutdown 2024-08-19T12:40:30.711 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:30.711 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:40:30.711 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:40:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.812+0000 7f25cfd04640 10 monclient: get_monmap_and_config 2024-08-19T12:40:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.812+0000 7f25cfd04640 10 monclient: build_initial_monmap 2024-08-19T12:40:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.812+0000 7f25cfd04640 10 monclient: monmap: 2024-08-19T12:40:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.814 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:30.814323+0000 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:30.814323+0000 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 1 Processor -- start 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 1 -- start start 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 10 monclient: init 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 10 monclient: _add_conns ranks=[0,3,1,6,4,2,7,8,5] 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 10 monclient(hunting): picked mon.noname-a con 0x7f25c806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 10 monclient(hunting): picked mon.noname-g con 0x7f25c80a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 10 monclient(hunting): picked mon.noname-f con 0x7f25c80a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c8136790 con 0x7f25c806e7b0 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c809ddf0 con 0x7f25c80a5d50 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c809e0d0 con 0x7f25c80a5920 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cfd04640 10 monclient(hunting): authenticate will time out at 824.901777s 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cdaa0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f25c806e7b0 0x7f25c8136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59824/0 (socket says 172.21.15.26:59824) 2024-08-19T12:40:30.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.813+0000 7f25cdaa0640 1 -- 172.21.15.26:0/698823802 learned_addr learned my addr 172.21.15.26:0/698823802 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:30.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cdaa0640 1 -- 172.21.15.26:0/698823802 >> v1:172.21.15.26:6789/0 conn(0x7f25c806e7b0 legacy=0x7f25c8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cd29f640 1 -- 172.21.15.26:0/698823802 >> v1:172.21.15.104:6790/0 conn(0x7f25c80a5920 legacy=0x7f25c813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25ce2a1640 1 -- 172.21.15.26:0/698823802 >> v1:172.21.15.104:6789/0 conn(0x7f25c80a5d50 legacy=0x7f25c8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1816395650 0 0) 0x7f25c8136790 con 0x7f25c806e7b0 2024-08-19T12:40:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient(hunting): my global_id is 26865 2024-08-19T12:40:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25b4003520 con 0x7f25c806e7b0 2024-08-19T12:40:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1973317885 0 0) 0x7f25c809e0d0 con 0x7f25c80a5920 2024-08-19T12:40:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient(hunting): my global_id is 25455 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c8136790 con 0x7f25c80a5920 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3994850514 0 0) 0x7f25c0001750 con 0x7f25c80a5d50 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient(hunting): my global_id is 25948 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c809e0d0 con 0x7f25c80a5d50 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1354459276 0 0) 0x7f25b4003520 con 0x7f25c806e7b0 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 >> v1:172.21.15.104:6790/0 conn(0x7f25c80a5920 legacy=0x7f25c813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 >> v1:172.21.15.104:6789/0 conn(0x7f25c80a5d50 legacy=0x7f25c8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient: _finish_hunting 0 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 monclient: found mon.noname-a 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:30.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c809e3b0 con 0x7f25c806e7b0 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient: _finish_auth 0 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient: _check_auth_tickets 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f25c4001ae0 con 0x7f25c806e7b0 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cfd04640 5 monclient: authenticate success, global_id 26865 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cfd04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient: dump: 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.824 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cfd04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f25c40048a0 con 0x7f25c806e7b0 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 1 -- 172.21.15.26:0/698823802 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f25c4004e10 con 0x7f25c806e7b0 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cfd04640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cfd04640 4 set_mon_vals no callback set 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.816+0000 7f25cca9e640 10 monclient: dump: 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.817+0000 7f25cfd04640 10 monclient: shutdown 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.817+0000 7f25cfd04640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.817+0000 7f25cfd04640 1 -- 172.21.15.26:0/698823802 >> v1:172.21.15.26:6789/0 conn(0x7f25c806e7b0 legacy=0x7f25c8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.817+0000 7f25cfd04640 1 -- 172.21.15.26:0/698823802 shutdown_connections 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.817+0000 7f25cfd04640 1 -- 172.21.15.26:0/698823802 >> 172.21.15.26:0/698823802 conn(0x7f25c80a8740 msgr2=0x7f25c80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.817+0000 7f25cfd04640 1 -- 172.21.15.26:0/698823802 shutdown_connections 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.817+0000 7f25cfd04640 1 -- 172.21.15.26:0/698823802 wait complete. 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.817+0000 7f25cfd04640 10 monclient: build_initial_monmap 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.817+0000 7f25cfd04640 10 monclient: monmap: 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:30.828 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.817+0000 7f25cfd04640 1 librados: starting msgr at 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.817+0000 7f25cfd04640 1 librados: starting objecter 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 1 Processor -- start 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 1 -- start start 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 1 librados: setting wanted keys 2024-08-19T12:40:30.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 1 librados: calling monclient init 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 10 monclient: init 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 10 monclient: _add_conns ranks=[7,8,4,3,0,5,2,6,1] 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 10 monclient(hunting): picked mon.noname-i con 0x7f25c806e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 10 monclient(hunting): picked mon.noname-e con 0x7f25c80a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 10 monclient(hunting): picked mon.noname-c con 0x7f25c80a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c8134a50 con 0x7f25c80a5d50 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c8134e80 con 0x7f25c806e7b0 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c8124b60 con 0x7f25c80a5920 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cfd04640 10 monclient(hunting): authenticate will time out at 824.906686s 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cd29f640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f25c80a5920 0x7f25c81d3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50184/0 (socket says 172.21.15.26:50184) 2024-08-19T12:40:30.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25ce2a1640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f25c80a5d50 0x7f25c81d69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:39084/0 (socket says 172.21.15.26:39084) 2024-08-19T12:40:30.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.818+0000 7f25cd29f640 1 -- 172.21.15.26:0/1098899346 learned_addr learned my addr 172.21.15.26:0/1098899346 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:30.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.820+0000 7f25cd29f640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.26:6793/0 conn(0x7f25c80a5920 legacy=0x7f25c81d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.820+0000 7f25be7fc640 10 client.?.objecter ms_handle_connect 0x7f25c80a5920 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.820+0000 7f25be7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.820+0000 7f25be7fc640 10 client.?.objecter ms_handle_connect 0x7f25c80a5d50 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.820+0000 7f25be7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.820+0000 7f25ce2a1640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.26:6791/0 conn(0x7f25c80a5d50 legacy=0x7f25c81d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25cdaa0640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.104:6792/0 conn(0x7f25c806e7b0 legacy=0x7f25c8128620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 client.?.objecter ms_handle_connect 0x7f25c806e7b0 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3083676067 0 0) 0x7f25c8124b60 con 0x7f25c80a5920 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): my global_id is 25172 2024-08-19T12:40:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25ac004eb0 con 0x7f25c80a5920 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4075221369 0 0) 0x7f25c8134a50 con 0x7f25c80a5d50 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): my global_id is 25573 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c8124b60 con 0x7f25c80a5d50 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 97473335 0 0) 0x7f25c8134e80 con 0x7f25c806e7b0 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): my global_id is 134107 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c8134a50 con 0x7f25c806e7b0 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3567759927 0 0) 0x7f25ac004eb0 con 0x7f25c80a5920 2024-08-19T12:40:30.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25c8134e80 con 0x7f25c80a5920 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f25b8003610 con 0x7f25c80a5920 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): dump: 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.839 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2776454925 0 0) 0x7f25c8124b60 con 0x7f25c80a5d50 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25ac004eb0 con 0x7f25c80a5d50 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f25c0001790 con 0x7f25c80a5d50 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): dump: 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.840 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1829845258 0 0) 0x7f25c8134a50 con 0x7f25c806e7b0 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25c8124b60 con 0x7f25c806e7b0 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f25c4004d90 con 0x7f25c806e7b0 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient(hunting): dump: 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.842 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2228504176 0 0) 0x7f25c8134e80 con 0x7f25c80a5920 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.104:6792/0 conn(0x7f25c806e7b0 legacy=0x7f25c8128620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.26:6791/0 conn(0x7f25c80a5d50 legacy=0x7f25c81d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 monclient: found mon.e 2024-08-19T12:40:30.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c8126620 con 0x7f25c80a5920 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient: _finish_auth 0 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25cfd04640 5 monclient: authenticate success, global_id 25172 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25cfd04640 10 monclient: _renew_subs 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25cfd04640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25cfd04640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f25c8126ec0 con 0x7f25c80a5920 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25cfd04640 10 client.25172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25cfd04640 10 monclient: _renew_subs 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25cfd04640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:30.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25cfd04640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f25c8128d30 con 0x7f25c80a5920 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25cfd04640 1 librados: init done 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f25b8003be0 con 0x7f25c80a5920 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f25b8004110 con 0x7f25c80a5920 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25bffff640 4 set_mon_vals no callback set 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.821+0000 7f25be7fc640 10 monclient: dump: 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.845 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.822+0000 7f25cfd04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.822+0000 7f25cfd04640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f25b00065c0 con 0x7f25b0002860 2024-08-19T12:40:30.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.822+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f25b80042b0 con 0x7f25c80a5920 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.822+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(74..74 src has 1..74) ==== 7453+0+0 (unknown 2159442305 0 0) 0x7f25b804eab0 con 0x7f25c80a5920 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.822+0000 7f25be7fc640 10 client.25172.objecter ms_dispatch 0x7f25c8000b90 osd_map(74..74 src has 1..74) 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.822+0000 7f25be7fc640 3 client.25172.objecter handle_osd_map got epochs [74,74] > 0 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.822+0000 7f25be7fc640 3 client.25172.objecter handle_osd_map decoding full epoch 74 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.823+0000 7f25be7fc640 20 client.25172.objecter dump_active .. 0 homeless 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.823+0000 7f25ce2a1640 10 monclient: get_auth_request con 0x7f25ac0418c0 auth_method 0 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.824+0000 7f25be7fc640 10 client.25172.objecter ms_handle_connect 0x7f25b0002860 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.824+0000 7f25be7fc640 10 client.25172.objecter resend_mon_ops 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.824+0000 7f25cdaa0640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.26:6791/0 conn(0x7f25b0002860 legacy=0x7f25b0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.824+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2400003877 0 0) 0x7f25b00065c0 con 0x7f25b0002860 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.824+0000 7f25be7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.824+0000 7f25be7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.824+0000 7f25be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.824+0000 7f25be7fc640 10 monclient(hunting): my global_id is 25582 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.824+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c8134e80 con 0x7f25b0002860 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2615489223 0 0) 0x7f25c8134e80 con 0x7f25b0002860 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25b00065c0 con 0x7f25b0002860 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f25c4001d30 con 0x7f25b0002860 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 10 monclient: dump: 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.848 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2426048979 0 0) 0x7f25b00065c0 con 0x7f25b0002860 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25b000a160 con 0x7f25b0002860 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 10 client.25172.objecter ms_handle_connect 0x7f25ac0418c0 2024-08-19T12:40:30.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f25c40048a0 con 0x7f25b0002860 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.26:6791/0 conn(0x7f25b0002860 legacy=0x7f25b0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.825+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.26:6791/0 conn(0x7f25b0002860 legacy=0x7f25b0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.831+0000 7f25cfd04640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.831+0000 7f25cfd04640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f25b0009e70 con 0x7f25b0009130 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25cdaa0640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.26:6791/0 conn(0x7f25b0009130 legacy=0x7f25b00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 10 client.25172.objecter ms_handle_connect 0x7f25b0009130 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 10 client.25172.objecter resend_mon_ops 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2249655381 0 0) 0x7f25b0009e70 con 0x7f25b0009130 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 10 monclient(hunting): my global_id is 25591 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25b00065c0 con 0x7f25b0009130 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3457632561 0 0) 0x7f25b00065c0 con 0x7f25b0009130 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25b0009e70 con 0x7f25b0009130 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f25c4005ac0 con 0x7f25b0009130 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 10 monclient: dump: 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.851 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.852 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.853 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 541203167 0 0) 0x7f25b0009e70 con 0x7f25b0009130 2024-08-19T12:40:30.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f25b000c410 con 0x7f25b0009130 2024-08-19T12:40:30.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7878 (unknown 0 0 1352762476) 0x7f25c4005ed0 con 0x7f25b0009130 2024-08-19T12:40:30.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:30.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:30.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.26:6791/0 conn(0x7f25b0009130 legacy=0x7f25b00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.834+0000 7f25be7fc640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.26:6791/0 conn(0x7f25b0009130 legacy=0x7f25b00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:30.853 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 143864, 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:30.854 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:30.855 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:30.856 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:30.857 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:30.858 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:30.859 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.862 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.863 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:30.864 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.865 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.867 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:30.870 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.871 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:30.872 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:30.873 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:30.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:30.874 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:30.874 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 10 librados: watch_flush enter 2024-08-19T12:40:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 10 librados: watch_flush exit 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 20 client.25172.objecter shutdown clearing up homeless session... 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 10 client.25172.objecter successfully canceled tick 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.104:6800/899887506 conn(0x7f25ac0418c0 legacy=0x7f25ac043d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 10 monclient: shutdown 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 1 -- 172.21.15.26:0/1098899346 >> v1:172.21.15.26:6793/0 conn(0x7f25c80a5920 legacy=0x7f25c81d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25ce2a1640 1 -- 172.21.15.26:0/1098899346 reap_dead start 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25ce2a1640 1 -- 172.21.15.26:0/1098899346 reap_dead start 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 1 -- 172.21.15.26:0/1098899346 shutdown_connections 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 1 -- 172.21.15.26:0/1098899346 >> 172.21.15.26:0/1098899346 conn(0x7f25c80a8740 msgr2=0x7f25c80aa870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 1 -- 172.21.15.26:0/1098899346 shutdown_connections 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 1 -- 172.21.15.26:0/1098899346 wait complete. 2024-08-19T12:40:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.835+0000 7f25cfd04640 1 librados: shutdown 2024-08-19T12:40:30.877 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:30.877 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:40:30.877 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:40:30.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.974+0000 7f3db46bc640 10 monclient: get_monmap_and_config 2024-08-19T12:40:30.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.974+0000 7f3db46bc640 10 monclient: build_initial_monmap 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.974+0000 7f3db46bc640 10 monclient: monmap: 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:30.976630+0000 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:30.976630+0000 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:30.978 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:30.978 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:30.978 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.975+0000 7f3db46bc640 1 Processor -- start 2024-08-19T12:40:30.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.975+0000 7f3db46bc640 1 -- start start 2024-08-19T12:40:30.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.975+0000 7f3db46bc640 10 monclient: init 2024-08-19T12:40:30.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.975+0000 7f3db46bc640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:30.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.976+0000 7f3db46bc640 10 monclient: _add_conns ranks=[2,6,3,7,4,0,8,5,1] 2024-08-19T12:40:30.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.976+0000 7f3db46bc640 10 monclient(hunting): picked mon.noname-b con 0x7f3dac06e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:30.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.976+0000 7f3db46bc640 10 monclient(hunting): picked mon.noname-d con 0x7f3dac0a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:30.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.976+0000 7f3db46bc640 10 monclient(hunting): picked mon.noname-g con 0x7f3dac0a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:30.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.976+0000 7f3db46bc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dac136790 con 0x7f3dac06e7b0 2024-08-19T12:40:30.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.976+0000 7f3db46bc640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dac09ddf0 con 0x7f3dac0a5d50 2024-08-19T12:40:30.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.976+0000 7f3db46bc640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dac09e0d0 con 0x7f3dac0a5920 2024-08-19T12:40:30.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.976+0000 7f3db46bc640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:30.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.976+0000 7f3db46bc640 10 monclient(hunting): authenticate will time out at 825.064100s 2024-08-19T12:40:30.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.976+0000 7f3db2458640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3dac06e7b0 0x7f3dac136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58210/0 (socket says 172.21.15.26:58210) 2024-08-19T12:40:30.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.976+0000 7f3db2458640 1 -- 172.21.15.26:0/1705079812 learned_addr learned my addr 172.21.15.26:0/1705079812 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:30.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.976+0000 7f3db1c57640 1 --1- 172.21.15.26:0/1705079812 >> v1:172.21.15.26:6792/0 conn(0x7f3dac0a5920 0x7f3dac13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39054/0 (socket says 172.21.15.26:39054) 2024-08-19T12:40:30.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db2458640 1 -- 172.21.15.26:0/1705079812 >> v1:172.21.15.26:6790/0 conn(0x7f3dac06e7b0 legacy=0x7f3dac136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1c57640 1 -- 172.21.15.26:0/1705079812 >> v1:172.21.15.26:6792/0 conn(0x7f3dac0a5920 legacy=0x7f3dac13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 1 -- 172.21.15.26:0/1705079812 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3128489839 0 0) 0x7f3dac136790 con 0x7f3dac06e7b0 2024-08-19T12:40:30.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient(hunting): my global_id is 25337 2024-08-19T12:40:30.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 1 -- 172.21.15.26:0/1705079812 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d94003520 con 0x7f3dac06e7b0 2024-08-19T12:40:30.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 1 -- 172.21.15.26:0/1705079812 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1909885730 0 0) 0x7f3dac09e0d0 con 0x7f3dac0a5920 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient(hunting): my global_id is 25566 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 1 -- 172.21.15.26:0/1705079812 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dac136790 con 0x7f3dac0a5920 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db2c59640 1 -- 172.21.15.26:0/1705079812 >> v1:172.21.15.104:6790/0 conn(0x7f3dac0a5d50 legacy=0x7f3dac141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 1 -- 172.21.15.26:0/1705079812 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3148596004 0 0) 0x7f3d94003520 con 0x7f3dac06e7b0 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 1 -- 172.21.15.26:0/1705079812 >> v1:172.21.15.26:6792/0 conn(0x7f3dac0a5920 legacy=0x7f3dac13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 1 -- 172.21.15.26:0/1705079812 >> v1:172.21.15.104:6790/0 conn(0x7f3dac0a5d50 legacy=0x7f3dac141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient: _finish_hunting 0 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 1 monclient: found mon.noname-b 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:30.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 1 -- 172.21.15.26:0/1705079812 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac09e3b0 con 0x7f3dac06e7b0 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient: _finish_auth 0 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient: _check_auth_tickets 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 1 -- 172.21.15.26:0/1705079812 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3da4001ae0 con 0x7f3dac06e7b0 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db46bc640 5 monclient: authenticate success, global_id 25337 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db46bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db1456640 10 monclient: dump: 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.986 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.978+0000 7f3db46bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db1456640 1 -- 172.21.15.26:0/1705079812 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3da40048a0 con 0x7f3dac06e7b0 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db1456640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db1456640 1 -- 172.21.15.26:0/1705079812 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3da4004e10 con 0x7f3dac06e7b0 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db1456640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db1456640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db1456640 10 monclient: dump: 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:30.988 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db46bc640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db46bc640 4 set_mon_vals no callback set 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db46bc640 10 monclient: shutdown 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db46bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db46bc640 1 -- 172.21.15.26:0/1705079812 >> v1:172.21.15.26:6790/0 conn(0x7f3dac06e7b0 legacy=0x7f3dac136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db46bc640 1 -- 172.21.15.26:0/1705079812 shutdown_connections 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db46bc640 1 -- 172.21.15.26:0/1705079812 >> 172.21.15.26:0/1705079812 conn(0x7f3dac0a8740 msgr2=0x7f3dac0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:30.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db46bc640 1 -- 172.21.15.26:0/1705079812 shutdown_connections 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.979+0000 7f3db46bc640 1 -- 172.21.15.26:0/1705079812 wait complete. 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 10 monclient: build_initial_monmap 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 10 monclient: monmap: 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:30.990 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 1 librados: starting msgr at 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 1 librados: starting objecter 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 1 Processor -- start 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 1 -- start start 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 1 librados: setting wanted keys 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 1 librados: calling monclient init 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 10 monclient: init 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:30.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 10 monclient: _add_conns ranks=[0,7,6,4,2,5,3,8,1] 2024-08-19T12:40:30.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 10 monclient(hunting): picked mon.noname-a con 0x7f3dac06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:30.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 10 monclient(hunting): picked mon.noname-i con 0x7f3dac0a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:30.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.980+0000 7f3db46bc640 10 monclient(hunting): picked mon.noname-d con 0x7f3dac0a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:30.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.981+0000 7f3db46bc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dac142050 con 0x7f3dac06e7b0 2024-08-19T12:40:30.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.981+0000 7f3db46bc640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dac13ae00 con 0x7f3dac0a5d50 2024-08-19T12:40:30.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.981+0000 7f3db46bc640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dac13b0e0 con 0x7f3dac0a5920 2024-08-19T12:40:30.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.981+0000 7f3db46bc640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:30.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.981+0000 7f3db46bc640 10 monclient(hunting): authenticate will time out at 825.069045s 2024-08-19T12:40:30.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.981+0000 7f3db2458640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3dac06e7b0 0x7f3dac1379e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59836/0 (socket says 172.21.15.26:59836) 2024-08-19T12:40:30.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.981+0000 7f3db2458640 1 -- 172.21.15.26:0/388425572 learned_addr learned my addr 172.21.15.26:0/388425572 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:30.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3db2c59640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.26:6792/0 conn(0x7f3dac0a5d50 legacy=0x7f3dac1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 client.?.objecter ms_handle_connect 0x7f3dac0a5d50 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 client.?.objecter ms_handle_connect 0x7f3dac06e7b0 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3db2458640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.26:6789/0 conn(0x7f3dac06e7b0 legacy=0x7f3dac1379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1947962056 0 0) 0x7f3dac13ae00 con 0x7f3dac0a5d50 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): my global_id is 25575 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d98004eb0 con 0x7f3dac0a5d50 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1583114050 0 0) 0x7f3dac142050 con 0x7f3dac06e7b0 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:30.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): my global_id is 26874 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dac13ae00 con 0x7f3dac06e7b0 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 client.?.objecter ms_handle_connect 0x7f3dac0a5920 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3db1c57640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.104:6792/0 conn(0x7f3dac0a5920 legacy=0x7f3dac1380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2295934061 0 0) 0x7f3dac13b0e0 con 0x7f3dac0a5920 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): my global_id is 134116 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dac142050 con 0x7f3dac0a5920 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3283346086 0 0) 0x7f3d98004eb0 con 0x7f3dac0a5d50 2024-08-19T12:40:31.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3dac13b0e0 con 0x7f3dac0a5d50 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3d9c003610 con 0x7f3dac0a5d50 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): dump: 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.001 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1215389164 0 0) 0x7f3dac13ae00 con 0x7f3dac06e7b0 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d98004eb0 con 0x7f3dac06e7b0 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3da4004d90 con 0x7f3dac06e7b0 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:31.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): dump: 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.003 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 88156984 0 0) 0x7f3dac142050 con 0x7f3dac0a5920 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3dac13ae00 con 0x7f3dac0a5920 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3da00016a0 con 0x7f3dac0a5920 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient(hunting): dump: 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.004 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 268627824 0 0) 0x7f3dac13b0e0 con 0x7f3dac0a5d50 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.104:6792/0 conn(0x7f3dac0a5920 legacy=0x7f3dac1380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.26:6789/0 conn(0x7f3dac06e7b0 legacy=0x7f3dac1379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient: _finish_hunting 0 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 monclient: found mon.d 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac139060 con 0x7f3dac0a5d50 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient: _finish_auth 0 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 10 monclient: _check_auth_tickets 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3daaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3db46bc640 5 monclient: authenticate success, global_id 25575 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3db46bc640 10 monclient: _renew_subs 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3db46bc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.983+0000 7f3db46bc640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3dac13b3c0 con 0x7f3dac0a5d50 2024-08-19T12:40:31.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3db46bc640 10 client.25575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3db46bc640 10 monclient: _renew_subs 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3db46bc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3db46bc640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f3dac13a6d0 con 0x7f3dac0a5d50 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3db46bc640 1 librados: init done 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d9c003b10 con 0x7f3dac0a5d50 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3daaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3d9c004040 con 0x7f3dac0a5d50 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3daaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3daaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3db0c55640 4 set_mon_vals no callback set 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3daaffd640 10 monclient: dump: 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.007 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.008 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3db46bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.984+0000 7f3db46bc640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d8c0065c0 con 0x7f3d8c002860 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.985+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3d9c0041e0 con 0x7f3dac0a5d50 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.985+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(74..74 src has 1..74) ==== 7453+0+0 (unknown 2159442305 0 0) 0x7f3d9c04e6e0 con 0x7f3dac0a5d50 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.985+0000 7f3daaffd640 10 client.25575.objecter ms_dispatch 0x7f3dac000b90 osd_map(74..74 src has 1..74) 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.985+0000 7f3daaffd640 3 client.25575.objecter handle_osd_map got epochs [74,74] > 0 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.985+0000 7f3daaffd640 3 client.25575.objecter handle_osd_map decoding full epoch 74 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.985+0000 7f3daaffd640 20 client.25575.objecter dump_active .. 0 homeless 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.985+0000 7f3db1c57640 10 monclient: get_auth_request con 0x7f3d98041a60 auth_method 0 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3db2458640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.26:6792/0 conn(0x7f3d8c002860 legacy=0x7f3d8c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 10 client.25575.objecter ms_handle_connect 0x7f3d8c002860 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 10 client.25575.objecter resend_mon_ops 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1942687283 0 0) 0x7f3d8c0065c0 con 0x7f3d8c002860 2024-08-19T12:40:31.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 10 monclient(hunting): my global_id is 25584 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dac13b0e0 con 0x7f3d8c002860 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1607820604 0 0) 0x7f3dac13b0e0 con 0x7f3d8c002860 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d8c0065c0 con 0x7f3d8c002860 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3da4001d30 con 0x7f3d8c002860 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 10 monclient: dump: 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.010 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 909993729 0 0) 0x7f3d8c0065c0 con 0x7f3d8c002860 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d8c00a160 con 0x7f3d8c002860 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3da4005740 con 0x7f3d8c002860 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.26:6792/0 conn(0x7f3d8c002860 legacy=0x7f3d8c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.26:6792/0 conn(0x7f3d8c002860 legacy=0x7f3d8c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.987+0000 7f3daaffd640 10 client.25575.objecter ms_handle_connect 0x7f3d98041a60 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.993+0000 7f3db46bc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.993+0000 7f3db46bc640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3d8c009e70 con 0x7f3d8c009130 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3db2458640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.26:6792/0 conn(0x7f3d8c009130 legacy=0x7f3d8c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 10 client.25575.objecter ms_handle_connect 0x7f3d8c009130 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 10 client.25575.objecter resend_mon_ops 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3953769251 0 0) 0x7f3d8c009e70 con 0x7f3d8c009130 2024-08-19T12:40:31.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 10 monclient(hunting): my global_id is 25593 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d8c0065c0 con 0x7f3d8c009130 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3800815821 0 0) 0x7f3d8c0065c0 con 0x7f3d8c009130 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d8c009e70 con 0x7f3d8c009130 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3da4005f20 con 0x7f3d8c009130 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 10 monclient: dump: 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.013 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.014 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2140470704 0 0) 0x7f3d8c009e70 con 0x7f3d8c009130 2024-08-19T12:40:31.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d8c00c3f0 con 0x7f3d8c009130 2024-08-19T12:40:31.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3206530716) 0x7f3da40082a0 con 0x7f3d8c009130 2024-08-19T12:40:31.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:31.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:31.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.26:6792/0 conn(0x7f3d8c009130 legacy=0x7f3d8c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.996+0000 7f3daaffd640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.26:6792/0 conn(0x7f3d8c009130 legacy=0x7f3d8c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:31.015 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:31.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 144037, 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:31.016 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:31.018 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:31.019 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:31.020 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:31.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.022 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:31.023 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.024 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.027 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:31.028 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:31.030 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:31.031 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:31.034 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:40:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.035 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:31.035 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 10 librados: watch_flush enter 2024-08-19T12:40:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 10 librados: watch_flush exit 2024-08-19T12:40:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 20 client.25575.objecter shutdown clearing up homeless session... 2024-08-19T12:40:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 10 client.25575.objecter successfully canceled tick 2024-08-19T12:40:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.104:6800/899887506 conn(0x7f3d98041a60 legacy=0x7f3d98043f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 10 monclient: shutdown 2024-08-19T12:40:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 1 -- 172.21.15.26:0/388425572 >> v1:172.21.15.26:6792/0 conn(0x7f3dac0a5d50 legacy=0x7f3dac1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db2c59640 1 -- 172.21.15.26:0/388425572 reap_dead start 2024-08-19T12:40:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db2c59640 1 -- 172.21.15.26:0/388425572 reap_dead start 2024-08-19T12:40:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 1 -- 172.21.15.26:0/388425572 shutdown_connections 2024-08-19T12:40:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 1 -- 172.21.15.26:0/388425572 >> 172.21.15.26:0/388425572 conn(0x7f3dac0a8740 msgr2=0x7f3dac0aa640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 1 -- 172.21.15.26:0/388425572 shutdown_connections 2024-08-19T12:40:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 1 -- 172.21.15.26:0/388425572 wait complete. 2024-08-19T12:40:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:30.997+0000 7f3db46bc640 1 librados: shutdown 2024-08-19T12:40:31.038 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:31.039 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:40:31.039 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:40:31.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.137+0000 7f8f8af53640 10 monclient: get_monmap_and_config 2024-08-19T12:40:31.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.137+0000 7f8f8af53640 10 monclient: build_initial_monmap 2024-08-19T12:40:31.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.137+0000 7f8f8af53640 10 monclient: monmap: 2024-08-19T12:40:31.139 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:31.139 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.139 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:31.139230+0000 2024-08-19T12:40:31.139 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:31.139230+0000 2024-08-19T12:40:31.139 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:31.139 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.139 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:31.140 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:31.140 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:31.140 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:31.140 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:31.140 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:31.140 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:31.140 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:31.140 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:31.140 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 1 Processor -- start 2024-08-19T12:40:31.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 1 -- start start 2024-08-19T12:40:31.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 10 monclient: init 2024-08-19T12:40:31.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:31.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 10 monclient: _add_conns ranks=[3,6,5,0,2,4,7,1,8] 2024-08-19T12:40:31.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 10 monclient(hunting): picked mon.noname-g con 0x7f8f8406e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:31.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 10 monclient(hunting): picked mon.noname-d con 0x7f8f840a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:31.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 10 monclient(hunting): picked mon.noname-h con 0x7f8f840a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:31.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f84136790 con 0x7f8f8406e7b0 2024-08-19T12:40:31.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f8409ddf0 con 0x7f8f840a5d50 2024-08-19T12:40:31.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f8409e0d0 con 0x7f8f840a5920 2024-08-19T12:40:31.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:31.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f8af53640 10 monclient(hunting): authenticate will time out at 825.226654s 2024-08-19T12:40:31.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f83fff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f8f840a5920 0x7f8f8413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39078/0 (socket says 172.21.15.26:39078) 2024-08-19T12:40:31.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.138+0000 7f8f83fff640 1 -- 172.21.15.26:0/157789410 learned_addr learned my addr 172.21.15.26:0/157789410 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:31.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.140+0000 7f8f83fff640 1 -- 172.21.15.26:0/157789410 >> v1:172.21.15.26:6792/0 conn(0x7f8f840a5920 legacy=0x7f8f8413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.141+0000 7f8f88cef640 1 -- 172.21.15.26:0/157789410 >> v1:172.21.15.104:6790/0 conn(0x7f8f8406e7b0 legacy=0x7f8f84136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.141+0000 7f8f894f0640 1 -- 172.21.15.26:0/157789410 >> v1:172.21.15.104:6791/0 conn(0x7f8f840a5d50 legacy=0x7f8f84141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.784+0000 7f8f837fe640 1 -- 172.21.15.26:0/157789410 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2255700589 0 0) 0x7f8f7c001820 con 0x7f8f840a5d50 2024-08-19T12:40:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.784+0000 7f8f837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.784+0000 7f8f837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.784+0000 7f8f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.784+0000 7f8f837fe640 10 monclient(hunting): my global_id is 25349 2024-08-19T12:40:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.784+0000 7f8f837fe640 1 -- 172.21.15.26:0/157789410 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f74003520 con 0x7f8f840a5d50 2024-08-19T12:40:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.784+0000 7f8f837fe640 1 -- 172.21.15.26:0/157789410 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2556696266 0 0) 0x7f8f8409e0d0 con 0x7f8f840a5920 2024-08-19T12:40:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.784+0000 7f8f837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.784+0000 7f8f837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.784+0000 7f8f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.784+0000 7f8f837fe640 10 monclient(hunting): my global_id is 25602 2024-08-19T12:40:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.784+0000 7f8f837fe640 1 -- 172.21.15.26:0/157789410 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f740048e0 con 0x7f8f840a5920 2024-08-19T12:40:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 1 -- 172.21.15.26:0/157789410 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 738392659 0 0) 0x7f8f740048e0 con 0x7f8f840a5920 2024-08-19T12:40:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 1 -- 172.21.15.26:0/157789410 >> v1:172.21.15.104:6791/0 conn(0x7f8f840a5d50 legacy=0x7f8f84141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 1 -- 172.21.15.26:0/157789410 >> v1:172.21.15.104:6790/0 conn(0x7f8f8406e7b0 legacy=0x7f8f84136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 10 monclient: _finish_hunting 0 2024-08-19T12:40:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 1 monclient: found mon.noname-d 2024-08-19T12:40:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:40:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 1 -- 172.21.15.26:0/157789410 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8409e3b0 con 0x7f8f840a5920 2024-08-19T12:40:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 10 monclient: _finish_auth 0 2024-08-19T12:40:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 10 monclient: _check_auth_tickets 2024-08-19T12:40:31.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 1 -- 172.21.15.26:0/157789410 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8f70001ac0 con 0x7f8f840a5920 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f837fe640 10 monclient: dump: 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.790 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.791 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f8af53640 5 monclient: authenticate success, global_id 25602 2024-08-19T12:40:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.785+0000 7f8f8af53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.788+0000 7f8f837fe640 1 -- 172.21.15.26:0/157789410 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8f700048a0 con 0x7f8f840a5920 2024-08-19T12:40:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.788+0000 7f8f837fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.788+0000 7f8f8af53640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.788+0000 7f8f8af53640 4 set_mon_vals no callback set 2024-08-19T12:40:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.789+0000 7f8f837fe640 1 -- 172.21.15.26:0/157789410 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8f70004d90 con 0x7f8f840a5920 2024-08-19T12:40:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.789+0000 7f8f837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.789+0000 7f8f837fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.789+0000 7f8f837fe640 10 monclient: dump: 2024-08-19T12:40:31.792 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.793 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.794 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.790+0000 7f8f8af53640 10 monclient: shutdown 2024-08-19T12:40:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.790+0000 7f8f8af53640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.790+0000 7f8f8af53640 1 -- 172.21.15.26:0/157789410 >> v1:172.21.15.26:6792/0 conn(0x7f8f840a5920 legacy=0x7f8f8413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.790+0000 7f8f8af53640 1 -- 172.21.15.26:0/157789410 shutdown_connections 2024-08-19T12:40:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.790+0000 7f8f8af53640 1 -- 172.21.15.26:0/157789410 >> 172.21.15.26:0/157789410 conn(0x7f8f840a8740 msgr2=0x7f8f840a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.790+0000 7f8f8af53640 1 -- 172.21.15.26:0/157789410 shutdown_connections 2024-08-19T12:40:31.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.790+0000 7f8f8af53640 1 -- 172.21.15.26:0/157789410 wait complete. 2024-08-19T12:40:31.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.791+0000 7f8f8af53640 10 monclient: build_initial_monmap 2024-08-19T12:40:31.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.791+0000 7f8f8af53640 10 monclient: monmap: 2024-08-19T12:40:31.795 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:31.795 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:31.796 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.792+0000 7f8f8af53640 1 librados: starting msgr at 2024-08-19T12:40:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.792+0000 7f8f8af53640 1 librados: starting objecter 2024-08-19T12:40:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.792+0000 7f8f8af53640 1 Processor -- start 2024-08-19T12:40:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.793+0000 7f8f8af53640 1 -- start start 2024-08-19T12:40:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.793+0000 7f8f8af53640 1 librados: setting wanted keys 2024-08-19T12:40:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.793+0000 7f8f8af53640 1 librados: calling monclient init 2024-08-19T12:40:31.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.793+0000 7f8f8af53640 10 monclient: init 2024-08-19T12:40:31.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.794+0000 7f8f8af53640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.794+0000 7f8f8af53640 10 monclient: _add_conns ranks=[3,1,2,8,4,7,0,5,6] 2024-08-19T12:40:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.794+0000 7f8f8af53640 10 monclient(hunting): picked mon.noname-g con 0x7f8f8406e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.794+0000 7f8f8af53640 10 monclient(hunting): picked mon.noname-f con 0x7f8f840a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.794+0000 7f8f8af53640 10 monclient(hunting): picked mon.noname-b con 0x7f8f840a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.794+0000 7f8f8af53640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f8413b620 con 0x7f8f840a5920 2024-08-19T12:40:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.794+0000 7f8f8af53640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f841349a0 con 0x7f8f840a5d50 2024-08-19T12:40:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.794+0000 7f8f8af53640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f84138270 con 0x7f8f8406e7b0 2024-08-19T12:40:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.794+0000 7f8f8af53640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.794+0000 7f8f8af53640 10 monclient(hunting): authenticate will time out at 825.882747s 2024-08-19T12:40:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.794+0000 7f8f88cef640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f8f8406e7b0 0x7f8f84137450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:46108/0 (socket says 172.21.15.26:46108) 2024-08-19T12:40:31.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.795+0000 7f8f88cef640 1 -- 172.21.15.26:0/914744235 learned_addr learned my addr 172.21.15.26:0/914744235 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.798+0000 7f8f88cef640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.104:6790/0 conn(0x7f8f8406e7b0 legacy=0x7f8f84137450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.798+0000 7f8f817fa640 10 client.?.objecter ms_handle_connect 0x7f8f8406e7b0 2024-08-19T12:40:31.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.798+0000 7f8f817fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.798+0000 7f8f894f0640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.26:6790/0 conn(0x7f8f840a5d50 legacy=0x7f8f841d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.798+0000 7f8f83fff640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.104:6789/0 conn(0x7f8f840a5920 legacy=0x7f8f84137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.799+0000 7f8f817fa640 10 client.?.objecter ms_handle_connect 0x7f8f840a5d50 2024-08-19T12:40:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.799+0000 7f8f817fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.799+0000 7f8f817fa640 10 client.?.objecter ms_handle_connect 0x7f8f840a5920 2024-08-19T12:40:31.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.799+0000 7f8f817fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:31.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.802+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3566776664 0 0) 0x7f8f8409e3b0 con 0x7f8f840a5920 2024-08-19T12:40:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.803+0000 7f8f817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.803+0000 7f8f817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.803+0000 7f8f817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.803+0000 7f8f817fa640 10 monclient(hunting): my global_id is 25957 2024-08-19T12:40:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.803+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f68004eb0 con 0x7f8f840a5920 2024-08-19T12:40:31.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.803+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1057598147 0 0) 0x7f8f84138270 con 0x7f8f8406e7b0 2024-08-19T12:40:31.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.803+0000 7f8f817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.805+0000 7f8f817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.805+0000 7f8f817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.805+0000 7f8f817fa640 10 monclient(hunting): my global_id is 25482 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.805+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f8409e3b0 con 0x7f8f8406e7b0 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.805+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3591538810 0 0) 0x7f8f68004eb0 con 0x7f8f840a5920 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.805+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f84138270 con 0x7f8f840a5920 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.805+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8f70001ac0 con 0x7f8f840a5920 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.805+0000 7f8f817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.806+0000 7f8f817fa640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.806+0000 7f8f817fa640 10 monclient(hunting): dump: 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.808 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.809 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.806+0000 7f8f817fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.806+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1712947520 0 0) 0x7f8f8409e3b0 con 0x7f8f8406e7b0 2024-08-19T12:40:31.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.806+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f68004eb0 con 0x7f8f8406e7b0 2024-08-19T12:40:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.806+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3187909571 0 0) 0x7f8f84138270 con 0x7f8f840a5920 2024-08-19T12:40:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.806+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.104:6790/0 conn(0x7f8f8406e7b0 legacy=0x7f8f84137450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.806+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.26:6790/0 conn(0x7f8f840a5d50 legacy=0x7f8f841d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.807+0000 7f8f817fa640 10 monclient: _finish_hunting 0 2024-08-19T12:40:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.807+0000 7f8f817fa640 1 monclient: found mon.f 2024-08-19T12:40:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.807+0000 7f8f817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.807+0000 7f8f817fa640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:40:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.807+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8413af20 con 0x7f8f840a5920 2024-08-19T12:40:31.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.807+0000 7f8f817fa640 10 monclient: _finish_auth 0 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.807+0000 7f8f817fa640 10 monclient: _check_auth_tickets 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.807+0000 7f8f817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.808+0000 7f8f8af53640 5 monclient: authenticate success, global_id 25957 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.808+0000 7f8f8af53640 10 monclient: _renew_subs 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.808+0000 7f8f8af53640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.808+0000 7f8f8af53640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8f8413b200 con 0x7f8f840a5920 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.808+0000 7f8f8af53640 10 client.25957.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.808+0000 7f8f8af53640 10 monclient: _renew_subs 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.808+0000 7f8f8af53640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.808+0000 7f8f8af53640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8f84139c70 con 0x7f8f840a5920 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.808+0000 7f8f8af53640 1 librados: init done 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.809+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8f70001320 con 0x7f8f840a5920 2024-08-19T12:40:31.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.809+0000 7f8f817fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.809+0000 7f8f827fc640 4 set_mon_vals no callback set 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.809+0000 7f8f8af53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.809+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8f700048a0 con 0x7f8f840a5920 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.810+0000 7f8f8af53640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8f640065c0 con 0x7f8f64002860 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.810+0000 7f8f817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.810+0000 7f8f817fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.810+0000 7f8f817fa640 10 monclient: dump: 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.814 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.815 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.815 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.811+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f8f70011880 con 0x7f8f840a5920 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.811+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(75..75 src has 1..75) ==== 7121+0+0 (unknown 2329720755 0 0) 0x7f8f7004e7f0 con 0x7f8f840a5920 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.811+0000 7f8f817fa640 10 client.25957.objecter ms_dispatch 0x7f8f84000b90 osd_map(75..75 src has 1..75) 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.811+0000 7f8f817fa640 3 client.25957.objecter handle_osd_map got epochs [75,75] > 0 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.811+0000 7f8f817fa640 3 client.25957.objecter handle_osd_map decoding full epoch 75 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.811+0000 7f8f894f0640 10 monclient: get_auth_request con 0x7f8f680413f0 auth_method 0 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.811+0000 7f8f817fa640 20 client.25957.objecter dump_active .. 0 homeless 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f88cef640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.26:6793/0 conn(0x7f8f64002860 legacy=0x7f8f64004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 10 client.25957.objecter ms_handle_connect 0x7f8f64002860 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 10 client.25957.objecter resend_mon_ops 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1037666964 0 0) 0x7f8f640065c0 con 0x7f8f64002860 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 10 monclient(hunting): my global_id is 25181 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f84138270 con 0x7f8f64002860 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 912158960 0 0) 0x7f8f84138270 con 0x7f8f64002860 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f640065c0 con 0x7f8f64002860 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8f780040f0 con 0x7f8f64002860 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 10 monclient: dump: 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.821 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1831140664 0 0) 0x7f8f640065c0 con 0x7f8f64002860 2024-08-19T12:40:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.813+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f6400a160 con 0x7f8f64002860 2024-08-19T12:40:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.814+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8f78004380 con 0x7f8f64002860 2024-08-19T12:40:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.814+0000 7f8f817fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.814+0000 7f8f817fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.814+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.26:6793/0 conn(0x7f8f64002860 legacy=0x7f8f64004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.814+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.26:6793/0 conn(0x7f8f64002860 legacy=0x7f8f64004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:31.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.815+0000 7f8f817fa640 10 client.25957.objecter ms_handle_connect 0x7f8f680413f0 2024-08-19T12:40:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.821+0000 7f8f8af53640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:31.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.821+0000 7f8f8af53640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8f64009e50 con 0x7f8f64009110 2024-08-19T12:40:31.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.823+0000 7f8f88cef640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.26:6793/0 conn(0x7f8f64009110 legacy=0x7f8f640095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.823+0000 7f8f817fa640 10 client.25957.objecter ms_handle_connect 0x7f8f64009110 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.823+0000 7f8f817fa640 10 client.25957.objecter resend_mon_ops 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2213882074 0 0) 0x7f8f64009e50 con 0x7f8f64009110 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 10 monclient(hunting): my global_id is 25190 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f640065c0 con 0x7f8f64009110 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1296186628 0 0) 0x7f8f640065c0 con 0x7f8f64009110 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f64009e50 con 0x7f8f64009110 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8f78005cf0 con 0x7f8f64009110 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:31.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 10 monclient: dump: 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.828 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.829 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3275600559 0 0) 0x7f8f64009e50 con 0x7f8f64009110 2024-08-19T12:40:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8f6400c3a0 con 0x7f8f64009110 2024-08-19T12:40:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1719448303) 0x7f8f78008010 con 0x7f8f64009110 2024-08-19T12:40:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.26:6793/0 conn(0x7f8f64009110 legacy=0x7f8f640095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.824+0000 7f8f817fa640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.26:6793/0 conn(0x7f8f64009110 legacy=0x7f8f640095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:31.829 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:31.829 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 144841, 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:31.830 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:31.831 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:31.832 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:31.833 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:31.834 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.835 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.838 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:31.844 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.845 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.845 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.845 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:31.845 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:31.845 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:31.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:31.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:31.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:31.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:31.847 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:31.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:31.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:31.847 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:31.847 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:31.847 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.848 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:40:31.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.850 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:31.850 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 10 librados: watch_flush enter 2024-08-19T12:40:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 10 librados: watch_flush exit 2024-08-19T12:40:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 20 client.25957.objecter shutdown clearing up homeless session... 2024-08-19T12:40:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 10 client.25957.objecter successfully canceled tick 2024-08-19T12:40:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.104:6800/899887506 conn(0x7f8f680413f0 legacy=0x7f8f680438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 10 monclient: shutdown 2024-08-19T12:40:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 1 -- 172.21.15.26:0/914744235 >> v1:172.21.15.104:6789/0 conn(0x7f8f840a5920 legacy=0x7f8f84137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f894f0640 1 -- 172.21.15.26:0/914744235 reap_dead start 2024-08-19T12:40:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f894f0640 1 -- 172.21.15.26:0/914744235 reap_dead start 2024-08-19T12:40:31.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 1 -- 172.21.15.26:0/914744235 shutdown_connections 2024-08-19T12:40:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 1 -- 172.21.15.26:0/914744235 >> 172.21.15.26:0/914744235 conn(0x7f8f840a8740 msgr2=0x7f8f8406dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 1 -- 172.21.15.26:0/914744235 shutdown_connections 2024-08-19T12:40:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 1 -- 172.21.15.26:0/914744235 wait complete. 2024-08-19T12:40:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.825+0000 7f8f8af53640 1 librados: shutdown 2024-08-19T12:40:31.853 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:31.853 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:40:31.853 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:40:31.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.954+0000 7f01cb186640 10 monclient: get_monmap_and_config 2024-08-19T12:40:31.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.954+0000 7f01cb186640 10 monclient: build_initial_monmap 2024-08-19T12:40:31.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.954+0000 7f01cb186640 10 monclient: monmap: 2024-08-19T12:40:31.956 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:31.956 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:31.956473+0000 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:31.956473+0000 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:31.957 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:31.958 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 1 Processor -- start 2024-08-19T12:40:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 1 -- start start 2024-08-19T12:40:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 10 monclient: init 2024-08-19T12:40:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 10 monclient: _add_conns ranks=[2,3,4,6,1,7,0,8,5] 2024-08-19T12:40:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 10 monclient(hunting): picked mon.noname-b con 0x7f01c406e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 10 monclient(hunting): picked mon.noname-g con 0x7f01c40a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 10 monclient(hunting): picked mon.noname-c con 0x7f01c40a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c4136790 con 0x7f01c406e7b0 2024-08-19T12:40:31.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c409ddf0 con 0x7f01c40a5920 2024-08-19T12:40:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c409e0d0 con 0x7f01c40a5d50 2024-08-19T12:40:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01cb186640 10 monclient(hunting): authenticate will time out at 826.043904s 2024-08-19T12:40:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.955+0000 7f01c8f22640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f01c406e7b0 0x7f01c4136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58224/0 (socket says 172.21.15.26:58224) 2024-08-19T12:40:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.956+0000 7f01c8f22640 1 -- 172.21.15.26:0/1655079096 learned_addr learned my addr 172.21.15.26:0/1655079096 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:31.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01c9723640 1 -- 172.21.15.26:0/1655079096 >> v1:172.21.15.26:6791/0 conn(0x7f01c40a5d50 legacy=0x7f01c4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01c8f22640 1 -- 172.21.15.26:0/1655079096 >> v1:172.21.15.26:6790/0 conn(0x7f01c406e7b0 legacy=0x7f01c4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 1 -- 172.21.15.26:0/1655079096 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2490118761 0 0) 0x7f01c4136790 con 0x7f01c406e7b0 2024-08-19T12:40:31.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient(hunting): my global_id is 25346 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 1 -- 172.21.15.26:0/1655079096 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01ac003520 con 0x7f01c406e7b0 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 1 -- 172.21.15.26:0/1655079096 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2127200498 0 0) 0x7f01c409e0d0 con 0x7f01c40a5d50 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient(hunting): my global_id is 25600 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 1 -- 172.21.15.26:0/1655079096 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01c4136790 con 0x7f01c40a5d50 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bbfff640 1 -- 172.21.15.26:0/1655079096 >> v1:172.21.15.104:6790/0 conn(0x7f01c40a5920 legacy=0x7f01c413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 1 -- 172.21.15.26:0/1655079096 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2366906813 0 0) 0x7f01ac003520 con 0x7f01c406e7b0 2024-08-19T12:40:31.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 1 -- 172.21.15.26:0/1655079096 >> v1:172.21.15.26:6791/0 conn(0x7f01c40a5d50 legacy=0x7f01c4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 1 -- 172.21.15.26:0/1655079096 >> v1:172.21.15.104:6790/0 conn(0x7f01c40a5920 legacy=0x7f01c413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 1 monclient: found mon.noname-b 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 1 -- 172.21.15.26:0/1655079096 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c409e3b0 con 0x7f01c406e7b0 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient: _finish_auth 0 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 1 -- 172.21.15.26:0/1655079096 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f01c0001a10 con 0x7f01c406e7b0 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01cb186640 5 monclient: authenticate success, global_id 25346 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01cb186640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient: dump: 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.970 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01cb186640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 1 -- 172.21.15.26:0/1655079096 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01c0001ce0 con 0x7f01c406e7b0 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 1 -- 172.21.15.26:0/1655079096 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f01c0004be0 con 0x7f01c406e7b0 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01bb7fe640 10 monclient: dump: 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01cb186640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:31.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.958+0000 7f01cb186640 4 set_mon_vals no callback set 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.959+0000 7f01cb186640 10 monclient: shutdown 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.959+0000 7f01cb186640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.959+0000 7f01cb186640 1 -- 172.21.15.26:0/1655079096 >> v1:172.21.15.26:6790/0 conn(0x7f01c406e7b0 legacy=0x7f01c4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.959+0000 7f01cb186640 1 -- 172.21.15.26:0/1655079096 shutdown_connections 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.959+0000 7f01cb186640 1 -- 172.21.15.26:0/1655079096 >> 172.21.15.26:0/1655079096 conn(0x7f01c40a8740 msgr2=0x7f01c40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.959+0000 7f01cb186640 1 -- 172.21.15.26:0/1655079096 shutdown_connections 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.959+0000 7f01cb186640 1 -- 172.21.15.26:0/1655079096 wait complete. 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.959+0000 7f01cb186640 10 monclient: build_initial_monmap 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.959+0000 7f01cb186640 10 monclient: monmap: 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:31.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 1 librados: starting msgr at 2024-08-19T12:40:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 1 librados: starting objecter 2024-08-19T12:40:31.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 1 Processor -- start 2024-08-19T12:40:31.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 1 -- start start 2024-08-19T12:40:31.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 1 librados: setting wanted keys 2024-08-19T12:40:31.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 1 librados: calling monclient init 2024-08-19T12:40:31.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 10 monclient: init 2024-08-19T12:40:31.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:31.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 10 monclient: _add_conns ranks=[8,6,1,4,2,3,5,0,7] 2024-08-19T12:40:31.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 10 monclient(hunting): picked mon.noname-e con 0x7f01c406e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:31.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 10 monclient(hunting): picked mon.noname-d con 0x7f01c40a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:31.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 10 monclient(hunting): picked mon.noname-f con 0x7f01c40a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c4132cb0 con 0x7f01c40a5d50 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c4142ec0 con 0x7f01c40a5920 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c40a04b0 con 0x7f01c406e7b0 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01cb186640 10 monclient(hunting): authenticate will time out at 826.048908s 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01c8f22640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f01c406e7b0 0x7f01c4131e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50216/0 (socket says 172.21.15.26:50216) 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.960+0000 7f01c8f22640 1 -- 172.21.15.26:0/552471304 learned_addr learned my addr 172.21.15.26:0/552471304 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01bbfff640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.26:6792/0 conn(0x7f01c40a5920 legacy=0x7f01c41325a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 client.?.objecter ms_handle_connect 0x7f01c40a5920 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 client.?.objecter ms_handle_connect 0x7f01c406e7b0 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:31.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01c8f22640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.26:6793/0 conn(0x7f01c406e7b0 legacy=0x7f01c4131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1910550572 0 0) 0x7f01c4142ec0 con 0x7f01c40a5920 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): my global_id is 25611 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01c9723640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.104:6789/0 conn(0x7f01c40a5d50 legacy=0x7f01c41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01b0004eb0 con 0x7f01c40a5920 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 client.?.objecter ms_handle_connect 0x7f01c40a5d50 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 421406312 0 0) 0x7f01c40a04b0 con 0x7f01c406e7b0 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): my global_id is 25199 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01c4142ec0 con 0x7f01c406e7b0 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3384469995 0 0) 0x7f01c4132cb0 con 0x7f01c40a5d50 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): my global_id is 25966 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01c40a04b0 con 0x7f01c40a5d50 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1773420219 0 0) 0x7f01c4142ec0 con 0x7f01c406e7b0 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01c4132cb0 con 0x7f01c406e7b0 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f01c0004b20 con 0x7f01c406e7b0 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:40:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): dump: 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.979 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1064816341 0 0) 0x7f01b0004eb0 con 0x7f01c40a5920 2024-08-19T12:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01c4142ec0 con 0x7f01c40a5920 2024-08-19T12:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f01b4003950 con 0x7f01c40a5920 2024-08-19T12:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient(hunting): dump: 2024-08-19T12:40:31.980 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.981 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 549146098 0 0) 0x7f01c4132cb0 con 0x7f01c406e7b0 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.26:6792/0 conn(0x7f01c40a5920 legacy=0x7f01c41325a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.104:6789/0 conn(0x7f01c40a5d50 legacy=0x7f01c41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient: _finish_hunting 0 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 monclient: found mon.e 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c40a0790 con 0x7f01c406e7b0 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient: _finish_auth 0 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient: _check_auth_tickets 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01cb186640 5 monclient: authenticate success, global_id 25199 2024-08-19T12:40:31.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01cb186640 10 monclient: _renew_subs 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01cb186640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01cb186640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f01c40a56a0 con 0x7f01c406e7b0 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01cb186640 10 client.25199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01cb186640 10 monclient: _renew_subs 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01cb186640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01cb186640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f01c409f550 con 0x7f01c406e7b0 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01cb186640 1 librados: init done 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01c00019f0 con 0x7f01c406e7b0 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f01c00014b0 con 0x7f01c406e7b0 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01baffd640 4 set_mon_vals no callback set 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:31.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.963+0000 7f01b97fa640 10 monclient: dump: 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.984 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.964+0000 7f01cb186640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.964+0000 7f01cb186640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f01a40065c0 con 0x7f01a4002860 2024-08-19T12:40:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.964+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f01c00117d0 con 0x7f01c406e7b0 2024-08-19T12:40:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.964+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(75..75 src has 1..75) ==== 7121+0+0 (unknown 2329720755 0 0) 0x7f01c004e8d0 con 0x7f01c406e7b0 2024-08-19T12:40:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.964+0000 7f01b97fa640 10 client.25199.objecter ms_dispatch 0x7f01c4000b90 osd_map(75..75 src has 1..75) 2024-08-19T12:40:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.964+0000 7f01b97fa640 3 client.25199.objecter handle_osd_map got epochs [75,75] > 0 2024-08-19T12:40:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.964+0000 7f01b97fa640 3 client.25199.objecter handle_osd_map decoding full epoch 75 2024-08-19T12:40:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.965+0000 7f01b97fa640 20 client.25199.objecter dump_active .. 0 homeless 2024-08-19T12:40:31.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.965+0000 7f01c9723640 10 monclient: get_auth_request con 0x7f01b00414c0 auth_method 0 2024-08-19T12:40:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 10 client.25199.objecter ms_handle_connect 0x7f01a4002860 2024-08-19T12:40:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 10 client.25199.objecter resend_mon_ops 2024-08-19T12:40:31.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01bbfff640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.104:6789/0 conn(0x7f01a4002860 legacy=0x7f01a4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3431216034 0 0) 0x7f01a40065c0 con 0x7f01a4002860 2024-08-19T12:40:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 10 monclient(hunting): my global_id is 25975 2024-08-19T12:40:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01c4132cb0 con 0x7f01a4002860 2024-08-19T12:40:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 10 client.25199.objecter ms_handle_connect 0x7f01b00414c0 2024-08-19T12:40:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 74550294 0 0) 0x7f01c4132cb0 con 0x7f01a4002860 2024-08-19T12:40:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01a40065c0 con 0x7f01a4002860 2024-08-19T12:40:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f01b4003e30 con 0x7f01a4002860 2024-08-19T12:40:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 10 monclient: dump: 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.991 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2036342514 0 0) 0x7f01a40065c0 con 0x7f01a4002860 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.967+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01a400a160 con 0x7f01a4002860 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.968+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f01b4003fb0 con 0x7f01a4002860 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.968+0000 7f01b97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.968+0000 7f01b97fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.968+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.104:6789/0 conn(0x7f01a4002860 legacy=0x7f01a4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.968+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.104:6789/0 conn(0x7f01a4002860 legacy=0x7f01a4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.974+0000 7f01cb186640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.974+0000 7f01cb186640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f01a4009e70 con 0x7f01a4009130 2024-08-19T12:40:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.976+0000 7f01bbfff640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.104:6789/0 conn(0x7f01a4009130 legacy=0x7f01a40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.976+0000 7f01b97fa640 10 client.25199.objecter ms_handle_connect 0x7f01a4009130 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.976+0000 7f01b97fa640 10 client.25199.objecter resend_mon_ops 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.976+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2616792746 0 0) 0x7f01a4009e70 con 0x7f01a4009130 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.976+0000 7f01b97fa640 10 monclient(hunting): _negotiate 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.976+0000 7f01b97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.976+0000 7f01b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.976+0000 7f01b97fa640 10 monclient(hunting): my global_id is 25984 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.976+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01a40065c0 con 0x7f01a4009130 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1759009322 0 0) 0x7f01a40065c0 con 0x7f01a4009130 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01a4009e70 con 0x7f01a4009130 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f01b4005b00 con 0x7f01a4009130 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:31.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 10 monclient: dump: 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:31.994 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:31.995 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3388806295 0 0) 0x7f01a4009e70 con 0x7f01a4009130 2024-08-19T12:40:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f01a400c3f0 con 0x7f01a4009130 2024-08-19T12:40:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 50560301) 0x7f01b4006f50 con 0x7f01a4009130 2024-08-19T12:40:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.104:6789/0 conn(0x7f01a4009130 legacy=0x7f01a40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:31.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01b97fa640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.104:6789/0 conn(0x7f01a4009130 legacy=0x7f01a40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:31.995 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 144982, 2024-08-19T12:40:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:31.996 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:31.997 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:31.998 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:31.999 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:32.000 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:32.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.004 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:32.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.012 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.015 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.016 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.017 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.018 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:32.018 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:32.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01cb186640 10 librados: watch_flush enter 2024-08-19T12:40:32.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01cb186640 10 librados: watch_flush exit 2024-08-19T12:40:32.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01cb186640 20 client.25199.objecter shutdown clearing up homeless session... 2024-08-19T12:40:32.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01cb186640 10 client.25199.objecter successfully canceled tick 2024-08-19T12:40:32.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01cb186640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.104:6800/899887506 conn(0x7f01b00414c0 legacy=0x7f01b0043980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01cb186640 10 monclient: shutdown 2024-08-19T12:40:32.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01cb186640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:32.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.977+0000 7f01cb186640 1 -- 172.21.15.26:0/552471304 >> v1:172.21.15.26:6793/0 conn(0x7f01c406e7b0 legacy=0x7f01c4131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.978+0000 7f01c9723640 1 -- 172.21.15.26:0/552471304 reap_dead start 2024-08-19T12:40:32.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.978+0000 7f01c9723640 1 -- 172.21.15.26:0/552471304 reap_dead start 2024-08-19T12:40:32.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.978+0000 7f01cb186640 1 -- 172.21.15.26:0/552471304 shutdown_connections 2024-08-19T12:40:32.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.978+0000 7f01cb186640 1 -- 172.21.15.26:0/552471304 >> 172.21.15.26:0/552471304 conn(0x7f01c40a8740 msgr2=0x7f01c406d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:32.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.978+0000 7f01cb186640 1 -- 172.21.15.26:0/552471304 shutdown_connections 2024-08-19T12:40:32.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.978+0000 7f01cb186640 1 -- 172.21.15.26:0/552471304 wait complete. 2024-08-19T12:40:32.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:31.978+0000 7f01cb186640 1 librados: shutdown 2024-08-19T12:40:32.021 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:32.021 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:40:32.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:40:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.120+0000 7f346e6da640 10 monclient: get_monmap_and_config 2024-08-19T12:40:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.120+0000 7f346e6da640 10 monclient: build_initial_monmap 2024-08-19T12:40:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.120+0000 7f346e6da640 10 monclient: monmap: 2024-08-19T12:40:32.122 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:32.122503+0000 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:32.122503+0000 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:32.124 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 1 Processor -- start 2024-08-19T12:40:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 1 -- start start 2024-08-19T12:40:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 10 monclient: init 2024-08-19T12:40:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 10 monclient: _add_conns ranks=[1,7,5,0,4,8,6,3,2] 2024-08-19T12:40:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 10 monclient(hunting): picked mon.noname-f con 0x7f3468073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 10 monclient(hunting): picked mon.noname-i con 0x7f34680a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 10 monclient(hunting): picked mon.noname-h con 0x7f346813f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f346809dae0 con 0x7f3468073470 2024-08-19T12:40:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3468136910 con 0x7f346813f2e0 2024-08-19T12:40:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f346809dd50 con 0x7f34680a57a0 2024-08-19T12:40:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.121+0000 7f346e6da640 10 monclient(hunting): authenticate will time out at 826.209930s 2024-08-19T12:40:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.122+0000 7f3467fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3468073470 0x7f3468136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:52366/0 (socket says 172.21.15.26:52366) 2024-08-19T12:40:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.122+0000 7f34677fe640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f34680a57a0 0x7f34680a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:51298/0 (socket says 172.21.15.26:51298) 2024-08-19T12:40:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.122+0000 7f3467fff640 1 -- 172.21.15.26:0/2134190415 learned_addr learned my addr 172.21.15.26:0/2134190415 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f34677fe640 1 -- 172.21.15.26:0/2134190415 >> v1:172.21.15.104:6792/0 conn(0x7f34680a57a0 legacy=0x7f34680a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f346cc77640 1 -- 172.21.15.26:0/2134190415 >> v1:172.21.15.104:6791/0 conn(0x7f346813f2e0 legacy=0x7f34681416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3467fff640 1 -- 172.21.15.26:0/2134190415 >> v1:172.21.15.104:6789/0 conn(0x7f3468073470 legacy=0x7f3468136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2371048730 0 0) 0x7f346809dd50 con 0x7f34680a57a0 2024-08-19T12:40:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 10 monclient(hunting): my global_id is 134125 2024-08-19T12:40:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3450003520 con 0x7f34680a57a0 2024-08-19T12:40:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560987555 0 0) 0x7f346809dae0 con 0x7f3468073470 2024-08-19T12:40:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 10 monclient(hunting): my global_id is 25993 2024-08-19T12:40:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f346809dd50 con 0x7f3468073470 2024-08-19T12:40:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3253807148 0 0) 0x7f3468136910 con 0x7f346813f2e0 2024-08-19T12:40:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 10 monclient(hunting): my global_id is 25358 2024-08-19T12:40:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.124+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f346809dae0 con 0x7f346813f2e0 2024-08-19T12:40:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4074341702 0 0) 0x7f3450003520 con 0x7f34680a57a0 2024-08-19T12:40:32.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 >> v1:172.21.15.104:6791/0 conn(0x7f346813f2e0 legacy=0x7f34681416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 >> v1:172.21.15.104:6789/0 conn(0x7f3468073470 legacy=0x7f3468136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:40:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 1 monclient: found mon.noname-i 2024-08-19T12:40:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:40:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f346809e030 con 0x7f34680a57a0 2024-08-19T12:40:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 10 monclient: _finish_auth 0 2024-08-19T12:40:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:40:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f345c001ae0 con 0x7f34680a57a0 2024-08-19T12:40:32.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f346e6da640 5 monclient: authenticate success, global_id 134125 2024-08-19T12:40:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f346e6da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:40:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 10 monclient: dump: 2024-08-19T12:40:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.134 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f346e6da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f345c0047e0 con 0x7f34680a57a0 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f346e6da640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 1 -- 172.21.15.26:0/2134190415 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f345c004d90 con 0x7f34680a57a0 2024-08-19T12:40:32.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f346e6da640 4 set_mon_vals no callback set 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f3466ffd640 10 monclient: dump: 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.136 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f346e6da640 10 monclient: shutdown 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f346e6da640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f346e6da640 1 -- 172.21.15.26:0/2134190415 >> v1:172.21.15.104:6792/0 conn(0x7f34680a57a0 legacy=0x7f34680a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f346e6da640 1 -- 172.21.15.26:0/2134190415 shutdown_connections 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f346e6da640 1 -- 172.21.15.26:0/2134190415 >> 172.21.15.26:0/2134190415 conn(0x7f34680abff0 msgr2=0x7f34680ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f346e6da640 1 -- 172.21.15.26:0/2134190415 shutdown_connections 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.125+0000 7f346e6da640 1 -- 172.21.15.26:0/2134190415 wait complete. 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.126+0000 7f346e6da640 10 monclient: build_initial_monmap 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.126+0000 7f346e6da640 10 monclient: monmap: 2024-08-19T12:40:32.137 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:32.138 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.126+0000 7f346e6da640 1 librados: starting msgr at 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.126+0000 7f346e6da640 1 librados: starting objecter 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.126+0000 7f346e6da640 1 Processor -- start 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.126+0000 7f346e6da640 1 -- start start 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.126+0000 7f346e6da640 1 librados: setting wanted keys 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.126+0000 7f346e6da640 1 librados: calling monclient init 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.126+0000 7f346e6da640 10 monclient: init 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.127+0000 7f346e6da640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.127+0000 7f346e6da640 10 monclient: _add_conns ranks=[1,5,0,6,3,4,7,2,8] 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.127+0000 7f346e6da640 10 monclient(hunting): picked mon.noname-f con 0x7f3468073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.127+0000 7f346e6da640 10 monclient(hunting): picked mon.noname-h con 0x7f34680a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.127+0000 7f346e6da640 10 monclient(hunting): picked mon.noname-a con 0x7f346813f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:32.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.127+0000 7f346e6da640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34680a3830 con 0x7f346813f2e0 2024-08-19T12:40:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.127+0000 7f346e6da640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3468142b80 con 0x7f3468073470 2024-08-19T12:40:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.127+0000 7f346e6da640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3468137400 con 0x7f34680a57a0 2024-08-19T12:40:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.127+0000 7f346e6da640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.127+0000 7f346e6da640 10 monclient(hunting): authenticate will time out at 826.215314s 2024-08-19T12:40:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.127+0000 7f346cc77640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f346813f2e0 0x7f34680a2bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59844/0 (socket says 172.21.15.26:59844) 2024-08-19T12:40:32.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.127+0000 7f346cc77640 1 -- 172.21.15.26:0/3474617250 learned_addr learned my addr 172.21.15.26:0/3474617250 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f346cc77640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.26:6789/0 conn(0x7f346813f2e0 legacy=0x7f34680a2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3467fff640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.104:6789/0 conn(0x7f3468073470 legacy=0x7f34680a1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 client.?.objecter ms_handle_connect 0x7f346813f2e0 2024-08-19T12:40:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 client.?.objecter ms_handle_connect 0x7f3468073470 2024-08-19T12:40:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 client.?.objecter ms_handle_connect 0x7f34680a57a0 2024-08-19T12:40:32.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f34677fe640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.104:6791/0 conn(0x7f34680a57a0 legacy=0x7f34680a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1334790847 0 0) 0x7f34680a3830 con 0x7f346813f2e0 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 monclient(hunting): my global_id is 26883 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3454004eb0 con 0x7f346813f2e0 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1400443837 0 0) 0x7f3468142b80 con 0x7f3468073470 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 monclient(hunting): my global_id is 26002 2024-08-19T12:40:32.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34680a3830 con 0x7f3468073470 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 381424525 0 0) 0x7f3468137400 con 0x7f34680a57a0 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 10 monclient(hunting): my global_id is 25367 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.129+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3468142b80 con 0x7f34680a57a0 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 365214492 0 0) 0x7f3454004eb0 con 0x7f346813f2e0 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3468137400 con 0x7f346813f2e0 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3460001ff0 con 0x7f346813f2e0 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient(hunting): dump: 2024-08-19T12:40:32.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.149 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1012655460 0 0) 0x7f34680a3830 con 0x7f3468073470 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3454004eb0 con 0x7f3468073470 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3048414727 0 0) 0x7f3468142b80 con 0x7f34680a57a0 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34680a3830 con 0x7f34680a57a0 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f34580047e0 con 0x7f3468073470 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient(hunting): dump: 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.150 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f345c001a60 con 0x7f34680a57a0 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient(hunting): dump: 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.152 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3305602208 0 0) 0x7f3468137400 con 0x7f346813f2e0 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.104:6791/0 conn(0x7f34680a57a0 legacy=0x7f34680a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.104:6789/0 conn(0x7f3468073470 legacy=0x7f34680a1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 monclient: found mon.a 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34680a4390 con 0x7f346813f2e0 2024-08-19T12:40:32.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient: _finish_auth 0 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f346e6da640 5 monclient: authenticate success, global_id 26883 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f346e6da640 10 monclient: _renew_subs 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f346e6da640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f346e6da640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f34680a4620 con 0x7f346813f2e0 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f346e6da640 10 client.26883.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f346e6da640 10 monclient: _renew_subs 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f346e6da640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f346e6da640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f34681d7340 con 0x7f346813f2e0 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f346e6da640 1 librados: init done 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3460003c40 con 0x7f346813f2e0 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f34600041b0 con 0x7f346813f2e0 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f34667fc640 4 set_mon_vals no callback set 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.130+0000 7f3464ff9640 10 monclient: dump: 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.131+0000 7f346e6da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.131+0000 7f346e6da640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f34480065c0 con 0x7f3448002860 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.131+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3460004350 con 0x7f346813f2e0 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.131+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(75..75 src has 1..75) ==== 7121+0+0 (unknown 2329720755 0 0) 0x7f346004ea50 con 0x7f346813f2e0 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.131+0000 7f3464ff9640 10 client.26883.objecter ms_dispatch 0x7f3468000b90 osd_map(75..75 src has 1..75) 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.131+0000 7f3464ff9640 3 client.26883.objecter handle_osd_map got epochs [75,75] > 0 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.131+0000 7f3464ff9640 3 client.26883.objecter handle_osd_map decoding full epoch 75 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.131+0000 7f3464ff9640 20 client.26883.objecter dump_active .. 0 homeless 2024-08-19T12:40:32.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.131+0000 7f34677fe640 10 monclient: get_auth_request con 0x7f3454041ab0 auth_method 0 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3467fff640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.104:6790/0 conn(0x7f3448002860 legacy=0x7f3448004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 10 client.26883.objecter ms_handle_connect 0x7f3448002860 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 10 client.26883.objecter resend_mon_ops 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1847638045 0 0) 0x7f34480065c0 con 0x7f3448002860 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 10 monclient(hunting): my global_id is 25500 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3468137400 con 0x7f3448002860 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2268396860 0 0) 0x7f3468137400 con 0x7f3448002860 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34480065c0 con 0x7f3448002860 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3458004eb0 con 0x7f3448002860 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:40:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 10 monclient: dump: 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.158 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 413430603 0 0) 0x7f34480065c0 con 0x7f3448002860 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.133+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f344800a160 con 0x7f3448002860 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.134+0000 7f3464ff9640 10 client.26883.objecter ms_handle_connect 0x7f3454041ab0 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.134+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3458006410 con 0x7f3448002860 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.134+0000 7f3464ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.134+0000 7f3464ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.134+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.104:6790/0 conn(0x7f3448002860 legacy=0x7f3448004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.134+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.104:6790/0 conn(0x7f3448002860 legacy=0x7f3448004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.140+0000 7f346e6da640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.140+0000 7f346e6da640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3448009e50 con 0x7f3448009110 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3467fff640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.104:6790/0 conn(0x7f3448009110 legacy=0x7f34480095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 10 client.26883.objecter ms_handle_connect 0x7f3448009110 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 10 client.26883.objecter resend_mon_ops 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832666041 0 0) 0x7f3448009e50 con 0x7f3448009110 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 10 monclient(hunting): my global_id is 25509 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34480065c0 con 0x7f3448009110 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1626929894 0 0) 0x7f34480065c0 con 0x7f3448009110 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3448009e50 con 0x7f3448009110 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3458005460 con 0x7f3448009110 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 10 monclient: dump: 2024-08-19T12:40:32.160 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.161 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3852294486 0 0) 0x7f3448009e50 con 0x7f3448009110 2024-08-19T12:40:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.143+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f344800c3a0 con 0x7f3448009110 2024-08-19T12:40:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1602394861) 0x7f3458005890 con 0x7f3448009110 2024-08-19T12:40:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f3464ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f3464ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.104:6790/0 conn(0x7f3448009110 legacy=0x7f34480095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f3464ff9640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.104:6790/0 conn(0x7f3448009110 legacy=0x7f34480095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:32.162 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:32.162 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 145157, 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:32.163 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:32.164 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:32.165 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:32.166 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:32.167 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.169 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.170 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.171 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:32.173 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.174 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:32.177 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.177 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.177 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:32.177 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:32.177 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.177 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.177 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:32.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:32.180 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:32.181 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:40:32.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.182 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:32.182 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 10 librados: watch_flush enter 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 10 librados: watch_flush exit 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 20 client.26883.objecter shutdown clearing up homeless session... 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 10 client.26883.objecter successfully canceled tick 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.104:6800/899887506 conn(0x7f3454041ab0 legacy=0x7f3454043f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 10 monclient: shutdown 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 1 -- 172.21.15.26:0/3474617250 >> v1:172.21.15.26:6789/0 conn(0x7f346813f2e0 legacy=0x7f34680a2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346cc77640 1 -- 172.21.15.26:0/3474617250 reap_dead start 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346cc77640 1 -- 172.21.15.26:0/3474617250 reap_dead start 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 1 -- 172.21.15.26:0/3474617250 shutdown_connections 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 1 -- 172.21.15.26:0/3474617250 >> 172.21.15.26:0/3474617250 conn(0x7f34680abff0 msgr2=0x7f34681338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 1 -- 172.21.15.26:0/3474617250 shutdown_connections 2024-08-19T12:40:32.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 1 -- 172.21.15.26:0/3474617250 wait complete. 2024-08-19T12:40:32.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.144+0000 7f346e6da640 1 librados: shutdown 2024-08-19T12:40:32.185 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:32.185 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:40:32.185 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.283+0000 7f8d1b9c9640 10 monclient: get_monmap_and_config 2024-08-19T12:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.283+0000 7f8d1b9c9640 10 monclient: build_initial_monmap 2024-08-19T12:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.283+0000 7f8d1b9c9640 10 monclient: monmap: 2024-08-19T12:40:32.285 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:32.285623+0000 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:32.285623+0000 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:32.286 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:32.287 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:32.287 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.284+0000 7f8d1b9c9640 1 Processor -- start 2024-08-19T12:40:32.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.284+0000 7f8d1b9c9640 1 -- start start 2024-08-19T12:40:32.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.284+0000 7f8d1b9c9640 10 monclient: init 2024-08-19T12:40:32.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.285+0000 7f8d1b9c9640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:32.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.285+0000 7f8d1b9c9640 10 monclient: _add_conns ranks=[5,7,1,3,2,4,0,6,8] 2024-08-19T12:40:32.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.285+0000 7f8d1b9c9640 10 monclient(hunting): picked mon.noname-h con 0x7f8d1406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:32.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.285+0000 7f8d1b9c9640 10 monclient(hunting): picked mon.noname-i con 0x7f8d140a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:32.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.285+0000 7f8d1b9c9640 10 monclient(hunting): picked mon.noname-f con 0x7f8d140a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:32.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.285+0000 7f8d1b9c9640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d14136790 con 0x7f8d140a5d50 2024-08-19T12:40:32.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.285+0000 7f8d1b9c9640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d1409ddf0 con 0x7f8d1406e7b0 2024-08-19T12:40:32.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.285+0000 7f8d1b9c9640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d1409e0d0 con 0x7f8d140a5920 2024-08-19T12:40:32.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.285+0000 7f8d1b9c9640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:32.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.285+0000 7f8d1b9c9640 10 monclient(hunting): authenticate will time out at 826.373130s 2024-08-19T12:40:32.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.285+0000 7f8d18f64640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f8d140a5920 0x7f8d1413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:51310/0 (socket says 172.21.15.26:51310) 2024-08-19T12:40:32.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.285+0000 7f8d18f64640 1 -- 172.21.15.26:0/3287291729 learned_addr learned my addr 172.21.15.26:0/3287291729 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.287+0000 7f8d19765640 1 -- 172.21.15.26:0/3287291729 >> v1:172.21.15.104:6791/0 conn(0x7f8d1406e7b0 legacy=0x7f8d14136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.287+0000 7f8d18f64640 1 -- 172.21.15.26:0/3287291729 >> v1:172.21.15.104:6792/0 conn(0x7f8d140a5920 legacy=0x7f8d1413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.287+0000 7f8d19f66640 1 -- 172.21.15.26:0/3287291729 >> v1:172.21.15.104:6789/0 conn(0x7f8d140a5d50 legacy=0x7f8d14141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.287+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 221933005 0 0) 0x7f8d1409e0d0 con 0x7f8d140a5920 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.287+0000 7f8d13fff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.287+0000 7f8d13fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.287+0000 7f8d13fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.287+0000 7f8d13fff640 10 monclient(hunting): my global_id is 134134 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cfc003520 con 0x7f8d140a5920 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2921963392 0 0) 0x7f8d04001820 con 0x7f8d1406e7b0 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient(hunting): my global_id is 25376 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d1409e0d0 con 0x7f8d1406e7b0 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 256092423 0 0) 0x7f8d14136790 con 0x7f8d140a5d50 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient(hunting): my global_id is 26011 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d04001820 con 0x7f8d140a5d50 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2165177200 0 0) 0x7f8cfc003520 con 0x7f8d140a5920 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 >> v1:172.21.15.104:6791/0 conn(0x7f8d1406e7b0 legacy=0x7f8d14136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 >> v1:172.21.15.104:6789/0 conn(0x7f8d140a5d50 legacy=0x7f8d14141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient: _finish_hunting 0 2024-08-19T12:40:32.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 monclient: found mon.noname-i 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1409e3b0 con 0x7f8d140a5920 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient: _finish_auth 0 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient: _check_auth_tickets 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8d08001a10 con 0x7f8d140a5920 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient: dump: 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.295 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d1b9c9640 5 monclient: authenticate success, global_id 134134 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d1b9c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:32.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8d08001ce0 con 0x7f8d140a5920 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 1 -- 172.21.15.26:0/3287291729 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8d08004c20 con 0x7f8d140a5920 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d1b9c9640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d1b9c9640 4 set_mon_vals no callback set 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d13fff640 10 monclient: dump: 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.297 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d1b9c9640 10 monclient: shutdown 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d1b9c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/3287291729 >> v1:172.21.15.104:6792/0 conn(0x7f8d140a5920 legacy=0x7f8d1413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/3287291729 shutdown_connections 2024-08-19T12:40:32.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.288+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/3287291729 >> 172.21.15.26:0/3287291729 conn(0x7f8d140a8740 msgr2=0x7f8d140a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.289+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/3287291729 shutdown_connections 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.289+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/3287291729 wait complete. 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.289+0000 7f8d1b9c9640 10 monclient: build_initial_monmap 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.289+0000 7f8d1b9c9640 10 monclient: monmap: 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:32.299 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:32.300 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:32.300 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:32.300 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:32.300 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:32.300 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.289+0000 7f8d1b9c9640 1 librados: starting msgr at 2024-08-19T12:40:32.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.289+0000 7f8d1b9c9640 1 librados: starting objecter 2024-08-19T12:40:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.289+0000 7f8d1b9c9640 1 Processor -- start 2024-08-19T12:40:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.289+0000 7f8d1b9c9640 1 -- start start 2024-08-19T12:40:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.289+0000 7f8d1b9c9640 1 librados: setting wanted keys 2024-08-19T12:40:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.289+0000 7f8d1b9c9640 1 librados: calling monclient init 2024-08-19T12:40:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.289+0000 7f8d1b9c9640 10 monclient: init 2024-08-19T12:40:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.290+0000 7f8d1b9c9640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.290+0000 7f8d1b9c9640 10 monclient: _add_conns ranks=[4,3,5,7,8,1,0,6,2] 2024-08-19T12:40:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.290+0000 7f8d1b9c9640 10 monclient(hunting): picked mon.noname-c con 0x7f8d1406e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.290+0000 7f8d1b9c9640 10 monclient(hunting): picked mon.noname-g con 0x7f8d140a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.290+0000 7f8d1b9c9640 10 monclient(hunting): picked mon.noname-h con 0x7f8d140a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.290+0000 7f8d1b9c9640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d140a3020 con 0x7f8d140a5920 2024-08-19T12:40:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.290+0000 7f8d1b9c9640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d140a3300 con 0x7f8d1406e7b0 2024-08-19T12:40:32.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.290+0000 7f8d1b9c9640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d14134a60 con 0x7f8d140a5d50 2024-08-19T12:40:32.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.290+0000 7f8d1b9c9640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:32.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.290+0000 7f8d1b9c9640 10 monclient(hunting): authenticate will time out at 826.378434s 2024-08-19T12:40:32.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.290+0000 7f8d19765640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f8d1406e7b0 0x7f8d140a4830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:39122/0 (socket says 172.21.15.26:39122) 2024-08-19T12:40:32.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.290+0000 7f8d19765640 1 -- 172.21.15.26:0/1087659499 learned_addr learned my addr 172.21.15.26:0/1087659499 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:32.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d19765640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.26:6791/0 conn(0x7f8d1406e7b0 legacy=0x7f8d140a4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d11ffb640 10 client.?.objecter ms_handle_connect 0x7f8d1406e7b0 2024-08-19T12:40:32.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d11ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:32.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1242041795 0 0) 0x7f8d140a3300 con 0x7f8d1406e7b0 2024-08-19T12:40:32.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d11ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d11ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d11ffb640 10 monclient(hunting): my global_id is 25609 2024-08-19T12:40:32.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d00004eb0 con 0x7f8d1406e7b0 2024-08-19T12:40:32.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d11ffb640 10 client.?.objecter ms_handle_connect 0x7f8d140a5d50 2024-08-19T12:40:32.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d11ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:32.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d11ffb640 10 client.?.objecter ms_handle_connect 0x7f8d140a5920 2024-08-19T12:40:32.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d11ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:32.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d19f66640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.104:6791/0 conn(0x7f8d140a5d50 legacy=0x7f8d141d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.292+0000 7f8d18f64640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.104:6790/0 conn(0x7f8d140a5920 legacy=0x7f8d141d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 911657263 0 0) 0x7f8d14134a60 con 0x7f8d140a5d50 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient(hunting): my global_id is 25385 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d140a3300 con 0x7f8d140a5d50 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2488487841 0 0) 0x7f8d140a3020 con 0x7f8d140a5920 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient(hunting): my global_id is 25518 2024-08-19T12:40:32.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d14134a60 con 0x7f8d140a5920 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2862686464 0 0) 0x7f8d00004eb0 con 0x7f8d1406e7b0 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d140a3020 con 0x7f8d1406e7b0 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8d04003cd0 con 0x7f8d1406e7b0 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient(hunting): dump: 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 525738229 0 0) 0x7f8d140a3020 con 0x7f8d1406e7b0 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.104:6791/0 conn(0x7f8d140a5d50 legacy=0x7f8d141d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.104:6790/0 conn(0x7f8d140a5920 legacy=0x7f8d141d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:40:32.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 monclient: found mon.c 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d14134630 con 0x7f8d1406e7b0 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient: _finish_auth 0 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d1b9c9640 5 monclient: authenticate success, global_id 25609 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d1b9c9640 10 monclient: _renew_subs 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d1b9c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8d140a1fc0 con 0x7f8d1406e7b0 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8d04003fa0 con 0x7f8d1406e7b0 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8d04004510 con 0x7f8d1406e7b0 2024-08-19T12:40:32.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d11ffb640 10 monclient: dump: 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.312 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d1b9c9640 10 client.25609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d1b9c9640 10 monclient: _renew_subs 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d1b9c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f8d140a52a0 con 0x7f8d1406e7b0 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d12ffd640 4 set_mon_vals no callback set 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.293+0000 7f8d1b9c9640 1 librados: init done 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.294+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f8d04011e00 con 0x7f8d1406e7b0 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.294+0000 7f8d1b9c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.294+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8d140a1bb0 con 0x7f8d1402e340 2024-08-19T12:40:32.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.294+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(75..75 src has 1..75) ==== 7121+0+0 (unknown 2329720755 0 0) 0x7f8d0404ecd0 con 0x7f8d1406e7b0 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.294+0000 7f8d11ffb640 10 client.25609.objecter ms_dispatch 0x7f8d14000b90 osd_map(75..75 src has 1..75) 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.294+0000 7f8d11ffb640 3 client.25609.objecter handle_osd_map got epochs [75,75] > 0 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.294+0000 7f8d11ffb640 3 client.25609.objecter handle_osd_map decoding full epoch 75 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.294+0000 7f8d19f66640 10 monclient: get_auth_request con 0x7f8d00041240 auth_method 0 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.294+0000 7f8d11ffb640 20 client.25609.objecter dump_active .. 0 homeless 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.296+0000 7f8d11ffb640 10 client.25609.objecter ms_handle_connect 0x7f8d1402e340 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.296+0000 7f8d18f64640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.104:6791/0 conn(0x7f8d1402e340 legacy=0x7f8d14063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.296+0000 7f8d11ffb640 10 client.25609.objecter resend_mon_ops 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2359532559 0 0) 0x7f8d14134a60 con 0x7f8d1402e340 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 10 monclient(hunting): my global_id is 25394 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d140a3020 con 0x7f8d1402e340 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 10 client.25609.objecter ms_handle_connect 0x7f8d00041240 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1536599770 0 0) 0x7f8d140a3020 con 0x7f8d1402e340 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d14134a60 con 0x7f8d1402e340 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8d080017f0 con 0x7f8d1402e340 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 10 monclient: dump: 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.315 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 865222199 0 0) 0x7f8d14134a60 con 0x7f8d1402e340 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d1402e720 con 0x7f8d1402e340 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8d08001a10 con 0x7f8d1402e340 2024-08-19T12:40:32.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:32.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:32.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.104:6791/0 conn(0x7f8d1402e340 legacy=0x7f8d14063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.297+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.104:6791/0 conn(0x7f8d1402e340 legacy=0x7f8d14063a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:32.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.303+0000 7f8d1b9c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:32.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.303+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8d141259c0 con 0x7f8d141327b0 2024-08-19T12:40:32.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.306+0000 7f8d11ffb640 10 client.25609.objecter ms_handle_connect 0x7f8d141327b0 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.306+0000 7f8d11ffb640 10 client.25609.objecter resend_mon_ops 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.306+0000 7f8d18f64640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.104:6791/0 conn(0x7f8d141327b0 legacy=0x7f8d141252b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.306+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 655893380 0 0) 0x7f8d141259c0 con 0x7f8d141327b0 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.306+0000 7f8d11ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.306+0000 7f8d11ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.306+0000 7f8d11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.306+0000 7f8d11ffb640 10 monclient(hunting): my global_id is 25403 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.306+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d14134a60 con 0x7f8d141327b0 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2315622068 0 0) 0x7f8d14134a60 con 0x7f8d141327b0 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d141259c0 con 0x7f8d141327b0 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8d08005340 con 0x7f8d141327b0 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 10 monclient: dump: 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.320 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.321 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 577658982 0 0) 0x7f8d141259c0 con 0x7f8d141327b0 2024-08-19T12:40:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8d14132b90 con 0x7f8d141327b0 2024-08-19T12:40:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1427558205) 0x7f8d08007520 con 0x7f8d141327b0 2024-08-19T12:40:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.104:6791/0 conn(0x7f8d141327b0 legacy=0x7f8d141252b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d11ffb640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.104:6791/0 conn(0x7f8d141327b0 legacy=0x7f8d141252b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:32.321 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 145331, 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:32.322 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:32.323 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:32.324 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:32.325 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:32.326 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.327 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:32.328 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.329 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.330 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:32.331 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:32.333 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.336 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.337 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.338 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:32.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.340 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.340 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:32.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.340 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:32.340 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:32.340 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-08-19T12:40:32.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.340 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.340 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:32.340 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:32.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d1b9c9640 10 librados: watch_flush enter 2024-08-19T12:40:32.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d1b9c9640 10 librados: watch_flush exit 2024-08-19T12:40:32.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d1b9c9640 20 client.25609.objecter shutdown clearing up homeless session... 2024-08-19T12:40:32.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.307+0000 7f8d1b9c9640 10 client.25609.objecter successfully canceled tick 2024-08-19T12:40:32.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.308+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.104:6800/899887506 conn(0x7f8d00041240 legacy=0x7f8d00043700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.308+0000 7f8d1b9c9640 10 monclient: shutdown 2024-08-19T12:40:32.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.308+0000 7f8d1b9c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:32.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.308+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/1087659499 >> v1:172.21.15.26:6791/0 conn(0x7f8d1406e7b0 legacy=0x7f8d140a4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.308+0000 7f8d19f66640 1 -- 172.21.15.26:0/1087659499 reap_dead start 2024-08-19T12:40:32.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.308+0000 7f8d19f66640 1 -- 172.21.15.26:0/1087659499 reap_dead start 2024-08-19T12:40:32.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.308+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/1087659499 shutdown_connections 2024-08-19T12:40:32.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.308+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/1087659499 >> 172.21.15.26:0/1087659499 conn(0x7f8d140a8740 msgr2=0x7f8d14135110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:32.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.308+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/1087659499 shutdown_connections 2024-08-19T12:40:32.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.308+0000 7f8d1b9c9640 1 -- 172.21.15.26:0/1087659499 wait complete. 2024-08-19T12:40:32.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.308+0000 7f8d1b9c9640 1 librados: shutdown 2024-08-19T12:40:32.343 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:32.344 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:40:32.344 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:40:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.442+0000 7f83894e5640 10 monclient: get_monmap_and_config 2024-08-19T12:40:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.442+0000 7f83894e5640 10 monclient: build_initial_monmap 2024-08-19T12:40:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.442+0000 7f83894e5640 10 monclient: monmap: 2024-08-19T12:40:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:32.443951+0000 2024-08-19T12:40:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:32.443951+0000 2024-08-19T12:40:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.444 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:32.445 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:32.445 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.442+0000 7f83894e5640 1 Processor -- start 2024-08-19T12:40:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.442+0000 7f83894e5640 1 -- start start 2024-08-19T12:40:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.442+0000 7f83894e5640 10 monclient: init 2024-08-19T12:40:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.443+0000 7f83894e5640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.443+0000 7f83894e5640 10 monclient: _add_conns ranks=[4,7,1,0,5,6,8,3,2] 2024-08-19T12:40:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.443+0000 7f83894e5640 10 monclient(hunting): picked mon.noname-c con 0x7f8384073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.443+0000 7f83894e5640 10 monclient(hunting): picked mon.noname-i con 0x7f83840a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.443+0000 7f83894e5640 10 monclient(hunting): picked mon.noname-f con 0x7f838413f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.443+0000 7f83894e5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f838409dae0 con 0x7f838413f2e0 2024-08-19T12:40:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.443+0000 7f83894e5640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8384136910 con 0x7f8384073470 2024-08-19T12:40:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.443+0000 7f83894e5640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f838409dd50 con 0x7f83840a57a0 2024-08-19T12:40:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.443+0000 7f83894e5640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:32.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.443+0000 7f83894e5640 10 monclient(hunting): authenticate will time out at 826.531365s 2024-08-19T12:40:32.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.443+0000 7f8382ffd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f8384073470 0x7f8384136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:39126/0 (socket says 172.21.15.26:39126) 2024-08-19T12:40:32.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.443+0000 7f8382ffd640 1 -- 172.21.15.26:0/579578095 learned_addr learned my addr 172.21.15.26:0/579578095 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:32.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.445+0000 7f8382ffd640 1 -- 172.21.15.26:0/579578095 >> v1:172.21.15.26:6791/0 conn(0x7f8384073470 legacy=0x7f8384136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.445+0000 7f83827fc640 1 -- 172.21.15.26:0/579578095 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3471254956 0 0) 0x7f8384136910 con 0x7f8384073470 2024-08-19T12:40:32.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.445+0000 7f83827fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.445+0000 7f83827fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.445+0000 7f83827fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.445+0000 7f83827fc640 10 monclient(hunting): my global_id is 25618 2024-08-19T12:40:32.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.445+0000 7f83827fc640 1 -- 172.21.15.26:0/579578095 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8370003520 con 0x7f8384073470 2024-08-19T12:40:32.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.445+0000 7f837bfff640 1 -- 172.21.15.26:0/579578095 >> v1:172.21.15.104:6792/0 conn(0x7f83840a57a0 legacy=0x7f83840a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.445+0000 7f83827fc640 1 -- 172.21.15.26:0/579578095 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3094462180 0 0) 0x7f8370003520 con 0x7f8384073470 2024-08-19T12:40:32.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.445+0000 7f83837fe640 1 -- 172.21.15.26:0/579578095 >> v1:172.21.15.104:6789/0 conn(0x7f838413f2e0 legacy=0x7f83841416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 1 -- 172.21.15.26:0/579578095 >> v1:172.21.15.104:6792/0 conn(0x7f83840a57a0 legacy=0x7f83840a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 1 -- 172.21.15.26:0/579578095 >> v1:172.21.15.104:6789/0 conn(0x7f838413f2e0 legacy=0x7f83841416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 10 monclient: _finish_hunting 0 2024-08-19T12:40:32.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 1 monclient: found mon.noname-c 2024-08-19T12:40:32.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:32.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:40:32.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 1 -- 172.21.15.26:0/579578095 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838409e030 con 0x7f8384073470 2024-08-19T12:40:32.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 10 monclient: _finish_auth 0 2024-08-19T12:40:32.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 10 monclient: _check_auth_tickets 2024-08-19T12:40:32.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:32.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83894e5640 5 monclient: authenticate success, global_id 25618 2024-08-19T12:40:32.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83894e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:32.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 1 -- 172.21.15.26:0/579578095 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8374001ae0 con 0x7f8384073470 2024-08-19T12:40:32.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:40:32.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 10 monclient: dump: 2024-08-19T12:40:32.454 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.454 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.454 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.454 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.454 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.454 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.455 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83894e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:32.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 1 -- 172.21.15.26:0/579578095 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f83740048a0 con 0x7f8384073470 2024-08-19T12:40:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 1 -- 172.21.15.26:0/579578095 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8374004e10 con 0x7f8384073470 2024-08-19T12:40:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83894e5640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83894e5640 4 set_mon_vals no callback set 2024-08-19T12:40:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83827fc640 10 monclient: dump: 2024-08-19T12:40:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.456 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.457 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83894e5640 10 monclient: shutdown 2024-08-19T12:40:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83894e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83894e5640 1 -- 172.21.15.26:0/579578095 >> v1:172.21.15.26:6791/0 conn(0x7f8384073470 legacy=0x7f8384136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83894e5640 1 -- 172.21.15.26:0/579578095 shutdown_connections 2024-08-19T12:40:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83894e5640 1 -- 172.21.15.26:0/579578095 >> 172.21.15.26:0/579578095 conn(0x7f83840abff0 msgr2=0x7f83840ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83894e5640 1 -- 172.21.15.26:0/579578095 shutdown_connections 2024-08-19T12:40:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.446+0000 7f83894e5640 1 -- 172.21.15.26:0/579578095 wait complete. 2024-08-19T12:40:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.447+0000 7f83894e5640 10 monclient: build_initial_monmap 2024-08-19T12:40:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.447+0000 7f83894e5640 10 monclient: monmap: 2024-08-19T12:40:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:32.458 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:32.459 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:32.459 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.447+0000 7f83894e5640 1 librados: starting msgr at 2024-08-19T12:40:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.447+0000 7f83894e5640 1 librados: starting objecter 2024-08-19T12:40:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.447+0000 7f83894e5640 1 Processor -- start 2024-08-19T12:40:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.447+0000 7f83894e5640 1 -- start start 2024-08-19T12:40:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.447+0000 7f83894e5640 1 librados: setting wanted keys 2024-08-19T12:40:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.447+0000 7f83894e5640 1 librados: calling monclient init 2024-08-19T12:40:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.447+0000 7f83894e5640 10 monclient: init 2024-08-19T12:40:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.448+0000 7f83894e5640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.448+0000 7f83894e5640 10 monclient: _add_conns ranks=[1,3,4,8,2,7,6,5,0] 2024-08-19T12:40:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.448+0000 7f83894e5640 10 monclient(hunting): picked mon.noname-f con 0x7f8384073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.448+0000 7f83894e5640 10 monclient(hunting): picked mon.noname-g con 0x7f83840a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:32.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.448+0000 7f83894e5640 10 monclient(hunting): picked mon.noname-c con 0x7f838413f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.448+0000 7f83894e5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83840a3b30 con 0x7f8384073470 2024-08-19T12:40:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.448+0000 7f83894e5640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83840a4690 con 0x7f83840a57a0 2024-08-19T12:40:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.448+0000 7f83894e5640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8384136f10 con 0x7f838413f2e0 2024-08-19T12:40:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.448+0000 7f83894e5640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.448+0000 7f83894e5640 10 monclient(hunting): authenticate will time out at 826.536290s 2024-08-19T12:40:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.448+0000 7f83837fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f838413f2e0 0x7f83841d61f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:39142/0 (socket says 172.21.15.26:39142) 2024-08-19T12:40:32.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.448+0000 7f83837fe640 1 -- 172.21.15.26:0/3599239815 learned_addr learned my addr 172.21.15.26:0/3599239815 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:32.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f83837fe640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.26:6791/0 conn(0x7f838413f2e0 legacy=0x7f83841d61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 client.?.objecter ms_handle_connect 0x7f838413f2e0 2024-08-19T12:40:32.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:32.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3984434474 0 0) 0x7f8384136f10 con 0x7f838413f2e0 2024-08-19T12:40:32.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): my global_id is 25627 2024-08-19T12:40:32.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8364004eb0 con 0x7f838413f2e0 2024-08-19T12:40:32.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 client.?.objecter ms_handle_connect 0x7f8384073470 2024-08-19T12:40:32.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:32.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f8382ffd640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.104:6789/0 conn(0x7f8384073470 legacy=0x7f83840a2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837bfff640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.104:6790/0 conn(0x7f83840a57a0 legacy=0x7f83840a2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 client.?.objecter ms_handle_connect 0x7f83840a57a0 2024-08-19T12:40:32.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:32.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 314922598 0 0) 0x7f83840a3b30 con 0x7f8384073470 2024-08-19T12:40:32.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): my global_id is 26020 2024-08-19T12:40:32.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8384136f10 con 0x7f8384073470 2024-08-19T12:40:32.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 551394067 0 0) 0x7f8364004eb0 con 0x7f838413f2e0 2024-08-19T12:40:32.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83840a3b30 con 0x7f838413f2e0 2024-08-19T12:40:32.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f837c002070 con 0x7f838413f2e0 2024-08-19T12:40:32.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:32.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:40:32.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): dump: 2024-08-19T12:40:32.471 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.471 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.471 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.471 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.471 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.472 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.472 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.472 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.472 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.472 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.472 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.473 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.473 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.473 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.473 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.473 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:32.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1388348628 0 0) 0x7f83840a4690 con 0x7f83840a57a0 2024-08-19T12:40:32.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 10 monclient(hunting): my global_id is 25527 2024-08-19T12:40:32.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.450+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8364004eb0 con 0x7f83840a57a0 2024-08-19T12:40:32.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3765744942 0 0) 0x7f83840a3b30 con 0x7f838413f2e0 2024-08-19T12:40:32.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.104:6790/0 conn(0x7f83840a57a0 legacy=0x7f83840a2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.104:6789/0 conn(0x7f8384073470 legacy=0x7f83840a2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:40:32.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 1 monclient: found mon.c 2024-08-19T12:40:32.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:32.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:40:32.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83840a52c0 con 0x7f838413f2e0 2024-08-19T12:40:32.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 10 monclient: _finish_auth 0 2024-08-19T12:40:32.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:40:32.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:32.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f83894e5640 5 monclient: authenticate success, global_id 25627 2024-08-19T12:40:32.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1435993834 0 0) 0x7f8384136f10 con 0x7f8384073470 2024-08-19T12:40:32.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:40:32.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f83894e5640 10 monclient: _renew_subs 2024-08-19T12:40:32.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8374004d90 con 0x7f8384073470 2024-08-19T12:40:32.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f83894e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:40:32.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f83894e5640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8384137eb0 con 0x7f838413f2e0 2024-08-19T12:40:32.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:40:32.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f83894e5640 10 client.25627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:32.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f83894e5640 10 monclient: _renew_subs 2024-08-19T12:40:32.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f83894e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:40:32.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f83894e5640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f83840a3340 con 0x7f838413f2e0 2024-08-19T12:40:32.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f83894e5640 1 librados: init done 2024-08-19T12:40:32.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f837c003db0 con 0x7f838413f2e0 2024-08-19T12:40:32.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:32.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f837c0042a0 con 0x7f838413f2e0 2024-08-19T12:40:32.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f8381ffb640 4 set_mon_vals no callback set 2024-08-19T12:40:32.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:32.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f837b7fe640 10 monclient: dump: 2024-08-19T12:40:32.476 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.477 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.477 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.477 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.477 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.477 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.477 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.477 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.477 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.477 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.477 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.477 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.477 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.478 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.478 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.478 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f83894e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:32.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.451+0000 7f83894e5640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f83680065c0 con 0x7f8368002860 2024-08-19T12:40:32.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.452+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f837c004540 con 0x7f838413f2e0 2024-08-19T12:40:32.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.452+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(75..75 src has 1..75) ==== 7121+0+0 (unknown 2329720755 0 0) 0x7f837c04eab0 con 0x7f838413f2e0 2024-08-19T12:40:32.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.452+0000 7f837b7fe640 10 client.25627.objecter ms_dispatch 0x7f8384000b90 osd_map(75..75 src has 1..75) 2024-08-19T12:40:32.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.452+0000 7f837b7fe640 3 client.25627.objecter handle_osd_map got epochs [75,75] > 0 2024-08-19T12:40:32.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.452+0000 7f837b7fe640 3 client.25627.objecter handle_osd_map decoding full epoch 75 2024-08-19T12:40:32.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.452+0000 7f837b7fe640 20 client.25627.objecter dump_active .. 0 homeless 2024-08-19T12:40:32.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.452+0000 7f837bfff640 10 monclient: get_auth_request con 0x7f8364040f20 auth_method 0 2024-08-19T12:40:32.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 10 client.25627.objecter ms_handle_connect 0x7f8368002860 2024-08-19T12:40:32.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 10 client.25627.objecter resend_mon_ops 2024-08-19T12:40:32.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f8382ffd640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.104:6792/0 conn(0x7f8368002860 legacy=0x7f8368004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2284972413 0 0) 0x7f83680065c0 con 0x7f8368002860 2024-08-19T12:40:32.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 10 monclient(hunting): my global_id is 134143 2024-08-19T12:40:32.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8384136f10 con 0x7f8368002860 2024-08-19T12:40:32.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 940228430 0 0) 0x7f8384136f10 con 0x7f8368002860 2024-08-19T12:40:32.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83680065c0 con 0x7f8368002860 2024-08-19T12:40:32.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8374001320 con 0x7f8368002860 2024-08-19T12:40:32.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:32.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 10 monclient: dump: 2024-08-19T12:40:32.480 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.480 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.481 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.481 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.481 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.481 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.481 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.481 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.481 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.481 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.481 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.481 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.481 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.482 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.482 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.482 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 10 client.25627.objecter ms_handle_connect 0x7f8364040f20 2024-08-19T12:40:32.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1270980116 0 0) 0x7f83680065c0 con 0x7f8368002860 2024-08-19T12:40:32.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.454+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f836800a160 con 0x7f8368002860 2024-08-19T12:40:32.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.455+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8374005520 con 0x7f8368002860 2024-08-19T12:40:32.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.455+0000 7f837b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:32.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.455+0000 7f837b7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:32.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.455+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.104:6792/0 conn(0x7f8368002860 legacy=0x7f8368004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.455+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.104:6792/0 conn(0x7f8368002860 legacy=0x7f8368004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:32.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.461+0000 7f83894e5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:32.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.461+0000 7f83894e5640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8368009e70 con 0x7f8368009130 2024-08-19T12:40:32.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f8382ffd640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.104:6792/0 conn(0x7f8368009130 legacy=0x7f83680095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 10 client.25627.objecter ms_handle_connect 0x7f8368009130 2024-08-19T12:40:32.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 10 client.25627.objecter resend_mon_ops 2024-08-19T12:40:32.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3531740369 0 0) 0x7f8368009e70 con 0x7f8368009130 2024-08-19T12:40:32.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 10 monclient(hunting): my global_id is 134152 2024-08-19T12:40:32.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83680065c0 con 0x7f8368009130 2024-08-19T12:40:32.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3054637403 0 0) 0x7f83680065c0 con 0x7f8368009130 2024-08-19T12:40:32.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8368009e70 con 0x7f8368009130 2024-08-19T12:40:32.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8374005e10 con 0x7f8368009130 2024-08-19T12:40:32.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:32.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.464+0000 7f837b7fe640 10 monclient: dump: 2024-08-19T12:40:32.484 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.484 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.485 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.485 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.485 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.485 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.485 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.485 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.485 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.485 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.485 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.485 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.485 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.486 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.486 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.486 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3519103410 0 0) 0x7f8368009e70 con 0x7f8368009130 2024-08-19T12:40:32.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f836800c410 con 0x7f8368009130 2024-08-19T12:40:32.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 3846879352) 0x7f83740073f0 con 0x7f8368009130 2024-08-19T12:40:32.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f837b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:32.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f837b7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:32.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.104:6792/0 conn(0x7f8368009130 legacy=0x7f83680095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f837b7fe640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.104:6792/0 conn(0x7f8368009130 legacy=0x7f83680095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:32.487 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:32.487 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:32.487 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:32.487 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:32.487 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:32.487 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 2823, 2024-08-19T12:40:32.487 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:32.487 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:32.488 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:32.488 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:32.488 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:32.488 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:32.488 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:32.488 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:32.488 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:32.488 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:32.488 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.488 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:40:32.488 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:32.488 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:32.489 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:32.489 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:32.489 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:32.489 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:32.489 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:32.489 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:32.489 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:32.489 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:32.489 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:32.489 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:32.489 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:32.489 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:32.490 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.490 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:32.490 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:32.490 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:32.490 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:32.490 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:32.490 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:32.490 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:32.490 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:32.490 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:32.490 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:32.491 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:32.491 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:32.491 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:32.491 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.491 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:32.491 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:32.491 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:32.491 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:32.491 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:32.491 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:32.492 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:32.492 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:32.492 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:32.492 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:32.492 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:32.492 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:32.492 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:32.492 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:32.492 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:32.492 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:32.492 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:32.493 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:32.493 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:32.493 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:32.493 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:32.493 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:32.493 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:32.493 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:32.493 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:32.493 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:32.494 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:32.494 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:32.494 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.494 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:32.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.494 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:32.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.494 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:32.494 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:32.494 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.495 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.495 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.495 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:32.495 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.495 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.495 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.495 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:32.495 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:32.496 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.496 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.496 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.496 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:32.496 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:32.496 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.496 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.496 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.498 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:32.498 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:32.498 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.498 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.498 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.498 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:32.498 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.499 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:32.499 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:32.499 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.499 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.501 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:32.503 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:32.505 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:32.506 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.507 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.508 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:32.509 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:32.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:32.510 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:32.510 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:32.510 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:32.510 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:32.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.511 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:32.511 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:32.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.511 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:32.511 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:32.511 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:32.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.511 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:32.511 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:32.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:32.512 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:32.512 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:32.512 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:32.512 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:32.512 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:32.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:32.512 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:32.512 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:32.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 10 librados: watch_flush enter 2024-08-19T12:40:32.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 10 librados: watch_flush exit 2024-08-19T12:40:32.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 20 client.25627.objecter shutdown clearing up homeless session... 2024-08-19T12:40:32.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 10 client.25627.objecter successfully canceled tick 2024-08-19T12:40:32.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.104:6800/899887506 conn(0x7f8364040f20 legacy=0x7f83640433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 10 monclient: shutdown 2024-08-19T12:40:32.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:32.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 1 -- 172.21.15.26:0/3599239815 >> v1:172.21.15.26:6791/0 conn(0x7f838413f2e0 legacy=0x7f83841d61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83837fe640 1 -- 172.21.15.26:0/3599239815 reap_dead start 2024-08-19T12:40:32.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83837fe640 1 -- 172.21.15.26:0/3599239815 reap_dead start 2024-08-19T12:40:32.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 1 -- 172.21.15.26:0/3599239815 shutdown_connections 2024-08-19T12:40:32.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 1 -- 172.21.15.26:0/3599239815 >> 172.21.15.26:0/3599239815 conn(0x7f83840abff0 msgr2=0x7f8384134980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:32.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 1 -- 172.21.15.26:0/3599239815 shutdown_connections 2024-08-19T12:40:32.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 1 -- 172.21.15.26:0/3599239815 wait complete. 2024-08-19T12:40:32.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.465+0000 7f83894e5640 1 librados: shutdown 2024-08-19T12:40:32.516 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:32.516 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T12:40:32.516 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T12:40:32.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.613+0000 7f27e99a1640 10 monclient: get_monmap_and_config 2024-08-19T12:40:32.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.613+0000 7f27e99a1640 10 monclient: build_initial_monmap 2024-08-19T12:40:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.613+0000 7f27e99a1640 10 monclient: monmap: 2024-08-19T12:40:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:32.615122+0000 2024-08-19T12:40:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:32.615122+0000 2024-08-19T12:40:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:32.616 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:32.617 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:32.617 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:32.617 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:32.617 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:32.617 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.613+0000 7f27e99a1640 1 Processor -- start 2024-08-19T12:40:32.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.613+0000 7f27e99a1640 1 -- start start 2024-08-19T12:40:32.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.613+0000 7f27e99a1640 10 monclient: init 2024-08-19T12:40:32.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e99a1640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:32.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e99a1640 10 monclient: _add_conns ranks=[1,0,8,3,5,4,7,2,6] 2024-08-19T12:40:32.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e99a1640 10 monclient(hunting): picked mon.noname-f con 0x7f27e4073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:32.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e99a1640 10 monclient(hunting): picked mon.noname-a con 0x7f27e40a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:32.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e99a1640 10 monclient(hunting): picked mon.noname-e con 0x7f27e413f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:32.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e99a1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e409dae0 con 0x7f27e40a57a0 2024-08-19T12:40:32.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e99a1640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e4136910 con 0x7f27e4073470 2024-08-19T12:40:32.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e99a1640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e409dd50 con 0x7f27e413f2e0 2024-08-19T12:40:32.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e99a1640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e99a1640 10 monclient(hunting): authenticate will time out at 826.702536s 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e27fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f27e40a57a0 0x7f27e40a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59854/0 (socket says 172.21.15.26:59854) 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e27fc640 1 -- 172.21.15.26:0/623977853 learned_addr learned my addr 172.21.15.26:0/623977853 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.614+0000 7f27e37fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f27e413f2e0 0x7f27e41416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50232/0 (socket says 172.21.15.26:50232) 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e37fe640 1 -- 172.21.15.26:0/623977853 >> v1:172.21.15.26:6793/0 conn(0x7f27e413f2e0 legacy=0x7f27e41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e27fc640 1 -- 172.21.15.26:0/623977853 >> v1:172.21.15.26:6789/0 conn(0x7f27e40a57a0 legacy=0x7f27e40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e1ffb640 1 -- 172.21.15.26:0/623977853 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 38210240 0 0) 0x7f27e409dd50 con 0x7f27e413f2e0 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e1ffb640 10 monclient(hunting): my global_id is 25208 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e1ffb640 1 -- 172.21.15.26:0/623977853 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27d0003520 con 0x7f27e413f2e0 2024-08-19T12:40:32.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e1ffb640 1 -- 172.21.15.26:0/623977853 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3396169026 0 0) 0x7f27e409dae0 con 0x7f27e40a57a0 2024-08-19T12:40:32.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e1ffb640 10 monclient(hunting): my global_id is 26892 2024-08-19T12:40:32.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.616+0000 7f27e1ffb640 1 -- 172.21.15.26:0/623977853 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27e409dd50 con 0x7f27e40a57a0 2024-08-19T12:40:32.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e2ffd640 1 -- 172.21.15.26:0/623977853 >> v1:172.21.15.104:6789/0 conn(0x7f27e4073470 legacy=0x7f27e4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 1 -- 172.21.15.26:0/623977853 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3462093722 0 0) 0x7f27d0003520 con 0x7f27e413f2e0 2024-08-19T12:40:32.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 1 -- 172.21.15.26:0/623977853 >> v1:172.21.15.104:6789/0 conn(0x7f27e4073470 legacy=0x7f27e4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 1 -- 172.21.15.26:0/623977853 >> v1:172.21.15.26:6789/0 conn(0x7f27e40a57a0 legacy=0x7f27e40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 1 monclient: found mon.noname-e 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 1 -- 172.21.15.26:0/623977853 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e409e030 con 0x7f27e413f2e0 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 10 monclient: _finish_auth 0 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 1 -- 172.21.15.26:0/623977853 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f27dc001e60 con 0x7f27e413f2e0 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e99a1640 5 monclient: authenticate success, global_id 25208 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e99a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:40:32.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 10 monclient: dump: 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.626 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e99a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 1 -- 172.21.15.26:0/623977853 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f27dc004b60 con 0x7f27e413f2e0 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 1 -- 172.21.15.26:0/623977853 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f27dc0050d0 con 0x7f27e413f2e0 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e99a1640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e99a1640 4 set_mon_vals no callback set 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e1ffb640 10 monclient: dump: 2024-08-19T12:40:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.628 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e99a1640 10 monclient: shutdown 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e99a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e99a1640 1 -- 172.21.15.26:0/623977853 >> v1:172.21.15.26:6793/0 conn(0x7f27e413f2e0 legacy=0x7f27e41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e99a1640 1 -- 172.21.15.26:0/623977853 shutdown_connections 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e99a1640 1 -- 172.21.15.26:0/623977853 >> 172.21.15.26:0/623977853 conn(0x7f27e40abff0 msgr2=0x7f27e40ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e99a1640 1 -- 172.21.15.26:0/623977853 shutdown_connections 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.617+0000 7f27e99a1640 1 -- 172.21.15.26:0/623977853 wait complete. 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.618+0000 7f27e99a1640 10 monclient: build_initial_monmap 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.618+0000 7f27e99a1640 10 monclient: monmap: 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:32.629 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.618+0000 7f27e99a1640 1 librados: starting msgr at 2024-08-19T12:40:32.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.618+0000 7f27e99a1640 1 librados: starting objecter 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.618+0000 7f27e99a1640 1 Processor -- start 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 1 -- start start 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 1 librados: setting wanted keys 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 1 librados: calling monclient init 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 10 monclient: init 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 10 monclient: _add_conns ranks=[1,2,5,0,6,3,8,7,4] 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 10 monclient(hunting): picked mon.noname-f con 0x7f27e4073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 10 monclient(hunting): picked mon.noname-b con 0x7f27e40a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 10 monclient(hunting): picked mon.noname-h con 0x7f27e413f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e40a9f00 con 0x7f27e4073470 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e41370c0 con 0x7f27e40a57a0 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e413e9b0 con 0x7f27e413f2e0 2024-08-19T12:40:32.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:32.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e99a1640 10 monclient(hunting): authenticate will time out at 826.707484s 2024-08-19T12:40:32.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e27fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f27e40a57a0 0x7f27e40a26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58234/0 (socket says 172.21.15.26:58234) 2024-08-19T12:40:32.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.619+0000 7f27e27fc640 1 -- 172.21.15.26:0/2988898064 learned_addr learned my addr 172.21.15.26:0/2988898064 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.621+0000 7f27e27fc640 1 -- 172.21.15.26:0/2988898064 >> v1:172.21.15.26:6790/0 conn(0x7f27e40a57a0 legacy=0x7f27e40a26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.621+0000 7f27dbfff640 10 client.?.objecter ms_handle_connect 0x7f27e40a57a0 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.621+0000 7f27dbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.622+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1916236246 0 0) 0x7f27e41370c0 con 0x7f27e40a57a0 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.622+0000 7f27dbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.622+0000 7f27dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.622+0000 7f27dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.622+0000 7f27dbfff640 10 monclient(hunting): my global_id is 25355 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.622+0000 7f27e37fe640 1 -- 172.21.15.26:0/2988898064 >> v1:172.21.15.104:6791/0 conn(0x7f27e413f2e0 legacy=0x7f27e41d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.622+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27c4004eb0 con 0x7f27e40a57a0 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.622+0000 7f27dbfff640 10 client.?.objecter ms_handle_connect 0x7f27e413f2e0 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.622+0000 7f27dbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:32.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.622+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.622+0000 7f27e2ffd640 1 -- 172.21.15.26:0/2988898064 >> v1:172.21.15.104:6789/0 conn(0x7f27e4073470 legacy=0x7f27e40a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 client.?.objecter ms_handle_connect 0x7f27e4073470 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1731133938 0 0) 0x7f27e4136910 con 0x7f27e4073470 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient(hunting): my global_id is 26038 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27e41370c0 con 0x7f27e4073470 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2399376378 0 0) 0x7f27c4004eb0 con 0x7f27e40a57a0 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27e4136910 con 0x7f27e40a57a0 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4112111221 0 0) 0x7f27e413e9b0 con 0x7f27e413f2e0 2024-08-19T12:40:32.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient(hunting): my global_id is 25412 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27c4004eb0 con 0x7f27e413f2e0 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f27cc001d00 con 0x7f27e40a57a0 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient(hunting): dump: 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.638 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3002063671 0 0) 0x7f27e4136910 con 0x7f27e40a57a0 2024-08-19T12:40:32.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 >> v1:172.21.15.104:6791/0 conn(0x7f27e413f2e0 legacy=0x7f27e41d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 >> v1:172.21.15.104:6789/0 conn(0x7f27e4073470 legacy=0x7f27e40a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient: _finish_hunting 0 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 monclient: found mon.b 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e4136b20 con 0x7f27e40a57a0 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient: _finish_auth 0 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient: _check_auth_tickets 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27e99a1640 5 monclient: authenticate success, global_id 25355 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27e99a1640 10 monclient: _renew_subs 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27e99a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27e99a1640 1 -- 172.21.15.26:0/2988898064 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f27e40a2e00 con 0x7f27e40a57a0 2024-08-19T12:40:32.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27e99a1640 10 client.25355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27e99a1640 10 monclient: _renew_subs 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27e99a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27e99a1640 1 -- 172.21.15.26:0/2988898064 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f27e40a4540 con 0x7f27e40a57a0 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27e99a1640 1 librados: init done 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f27cc004bb0 con 0x7f27e40a57a0 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f27cc0050a0 con 0x7f27e40a57a0 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27e17fa640 4 set_mon_vals no callback set 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.624+0000 7f27dbfff640 10 monclient: dump: 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:32.641 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.642 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:32.642 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:32.642 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:32.642 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:32.642 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:32.642 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:32.642 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:32.642 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:32.642 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:32.642 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:32.642 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:32.642 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.625+0000 7f27e99a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.625+0000 7f27e99a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.625+0000 7f27e99a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.625+0000 7f27e99a1640 1 -- 172.21.15.26:0/2988898064 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27c8005080 con 0x7f27e40a57a0 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.625+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f27cc005240 con 0x7f27e40a57a0 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.625+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(75..75 src has 1..75) ==== 7121+0+0 (unknown 2329720755 0 0) 0x7f27cc04e7b0 con 0x7f27e40a57a0 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.625+0000 7f27dbfff640 10 client.25355.objecter ms_dispatch 0x7f27e4000b90 osd_map(75..75 src has 1..75) 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.625+0000 7f27dbfff640 3 client.25355.objecter handle_osd_map got epochs [75,75] > 0 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.625+0000 7f27dbfff640 3 client.25355.objecter handle_osd_map decoding full epoch 75 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.626+0000 7f27dbfff640 20 client.25355.objecter dump_active .. 0 homeless 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.626+0000 7f27e2ffd640 10 monclient: get_auth_request con 0x7f27c4041280 auth_method 0 2024-08-19T12:40:32.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.628+0000 7f27dbfff640 10 client.25355.objecter ms_handle_connect 0x7f27c4041280 2024-08-19T12:40:32.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.630+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f27cc0175b0 con 0x7f27e40a57a0 2024-08-19T12:40:32.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.630+0000 7f27dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:32.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.630+0000 7f27dbfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:32.791 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: Running main() from gmock_main.cc 2024-08-19T12:40:32.791 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-08-19T12:40:32.791 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [----------] Global test environment set-up. 2024-08-19T12:40:32.791 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-08-19T12:40:32.791 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2073 ms) 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2079 ms) 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2005 ms) 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2014 ms) 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8172 ms total) 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4046 ms) 2024-08-19T12:40:32.792 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-08-19T12:40:32.793 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4519 ms) 2024-08-19T12:40:32.793 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-08-19T12:40:32.793 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (4482 ms) 2024-08-19T12:40:32.793 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (13047 ms total) 2024-08-19T12:40:32.793 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: 2024-08-19T12:40:32.793 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-08-19T12:40:32.793 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-08-19T12:40:32.793 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2969 ms) 2024-08-19T12:40:32.793 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-08-19T12:40:32.793 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2043 ms) 2024-08-19T12:40:32.793 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-08-19T12:40:32.793 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1935 ms) 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2072 ms) 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9019 ms total) 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (3933 ms) 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4015 ms) 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (7948 ms total) 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [----------] Global test environment tear-down 2024-08-19T12:40:32.794 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (57165 ms total) 2024-08-19T12:40:32.795 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-08-19T12:40:32.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.900+0000 7f27e99a1640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T12:40:32.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.900+0000 7f27e99a1640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T12:40:32.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.900+0000 7f27e99a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:32.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.900+0000 7f27e99a1640 1 -- 172.21.15.26:0/2988898064 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f27c8005c40 con 0x7f27e40a57a0 2024-08-19T12:40:32.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.902+0000 7f27dbfff640 1 -- 172.21.15.26:0/2988898064 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f27cc016eb0 con 0x7f27e40a57a0 2024-08-19T12:40:32.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.902+0000 7f27dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T12:40:32.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.902+0000 7f27dbfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:32.905 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:32.905 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=1,health=12,kv=11,logm=57} crc {auth=1172049513,config=3009771555,health=4142905394,kv=959342030,logm=1761305446}) 2024-08-19T12:40:32.906 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=12,kv=12,logm=54} crc {auth=2478323150,config=1097430493,health=4142905394,kv=2934104134,logm=3423998784}) 2024-08-19T12:40:32.906 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:32.906 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=1,health=12,kv=11,logm=57} crc {auth=1172049513,config=3009771555,health=4142905394,kv=959342030,logm=1761305446}) 2024-08-19T12:40:32.906 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=1319256804,config=1097430494,health=4095691973,kv=333004542,logm=2691270319}) 2024-08-19T12:40:32.906 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:32.906 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=1,health=12,kv=11,logm=57} crc {auth=1172049513,config=3009771555,health=4142905394,kv=959342030,logm=1761305446}) 2024-08-19T12:40:32.906 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=19,config=2,health=12,kv=13,logm=54} crc {auth=3943396684,config=1097430493,health=4142905394,kv=3342064064,logm=2853430163}) 2024-08-19T12:40:32.906 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:32.906 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=1,health=12,kv=11,logm=57} crc {auth=1172049513,config=3009771555,health=4142905394,kv=959342030,logm=1761305446}) 2024-08-19T12:40:32.906 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=20,config=1,health=12,kv=13,logm=54} crc {auth=3127897874,config=3009771555,health=4142905394,kv=1344821656,logm=1533664741}) 2024-08-19T12:40:32.906 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:32.906 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=1,health=12,kv=11,logm=57} crc {auth=1172049513,config=3009771555,health=4142905394,kv=959342030,logm=1761305446}) 2024-08-19T12:40:32.907 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=19,config=1,health=12,kv=12,logm=56} crc {auth=2135785899,config=1228700967,health=4142905394,kv=2691022117,logm=546629682}) 2024-08-19T12:40:32.907 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:32.907 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=1,health=12,kv=11,logm=57} crc {auth=1172049513,config=3009771555,health=4142905394,kv=959342030,logm=1761305446}) 2024-08-19T12:40:32.907 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=19,config=2,health=11,kv=13,logm=55} crc {auth=3667121500,config=1097430493,health=1760202739,kv=1344821656,logm=550897820}) 2024-08-19T12:40:32.907 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:32.907 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=1,health=12,kv=11,logm=57} crc {auth=1172049513,config=3009771555,health=4142905394,kv=959342030,logm=1761305446}) 2024-08-19T12:40:32.907 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=18,config=2,health=12,kv=11,logm=57} crc {auth=1796364498,config=4178277379,health=4142905394,kv=733351094,logm=502598886}) 2024-08-19T12:40:32.907 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:32.907 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=1,health=12,kv=11,logm=57} crc {auth=1172049513,config=3009771555,health=4142905394,kv=959342030,logm=1761305446}) 2024-08-19T12:40:32.907 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:32.902+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=20,config=1,health=12,kv=13,logm=54} crc {auth=3127897874,config=3009771555,health=4142905394,kv=1344821656,logm=3672209637}) 2024-08-19T12:40:32.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.905+0000 7f27e99a1640 10 librados: watch_flush enter 2024-08-19T12:40:32.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.906+0000 7f27e99a1640 10 librados: watch_flush exit 2024-08-19T12:40:32.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.906+0000 7f27e99a1640 20 client.25355.objecter shutdown clearing up homeless session... 2024-08-19T12:40:32.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.906+0000 7f27e99a1640 10 client.25355.objecter successfully canceled tick 2024-08-19T12:40:32.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.906+0000 7f27e99a1640 1 -- 172.21.15.26:0/2988898064 >> v1:172.21.15.104:6800/899887506 conn(0x7f27c4041280 legacy=0x7f27c4043740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.906+0000 7f27e99a1640 10 monclient: shutdown 2024-08-19T12:40:32.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.906+0000 7f27e99a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:32.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.906+0000 7f27e99a1640 1 -- 172.21.15.26:0/2988898064 >> v1:172.21.15.26:6790/0 conn(0x7f27e40a57a0 legacy=0x7f27e40a26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:32.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.906+0000 7f27e99a1640 1 -- 172.21.15.26:0/2988898064 shutdown_connections 2024-08-19T12:40:32.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.906+0000 7f27e99a1640 1 -- 172.21.15.26:0/2988898064 >> 172.21.15.26:0/2988898064 conn(0x7f27e40abff0 msgr2=0x7f27e4134360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:32.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.906+0000 7f27e99a1640 1 -- 172.21.15.26:0/2988898064 shutdown_connections 2024-08-19T12:40:32.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.906+0000 7f27e99a1640 1 -- 172.21.15.26:0/2988898064 wait complete. 2024-08-19T12:40:32.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:32.907+0000 7f27e99a1640 1 librados: shutdown 2024-08-19T12:40:32.922 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T12:40:33.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4038658259}) 2024-08-19T12:40:33.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2078842379}) 2024-08-19T12:40:33.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4038658259}) 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1098823689}) 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4038658259}) 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3505384609}) 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4038658259}) 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2740298463}) 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4038658259}) 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1829974478}) 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4038658259}) 2024-08-19T12:40:33.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1416648248}) 2024-08-19T12:40:33.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4038658259}) 2024-08-19T12:40:33.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3550156204}) 2024-08-19T12:40:33.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4038658259}) 2024-08-19T12:40:33.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.017+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1854968342}) 2024-08-19T12:40:33.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082621136}) 2024-08-19T12:40:33.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181599249}) 2024-08-19T12:40:33.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082621136}) 2024-08-19T12:40:33.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1118846870}) 2024-08-19T12:40:33.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082621136}) 2024-08-19T12:40:33.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2729150153}) 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082621136}) 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3941763562}) 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082621136}) 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3375007527}) 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082621136}) 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2262871139}) 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082621136}) 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2967400711}) 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082621136}) 2024-08-19T12:40:33.078 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.073+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3329338053}) 2024-08-19T12:40:33.125 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.125 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372479292}) 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3628911103}) 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372479292}) 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3460882245}) 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372479292}) 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=161368810}) 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372479292}) 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1292235512}) 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372479292}) 2024-08-19T12:40:33.126 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=336644617}) 2024-08-19T12:40:33.127 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.127 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372479292}) 2024-08-19T12:40:33.127 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4087437768}) 2024-08-19T12:40:33.127 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.127 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372479292}) 2024-08-19T12:40:33.127 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=505232095}) 2024-08-19T12:40:33.127 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.127 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372479292}) 2024-08-19T12:40:33.127 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.122+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=895606883}) 2024-08-19T12:40:33.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126590215}) 2024-08-19T12:40:33.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3884691930}) 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126590215}) 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3474159906}) 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126590215}) 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4069326012}) 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126590215}) 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3844635168}) 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126590215}) 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2402880005}) 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126590215}) 2024-08-19T12:40:33.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3544622786}) 2024-08-19T12:40:33.132 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.132 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126590215}) 2024-08-19T12:40:33.132 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3258310067}) 2024-08-19T12:40:33.132 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.132 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126590215}) 2024-08-19T12:40:33.132 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.127+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=11174497}) 2024-08-19T12:40:33.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895579348}) 2024-08-19T12:40:33.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=601772771}) 2024-08-19T12:40:33.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895579348}) 2024-08-19T12:40:33.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3627398104}) 2024-08-19T12:40:33.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895579348}) 2024-08-19T12:40:33.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1548885486}) 2024-08-19T12:40:33.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895579348}) 2024-08-19T12:40:33.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1904820160}) 2024-08-19T12:40:33.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895579348}) 2024-08-19T12:40:33.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3876403438}) 2024-08-19T12:40:33.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895579348}) 2024-08-19T12:40:33.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3700316179}) 2024-08-19T12:40:33.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895579348}) 2024-08-19T12:40:33.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3031666336}) 2024-08-19T12:40:33.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895579348}) 2024-08-19T12:40:33.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.803+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1330111071}) 2024-08-19T12:40:33.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858329343}) 2024-08-19T12:40:33.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4081042595}) 2024-08-19T12:40:33.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858329343}) 2024-08-19T12:40:33.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905901186}) 2024-08-19T12:40:33.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858329343}) 2024-08-19T12:40:33.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2947037163}) 2024-08-19T12:40:33.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858329343}) 2024-08-19T12:40:33.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1853767568}) 2024-08-19T12:40:33.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858329343}) 2024-08-19T12:40:33.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3039530631}) 2024-08-19T12:40:33.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858329343}) 2024-08-19T12:40:33.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3077401980}) 2024-08-19T12:40:33.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858329343}) 2024-08-19T12:40:33.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4062799194}) 2024-08-19T12:40:33.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858329343}) 2024-08-19T12:40:33.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.809+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1493201496}) 2024-08-19T12:40:33.837 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.837 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501305027}) 2024-08-19T12:40:33.837 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=869250656}) 2024-08-19T12:40:33.837 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.838 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501305027}) 2024-08-19T12:40:33.838 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3495333168}) 2024-08-19T12:40:33.838 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.838 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501305027}) 2024-08-19T12:40:33.838 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=732249670}) 2024-08-19T12:40:33.838 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.838 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501305027}) 2024-08-19T12:40:33.838 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2062871797}) 2024-08-19T12:40:33.838 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.838 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501305027}) 2024-08-19T12:40:33.838 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3356258400}) 2024-08-19T12:40:33.838 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.839 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501305027}) 2024-08-19T12:40:33.839 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1432543221}) 2024-08-19T12:40:33.839 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.839 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501305027}) 2024-08-19T12:40:33.839 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2587147064}) 2024-08-19T12:40:33.839 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.839 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501305027}) 2024-08-19T12:40:33.839 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.832+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1297637850}) 2024-08-19T12:40:33.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.888+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.888+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227576031}) 2024-08-19T12:40:33.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.888+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3028670245}) 2024-08-19T12:40:33.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.888+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.888+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227576031}) 2024-08-19T12:40:33.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.888+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4205570386}) 2024-08-19T12:40:33.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.888+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.888+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227576031}) 2024-08-19T12:40:33.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2752644961}) 2024-08-19T12:40:33.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227576031}) 2024-08-19T12:40:33.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=368475870}) 2024-08-19T12:40:33.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227576031}) 2024-08-19T12:40:33.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=251707546}) 2024-08-19T12:40:33.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227576031}) 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=233511783}) 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227576031}) 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2405129548}) 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227576031}) 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.889+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2006742107}) 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412687369}) 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=828676794}) 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412687369}) 2024-08-19T12:40:33.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1031466348}) 2024-08-19T12:40:33.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412687369}) 2024-08-19T12:40:33.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=467880727}) 2024-08-19T12:40:33.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412687369}) 2024-08-19T12:40:33.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3463532492}) 2024-08-19T12:40:33.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412687369}) 2024-08-19T12:40:33.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=316476787}) 2024-08-19T12:40:33.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412687369}) 2024-08-19T12:40:33.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3265540898}) 2024-08-19T12:40:33.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412687369}) 2024-08-19T12:40:33.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=85432407}) 2024-08-19T12:40:33.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:33.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412687369}) 2024-08-19T12:40:33.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:33.890+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2272660940}) 2024-08-19T12:40:33.923 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:40:33.924 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:40:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.026+0000 7f08c1256640 10 monclient: get_monmap_and_config 2024-08-19T12:40:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.026+0000 7f08c1256640 10 monclient: build_initial_monmap 2024-08-19T12:40:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.026+0000 7f08c1256640 10 monclient: monmap: 2024-08-19T12:40:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:34.028550+0000 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:34.028550+0000 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:34.030 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 1 Processor -- start 2024-08-19T12:40:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 1 -- start start 2024-08-19T12:40:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 10 monclient: init 2024-08-19T12:40:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 10 monclient: _add_conns ranks=[0,2,6,3,5,1,7,4,8] 2024-08-19T12:40:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 10 monclient(hunting): picked mon.noname-a con 0x7f08bc06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 10 monclient(hunting): picked mon.noname-b con 0x7f08bc0a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 10 monclient(hunting): picked mon.noname-d con 0x7f08bc0a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08bc136790 con 0x7f08bc06e7b0 2024-08-19T12:40:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08bc09ddf0 con 0x7f08bc0a5920 2024-08-19T12:40:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08bc09e0d0 con 0x7f08bc0a5d50 2024-08-19T12:40:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08c1256640 10 monclient(hunting): authenticate will time out at 828.115833s 2024-08-19T12:40:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08ba59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f08bc0a5920 0x7f08bc13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58246/0 (socket says 172.21.15.26:58246) 2024-08-19T12:40:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08ba59c640 1 -- 172.21.15.26:0/1006916347 learned_addr learned my addr 172.21.15.26:0/1006916347 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.027+0000 7f08bb59e640 1 --1- 172.21.15.26:0/1006916347 >> v1:172.21.15.26:6792/0 conn(0x7f08bc0a5d50 0x7f08bc141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39088/0 (socket says 172.21.15.26:39088) 2024-08-19T12:40:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08ba59c640 1 -- 172.21.15.26:0/1006916347 >> v1:172.21.15.26:6790/0 conn(0x7f08bc0a5920 legacy=0x7f08bc13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08bb59e640 1 -- 172.21.15.26:0/1006916347 >> v1:172.21.15.26:6792/0 conn(0x7f08bc0a5d50 legacy=0x7f08bc141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08bad9d640 1 -- 172.21.15.26:0/1006916347 >> v1:172.21.15.26:6789/0 conn(0x7f08bc06e7b0 legacy=0x7f08bc136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2881254553 0 0) 0x7f08bc09e0d0 con 0x7f08bc0a5d50 2024-08-19T12:40:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient(hunting): my global_id is 25620 2024-08-19T12:40:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08a4003520 con 0x7f08bc0a5d50 2024-08-19T12:40:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 573469200 0 0) 0x7f08b0001450 con 0x7f08bc0a5920 2024-08-19T12:40:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient(hunting): my global_id is 25391 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08bc09e0d0 con 0x7f08bc0a5920 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2874453836 0 0) 0x7f08bc136790 con 0x7f08bc06e7b0 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient(hunting): my global_id is 26910 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08b0001450 con 0x7f08bc06e7b0 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3594782236 0 0) 0x7f08bc09e0d0 con 0x7f08bc0a5920 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 >> v1:172.21.15.26:6792/0 conn(0x7f08bc0a5d50 legacy=0x7f08bc141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 >> v1:172.21.15.26:6789/0 conn(0x7f08bc06e7b0 legacy=0x7f08bc136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:40:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 monclient: found mon.noname-b 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08bc09e3b0 con 0x7f08bc0a5920 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient: _finish_auth 0 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f08b0001c90 con 0x7f08bc0a5920 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient: dump: 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.038 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.039 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.039 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.039 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.039 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.041 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.041 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.041 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.041 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.041 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08c1256640 5 monclient: authenticate success, global_id 25391 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08c1256640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f08b0004920 con 0x7f08bc0a5920 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 1 -- 172.21.15.26:0/1006916347 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f08b0004e90 con 0x7f08bc0a5920 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08b9d9b640 10 monclient: dump: 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.043 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08c1256640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.030+0000 7f08c1256640 4 set_mon_vals no callback set 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.031+0000 7f08c1256640 10 monclient: shutdown 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.031+0000 7f08c1256640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.031+0000 7f08c1256640 1 -- 172.21.15.26:0/1006916347 >> v1:172.21.15.26:6790/0 conn(0x7f08bc0a5920 legacy=0x7f08bc13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.031+0000 7f08c1256640 1 -- 172.21.15.26:0/1006916347 shutdown_connections 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.031+0000 7f08c1256640 1 -- 172.21.15.26:0/1006916347 >> 172.21.15.26:0/1006916347 conn(0x7f08bc0a8740 msgr2=0x7f08bc0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.031+0000 7f08c1256640 1 -- 172.21.15.26:0/1006916347 shutdown_connections 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.031+0000 7f08c1256640 1 -- 172.21.15.26:0/1006916347 wait complete. 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.032+0000 7f08c1256640 10 monclient: build_initial_monmap 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.032+0000 7f08c1256640 10 monclient: monmap: 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:34.044 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.032+0000 7f08c1256640 1 librados: starting msgr at 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.032+0000 7f08c1256640 1 librados: starting objecter 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.032+0000 7f08c1256640 1 Processor -- start 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.032+0000 7f08c1256640 1 -- start start 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.032+0000 7f08c1256640 1 librados: setting wanted keys 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.032+0000 7f08c1256640 1 librados: calling monclient init 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.032+0000 7f08c1256640 10 monclient: init 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08c1256640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08c1256640 10 monclient: _add_conns ranks=[6,8,0,3,5,4,1,2,7] 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08c1256640 10 monclient(hunting): picked mon.noname-d con 0x7f08bc06e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08c1256640 10 monclient(hunting): picked mon.noname-e con 0x7f08bc0a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08c1256640 10 monclient(hunting): picked mon.noname-a con 0x7f08bc0a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08c1256640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08bc0a0450 con 0x7f08bc0a5d50 2024-08-19T12:40:34.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08c1256640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08bc0a0a10 con 0x7f08bc06e7b0 2024-08-19T12:40:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08c1256640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08bc0a0cf0 con 0x7f08bc0a5920 2024-08-19T12:40:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08c1256640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08c1256640 10 monclient(hunting): authenticate will time out at 828.121305s 2024-08-19T12:40:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08ba59c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f08bc0a5920 0x7f08bc1323e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50246/0 (socket says 172.21.15.26:50246) 2024-08-19T12:40:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08ba59c640 1 -- 172.21.15.26:0/2228232996 learned_addr learned my addr 172.21.15.26:0/2228232996 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:34.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.033+0000 7f08bad9d640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f08bc06e7b0 0x7f08bc131cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39102/0 (socket says 172.21.15.26:39102) 2024-08-19T12:40:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08bad9d640 1 -- 172.21.15.26:0/2228232996 >> v1:172.21.15.26:6792/0 conn(0x7f08bc06e7b0 legacy=0x7f08bc131cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08bb59e640 1 -- 172.21.15.26:0/2228232996 >> v1:172.21.15.26:6789/0 conn(0x7f08bc0a5d50 legacy=0x7f08bc132af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 client.?.objecter ms_handle_connect 0x7f08bc06e7b0 2024-08-19T12:40:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08ba59c640 1 -- 172.21.15.26:0/2228232996 >> v1:172.21.15.26:6793/0 conn(0x7f08bc0a5920 legacy=0x7f08bc1323e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 client.?.objecter ms_handle_connect 0x7f08bc0a5d50 2024-08-19T12:40:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 client.?.objecter ms_handle_connect 0x7f08bc0a5920 2024-08-19T12:40:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 974300686 0 0) 0x7f08bc0a0a10 con 0x7f08bc06e7b0 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 monclient(hunting): my global_id is 25629 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08a8004eb0 con 0x7f08bc06e7b0 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1546795412 0 0) 0x7f08bc0a0cf0 con 0x7f08bc0a5920 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 monclient(hunting): my global_id is 25226 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08bc0a0a10 con 0x7f08bc0a5920 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 570137125 0 0) 0x7f08bc0a0450 con 0x7f08bc0a5d50 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 10 monclient(hunting): my global_id is 26919 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.035+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08bc0a0cf0 con 0x7f08bc0a5d50 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3375539823 0 0) 0x7f08bc0a0a10 con 0x7f08bc0a5920 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08bc0a0450 con 0x7f08bc0a5920 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f08b0001cd0 con 0x7f08bc0a5920 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient(hunting): dump: 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.053 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:34.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4138973769 0 0) 0x7f08bc0a0cf0 con 0x7f08bc0a5d50 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08bc0a0a10 con 0x7f08bc0a5d50 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1240837466 0 0) 0x7f08a8004eb0 con 0x7f08bc06e7b0 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08bc0a0cf0 con 0x7f08bc06e7b0 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f08b40020b0 con 0x7f08bc0a5d50 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient(hunting): dump: 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.055 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f08ac003950 con 0x7f08bc06e7b0 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient(hunting): dump: 2024-08-19T12:40:34.056 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.057 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 210191391 0 0) 0x7f08bc0a0450 con 0x7f08bc0a5920 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 >> v1:172.21.15.26:6792/0 conn(0x7f08bc06e7b0 legacy=0x7f08bc131cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 >> v1:172.21.15.26:6789/0 conn(0x7f08bc0a5d50 legacy=0x7f08bc132af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient: _finish_hunting 0 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 monclient: found mon.e 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08bc09f790 con 0x7f08bc0a5920 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient: _finish_auth 0 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient: _check_auth_tickets 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:34.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08c1256640 5 monclient: authenticate success, global_id 25226 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08c1256640 10 monclient: _renew_subs 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08c1256640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08c1256640 1 -- 172.21.15.26:0/2228232996 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f08bc0a1110 con 0x7f08bc0a5920 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08c1256640 10 client.25226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08c1256640 10 monclient: _renew_subs 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f08b0004e90 con 0x7f08bc0a5920 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08c1256640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08c1256640 1 -- 172.21.15.26:0/2228232996 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f08bc1d7480 con 0x7f08bc0a5920 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08c1256640 1 librados: init done 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f08b0005510 con 0x7f08bc0a5920 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08a37fe640 10 monclient: dump: 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.036+0000 7f08b959a640 4 set_mon_vals no callback set 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.037+0000 7f08c1256640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.037+0000 7f08c1256640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.037+0000 7f08c1256640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.037+0000 7f08c1256640 1 -- 172.21.15.26:0/2228232996 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f089c005080 con 0x7f08bc0a5920 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.037+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f08b0011970 con 0x7f08bc0a5920 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.037+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(77..77 src has 1..77) ==== 6966+0+0 (unknown 2975105348 0 0) 0x7f08b004e6a0 con 0x7f08bc0a5920 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.037+0000 7f08a37fe640 10 client.25226.objecter ms_dispatch 0x7f08bc000b90 osd_map(77..77 src has 1..77) 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.037+0000 7f08a37fe640 3 client.25226.objecter handle_osd_map got epochs [77,77] > 0 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.037+0000 7f08a37fe640 3 client.25226.objecter handle_osd_map decoding full epoch 77 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.037+0000 7f08a37fe640 20 client.25226.objecter dump_active .. 0 homeless 2024-08-19T12:40:34.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.037+0000 7f08bad9d640 10 monclient: get_auth_request con 0x7f08a8041b50 auth_method 0 2024-08-19T12:40:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.040+0000 7f08a37fe640 10 client.25226.objecter ms_handle_connect 0x7f08a8041b50 2024-08-19T12:40:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.042+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f08bc09ddf0 con 0x7f08bc0a5920 2024-08-19T12:40:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.042+0000 7f08a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.042+0000 7f08a37fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227174040}) 2024-08-19T12:40:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=557287340}) 2024-08-19T12:40:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227174040}) 2024-08-19T12:40:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3988380629}) 2024-08-19T12:40:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227174040}) 2024-08-19T12:40:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3882377397}) 2024-08-19T12:40:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227174040}) 2024-08-19T12:40:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2445929861}) 2024-08-19T12:40:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.084+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.085+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227174040}) 2024-08-19T12:40:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.085+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2039062065}) 2024-08-19T12:40:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.085+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.085+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227174040}) 2024-08-19T12:40:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.085+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=76510866}) 2024-08-19T12:40:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.085+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.085+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227174040}) 2024-08-19T12:40:34.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.085+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=810341583}) 2024-08-19T12:40:34.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.085+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.085+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227174040}) 2024-08-19T12:40:34.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.085+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1164919203}) 2024-08-19T12:40:34.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612509179}) 2024-08-19T12:40:34.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2931693942}) 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612509179}) 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=702471881}) 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612509179}) 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=178346485}) 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612509179}) 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1333473989}) 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612509179}) 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1440878568}) 2024-08-19T12:40:34.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612509179}) 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2776496177}) 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612509179}) 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=910915278}) 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612509179}) 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.086+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1865525340}) 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3123919675}) 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1473140115}) 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3123919675}) 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2033446051}) 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3123919675}) 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2694812755}) 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3123919675}) 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3561555670}) 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3123919675}) 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3722583565}) 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3123919675}) 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2938001976}) 2024-08-19T12:40:34.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3123919675}) 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2118687644}) 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3123919675}) 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.088+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1925747725}) 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002975772}) 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=782159738}) 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002975772}) 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2804754835}) 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002975772}) 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4187091816}) 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002975772}) 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1721042453}) 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002975772}) 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3445496274}) 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002975772}) 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=75862021}) 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002975772}) 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3741420080}) 2024-08-19T12:40:34.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002975772}) 2024-08-19T12:40:34.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.090+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2642952359}) 2024-08-19T12:40:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.307+0000 7f08c1256640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:40:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.307+0000 7f08c1256640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:40:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.307+0000 7f08c1256640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.307+0000 7f08c1256640 1 -- 172.21.15.26:0/2228232996 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f089c005370 con 0x7f08bc0a5920 2024-08-19T12:40:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.307+0000 7f08a37fe640 1 -- 172.21.15.26:0/2228232996 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 3970115070) 0x7f08b0017450 con 0x7f08bc0a5920 2024-08-19T12:40:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.307+0000 7f08a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:40:34.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.307+0000 7f08a37fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:34.310 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":10,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:40:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 10 librados: watch_flush enter 2024-08-19T12:40:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 10 librados: watch_flush exit 2024-08-19T12:40:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 20 client.25226.objecter shutdown clearing up homeless session... 2024-08-19T12:40:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 10 client.25226.objecter successfully canceled tick 2024-08-19T12:40:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 1 -- 172.21.15.26:0/2228232996 >> v1:172.21.15.104:6800/899887506 conn(0x7f08a8041b50 legacy=0x7f08a8044010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 10 monclient: shutdown 2024-08-19T12:40:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 1 -- 172.21.15.26:0/2228232996 >> v1:172.21.15.26:6793/0 conn(0x7f08bc0a5920 legacy=0x7f08bc1323e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 1 -- 172.21.15.26:0/2228232996 shutdown_connections 2024-08-19T12:40:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 1 -- 172.21.15.26:0/2228232996 >> 172.21.15.26:0/2228232996 conn(0x7f08bc0a8740 msgr2=0x7f08bc06d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 1 -- 172.21.15.26:0/2228232996 shutdown_connections 2024-08-19T12:40:34.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 1 -- 172.21.15.26:0/2228232996 wait complete. 2024-08-19T12:40:34.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.309+0000 7f08c1256640 1 librados: shutdown 2024-08-19T12:40:34.322 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:40:34.322 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T12:40:34.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:40:34.322 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:40:34.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.419+0000 7efe144cc640 10 monclient: get_monmap_and_config 2024-08-19T12:40:34.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.419+0000 7efe144cc640 10 monclient: build_initial_monmap 2024-08-19T12:40:34.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.419+0000 7efe144cc640 10 monclient: monmap: 2024-08-19T12:40:34.421 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:34.421527+0000 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:34.421527+0000 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:34.422 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:34.423 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:34.423 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.420+0000 7efe144cc640 1 Processor -- start 2024-08-19T12:40:34.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.420+0000 7efe144cc640 1 -- start start 2024-08-19T12:40:34.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.420+0000 7efe144cc640 10 monclient: init 2024-08-19T12:40:34.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.420+0000 7efe144cc640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:34.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.420+0000 7efe144cc640 10 monclient: _add_conns ranks=[0,1,3,8,6,2,4,7,5] 2024-08-19T12:40:34.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.420+0000 7efe144cc640 10 monclient(hunting): picked mon.noname-a con 0x7efe0c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:34.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.420+0000 7efe144cc640 10 monclient(hunting): picked mon.noname-f con 0x7efe0c0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:34.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.420+0000 7efe144cc640 10 monclient(hunting): picked mon.noname-g con 0x7efe0c0a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:34.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.420+0000 7efe144cc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0c136790 con 0x7efe0c06e7b0 2024-08-19T12:40:34.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.421+0000 7efe144cc640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0c09ddf0 con 0x7efe0c0a5920 2024-08-19T12:40:34.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.421+0000 7efe144cc640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0c09e0d0 con 0x7efe0c0a5d50 2024-08-19T12:40:34.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.421+0000 7efe144cc640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:34.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.421+0000 7efe144cc640 10 monclient(hunting): authenticate will time out at 828.509071s 2024-08-19T12:40:34.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.421+0000 7efe12268640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7efe0c06e7b0 0x7efe0c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59880/0 (socket says 172.21.15.26:59880) 2024-08-19T12:40:34.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.421+0000 7efe12268640 1 -- 172.21.15.26:0/1247118612 learned_addr learned my addr 172.21.15.26:0/1247118612 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:34.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe12268640 1 -- 172.21.15.26:0/1247118612 >> v1:172.21.15.26:6789/0 conn(0x7efe0c06e7b0 legacy=0x7efe0c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 1 -- 172.21.15.26:0/1247118612 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3003908311 0 0) 0x7efe0c136790 con 0x7efe0c06e7b0 2024-08-19T12:40:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient(hunting): my global_id is 26928 2024-08-19T12:40:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 1 -- 172.21.15.26:0/1247118612 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdf8003520 con 0x7efe0c06e7b0 2024-08-19T12:40:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe12a69640 1 -- 172.21.15.26:0/1247118612 >> v1:172.21.15.104:6790/0 conn(0x7efe0c0a5d50 legacy=0x7efe0c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11a67640 1 -- 172.21.15.26:0/1247118612 >> v1:172.21.15.104:6789/0 conn(0x7efe0c0a5920 legacy=0x7efe0c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 1 -- 172.21.15.26:0/1247118612 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 562389113 0 0) 0x7efdf8003520 con 0x7efe0c06e7b0 2024-08-19T12:40:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 1 -- 172.21.15.26:0/1247118612 >> v1:172.21.15.104:6790/0 conn(0x7efe0c0a5d50 legacy=0x7efe0c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 1 -- 172.21.15.26:0/1247118612 >> v1:172.21.15.104:6789/0 conn(0x7efe0c0a5920 legacy=0x7efe0c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient: _finish_hunting 0 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 1 monclient: found mon.noname-a 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 1 -- 172.21.15.26:0/1247118612 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0c09e3b0 con 0x7efe0c06e7b0 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient: _finish_auth 0 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient: _check_auth_tickets 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 1 -- 172.21.15.26:0/1247118612 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efe08001ae0 con 0x7efe0c06e7b0 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient: dump: 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.430 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.431 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe144cc640 5 monclient: authenticate success, global_id 26928 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe144cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 1 -- 172.21.15.26:0/1247118612 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe080048a0 con 0x7efe0c06e7b0 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 1 -- 172.21.15.26:0/1247118612 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efe08004dd0 con 0x7efe0c06e7b0 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe144cc640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe144cc640 4 set_mon_vals no callback set 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.423+0000 7efe11266640 10 monclient: dump: 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.432 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.424+0000 7efe144cc640 10 monclient: shutdown 2024-08-19T12:40:34.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.424+0000 7efe144cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:34.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.424+0000 7efe144cc640 1 -- 172.21.15.26:0/1247118612 >> v1:172.21.15.26:6789/0 conn(0x7efe0c06e7b0 legacy=0x7efe0c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.424+0000 7efe144cc640 1 -- 172.21.15.26:0/1247118612 shutdown_connections 2024-08-19T12:40:34.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.424+0000 7efe144cc640 1 -- 172.21.15.26:0/1247118612 >> 172.21.15.26:0/1247118612 conn(0x7efe0c0a8740 msgr2=0x7efe0c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:34.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.424+0000 7efe144cc640 1 -- 172.21.15.26:0/1247118612 shutdown_connections 2024-08-19T12:40:34.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.424+0000 7efe144cc640 1 -- 172.21.15.26:0/1247118612 wait complete. 2024-08-19T12:40:34.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.424+0000 7efe144cc640 10 monclient: build_initial_monmap 2024-08-19T12:40:34.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.424+0000 7efe144cc640 10 monclient: monmap: 2024-08-19T12:40:34.434 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:34.434 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:34.434 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:34.434 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:34.434 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 1 librados: starting msgr at 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 1 librados: starting objecter 2024-08-19T12:40:34.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 1 Processor -- start 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 1 -- start start 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 1 librados: setting wanted keys 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 1 librados: calling monclient init 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 10 monclient: init 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 10 monclient: _add_conns ranks=[2,5,6,8,3,1,4,7,0] 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 10 monclient(hunting): picked mon.noname-b con 0x7efe0c06e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 10 monclient(hunting): picked mon.noname-h con 0x7efe0c0a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 10 monclient(hunting): picked mon.noname-d con 0x7efe0c0a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0c1349d0 con 0x7efe0c06e7b0 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0c0a2e00 con 0x7efe0c0a5920 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0c0a1bc0 con 0x7efe0c0a5d50 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:34.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe144cc640 10 monclient(hunting): authenticate will time out at 828.513843s 2024-08-19T12:40:34.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe12268640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7efe0c06e7b0 0x7efe0c0a48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58256/0 (socket says 172.21.15.26:58256) 2024-08-19T12:40:34.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.425+0000 7efe12268640 1 -- 172.21.15.26:0/2414785210 learned_addr learned my addr 172.21.15.26:0/2414785210 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:34.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe12a69640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.26:6792/0 conn(0x7efe0c0a5d50 legacy=0x7efe0c1d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 client.?.objecter ms_handle_connect 0x7efe0c0a5d50 2024-08-19T12:40:34.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe12268640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.26:6790/0 conn(0x7efe0c06e7b0 legacy=0x7efe0c0a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 client.?.objecter ms_handle_connect 0x7efe0c06e7b0 2024-08-19T12:40:34.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe11a67640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.104:6791/0 conn(0x7efe0c0a5920 legacy=0x7efe0c1d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3111169346 0 0) 0x7efe0c1349d0 con 0x7efe0c06e7b0 2024-08-19T12:40:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): my global_id is 25400 2024-08-19T12:40:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdf0004eb0 con 0x7efe0c06e7b0 2024-08-19T12:40:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 client.?.objecter ms_handle_connect 0x7efe0c0a5920 2024-08-19T12:40:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2706906466 0 0) 0x7efe0c0a1bc0 con 0x7efe0c0a5d50 2024-08-19T12:40:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): my global_id is 25638 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe0c1349d0 con 0x7efe0c0a5d50 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3708080873 0 0) 0x7efe0c0a2e00 con 0x7efe0c0a5920 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): my global_id is 25439 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe0c0a1bc0 con 0x7efe0c0a5920 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3236777906 0 0) 0x7efdf0004eb0 con 0x7efe0c06e7b0 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe0c0a2e00 con 0x7efe0c06e7b0 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efe08004d90 con 0x7efe0c06e7b0 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:34.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): dump: 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.444 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 230218161 0 0) 0x7efe0c1349d0 con 0x7efe0c0a5d50 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efdf0004eb0 con 0x7efe0c0a5d50 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efe04002030 con 0x7efe0c0a5d50 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient(hunting): dump: 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.445 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.446 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3264459534 0 0) 0x7efe0c0a2e00 con 0x7efe0c06e7b0 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.26:6792/0 conn(0x7efe0c0a5d50 legacy=0x7efe0c1d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.104:6791/0 conn(0x7efe0c0a5920 legacy=0x7efe0c1d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 monclient: found mon.b 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0c0a2b20 con 0x7efe0c06e7b0 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient: _finish_auth 0 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe144cc640 5 monclient: authenticate success, global_id 25400 2024-08-19T12:40:34.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe144cc640 10 monclient: _renew_subs 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe144cc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe144cc640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efe0c0a2810 con 0x7efe0c06e7b0 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe144cc640 10 client.25400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe144cc640 10 monclient: _renew_subs 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe144cc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe144cc640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efe0c0a51e0 con 0x7efe0c06e7b0 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe144cc640 1 librados: init done 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe080048a0 con 0x7efe0c06e7b0 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efe08001b00 con 0x7efe0c06e7b0 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe10a65640 4 set_mon_vals no callback set 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.428+0000 7efe02ffd640 10 monclient: dump: 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.450 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.429+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7efe080119c0 con 0x7efe0c06e7b0 2024-08-19T12:40:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.429+0000 7efe144cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.429+0000 7efe144cc640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efdf40065c0 con 0x7efdf4002860 2024-08-19T12:40:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.429+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(77..77 src has 1..77) ==== 6966+0+0 (unknown 2975105348 0 0) 0x7efe0804e940 con 0x7efe0c06e7b0 2024-08-19T12:40:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.429+0000 7efe02ffd640 10 client.25400.objecter ms_dispatch 0x7efe0c000b90 osd_map(77..77 src has 1..77) 2024-08-19T12:40:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.429+0000 7efe02ffd640 3 client.25400.objecter handle_osd_map got epochs [77,77] > 0 2024-08-19T12:40:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.429+0000 7efe02ffd640 3 client.25400.objecter handle_osd_map decoding full epoch 77 2024-08-19T12:40:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.429+0000 7efe02ffd640 20 client.25400.objecter dump_active .. 0 homeless 2024-08-19T12:40:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.430+0000 7efe12a69640 10 monclient: get_auth_request con 0x7efdf0041440 auth_method 0 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.431+0000 7efe11a67640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.26:6789/0 conn(0x7efdf4002860 legacy=0x7efdf4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 10 client.25400.objecter ms_handle_connect 0x7efdf4002860 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 10 client.25400.objecter resend_mon_ops 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713801531 0 0) 0x7efdf40065c0 con 0x7efdf4002860 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 10 monclient(hunting): my global_id is 26937 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe0c0a2e00 con 0x7efdf4002860 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 638411190 0 0) 0x7efe0c0a2e00 con 0x7efdf4002860 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efdf40065c0 con 0x7efdf4002860 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efdfc003d10 con 0x7efdf4002860 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 10 monclient: dump: 2024-08-19T12:40:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.452 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.453 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 10 client.25400.objecter ms_handle_connect 0x7efdf0041440 2024-08-19T12:40:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2313856622 0 0) 0x7efdf40065c0 con 0x7efdf4002860 2024-08-19T12:40:34.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efdf400a160 con 0x7efdf4002860 2024-08-19T12:40:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7efdfc004040 con 0x7efdf4002860 2024-08-19T12:40:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.26:6789/0 conn(0x7efdf4002860 legacy=0x7efdf4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.432+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.26:6789/0 conn(0x7efdf4002860 legacy=0x7efdf4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.439+0000 7efe144cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:34.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.439+0000 7efe144cc640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efdf4009e50 con 0x7efdf4009110 2024-08-19T12:40:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe11a67640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.26:6789/0 conn(0x7efdf4009110 legacy=0x7efdf40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 10 client.25400.objecter ms_handle_connect 0x7efdf4009110 2024-08-19T12:40:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 10 client.25400.objecter resend_mon_ops 2024-08-19T12:40:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2688585480 0 0) 0x7efdf4009e50 con 0x7efdf4009110 2024-08-19T12:40:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 10 monclient(hunting): my global_id is 26946 2024-08-19T12:40:34.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdf40065c0 con 0x7efdf4009110 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2916709079 0 0) 0x7efdf40065c0 con 0x7efdf4009110 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efdf4009e50 con 0x7efdf4009110 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efdfc005b00 con 0x7efdf4009110 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 10 monclient: dump: 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.457 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 138540298 0 0) 0x7efdf4009e50 con 0x7efdf4009110 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.441+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efdf400c3f0 con 0x7efdf4009110 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7546 (unknown 0 0 1206527626) 0x7efdfc005de0 con 0x7efdf4009110 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe02ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:34.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe02ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.26:6789/0 conn(0x7efdf4009110 legacy=0x7efdf40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe02ffd640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.26:6789/0 conn(0x7efdf4009110 legacy=0x7efdf40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:34.459 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:34.459 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:34.459 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:34.459 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:40:34.459 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:34.459 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 147484, 2024-08-19T12:40:34.459 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:34.459 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:34.460 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.461 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:34.462 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:34.463 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:34.464 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.465 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:34.466 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.467 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.469 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:34.470 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.471 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.472 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.473 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:34.474 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.475 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.476 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:34.477 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.478 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:34.478 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:34.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 10 librados: watch_flush enter 2024-08-19T12:40:34.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 10 librados: watch_flush exit 2024-08-19T12:40:34.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 20 client.25400.objecter shutdown clearing up homeless session... 2024-08-19T12:40:34.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 10 client.25400.objecter successfully canceled tick 2024-08-19T12:40:34.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.104:6800/899887506 conn(0x7efdf0041440 legacy=0x7efdf0043900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 10 monclient: shutdown 2024-08-19T12:40:34.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:34.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 1 -- 172.21.15.26:0/2414785210 >> v1:172.21.15.26:6790/0 conn(0x7efe0c06e7b0 legacy=0x7efe0c0a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe12a69640 1 -- 172.21.15.26:0/2414785210 reap_dead start 2024-08-19T12:40:34.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe12a69640 1 -- 172.21.15.26:0/2414785210 reap_dead start 2024-08-19T12:40:34.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 1 -- 172.21.15.26:0/2414785210 shutdown_connections 2024-08-19T12:40:34.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 1 -- 172.21.15.26:0/2414785210 >> 172.21.15.26:0/2414785210 conn(0x7efe0c0a8740 msgr2=0x7efe0c0aa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:34.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 1 -- 172.21.15.26:0/2414785210 shutdown_connections 2024-08-19T12:40:34.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 1 -- 172.21.15.26:0/2414785210 wait complete. 2024-08-19T12:40:34.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.442+0000 7efe144cc640 1 librados: shutdown 2024-08-19T12:40:34.482 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:34.482 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:40:34.482 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:40:34.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.579+0000 7f0052a60640 10 monclient: get_monmap_and_config 2024-08-19T12:40:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.579+0000 7f0052a60640 10 monclient: build_initial_monmap 2024-08-19T12:40:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.579+0000 7f0052a60640 10 monclient: monmap: 2024-08-19T12:40:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:34.581637+0000 2024-08-19T12:40:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:34.581637+0000 2024-08-19T12:40:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:34.614 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.580+0000 7f0052a60640 1 Processor -- start 2024-08-19T12:40:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.580+0000 7f0052a60640 1 -- start start 2024-08-19T12:40:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.580+0000 7f0052a60640 10 monclient: init 2024-08-19T12:40:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.580+0000 7f0052a60640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.580+0000 7f0052a60640 10 monclient: _add_conns ranks=[1,7,6,4,5,0,8,2,3] 2024-08-19T12:40:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.581+0000 7f0052a60640 10 monclient(hunting): picked mon.noname-f con 0x7f004c06e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.581+0000 7f0052a60640 10 monclient(hunting): picked mon.noname-i con 0x7f004c0a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.581+0000 7f0052a60640 10 monclient(hunting): picked mon.noname-d con 0x7f004c0a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.581+0000 7f0052a60640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004c136790 con 0x7f004c06e7b0 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.581+0000 7f0052a60640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004c09ddf0 con 0x7f004c0a5d50 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.581+0000 7f0052a60640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004c09e0d0 con 0x7f004c0a5920 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.581+0000 7f0052a60640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.581+0000 7f0052a60640 10 monclient(hunting): authenticate will time out at 828.669065s 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.581+0000 7f0050ffd640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f004c0a5d50 0x7f004c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39120/0 (socket says 172.21.15.26:39120) 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.581+0000 7f0050ffd640 1 -- 172.21.15.26:0/2162270011 learned_addr learned my addr 172.21.15.26:0/2162270011 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f0050ffd640 1 -- 172.21.15.26:0/2162270011 >> v1:172.21.15.26:6792/0 conn(0x7f004c0a5d50 legacy=0x7f004c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004b7fe640 1 -- 172.21.15.26:0/2162270011 >> v1:172.21.15.104:6792/0 conn(0x7f004c0a5920 legacy=0x7f004c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004bfff640 1 -- 172.21.15.26:0/2162270011 >> v1:172.21.15.104:6789/0 conn(0x7f004c06e7b0 legacy=0x7f004c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2617195630 0 0) 0x7f00440017d0 con 0x7f004c0a5d50 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient(hunting): my global_id is 25647 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0034003520 con 0x7f004c0a5d50 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2676487534 0 0) 0x7f004c09e0d0 con 0x7f004c0a5920 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient(hunting): my global_id is 134161 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00440017d0 con 0x7f004c0a5920 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3029214526 0 0) 0x7f004c136790 con 0x7f004c06e7b0 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient(hunting): my global_id is 26083 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f004c09e0d0 con 0x7f004c06e7b0 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3258940593 0 0) 0x7f0034003520 con 0x7f004c0a5d50 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 >> v1:172.21.15.104:6792/0 conn(0x7f004c0a5920 legacy=0x7f004c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 >> v1:172.21.15.104:6789/0 conn(0x7f004c06e7b0 legacy=0x7f004c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient: _finish_hunting 0 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 monclient: found mon.noname-d 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004c09e3b0 con 0x7f004c0a5d50 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient: _finish_auth 0 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient: _check_auth_tickets 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0044002030 con 0x7f004c0a5d50 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f0052a60640 5 monclient: authenticate success, global_id 25647 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f0052a60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient: dump: 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f0052a60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0044004cd0 con 0x7f004c0a5d50 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 1 -- 172.21.15.26:0/2162270011 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0044005200 con 0x7f004c0a5d50 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f0052a60640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f0052a60640 4 set_mon_vals no callback set 2024-08-19T12:40:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.583+0000 7f004affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.584+0000 7f004affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.584+0000 7f004affd640 10 monclient: dump: 2024-08-19T12:40:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.584+0000 7f0052a60640 10 monclient: shutdown 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.584+0000 7f0052a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.584+0000 7f0052a60640 1 -- 172.21.15.26:0/2162270011 >> v1:172.21.15.26:6792/0 conn(0x7f004c0a5d50 legacy=0x7f004c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.584+0000 7f0052a60640 1 -- 172.21.15.26:0/2162270011 shutdown_connections 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.584+0000 7f0052a60640 1 -- 172.21.15.26:0/2162270011 >> 172.21.15.26:0/2162270011 conn(0x7f004c0a8740 msgr2=0x7f004c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:34.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.584+0000 7f0052a60640 1 -- 172.21.15.26:0/2162270011 shutdown_connections 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.584+0000 7f0052a60640 1 -- 172.21.15.26:0/2162270011 wait complete. 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.585+0000 7f0052a60640 10 monclient: build_initial_monmap 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.585+0000 7f0052a60640 10 monclient: monmap: 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.585+0000 7f0052a60640 1 librados: starting msgr at 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.585+0000 7f0052a60640 1 librados: starting objecter 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.585+0000 7f0052a60640 1 Processor -- start 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.585+0000 7f0052a60640 1 -- start start 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.585+0000 7f0052a60640 1 librados: setting wanted keys 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.585+0000 7f0052a60640 1 librados: calling monclient init 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.585+0000 7f0052a60640 10 monclient: init 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f0052a60640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f0052a60640 10 monclient: _add_conns ranks=[5,8,4,6,1,2,0,7,3] 2024-08-19T12:40:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f0052a60640 10 monclient(hunting): picked mon.noname-h con 0x7f004c06e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f0052a60640 10 monclient(hunting): picked mon.noname-e con 0x7f004c0a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f0052a60640 10 monclient(hunting): picked mon.noname-c con 0x7f004c0a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f0052a60640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004c13b340 con 0x7f004c0a5d50 2024-08-19T12:40:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f0052a60640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004c1cf060 con 0x7f004c06e7b0 2024-08-19T12:40:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f0052a60640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004c138fe0 con 0x7f004c0a5920 2024-08-19T12:40:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f0052a60640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f0052a60640 10 monclient(hunting): authenticate will time out at 828.674149s 2024-08-19T12:40:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f0050ffd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f004c0a5d50 0x7f004c1d64a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:39156/0 (socket says 172.21.15.26:39156) 2024-08-19T12:40:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f004b7fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f004c0a5920 0x7f004c1ce950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50250/0 (socket says 172.21.15.26:50250) 2024-08-19T12:40:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.586+0000 7f0050ffd640 1 -- 172.21.15.26:0/2196133817 learned_addr learned my addr 172.21.15.26:0/2196133817 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f004b7fe640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.26:6793/0 conn(0x7f004c0a5920 legacy=0x7f004c1ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0050ffd640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.26:6791/0 conn(0x7f004c0a5d50 legacy=0x7f004c1d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 client.?.objecter ms_handle_connect 0x7f004c0a5920 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 client.?.objecter ms_handle_connect 0x7f004c0a5d50 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f004bfff640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.104:6791/0 conn(0x7f004c06e7b0 legacy=0x7f004c1ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 client.?.objecter ms_handle_connect 0x7f004c06e7b0 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543483644 0 0) 0x7f004c138fe0 con 0x7f004c0a5920 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): my global_id is 25235 2024-08-19T12:40:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0038004ee0 con 0x7f004c0a5920 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3224566478 0 0) 0x7f004c13b340 con 0x7f004c0a5d50 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): my global_id is 25663 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f004c138fe0 con 0x7f004c0a5d50 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3674717559 0 0) 0x7f004c1cf060 con 0x7f004c06e7b0 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): my global_id is 25448 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f004c13b340 con 0x7f004c06e7b0 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4199350053 0 0) 0x7f0038004ee0 con 0x7f004c0a5920 2024-08-19T12:40:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f004c1cf060 con 0x7f004c0a5920 2024-08-19T12:40:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0040003610 con 0x7f004c0a5920 2024-08-19T12:40:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:40:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): dump: 2024-08-19T12:40:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3554378131 0 0) 0x7f004c138fe0 con 0x7f004c0a5d50 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0038004ee0 con 0x7f004c0a5d50 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f00440051c0 con 0x7f004c0a5d50 2024-08-19T12:40:34.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): dump: 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.639 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1209126019 0 0) 0x7f004c13b340 con 0x7f004c06e7b0 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f004c138fe0 con 0x7f004c06e7b0 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f003c001cc0 con 0x7f004c06e7b0 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 10 monclient(hunting): dump: 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.640 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.588+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3290283655 0 0) 0x7f004c1cf060 con 0x7f004c0a5920 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.104:6791/0 conn(0x7f004c06e7b0 legacy=0x7f004c1ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.26:6791/0 conn(0x7f004c0a5d50 legacy=0x7f004c1d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 1 monclient: found mon.e 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004c13aaa0 con 0x7f004c0a5920 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 10 monclient: _finish_auth 0 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0052a60640 5 monclient: authenticate success, global_id 25235 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0052a60640 10 monclient: _renew_subs 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0052a60640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:34.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0052a60640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f004c13ad80 con 0x7f004c0a5920 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0052a60640 10 client.25235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0052a60640 10 monclient: _renew_subs 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0052a60640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0052a60640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f004c138550 con 0x7f004c0a5920 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0052a60640 1 librados: init done 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0040003b90 con 0x7f004c0a5920 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f00400040c0 con 0x7f004c0a5920 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f004a7fc640 4 set_mon_vals no callback set 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0048ff9640 10 monclient: dump: 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.643 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0052a60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.589+0000 7f0052a60640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f002c0065c0 con 0x7f002c002860 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.590+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f0040004260 con 0x7f004c0a5920 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.590+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(77..77 src has 1..77) ==== 6966+0+0 (unknown 2975105348 0 0) 0x7f004004e850 con 0x7f004c0a5920 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.590+0000 7f0048ff9640 10 client.25235.objecter ms_dispatch 0x7f004c000b90 osd_map(77..77 src has 1..77) 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.590+0000 7f0048ff9640 3 client.25235.objecter handle_osd_map got epochs [77,77] > 0 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.590+0000 7f0048ff9640 3 client.25235.objecter handle_osd_map decoding full epoch 77 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.590+0000 7f0048ff9640 20 client.25235.objecter dump_active .. 0 homeless 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.590+0000 7f0050ffd640 10 monclient: get_auth_request con 0x7f0038041920 auth_method 0 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 10 client.25235.objecter ms_handle_connect 0x7f002c002860 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 10 client.25235.objecter resend_mon_ops 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f004bfff640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.26:6790/0 conn(0x7f002c002860 legacy=0x7f002c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1552671383 0 0) 0x7f002c0065c0 con 0x7f002c002860 2024-08-19T12:40:34.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 10 monclient(hunting): my global_id is 25409 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f004c1cf060 con 0x7f002c002860 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1920052331 0 0) 0x7f004c1cf060 con 0x7f002c002860 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f002c0065c0 con 0x7f002c002860 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f003c003c50 con 0x7f002c002860 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 10 monclient: dump: 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.646 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4081644701 0 0) 0x7f002c0065c0 con 0x7f002c002860 2024-08-19T12:40:34.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f002c00a160 con 0x7f002c002860 2024-08-19T12:40:34.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 10 client.25235.objecter ms_handle_connect 0x7f0038041920 2024-08-19T12:40:34.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f003c003f00 con 0x7f002c002860 2024-08-19T12:40:34.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:34.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:34.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.26:6790/0 conn(0x7f002c002860 legacy=0x7f002c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.592+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.26:6790/0 conn(0x7f002c002860 legacy=0x7f002c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:34.651 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.651 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528217000}) 2024-08-19T12:40:34.651 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863376699}) 2024-08-19T12:40:34.651 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.651 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528217000}) 2024-08-19T12:40:34.651 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1716022423}) 2024-08-19T12:40:34.651 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.651 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528217000}) 2024-08-19T12:40:34.651 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1503529204}) 2024-08-19T12:40:34.651 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.651 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528217000}) 2024-08-19T12:40:34.651 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2797465753}) 2024-08-19T12:40:34.653 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.653 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528217000}) 2024-08-19T12:40:34.653 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3472343382}) 2024-08-19T12:40:34.653 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.653 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528217000}) 2024-08-19T12:40:34.654 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3242108412}) 2024-08-19T12:40:34.654 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.654 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528217000}) 2024-08-19T12:40:34.654 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3117805367}) 2024-08-19T12:40:34.654 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.654 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528217000}) 2024-08-19T12:40:34.654 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.597+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3749958225}) 2024-08-19T12:40:34.654 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.598+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.654 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.598+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=1917747916,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2037334906,mon_config_key=2171442413,monmap=1261126655,osd_metadata=2072018962,osd_pg_creating=3994032820,osd_snap=1167059780}) 2024-08-19T12:40:34.654 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.598+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=19,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=108901970,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=1257060107,mon_config_key=3689853235,monmap=1261126655,osd_metadata=3817445533,osd_pg_creating=3994032820,osd_snap=3669671379}) 2024-08-19T12:40:34.654 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.598+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.654 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.598+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=1917747916,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2037334906,mon_config_key=2171442413,monmap=1261126655,osd_metadata=2072018962,osd_pg_creating=3994032820,osd_snap=1167059780}) 2024-08-19T12:40:34.655 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.598+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=11} crc {logm=860164428,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2244450244,mon_config_key=3222896933,monmap=2932333919,osd_metadata=3470940196,osd_pg_creating=3994032820,osd_snap=131998043}) 2024-08-19T12:40:34.655 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.598+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.655 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.598+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=1917747916,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2037334906,mon_config_key=2171442413,monmap=1261126655,osd_metadata=2072018962,osd_pg_creating=3994032820,osd_snap=1167059780}) 2024-08-19T12:40:34.655 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.598+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=7} crc {logm=3833473958,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2244450244,mon_config_key=2171442413,monmap=1261126655,osd_metadata=751688751,osd_pg_creating=3994032820,osd_snap=608958682}) 2024-08-19T12:40:34.655 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.655 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=1917747916,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2037334906,mon_config_key=2171442413,monmap=1261126655,osd_metadata=2072018962,osd_pg_creating=3994032820,osd_snap=1167059780}) 2024-08-19T12:40:34.655 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9} crc {logm=1890990692,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2244450244,mon_config_key=2577291069,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=3994032820,osd_snap=3737864137}) 2024-08-19T12:40:34.655 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.655 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=1917747916,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2037334906,mon_config_key=2171442413,monmap=1261126655,osd_metadata=2072018962,osd_pg_creating=3994032820,osd_snap=1167059780}) 2024-08-19T12:40:34.655 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8} crc {logm=4266453157,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=1311991692,mon_config_key=2171442413,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=3994032820,osd_snap=3924052942}) 2024-08-19T12:40:34.655 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.655 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=1917747916,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2037334906,mon_config_key=2171442413,monmap=1261126655,osd_metadata=2072018962,osd_pg_creating=3994032820,osd_snap=1167059780}) 2024-08-19T12:40:34.656 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=81493846,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2244450244,mon_config_key=3918074939,monmap=1261126655,osd_metadata=751688751,osd_pg_creating=3994032820,osd_snap=3737864137}) 2024-08-19T12:40:34.656 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.656 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=1917747916,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2037334906,mon_config_key=2171442413,monmap=1261126655,osd_metadata=2072018962,osd_pg_creating=3994032820,osd_snap=1167059780}) 2024-08-19T12:40:34.656 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=20,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=9} crc {logm=1536424249,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2244450244,mon_config_key=3397144592,monmap=1261126655,osd_metadata=1059839136,osd_pg_creating=3994032820,osd_snap=2414122996}) 2024-08-19T12:40:34.656 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.656 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=1917747916,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2037334906,mon_config_key=2171442413,monmap=1261126655,osd_metadata=2072018962,osd_pg_creating=3994032820,osd_snap=1167059780}) 2024-08-19T12:40:34.656 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.599+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=41,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=12} crc {logm=2458269407,mdsmap=3707151070,mgr=2385552920,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2244450244,mon_config_key=196102489,monmap=1261126655,osd_metadata=847310766,osd_pg_creating=3994032820,osd_snap=2892072623}) 2024-08-19T12:40:34.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.600+0000 7f0052a60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:34.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.600+0000 7f0052a60640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f002c009e50 con 0x7f002c009110 2024-08-19T12:40:34.658 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.658 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=1822077521,osdmap=1023911815}) 2024-08-19T12:40:34.658 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=499763642,osdmap=2582752783}) 2024-08-19T12:40:34.659 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.659 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=1822077521,osdmap=1023911815}) 2024-08-19T12:40:34.659 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=3635964529,osdmap=1043055373}) 2024-08-19T12:40:34.659 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.659 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=1822077521,osdmap=1023911815}) 2024-08-19T12:40:34.659 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=499763643,osdmap=3484962472}) 2024-08-19T12:40:34.659 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.659 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=1822077521,osdmap=1023911815}) 2024-08-19T12:40:34.659 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=499763642,osdmap=4258523825}) 2024-08-19T12:40:34.659 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.659 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=1822077521,osdmap=1023911815}) 2024-08-19T12:40:34.660 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=499763642,osdmap=1931943797}) 2024-08-19T12:40:34.660 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.660 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=1822077521,osdmap=1023911815}) 2024-08-19T12:40:34.660 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=499763642,osdmap=4023295086}) 2024-08-19T12:40:34.660 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.660 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=1822077521,osdmap=1023911815}) 2024-08-19T12:40:34.660 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=499763642,osdmap=1215957165}) 2024-08-19T12:40:34.660 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.660 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=1822077521,osdmap=1023911815}) 2024-08-19T12:40:34.660 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.601+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=499763642,osdmap=1038134325}) 2024-08-19T12:40:34.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.603+0000 7f0048ff9640 10 client.25235.objecter ms_handle_connect 0x7f002c009110 2024-08-19T12:40:34.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.603+0000 7f0048ff9640 10 client.25235.objecter resend_mon_ops 2024-08-19T12:40:34.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.603+0000 7f004bfff640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.26:6790/0 conn(0x7f002c009110 legacy=0x7f002c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.661 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.603+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.661 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.603+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=1331789441}) 2024-08-19T12:40:34.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.603+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3871544465 0 0) 0x7f002c009e50 con 0x7f002c009110 2024-08-19T12:40:34.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.603+0000 7f0048ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.662 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.603+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=31} crc {osdmap=1049892414}) 2024-08-19T12:40:34.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.603+0000 7f0048ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.603+0000 7f0048ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.603+0000 7f0048ff9640 10 monclient(hunting): my global_id is 25418 2024-08-19T12:40:34.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.603+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f002c0065c0 con 0x7f002c009110 2024-08-19T12:40:34.664 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.603+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.664 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.603+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=1331789441}) 2024-08-19T12:40:34.664 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.603+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=31} crc {osdmap=1416056220}) 2024-08-19T12:40:34.664 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.603+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.664 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.603+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=1331789441}) 2024-08-19T12:40:34.664 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.603+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=33} crc {osdmap=414837209}) 2024-08-19T12:40:34.664 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.603+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.664 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.603+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=1331789441}) 2024-08-19T12:40:34.665 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=33} crc {osdmap=1441649038}) 2024-08-19T12:40:34.665 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.665 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=1331789441}) 2024-08-19T12:40:34.665 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=32} crc {osdmap=637233902}) 2024-08-19T12:40:34.665 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.665 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=1331789441}) 2024-08-19T12:40:34.665 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=32} crc {osdmap=559637637}) 2024-08-19T12:40:34.665 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 866060279 0 0) 0x7f002c0065c0 con 0x7f002c009110 2024-08-19T12:40:34.666 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=1331789441}) 2024-08-19T12:40:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f002c009e50 con 0x7f002c009110 2024-08-19T12:40:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f003c004420 con 0x7f002c009110 2024-08-19T12:40:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.666 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=32} crc {osdmap=637233902}) 2024-08-19T12:40:34.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:34.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 10 monclient: dump: 2024-08-19T12:40:34.667 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.667 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.667 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.667 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.667 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.667 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.667 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.667 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.668 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.668 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.668 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.668 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.668 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.668 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.668 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.668 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.668 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:40:34.668 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=1331789441}) 2024-08-19T12:40:34.669 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:40:34.604+0000 7fb087852640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=32} crc {osdmap=3789299151}) 2024-08-19T12:40:34.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4059237323 0 0) 0x7f002c009e50 con 0x7f002c009110 2024-08-19T12:40:34.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f002c00c3a0 con 0x7f002c009110 2024-08-19T12:40:34.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 2969475140) 0x7f003c007b80 con 0x7f002c009110 2024-08-19T12:40:34.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:34.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:34.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.26:6790/0 conn(0x7f002c009110 legacy=0x7f002c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.604+0000 7f0048ff9640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.26:6790/0 conn(0x7f002c009110 legacy=0x7f002c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:34.670 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:34.670 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:34.670 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:34.670 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:34.670 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:34.670 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 147659, 2024-08-19T12:40:34.670 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:34.670 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:34.670 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:34.671 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:34.672 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:34.674 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:34.675 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:34.677 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.678 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.681 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:34.682 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:34.686 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.688 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.689 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:34.690 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:34.690 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:34.690 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.690 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.690 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:34.690 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:34.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 10 librados: watch_flush enter 2024-08-19T12:40:34.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 10 librados: watch_flush exit 2024-08-19T12:40:34.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 20 client.25235.objecter shutdown clearing up homeless session... 2024-08-19T12:40:34.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 10 client.25235.objecter successfully canceled tick 2024-08-19T12:40:34.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.104:6800/899887506 conn(0x7f0038041920 legacy=0x7f0038043de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 10 monclient: shutdown 2024-08-19T12:40:34.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:34.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 1 -- 172.21.15.26:0/2196133817 >> v1:172.21.15.26:6793/0 conn(0x7f004c0a5920 legacy=0x7f004c1ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0050ffd640 1 -- 172.21.15.26:0/2196133817 reap_dead start 2024-08-19T12:40:34.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0050ffd640 1 -- 172.21.15.26:0/2196133817 reap_dead start 2024-08-19T12:40:34.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 1 -- 172.21.15.26:0/2196133817 shutdown_connections 2024-08-19T12:40:34.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 1 -- 172.21.15.26:0/2196133817 >> 172.21.15.26:0/2196133817 conn(0x7f004c0a8740 msgr2=0x7f004c135170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:34.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 1 -- 172.21.15.26:0/2196133817 shutdown_connections 2024-08-19T12:40:34.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 1 -- 172.21.15.26:0/2196133817 wait complete. 2024-08-19T12:40:34.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.605+0000 7f0052a60640 1 librados: shutdown 2024-08-19T12:40:34.693 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:34.693 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:40:34.693 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:40:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.791+0000 7f2360eda640 10 monclient: get_monmap_and_config 2024-08-19T12:40:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.791+0000 7f2360eda640 10 monclient: build_initial_monmap 2024-08-19T12:40:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.791+0000 7f2360eda640 10 monclient: monmap: 2024-08-19T12:40:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:34.793340+0000 2024-08-19T12:40:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:34.793340+0000 2024-08-19T12:40:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:34.795 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:34.795 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 1 Processor -- start 2024-08-19T12:40:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 1 -- start start 2024-08-19T12:40:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 10 monclient: init 2024-08-19T12:40:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 10 monclient: _add_conns ranks=[6,1,2,8,5,0,4,3,7] 2024-08-19T12:40:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 10 monclient(hunting): picked mon.noname-d con 0x7f235c06e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 10 monclient(hunting): picked mon.noname-f con 0x7f235c0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 10 monclient(hunting): picked mon.noname-b con 0x7f235c0a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f235c136790 con 0x7f235c0a5920 2024-08-19T12:40:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f235c09ddf0 con 0x7f235c0a5d50 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f235c09e0d0 con 0x7f235c06e7b0 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f2360eda640 10 monclient(hunting): authenticate will time out at 828.880853s 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f235a59c640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f235c06e7b0 0x7f235c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39134/0 (socket says 172.21.15.26:39134) 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f235ad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f235c0a5d50 0x7f235c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58292/0 (socket says 172.21.15.26:58292) 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.792+0000 7f235a59c640 1 -- 172.21.15.26:0/3944407510 learned_addr learned my addr 172.21.15.26:0/3944407510 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235a59c640 1 -- 172.21.15.26:0/3944407510 >> v1:172.21.15.26:6792/0 conn(0x7f235c06e7b0 legacy=0x7f235c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235ad9d640 1 -- 172.21.15.26:0/3944407510 >> v1:172.21.15.26:6790/0 conn(0x7f235c0a5d50 legacy=0x7f235c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f2359d9b640 1 -- 172.21.15.26:0/3944407510 >> v1:172.21.15.104:6789/0 conn(0x7f235c0a5920 legacy=0x7f235c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3776141623 0 0) 0x7f235c09e0d0 con 0x7f235c06e7b0 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient(hunting): my global_id is 25656 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2344003520 con 0x7f235c06e7b0 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 482010696 0 0) 0x7f23540017d0 con 0x7f235c0a5d50 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient(hunting): my global_id is 25427 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f235c09e0d0 con 0x7f235c0a5d50 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 999346573 0 0) 0x7f235c136790 con 0x7f235c0a5920 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient(hunting): my global_id is 26092 2024-08-19T12:40:34.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23540017d0 con 0x7f235c0a5920 2024-08-19T12:40:34.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 40748972 0 0) 0x7f235c09e0d0 con 0x7f235c0a5d50 2024-08-19T12:40:34.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 >> v1:172.21.15.26:6792/0 conn(0x7f235c06e7b0 legacy=0x7f235c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 >> v1:172.21.15.104:6789/0 conn(0x7f235c0a5920 legacy=0x7f235c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient: _finish_hunting 0 2024-08-19T12:40:34.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 monclient: found mon.noname-b 2024-08-19T12:40:34.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:34.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:40:34.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235c09e3b0 con 0x7f235c0a5d50 2024-08-19T12:40:34.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient: _finish_auth 0 2024-08-19T12:40:34.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient: _check_auth_tickets 2024-08-19T12:40:34.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2354002010 con 0x7f235c0a5d50 2024-08-19T12:40:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f2360eda640 5 monclient: authenticate success, global_id 25427 2024-08-19T12:40:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f2360eda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient: dump: 2024-08-19T12:40:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f2360eda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2354004be0 con 0x7f235c0a5d50 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 1 -- 172.21.15.26:0/3944407510 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2354005150 con 0x7f235c0a5d50 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f2360eda640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f2360eda640 4 set_mon_vals no callback set 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.795+0000 7f235959a640 10 monclient: dump: 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.807 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.796+0000 7f2360eda640 10 monclient: shutdown 2024-08-19T12:40:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.796+0000 7f2360eda640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.796+0000 7f2360eda640 1 -- 172.21.15.26:0/3944407510 >> v1:172.21.15.26:6790/0 conn(0x7f235c0a5d50 legacy=0x7f235c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.796+0000 7f2360eda640 1 -- 172.21.15.26:0/3944407510 shutdown_connections 2024-08-19T12:40:34.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.796+0000 7f2360eda640 1 -- 172.21.15.26:0/3944407510 >> 172.21.15.26:0/3944407510 conn(0x7f235c0a8740 msgr2=0x7f235c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:34.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.796+0000 7f2360eda640 1 -- 172.21.15.26:0/3944407510 shutdown_connections 2024-08-19T12:40:34.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.796+0000 7f2360eda640 1 -- 172.21.15.26:0/3944407510 wait complete. 2024-08-19T12:40:34.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.796+0000 7f2360eda640 10 monclient: build_initial_monmap 2024-08-19T12:40:34.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.796+0000 7f2360eda640 10 monclient: monmap: 2024-08-19T12:40:34.808 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:34.808 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:34.808 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:34.808 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:34.808 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:34.808 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 1 librados: starting msgr at 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 1 librados: starting objecter 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 1 Processor -- start 2024-08-19T12:40:34.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 1 -- start start 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 1 librados: setting wanted keys 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 1 librados: calling monclient init 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 10 monclient: init 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 10 monclient: _add_conns ranks=[8,1,7,4,3,5,2,0,6] 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 10 monclient(hunting): picked mon.noname-e con 0x7f235c06e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 10 monclient(hunting): picked mon.noname-f con 0x7f235c0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 10 monclient(hunting): picked mon.noname-i con 0x7f235c0a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f235c134600 con 0x7f235c0a5920 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f235c142e30 con 0x7f235c0a5d50 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f235c1cf060 con 0x7f235c06e7b0 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f2360eda640 10 monclient(hunting): authenticate will time out at 828.885774s 2024-08-19T12:40:34.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f235a59c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f235c06e7b0 0x7f235c1ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50260/0 (socket says 172.21.15.26:50260) 2024-08-19T12:40:34.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.797+0000 7f235a59c640 1 -- 172.21.15.26:0/274832628 learned_addr learned my addr 172.21.15.26:0/274832628 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:34.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f235a59c640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.26:6793/0 conn(0x7f235c06e7b0 legacy=0x7f235c1ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 client.?.objecter ms_handle_connect 0x7f235c06e7b0 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f2359d9b640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.104:6789/0 conn(0x7f235c0a5920 legacy=0x7f235c1ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 client.?.objecter ms_handle_connect 0x7f235c0a5920 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 client.?.objecter ms_handle_connect 0x7f235c0a5d50 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f235ad9d640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.104:6792/0 conn(0x7f235c0a5d50 legacy=0x7f235c1d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3787011654 0 0) 0x7f235c1cf060 con 0x7f235c06e7b0 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): my global_id is 25244 2024-08-19T12:40:34.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f233c004ee0 con 0x7f235c06e7b0 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4185562856 0 0) 0x7f235c09e3b0 con 0x7f235c0a5d50 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): my global_id is 134170 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f235c1cf060 con 0x7f235c0a5d50 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1956082563 0 0) 0x7f23540017d0 con 0x7f235c0a5920 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): my global_id is 26101 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f235c09e3b0 con 0x7f235c0a5920 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 372104157 0 0) 0x7f233c004ee0 con 0x7f235c06e7b0 2024-08-19T12:40:34.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23540017d0 con 0x7f235c06e7b0 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f234c0019d0 con 0x7f235c06e7b0 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): dump: 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.821 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2379387978 0 0) 0x7f235c1cf060 con 0x7f235c0a5d50 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f233c004ee0 con 0x7f235c0a5d50 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2354002070 con 0x7f235c0a5d50 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:34.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient(hunting): dump: 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.823 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 539203119 0 0) 0x7f23540017d0 con 0x7f235c06e7b0 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.104:6792/0 conn(0x7f235c0a5d50 legacy=0x7f235c1d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.104:6789/0 conn(0x7f235c0a5920 legacy=0x7f235c1ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient: _finish_hunting 0 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 monclient: found mon.e 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235c13ad20 con 0x7f235c06e7b0 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient: _finish_auth 0 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 10 monclient: _check_auth_tickets 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f234affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f2360eda640 5 monclient: authenticate success, global_id 25244 2024-08-19T12:40:34.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f2360eda640 10 monclient: _renew_subs 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f2360eda640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.800+0000 7f2360eda640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f235c13b000 con 0x7f235c06e7b0 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f2360eda640 10 client.25244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f2360eda640 10 monclient: _renew_subs 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f2360eda640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f2360eda640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f235c139ae0 con 0x7f235c06e7b0 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f2360eda640 1 librados: init done 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f234c004dd0 con 0x7f235c06e7b0 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f234affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f234c0052c0 con 0x7f235c06e7b0 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f234affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f2358d99640 4 set_mon_vals no callback set 2024-08-19T12:40:34.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f234affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f234affd640 10 monclient: dump: 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.826 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f2360eda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f2360eda640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f23400065c0 con 0x7f2340002860 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.801+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f234c011720 con 0x7f235c06e7b0 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.802+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(77..77 src has 1..77) ==== 6966+0+0 (unknown 2975105348 0 0) 0x7f234c04e800 con 0x7f235c06e7b0 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.802+0000 7f234affd640 10 client.25244.objecter ms_dispatch 0x7f235c000b90 osd_map(77..77 src has 1..77) 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.802+0000 7f234affd640 3 client.25244.objecter handle_osd_map got epochs [77,77] > 0 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.802+0000 7f234affd640 3 client.25244.objecter handle_osd_map decoding full epoch 77 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.802+0000 7f234affd640 20 client.25244.objecter dump_active .. 0 homeless 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.802+0000 7f235ad9d640 10 monclient: get_auth_request con 0x7f233c041510 auth_method 0 2024-08-19T12:40:34.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 10 client.25244.objecter ms_handle_connect 0x7f2340002860 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f2359d9b640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.26:6791/0 conn(0x7f2340002860 legacy=0x7f2340004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 10 client.25244.objecter resend_mon_ops 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2480756312 0 0) 0x7f23400065c0 con 0x7f2340002860 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 10 monclient(hunting): my global_id is 25672 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23540017d0 con 0x7f2340002860 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1723205202 0 0) 0x7f23540017d0 con 0x7f2340002860 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23400065c0 con 0x7f2340002860 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2350003e10 con 0x7f2340002860 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 10 monclient: dump: 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 395641501 0 0) 0x7f23400065c0 con 0x7f2340002860 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f234000a160 con 0x7f2340002860 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 10 client.25244.objecter ms_handle_connect 0x7f233c041510 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f23500040c0 con 0x7f2340002860 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.26:6791/0 conn(0x7f2340002860 legacy=0x7f2340004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.804+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.26:6791/0 conn(0x7f2340002860 legacy=0x7f2340004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.811+0000 7f2360eda640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.811+0000 7f2360eda640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2340009e70 con 0x7f2340009130 2024-08-19T12:40:34.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f2359d9b640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.26:6791/0 conn(0x7f2340009130 legacy=0x7f23400095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 10 client.25244.objecter ms_handle_connect 0x7f2340009130 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 10 client.25244.objecter resend_mon_ops 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1917942780 0 0) 0x7f2340009e70 con 0x7f2340009130 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 10 monclient(hunting): my global_id is 25681 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23400065c0 con 0x7f2340009130 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2936346495 0 0) 0x7f23400065c0 con 0x7f2340009130 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2340009e70 con 0x7f2340009130 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2350005b00 con 0x7f2340009130 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:34.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 10 monclient: dump: 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.832 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.833 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.833 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3231190769 0 0) 0x7f2340009e70 con 0x7f2340009130 2024-08-19T12:40:34.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.813+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f234000c410 con 0x7f2340009130 2024-08-19T12:40:34.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7878 (unknown 0 0 1639564684) 0x7f2350007ed0 con 0x7f2340009130 2024-08-19T12:40:34.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f234affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:34.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f234affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:34.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.26:6791/0 conn(0x7f2340009130 legacy=0x7f23400095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f234affd640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.26:6791/0 conn(0x7f2340009130 legacy=0x7f23400095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:34.833 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 147843, 2024-08-19T12:40:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:34.834 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:34.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:34.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:34.837 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:34.838 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:34.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:34.840 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.841 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:34.844 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:34.851 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.852 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:34.853 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:40:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:34.854 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:34.854 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:34.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 10 librados: watch_flush enter 2024-08-19T12:40:34.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 10 librados: watch_flush exit 2024-08-19T12:40:34.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 20 client.25244.objecter shutdown clearing up homeless session... 2024-08-19T12:40:34.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 10 client.25244.objecter successfully canceled tick 2024-08-19T12:40:34.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.104:6800/899887506 conn(0x7f233c041510 legacy=0x7f233c0439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 10 monclient: shutdown 2024-08-19T12:40:34.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:34.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 1 -- 172.21.15.26:0/274832628 >> v1:172.21.15.26:6793/0 conn(0x7f235c06e7b0 legacy=0x7f235c1ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f235ad9d640 1 -- 172.21.15.26:0/274832628 reap_dead start 2024-08-19T12:40:34.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f235ad9d640 1 -- 172.21.15.26:0/274832628 reap_dead start 2024-08-19T12:40:34.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 1 -- 172.21.15.26:0/274832628 shutdown_connections 2024-08-19T12:40:34.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 1 -- 172.21.15.26:0/274832628 >> 172.21.15.26:0/274832628 conn(0x7f235c0a8740 msgr2=0x7f235c0ab030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:34.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 1 -- 172.21.15.26:0/274832628 shutdown_connections 2024-08-19T12:40:34.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 1 -- 172.21.15.26:0/274832628 wait complete. 2024-08-19T12:40:34.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.814+0000 7f2360eda640 1 librados: shutdown 2024-08-19T12:40:34.857 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:34.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:40:34.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:40:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.969+0000 7f31e584f640 10 monclient: get_monmap_and_config 2024-08-19T12:40:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.969+0000 7f31e584f640 10 monclient: build_initial_monmap 2024-08-19T12:40:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.969+0000 7f31e584f640 10 monclient: monmap: 2024-08-19T12:40:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:34.970929+0000 2024-08-19T12:40:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:34.970929+0000 2024-08-19T12:40:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:34.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:34.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:34.972 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:34.972 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:34.972 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:34.972 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:34.972 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:34.972 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:34.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.969+0000 7f31e584f640 1 Processor -- start 2024-08-19T12:40:34.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.969+0000 7f31e584f640 1 -- start start 2024-08-19T12:40:34.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.969+0000 7f31e584f640 10 monclient: init 2024-08-19T12:40:34.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.970+0000 7f31e584f640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:34.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.970+0000 7f31e584f640 10 monclient: _add_conns ranks=[1,3,2,8,5,4,0,7,6] 2024-08-19T12:40:34.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.970+0000 7f31e584f640 10 monclient(hunting): picked mon.noname-f con 0x7f31e0073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:34.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.970+0000 7f31e584f640 10 monclient(hunting): picked mon.noname-g con 0x7f31e00a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:34.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.970+0000 7f31e584f640 10 monclient(hunting): picked mon.noname-b con 0x7f31e013f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:34.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.970+0000 7f31e584f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e009dae0 con 0x7f31e0073470 2024-08-19T12:40:34.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.970+0000 7f31e584f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e0136910 con 0x7f31e013f2e0 2024-08-19T12:40:34.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.970+0000 7f31e584f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e009dd50 con 0x7f31e00a57a0 2024-08-19T12:40:34.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.970+0000 7f31e584f640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:34.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.970+0000 7f31e584f640 10 monclient(hunting): authenticate will time out at 829.058279s 2024-08-19T12:40:34.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.970+0000 7f31df7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f31e013f2e0 0x7f31e01416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58300/0 (socket says 172.21.15.26:58300) 2024-08-19T12:40:34.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.970+0000 7f31df7fe640 1 -- 172.21.15.26:0/4247447626 learned_addr learned my addr 172.21.15.26:0/4247447626 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:34.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31df7fe640 1 -- 172.21.15.26:0/4247447626 >> v1:172.21.15.26:6790/0 conn(0x7f31e013f2e0 legacy=0x7f31e01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31de7fc640 1 -- 172.21.15.26:0/4247447626 >> v1:172.21.15.104:6790/0 conn(0x7f31e00a57a0 legacy=0x7f31e00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1206353774 0 0) 0x7f31e0136910 con 0x7f31e013f2e0 2024-08-19T12:40:34.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 10 monclient(hunting): my global_id is 25436 2024-08-19T12:40:34.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31c8003520 con 0x7f31e013f2e0 2024-08-19T12:40:34.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31deffd640 1 -- 172.21.15.26:0/4247447626 >> v1:172.21.15.104:6789/0 conn(0x7f31e0073470 legacy=0x7f31e0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3974013277 0 0) 0x7f31e009dae0 con 0x7f31e0073470 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 10 monclient(hunting): my global_id is 26110 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31e0136910 con 0x7f31e0073470 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 497430360 0 0) 0x7f31e009dd50 con 0x7f31e00a57a0 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 10 monclient(hunting): my global_id is 25572 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31e009dae0 con 0x7f31e00a57a0 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2052108806 0 0) 0x7f31c8003520 con 0x7f31e013f2e0 2024-08-19T12:40:34.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.972+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 >> v1:172.21.15.104:6790/0 conn(0x7f31e00a57a0 legacy=0x7f31e00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 >> v1:172.21.15.104:6789/0 conn(0x7f31e0073470 legacy=0x7f31e0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 10 monclient: _finish_hunting 0 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 1 monclient: found mon.noname-b 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e009e030 con 0x7f31e013f2e0 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 10 monclient: _finish_auth 0 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 10 monclient: _check_auth_tickets 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f31d8001e60 con 0x7f31e013f2e0 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 10 monclient: dump: 2024-08-19T12:40:34.980 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.981 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31e584f640 5 monclient: authenticate success, global_id 25436 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31e584f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f31d8004c20 con 0x7f31e013f2e0 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 1 -- 172.21.15.26:0/4247447626 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f31d8005190 con 0x7f31e013f2e0 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31e584f640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31e584f640 4 set_mon_vals no callback set 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31ddffb640 10 monclient: dump: 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.982 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.983 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31e584f640 10 monclient: shutdown 2024-08-19T12:40:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31e584f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31e584f640 1 -- 172.21.15.26:0/4247447626 >> v1:172.21.15.26:6790/0 conn(0x7f31e013f2e0 legacy=0x7f31e01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31e584f640 1 -- 172.21.15.26:0/4247447626 shutdown_connections 2024-08-19T12:40:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31e584f640 1 -- 172.21.15.26:0/4247447626 >> 172.21.15.26:0/4247447626 conn(0x7f31e00abff0 msgr2=0x7f31e00ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31e584f640 1 -- 172.21.15.26:0/4247447626 shutdown_connections 2024-08-19T12:40:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.973+0000 7f31e584f640 1 -- 172.21.15.26:0/4247447626 wait complete. 2024-08-19T12:40:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.974+0000 7f31e584f640 10 monclient: build_initial_monmap 2024-08-19T12:40:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.974+0000 7f31e584f640 10 monclient: monmap: 2024-08-19T12:40:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.974+0000 7f31e584f640 1 librados: starting msgr at 2024-08-19T12:40:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.974+0000 7f31e584f640 1 librados: starting objecter 2024-08-19T12:40:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.974+0000 7f31e584f640 1 Processor -- start 2024-08-19T12:40:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.974+0000 7f31e584f640 1 -- start start 2024-08-19T12:40:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.974+0000 7f31e584f640 1 librados: setting wanted keys 2024-08-19T12:40:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.974+0000 7f31e584f640 1 librados: calling monclient init 2024-08-19T12:40:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.974+0000 7f31e584f640 10 monclient: init 2024-08-19T12:40:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.975+0000 7f31e584f640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.975+0000 7f31e584f640 10 monclient: _add_conns ranks=[4,2,0,7,5,6,8,3,1] 2024-08-19T12:40:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.975+0000 7f31e584f640 10 monclient(hunting): picked mon.noname-c con 0x7f31e0073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.975+0000 7f31e584f640 10 monclient(hunting): picked mon.noname-b con 0x7f31e00a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.975+0000 7f31e584f640 10 monclient(hunting): picked mon.noname-a con 0x7f31e013f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.975+0000 7f31e584f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e0137540 con 0x7f31e013f2e0 2024-08-19T12:40:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.975+0000 7f31e584f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e0137260 con 0x7f31e00a57a0 2024-08-19T12:40:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.975+0000 7f31e584f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e00a46e0 con 0x7f31e0073470 2024-08-19T12:40:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.975+0000 7f31e584f640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.975+0000 7f31de7fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f31e00a57a0 0x7f31e01d3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58314/0 (socket says 172.21.15.26:58314) 2024-08-19T12:40:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.975+0000 7f31e584f640 10 monclient(hunting): authenticate will time out at 829.063340s 2024-08-19T12:40:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.975+0000 7f31de7fc640 1 -- 172.21.15.26:0/3567017024 learned_addr learned my addr 172.21.15.26:0/3567017024 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31deffd640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.26:6791/0 conn(0x7f31e0073470 legacy=0x7f31e01ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31de7fc640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.26:6790/0 conn(0x7f31e00a57a0 legacy=0x7f31e01d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 client.?.objecter ms_handle_connect 0x7f31e0073470 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 client.?.objecter ms_handle_connect 0x7f31e00a57a0 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 client.?.objecter ms_handle_connect 0x7f31e013f2e0 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31df7fe640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.26:6789/0 conn(0x7f31e013f2e0 legacy=0x7f31e01d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1398227324 0 0) 0x7f31e00a46e0 con 0x7f31e0073470 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): my global_id is 25690 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31cc004eb0 con 0x7f31e0073470 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506089228 0 0) 0x7f31e0137260 con 0x7f31e00a57a0 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): my global_id is 25445 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31e00a46e0 con 0x7f31e00a57a0 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 864348074 0 0) 0x7f31e0137540 con 0x7f31e013f2e0 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): my global_id is 26955 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31e0137260 con 0x7f31e013f2e0 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3074975939 0 0) 0x7f31e00a46e0 con 0x7f31e00a57a0 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31e0137540 con 0x7f31e00a57a0 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1711322897 0 0) 0x7f31cc004eb0 con 0x7f31e0073470 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31e00a46e0 con 0x7f31e0073470 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f31d4003610 con 0x7f31e00a57a0 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.977+0000 7f31c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient(hunting): dump: 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 63303712 0 0) 0x7f31e0137260 con 0x7f31e013f2e0 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31cc004eb0 con 0x7f31e013f2e0 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f31d0001d00 con 0x7f31e0073470 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient(hunting): dump: 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f31d80016a0 con 0x7f31e013f2e0 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient(hunting): dump: 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:34.999 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4004521451 0 0) 0x7f31e0137540 con 0x7f31e00a57a0 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.26:6791/0 conn(0x7f31e0073470 legacy=0x7f31e01ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.26:6789/0 conn(0x7f31e013f2e0 legacy=0x7f31e01d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient: _finish_hunting 0 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 1 monclient: found mon.b 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e00a3180 con 0x7f31e00a57a0 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient: _finish_auth 0 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient: _check_auth_tickets 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31e584f640 5 monclient: authenticate success, global_id 25445 2024-08-19T12:40:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31e584f640 10 monclient: _renew_subs 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31e584f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31e584f640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f31e00a3410 con 0x7f31e00a57a0 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31e584f640 10 client.25445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31e584f640 10 monclient: _renew_subs 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31e584f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31e584f640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f31e01cec60 con 0x7f31e00a57a0 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31e584f640 1 librados: init done 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f31d4003b60 con 0x7f31e00a57a0 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f31d4004050 con 0x7f31e00a57a0 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31dd7fa640 4 set_mon_vals no callback set 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.978+0000 7f31c7fff640 10 monclient: dump: 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.002 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.979+0000 7f31e584f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.979+0000 7f31e584f640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f31c00065c0 con 0x7f31c0002860 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.979+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f31d40042a0 con 0x7f31e00a57a0 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.979+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(78..78 src has 1..78) ==== 6992+0+0 (unknown 3025720083 0 0) 0x7f31d404e890 con 0x7f31e00a57a0 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.979+0000 7f31c7fff640 10 client.25445.objecter ms_dispatch 0x7f31e0000b90 osd_map(78..78 src has 1..78) 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.979+0000 7f31c7fff640 3 client.25445.objecter handle_osd_map got epochs [78,78] > 0 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.979+0000 7f31c7fff640 3 client.25445.objecter handle_osd_map decoding full epoch 78 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.979+0000 7f31c7fff640 20 client.25445.objecter dump_active .. 0 homeless 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.979+0000 7f31df7fe640 10 monclient: get_auth_request con 0x7f31cc041ae0 auth_method 0 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 10 client.25445.objecter ms_handle_connect 0x7f31c0002860 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 10 client.25445.objecter resend_mon_ops 2024-08-19T12:40:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31deffd640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.26:6792/0 conn(0x7f31c0002860 legacy=0x7f31c0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182996567 0 0) 0x7f31c00065c0 con 0x7f31c0002860 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 10 monclient(hunting): my global_id is 25665 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31e0137540 con 0x7f31c0002860 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3146036507 0 0) 0x7f31e0137540 con 0x7f31c0002860 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31c00065c0 con 0x7f31c0002860 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f31d0004060 con 0x7f31c0002860 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 10 monclient: dump: 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.004 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.005 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 18462326 0 0) 0x7f31c00065c0 con 0x7f31c0002860 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.981+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31c000a160 con 0x7f31c0002860 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.982+0000 7f31c7fff640 10 client.25445.objecter ms_handle_connect 0x7f31cc041ae0 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.982+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f31d0004330 con 0x7f31c0002860 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.982+0000 7f31c7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.982+0000 7f31c7fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.982+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.26:6792/0 conn(0x7f31c0002860 legacy=0x7f31c0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.982+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.26:6792/0 conn(0x7f31c0002860 legacy=0x7f31c0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.988+0000 7f31e584f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.988+0000 7f31e584f640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f31c0009e50 con 0x7f31c0009110 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31deffd640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.26:6792/0 conn(0x7f31c0009110 legacy=0x7f31c00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 10 client.25445.objecter ms_handle_connect 0x7f31c0009110 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 10 client.25445.objecter resend_mon_ops 2024-08-19T12:40:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3522531707 0 0) 0x7f31c0009e50 con 0x7f31c0009110 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 10 monclient(hunting): my global_id is 25674 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31c00065c0 con 0x7f31c0009110 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2966624409 0 0) 0x7f31c00065c0 con 0x7f31c0009110 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31c0009e50 con 0x7f31c0009110 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f31d0005cf0 con 0x7f31c0009110 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.990+0000 7f31c7fff640 10 monclient: dump: 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.007 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.008 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 645022289 0 0) 0x7f31c0009e50 con 0x7f31c0009110 2024-08-19T12:40:35.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f31c000c3a0 con 0x7f31c0009110 2024-08-19T12:40:35.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1981145689) 0x7f31d0006080 con 0x7f31c0009110 2024-08-19T12:40:35.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31c7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:35.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31c7fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:35.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.26:6792/0 conn(0x7f31c0009110 legacy=0x7f31c00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31c7fff640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.26:6792/0 conn(0x7f31c0009110 legacy=0x7f31c00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.009 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:35.009 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:35.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 148032, 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:35.010 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.011 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.012 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:35.013 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.014 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:35.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.016 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.018 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:35.019 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:35.021 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:35.022 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:35.023 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:35.029 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:35.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 10 librados: watch_flush enter 2024-08-19T12:40:35.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 10 librados: watch_flush exit 2024-08-19T12:40:35.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 20 client.25445.objecter shutdown clearing up homeless session... 2024-08-19T12:40:35.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 10 client.25445.objecter successfully canceled tick 2024-08-19T12:40:35.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.104:6800/899887506 conn(0x7f31cc041ae0 legacy=0x7f31cc043fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 10 monclient: shutdown 2024-08-19T12:40:35.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:35.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 1 -- 172.21.15.26:0/3567017024 >> v1:172.21.15.26:6790/0 conn(0x7f31e00a57a0 legacy=0x7f31e01d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31df7fe640 1 -- 172.21.15.26:0/3567017024 reap_dead start 2024-08-19T12:40:35.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31df7fe640 1 -- 172.21.15.26:0/3567017024 reap_dead start 2024-08-19T12:40:35.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 1 -- 172.21.15.26:0/3567017024 shutdown_connections 2024-08-19T12:40:35.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 1 -- 172.21.15.26:0/3567017024 >> 172.21.15.26:0/3567017024 conn(0x7f31e00abff0 msgr2=0x7f31e0133c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:35.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 1 -- 172.21.15.26:0/3567017024 shutdown_connections 2024-08-19T12:40:35.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 1 -- 172.21.15.26:0/3567017024 wait complete. 2024-08-19T12:40:35.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:34.991+0000 7f31e584f640 1 librados: shutdown 2024-08-19T12:40:35.032 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:35.032 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:40:35.032 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:40:35.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.131+0000 7fc80a16b640 10 monclient: get_monmap_and_config 2024-08-19T12:40:35.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.131+0000 7fc80a16b640 10 monclient: build_initial_monmap 2024-08-19T12:40:35.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.131+0000 7fc80a16b640 10 monclient: monmap: 2024-08-19T12:40:35.133 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:35.133 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.133 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:35.133117+0000 2024-08-19T12:40:35.133 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:35.133117+0000 2024-08-19T12:40:35.133 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:35.133 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.133 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:35.133 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:35.134 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:35.134 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:35.134 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:35.134 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:35.134 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:35.134 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:35.134 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:35.134 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.131+0000 7fc80a16b640 1 Processor -- start 2024-08-19T12:40:35.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc80a16b640 1 -- start start 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc80a16b640 10 monclient: init 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc80a16b640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc80a16b640 10 monclient: _add_conns ranks=[0,8,3,1,6,4,7,5,2] 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc80a16b640 10 monclient(hunting): picked mon.noname-a con 0x7fc804073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc80a16b640 10 monclient(hunting): picked mon.noname-e con 0x7fc8040a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc80a16b640 10 monclient(hunting): picked mon.noname-g con 0x7fc80413f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc80a16b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc80409dae0 con 0x7fc804073470 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc80a16b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc804136910 con 0x7fc80413f2e0 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc80a16b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc80409dd50 con 0x7fc8040a57a0 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc80a16b640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc80a16b640 10 monclient(hunting): authenticate will time out at 829.220614s 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc8037fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc804073470 0x7fc804136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:59914/0 (socket says 172.21.15.26:59914) 2024-08-19T12:40:35.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.132+0000 7fc8037fe640 1 -- 172.21.15.26:0/2279288269 learned_addr learned my addr 172.21.15.26:0/2279288269 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:35.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.134+0000 7fc8037fe640 1 -- 172.21.15.26:0/2279288269 >> v1:172.21.15.26:6789/0 conn(0x7fc804073470 legacy=0x7fc804136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.134+0000 7fc802ffd640 1 -- 172.21.15.26:0/2279288269 >> v1:172.21.15.26:6793/0 conn(0x7fc8040a57a0 legacy=0x7fc8040a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.134+0000 7fc803fff640 1 -- 172.21.15.26:0/2279288269 >> v1:172.21.15.104:6790/0 conn(0x7fc80413f2e0 legacy=0x7fc8041416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.134+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2764250394 0 0) 0x7fc80409dae0 con 0x7fc804073470 2024-08-19T12:40:35.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient(hunting): my global_id is 26964 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7f0003520 con 0x7fc804073470 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101914120 0 0) 0x7fc80409dd50 con 0x7fc8040a57a0 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient(hunting): my global_id is 25253 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc80409dae0 con 0x7fc8040a57a0 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1862682270 0 0) 0x7fc804136910 con 0x7fc80413f2e0 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient(hunting): my global_id is 25581 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc80409dd50 con 0x7fc80413f2e0 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2144118916 0 0) 0x7fc7f0003520 con 0x7fc804073470 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 >> v1:172.21.15.26:6793/0 conn(0x7fc8040a57a0 legacy=0x7fc8040a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 >> v1:172.21.15.104:6790/0 conn(0x7fc80413f2e0 legacy=0x7fc8041416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient: _finish_hunting 0 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 monclient: found mon.noname-a 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80409e030 con 0x7fc804073470 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient: _finish_auth 0 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient: _check_auth_tickets 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc7f4001ae0 con 0x7fc804073470 2024-08-19T12:40:35.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc80a16b640 5 monclient: authenticate success, global_id 26964 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc80a16b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient: dump: 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.143 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc80a16b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc7f40048a0 con 0x7fc804073470 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 1 -- 172.21.15.26:0/2279288269 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc7f4004e10 con 0x7fc804073470 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc80a16b640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc80a16b640 4 set_mon_vals no callback set 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc8027fc640 10 monclient: dump: 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.145 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc80a16b640 10 monclient: shutdown 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc80a16b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc80a16b640 1 -- 172.21.15.26:0/2279288269 >> v1:172.21.15.26:6789/0 conn(0x7fc804073470 legacy=0x7fc804136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc80a16b640 1 -- 172.21.15.26:0/2279288269 shutdown_connections 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.135+0000 7fc80a16b640 1 -- 172.21.15.26:0/2279288269 >> 172.21.15.26:0/2279288269 conn(0x7fc8040abff0 msgr2=0x7fc8040ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.136+0000 7fc80a16b640 1 -- 172.21.15.26:0/2279288269 shutdown_connections 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.136+0000 7fc80a16b640 1 -- 172.21.15.26:0/2279288269 wait complete. 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.136+0000 7fc80a16b640 10 monclient: build_initial_monmap 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.136+0000 7fc80a16b640 10 monclient: monmap: 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:35.146 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:35.147 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.136+0000 7fc80a16b640 1 librados: starting msgr at 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.136+0000 7fc80a16b640 1 librados: starting objecter 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.136+0000 7fc80a16b640 1 Processor -- start 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 1 -- start start 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 1 librados: setting wanted keys 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 1 librados: calling monclient init 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 10 monclient: init 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 10 monclient: _add_conns ranks=[7,4,0,5,3,6,2,8,1] 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 10 monclient(hunting): picked mon.noname-i con 0x7fc804073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 10 monclient(hunting): picked mon.noname-c con 0x7fc8040a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 10 monclient(hunting): picked mon.noname-a con 0x7fc80413f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc804131830 con 0x7fc80413f2e0 2024-08-19T12:40:35.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc80412ea60 con 0x7fc8040a57a0 2024-08-19T12:40:35.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc80413f0b0 con 0x7fc804073470 2024-08-19T12:40:35.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:35.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc80a16b640 10 monclient(hunting): authenticate will time out at 829.225648s 2024-08-19T12:40:35.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc802ffd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fc8040a57a0 0x7fc80412e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:39182/0 (socket says 172.21.15.26:39182) 2024-08-19T12:40:35.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.137+0000 7fc802ffd640 1 -- 172.21.15.26:0/729011935 learned_addr learned my addr 172.21.15.26:0/729011935 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.139+0000 7fc803fff640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.26:6789/0 conn(0x7fc80413f2e0 legacy=0x7fc8041d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.139+0000 7fc802ffd640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.26:6791/0 conn(0x7fc8040a57a0 legacy=0x7fc80412e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc8037fe640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.104:6792/0 conn(0x7fc804073470 legacy=0x7fc80412dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.139+0000 7fc7fbfff640 10 client.?.objecter ms_handle_connect 0x7fc80413f2e0 2024-08-19T12:40:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 client.?.objecter ms_handle_connect 0x7fc8040a57a0 2024-08-19T12:40:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 client.?.objecter ms_handle_connect 0x7fc804073470 2024-08-19T12:40:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1944674460 0 0) 0x7fc804131830 con 0x7fc80413f2e0 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): my global_id is 26973 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7e4004eb0 con 0x7fc80413f2e0 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3095483420 0 0) 0x7fc80412ea60 con 0x7fc8040a57a0 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): my global_id is 25699 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc804131830 con 0x7fc8040a57a0 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475800137 0 0) 0x7fc80413f0b0 con 0x7fc804073470 2024-08-19T12:40:35.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): my global_id is 134179 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc80412ea60 con 0x7fc804073470 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4256486819 0 0) 0x7fc7e4004eb0 con 0x7fc80413f2e0 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc80413f0b0 con 0x7fc80413f2e0 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc7fc001cf0 con 0x7fc80413f2e0 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): dump: 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.158 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:35.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1155204680 0 0) 0x7fc804131830 con 0x7fc8040a57a0 2024-08-19T12:40:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7e4004eb0 con 0x7fc8040a57a0 2024-08-19T12:40:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc7ec003990 con 0x7fc8040a57a0 2024-08-19T12:40:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): dump: 2024-08-19T12:40:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.160 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 600388415 0 0) 0x7fc80412ea60 con 0x7fc804073470 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc804131830 con 0x7fc804073470 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc7f4004d90 con 0x7fc804073470 2024-08-19T12:40:35.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient(hunting): dump: 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.162 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2731515802 0 0) 0x7fc80413f0b0 con 0x7fc80413f2e0 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.104:6792/0 conn(0x7fc804073470 legacy=0x7fc80412dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.26:6791/0 conn(0x7fc8040a57a0 legacy=0x7fc80412e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient: _finish_hunting 0 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 monclient: found mon.a 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc804142b80 con 0x7fc80413f2e0 2024-08-19T12:40:35.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient: _finish_auth 0 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient: _check_auth_tickets 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc80a16b640 5 monclient: authenticate success, global_id 26973 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc80a16b640 10 monclient: _renew_subs 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc80a16b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc80a16b640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc804133210 con 0x7fc80413f2e0 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc80a16b640 10 client.26973.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc80a16b640 10 monclient: _renew_subs 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc80a16b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc80a16b640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc804131550 con 0x7fc80413f2e0 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc80a16b640 1 librados: init done 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc7fc003a30 con 0x7fc80413f2e0 2024-08-19T12:40:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc7fc003f20 con 0x7fc80413f2e0 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc801ffb640 4 set_mon_vals no callback set 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.140+0000 7fc7fbfff640 10 monclient: dump: 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.165 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.141+0000 7fc80a16b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.141+0000 7fc80a16b640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc7e80065c0 con 0x7fc7e8002860 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.141+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fc7fc0040c0 con 0x7fc80413f2e0 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.142+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(78..78 src has 1..78) ==== 6992+0+0 (unknown 3025720083 0 0) 0x7fc7fc04e6d0 con 0x7fc80413f2e0 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.142+0000 7fc7fbfff640 10 client.26973.objecter ms_dispatch 0x7fc804000b90 osd_map(78..78 src has 1..78) 2024-08-19T12:40:35.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.142+0000 7fc7fbfff640 3 client.26973.objecter handle_osd_map got epochs [78,78] > 0 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.142+0000 7fc7fbfff640 3 client.26973.objecter handle_osd_map decoding full epoch 78 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.142+0000 7fc7fbfff640 20 client.26973.objecter dump_active .. 0 homeless 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.142+0000 7fc802ffd640 10 monclient: get_auth_request con 0x7fc7e4041940 auth_method 0 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.143+0000 7fc8037fe640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.26:6793/0 conn(0x7fc7e8002860 legacy=0x7fc7e8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.143+0000 7fc7fbfff640 10 client.26973.objecter ms_handle_connect 0x7fc7e8002860 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.143+0000 7fc7fbfff640 10 client.26973.objecter resend_mon_ops 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1368595352 0 0) 0x7fc7e80065c0 con 0x7fc7e8002860 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 10 monclient(hunting): my global_id is 25262 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc80413f0b0 con 0x7fc7e8002860 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 828931878 0 0) 0x7fc80413f0b0 con 0x7fc7e8002860 2024-08-19T12:40:35.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7e80065c0 con 0x7fc7e8002860 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc7f4001580 con 0x7fc7e8002860 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 10 monclient: dump: 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.168 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1176265777 0 0) 0x7fc7e80065c0 con 0x7fc7e8002860 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc7e800a160 con 0x7fc7e8002860 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 10 client.26973.objecter ms_handle_connect 0x7fc7e4041940 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc7f4001d30 con 0x7fc7e8002860 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:35.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.26:6793/0 conn(0x7fc7e8002860 legacy=0x7fc7e8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.144+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.26:6793/0 conn(0x7fc7e8002860 legacy=0x7fc7e8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.151+0000 7fc80a16b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.151+0000 7fc80a16b640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc7e8009e70 con 0x7fc7e8009130 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 10 client.26973.objecter ms_handle_connect 0x7fc7e8009130 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 10 client.26973.objecter resend_mon_ops 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc8037fe640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.26:6793/0 conn(0x7fc7e8009130 legacy=0x7fc7e80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 121687658 0 0) 0x7fc7e8009e70 con 0x7fc7e8009130 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 10 monclient(hunting): my global_id is 25271 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7e80065c0 con 0x7fc7e8009130 2024-08-19T12:40:35.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2706341542 0 0) 0x7fc7e80065c0 con 0x7fc7e8009130 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7e8009e70 con 0x7fc7e8009130 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc7f40059c0 con 0x7fc7e8009130 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 10 monclient: dump: 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.171 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1738343804 0 0) 0x7fc7e8009e70 con 0x7fc7e8009130 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc7e800c410 con 0x7fc7e8009130 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 451435780) 0x7fc7f40048a0 con 0x7fc7e8009130 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.26:6793/0 conn(0x7fc7e8009130 legacy=0x7fc7e80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.153+0000 7fc7fbfff640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.26:6793/0 conn(0x7fc7e8009130 legacy=0x7fc7e80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.173 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 148170, 2024-08-19T12:40:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:35.193 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:35.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 10 librados: watch_flush enter 2024-08-19T12:40:35.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 10 librados: watch_flush exit 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 20 client.26973.objecter shutdown clearing up homeless session... 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 10 client.26973.objecter successfully canceled tick 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.104:6800/899887506 conn(0x7fc7e4041940 legacy=0x7fc7e4043e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 10 monclient: shutdown 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 1 -- 172.21.15.26:0/729011935 >> v1:172.21.15.26:6789/0 conn(0x7fc80413f2e0 legacy=0x7fc8041d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc803fff640 1 -- 172.21.15.26:0/729011935 reap_dead start 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc803fff640 1 -- 172.21.15.26:0/729011935 reap_dead start 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 1 -- 172.21.15.26:0/729011935 shutdown_connections 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 1 -- 172.21.15.26:0/729011935 >> 172.21.15.26:0/729011935 conn(0x7fc8040abff0 msgr2=0x7fc8040a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 1 -- 172.21.15.26:0/729011935 shutdown_connections 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 1 -- 172.21.15.26:0/729011935 wait complete. 2024-08-19T12:40:35.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.154+0000 7fc80a16b640 1 librados: shutdown 2024-08-19T12:40:35.196 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:35.196 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:40:35.196 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:40:35.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.293+0000 7f6ff8588640 10 monclient: get_monmap_and_config 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.293+0000 7f6ff8588640 10 monclient: build_initial_monmap 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.294+0000 7f6ff8588640 10 monclient: monmap: 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:35.295851+0000 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:35.295851+0000 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:35.296 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:35.297 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:35.297 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:35.297 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:35.297 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.294+0000 7f6ff8588640 1 Processor -- start 2024-08-19T12:40:35.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.294+0000 7f6ff8588640 1 -- start start 2024-08-19T12:40:35.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.294+0000 7f6ff8588640 10 monclient: init 2024-08-19T12:40:35.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.295+0000 7f6ff8588640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:35.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.295+0000 7f6ff8588640 10 monclient: _add_conns ranks=[1,5,4,6,7,3,0,8,2] 2024-08-19T12:40:35.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.295+0000 7f6ff8588640 10 monclient(hunting): picked mon.noname-f con 0x7f6ff006e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:35.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.295+0000 7f6ff8588640 10 monclient(hunting): picked mon.noname-h con 0x7f6ff00a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:35.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.295+0000 7f6ff8588640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff00a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:35.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.295+0000 7f6ff8588640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff0136790 con 0x7f6ff006e7b0 2024-08-19T12:40:35.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.295+0000 7f6ff8588640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff009ddf0 con 0x7f6ff00a5d50 2024-08-19T12:40:35.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.295+0000 7f6ff8588640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff009e0d0 con 0x7f6ff00a5920 2024-08-19T12:40:35.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.295+0000 7f6ff8588640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:35.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.295+0000 7f6ff8588640 10 monclient(hunting): authenticate will time out at 829.383285s 2024-08-19T12:40:35.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.295+0000 7f6ff6b25640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f6ff00a5d50 0x7f6ff0141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:39190/0 (socket says 172.21.15.26:39190) 2024-08-19T12:40:35.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.295+0000 7f6ff6b25640 1 -- 172.21.15.26:0/4015179041 learned_addr learned my addr 172.21.15.26:0/4015179041 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:35.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff6b25640 1 -- 172.21.15.26:0/4015179041 >> v1:172.21.15.26:6791/0 conn(0x7f6ff00a5d50 legacy=0x7f6ff0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff6324640 1 -- 172.21.15.26:0/4015179041 >> v1:172.21.15.104:6789/0 conn(0x7f6ff006e7b0 legacy=0x7f6ff0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5b23640 1 -- 172.21.15.26:0/4015179041 >> v1:172.21.15.104:6791/0 conn(0x7f6ff00a5920 legacy=0x7f6ff013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2271281690 0 0) 0x7f6fe80017d0 con 0x7f6ff00a5d50 2024-08-19T12:40:35.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 10 monclient(hunting): my global_id is 25708 2024-08-19T12:40:35.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fdc003520 con 0x7f6ff00a5d50 2024-08-19T12:40:35.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2258798112 0 0) 0x7f6ff009e0d0 con 0x7f6ff00a5920 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 10 monclient(hunting): my global_id is 25457 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe80017d0 con 0x7f6ff00a5920 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1651369715 0 0) 0x7f6ff0136790 con 0x7f6ff006e7b0 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 10 monclient(hunting): my global_id is 26119 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.297+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff009e0d0 con 0x7f6ff006e7b0 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2907588113 0 0) 0x7f6fdc003520 con 0x7f6ff00a5d50 2024-08-19T12:40:35.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 >> v1:172.21.15.104:6791/0 conn(0x7f6ff00a5920 legacy=0x7f6ff013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 >> v1:172.21.15.104:6789/0 conn(0x7f6ff006e7b0 legacy=0x7f6ff0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 10 monclient: _finish_hunting 0 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 1 monclient: found mon.noname-c 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff009e3b0 con 0x7f6ff00a5d50 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 10 monclient: _finish_auth 0 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 10 monclient: _check_auth_tickets 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6fe8002030 con 0x7f6ff00a5d50 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 10 monclient: dump: 2024-08-19T12:40:35.305 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.306 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff8588640 5 monclient: authenticate success, global_id 25708 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff8588640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fe8004cd0 con 0x7f6ff00a5d50 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 1 -- 172.21.15.26:0/4015179041 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6fe8005200 con 0x7f6ff00a5d50 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff8588640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff8588640 4 set_mon_vals no callback set 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff5322640 10 monclient: dump: 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.307 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff8588640 10 monclient: shutdown 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff8588640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff8588640 1 -- 172.21.15.26:0/4015179041 >> v1:172.21.15.26:6791/0 conn(0x7f6ff00a5d50 legacy=0x7f6ff0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff8588640 1 -- 172.21.15.26:0/4015179041 shutdown_connections 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff8588640 1 -- 172.21.15.26:0/4015179041 >> 172.21.15.26:0/4015179041 conn(0x7f6ff00a8740 msgr2=0x7f6ff00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff8588640 1 -- 172.21.15.26:0/4015179041 shutdown_connections 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.298+0000 7f6ff8588640 1 -- 172.21.15.26:0/4015179041 wait complete. 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.299+0000 7f6ff8588640 10 monclient: build_initial_monmap 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.299+0000 7f6ff8588640 10 monclient: monmap: 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.309 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.299+0000 7f6ff8588640 1 librados: starting msgr at 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.299+0000 7f6ff8588640 1 librados: starting objecter 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.299+0000 7f6ff8588640 1 Processor -- start 2024-08-19T12:40:35.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.299+0000 7f6ff8588640 1 -- start start 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.299+0000 7f6ff8588640 1 librados: setting wanted keys 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.299+0000 7f6ff8588640 1 librados: calling monclient init 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.299+0000 7f6ff8588640 10 monclient: init 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.300+0000 7f6ff8588640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.300+0000 7f6ff8588640 10 monclient: _add_conns ranks=[7,1,8,3,2,0,5,6,4] 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.300+0000 7f6ff8588640 10 monclient(hunting): picked mon.noname-i con 0x7f6ff006e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.300+0000 7f6ff8588640 10 monclient(hunting): picked mon.noname-f con 0x7f6ff00a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.300+0000 7f6ff8588640 10 monclient(hunting): picked mon.noname-e con 0x7f6ff00a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.300+0000 7f6ff8588640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff0139290 con 0x7f6ff00a5920 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.300+0000 7f6ff8588640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff013b5f0 con 0x7f6ff006e7b0 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.300+0000 7f6ff8588640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff013ad50 con 0x7f6ff00a5d50 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.300+0000 7f6ff8588640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.300+0000 7f6ff8588640 10 monclient(hunting): authenticate will time out at 829.388352s 2024-08-19T12:40:35.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.300+0000 7f6ff6b25640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f6ff00a5d50 0x7f6ff01d6dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50280/0 (socket says 172.21.15.26:50280) 2024-08-19T12:40:35.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.300+0000 7f6ff6b25640 1 -- 172.21.15.26:0/1457569230 learned_addr learned my addr 172.21.15.26:0/1457569230 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:35.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6ff6b25640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.26:6793/0 conn(0x7f6ff00a5d50 legacy=0x7f6ff01d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 client.?.objecter ms_handle_connect 0x7f6ff00a5d50 2024-08-19T12:40:35.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6ff6324640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.104:6792/0 conn(0x7f6ff006e7b0 legacy=0x7f6ff01326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 client.?.objecter ms_handle_connect 0x7f6ff006e7b0 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 client.?.objecter ms_handle_connect 0x7f6ff00a5920 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6ff5b23640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.104:6789/0 conn(0x7f6ff00a5920 legacy=0x7f6ff01d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4229442020 0 0) 0x7f6ff013ad50 con 0x7f6ff00a5d50 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 monclient(hunting): my global_id is 25289 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd4004eb0 con 0x7f6ff00a5d50 2024-08-19T12:40:35.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3566582127 0 0) 0x7f6ff013b5f0 con 0x7f6ff006e7b0 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 monclient(hunting): my global_id is 134188 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff013ad50 con 0x7f6ff006e7b0 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3867377509 0 0) 0x7f6ff0139290 con 0x7f6ff00a5920 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 10 monclient(hunting): my global_id is 26128 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff013b5f0 con 0x7f6ff00a5920 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2592395389 0 0) 0x7f6fd4004eb0 con 0x7f6ff00a5d50 2024-08-19T12:40:35.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.302+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ff0139290 con 0x7f6ff00a5d50 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6fe8002050 con 0x7f6ff00a5d50 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient(hunting): dump: 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.319 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1118199296 0 0) 0x7f6ff013ad50 con 0x7f6ff006e7b0 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fd4004eb0 con 0x7f6ff006e7b0 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6fec001d10 con 0x7f6ff006e7b0 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:35.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient(hunting): dump: 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.321 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1546332262 0 0) 0x7f6ff013b5f0 con 0x7f6ff00a5920 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ff013ad50 con 0x7f6ff00a5920 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6fe0003610 con 0x7f6ff00a5920 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient(hunting): dump: 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.322 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 611933586 0 0) 0x7f6ff0139290 con 0x7f6ff00a5d50 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.104:6792/0 conn(0x7f6ff006e7b0 legacy=0x7f6ff01326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.104:6789/0 conn(0x7f6ff00a5920 legacy=0x7f6ff01d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 monclient: found mon.e 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff0139df0 con 0x7f6ff00a5d50 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient: _finish_auth 0 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6ff8588640 5 monclient: authenticate success, global_id 25289 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6ff8588640 10 monclient: _renew_subs 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6ff8588640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6ff8588640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff013bfd0 con 0x7f6ff00a5d50 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6ff8588640 10 client.25289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:35.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6ff8588640 10 monclient: _renew_subs 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6ff8588640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6ff8588640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f6ff01389c0 con 0x7f6ff00a5d50 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6ff8588640 1 librados: init done 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fe8004cd0 con 0x7f6ff00a5d50 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6fe80057b0 con 0x7f6ff00a5d50 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6ff4b21640 4 set_mon_vals no callback set 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6fe6ffd640 10 monclient: dump: 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.325 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.303+0000 7f6ff8588640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.304+0000 7f6ff8588640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fd80065c0 con 0x7f6fd8002860 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.304+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6fe8011c10 con 0x7f6ff00a5d50 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.304+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(78..78 src has 1..78) ==== 6992+0+0 (unknown 3025720083 0 0) 0x7f6fe804ec00 con 0x7f6ff00a5d50 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.304+0000 7f6fe6ffd640 10 client.25289.objecter ms_dispatch 0x7f6ff0000b90 osd_map(78..78 src has 1..78) 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.304+0000 7f6fe6ffd640 3 client.25289.objecter handle_osd_map got epochs [78,78] > 0 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.304+0000 7f6fe6ffd640 3 client.25289.objecter handle_osd_map decoding full epoch 78 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.304+0000 7f6fe6ffd640 20 client.25289.objecter dump_active .. 0 homeless 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.304+0000 7f6ff5b23640 10 monclient: get_auth_request con 0x7f6fd40419f0 auth_method 0 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.306+0000 7f6fe6ffd640 10 client.25289.objecter ms_handle_connect 0x7f6fd8002860 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.306+0000 7f6ff6324640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.104:6789/0 conn(0x7f6fd8002860 legacy=0x7f6fd8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.306+0000 7f6fe6ffd640 10 client.25289.objecter resend_mon_ops 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.306+0000 7f6fe6ffd640 10 client.25289.objecter ms_handle_connect 0x7f6fd40419f0 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1084593407 0 0) 0x7f6fd80065c0 con 0x7f6fd8002860 2024-08-19T12:40:35.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 10 monclient(hunting): my global_id is 26137 2024-08-19T12:40:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff0139290 con 0x7f6fd8002860 2024-08-19T12:40:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 14832308 0 0) 0x7f6ff0139290 con 0x7f6fd8002860 2024-08-19T12:40:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fd80065c0 con 0x7f6fd8002860 2024-08-19T12:40:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6fec003ce0 con 0x7f6fd8002860 2024-08-19T12:40:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 10 monclient: dump: 2024-08-19T12:40:35.328 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.329 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.330 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 714408159 0 0) 0x7f6fd80065c0 con 0x7f6fd8002860 2024-08-19T12:40:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fd800a160 con 0x7f6fd8002860 2024-08-19T12:40:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6fec003f70 con 0x7f6fd8002860 2024-08-19T12:40:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.104:6789/0 conn(0x7f6fd8002860 legacy=0x7f6fd8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.309+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.104:6789/0 conn(0x7f6fd8002860 legacy=0x7f6fd8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.315+0000 7f6ff8588640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.315+0000 7f6ff8588640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fd8009e30 con 0x7f6fd8009110 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6ff6324640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.104:6789/0 conn(0x7f6fd8009110 legacy=0x7f6fd8009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 10 client.25289.objecter ms_handle_connect 0x7f6fd8009110 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 10 client.25289.objecter resend_mon_ops 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2929723176 0 0) 0x7f6fd8009e30 con 0x7f6fd8009110 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 10 monclient(hunting): my global_id is 26146 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd80065c0 con 0x7f6fd8009110 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3068233208 0 0) 0x7f6fd80065c0 con 0x7f6fd8009110 2024-08-19T12:40:35.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fd8009e30 con 0x7f6fd8009110 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6fec005b00 con 0x7f6fd8009110 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 10 monclient: dump: 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.333 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 420972809 0 0) 0x7f6fd8009e30 con 0x7f6fd8009110 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.318+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6fd800c3f0 con 0x7f6fd8009110 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1885783945) 0x7f6fec007e30 con 0x7f6fd8009110 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6fe6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6fe6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.104:6789/0 conn(0x7f6fd8009110 legacy=0x7f6fd8009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6fe6ffd640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.104:6789/0 conn(0x7f6fd8009110 legacy=0x7f6fd8009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.335 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 148323, 2024-08-19T12:40:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:35.335 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.336 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.337 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:35.338 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:35.339 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.340 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.341 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.342 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.343 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.344 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.346 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:35.347 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:35.349 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:35.350 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:35.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.354 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.354 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:35.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.354 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.354 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.354 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-08-19T12:40:35.354 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.354 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.354 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:35.354 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 10 librados: watch_flush enter 2024-08-19T12:40:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 10 librados: watch_flush exit 2024-08-19T12:40:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 20 client.25289.objecter shutdown clearing up homeless session... 2024-08-19T12:40:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 10 client.25289.objecter successfully canceled tick 2024-08-19T12:40:35.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.104:6800/899887506 conn(0x7f6fd40419f0 legacy=0x7f6fd4043eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 10 monclient: shutdown 2024-08-19T12:40:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 1 -- 172.21.15.26:0/1457569230 >> v1:172.21.15.26:6793/0 conn(0x7f6ff00a5d50 legacy=0x7f6ff01d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff6b25640 1 -- 172.21.15.26:0/1457569230 reap_dead start 2024-08-19T12:40:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff6b25640 1 -- 172.21.15.26:0/1457569230 reap_dead start 2024-08-19T12:40:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 1 -- 172.21.15.26:0/1457569230 shutdown_connections 2024-08-19T12:40:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 1 -- 172.21.15.26:0/1457569230 >> 172.21.15.26:0/1457569230 conn(0x7f6ff00a8740 msgr2=0x7f6ff01350d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 1 -- 172.21.15.26:0/1457569230 shutdown_connections 2024-08-19T12:40:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 1 -- 172.21.15.26:0/1457569230 wait complete. 2024-08-19T12:40:35.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.319+0000 7f6ff8588640 1 librados: shutdown 2024-08-19T12:40:35.357 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:35.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:40:35.358 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:40:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.455+0000 7f721e1aa640 10 monclient: get_monmap_and_config 2024-08-19T12:40:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.455+0000 7f721e1aa640 10 monclient: build_initial_monmap 2024-08-19T12:40:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.455+0000 7f721e1aa640 10 monclient: monmap: 2024-08-19T12:40:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:35.457431+0000 2024-08-19T12:40:35.457 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:35.457431+0000 2024-08-19T12:40:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:35.458 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:35.458 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 1 Processor -- start 2024-08-19T12:40:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 1 -- start start 2024-08-19T12:40:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 10 monclient: init 2024-08-19T12:40:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 10 monclient: _add_conns ranks=[6,8,4,7,1,5,2,3,0] 2024-08-19T12:40:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 10 monclient(hunting): picked mon.noname-d con 0x7f721806e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 10 monclient(hunting): picked mon.noname-e con 0x7f72180a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 10 monclient(hunting): picked mon.noname-c con 0x7f72180a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:35.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7218136790 con 0x7f72180a5d50 2024-08-19T12:40:35.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f721809ddf0 con 0x7f721806e7b0 2024-08-19T12:40:35.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f721809e0d0 con 0x7f72180a5920 2024-08-19T12:40:35.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:35.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f721e1aa640 10 monclient(hunting): authenticate will time out at 829.544896s 2024-08-19T12:40:35.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f7216ffd640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f72180a5920 0x7f721813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50282/0 (socket says 172.21.15.26:50282) 2024-08-19T12:40:35.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f7216ffd640 1 -- 172.21.15.26:0/1151679155 learned_addr learned my addr 172.21.15.26:0/1151679155 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:35.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.456+0000 7f72177fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f721806e7b0 0x7f7218136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39164/0 (socket says 172.21.15.26:39164) 2024-08-19T12:40:35.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f7216ffd640 1 -- 172.21.15.26:0/1151679155 >> v1:172.21.15.26:6793/0 conn(0x7f72180a5920 legacy=0x7f721813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f7217fff640 1 -- 172.21.15.26:0/1151679155 >> v1:172.21.15.26:6791/0 conn(0x7f72180a5d50 legacy=0x7f7218141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72177fe640 1 -- 172.21.15.26:0/1151679155 >> v1:172.21.15.26:6792/0 conn(0x7f721806e7b0 legacy=0x7f7218136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3212868366 0 0) 0x7f721809e0d0 con 0x7f72180a5920 2024-08-19T12:40:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient(hunting): my global_id is 25298 2024-08-19T12:40:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7200003520 con 0x7f72180a5920 2024-08-19T12:40:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 336174758 0 0) 0x7f7218136790 con 0x7f72180a5d50 2024-08-19T12:40:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient(hunting): my global_id is 25717 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f721809e0d0 con 0x7f72180a5d50 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 32259275 0 0) 0x7f7208001380 con 0x7f721806e7b0 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient(hunting): my global_id is 25701 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7218136790 con 0x7f721806e7b0 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72177fe640 1 --1- 172.21.15.26:0/1151679155 >> v1:172.21.15.26:6792/0 conn(0x7f721806e7b0 0x7f7218136080 :-1 s=OPENED pgs=3018 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f7218136790 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4206176759 0 0) 0x7f7200003520 con 0x7f72180a5920 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 >> v1:172.21.15.26:6792/0 conn(0x7f721806e7b0 legacy=0x7f7218136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 >> v1:172.21.15.26:6791/0 conn(0x7f72180a5d50 legacy=0x7f7218141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient: _finish_hunting 0 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 monclient: found mon.noname-e 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f721809e3b0 con 0x7f72180a5920 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient: _finish_auth 0 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient: _check_auth_tickets 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7208001b00 con 0x7f721806e7b0 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f720c001ae0 con 0x7f72180a5920 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:40:35.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient: dump: 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.468 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.469 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f721e1aa640 5 monclient: authenticate success, global_id 25298 2024-08-19T12:40:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f721e1aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f720c0048a0 con 0x7f72180a5920 2024-08-19T12:40:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 1 -- 172.21.15.26:0/1151679155 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f720c004d90 con 0x7f72180a5920 2024-08-19T12:40:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f721e1aa640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f721e1aa640 4 set_mon_vals no callback set 2024-08-19T12:40:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.459+0000 7f72167fc640 10 monclient: dump: 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.470 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.460+0000 7f721e1aa640 10 monclient: shutdown 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.460+0000 7f721e1aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.460+0000 7f721e1aa640 1 -- 172.21.15.26:0/1151679155 >> v1:172.21.15.26:6793/0 conn(0x7f72180a5920 legacy=0x7f721813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.460+0000 7f721e1aa640 1 -- 172.21.15.26:0/1151679155 shutdown_connections 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.460+0000 7f721e1aa640 1 -- 172.21.15.26:0/1151679155 >> 172.21.15.26:0/1151679155 conn(0x7f72180a8740 msgr2=0x7f72180a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.460+0000 7f721e1aa640 1 -- 172.21.15.26:0/1151679155 shutdown_connections 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.460+0000 7f721e1aa640 1 -- 172.21.15.26:0/1151679155 wait complete. 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.460+0000 7f721e1aa640 10 monclient: build_initial_monmap 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.460+0000 7f721e1aa640 10 monclient: monmap: 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:35.471 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:35.472 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 1 librados: starting msgr at 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 1 librados: starting objecter 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 1 Processor -- start 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 1 -- start start 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 1 librados: setting wanted keys 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 1 librados: calling monclient init 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 10 monclient: init 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 10 monclient: _add_conns ranks=[6,7,5,8,3,1,2,0,4] 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 10 monclient(hunting): picked mon.noname-d con 0x7f721806e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 10 monclient(hunting): picked mon.noname-i con 0x7f72180a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 10 monclient(hunting): picked mon.noname-h con 0x7f72180a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7218142ec0 con 0x7f72180a5d50 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7218138fc0 con 0x7f721806e7b0 2024-08-19T12:40:35.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f721813aa80 con 0x7f72180a5920 2024-08-19T12:40:35.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:35.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f721e1aa640 10 monclient(hunting): authenticate will time out at 829.549894s 2024-08-19T12:40:35.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f72177fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f721806e7b0 0x7f72181ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39168/0 (socket says 172.21.15.26:39168) 2024-08-19T12:40:35.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.461+0000 7f72177fe640 1 -- 172.21.15.26:0/2146681143 learned_addr learned my addr 172.21.15.26:0/2146681143 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f72177fe640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.26:6792/0 conn(0x7f721806e7b0 legacy=0x7f72181ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 client.?.objecter ms_handle_connect 0x7f721806e7b0 2024-08-19T12:40:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3666208404 0 0) 0x7f7218138fc0 con 0x7f721806e7b0 2024-08-19T12:40:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient(hunting): my global_id is 25710 2024-08-19T12:40:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7204004eb0 con 0x7f721806e7b0 2024-08-19T12:40:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 client.?.objecter ms_handle_connect 0x7f72180a5d50 2024-08-19T12:40:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f7217fff640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.104:6791/0 conn(0x7f72180a5d50 legacy=0x7f72181d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3678744736 0 0) 0x7f7204004eb0 con 0x7f721806e7b0 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f7216ffd640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.104:6792/0 conn(0x7f72180a5920 legacy=0x7f72181d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7218138fc0 con 0x7f721806e7b0 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 client.?.objecter ms_handle_connect 0x7f72180a5920 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7208001b40 con 0x7f721806e7b0 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient(hunting): dump: 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1011026047 0 0) 0x7f7218138fc0 con 0x7f721806e7b0 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.104:6792/0 conn(0x7f72180a5920 legacy=0x7f72181d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.104:6791/0 conn(0x7f72180a5d50 legacy=0x7f72181d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient: _finish_hunting 0 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 1 monclient: found mon.d 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f721813ad60 con 0x7f721806e7b0 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient: _finish_auth 0 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient: _check_auth_tickets 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f721e1aa640 5 monclient: authenticate success, global_id 25710 2024-08-19T12:40:35.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f721e1aa640 10 monclient: _renew_subs 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f721e1aa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f721e1aa640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f721813a650 con 0x7f721806e7b0 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f721e1aa640 10 client.25710.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f721e1aa640 10 monclient: _renew_subs 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f721e1aa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f721e1aa640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f72181386b0 con 0x7f721806e7b0 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f721e1aa640 1 librados: init done 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7208004b90 con 0x7f721806e7b0 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7208005080 con 0x7f721806e7b0 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f7215ffb640 4 set_mon_vals no callback set 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:35.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.464+0000 7f71fffff640 10 monclient: dump: 2024-08-19T12:40:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.485 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.486 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.465+0000 7f721e1aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:35.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.465+0000 7f721e1aa640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f71f80065c0 con 0x7f71f8002860 2024-08-19T12:40:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.465+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f7208005320 con 0x7f721806e7b0 2024-08-19T12:40:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.465+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(78..78 src has 1..78) ==== 6992+0+0 (unknown 3025720083 0 0) 0x7f720804e870 con 0x7f721806e7b0 2024-08-19T12:40:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.465+0000 7f71fffff640 10 client.25710.objecter ms_dispatch 0x7f7218000b90 osd_map(78..78 src has 1..78) 2024-08-19T12:40:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.465+0000 7f71fffff640 3 client.25710.objecter handle_osd_map got epochs [78,78] > 0 2024-08-19T12:40:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.465+0000 7f71fffff640 3 client.25710.objecter handle_osd_map decoding full epoch 78 2024-08-19T12:40:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.465+0000 7f71fffff640 20 client.25710.objecter dump_active .. 0 homeless 2024-08-19T12:40:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.465+0000 7f7217fff640 10 monclient: get_auth_request con 0x7f7204040e90 auth_method 0 2024-08-19T12:40:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.467+0000 7f7216ffd640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.104:6790/0 conn(0x7f71f8002860 legacy=0x7f71f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.467+0000 7f71fffff640 10 client.25710.objecter ms_handle_connect 0x7f71f8002860 2024-08-19T12:40:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.467+0000 7f71fffff640 10 client.25710.objecter resend_mon_ops 2024-08-19T12:40:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.467+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2230312069 0 0) 0x7f71f80065c0 con 0x7f71f8002860 2024-08-19T12:40:35.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.467+0000 7f71fffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.467+0000 7f71fffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.467+0000 7f71fffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.467+0000 7f71fffff640 10 monclient(hunting): my global_id is 25599 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.467+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7218138fc0 con 0x7f71f8002860 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 27489246 0 0) 0x7f7218138fc0 con 0x7f71f8002860 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71f80065c0 con 0x7f71f8002860 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f720c001d30 con 0x7f71f8002860 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 10 monclient: dump: 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.488 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 10 client.25710.objecter ms_handle_connect 0x7f7204040e90 2024-08-19T12:40:35.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2298844692 0 0) 0x7f71f80065c0 con 0x7f71f8002860 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71f800a160 con 0x7f71f8002860 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f720c0049a0 con 0x7f71f8002860 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.104:6790/0 conn(0x7f71f8002860 legacy=0x7f71f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.468+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.104:6790/0 conn(0x7f71f8002860 legacy=0x7f71f8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.475+0000 7f721e1aa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.475+0000 7f721e1aa640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f71f8009e70 con 0x7f71f8009130 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f7216ffd640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.104:6790/0 conn(0x7f71f8009130 legacy=0x7f71f80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 10 client.25710.objecter ms_handle_connect 0x7f71f8009130 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 10 client.25710.objecter resend_mon_ops 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2706114540 0 0) 0x7f71f8009e70 con 0x7f71f8009130 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 10 monclient(hunting): my global_id is 25608 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71f80065c0 con 0x7f71f8009130 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1919198006 0 0) 0x7f71f80065c0 con 0x7f71f8009130 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71f8009e70 con 0x7f71f8009130 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f720c0053d0 con 0x7f71f8009130 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 10 monclient: dump: 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.491 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2828860401 0 0) 0x7f71f8009e70 con 0x7f71f8009130 2024-08-19T12:40:35.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f71f800c410 con 0x7f71f8009130 2024-08-19T12:40:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1907278766) 0x7f720c008540 con 0x7f71f8009130 2024-08-19T12:40:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.104:6790/0 conn(0x7f71f8009130 legacy=0x7f71f80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.478+0000 7f71fffff640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.104:6790/0 conn(0x7f71f8009130 legacy=0x7f71f80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.494 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:35.494 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:35.494 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:35.494 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:35.494 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:35.494 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 148492, 2024-08-19T12:40:35.494 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:35.494 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:35.494 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:35.494 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:35.494 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.495 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.496 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:35.497 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:35.498 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:35.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:35.502 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.503 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.508 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:35.509 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.510 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.512 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:35.513 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.513 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.513 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:35.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.513 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.513 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.513 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-08-19T12:40:35.513 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.513 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.514 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:35.514 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 10 librados: watch_flush enter 2024-08-19T12:40:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 10 librados: watch_flush exit 2024-08-19T12:40:35.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 20 client.25710.objecter shutdown clearing up homeless session... 2024-08-19T12:40:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 10 client.25710.objecter successfully canceled tick 2024-08-19T12:40:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.104:6800/899887506 conn(0x7f7204040e90 legacy=0x7f7204043350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 10 monclient: shutdown 2024-08-19T12:40:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 1 -- 172.21.15.26:0/2146681143 >> v1:172.21.15.26:6792/0 conn(0x7f721806e7b0 legacy=0x7f72181ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f7217fff640 1 -- 172.21.15.26:0/2146681143 reap_dead start 2024-08-19T12:40:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f7217fff640 1 -- 172.21.15.26:0/2146681143 reap_dead start 2024-08-19T12:40:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 1 -- 172.21.15.26:0/2146681143 shutdown_connections 2024-08-19T12:40:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 1 -- 172.21.15.26:0/2146681143 >> 172.21.15.26:0/2146681143 conn(0x7f72180a8740 msgr2=0x7f72180ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 1 -- 172.21.15.26:0/2146681143 shutdown_connections 2024-08-19T12:40:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 1 -- 172.21.15.26:0/2146681143 wait complete. 2024-08-19T12:40:35.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.479+0000 7f721e1aa640 1 librados: shutdown 2024-08-19T12:40:35.517 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:35.517 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:40:35.517 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:40:35.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.615+0000 7f419bcde640 10 monclient: get_monmap_and_config 2024-08-19T12:40:35.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.615+0000 7f419bcde640 10 monclient: build_initial_monmap 2024-08-19T12:40:35.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.615+0000 7f419bcde640 10 monclient: monmap: 2024-08-19T12:40:35.617 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:35.617593+0000 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:35.617593+0000 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:35.618 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:35.619 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:35.619 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.616+0000 7f419bcde640 1 Processor -- start 2024-08-19T12:40:35.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.616+0000 7f419bcde640 1 -- start start 2024-08-19T12:40:35.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.616+0000 7f419bcde640 10 monclient: init 2024-08-19T12:40:35.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.616+0000 7f419bcde640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:35.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.617+0000 7f419bcde640 10 monclient: _add_conns ranks=[2,6,1,7,5,4,0,8,3] 2024-08-19T12:40:35.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.617+0000 7f419bcde640 10 monclient(hunting): picked mon.noname-b con 0x7f419406e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:35.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.617+0000 7f419bcde640 10 monclient(hunting): picked mon.noname-d con 0x7f41940a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:35.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.617+0000 7f419bcde640 10 monclient(hunting): picked mon.noname-f con 0x7f41940a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:35.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.617+0000 7f419bcde640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4194136790 con 0x7f41940a5d50 2024-08-19T12:40:35.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.617+0000 7f419bcde640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419409ddf0 con 0x7f419406e7b0 2024-08-19T12:40:35.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.617+0000 7f419bcde640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419409e0d0 con 0x7f41940a5920 2024-08-19T12:40:35.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.617+0000 7f419bcde640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:35.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.617+0000 7f419bcde640 10 monclient(hunting): authenticate will time out at 829.705097s 2024-08-19T12:40:35.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.617+0000 7f4199a7a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f419406e7b0 0x7f4194136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58318/0 (socket says 172.21.15.26:58318) 2024-08-19T12:40:35.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.617+0000 7f4199a7a640 1 -- 172.21.15.26:0/2410571354 learned_addr learned my addr 172.21.15.26:0/2410571354 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:35.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.617+0000 7f4199279640 1 --1- 172.21.15.26:0/2410571354 >> v1:172.21.15.26:6792/0 conn(0x7f41940a5920 0x7f419413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39178/0 (socket says 172.21.15.26:39178) 2024-08-19T12:40:35.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4199a7a640 1 -- 172.21.15.26:0/2410571354 >> v1:172.21.15.26:6790/0 conn(0x7f419406e7b0 legacy=0x7f4194136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4199279640 1 -- 172.21.15.26:0/2410571354 >> v1:172.21.15.26:6792/0 conn(0x7f41940a5920 legacy=0x7f419413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2393365124 0 0) 0x7f4190001450 con 0x7f419406e7b0 2024-08-19T12:40:35.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient(hunting): my global_id is 25472 2024-08-19T12:40:35.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4180003520 con 0x7f419406e7b0 2024-08-19T12:40:35.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f419a27b640 1 -- 172.21.15.26:0/2410571354 >> v1:172.21.15.104:6789/0 conn(0x7f41940a5d50 legacy=0x7f4194141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2561742478 0 0) 0x7f419409e0d0 con 0x7f41940a5920 2024-08-19T12:40:35.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient(hunting): my global_id is 25719 2024-08-19T12:40:35.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4190001450 con 0x7f41940a5920 2024-08-19T12:40:35.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 606046336 0 0) 0x7f4194136790 con 0x7f41940a5d50 2024-08-19T12:40:35.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient(hunting): my global_id is 26155 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f419409e0d0 con 0x7f41940a5d50 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 788244505 0 0) 0x7f4180003520 con 0x7f419406e7b0 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 >> v1:172.21.15.26:6792/0 conn(0x7f41940a5920 legacy=0x7f419413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 >> v1:172.21.15.104:6789/0 conn(0x7f41940a5d50 legacy=0x7f4194141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient: _finish_hunting 0 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 monclient: found mon.noname-b 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419409e3b0 con 0x7f419406e7b0 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient: _finish_auth 0 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient: _check_auth_tickets 2024-08-19T12:40:35.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f419bcde640 5 monclient: authenticate success, global_id 25472 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4190001cb0 con 0x7f419406e7b0 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f419bcde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient: dump: 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.632 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f419bcde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4190004950 con 0x7f419406e7b0 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f419bcde640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:35.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 1 -- 172.21.15.26:0/2410571354 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4190004ec0 con 0x7f419406e7b0 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f419bcde640 4 set_mon_vals no callback set 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.619+0000 7f4198a78640 10 monclient: dump: 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.634 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.620+0000 7f419bcde640 10 monclient: shutdown 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.620+0000 7f419bcde640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.620+0000 7f419bcde640 1 -- 172.21.15.26:0/2410571354 >> v1:172.21.15.26:6790/0 conn(0x7f419406e7b0 legacy=0x7f4194136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.620+0000 7f419bcde640 1 -- 172.21.15.26:0/2410571354 shutdown_connections 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.620+0000 7f419bcde640 1 -- 172.21.15.26:0/2410571354 >> 172.21.15.26:0/2410571354 conn(0x7f41940a8740 msgr2=0x7f41940a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.620+0000 7f419bcde640 1 -- 172.21.15.26:0/2410571354 shutdown_connections 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.620+0000 7f419bcde640 1 -- 172.21.15.26:0/2410571354 wait complete. 2024-08-19T12:40:35.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 10 monclient: build_initial_monmap 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 10 monclient: monmap: 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:35.636 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 1 librados: starting msgr at 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 1 librados: starting objecter 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 1 Processor -- start 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 1 -- start start 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 1 librados: setting wanted keys 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 1 librados: calling monclient init 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 10 monclient: init 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 10 monclient: _add_conns ranks=[2,6,7,8,1,4,0,3,5] 2024-08-19T12:40:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 10 monclient(hunting): picked mon.noname-b con 0x7f419406e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 10 monclient(hunting): picked mon.noname-d con 0x7f41940a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 10 monclient(hunting): picked mon.noname-i con 0x7f41940a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419413b340 con 0x7f419406e7b0 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41941cf060 con 0x7f41940a5920 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4194138fe0 con 0x7f41940a5d50 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.621+0000 7f419bcde640 10 monclient(hunting): authenticate will time out at 829.709997s 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.622+0000 7f4199a7a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f419406e7b0 0x7f41941ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:58326/0 (socket says 172.21.15.26:58326) 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.622+0000 7f4199a7a640 1 -- 172.21.15.26:0/74760331 learned_addr learned my addr 172.21.15.26:0/74760331 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.622+0000 7f4199279640 1 --1- 172.21.15.26:0/74760331 >> v1:172.21.15.26:6792/0 conn(0x7f41940a5920 0x7f41941ce950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39186/0 (socket says 172.21.15.26:39186) 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f4199a7a640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.26:6790/0 conn(0x7f419406e7b0 legacy=0x7f41941ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f4199279640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.26:6792/0 conn(0x7f41940a5920 legacy=0x7f41941ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 client.?.objecter ms_handle_connect 0x7f419406e7b0 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 client.?.objecter ms_handle_connect 0x7f41940a5920 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1816306796 0 0) 0x7f41941cf060 con 0x7f41940a5920 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): my global_id is 25728 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4178004ee0 con 0x7f41940a5920 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3373805057 0 0) 0x7f419413b340 con 0x7f419406e7b0 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): my global_id is 25481 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41941cf060 con 0x7f419406e7b0 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f419a27b640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.104:6792/0 conn(0x7f41940a5d50 legacy=0x7f41941d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 client.?.objecter ms_handle_connect 0x7f41940a5d50 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3568591915 0 0) 0x7f4194138fe0 con 0x7f41940a5d50 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): my global_id is 134206 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f419413b340 con 0x7f41940a5d50 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 82756955 0 0) 0x7f4178004ee0 con 0x7f41940a5920 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4194138fe0 con 0x7f41940a5920 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f41840047e0 con 0x7f41940a5920 2024-08-19T12:40:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): dump: 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.641 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1222662212 0 0) 0x7f41941cf060 con 0x7f419406e7b0 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4178004ee0 con 0x7f419406e7b0 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4190001cb0 con 0x7f419406e7b0 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient(hunting): dump: 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.642 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.643 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 308006838 0 0) 0x7f4194138fe0 con 0x7f41940a5920 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.104:6792/0 conn(0x7f41940a5d50 legacy=0x7f41941d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.26:6790/0 conn(0x7f419406e7b0 legacy=0x7f41941ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 monclient: found mon.d 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419413aaa0 con 0x7f41940a5920 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f418a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f419bcde640 5 monclient: authenticate success, global_id 25728 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f419bcde640 10 monclient: _renew_subs 2024-08-19T12:40:35.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f419bcde640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f419bcde640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f419413ad80 con 0x7f41940a5920 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f419bcde640 10 client.25728.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f419bcde640 10 monclient: _renew_subs 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f419bcde640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f419bcde640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f4194138550 con 0x7f41940a5920 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.624+0000 7f419bcde640 1 librados: init done 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.625+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4184004c30 con 0x7f41940a5920 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.625+0000 7f418a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.625+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4184005120 con 0x7f41940a5920 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.625+0000 7f418a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.625+0000 7f418bfff640 4 set_mon_vals no callback set 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.625+0000 7f418a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.625+0000 7f418a7fc640 10 monclient: dump: 2024-08-19T12:40:35.645 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.646 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.647 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.625+0000 7f419bcde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.625+0000 7f419bcde640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f417c0065c0 con 0x7f417c002860 2024-08-19T12:40:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.625+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f41840052c0 con 0x7f41940a5920 2024-08-19T12:40:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.626+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(78..78 src has 1..78) ==== 6992+0+0 (unknown 3025720083 0 0) 0x7f418404e7d0 con 0x7f41940a5920 2024-08-19T12:40:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.626+0000 7f418a7fc640 10 client.25728.objecter ms_dispatch 0x7f4194000b90 osd_map(78..78 src has 1..78) 2024-08-19T12:40:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.626+0000 7f418a7fc640 3 client.25728.objecter handle_osd_map got epochs [78,78] > 0 2024-08-19T12:40:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.626+0000 7f418a7fc640 3 client.25728.objecter handle_osd_map decoding full epoch 78 2024-08-19T12:40:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.626+0000 7f418a7fc640 20 client.25728.objecter dump_active .. 0 homeless 2024-08-19T12:40:35.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.626+0000 7f419a27b640 10 monclient: get_auth_request con 0x7f4178041750 auth_method 0 2024-08-19T12:40:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 10 client.25728.objecter ms_handle_connect 0x7f417c002860 2024-08-19T12:40:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 10 client.25728.objecter resend_mon_ops 2024-08-19T12:40:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f4199a7a640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.104:6791/0 conn(0x7f417c002860 legacy=0x7f417c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3069463081 0 0) 0x7f417c0065c0 con 0x7f417c002860 2024-08-19T12:40:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 10 monclient(hunting): my global_id is 25475 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4194138fe0 con 0x7f417c002860 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 10 client.25728.objecter ms_handle_connect 0x7f4178041750 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 11758461 0 0) 0x7f4194138fe0 con 0x7f417c002860 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f417c0065c0 con 0x7f417c002860 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4190004f00 con 0x7f417c002860 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 10 monclient: dump: 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.651 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3436209600 0 0) 0x7f417c0065c0 con 0x7f417c002860 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.628+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f417c00a160 con 0x7f417c002860 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.629+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4190005200 con 0x7f417c002860 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.629+0000 7f418a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.629+0000 7f418a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.629+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.104:6791/0 conn(0x7f417c002860 legacy=0x7f417c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.629+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.104:6791/0 conn(0x7f417c002860 legacy=0x7f417c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.635+0000 7f419bcde640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.635+0000 7f419bcde640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f417c009e70 con 0x7f417c009130 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.637+0000 7f4199a7a640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.104:6791/0 conn(0x7f417c009130 legacy=0x7f417c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.637+0000 7f418a7fc640 10 client.25728.objecter ms_handle_connect 0x7f417c009130 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.637+0000 7f418a7fc640 10 client.25728.objecter resend_mon_ops 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.637+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3153635739 0 0) 0x7f417c009e70 con 0x7f417c009130 2024-08-19T12:40:35.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.637+0000 7f418a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.637+0000 7f418a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.637+0000 7f418a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.637+0000 7f418a7fc640 10 monclient(hunting): my global_id is 25484 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.637+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f417c0065c0 con 0x7f417c009130 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3557256830 0 0) 0x7f417c0065c0 con 0x7f417c009130 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f417c009e70 con 0x7f417c009130 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4190005ad0 con 0x7f417c009130 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 10 monclient: dump: 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.654 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1651295982 0 0) 0x7f417c009e70 con 0x7f417c009130 2024-08-19T12:40:35.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f417c00c410 con 0x7f417c009130 2024-08-19T12:40:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 3379390165) 0x7f4190007eb0 con 0x7f417c009130 2024-08-19T12:40:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.104:6791/0 conn(0x7f417c009130 legacy=0x7f417c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f418a7fc640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.104:6791/0 conn(0x7f417c009130 legacy=0x7f417c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.656 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:35.656 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:35.656 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:35.656 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 148662, 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:35.657 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.658 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.659 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:35.660 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.661 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:35.662 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.663 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.664 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:35.666 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:35.668 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:35.669 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:35.672 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:35.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.675 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:35.676 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f419bcde640 10 librados: watch_flush enter 2024-08-19T12:40:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f419bcde640 10 librados: watch_flush exit 2024-08-19T12:40:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f419bcde640 20 client.25728.objecter shutdown clearing up homeless session... 2024-08-19T12:40:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.638+0000 7f419bcde640 10 client.25728.objecter successfully canceled tick 2024-08-19T12:40:35.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.639+0000 7f419bcde640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.104:6800/899887506 conn(0x7f4178041750 legacy=0x7f4178043c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.639+0000 7f419bcde640 10 monclient: shutdown 2024-08-19T12:40:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.639+0000 7f419bcde640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.639+0000 7f419bcde640 1 -- 172.21.15.26:0/74760331 >> v1:172.21.15.26:6792/0 conn(0x7f41940a5920 legacy=0x7f41941ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.639+0000 7f419a27b640 1 -- 172.21.15.26:0/74760331 reap_dead start 2024-08-19T12:40:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.639+0000 7f419a27b640 1 -- 172.21.15.26:0/74760331 reap_dead start 2024-08-19T12:40:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.639+0000 7f419bcde640 1 -- 172.21.15.26:0/74760331 shutdown_connections 2024-08-19T12:40:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.639+0000 7f419bcde640 1 -- 172.21.15.26:0/74760331 >> 172.21.15.26:0/74760331 conn(0x7f41940a8740 msgr2=0x7f4194135170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.639+0000 7f419bcde640 1 -- 172.21.15.26:0/74760331 shutdown_connections 2024-08-19T12:40:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.639+0000 7f419bcde640 1 -- 172.21.15.26:0/74760331 wait complete. 2024-08-19T12:40:35.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.639+0000 7f419bcde640 1 librados: shutdown 2024-08-19T12:40:35.679 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:35.679 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:40:35.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:40:35.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.776+0000 7f339e260640 10 monclient: get_monmap_and_config 2024-08-19T12:40:35.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.776+0000 7f339e260640 10 monclient: build_initial_monmap 2024-08-19T12:40:35.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.776+0000 7f339e260640 10 monclient: monmap: 2024-08-19T12:40:35.778 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:35.778381+0000 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:35.778381+0000 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:35.779 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:35.780 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 1 Processor -- start 2024-08-19T12:40:35.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 1 -- start start 2024-08-19T12:40:35.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 10 monclient: init 2024-08-19T12:40:35.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:35.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 10 monclient: _add_conns ranks=[3,5,7,4,8,1,0,6,2] 2024-08-19T12:40:35.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 10 monclient(hunting): picked mon.noname-g con 0x7f339806e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:35.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 10 monclient(hunting): picked mon.noname-h con 0x7f33980a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:35.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 10 monclient(hunting): picked mon.noname-i con 0x7f33980a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:35.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3398136790 con 0x7f339806e7b0 2024-08-19T12:40:35.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f339809ddf0 con 0x7f33980a5920 2024-08-19T12:40:35.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f339809e0d0 con 0x7f33980a5d50 2024-08-19T12:40:35.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:35.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.777+0000 7f339e260640 10 monclient(hunting): authenticate will time out at 829.865850s 2024-08-19T12:40:35.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.778+0000 7f33977fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f339806e7b0 0x7f3398136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:46212/0 (socket says 172.21.15.26:46212) 2024-08-19T12:40:35.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.778+0000 7f33977fe640 1 -- 172.21.15.26:0/2353258199 learned_addr learned my addr 172.21.15.26:0/2353258199 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:35.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.778+0000 7f3396ffd640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f33980a5920 0x7f339813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:35016/0 (socket says 172.21.15.26:35016) 2024-08-19T12:40:35.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.778+0000 7f3397fff640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f33980a5d50 0x7f3398141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:51386/0 (socket says 172.21.15.26:51386) 2024-08-19T12:40:35.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33977fe640 1 -- 172.21.15.26:0/2353258199 >> v1:172.21.15.104:6790/0 conn(0x7f339806e7b0 legacy=0x7f3398136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f3397fff640 1 -- 172.21.15.26:0/2353258199 >> v1:172.21.15.104:6792/0 conn(0x7f33980a5d50 legacy=0x7f3398141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f3396ffd640 1 -- 172.21.15.26:0/2353258199 >> v1:172.21.15.104:6791/0 conn(0x7f33980a5920 legacy=0x7f339813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4102538495 0 0) 0x7f3398136790 con 0x7f339806e7b0 2024-08-19T12:40:35.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient(hunting): my global_id is 25617 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3380003520 con 0x7f339806e7b0 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 681627732 0 0) 0x7f339809e0d0 con 0x7f33980a5d50 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient(hunting): my global_id is 134215 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3398136790 con 0x7f33980a5d50 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2108189100 0 0) 0x7f338c001450 con 0x7f33980a5920 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient(hunting): my global_id is 25493 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f339809e0d0 con 0x7f33980a5920 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1496413303 0 0) 0x7f3380003520 con 0x7f339806e7b0 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 >> v1:172.21.15.104:6792/0 conn(0x7f33980a5d50 legacy=0x7f3398141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 >> v1:172.21.15.104:6791/0 conn(0x7f33980a5920 legacy=0x7f339813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient: _finish_hunting 0 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 monclient: found mon.noname-g 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339809e3b0 con 0x7f339806e7b0 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient: _finish_auth 0 2024-08-19T12:40:35.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient: _check_auth_tickets 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3388001ba0 con 0x7f339806e7b0 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient: dump: 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f339e260640 5 monclient: authenticate success, global_id 25617 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f339e260640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33977fe640 1 --1- 172.21.15.26:0/2353258199 >> v1:172.21.15.104:6790/0 conn(0x7f339806e7b0 0x7f3398136080 :-1 s=OPENED pgs=3027 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f33880048a0 config(0 keys) 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f33880048a0 con 0x7f339806e7b0 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f33967fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f339e260640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:35.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.780+0000 7f339e260640 4 set_mon_vals no callback set 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.781+0000 7f339e260640 10 monclient: shutdown 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.781+0000 7f339e260640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.781+0000 7f339e260640 1 -- 172.21.15.26:0/2353258199 >> v1:172.21.15.104:6790/0 conn(0x7f339806e7b0 legacy=0x7f3398136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.781+0000 7f33967fc640 1 -- 172.21.15.26:0/2353258199 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3388004e10 con 0x7f339806e7b0 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.781+0000 7f33967fc640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.781+0000 7f339e260640 1 -- 172.21.15.26:0/2353258199 shutdown_connections 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.781+0000 7f339e260640 1 -- 172.21.15.26:0/2353258199 >> 172.21.15.26:0/2353258199 conn(0x7f33980a8740 msgr2=0x7f33980a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.781+0000 7f339e260640 1 -- 172.21.15.26:0/2353258199 shutdown_connections 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.781+0000 7f339e260640 1 -- 172.21.15.26:0/2353258199 wait complete. 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.781+0000 7f339e260640 10 monclient: build_initial_monmap 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.781+0000 7f339e260640 10 monclient: monmap: 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:35.791 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 1 librados: starting msgr at 2024-08-19T12:40:35.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 1 librados: starting objecter 2024-08-19T12:40:35.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 1 Processor -- start 2024-08-19T12:40:35.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 1 -- start start 2024-08-19T12:40:35.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 1 librados: setting wanted keys 2024-08-19T12:40:35.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 1 librados: calling monclient init 2024-08-19T12:40:35.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 10 monclient: init 2024-08-19T12:40:35.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:35.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 10 monclient: _add_conns ranks=[8,6,0,1,4,2,3,7,5] 2024-08-19T12:40:35.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 10 monclient(hunting): picked mon.noname-e con 0x7f339806e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:35.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 10 monclient(hunting): picked mon.noname-d con 0x7f33980a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:35.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 10 monclient(hunting): picked mon.noname-a con 0x7f33980a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:35.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33980a2e30 con 0x7f33980a5d50 2024-08-19T12:40:35.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33980a3110 con 0x7f33980a5920 2024-08-19T12:40:35.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33980a33f0 con 0x7f339806e7b0 2024-08-19T12:40:35.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:35.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f339e260640 10 monclient(hunting): authenticate will time out at 829.870933s 2024-08-19T12:40:35.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f33977fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f339806e7b0 0x7f33981ce670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50296/0 (socket says 172.21.15.26:50296) 2024-08-19T12:40:35.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.782+0000 7f33977fe640 1 -- 172.21.15.26:0/2474519589 learned_addr learned my addr 172.21.15.26:0/2474519589 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:35.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f3396ffd640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.26:6792/0 conn(0x7f33980a5920 legacy=0x7f33981d3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 client.?.objecter ms_handle_connect 0x7f33980a5920 2024-08-19T12:40:35.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f33977fe640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.26:6793/0 conn(0x7f339806e7b0 legacy=0x7f33981ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 client.?.objecter ms_handle_connect 0x7f339806e7b0 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 client.?.objecter ms_handle_connect 0x7f33980a5d50 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f3397fff640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.26:6789/0 conn(0x7f33980a5d50 legacy=0x7f33981d6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f33977fe640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.26:6793/0 conn(0x7f339806e7b0 legacy=0x7f33981ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f33977fe640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.26:6793/0 conn(0x7f339806e7b0 legacy=0x7f33981ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f33977fe640 1 --1- 172.21.15.26:0/2474519589 >> v1:172.21.15.26:6793/0 conn(0x7f339806e7b0 0x7f33981ce670 :-1 s=OPENED pgs=3089 cs=1 l=1).handle_message read tag failed 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f33977fe640 1 --1- 172.21.15.26:0/2474519589 >> v1:172.21.15.26:6793/0 conn(0x7f339806e7b0 0x7f33981ce670 :-1 s=OPENED pgs=3089 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6793/0 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3156396702 0 0) 0x7f33980a3110 con 0x7f33980a5920 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): my global_id is 25737 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3384004eb0 con 0x7f33980a5920 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2517712013 0 0) 0x7f33980a2e30 con 0x7f33980a5d50 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): my global_id is 26982 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33980a3110 con 0x7f33980a5d50 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3484621078 0 0) 0x7f3384004eb0 con 0x7f33980a5920 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f33980a2e30 con 0x7f33980a5920 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2730517052 0 0) 0x7f33980a3110 con 0x7f33980a5d50 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3384004eb0 con 0x7f33980a5d50 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f338c004b20 con 0x7f33980a5920 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:35.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): dump: 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.805 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.806 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.806 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.806 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.806 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:35.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3390002020 con 0x7f33980a5d50 2024-08-19T12:40:35.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:35.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:40:35.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient(hunting): dump: 2024-08-19T12:40:35.806 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.807 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1358996643 0 0) 0x7f33980a2e30 con 0x7f33980a5920 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.26:6793/0 conn(0x7f339806e7b0 legacy=0x7f33981ce670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.26:6789/0 conn(0x7f33980a5d50 legacy=0x7f33981d6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient: _finish_hunting 0 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 monclient: found mon.d 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33981347a0 con 0x7f33980a5920 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient: _finish_auth 0 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 10 monclient: _check_auth_tickets 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f337ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f339e260640 5 monclient: authenticate success, global_id 25737 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f339e260640 10 monclient: _renew_subs 2024-08-19T12:40:35.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f339e260640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f339e260640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f33980a3dd0 con 0x7f33980a5920 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f339e260640 10 client.25737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f339e260640 10 monclient: _renew_subs 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f339e260640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f339e260640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f33980a09d0 con 0x7f33980a5920 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.785+0000 7f339e260640 1 librados: init done 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.786+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f338c001c50 con 0x7f33980a5920 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.786+0000 7f337ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.786+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f338c0052b0 con 0x7f33980a5920 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.786+0000 7f337ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.786+0000 7f337ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.786+0000 7f3395ffb640 4 set_mon_vals no callback set 2024-08-19T12:40:35.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.786+0000 7f337ffff640 10 monclient: dump: 2024-08-19T12:40:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.810 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.786+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f338c011710 con 0x7f33980a5920 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.786+0000 7f339e260640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.786+0000 7f339e260640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f33780065c0 con 0x7f3378002860 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.787+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(78..78 src has 1..78) ==== 6992+0+0 (unknown 3025720083 0 0) 0x7f338c04e830 con 0x7f33980a5920 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.787+0000 7f337ffff640 10 client.25737.objecter ms_dispatch 0x7f3398000b90 osd_map(78..78 src has 1..78) 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.787+0000 7f337ffff640 3 client.25737.objecter handle_osd_map got epochs [78,78] > 0 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.787+0000 7f337ffff640 3 client.25737.objecter handle_osd_map decoding full epoch 78 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.787+0000 7f337ffff640 20 client.25737.objecter dump_active .. 0 homeless 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.787+0000 7f3397fff640 10 monclient: get_auth_request con 0x7f33840414a0 auth_method 0 2024-08-19T12:40:35.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f33977fe640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.104:6792/0 conn(0x7f3378002860 legacy=0x7f3378004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 10 client.25737.objecter ms_handle_connect 0x7f3378002860 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 10 client.25737.objecter resend_mon_ops 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3698552507 0 0) 0x7f33780065c0 con 0x7f3378002860 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 10 monclient(hunting): my global_id is 134224 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33980a2e30 con 0x7f3378002860 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 10 client.25737.objecter ms_handle_connect 0x7f33840414a0 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 229429936 0 0) 0x7f33980a2e30 con 0x7f3378002860 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f33780065c0 con 0x7f3378002860 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3388004e10 con 0x7f3378002860 2024-08-19T12:40:35.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 10 monclient: dump: 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.813 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1732075901 0 0) 0x7f33780065c0 con 0x7f3378002860 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.789+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f337800a160 con 0x7f3378002860 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.790+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3388001820 con 0x7f3378002860 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.790+0000 7f337ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.790+0000 7f337ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.790+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.104:6792/0 conn(0x7f3378002860 legacy=0x7f3378004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.790+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.104:6792/0 conn(0x7f3378002860 legacy=0x7f3378004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.796+0000 7f339e260640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.796+0000 7f339e260640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3378009e70 con 0x7f3378009130 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f33977fe640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.104:6792/0 conn(0x7f3378009130 legacy=0x7f33780095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:35.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 10 client.25737.objecter ms_handle_connect 0x7f3378009130 2024-08-19T12:40:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 10 client.25737.objecter resend_mon_ops 2024-08-19T12:40:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4018962586 0 0) 0x7f3378009e70 con 0x7f3378009130 2024-08-19T12:40:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 10 monclient(hunting): my global_id is 134233 2024-08-19T12:40:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33780065c0 con 0x7f3378009130 2024-08-19T12:40:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4129356754 0 0) 0x7f33780065c0 con 0x7f3378009130 2024-08-19T12:40:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3378009e70 con 0x7f3378009130 2024-08-19T12:40:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f33880051c0 con 0x7f3378009130 2024-08-19T12:40:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:35.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 10 monclient: dump: 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:35.816 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:35.817 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 755847642 0 0) 0x7f3378009e70 con 0x7f3378009130 2024-08-19T12:40:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f337800c410 con 0x7f3378009130 2024-08-19T12:40:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 3648647506) 0x7f33880080a0 con 0x7f3378009130 2024-08-19T12:40:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.104:6792/0 conn(0x7f3378009130 legacy=0x7f33780095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f337ffff640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.104:6792/0 conn(0x7f3378009130 legacy=0x7f33780095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:35.817 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 10, 2024-08-19T12:40:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 6157, 2024-08-19T12:40:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:40:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:35.837 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 10 librados: watch_flush enter 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 10 librados: watch_flush exit 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 20 client.25737.objecter shutdown clearing up homeless session... 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 10 client.25737.objecter successfully canceled tick 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.104:6800/899887506 conn(0x7f33840414a0 legacy=0x7f3384043960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 10 monclient: shutdown 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 1 -- 172.21.15.26:0/2474519589 >> v1:172.21.15.26:6792/0 conn(0x7f33980a5920 legacy=0x7f33981d3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f3397fff640 1 -- 172.21.15.26:0/2474519589 reap_dead start 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f3397fff640 1 -- 172.21.15.26:0/2474519589 reap_dead start 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 1 -- 172.21.15.26:0/2474519589 shutdown_connections 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 1 -- 172.21.15.26:0/2474519589 >> 172.21.15.26:0/2474519589 conn(0x7f33980a8740 msgr2=0x7f33980aa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:35.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 1 -- 172.21.15.26:0/2474519589 shutdown_connections 2024-08-19T12:40:35.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 1 -- 172.21.15.26:0/2474519589 wait complete. 2024-08-19T12:40:35.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:35.800+0000 7f339e260640 1 librados: shutdown 2024-08-19T12:40:35.840 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:40:35.840 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-08-19T12:40:35.840 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c', 'h'] 2024-08-19T12:40:35.840 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-08-19T12:40:35.840 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-08-19T12:40:35.841 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-19T12:40:35.841 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:40:35.853 INFO:tasks.ceph.mon.a:Stopped 2024-08-19T12:40:35.853 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-08-19T12:40:35.853 INFO:tasks.ceph.mon.c:Sent signal 19 2024-08-19T12:40:35.853 INFO:tasks.mon_thrash:Sending STOP to mon h 2024-08-19T12:40:35.854 INFO:tasks.ceph.mon.h:Sent signal 19 2024-08-19T12:40:35.854 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:40:50.855 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-08-19T12:40:50.855 INFO:tasks.ceph.mon.c:Sent signal 18 2024-08-19T12:40:50.855 INFO:tasks.mon_thrash:Sending CONT to mon h 2024-08-19T12:40:50.856 INFO:tasks.ceph.mon.h:Sent signal 18 2024-08-19T12:40:50.856 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-08-19T12:40:50.856 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:40:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.967+0000 7faa49586640 10 monclient: get_monmap_and_config 2024-08-19T12:40:50.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.967+0000 7faa49586640 10 monclient: build_initial_monmap 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.967+0000 7faa49586640 10 monclient: monmap: 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:50.968608+0000 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:50.968608+0000 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:50.969 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:50.970 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:50.970 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.968+0000 7faa49586640 1 Processor -- start 2024-08-19T12:40:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.968+0000 7faa49586640 1 -- start start 2024-08-19T12:40:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.968+0000 7faa49586640 10 monclient: init 2024-08-19T12:40:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa49586640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa49586640 10 monclient: _add_conns ranks=[5,0,3,2,8,6,4,1,7] 2024-08-19T12:40:50.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa49586640 10 monclient(hunting): picked mon.noname-h con 0x7faa4406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa49586640 10 monclient(hunting): picked mon.noname-a con 0x7faa440a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa49586640 10 monclient(hunting): picked mon.noname-g con 0x7faa440a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa49586640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa44136790 con 0x7faa440a5920 2024-08-19T12:40:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa49586640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4409ddf0 con 0x7faa440a5d50 2024-08-19T12:40:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa49586640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4409e0d0 con 0x7faa4406e7b0 2024-08-19T12:40:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa49586640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa49586640 10 monclient(hunting): authenticate will time out at 845.056141s 2024-08-19T12:40:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa3bfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7faa440a5920 legacy=0x7faa4413e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:40:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa42ffd640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7faa4406e7b0 0x7faa44136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:57826/0 (socket says 172.21.15.26:57826) 2024-08-19T12:40:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa42ffd640 1 -- 172.21.15.26:0/2517724575 learned_addr learned my addr 172.21.15.26:0/2517724575 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:50.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.969+0000 7faa437fe640 1 --1- 172.21.15.26:0/2517724575 >> v1:172.21.15.104:6790/0 conn(0x7faa440a5d50 0x7faa44141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:56290/0 (socket says 172.21.15.26:56290) 2024-08-19T12:40:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.971+0000 7faa437fe640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.104:6790/0 conn(0x7faa440a5d50 legacy=0x7faa44141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:50.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:50.971+0000 7faa42ffd640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.104:6791/0 conn(0x7faa4406e7b0 legacy=0x7faa44136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:51.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:51.170+0000 7faa3bfff640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.26:6789/0 conn(0x7faa440a5920 legacy=0x7faa4413e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:40:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:51.571+0000 7faa3bfff640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.26:6789/0 conn(0x7faa440a5920 legacy=0x7faa4413e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:40:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:52.372+0000 7faa3bfff640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.26:6789/0 conn(0x7faa440a5920 legacy=0x7faa4413e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:40:53.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 10 monclient(hunting): tick 2024-08-19T12:40:53.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:40:53.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 1 monclient(hunting): continuing hunt 2024-08-19T12:40:53.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:40:53.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.104:6791/0 conn(0x7faa4406e7b0 legacy=0x7faa44136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:53.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.104:6790/0 conn(0x7faa440a5d50 legacy=0x7faa44141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:53.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.26:6789/0 conn(0x7faa440a5920 legacy=0x7faa4413e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:40:53.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 10 monclient: _add_conns ranks=[1,8,7,6,4,2] 2024-08-19T12:40:53.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 10 monclient(hunting): picked mon.noname-f con 0x7faa240038c0 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:53.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 10 monclient(hunting): picked mon.noname-e con 0x7faa24007530 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:53.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 10 monclient(hunting): picked mon.noname-i con 0x7faa2400b2a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:53.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 1 -- 172.21.15.26:0/2517724575 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4409e3b0 con 0x7faa240038c0 2024-08-19T12:40:53.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 1 -- 172.21.15.26:0/2517724575 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa44136790 con 0x7faa2400b2a0 2024-08-19T12:40:53.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 1 -- 172.21.15.26:0/2517724575 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa24011110 con 0x7faa24007530 2024-08-19T12:40:53.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.969+0000 7faa417fa640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:53.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.971+0000 7faa3bfff640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.26:6793/0 conn(0x7faa24007530 legacy=0x7faa240099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:53.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.971+0000 7faa437fe640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.104:6792/0 conn(0x7faa2400b2a0 legacy=0x7faa2400d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:53.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:53.971+0000 7faa42ffd640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.104:6789/0 conn(0x7faa240038c0 legacy=0x7faa24005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:55.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 1 -- 172.21.15.26:0/2517724575 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1660987253 0 0) 0x7faa44136790 con 0x7faa2400b2a0 2024-08-19T12:40:55.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:55.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:55.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:55.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient(hunting): my global_id is 134269 2024-08-19T12:40:55.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 1 -- 172.21.15.26:0/2517724575 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa30003520 con 0x7faa2400b2a0 2024-08-19T12:40:55.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 1 -- 172.21.15.26:0/2517724575 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3877911322 0 0) 0x7faa24011110 con 0x7faa24007530 2024-08-19T12:40:55.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:55.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient(hunting): my global_id is 25325 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 1 -- 172.21.15.26:0/2517724575 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa30003880 con 0x7faa24007530 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 1 -- 172.21.15.26:0/2517724575 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2663613747 0 0) 0x7faa30003520 con 0x7faa2400b2a0 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.26:6793/0 conn(0x7faa24007530 legacy=0x7faa240099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.104:6789/0 conn(0x7faa240038c0 legacy=0x7faa24005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient: _finish_hunting 0 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 1 monclient: found mon.noname-i 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 1 -- 172.21.15.26:0/2517724575 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa24012470 con 0x7faa2400b2a0 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient: _finish_auth 0 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient: _check_auth_tickets 2024-08-19T12:40:55.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa49586640 5 monclient: authenticate success, global_id 134269 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa49586640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa437fe640 1 -- 172.21.15.26:0/2517724575 reap_dead start 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 1 -- 172.21.15.26:0/2517724575 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faa3c001da0 con 0x7faa2400b2a0 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient: dump: 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:55.873 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa49586640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 1 -- 172.21.15.26:0/2517724575 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa3c0020b0 con 0x7faa2400b2a0 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 1 -- 172.21.15.26:0/2517724575 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faa3c004ea0 con 0x7faa2400b2a0 2024-08-19T12:40:55.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.869+0000 7faa427fc640 10 monclient: dump: 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:55.875 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:55.876 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:55.876 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:55.876 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:55.876 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:55.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.870+0000 7faa49586640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:55.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.870+0000 7faa49586640 4 set_mon_vals no callback set 2024-08-19T12:40:55.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.871+0000 7faa49586640 10 monclient: shutdown 2024-08-19T12:40:55.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.871+0000 7faa49586640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:55.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.871+0000 7faa49586640 1 -- 172.21.15.26:0/2517724575 >> v1:172.21.15.104:6792/0 conn(0x7faa2400b2a0 legacy=0x7faa2400d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:55.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.871+0000 7faa49586640 1 -- 172.21.15.26:0/2517724575 shutdown_connections 2024-08-19T12:40:55.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.871+0000 7faa49586640 1 -- 172.21.15.26:0/2517724575 >> 172.21.15.26:0/2517724575 conn(0x7faa440a8740 msgr2=0x7faa440a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:55.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.871+0000 7faa49586640 1 -- 172.21.15.26:0/2517724575 shutdown_connections 2024-08-19T12:40:55.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.871+0000 7faa49586640 1 -- 172.21.15.26:0/2517724575 wait complete. 2024-08-19T12:40:55.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.872+0000 7faa49586640 10 monclient: build_initial_monmap 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.872+0000 7faa49586640 10 monclient: monmap: 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:55.879 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 1 librados: starting msgr at 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 1 librados: starting objecter 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 1 Processor -- start 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 1 -- start start 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 1 librados: setting wanted keys 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 1 librados: calling monclient init 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 10 monclient: init 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 10 monclient: _add_conns ranks=[6,3,2,7,5,0,4,1,8] 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 10 monclient(hunting): picked mon.noname-d con 0x7faa2400b2a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 10 monclient(hunting): picked mon.noname-g con 0x7faa441ce130 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 10 monclient(hunting): picked mon.noname-b con 0x7faa44132ed0 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa441cedb0 con 0x7faa44132ed0 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa440ab230 con 0x7faa441ce130 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa441259f0 con 0x7faa2400b2a0 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.873+0000 7faa49586640 10 monclient(hunting): authenticate will time out at 849.960929s 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.874+0000 7faa42ffd640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7faa2400b2a0 0x7faa441327c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:44138/0 (socket says 172.21.15.26:44138) 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.874+0000 7faa437fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7faa44132ed0 0x7faa441d6900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33352/0 (socket says 172.21.15.26:33352) 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.874+0000 7faa42ffd640 1 -- 172.21.15.26:0/911418121 learned_addr learned my addr 172.21.15.26:0/911418121 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.876+0000 7faa42ffd640 1 -- 172.21.15.26:0/911418121 >> v1:172.21.15.26:6792/0 conn(0x7faa2400b2a0 legacy=0x7faa441327c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.876+0000 7faa3b7fe640 10 client.?.objecter ms_handle_connect 0x7faa2400b2a0 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.876+0000 7faa3b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.876+0000 7faa3b7fe640 10 client.?.objecter ms_handle_connect 0x7faa44132ed0 2024-08-19T12:40:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.876+0000 7faa3b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:55.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.876+0000 7faa3b7fe640 10 client.?.objecter ms_handle_connect 0x7faa441ce130 2024-08-19T12:40:55.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.876+0000 7faa3b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:55.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.876+0000 7faa3bfff640 1 -- 172.21.15.26:0/911418121 >> v1:172.21.15.104:6790/0 conn(0x7faa441ce130 legacy=0x7faa441d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:55.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.876+0000 7faa437fe640 1 -- 172.21.15.26:0/911418121 >> v1:172.21.15.26:6790/0 conn(0x7faa44132ed0 legacy=0x7faa441d6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:55.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1012642597 0 0) 0x7faa440ab230 con 0x7faa441ce130 2024-08-19T12:40:55.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:55.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:55.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): my global_id is 25644 2024-08-19T12:40:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa2400aa40 con 0x7faa441ce130 2024-08-19T12:40:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701043899 0 0) 0x7faa441cedb0 con 0x7faa44132ed0 2024-08-19T12:40:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): my global_id is 25499 2024-08-19T12:40:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa440ab230 con 0x7faa44132ed0 2024-08-19T12:40:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2085680704 0 0) 0x7faa441259f0 con 0x7faa2400b2a0 2024-08-19T12:40:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:40:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:55.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): my global_id is 25746 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa441cedb0 con 0x7faa2400b2a0 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2666048094 0 0) 0x7faa440ab230 con 0x7faa44132ed0 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa441259f0 con 0x7faa44132ed0 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faa3c004ea0 con 0x7faa44132ed0 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): dump: 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:55.897 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:55.898 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:55.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1845187834 0 0) 0x7faa441cedb0 con 0x7faa2400b2a0 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa440ab230 con 0x7faa2400b2a0 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 744893348 0 0) 0x7faa2400aa40 con 0x7faa441ce130 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa441cedb0 con 0x7faa441ce130 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faa34001b70 con 0x7faa2400b2a0 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.892+0000 7faa3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient(hunting): dump: 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:55.899 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faa2c004750 con 0x7faa441ce130 2024-08-19T12:40:55.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient(hunting): dump: 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:55.901 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2399558120 0 0) 0x7faa441259f0 con 0x7faa44132ed0 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 >> v1:172.21.15.26:6792/0 conn(0x7faa2400b2a0 legacy=0x7faa441327c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 >> v1:172.21.15.104:6790/0 conn(0x7faa441ce130 legacy=0x7faa441d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 1 monclient: found mon.b 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa44124e90 con 0x7faa44132ed0 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient: _finish_auth 0 2024-08-19T12:40:55.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa49586640 5 monclient: authenticate success, global_id 25499 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa49586640 10 monclient: _renew_subs 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa49586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa49586640 1 -- 172.21.15.26:0/911418121 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faa441271f0 con 0x7faa44132ed0 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa49586640 10 client.25499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa49586640 10 monclient: _renew_subs 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa49586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa49586640 1 -- 172.21.15.26:0/911418121 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7faa44126c30 con 0x7faa44132ed0 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa49586640 1 librados: init done 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa3c001ff0 con 0x7faa44132ed0 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:55.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faa3c005630 con 0x7faa44132ed0 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa41ffb640 4 set_mon_vals no callback set 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.893+0000 7faa3b7fe640 10 monclient: dump: 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:55.904 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.894+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7faa3c011a90 con 0x7faa44132ed0 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.894+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(79..79 src has 1..79) ==== 6986+0+0 (unknown 1453033598 0 0) 0x7faa3c04e600 con 0x7faa44132ed0 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.894+0000 7faa3b7fe640 10 client.25499.objecter ms_dispatch 0x7faa44000b90 osd_map(79..79 src has 1..79) 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.894+0000 7faa3b7fe640 3 client.25499.objecter handle_osd_map got epochs [79,79] > 0 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.894+0000 7faa3b7fe640 3 client.25499.objecter handle_osd_map decoding full epoch 79 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.894+0000 7faa49586640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.894+0000 7faa49586640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:55.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.894+0000 7faa49586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:55.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.894+0000 7faa49586640 1 -- 172.21.15.26:0/911418121 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa28005080 con 0x7faa44132ed0 2024-08-19T12:40:55.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.894+0000 7faa3b7fe640 20 client.25499.objecter dump_active .. 0 homeless 2024-08-19T12:40:55.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.894+0000 7faa42ffd640 10 monclient: get_auth_request con 0x7faa24042f10 auth_method 0 2024-08-19T12:40:55.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.897+0000 7faa3b7fe640 10 client.25499.objecter ms_handle_connect 0x7faa24042f10 2024-08-19T12:40:55.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.899+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7faa4409ddf0 con 0x7faa44132ed0 2024-08-19T12:40:55.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.899+0000 7faa3b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:55.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:55.899+0000 7faa3b7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:56.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.178+0000 7faa49586640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:40:56.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.178+0000 7faa49586640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:40:56.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.178+0000 7faa49586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:40:56.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.178+0000 7faa49586640 1 -- 172.21.15.26:0/911418121 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faa28005370 con 0x7faa44132ed0 2024-08-19T12:40:56.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.179+0000 7faa3b7fe640 1 -- 172.21.15.26:0/911418121 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2723 (unknown 254746657 0 73996743) 0x7faa3c016e70 con 0x7faa44132ed0 2024-08-19T12:40:56.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.179+0000 7faa3b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:40:56.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.179+0000 7faa3b7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:56.180 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":14,"quorum":[1,2,3,4,5,6,7,8],"quorum_names":["f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:40:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.180+0000 7faa49586640 10 librados: watch_flush enter 2024-08-19T12:40:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.180+0000 7faa49586640 10 librados: watch_flush exit 2024-08-19T12:40:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.180+0000 7faa49586640 20 client.25499.objecter shutdown clearing up homeless session... 2024-08-19T12:40:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.180+0000 7faa49586640 10 client.25499.objecter successfully canceled tick 2024-08-19T12:40:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.181+0000 7faa49586640 1 -- 172.21.15.26:0/911418121 >> v1:172.21.15.104:6800/899887506 conn(0x7faa24042f10 legacy=0x7faa240453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.181+0000 7faa49586640 10 monclient: shutdown 2024-08-19T12:40:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.181+0000 7faa49586640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.181+0000 7faa49586640 1 -- 172.21.15.26:0/911418121 >> v1:172.21.15.26:6790/0 conn(0x7faa44132ed0 legacy=0x7faa441d6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.181+0000 7faa49586640 1 -- 172.21.15.26:0/911418121 shutdown_connections 2024-08-19T12:40:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.181+0000 7faa49586640 1 -- 172.21.15.26:0/911418121 >> 172.21.15.26:0/911418121 conn(0x7faa440a8740 msgr2=0x7faa44133970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.181+0000 7faa49586640 1 -- 172.21.15.26:0/911418121 shutdown_connections 2024-08-19T12:40:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.181+0000 7faa49586640 1 -- 172.21.15.26:0/911418121 wait complete. 2024-08-19T12:40:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.181+0000 7faa49586640 1 librados: shutdown 2024-08-19T12:40:56.192 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-08-19T12:40:56.192 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-08-19T12:40:56.193 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:40:56.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.291+0000 7fa84c1b8640 10 monclient: get_monmap_and_config 2024-08-19T12:40:56.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.291+0000 7fa84c1b8640 10 monclient: build_initial_monmap 2024-08-19T12:40:56.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.291+0000 7fa84c1b8640 10 monclient: monmap: 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:56.292709+0000 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:56.292709+0000 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:56.294 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:56.294 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.292+0000 7fa84c1b8640 1 Processor -- start 2024-08-19T12:40:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.292+0000 7fa84c1b8640 1 -- start start 2024-08-19T12:40:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.292+0000 7fa84c1b8640 10 monclient: init 2024-08-19T12:40:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa84c1b8640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa84c1b8640 10 monclient: _add_conns ranks=[1,0,5,2,7,6,3,8,4] 2024-08-19T12:40:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa84c1b8640 10 monclient(hunting): picked mon.noname-f con 0x7fa84406e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa84c1b8640 10 monclient(hunting): picked mon.noname-a con 0x7fa8440a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:56.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa84c1b8640 10 monclient(hunting): picked mon.noname-h con 0x7fa8440a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa84c1b8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa844136790 con 0x7fa8440a5920 2024-08-19T12:40:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa84c1b8640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa84409ddf0 con 0x7fa84406e7b0 2024-08-19T12:40:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa849753640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa8440a5920 legacy=0x7fa84413e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:40:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa84c1b8640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa84409e0d0 con 0x7fa8440a5d50 2024-08-19T12:40:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa84c1b8640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa84c1b8640 10 monclient(hunting): authenticate will time out at 850.380288s 2024-08-19T12:40:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa84a755640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7fa8440a5d50 0x7fa844141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:57838/0 (socket says 172.21.15.26:57838) 2024-08-19T12:40:56.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.293+0000 7fa84a755640 1 -- 172.21.15.26:0/2582689514 learned_addr learned my addr 172.21.15.26:0/2582689514 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:56.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.295+0000 7fa84a755640 1 -- 172.21.15.26:0/2582689514 >> v1:172.21.15.104:6791/0 conn(0x7fa8440a5d50 legacy=0x7fa844141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.295+0000 7fa849f54640 1 -- 172.21.15.26:0/2582689514 >> v1:172.21.15.104:6789/0 conn(0x7fa84406e7b0 legacy=0x7fa844136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.295+0000 7fa848f52640 1 -- 172.21.15.26:0/2582689514 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3058391212 0 0) 0x7fa834001820 con 0x7fa84406e7b0 2024-08-19T12:40:56.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.295+0000 7fa848f52640 10 monclient(hunting): _negotiate 2024-08-19T12:40:56.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.295+0000 7fa848f52640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:56.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.295+0000 7fa848f52640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:56.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.295+0000 7fa848f52640 10 monclient(hunting): my global_id is 26173 2024-08-19T12:40:56.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.295+0000 7fa848f52640 1 -- 172.21.15.26:0/2582689514 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa82c003520 con 0x7fa84406e7b0 2024-08-19T12:40:56.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 1 -- 172.21.15.26:0/2582689514 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 795806639 0 0) 0x7fa82c003520 con 0x7fa84406e7b0 2024-08-19T12:40:56.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 1 -- 172.21.15.26:0/2582689514 >> v1:172.21.15.104:6791/0 conn(0x7fa8440a5d50 legacy=0x7fa844141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 1 -- 172.21.15.26:0/2582689514 >> v1:172.21.15.26:6789/0 conn(0x7fa8440a5920 legacy=0x7fa84413e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 10 monclient: _finish_hunting 0 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 1 monclient: found mon.noname-f 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 1 -- 172.21.15.26:0/2582689514 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa84409e3b0 con 0x7fa84406e7b0 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 10 monclient: _finish_auth 0 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 10 monclient: _check_auth_tickets 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 1 -- 172.21.15.26:0/2582689514 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa834002080 con 0x7fa84406e7b0 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa84c1b8640 5 monclient: authenticate success, global_id 26173 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa84c1b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:56.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 10 monclient: dump: 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa84c1b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 1 -- 172.21.15.26:0/2582689514 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa834003ac0 con 0x7fa84406e7b0 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 1 -- 172.21.15.26:0/2582689514 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa834003ff0 con 0x7fa84406e7b0 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa848f52640 10 monclient: dump: 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.304 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa84c1b8640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa84c1b8640 4 set_mon_vals no callback set 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa84c1b8640 10 monclient: shutdown 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa84c1b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa84c1b8640 1 -- 172.21.15.26:0/2582689514 >> v1:172.21.15.104:6789/0 conn(0x7fa84406e7b0 legacy=0x7fa844136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa84c1b8640 1 -- 172.21.15.26:0/2582689514 shutdown_connections 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.296+0000 7fa84c1b8640 1 -- 172.21.15.26:0/2582689514 >> 172.21.15.26:0/2582689514 conn(0x7fa8440a8740 msgr2=0x7fa8440a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.297+0000 7fa84c1b8640 1 -- 172.21.15.26:0/2582689514 shutdown_connections 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.297+0000 7fa84c1b8640 1 -- 172.21.15.26:0/2582689514 wait complete. 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.297+0000 7fa84c1b8640 10 monclient: build_initial_monmap 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.297+0000 7fa84c1b8640 10 monclient: monmap: 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:56.305 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:56.306 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.297+0000 7fa84c1b8640 1 librados: starting msgr at 2024-08-19T12:40:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.297+0000 7fa84c1b8640 1 librados: starting objecter 2024-08-19T12:40:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.297+0000 7fa84c1b8640 1 Processor -- start 2024-08-19T12:40:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.297+0000 7fa84c1b8640 1 -- start start 2024-08-19T12:40:56.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.297+0000 7fa84c1b8640 1 librados: setting wanted keys 2024-08-19T12:40:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.297+0000 7fa84c1b8640 1 librados: calling monclient init 2024-08-19T12:40:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.297+0000 7fa84c1b8640 10 monclient: init 2024-08-19T12:40:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa84c1b8640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa84c1b8640 10 monclient: _add_conns ranks=[8,0,7,3,5,4,1,2,6] 2024-08-19T12:40:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa84c1b8640 10 monclient(hunting): picked mon.noname-e con 0x7fa84406e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa84c1b8640 10 monclient(hunting): picked mon.noname-a con 0x7fa8440a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa84c1b8640 10 monclient(hunting): picked mon.noname-i con 0x7fa8440a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa84c1b8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8441349d0 con 0x7fa8440a5920 2024-08-19T12:40:56.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa84c1b8640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8440a2e00 con 0x7fa8440a5d50 2024-08-19T12:40:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa84c1b8640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8440a1bc0 con 0x7fa84406e7b0 2024-08-19T12:40:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa84c1b8640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa849753640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa8440a5920 legacy=0x7fa8441d3270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:40:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa84c1b8640 10 monclient(hunting): authenticate will time out at 850.385448s 2024-08-19T12:40:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa849f54640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fa84406e7b0 0x7fa8440a48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:45534/0 (socket says 172.21.15.26:45534) 2024-08-19T12:40:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.298+0000 7fa849f54640 1 -- 172.21.15.26:0/142735785 learned_addr learned my addr 172.21.15.26:0/142735785 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.300+0000 7fa849f54640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.26:6793/0 conn(0x7fa84406e7b0 legacy=0x7fa8440a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.300+0000 7fa8427fc640 10 client.?.objecter ms_handle_connect 0x7fa84406e7b0 2024-08-19T12:40:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.300+0000 7fa8427fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.300+0000 7fa8427fc640 10 client.?.objecter ms_handle_connect 0x7fa8440a5d50 2024-08-19T12:40:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.300+0000 7fa8427fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.300+0000 7fa84a755640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.104:6792/0 conn(0x7fa8440a5d50 legacy=0x7fa8441d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.499+0000 7fa849753640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.26:6789/0 conn(0x7fa8440a5920 legacy=0x7fa8441d3270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:40:56.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1250882399 0 0) 0x7fa8440a2e00 con 0x7fa8440a5d50 2024-08-19T12:40:56.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:56.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:56.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:56.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): my global_id is 134278 2024-08-19T12:40:56.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa830004eb0 con 0x7fa8440a5d50 2024-08-19T12:40:56.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1501168246 0 0) 0x7fa8440a1bc0 con 0x7fa84406e7b0 2024-08-19T12:40:56.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:56.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:56.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:56.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): my global_id is 25343 2024-08-19T12:40:56.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8440a2e00 con 0x7fa84406e7b0 2024-08-19T12:40:56.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3471824814 0 0) 0x7fa830004eb0 con 0x7fa8440a5d50 2024-08-19T12:40:56.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8440a1bc0 con 0x7fa8440a5d50 2024-08-19T12:40:56.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa83c001ca0 con 0x7fa8440a5d50 2024-08-19T12:40:56.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:56.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:40:56.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): dump: 2024-08-19T12:40:56.734 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.734 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.735 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.735 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.735 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.735 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.735 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.735 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.736 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.736 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.736 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.736 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.736 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.736 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.737 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.737 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:56.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3674780106 0 0) 0x7fa8440a2e00 con 0x7fa84406e7b0 2024-08-19T12:40:56.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa830004eb0 con 0x7fa84406e7b0 2024-08-19T12:40:56.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa834003fb0 con 0x7fa84406e7b0 2024-08-19T12:40:56.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:56.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:56.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient(hunting): dump: 2024-08-19T12:40:56.738 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.738 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.739 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.739 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.739 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.739 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.739 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.739 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.740 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.740 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.740 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.740 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.740 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.740 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.741 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.741 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3253366420 0 0) 0x7fa8440a1bc0 con 0x7fa8440a5d50 2024-08-19T12:40:56.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.26:6793/0 conn(0x7fa84406e7b0 legacy=0x7fa8440a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.26:6789/0 conn(0x7fa8440a5920 legacy=0x7fa8441d3270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:40:56.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient: _finish_hunting 0 2024-08-19T12:40:56.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 monclient: found mon.i 2024-08-19T12:40:56.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:56.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:40:56.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8440a2b20 con 0x7fa8440a5d50 2024-08-19T12:40:56.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient: _finish_auth 0 2024-08-19T12:40:56.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 10 monclient: _check_auth_tickets 2024-08-19T12:40:56.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa8427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:56.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa84c1b8640 5 monclient: authenticate success, global_id 134278 2024-08-19T12:40:56.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa84c1b8640 10 monclient: _renew_subs 2024-08-19T12:40:56.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa84c1b8640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:40:56.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.721+0000 7fa84c1b8640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa8440a2810 con 0x7fa8440a5d50 2024-08-19T12:40:56.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa84c1b8640 10 client.134278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:56.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa84c1b8640 10 monclient: _renew_subs 2024-08-19T12:40:56.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa84c1b8640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:40:56.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa84c1b8640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa8440a51e0 con 0x7fa8440a5d50 2024-08-19T12:40:56.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa83c004b10 con 0x7fa8440a5d50 2024-08-19T12:40:56.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa84c1b8640 1 librados: init done 2024-08-19T12:40:56.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa8427fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:56.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa83c005000 con 0x7fa8440a5d50 2024-08-19T12:40:56.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa8427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:56.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa8427fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:56.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa843fff640 4 set_mon_vals no callback set 2024-08-19T12:40:56.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa8427fc640 10 monclient: dump: 2024-08-19T12:40:56.746 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.746 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.746 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.746 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.746 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.747 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.747 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.747 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.747 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.747 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.747 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.748 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.748 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.748 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.748 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.748 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa84c1b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:56.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.722+0000 7fa84c1b8640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa844077810 con 0x7fa844132a20 2024-08-19T12:40:56.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.723+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa83c0051a0 con 0x7fa8440a5d50 2024-08-19T12:40:56.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.723+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(80..80 src has 1..80) ==== 7501+0+0 (unknown 3349506614 0 0) 0x7fa83c04e950 con 0x7fa8440a5d50 2024-08-19T12:40:56.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.723+0000 7fa8427fc640 10 client.134278.objecter ms_dispatch 0x7fa844000b90 osd_map(80..80 src has 1..80) 2024-08-19T12:40:56.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.723+0000 7fa8427fc640 3 client.134278.objecter handle_osd_map got epochs [80,80] > 0 2024-08-19T12:40:56.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.723+0000 7fa8427fc640 3 client.134278.objecter handle_osd_map decoding full epoch 80 2024-08-19T12:40:56.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.723+0000 7fa8427fc640 20 client.134278.objecter dump_active .. 0 homeless 2024-08-19T12:40:56.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.723+0000 7fa849753640 10 monclient: get_auth_request con 0x7fa8300413a0 auth_method 0 2024-08-19T12:40:56.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 10 client.134278.objecter ms_handle_connect 0x7fa844132a20 2024-08-19T12:40:56.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 10 client.134278.objecter resend_mon_ops 2024-08-19T12:40:56.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa849f54640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.26:6790/0 conn(0x7fa844132a20 legacy=0x7fa8441253f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 616097769 0 0) 0x7fa844077810 con 0x7fa844132a20 2024-08-19T12:40:56.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:56.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:56.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:56.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 10 monclient(hunting): my global_id is 25508 2024-08-19T12:40:56.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8440a1bc0 con 0x7fa844132a20 2024-08-19T12:40:56.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1571314697 0 0) 0x7fa8440a1bc0 con 0x7fa844132a20 2024-08-19T12:40:56.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa844077810 con 0x7fa844132a20 2024-08-19T12:40:56.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa834001b20 con 0x7fa844132a20 2024-08-19T12:40:56.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:56.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:56.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 10 monclient: dump: 2024-08-19T12:40:56.753 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.753 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.753 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.753 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.753 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.754 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.754 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.754 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.754 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.754 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.754 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.755 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.755 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.755 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.755 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.755 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 717638676 0 0) 0x7fa844077810 con 0x7fa844132a20 2024-08-19T12:40:56.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa8441345a0 con 0x7fa844132a20 2024-08-19T12:40:56.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.725+0000 7fa8427fc640 10 client.134278.objecter ms_handle_connect 0x7fa8300413a0 2024-08-19T12:40:56.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.726+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa834003ac0 con 0x7fa844132a20 2024-08-19T12:40:56.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.726+0000 7fa8427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:56.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.726+0000 7fa8427fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:56.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.726+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.26:6790/0 conn(0x7fa844132a20 legacy=0x7fa8441253f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.726+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.26:6790/0 conn(0x7fa844132a20 legacy=0x7fa8441253f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:56.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.733+0000 7fa84c1b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:56.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.733+0000 7fa84c1b8640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa844132e00 con 0x7fa84402e1d0 2024-08-19T12:40:56.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa849f54640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.26:6790/0 conn(0x7fa84402e1d0 legacy=0x7fa844125b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 10 client.134278.objecter ms_handle_connect 0x7fa84402e1d0 2024-08-19T12:40:56.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 10 client.134278.objecter resend_mon_ops 2024-08-19T12:40:56.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2224828213 0 0) 0x7fa844132e00 con 0x7fa84402e1d0 2024-08-19T12:40:56.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:56.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:56.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:56.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 10 monclient(hunting): my global_id is 25517 2024-08-19T12:40:56.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa844077810 con 0x7fa84402e1d0 2024-08-19T12:40:56.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3132611280 0 0) 0x7fa844077810 con 0x7fa84402e1d0 2024-08-19T12:40:56.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa844132e00 con 0x7fa84402e1d0 2024-08-19T12:40:56.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa8340060f0 con 0x7fa84402e1d0 2024-08-19T12:40:56.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:56.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:56.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 10 monclient: dump: 2024-08-19T12:40:56.763 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.763 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.763 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.763 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.764 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.764 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.764 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.764 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.764 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.765 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.765 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.765 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.765 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.765 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.765 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3001936071 0 0) 0x7fa844132e00 con 0x7fa84402e1d0 2024-08-19T12:40:56.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.735+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa844068df0 con 0x7fa84402e1d0 2024-08-19T12:40:56.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7700 (unknown 0 0 2991022255) 0x7fa834008470 con 0x7fa84402e1d0 2024-08-19T12:40:56.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa8427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:56.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa8427fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:56.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.26:6790/0 conn(0x7fa84402e1d0 legacy=0x7fa844125b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa8427fc640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.26:6790/0 conn(0x7fa84402e1d0 legacy=0x7fa844125b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:56.767 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:56.767 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 14, 2024-08-19T12:40:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 169791, 2024-08-19T12:40:56.768 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:56.769 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:40:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:56.770 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:56.771 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:56.772 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:56.773 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:56.774 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:56.775 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:56.776 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:56.777 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.778 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.781 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:56.782 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:56.783 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:56.784 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.785 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:56.786 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.787 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.789 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:56.790 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.792 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.793 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:56.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.795 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:56.796 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:56.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 10 librados: watch_flush enter 2024-08-19T12:40:56.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 10 librados: watch_flush exit 2024-08-19T12:40:56.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 20 client.134278.objecter shutdown clearing up homeless session... 2024-08-19T12:40:56.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 10 client.134278.objecter successfully canceled tick 2024-08-19T12:40:56.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.104:6800/899887506 conn(0x7fa8300413a0 legacy=0x7fa830043860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 10 monclient: shutdown 2024-08-19T12:40:56.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:56.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 1 -- 172.21.15.26:0/142735785 >> v1:172.21.15.104:6792/0 conn(0x7fa8440a5d50 legacy=0x7fa8441d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84a755640 1 -- 172.21.15.26:0/142735785 reap_dead start 2024-08-19T12:40:56.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84a755640 1 -- 172.21.15.26:0/142735785 reap_dead start 2024-08-19T12:40:56.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 1 -- 172.21.15.26:0/142735785 shutdown_connections 2024-08-19T12:40:56.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 1 -- 172.21.15.26:0/142735785 >> 172.21.15.26:0/142735785 conn(0x7fa8440a8740 msgr2=0x7fa8440aa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:56.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 1 -- 172.21.15.26:0/142735785 shutdown_connections 2024-08-19T12:40:56.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 1 -- 172.21.15.26:0/142735785 wait complete. 2024-08-19T12:40:56.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.736+0000 7fa84c1b8640 1 librados: shutdown 2024-08-19T12:40:56.799 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:40:56.799 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-08-19T12:40:56.799 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:40:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.900+0000 7fa7ee104640 10 monclient: get_monmap_and_config 2024-08-19T12:40:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.900+0000 7fa7ee104640 10 monclient: build_initial_monmap 2024-08-19T12:40:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.900+0000 7fa7ee104640 10 monclient: monmap: 2024-08-19T12:40:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:56.901140+0000 2024-08-19T12:40:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:56.901140+0000 2024-08-19T12:40:56.901 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:56.902 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:56.902 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 1 Processor -- start 2024-08-19T12:40:56.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 1 -- start start 2024-08-19T12:40:56.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 10 monclient: init 2024-08-19T12:40:56.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:56.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 10 monclient: _add_conns ranks=[4,6,5,3,0,8,2,7,1] 2024-08-19T12:40:56.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e806e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:56.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 10 monclient(hunting): picked mon.noname-d con 0x7fa7e80a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:56.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 10 monclient(hunting): picked mon.noname-h con 0x7fa7e80a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:56.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e8136790 con 0x7fa7e806e7b0 2024-08-19T12:40:56.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e809ddf0 con 0x7fa7e80a5d50 2024-08-19T12:40:56.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e809e0d0 con 0x7fa7e80a5920 2024-08-19T12:40:56.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:56.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7e77fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fa7e806e7b0 0x7fa7e8136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:55348/0 (socket says 172.21.15.26:55348) 2024-08-19T12:40:56.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7ee104640 10 monclient(hunting): authenticate will time out at 850.988759s 2024-08-19T12:40:56.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7e77fe640 1 -- 172.21.15.26:0/949487866 learned_addr learned my addr 172.21.15.26:0/949487866 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:56.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.901+0000 7fa7e6ffd640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fa7e80a5920 0x7fa7e813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:44148/0 (socket says 172.21.15.26:44148) 2024-08-19T12:40:56.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.903+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/949487866 >> v1:172.21.15.26:6792/0 conn(0x7fa7e80a5920 legacy=0x7fa7e813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.903+0000 7fa7e77fe640 1 -- 172.21.15.26:0/949487866 >> v1:172.21.15.26:6791/0 conn(0x7fa7e806e7b0 legacy=0x7fa7e8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 1 -- 172.21.15.26:0/949487866 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 134824431 0 0) 0x7fa7e809e0d0 con 0x7fa7e80a5920 2024-08-19T12:40:56.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:56.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:56.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:56.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient(hunting): my global_id is 25764 2024-08-19T12:40:56.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 1 -- 172.21.15.26:0/949487866 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7d0003520 con 0x7fa7e80a5920 2024-08-19T12:40:56.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 1 -- 172.21.15.26:0/949487866 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1755789099 0 0) 0x7fa7e8136790 con 0x7fa7e806e7b0 2024-08-19T12:40:56.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:56.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient(hunting): my global_id is 25735 2024-08-19T12:40:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 1 -- 172.21.15.26:0/949487866 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e809e0d0 con 0x7fa7e806e7b0 2024-08-19T12:40:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e7fff640 1 -- 172.21.15.26:0/949487866 >> v1:172.21.15.104:6791/0 conn(0x7fa7e80a5d50 legacy=0x7fa7e8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 1 -- 172.21.15.26:0/949487866 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 58339348 0 0) 0x7fa7d0003520 con 0x7fa7e80a5920 2024-08-19T12:40:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 1 -- 172.21.15.26:0/949487866 >> v1:172.21.15.104:6791/0 conn(0x7fa7e80a5d50 legacy=0x7fa7e8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 1 -- 172.21.15.26:0/949487866 >> v1:172.21.15.26:6791/0 conn(0x7fa7e806e7b0 legacy=0x7fa7e8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient: _finish_hunting 0 2024-08-19T12:40:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 1 monclient: found mon.noname-d 2024-08-19T12:40:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:40:56.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 1 -- 172.21.15.26:0/949487866 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e809e3b0 con 0x7fa7e80a5920 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient: _finish_auth 0 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient: _check_auth_tickets 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 1 -- 172.21.15.26:0/949487866 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7dc001ae0 con 0x7fa7e80a5920 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient: dump: 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.915 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7ee104640 5 monclient: authenticate success, global_id 25764 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7ee104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:56.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 1 -- 172.21.15.26:0/949487866 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7dc0047e0 con 0x7fa7e80a5920 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 1 -- 172.21.15.26:0/949487866 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7dc004d50 con 0x7fa7e80a5920 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7e67fc640 10 monclient: dump: 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.917 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7ee104640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7ee104640 4 set_mon_vals no callback set 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7ee104640 10 monclient: shutdown 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7ee104640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.904+0000 7fa7ee104640 1 -- 172.21.15.26:0/949487866 >> v1:172.21.15.26:6792/0 conn(0x7fa7e80a5920 legacy=0x7fa7e813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.905+0000 7fa7ee104640 1 -- 172.21.15.26:0/949487866 shutdown_connections 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.905+0000 7fa7ee104640 1 -- 172.21.15.26:0/949487866 >> 172.21.15.26:0/949487866 conn(0x7fa7e80a8740 msgr2=0x7fa7e80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.905+0000 7fa7ee104640 1 -- 172.21.15.26:0/949487866 shutdown_connections 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.905+0000 7fa7ee104640 1 -- 172.21.15.26:0/949487866 wait complete. 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.905+0000 7fa7ee104640 10 monclient: build_initial_monmap 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.905+0000 7fa7ee104640 10 monclient: monmap: 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:56.919 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.905+0000 7fa7ee104640 1 librados: starting msgr at 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.905+0000 7fa7ee104640 1 librados: starting objecter 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.905+0000 7fa7ee104640 1 Processor -- start 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 1 -- start start 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 1 librados: setting wanted keys 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 1 librados: calling monclient init 2024-08-19T12:40:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 10 monclient: init 2024-08-19T12:40:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 10 monclient: _add_conns ranks=[4,2,1,3,6,5,8,7,0] 2024-08-19T12:40:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e806e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 10 monclient(hunting): picked mon.noname-b con 0x7fa7e80a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 10 monclient(hunting): picked mon.noname-f con 0x7fa7e80a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e813b450 con 0x7fa7e80a5d50 2024-08-19T12:40:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e81390f0 con 0x7fa7e80a5920 2024-08-19T12:40:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e8139c50 con 0x7fa7e806e7b0 2024-08-19T12:40:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7ee104640 10 monclient(hunting): authenticate will time out at 850.993664s 2024-08-19T12:40:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7e77fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fa7e806e7b0 0x7fa7e81ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:55362/0 (socket says 172.21.15.26:55362) 2024-08-19T12:40:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.906+0000 7fa7e77fe640 1 -- 172.21.15.26:0/2381036748 learned_addr learned my addr 172.21.15.26:0/2381036748 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.908+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.26:6790/0 conn(0x7fa7e80a5920 legacy=0x7fa7e81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.908+0000 7fa7cffff640 10 client.?.objecter ms_handle_connect 0x7fa7e80a5920 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.908+0000 7fa7cffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.908+0000 7fa7e77fe640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.26:6791/0 conn(0x7fa7e806e7b0 legacy=0x7fa7e81ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.908+0000 7fa7cffff640 10 client.?.objecter ms_handle_connect 0x7fa7e806e7b0 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.908+0000 7fa7cffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7e7fff640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.104:6789/0 conn(0x7fa7e80a5d50 legacy=0x7fa7e81d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 client.?.objecter ms_handle_connect 0x7fa7e80a5d50 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 411214265 0 0) 0x7fa7e81390f0 con 0x7fa7e80a5920 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:56.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): my global_id is 25526 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7d4004eb0 con 0x7fa7e80a5920 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 51997848 0 0) 0x7fa7e8139c50 con 0x7fa7e806e7b0 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): my global_id is 25744 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e81390f0 con 0x7fa7e806e7b0 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1200297638 0 0) 0x7fa7e813b450 con 0x7fa7e80a5d50 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:56.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): my global_id is 26182 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e8139c50 con 0x7fa7e80a5d50 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2965629252 0 0) 0x7fa7e81390f0 con 0x7fa7e806e7b0 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7e813b450 con 0x7fa7e806e7b0 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7d8004b60 con 0x7fa7e806e7b0 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): dump: 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.924 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3080051262 0 0) 0x7fa7d4004eb0 con 0x7fa7e80a5920 2024-08-19T12:40:56.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7e81390f0 con 0x7fa7e80a5920 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7dc004cd0 con 0x7fa7e80a5920 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): dump: 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.926 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3152135652 0 0) 0x7fa7e8139c50 con 0x7fa7e80a5d50 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7d4004eb0 con 0x7fa7e80a5d50 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7e0001c30 con 0x7fa7e80a5d50 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient(hunting): dump: 2024-08-19T12:40:56.927 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.928 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1576986521 0 0) 0x7fa7e813b450 con 0x7fa7e806e7b0 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.26:6790/0 conn(0x7fa7e80a5920 legacy=0x7fa7e81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.104:6789/0 conn(0x7fa7e80a5d50 legacy=0x7fa7e81d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient: _finish_hunting 0 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 monclient: found mon.c 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e813abb0 con 0x7fa7e806e7b0 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient: _finish_auth 0 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient: _check_auth_tickets 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:56.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7ee104640 5 monclient: authenticate success, global_id 25744 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7ee104640 10 monclient: _renew_subs 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7ee104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7ee104640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e813ae90 con 0x7fa7e806e7b0 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7ee104640 10 client.25744.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7ee104640 10 monclient: _renew_subs 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7ee104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7ee104640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fa7e8138690 con 0x7fa7e806e7b0 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7ee104640 1 librados: init done 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7d8004fb0 con 0x7fa7e806e7b0 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7d80054e0 con 0x7fa7e806e7b0 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:56.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7e5ffb640 4 set_mon_vals no callback set 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.909+0000 7fa7cffff640 10 monclient: dump: 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.931 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.910+0000 7fa7ee104640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.910+0000 7fa7ee104640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa7c80065c0 con 0x7fa7c8002860 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.910+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa7d8005680 con 0x7fa7e806e7b0 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.910+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(80..80 src has 1..80) ==== 7501+0+0 (unknown 3349506614 0 0) 0x7fa7d804ed60 con 0x7fa7e806e7b0 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.910+0000 7fa7cffff640 10 client.25744.objecter ms_dispatch 0x7fa7e8000b90 osd_map(80..80 src has 1..80) 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.910+0000 7fa7cffff640 3 client.25744.objecter handle_osd_map got epochs [80,80] > 0 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.910+0000 7fa7cffff640 3 client.25744.objecter handle_osd_map decoding full epoch 80 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.911+0000 7fa7cffff640 20 client.25744.objecter dump_active .. 0 homeless 2024-08-19T12:40:56.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.911+0000 7fa7e7fff640 10 monclient: get_auth_request con 0x7fa7d40419c0 auth_method 0 2024-08-19T12:40:56.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.912+0000 7fa7cffff640 10 client.25744.objecter ms_handle_connect 0x7fa7c8002860 2024-08-19T12:40:56.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.912+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.26:6791/0 conn(0x7fa7c8002860 legacy=0x7fa7c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.912+0000 7fa7cffff640 10 client.25744.objecter resend_mon_ops 2024-08-19T12:40:56.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.912+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 353532003 0 0) 0x7fa7c80065c0 con 0x7fa7c8002860 2024-08-19T12:40:56.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.912+0000 7fa7cffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:56.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.912+0000 7fa7cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:56.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.912+0000 7fa7cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:56.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.912+0000 7fa7cffff640 10 monclient(hunting): my global_id is 25753 2024-08-19T12:40:56.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.912+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e813b450 con 0x7fa7c8002860 2024-08-19T12:40:56.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 625778722 0 0) 0x7fa7e813b450 con 0x7fa7c8002860 2024-08-19T12:40:56.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7c80065c0 con 0x7fa7c8002860 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7dc001d30 con 0x7fa7c8002860 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 10 monclient: dump: 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.937 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2453671808 0 0) 0x7fa7c80065c0 con 0x7fa7c8002860 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7c800a160 con 0x7fa7c8002860 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 10 client.25744.objecter ms_handle_connect 0x7fa7d40419c0 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa7dc0053d0 con 0x7fa7c8002860 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:56.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.26:6791/0 conn(0x7fa7c8002860 legacy=0x7fa7c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.913+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.26:6791/0 conn(0x7fa7c8002860 legacy=0x7fa7c8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.919+0000 7fa7ee104640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.919+0000 7fa7ee104640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa7c8009e70 con 0x7fa7c8009130 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.26:6791/0 conn(0x7fa7c8009130 legacy=0x7fa7c80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 10 client.25744.objecter ms_handle_connect 0x7fa7c8009130 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 10 client.25744.objecter resend_mon_ops 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2043168878 0 0) 0x7fa7c8009e70 con 0x7fa7c8009130 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 10 monclient(hunting): my global_id is 25762 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7c80065c0 con 0x7fa7c8009130 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 206702093 0 0) 0x7fa7c80065c0 con 0x7fa7c8009130 2024-08-19T12:40:56.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7c8009e70 con 0x7fa7c8009130 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7dc005b30 con 0x7fa7c8009130 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 10 monclient: dump: 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1303535504 0 0) 0x7fa7c8009e70 con 0x7fa7c8009130 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7c800c3f0 con 0x7fa7c8009130 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7867 (unknown 0 0 2804730285) 0x7fa7dc008880 con 0x7fa7c8009130 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.26:6791/0 conn(0x7fa7c8009130 legacy=0x7fa7c80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.922+0000 7fa7cffff640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.26:6791/0 conn(0x7fa7c8009130 legacy=0x7fa7c80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:56.942 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 14, 2024-08-19T12:40:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 169952, 2024-08-19T12:40:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:40:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:56.962 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:56.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 10 librados: watch_flush enter 2024-08-19T12:40:56.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 10 librados: watch_flush exit 2024-08-19T12:40:56.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 20 client.25744.objecter shutdown clearing up homeless session... 2024-08-19T12:40:56.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 10 client.25744.objecter successfully canceled tick 2024-08-19T12:40:56.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.104:6800/899887506 conn(0x7fa7d40419c0 legacy=0x7fa7d4043e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 10 monclient: shutdown 2024-08-19T12:40:56.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:56.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 1 -- 172.21.15.26:0/2381036748 >> v1:172.21.15.26:6791/0 conn(0x7fa7e806e7b0 legacy=0x7fa7e81ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:56.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7e7fff640 1 -- 172.21.15.26:0/2381036748 reap_dead start 2024-08-19T12:40:56.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7e7fff640 1 -- 172.21.15.26:0/2381036748 reap_dead start 2024-08-19T12:40:56.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 1 -- 172.21.15.26:0/2381036748 shutdown_connections 2024-08-19T12:40:56.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 1 -- 172.21.15.26:0/2381036748 >> 172.21.15.26:0/2381036748 conn(0x7fa7e80a8740 msgr2=0x7fa7e80ab120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:56.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 1 -- 172.21.15.26:0/2381036748 shutdown_connections 2024-08-19T12:40:56.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 1 -- 172.21.15.26:0/2381036748 wait complete. 2024-08-19T12:40:56.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:56.923+0000 7fa7ee104640 1 librados: shutdown 2024-08-19T12:40:56.965 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:40:56.965 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon d 2024-08-19T12:40:56.966 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:40:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.079+0000 7f2a2a84f640 10 monclient: get_monmap_and_config 2024-08-19T12:40:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.079+0000 7f2a2a84f640 10 monclient: build_initial_monmap 2024-08-19T12:40:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.079+0000 7f2a2a84f640 10 monclient: monmap: 2024-08-19T12:40:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:57.079907+0000 2024-08-19T12:40:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:57.079907+0000 2024-08-19T12:40:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:57.080 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:57.081 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:57.081 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:57.081 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:57.081 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:57.081 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:57.081 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:57.082 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:57.082 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.079+0000 7f2a2a84f640 1 Processor -- start 2024-08-19T12:40:57.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.079+0000 7f2a2a84f640 1 -- start start 2024-08-19T12:40:57.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.079+0000 7f2a2a84f640 10 monclient: init 2024-08-19T12:40:57.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a2a84f640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a2a84f640 10 monclient: _add_conns ranks=[2,0,6,7,1,4,3,5,8] 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a2a84f640 10 monclient(hunting): picked mon.noname-b con 0x7f2a24073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a2a84f640 10 monclient(hunting): picked mon.noname-a con 0x7f2a240a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a2a84f640 10 monclient(hunting): picked mon.noname-d con 0x7f2a2413f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a2a84f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a2409dae0 con 0x7f2a240a57a0 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a2a84f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a24136910 con 0x7f2a24073470 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a2a84f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a2409dd50 con 0x7f2a2413f2e0 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a2a84f640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a2a84f640 10 monclient(hunting): authenticate will time out at 851.167412s 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a237fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2a240a57a0 legacy=0x7f2a240a5b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a23fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2a24073470 0x7f2a24136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33386/0 (socket says 172.21.15.26:33386) 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a28dec640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f2a2413f2e0 0x7f2a241416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:44156/0 (socket says 172.21.15.26:44156) 2024-08-19T12:40:57.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.080+0000 7f2a28dec640 1 -- 172.21.15.26:0/861532596 learned_addr learned my addr 172.21.15.26:0/861532596 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:57.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.082+0000 7f2a23fff640 1 -- 172.21.15.26:0/861532596 >> v1:172.21.15.26:6790/0 conn(0x7f2a24073470 legacy=0x7f2a24136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.082+0000 7f2a28dec640 1 -- 172.21.15.26:0/861532596 >> v1:172.21.15.26:6792/0 conn(0x7f2a2413f2e0 legacy=0x7f2a241416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 1 -- 172.21.15.26:0/861532596 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2099882404 0 0) 0x7f2a2409dd50 con 0x7f2a2413f2e0 2024-08-19T12:40:57.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient(hunting): my global_id is 25791 2024-08-19T12:40:57.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 1 -- 172.21.15.26:0/861532596 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a0c003520 con 0x7f2a2413f2e0 2024-08-19T12:40:57.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 1 -- 172.21.15.26:0/861532596 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 989820929 0 0) 0x7f2a24136910 con 0x7f2a24073470 2024-08-19T12:40:57.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient(hunting): my global_id is 25544 2024-08-19T12:40:57.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 1 -- 172.21.15.26:0/861532596 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a2409dd50 con 0x7f2a24073470 2024-08-19T12:40:57.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 1 -- 172.21.15.26:0/861532596 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1198114153 0 0) 0x7f2a0c003520 con 0x7f2a2413f2e0 2024-08-19T12:40:57.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 1 -- 172.21.15.26:0/861532596 >> v1:172.21.15.26:6790/0 conn(0x7f2a24073470 legacy=0x7f2a24136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 1 -- 172.21.15.26:0/861532596 >> v1:172.21.15.26:6789/0 conn(0x7f2a240a57a0 legacy=0x7f2a240a5b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:40:57.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:40:57.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 1 monclient: found mon.noname-d 2024-08-19T12:40:57.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:57.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:40:57.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 1 -- 172.21.15.26:0/861532596 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2409e030 con 0x7f2a2413f2e0 2024-08-19T12:40:57.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient: _finish_auth 0 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 1 -- 172.21.15.26:0/861532596 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2a1c001e60 con 0x7f2a2413f2e0 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.119+0000 7f2a22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.120+0000 7f2a22ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.120+0000 7f2a22ffd640 10 monclient: dump: 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.124 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.120+0000 7f2a2a84f640 5 monclient: authenticate success, global_id 25791 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.120+0000 7f2a2a84f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.120+0000 7f2a22ffd640 1 -- 172.21.15.26:0/861532596 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a1c004c20 con 0x7f2a2413f2e0 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.120+0000 7f2a22ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:57.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.120+0000 7f2a2a84f640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.120+0000 7f2a2a84f640 4 set_mon_vals no callback set 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.120+0000 7f2a22ffd640 1 -- 172.21.15.26:0/861532596 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2a1c0051d0 con 0x7f2a2413f2e0 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.120+0000 7f2a22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.120+0000 7f2a22ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.120+0000 7f2a22ffd640 10 monclient: dump: 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.126 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.121+0000 7f2a2a84f640 10 monclient: shutdown 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.121+0000 7f2a2a84f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.121+0000 7f2a2a84f640 1 -- 172.21.15.26:0/861532596 >> v1:172.21.15.26:6792/0 conn(0x7f2a2413f2e0 legacy=0x7f2a241416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.121+0000 7f2a2a84f640 1 -- 172.21.15.26:0/861532596 shutdown_connections 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.121+0000 7f2a2a84f640 1 -- 172.21.15.26:0/861532596 >> 172.21.15.26:0/861532596 conn(0x7f2a240abff0 msgr2=0x7f2a240ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.121+0000 7f2a2a84f640 1 -- 172.21.15.26:0/861532596 shutdown_connections 2024-08-19T12:40:57.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.121+0000 7f2a2a84f640 1 -- 172.21.15.26:0/861532596 wait complete. 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.121+0000 7f2a2a84f640 10 monclient: build_initial_monmap 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.122+0000 7f2a2a84f640 10 monclient: monmap: 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:57.129 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:57.129 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:57.129 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:57.129 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.122+0000 7f2a2a84f640 1 librados: starting msgr at 2024-08-19T12:40:57.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.122+0000 7f2a2a84f640 1 librados: starting objecter 2024-08-19T12:40:57.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.122+0000 7f2a2a84f640 1 Processor -- start 2024-08-19T12:40:57.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.122+0000 7f2a2a84f640 1 -- start start 2024-08-19T12:40:57.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.122+0000 7f2a2a84f640 1 librados: setting wanted keys 2024-08-19T12:40:57.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.122+0000 7f2a2a84f640 1 librados: calling monclient init 2024-08-19T12:40:57.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.122+0000 7f2a2a84f640 10 monclient: init 2024-08-19T12:40:57.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.123+0000 7f2a2a84f640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:57.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.123+0000 7f2a2a84f640 10 monclient: _add_conns ranks=[7,3,8,5,1,6,2,0,4] 2024-08-19T12:40:57.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.123+0000 7f2a2a84f640 10 monclient(hunting): picked mon.noname-i con 0x7f2a24073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:57.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.123+0000 7f2a2a84f640 10 monclient(hunting): picked mon.noname-g con 0x7f2a240a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.123+0000 7f2a2a84f640 10 monclient(hunting): picked mon.noname-e con 0x7f2a2413f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.123+0000 7f2a2a84f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a24135200 con 0x7f2a240a57a0 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.123+0000 7f2a2a84f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a2412eb40 con 0x7f2a24073470 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.123+0000 7f2a2a84f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a24131a00 con 0x7f2a2413f2e0 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.123+0000 7f2a2a84f640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.123+0000 7f2a2a84f640 10 monclient(hunting): authenticate will time out at 851.210255s 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.123+0000 7f2a28dec640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f2a2413f2e0 0x7f2a241d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:45536/0 (socket says 172.21.15.26:45536) 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.123+0000 7f2a28dec640 1 -- 172.21.15.26:0/2827268176 learned_addr learned my addr 172.21.15.26:0/2827268176 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a28dec640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.26:6793/0 conn(0x7f2a2413f2e0 legacy=0x7f2a241d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 client.?.objecter ms_handle_connect 0x7f2a2413f2e0 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 client.?.objecter ms_handle_connect 0x7f2a24073470 2024-08-19T12:40:57.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 client.?.objecter ms_handle_connect 0x7f2a240a57a0 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a23fff640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.104:6792/0 conn(0x7f2a24073470 legacy=0x7f2a2412dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a237fe640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.104:6790/0 conn(0x7f2a240a57a0 legacy=0x7f2a2412e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 101692814 0 0) 0x7f2a24131a00 con 0x7f2a2413f2e0 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 monclient(hunting): my global_id is 25361 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a10004ee0 con 0x7f2a2413f2e0 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3482107591 0 0) 0x7f2a2412eb40 con 0x7f2a24073470 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 monclient(hunting): my global_id is 134287 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a24131a00 con 0x7f2a24073470 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588093981 0 0) 0x7f2a18003d30 con 0x7f2a240a57a0 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 10 monclient(hunting): my global_id is 25653 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a2412eb40 con 0x7f2a240a57a0 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.125+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3235082870 0 0) 0x7f2a10004ee0 con 0x7f2a2413f2e0 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a18003d30 con 0x7f2a2413f2e0 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2a1c0020b0 con 0x7f2a2413f2e0 2024-08-19T12:40:57.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient(hunting): dump: 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.139 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2953836546 0 0) 0x7f2a2412eb40 con 0x7f2a240a57a0 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a10004ee0 con 0x7f2a240a57a0 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2a18004620 con 0x7f2a240a57a0 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient(hunting): dump: 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.140 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.141 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 979142692 0 0) 0x7f2a18003d30 con 0x7f2a2413f2e0 2024-08-19T12:40:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.104:6792/0 conn(0x7f2a24073470 legacy=0x7f2a2412dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.104:6790/0 conn(0x7f2a240a57a0 legacy=0x7f2a2412e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:40:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 1 monclient: found mon.e 2024-08-19T12:40:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2412ff80 con 0x7f2a2413f2e0 2024-08-19T12:40:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient: _finish_auth 0 2024-08-19T12:40:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:40:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a2a84f640 5 monclient: authenticate success, global_id 25361 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a2a84f640 10 monclient: _renew_subs 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a2a84f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a2a84f640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2a241303f0 con 0x7f2a2413f2e0 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a2a84f640 10 client.25361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a2a84f640 10 monclient: _renew_subs 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a2a84f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a2a84f640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f2a241320e0 con 0x7f2a2413f2e0 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a2a84f640 1 librados: init done 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a1c0053c0 con 0x7f2a2413f2e0 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2a1c0058b0 con 0x7f2a2413f2e0 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a227fc640 4 set_mon_vals no callback set 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.126+0000 7f2a20ff9640 10 monclient: dump: 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.144 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.127+0000 7f2a2a84f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.127+0000 7f2a2a84f640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2a040065c0 con 0x7f2a04002860 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.127+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2a1c011d10 con 0x7f2a2413f2e0 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.127+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(81..81 src has 1..81) ==== 6713+0+0 (unknown 3862053268 0 0) 0x7f2a1c04e950 con 0x7f2a2413f2e0 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.127+0000 7f2a20ff9640 10 client.25361.objecter ms_dispatch 0x7f2a24000b90 osd_map(81..81 src has 1..81) 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.127+0000 7f2a20ff9640 3 client.25361.objecter handle_osd_map got epochs [81,81] > 0 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.127+0000 7f2a20ff9640 3 client.25361.objecter handle_osd_map decoding full epoch 81 2024-08-19T12:40:57.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.127+0000 7f2a20ff9640 20 client.25361.objecter dump_active .. 0 homeless 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.127+0000 7f2a237fe640 10 monclient: get_auth_request con 0x7f2a10041950 auth_method 0 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a23fff640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.26:6792/0 conn(0x7f2a04002860 legacy=0x7f2a04004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 10 client.25361.objecter ms_handle_connect 0x7f2a04002860 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 10 client.25361.objecter resend_mon_ops 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3539663133 0 0) 0x7f2a040065c0 con 0x7f2a04002860 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 10 monclient(hunting): my global_id is 25800 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a18003d30 con 0x7f2a04002860 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3786414436 0 0) 0x7f2a18003d30 con 0x7f2a04002860 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a040065c0 con 0x7f2a04002860 2024-08-19T12:40:57.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2a14001c10 con 0x7f2a04002860 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 10 monclient: dump: 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.147 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4082038056 0 0) 0x7f2a040065c0 con 0x7f2a04002860 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.129+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a0400a160 con 0x7f2a04002860 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.130+0000 7f2a20ff9640 10 client.25361.objecter ms_handle_connect 0x7f2a10041950 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.130+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2a14006860 con 0x7f2a04002860 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.130+0000 7f2a20ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.130+0000 7f2a20ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.130+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.26:6792/0 conn(0x7f2a04002860 legacy=0x7f2a04004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.130+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.26:6792/0 conn(0x7f2a04002860 legacy=0x7f2a04004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:57.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.136+0000 7f2a2a84f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:57.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.136+0000 7f2a2a84f640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2a04009e70 con 0x7f2a04009130 2024-08-19T12:40:57.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 10 client.25361.objecter ms_handle_connect 0x7f2a04009130 2024-08-19T12:40:57.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 10 client.25361.objecter resend_mon_ops 2024-08-19T12:40:57.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a23fff640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.26:6792/0 conn(0x7f2a04009130 legacy=0x7f2a040095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1590792979 0 0) 0x7f2a04009e70 con 0x7f2a04009130 2024-08-19T12:40:57.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 10 monclient(hunting): my global_id is 25809 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a040065c0 con 0x7f2a04009130 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2471499434 0 0) 0x7f2a040065c0 con 0x7f2a04009130 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a04009e70 con 0x7f2a04009130 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2a14004f10 con 0x7f2a04009130 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 10 monclient: dump: 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.151 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3672567773 0 0) 0x7f2a04009e70 con 0x7f2a04009130 2024-08-19T12:40:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2a0400c3f0 con 0x7f2a04009130 2024-08-19T12:40:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7700 (unknown 0 0 4211595154) 0x7f2a14006b10 con 0x7f2a04009130 2024-08-19T12:40:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.26:6792/0 conn(0x7f2a04009130 legacy=0x7f2a040095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a20ff9640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.26:6792/0 conn(0x7f2a04009130 legacy=0x7f2a040095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:57.153 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:57.153 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 14, 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 170180, 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.154 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.155 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.156 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:57.157 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.158 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.159 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.161 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:57.163 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.164 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:57.167 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:57.171 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.172 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.173 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:57.173 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:57.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.139+0000 7f2a2a84f640 10 librados: watch_flush enter 2024-08-19T12:40:57.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a2a84f640 10 librados: watch_flush exit 2024-08-19T12:40:57.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a2a84f640 20 client.25361.objecter shutdown clearing up homeless session... 2024-08-19T12:40:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a2a84f640 10 client.25361.objecter successfully canceled tick 2024-08-19T12:40:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a2a84f640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.104:6800/899887506 conn(0x7f2a10041950 legacy=0x7f2a10043e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a2a84f640 10 monclient: shutdown 2024-08-19T12:40:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a2a84f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a2a84f640 1 -- 172.21.15.26:0/2827268176 >> v1:172.21.15.26:6793/0 conn(0x7f2a2413f2e0 legacy=0x7f2a241d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a28dec640 1 -- 172.21.15.26:0/2827268176 reap_dead start 2024-08-19T12:40:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a28dec640 1 -- 172.21.15.26:0/2827268176 reap_dead start 2024-08-19T12:40:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a2a84f640 1 -- 172.21.15.26:0/2827268176 shutdown_connections 2024-08-19T12:40:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a2a84f640 1 -- 172.21.15.26:0/2827268176 >> 172.21.15.26:0/2827268176 conn(0x7f2a240abff0 msgr2=0x7f2a2412c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a2a84f640 1 -- 172.21.15.26:0/2827268176 shutdown_connections 2024-08-19T12:40:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a2a84f640 1 -- 172.21.15.26:0/2827268176 wait complete. 2024-08-19T12:40:57.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.140+0000 7f2a2a84f640 1 librados: shutdown 2024-08-19T12:40:57.177 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:40:57.177 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-08-19T12:40:57.177 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:40:57.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.275+0000 7f3c49c32640 10 monclient: get_monmap_and_config 2024-08-19T12:40:57.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.275+0000 7f3c49c32640 10 monclient: build_initial_monmap 2024-08-19T12:40:57.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.275+0000 7f3c49c32640 10 monclient: monmap: 2024-08-19T12:40:57.277 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:57.277 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.277 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:57.276770+0000 2024-08-19T12:40:57.277 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:57.276770+0000 2024-08-19T12:40:57.277 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:57.277 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.277 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:57.277 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:57.277 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:57.277 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:57.278 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:57.278 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:57.278 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:57.278 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:57.278 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:57.278 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.276+0000 7f3c49c32640 1 Processor -- start 2024-08-19T12:40:57.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.276+0000 7f3c49c32640 1 -- start start 2024-08-19T12:40:57.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.276+0000 7f3c49c32640 10 monclient: init 2024-08-19T12:40:57.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.277+0000 7f3c49c32640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:57.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.277+0000 7f3c49c32640 10 monclient: _add_conns ranks=[3,2,5,7,0,4,8,1,6] 2024-08-19T12:40:57.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.277+0000 7f3c49c32640 10 monclient(hunting): picked mon.noname-g con 0x7f3c4406e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:57.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.277+0000 7f3c49c32640 10 monclient(hunting): picked mon.noname-b con 0x7f3c440a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:57.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.277+0000 7f3c49c32640 10 monclient(hunting): picked mon.noname-h con 0x7f3c440a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:57.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.277+0000 7f3c49c32640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c44136790 con 0x7f3c440a5920 2024-08-19T12:40:57.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.277+0000 7f3c49c32640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c4409ddf0 con 0x7f3c4406e7b0 2024-08-19T12:40:57.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.277+0000 7f3c49c32640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c4409e0d0 con 0x7f3c440a5d50 2024-08-19T12:40:57.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.277+0000 7f3c49c32640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:57.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.277+0000 7f3c49c32640 10 monclient(hunting): authenticate will time out at 851.364228s 2024-08-19T12:40:57.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.277+0000 7f3c42ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3c440a5920 0x7f3c4413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33398/0 (socket says 172.21.15.26:33398) 2024-08-19T12:40:57.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.277+0000 7f3c42ffd640 1 -- 172.21.15.26:0/908383275 learned_addr learned my addr 172.21.15.26:0/908383275 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c42ffd640 1 -- 172.21.15.26:0/908383275 >> v1:172.21.15.26:6790/0 conn(0x7f3c440a5920 legacy=0x7f3c4413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c43fff640 1 -- 172.21.15.26:0/908383275 >> v1:172.21.15.104:6791/0 conn(0x7f3c440a5d50 legacy=0x7f3c44141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2104565631 0 0) 0x7f3c44136790 con 0x7f3c440a5920 2024-08-19T12:40:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c437fe640 1 -- 172.21.15.26:0/908383275 >> v1:172.21.15.104:6790/0 conn(0x7f3c4406e7b0 legacy=0x7f3c44136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient(hunting): my global_id is 25571 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c2c003520 con 0x7f3c440a5920 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1489194165 0 0) 0x7f3c4409e0d0 con 0x7f3c440a5d50 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient(hunting): my global_id is 25538 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c44136790 con 0x7f3c440a5d50 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2824810970 0 0) 0x7f3c340013d0 con 0x7f3c4406e7b0 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient(hunting): my global_id is 25671 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c4409e0d0 con 0x7f3c4406e7b0 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 207737459 0 0) 0x7f3c2c003520 con 0x7f3c440a5920 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 >> v1:172.21.15.104:6791/0 conn(0x7f3c440a5d50 legacy=0x7f3c44141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 >> v1:172.21.15.104:6790/0 conn(0x7f3c4406e7b0 legacy=0x7f3c44136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient: _finish_hunting 0 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 1 monclient: found mon.noname-b 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4409e3b0 con 0x7f3c440a5920 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient: _finish_auth 0 2024-08-19T12:40:57.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient: _check_auth_tickets 2024-08-19T12:40:57.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:57.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3c38001ae0 con 0x7f3c440a5920 2024-08-19T12:40:57.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c49c32640 5 monclient: authenticate success, global_id 25571 2024-08-19T12:40:57.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c49c32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:57.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.279+0000 7f3c427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:57.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c427fc640 10 monclient: dump: 2024-08-19T12:40:57.293 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.293 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.294 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c49c32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3c380048a0 con 0x7f3c440a5920 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c427fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c427fc640 1 -- 172.21.15.26:0/908383275 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3c38004e10 con 0x7f3c440a5920 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c427fc640 10 monclient: dump: 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.295 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c49c32640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:57.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c49c32640 4 set_mon_vals no callback set 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c49c32640 10 monclient: shutdown 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c49c32640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c49c32640 1 -- 172.21.15.26:0/908383275 >> v1:172.21.15.26:6790/0 conn(0x7f3c440a5920 legacy=0x7f3c4413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c49c32640 1 -- 172.21.15.26:0/908383275 shutdown_connections 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c49c32640 1 -- 172.21.15.26:0/908383275 >> 172.21.15.26:0/908383275 conn(0x7f3c440a8740 msgr2=0x7f3c440a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c49c32640 1 -- 172.21.15.26:0/908383275 shutdown_connections 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.280+0000 7f3c49c32640 1 -- 172.21.15.26:0/908383275 wait complete. 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.281+0000 7f3c49c32640 10 monclient: build_initial_monmap 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.281+0000 7f3c49c32640 10 monclient: monmap: 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:57.297 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.281+0000 7f3c49c32640 1 librados: starting msgr at 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.281+0000 7f3c49c32640 1 librados: starting objecter 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.281+0000 7f3c49c32640 1 Processor -- start 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.281+0000 7f3c49c32640 1 -- start start 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.281+0000 7f3c49c32640 1 librados: setting wanted keys 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.281+0000 7f3c49c32640 1 librados: calling monclient init 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.281+0000 7f3c49c32640 10 monclient: init 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c49c32640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c49c32640 10 monclient: _add_conns ranks=[0,6,8,7,5,2,4,1,3] 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c49c32640 10 monclient(hunting): picked mon.noname-a con 0x7f3c4406e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c49c32640 10 monclient(hunting): picked mon.noname-d con 0x7f3c440a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c49c32640 10 monclient(hunting): picked mon.noname-e con 0x7f3c440a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c49c32640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c44134b20 con 0x7f3c4406e7b0 2024-08-19T12:40:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c49c32640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c440a3330 con 0x7f3c440a5920 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c437fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f3c4406e7b0 legacy=0x7f3c440a4a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c49c32640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c440a2a20 con 0x7f3c440a5d50 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c49c32640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c49c32640 10 monclient(hunting): authenticate will time out at 851.369144s 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c42ffd640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f3c440a5920 0x7f3c441d3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:44188/0 (socket says 172.21.15.26:44188) 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.282+0000 7f3c42ffd640 1 -- 172.21.15.26:0/2920760918 learned_addr learned my addr 172.21.15.26:0/2920760918 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c43fff640 1 -- 172.21.15.26:0/2920760918 >> v1:172.21.15.26:6793/0 conn(0x7f3c440a5d50 legacy=0x7f3c441d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 client.?.objecter ms_handle_connect 0x7f3c440a5d50 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 client.?.objecter ms_handle_connect 0x7f3c440a5920 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c42ffd640 1 -- 172.21.15.26:0/2920760918 >> v1:172.21.15.26:6792/0 conn(0x7f3c440a5920 legacy=0x7f3c441d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2692717194 0 0) 0x7f3c440a3330 con 0x7f3c440a5920 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): my global_id is 25818 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c30004eb0 con 0x7f3c440a5920 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 844998401 0 0) 0x7f3c440a2a20 con 0x7f3c440a5d50 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): my global_id is 25379 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c440a3330 con 0x7f3c440a5d50 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2616688795 0 0) 0x7f3c30004eb0 con 0x7f3c440a5920 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c440a2a20 con 0x7f3c440a5920 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3c38004d90 con 0x7f3c440a5920 2024-08-19T12:40:57.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:57.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:40:57.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): dump: 2024-08-19T12:40:57.302 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.302 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.302 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.302 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.302 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.302 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.302 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.302 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.302 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.303 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.303 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.303 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.303 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.303 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.303 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.303 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:57.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3175422695 0 0) 0x7f3c440a3330 con 0x7f3c440a5d50 2024-08-19T12:40:57.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c30004eb0 con 0x7f3c440a5d50 2024-08-19T12:40:57.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3c3c0020b0 con 0x7f3c440a5d50 2024-08-19T12:40:57.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:57.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:57.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.284+0000 7f3c2bfff640 10 monclient(hunting): dump: 2024-08-19T12:40:57.304 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.304 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.304 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.304 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.304 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.304 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.304 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.304 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.304 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1462036267 0 0) 0x7f3c440a2a20 con 0x7f3c440a5920 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 >> v1:172.21.15.26:6793/0 conn(0x7f3c440a5d50 legacy=0x7f3c441d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 >> v1:172.21.15.26:6789/0 conn(0x7f3c4406e7b0 legacy=0x7f3c440a4a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 1 monclient: found mon.d 2024-08-19T12:40:57.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c440a1e10 con 0x7f3c440a5920 2024-08-19T12:40:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 10 monclient: _finish_auth 0 2024-08-19T12:40:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:40:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c49c32640 5 monclient: authenticate success, global_id 25818 2024-08-19T12:40:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c49c32640 10 monclient: _renew_subs 2024-08-19T12:40:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c49c32640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c49c32640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3c440a5440 con 0x7f3c440a5920 2024-08-19T12:40:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c49c32640 10 client.25818.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c49c32640 10 monclient: _renew_subs 2024-08-19T12:40:57.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c49c32640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c49c32640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f3c440a5150 con 0x7f3c440a5920 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c49c32640 1 librados: init done 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3c38001d30 con 0x7f3c440a5920 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3c380048a0 con 0x7f3c440a5920 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c41ffb640 4 set_mon_vals no callback set 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c2bfff640 10 monclient: dump: 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.308 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.308 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c49c32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.285+0000 7f3c49c32640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3c240065c0 con 0x7f3c24002860 2024-08-19T12:40:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.286+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3c38011a00 con 0x7f3c440a5920 2024-08-19T12:40:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.286+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(81..81 src has 1..81) ==== 6713+0+0 (unknown 3862053268 0 0) 0x7f3c3804e740 con 0x7f3c440a5920 2024-08-19T12:40:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.286+0000 7f3c2bfff640 10 client.25818.objecter ms_dispatch 0x7f3c44000b90 osd_map(81..81 src has 1..81) 2024-08-19T12:40:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.286+0000 7f3c2bfff640 3 client.25818.objecter handle_osd_map got epochs [81,81] > 0 2024-08-19T12:40:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.286+0000 7f3c2bfff640 3 client.25818.objecter handle_osd_map decoding full epoch 81 2024-08-19T12:40:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.286+0000 7f3c2bfff640 20 client.25818.objecter dump_active .. 0 homeless 2024-08-19T12:40:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.286+0000 7f3c43fff640 10 monclient: get_auth_request con 0x7f3c30041880 auth_method 0 2024-08-19T12:40:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 10 client.25818.objecter ms_handle_connect 0x7f3c24002860 2024-08-19T12:40:57.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 10 client.25818.objecter resend_mon_ops 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c437fe640 1 -- 172.21.15.26:0/2920760918 >> v1:172.21.15.26:6793/0 conn(0x7f3c24002860 legacy=0x7f3c24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4274194296 0 0) 0x7f3c240065c0 con 0x7f3c24002860 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 10 monclient(hunting): my global_id is 25388 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c440a2a20 con 0x7f3c24002860 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4044731108 0 0) 0x7f3c440a2a20 con 0x7f3c24002860 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c240065c0 con 0x7f3c24002860 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3c34001b70 con 0x7f3c24002860 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:57.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 10 monclient: dump: 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.311 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.312 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.312 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.312 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3358310320 0 0) 0x7f3c240065c0 con 0x7f3c24002860 2024-08-19T12:40:57.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c2400a160 con 0x7f3c24002860 2024-08-19T12:40:57.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 10 client.25818.objecter ms_handle_connect 0x7f3c30041880 2024-08-19T12:40:57.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3c340057c0 con 0x7f3c24002860 2024-08-19T12:40:57.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:57.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:57.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 >> v1:172.21.15.26:6793/0 conn(0x7f3c24002860 legacy=0x7f3c24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.288+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 >> v1:172.21.15.26:6793/0 conn(0x7f3c24002860 legacy=0x7f3c24004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:57.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.295+0000 7f3c49c32640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:57.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.295+0000 7f3c49c32640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3c24009e70 con 0x7f3c24009130 2024-08-19T12:40:57.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c437fe640 1 -- 172.21.15.26:0/2920760918 >> v1:172.21.15.26:6793/0 conn(0x7f3c24009130 legacy=0x7f3c240095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 10 client.25818.objecter ms_handle_connect 0x7f3c24009130 2024-08-19T12:40:57.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 10 client.25818.objecter resend_mon_ops 2024-08-19T12:40:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2145548157 0 0) 0x7f3c24009e70 con 0x7f3c24009130 2024-08-19T12:40:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 10 monclient(hunting): my global_id is 25397 2024-08-19T12:40:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c240065c0 con 0x7f3c24009130 2024-08-19T12:40:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2030313327 0 0) 0x7f3c240065c0 con 0x7f3c24009130 2024-08-19T12:40:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c24009e70 con 0x7f3c24009130 2024-08-19T12:40:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3c34003c80 con 0x7f3c24009130 2024-08-19T12:40:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:57.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 10 monclient: dump: 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.316 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.317 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1244220986 0 0) 0x7f3c24009e70 con 0x7f3c24009130 2024-08-19T12:40:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.297+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3c2400c410 con 0x7f3c24009130 2024-08-19T12:40:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7700 (unknown 0 0 116784544) 0x7f3c34005a10 con 0x7f3c24009130 2024-08-19T12:40:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c2bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c2bfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 >> v1:172.21.15.26:6793/0 conn(0x7f3c24009130 legacy=0x7f3c240095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c2bfff640 1 -- 172.21.15.26:0/2920760918 >> v1:172.21.15.26:6793/0 conn(0x7f3c24009130 legacy=0x7f3c240095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:57.317 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 14, 2024-08-19T12:40:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 170314, 2024-08-19T12:40:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:57.318 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.319 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.320 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:57.321 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:57.322 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.323 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:57.324 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:57.325 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.326 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.327 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:57.328 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:57.329 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.330 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.332 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:57.333 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.334 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.336 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.337 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:40:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.338 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:57.338 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:57.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 10 librados: watch_flush enter 2024-08-19T12:40:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 10 librados: watch_flush exit 2024-08-19T12:40:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 20 client.25818.objecter shutdown clearing up homeless session... 2024-08-19T12:40:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 10 client.25818.objecter successfully canceled tick 2024-08-19T12:40:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 1 -- 172.21.15.26:0/2920760918 >> v1:172.21.15.104:6800/899887506 conn(0x7f3c30041880 legacy=0x7f3c30043d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 10 monclient: shutdown 2024-08-19T12:40:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 1 -- 172.21.15.26:0/2920760918 >> v1:172.21.15.26:6792/0 conn(0x7f3c440a5920 legacy=0x7f3c441d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c43fff640 1 -- 172.21.15.26:0/2920760918 reap_dead start 2024-08-19T12:40:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c43fff640 1 -- 172.21.15.26:0/2920760918 reap_dead start 2024-08-19T12:40:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 1 -- 172.21.15.26:0/2920760918 shutdown_connections 2024-08-19T12:40:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 1 -- 172.21.15.26:0/2920760918 >> 172.21.15.26:0/2920760918 conn(0x7f3c440a8740 msgr2=0x7f3c4406dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:57.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 1 -- 172.21.15.26:0/2920760918 shutdown_connections 2024-08-19T12:40:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 1 -- 172.21.15.26:0/2920760918 wait complete. 2024-08-19T12:40:57.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.298+0000 7f3c49c32640 1 librados: shutdown 2024-08-19T12:40:57.341 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:40:57.341 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-08-19T12:40:57.341 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:40:57.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.439+0000 7f4349d7f640 10 monclient: get_monmap_and_config 2024-08-19T12:40:57.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.439+0000 7f4349d7f640 10 monclient: build_initial_monmap 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.439+0000 7f4349d7f640 10 monclient: monmap: 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:57.440525+0000 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:57.440525+0000 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:57.441 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:57.442 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:57.442 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:57.442 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:57.442 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.440+0000 7f4349d7f640 1 Processor -- start 2024-08-19T12:40:57.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.440+0000 7f4349d7f640 1 -- start start 2024-08-19T12:40:57.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.440+0000 7f4349d7f640 10 monclient: init 2024-08-19T12:40:57.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.440+0000 7f4349d7f640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:57.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.440+0000 7f4349d7f640 10 monclient: _add_conns ranks=[3,1,2,8,5,0,7,4,6] 2024-08-19T12:40:57.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.441+0000 7f4349d7f640 10 monclient(hunting): picked mon.noname-g con 0x7f434406e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:57.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.441+0000 7f4349d7f640 10 monclient(hunting): picked mon.noname-f con 0x7f43440a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:57.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.441+0000 7f4349d7f640 10 monclient(hunting): picked mon.noname-b con 0x7f43440a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:57.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.441+0000 7f4349d7f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4344136790 con 0x7f43440a5920 2024-08-19T12:40:57.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.441+0000 7f4349d7f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f434409ddf0 con 0x7f43440a5d50 2024-08-19T12:40:57.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.441+0000 7f4349d7f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f434409e0d0 con 0x7f434406e7b0 2024-08-19T12:40:57.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.441+0000 7f4349d7f640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:57.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.441+0000 7f4349d7f640 10 monclient(hunting): authenticate will time out at 851.528070s 2024-08-19T12:40:57.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.441+0000 7f4343fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f43440a5d50 0x7f4344141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33414/0 (socket says 172.21.15.26:33414) 2024-08-19T12:40:57.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.441+0000 7f4343fff640 1 -- 172.21.15.26:0/4151200035 learned_addr learned my addr 172.21.15.26:0/4151200035 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:57.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4343fff640 1 -- 172.21.15.26:0/4151200035 >> v1:172.21.15.26:6790/0 conn(0x7f43440a5d50 legacy=0x7f4344141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 1 -- 172.21.15.26:0/4151200035 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3995140063 0 0) 0x7f433c0017d0 con 0x7f43440a5d50 2024-08-19T12:40:57.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient(hunting): my global_id is 25580 2024-08-19T12:40:57.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 1 -- 172.21.15.26:0/4151200035 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4330003520 con 0x7f43440a5d50 2024-08-19T12:40:57.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f433bfff640 1 -- 172.21.15.26:0/4151200035 >> v1:172.21.15.104:6789/0 conn(0x7f43440a5920 legacy=0x7f434413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f43437fe640 1 -- 172.21.15.26:0/4151200035 >> v1:172.21.15.104:6790/0 conn(0x7f434406e7b0 legacy=0x7f4344136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 1 -- 172.21.15.26:0/4151200035 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 688595436 0 0) 0x7f4330003520 con 0x7f43440a5d50 2024-08-19T12:40:57.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 1 -- 172.21.15.26:0/4151200035 >> v1:172.21.15.104:6790/0 conn(0x7f434406e7b0 legacy=0x7f4344136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 1 -- 172.21.15.26:0/4151200035 >> v1:172.21.15.104:6789/0 conn(0x7f43440a5920 legacy=0x7f434413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 1 monclient: found mon.noname-b 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 1 -- 172.21.15.26:0/4151200035 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434409e3b0 con 0x7f43440a5d50 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient: _finish_auth 0 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4349d7f640 5 monclient: authenticate success, global_id 25580 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4349d7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 1 -- 172.21.15.26:0/4151200035 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f433c002030 con 0x7f43440a5d50 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient: dump: 2024-08-19T12:40:57.449 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.450 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4349d7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 1 -- 172.21.15.26:0/4151200035 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f433c004d10 con 0x7f43440a5d50 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 1 -- 172.21.15.26:0/4151200035 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f433c005280 con 0x7f43440a5d50 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4349d7f640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4349d7f640 4 set_mon_vals no callback set 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.443+0000 7f4342ffd640 10 monclient: dump: 2024-08-19T12:40:57.451 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.452 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.444+0000 7f4349d7f640 10 monclient: shutdown 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.444+0000 7f4349d7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.444+0000 7f4349d7f640 1 -- 172.21.15.26:0/4151200035 >> v1:172.21.15.26:6790/0 conn(0x7f43440a5d50 legacy=0x7f4344141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.444+0000 7f4349d7f640 1 -- 172.21.15.26:0/4151200035 shutdown_connections 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.444+0000 7f4349d7f640 1 -- 172.21.15.26:0/4151200035 >> 172.21.15.26:0/4151200035 conn(0x7f43440a8740 msgr2=0x7f43440a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.444+0000 7f4349d7f640 1 -- 172.21.15.26:0/4151200035 shutdown_connections 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.444+0000 7f4349d7f640 1 -- 172.21.15.26:0/4151200035 wait complete. 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 10 monclient: build_initial_monmap 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 10 monclient: monmap: 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:57.453 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 1 librados: starting msgr at 2024-08-19T12:40:57.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 1 librados: starting objecter 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 1 Processor -- start 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 1 -- start start 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 1 librados: setting wanted keys 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 1 librados: calling monclient init 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 10 monclient: init 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 10 monclient: _add_conns ranks=[3,0,8,7,1,4,5,6,2] 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 10 monclient(hunting): picked mon.noname-g con 0x7f434406e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 10 monclient(hunting): picked mon.noname-a con 0x7f43440a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 10 monclient(hunting): picked mon.noname-e con 0x7f43440a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43440a34d0 con 0x7f43440a5920 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43441349c0 con 0x7f434406e7b0 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f433bfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f43440a5920 legacy=0x7f43441d36d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:40:57.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43440a1170 con 0x7f43440a5d50 2024-08-19T12:40:57.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:57.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.445+0000 7f4349d7f640 10 monclient(hunting): authenticate will time out at 851.532935s 2024-08-19T12:40:57.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.446+0000 7f4343fff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f43440a5d50 0x7f43441d6e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:45574/0 (socket says 172.21.15.26:45574) 2024-08-19T12:40:57.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.446+0000 7f4343fff640 1 -- 172.21.15.26:0/3719034875 learned_addr learned my addr 172.21.15.26:0/3719034875 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:57.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4343fff640 1 -- 172.21.15.26:0/3719034875 >> v1:172.21.15.26:6793/0 conn(0x7f43440a5d50 legacy=0x7f43441d6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 client.?.objecter ms_handle_connect 0x7f43440a5d50 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 client.?.objecter ms_handle_connect 0x7f434406e7b0 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f43437fe640 1 -- 172.21.15.26:0/3719034875 >> v1:172.21.15.104:6790/0 conn(0x7f434406e7b0 legacy=0x7f43441ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2725194937 0 0) 0x7f43440a1170 con 0x7f43440a5d50 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient(hunting): my global_id is 25406 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4324004eb0 con 0x7f43440a5d50 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1627828309 0 0) 0x7f43441349c0 con 0x7f434406e7b0 2024-08-19T12:40:57.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient(hunting): my global_id is 25689 2024-08-19T12:40:57.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43440a1170 con 0x7f434406e7b0 2024-08-19T12:40:57.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3060124933 0 0) 0x7f4324004eb0 con 0x7f43440a5d50 2024-08-19T12:40:57.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43441349c0 con 0x7f43440a5d50 2024-08-19T12:40:57.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f433c001700 con 0x7f43440a5d50 2024-08-19T12:40:57.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:57.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:40:57.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient(hunting): dump: 2024-08-19T12:40:57.462 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.462 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.463 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f43437fe640 1 --1- 172.21.15.26:0/3719034875 >> v1:172.21.15.104:6790/0 conn(0x7f434406e7b0 0x7f43441ce690 :-1 s=OPENED pgs=3052 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f43440a1170 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 283404152 0 0) 0x7f43440a1170 con 0x7f434406e7b0 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4324004eb0 con 0x7f434406e7b0 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2296550627 0 0) 0x7f43441349c0 con 0x7f43440a5d50 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 >> v1:172.21.15.104:6790/0 conn(0x7f434406e7b0 legacy=0x7f43441ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 >> v1:172.21.15.26:6789/0 conn(0x7f43440a5920 legacy=0x7f43441d36d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 monclient: found mon.e 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43440a31f0 con 0x7f43440a5d50 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient: _finish_auth 0 2024-08-19T12:40:57.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4334001c80 con 0x7f434406e7b0 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4349d7f640 5 monclient: authenticate success, global_id 25406 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4349d7f640 10 monclient: _renew_subs 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4349d7f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4349d7f640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f43440a5300 con 0x7f43440a5d50 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4349d7f640 10 client.25406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4349d7f640 10 monclient: _renew_subs 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4349d7f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4349d7f640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f43440a0910 con 0x7f43440a5d50 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4349d7f640 1 librados: init done 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f433c0054b0 con 0x7f43440a5d50 2024-08-19T12:40:57.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f433c004d10 con 0x7f43440a5d50 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f43427fc640 4 set_mon_vals no callback set 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.448+0000 7f4340ff9640 10 monclient: dump: 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.466 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.449+0000 7f4349d7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.449+0000 7f4349d7f640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f43280065c0 con 0x7f4328002860 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.449+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f433c011c60 con 0x7f43440a5d50 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.449+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(81..81 src has 1..81) ==== 6713+0+0 (unknown 3862053268 0 0) 0x7f433c04e9e0 con 0x7f43440a5d50 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.449+0000 7f4340ff9640 10 client.25406.objecter ms_dispatch 0x7f4344000b90 osd_map(81..81 src has 1..81) 2024-08-19T12:40:57.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.449+0000 7f4340ff9640 3 client.25406.objecter handle_osd_map got epochs [81,81] > 0 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.449+0000 7f4340ff9640 3 client.25406.objecter handle_osd_map decoding full epoch 81 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.450+0000 7f433bfff640 10 monclient: get_auth_request con 0x7f4324041200 auth_method 0 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.450+0000 7f4340ff9640 20 client.25406.objecter dump_active .. 0 homeless 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.451+0000 7f43437fe640 1 -- 172.21.15.26:0/3719034875 >> v1:172.21.15.104:6789/0 conn(0x7f4328002860 legacy=0x7f4328004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.451+0000 7f4340ff9640 10 client.25406.objecter ms_handle_connect 0x7f4328002860 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.451+0000 7f4340ff9640 10 client.25406.objecter resend_mon_ops 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1026554752 0 0) 0x7f43280065c0 con 0x7f4328002860 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 10 monclient(hunting): my global_id is 26200 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43441349c0 con 0x7f4328002860 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2756248641 0 0) 0x7f43441349c0 con 0x7f4328002860 2024-08-19T12:40:57.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43280065c0 con 0x7f4328002860 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4334003bc0 con 0x7f4328002860 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 10 monclient: dump: 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.469 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 10 client.25406.objecter ms_handle_connect 0x7f4324041200 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3337603543 0 0) 0x7f43280065c0 con 0x7f4328002860 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f432800a160 con 0x7f4328002860 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4334003f90 con 0x7f4328002860 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 >> v1:172.21.15.104:6789/0 conn(0x7f4328002860 legacy=0x7f4328004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.452+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 >> v1:172.21.15.104:6789/0 conn(0x7f4328002860 legacy=0x7f4328004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:57.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.458+0000 7f4349d7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:57.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.458+0000 7f4349d7f640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4328009e50 con 0x7f4328009110 2024-08-19T12:40:57.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f43437fe640 1 -- 172.21.15.26:0/3719034875 >> v1:172.21.15.104:6789/0 conn(0x7f4328009110 legacy=0x7f43280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 10 client.25406.objecter ms_handle_connect 0x7f4328009110 2024-08-19T12:40:57.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 10 client.25406.objecter resend_mon_ops 2024-08-19T12:40:57.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2309614354 0 0) 0x7f4328009e50 con 0x7f4328009110 2024-08-19T12:40:57.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 10 monclient(hunting): my global_id is 26209 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43280065c0 con 0x7f4328009110 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 684322349 0 0) 0x7f43280065c0 con 0x7f4328009110 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4328009e50 con 0x7f4328009110 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4334004470 con 0x7f4328009110 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.461+0000 7f4340ff9640 10 monclient: dump: 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.473 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 197569732 0 0) 0x7f4328009e50 con 0x7f4328009110 2024-08-19T12:40:57.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f432800c3f0 con 0x7f4328009110 2024-08-19T12:40:57.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7702 (unknown 0 0 645338106) 0x7f4334005cd0 con 0x7f4328009110 2024-08-19T12:40:57.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4340ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:57.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4340ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:57.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 >> v1:172.21.15.104:6789/0 conn(0x7f4328009110 legacy=0x7f43280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4340ff9640 1 -- 172.21.15.26:0/3719034875 >> v1:172.21.15.104:6789/0 conn(0x7f4328009110 legacy=0x7f43280095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:57.475 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:57.475 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:57.475 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:57.475 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:40:57.475 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 14, 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 170466, 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:40:57.476 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.477 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.478 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:57.479 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.480 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.482 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:57.483 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.484 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.486 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:57.487 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:57.487 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:57.487 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.487 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.487 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.488 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:57.489 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.490 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.491 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:57.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.495 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.495 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:57.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.495 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.495 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.495 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-08-19T12:40:57.495 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.495 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.495 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:57.495 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:57.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 10 librados: watch_flush enter 2024-08-19T12:40:57.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 10 librados: watch_flush exit 2024-08-19T12:40:57.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 20 client.25406.objecter shutdown clearing up homeless session... 2024-08-19T12:40:57.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 10 client.25406.objecter successfully canceled tick 2024-08-19T12:40:57.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 1 -- 172.21.15.26:0/3719034875 >> v1:172.21.15.104:6800/899887506 conn(0x7f4324041200 legacy=0x7f43240436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 10 monclient: shutdown 2024-08-19T12:40:57.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:57.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 1 -- 172.21.15.26:0/3719034875 >> v1:172.21.15.26:6793/0 conn(0x7f43440a5d50 legacy=0x7f43441d6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4343fff640 1 -- 172.21.15.26:0/3719034875 reap_dead start 2024-08-19T12:40:57.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4343fff640 1 -- 172.21.15.26:0/3719034875 reap_dead start 2024-08-19T12:40:57.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 1 -- 172.21.15.26:0/3719034875 shutdown_connections 2024-08-19T12:40:57.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 1 -- 172.21.15.26:0/3719034875 >> 172.21.15.26:0/3719034875 conn(0x7f43440a8740 msgr2=0x7f43440aa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:57.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 1 -- 172.21.15.26:0/3719034875 shutdown_connections 2024-08-19T12:40:57.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 1 -- 172.21.15.26:0/3719034875 wait complete. 2024-08-19T12:40:57.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.462+0000 7f4349d7f640 1 librados: shutdown 2024-08-19T12:40:57.499 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:40:57.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-08-19T12:40:57.499 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:40:57.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.598+0000 7f030e4de640 10 monclient: get_monmap_and_config 2024-08-19T12:40:57.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.598+0000 7f030e4de640 10 monclient: build_initial_monmap 2024-08-19T12:40:57.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.598+0000 7f030e4de640 10 monclient: monmap: 2024-08-19T12:40:57.599 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:57.599 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.599 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:57.599070+0000 2024-08-19T12:40:57.599 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:57.599070+0000 2024-08-19T12:40:57.599 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:57.599 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.599 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:57.599 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:57.600 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:57.600 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:57.600 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:57.600 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:57.600 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:57.600 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:57.600 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:57.600 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.598+0000 7f030e4de640 1 Processor -- start 2024-08-19T12:40:57.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f030e4de640 1 -- start start 2024-08-19T12:40:57.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f030e4de640 10 monclient: init 2024-08-19T12:40:57.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f030e4de640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:57.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f030e4de640 10 monclient: _add_conns ranks=[2,5,1,7,3,8,6,0,4] 2024-08-19T12:40:57.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f030e4de640 10 monclient(hunting): picked mon.noname-b con 0x7f0308073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:57.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f030e4de640 10 monclient(hunting): picked mon.noname-h con 0x7f03080a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:57.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f030e4de640 10 monclient(hunting): picked mon.noname-f con 0x7f030813f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:40:57.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f030e4de640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f030809dae0 con 0x7f030813f2e0 2024-08-19T12:40:57.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f030e4de640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0308136910 con 0x7f0308073470 2024-08-19T12:40:57.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f030e4de640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f030809dd50 con 0x7f03080a57a0 2024-08-19T12:40:57.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f030e4de640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:57.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f030e4de640 10 monclient(hunting): authenticate will time out at 851.686603s 2024-08-19T12:40:57.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f0307fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0308073470 0x7f0308136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33426/0 (socket says 172.21.15.26:33426) 2024-08-19T12:40:57.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.599+0000 7f0307fff640 1 -- 172.21.15.26:0/1327861171 learned_addr learned my addr 172.21.15.26:0/1327861171 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:57.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.601+0000 7f0307fff640 1 -- 172.21.15.26:0/1327861171 >> v1:172.21.15.26:6790/0 conn(0x7f0308073470 legacy=0x7f0308136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.601+0000 7f0306ffd640 1 -- 172.21.15.26:0/1327861171 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530508539 0 0) 0x7f0308136910 con 0x7f0308073470 2024-08-19T12:40:57.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient(hunting): my global_id is 25589 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 1 -- 172.21.15.26:0/1327861171 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02f0003520 con 0x7f0308073470 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f030ca7b640 1 -- 172.21.15.26:0/1327861171 >> v1:172.21.15.104:6789/0 conn(0x7f030813f2e0 legacy=0x7f03081416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f03077fe640 1 -- 172.21.15.26:0/1327861171 >> v1:172.21.15.104:6791/0 conn(0x7f03080a57a0 legacy=0x7f03080a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 1 -- 172.21.15.26:0/1327861171 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3741844919 0 0) 0x7f02f0003520 con 0x7f0308073470 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 1 -- 172.21.15.26:0/1327861171 >> v1:172.21.15.104:6791/0 conn(0x7f03080a57a0 legacy=0x7f03080a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 1 -- 172.21.15.26:0/1327861171 >> v1:172.21.15.104:6789/0 conn(0x7f030813f2e0 legacy=0x7f03081416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 1 monclient: found mon.noname-b 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:57.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:40:57.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 1 -- 172.21.15.26:0/1327861171 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f030809e030 con 0x7f0308073470 2024-08-19T12:40:57.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient: _finish_auth 0 2024-08-19T12:40:57.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:40:57.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:57.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 1 -- 172.21.15.26:0/1327861171 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f02f8001ae0 con 0x7f0308073470 2024-08-19T12:40:57.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:57.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient: dump: 2024-08-19T12:40:57.612 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.612 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.613 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f030e4de640 5 monclient: authenticate success, global_id 25589 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f030e4de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 1 -- 172.21.15.26:0/1327861171 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f02f80048a0 con 0x7f0308073470 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 1 -- 172.21.15.26:0/1327861171 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f02f8004e10 con 0x7f0308073470 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f030e4de640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f030e4de640 4 set_mon_vals no callback set 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f0306ffd640 10 monclient: dump: 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.614 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.615 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f030e4de640 10 monclient: shutdown 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f030e4de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f030e4de640 1 -- 172.21.15.26:0/1327861171 >> v1:172.21.15.26:6790/0 conn(0x7f0308073470 legacy=0x7f0308136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f030e4de640 1 -- 172.21.15.26:0/1327861171 shutdown_connections 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f030e4de640 1 -- 172.21.15.26:0/1327861171 >> 172.21.15.26:0/1327861171 conn(0x7f03080abff0 msgr2=0x7f03080ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f030e4de640 1 -- 172.21.15.26:0/1327861171 shutdown_connections 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.602+0000 7f030e4de640 1 -- 172.21.15.26:0/1327861171 wait complete. 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.603+0000 7f030e4de640 10 monclient: build_initial_monmap 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.603+0000 7f030e4de640 10 monclient: monmap: 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:57.616 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.603+0000 7f030e4de640 1 librados: starting msgr at 2024-08-19T12:40:57.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.603+0000 7f030e4de640 1 librados: starting objecter 2024-08-19T12:40:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.603+0000 7f030e4de640 1 Processor -- start 2024-08-19T12:40:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 1 -- start start 2024-08-19T12:40:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 1 librados: setting wanted keys 2024-08-19T12:40:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 1 librados: calling monclient init 2024-08-19T12:40:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 10 monclient: init 2024-08-19T12:40:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 10 monclient: _add_conns ranks=[4,7,3,0,1,2,5,6,8] 2024-08-19T12:40:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 10 monclient(hunting): picked mon.noname-c con 0x7f0308073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 10 monclient(hunting): picked mon.noname-i con 0x7f03080a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 10 monclient(hunting): picked mon.noname-g con 0x7f030813f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0308142ba0 con 0x7f030813f2e0 2024-08-19T12:40:57.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f030812e9f0 con 0x7f0308073470 2024-08-19T12:40:57.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0308131510 con 0x7f03080a57a0 2024-08-19T12:40:57.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:57.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f030e4de640 10 monclient(hunting): authenticate will time out at 851.691578s 2024-08-19T12:40:57.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f0307fff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f0308073470 0x7f030812dbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:55372/0 (socket says 172.21.15.26:55372) 2024-08-19T12:40:57.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.604+0000 7f0307fff640 1 -- 172.21.15.26:0/2931861940 learned_addr learned my addr 172.21.15.26:0/2931861940 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:57.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0307fff640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.26:6791/0 conn(0x7f0308073470 legacy=0x7f030812dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0304ff9640 10 client.?.objecter ms_handle_connect 0x7f0308073470 2024-08-19T12:40:57.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0304ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f03077fe640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.104:6792/0 conn(0x7f03080a57a0 legacy=0x7f030812e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0304ff9640 10 client.?.objecter ms_handle_connect 0x7f03080a57a0 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0304ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0304ff9640 10 client.?.objecter ms_handle_connect 0x7f030813f2e0 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0304ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f030ca7b640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.104:6790/0 conn(0x7f030813f2e0 legacy=0x7f03081d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3660294118 0 0) 0x7f030812e9f0 con 0x7f0308073470 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0304ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0304ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0304ff9640 10 monclient(hunting): my global_id is 25780 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.606+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02f4004eb0 con 0x7f0308073470 2024-08-19T12:40:57.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659534332 0 0) 0x7f0308131510 con 0x7f03080a57a0 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): my global_id is 134314 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f030812e9f0 con 0x7f03080a57a0 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2146292080 0 0) 0x7f03000018d0 con 0x7f030813f2e0 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): my global_id is 25698 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0308131510 con 0x7f030813f2e0 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2602881304 0 0) 0x7f02f4004eb0 con 0x7f0308073470 2024-08-19T12:40:57.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03000018d0 con 0x7f0308073470 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f02f8004d90 con 0x7f0308073470 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): dump: 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.622 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 317023629 0 0) 0x7f030812e9f0 con 0x7f03080a57a0 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02f4004eb0 con 0x7f03080a57a0 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f02fc003610 con 0x7f03080a57a0 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:57.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient(hunting): dump: 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.624 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 23290581 0 0) 0x7f03000018d0 con 0x7f0308073470 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.104:6792/0 conn(0x7f03080a57a0 legacy=0x7f030812e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.104:6790/0 conn(0x7f030813f2e0 legacy=0x7f03081d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 monclient: found mon.c 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03081317f0 con 0x7f0308073470 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient: _finish_auth 0 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:40:57.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f030e4de640 5 monclient: authenticate success, global_id 25780 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f030e4de640 10 monclient: _renew_subs 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f030e4de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f030e4de640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f030812fff0 con 0x7f0308073470 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f030e4de640 10 client.25780.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f030e4de640 10 monclient: _renew_subs 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f030e4de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f030e4de640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f03081330c0 con 0x7f0308073470 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f030e4de640 1 librados: init done 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f02f8001d30 con 0x7f0308073470 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f02f80048a0 con 0x7f0308073470 2024-08-19T12:40:57.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f03067fc640 4 set_mon_vals no callback set 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.607+0000 7f0304ff9640 10 monclient: dump: 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.627 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.608+0000 7f030e4de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.608+0000 7f030e4de640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f02e80065c0 con 0x7f02e8002860 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.608+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f02f8011a00 con 0x7f0308073470 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.608+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(81..81 src has 1..81) ==== 6713+0+0 (unknown 3862053268 0 0) 0x7f02f804e7d0 con 0x7f0308073470 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.608+0000 7f0304ff9640 10 client.25780.objecter ms_dispatch 0x7f0308000b90 osd_map(81..81 src has 1..81) 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.608+0000 7f0304ff9640 3 client.25780.objecter handle_osd_map got epochs [81,81] > 0 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.608+0000 7f0304ff9640 3 client.25780.objecter handle_osd_map decoding full epoch 81 2024-08-19T12:40:57.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.608+0000 7f0304ff9640 20 client.25780.objecter dump_active .. 0 homeless 2024-08-19T12:40:57.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.608+0000 7f030ca7b640 10 monclient: get_auth_request con 0x7f02f4041580 auth_method 0 2024-08-19T12:40:57.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.610+0000 7f0304ff9640 10 client.25780.objecter ms_handle_connect 0x7f02e8002860 2024-08-19T12:40:57.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.610+0000 7f0304ff9640 10 client.25780.objecter resend_mon_ops 2024-08-19T12:40:57.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.610+0000 7f03077fe640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.104:6790/0 conn(0x7f02e8002860 legacy=0x7f02e8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.610+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 284696140 0 0) 0x7f02e80065c0 con 0x7f02e8002860 2024-08-19T12:40:57.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.610+0000 7f0304ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.610+0000 7f0304ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.610+0000 7f0304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.610+0000 7f0304ff9640 10 monclient(hunting): my global_id is 25707 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.610+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03000018d0 con 0x7f02e8002860 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 540191185 0 0) 0x7f03000018d0 con 0x7f02e8002860 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02e80065c0 con 0x7f02e8002860 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f02fc003e20 con 0x7f02e8002860 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 10 monclient: dump: 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.632 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 10 client.25780.objecter ms_handle_connect 0x7f02f4041580 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2371559159 0 0) 0x7f02e80065c0 con 0x7f02e8002860 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02e800a160 con 0x7f02e8002860 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f02fc004050 con 0x7f02e8002860 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.104:6790/0 conn(0x7f02e8002860 legacy=0x7f02e8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.611+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.104:6790/0 conn(0x7f02e8002860 legacy=0x7f02e8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.617+0000 7f030e4de640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.617+0000 7f030e4de640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f02e8009e70 con 0x7f02e8009130 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f03077fe640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.104:6790/0 conn(0x7f02e8009130 legacy=0x7f02e80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 10 client.25780.objecter ms_handle_connect 0x7f02e8009130 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 10 client.25780.objecter resend_mon_ops 2024-08-19T12:40:57.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2476492268 0 0) 0x7f02e8009e70 con 0x7f02e8009130 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 10 monclient(hunting): my global_id is 25716 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02e80065c0 con 0x7f02e8009130 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3503967733 0 0) 0x7f02e80065c0 con 0x7f02e8009130 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02e8009e70 con 0x7f02e8009130 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f02fc005b00 con 0x7f02e8009130 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 10 monclient: dump: 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.636 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3097696178 0 0) 0x7f02e8009e70 con 0x7f02e8009130 2024-08-19T12:40:57.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.620+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f02e800c410 con 0x7f02e8009130 2024-08-19T12:40:57.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 3123279905) 0x7f02fc005e10 con 0x7f02e8009130 2024-08-19T12:40:57.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f0304ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:57.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f0304ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:57.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.104:6790/0 conn(0x7f02e8009130 legacy=0x7f02e80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f0304ff9640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.104:6790/0 conn(0x7f02e8009130 legacy=0x7f02e80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:57.637 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:57.637 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:57.637 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:57.638 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:57.638 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 14, 2024-08-19T12:40:57.638 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 170633, 2024-08-19T12:40:57.638 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:57.638 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:57.638 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:57.638 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:57.638 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:57.638 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:57.638 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:57.638 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:57.638 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.639 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.640 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:57.641 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.642 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.643 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:57.644 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:57.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:57.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:57.650 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:57.650 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.650 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.650 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.650 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:57.651 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.652 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.654 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.654 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:57.654 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:57.654 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:57.655 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.656 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.658 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 6 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:57.659 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:57.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030e4de640 10 librados: watch_flush enter 2024-08-19T12:40:57.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030e4de640 10 librados: watch_flush exit 2024-08-19T12:40:57.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030e4de640 20 client.25780.objecter shutdown clearing up homeless session... 2024-08-19T12:40:57.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030e4de640 10 client.25780.objecter successfully canceled tick 2024-08-19T12:40:57.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030e4de640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.104:6800/899887506 conn(0x7f02f4041580 legacy=0x7f02f4043a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030e4de640 10 monclient: shutdown 2024-08-19T12:40:57.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030e4de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:57.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030e4de640 1 -- 172.21.15.26:0/2931861940 >> v1:172.21.15.26:6791/0 conn(0x7f0308073470 legacy=0x7f030812dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030ca7b640 1 -- 172.21.15.26:0/2931861940 reap_dead start 2024-08-19T12:40:57.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030ca7b640 1 -- 172.21.15.26:0/2931861940 reap_dead start 2024-08-19T12:40:57.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030e4de640 1 -- 172.21.15.26:0/2931861940 shutdown_connections 2024-08-19T12:40:57.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030e4de640 1 -- 172.21.15.26:0/2931861940 >> 172.21.15.26:0/2931861940 conn(0x7f03080abff0 msgr2=0x7f03080a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:57.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030e4de640 1 -- 172.21.15.26:0/2931861940 shutdown_connections 2024-08-19T12:40:57.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.621+0000 7f030e4de640 1 -- 172.21.15.26:0/2931861940 wait complete. 2024-08-19T12:40:57.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.622+0000 7f030e4de640 1 librados: shutdown 2024-08-19T12:40:57.663 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:40:57.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-08-19T12:40:57.663 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:40:57.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.762+0000 7f664b79f640 10 monclient: get_monmap_and_config 2024-08-19T12:40:57.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.762+0000 7f664b79f640 10 monclient: build_initial_monmap 2024-08-19T12:40:57.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.762+0000 7f664b79f640 10 monclient: monmap: 2024-08-19T12:40:57.763 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:57.763 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.763 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:57.763182+0000 2024-08-19T12:40:57.763 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:57.763182+0000 2024-08-19T12:40:57.763 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:57.763 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.763 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:57.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:57.764 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:57.764 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:57.764 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:57.764 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:57.764 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:57.764 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:57.764 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:57.764 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.762+0000 7f664b79f640 1 Processor -- start 2024-08-19T12:40:57.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664b79f640 1 -- start start 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664b79f640 10 monclient: init 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664b79f640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664b79f640 10 monclient: _add_conns ranks=[0,7,2,1,5,8,3,6,4] 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664b79f640 10 monclient(hunting): picked mon.noname-a con 0x7f664406e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664b79f640 10 monclient(hunting): picked mon.noname-i con 0x7f66440a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664b79f640 10 monclient(hunting): picked mon.noname-b con 0x7f66440a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664b79f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6644136790 con 0x7f664406e7b0 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664b79f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f664409ddf0 con 0x7f66440a5d50 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664b79f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f664409e0d0 con 0x7f66440a5920 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664b79f640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664b79f640 10 monclient(hunting): authenticate will time out at 851.850651s 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f664953b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f664406e7b0 legacy=0x7f6644136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:40:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f6649d3c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f66440a5d50 0x7f6644141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33430/0 (socket says 172.21.15.26:33430) 2024-08-19T12:40:57.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.763+0000 7f6649d3c640 1 -- 172.21.15.26:0/420482225 learned_addr learned my addr 172.21.15.26:0/420482225 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.765+0000 7f6649d3c640 1 -- 172.21.15.26:0/420482225 >> v1:172.21.15.26:6790/0 conn(0x7f66440a5d50 legacy=0x7f6644141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f6648d3a640 1 -- 172.21.15.26:0/420482225 >> v1:172.21.15.104:6792/0 conn(0x7f66440a5920 legacy=0x7f664413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 1 -- 172.21.15.26:0/420482225 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1882782447 0 0) 0x7f663c0017d0 con 0x7f66440a5d50 2024-08-19T12:40:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient(hunting): my global_id is 25598 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 1 -- 172.21.15.26:0/420482225 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6630003520 con 0x7f66440a5d50 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 1 -- 172.21.15.26:0/420482225 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101602919 0 0) 0x7f664409e0d0 con 0x7f66440a5920 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient(hunting): my global_id is 134323 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 1 -- 172.21.15.26:0/420482225 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f663c0017d0 con 0x7f66440a5920 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 1 -- 172.21.15.26:0/420482225 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 293598752 0 0) 0x7f6630003520 con 0x7f66440a5d50 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 1 -- 172.21.15.26:0/420482225 >> v1:172.21.15.104:6792/0 conn(0x7f66440a5920 legacy=0x7f664413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 1 -- 172.21.15.26:0/420482225 >> v1:172.21.15.26:6789/0 conn(0x7f664406e7b0 legacy=0x7f6644136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:40:57.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 1 monclient: found mon.noname-b 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 1 -- 172.21.15.26:0/420482225 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f664409e3b0 con 0x7f66440a5d50 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient: _finish_auth 0 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 1 -- 172.21.15.26:0/420482225 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f663c002030 con 0x7f66440a5d50 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient: dump: 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.772 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f664b79f640 5 monclient: authenticate success, global_id 25598 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f664b79f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 1 -- 172.21.15.26:0/420482225 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f663c004cd0 con 0x7f66440a5d50 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f664b79f640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 1 -- 172.21.15.26:0/420482225 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f663c005200 con 0x7f66440a5d50 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f664b79f640 4 set_mon_vals no callback set 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f663bfff640 10 monclient: dump: 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.774 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f664b79f640 10 monclient: shutdown 2024-08-19T12:40:57.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f664b79f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f664b79f640 1 -- 172.21.15.26:0/420482225 >> v1:172.21.15.26:6790/0 conn(0x7f66440a5d50 legacy=0x7f6644141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f664b79f640 1 -- 172.21.15.26:0/420482225 shutdown_connections 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.766+0000 7f664b79f640 1 -- 172.21.15.26:0/420482225 >> 172.21.15.26:0/420482225 conn(0x7f66440a8740 msgr2=0x7f66440a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.767+0000 7f664b79f640 1 -- 172.21.15.26:0/420482225 shutdown_connections 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.767+0000 7f664b79f640 1 -- 172.21.15.26:0/420482225 wait complete. 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.767+0000 7f664b79f640 10 monclient: build_initial_monmap 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.767+0000 7f664b79f640 10 monclient: monmap: 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.776 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.767+0000 7f664b79f640 1 librados: starting msgr at 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.767+0000 7f664b79f640 1 librados: starting objecter 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.767+0000 7f664b79f640 1 Processor -- start 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.767+0000 7f664b79f640 1 -- start start 2024-08-19T12:40:57.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.767+0000 7f664b79f640 1 librados: setting wanted keys 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.767+0000 7f664b79f640 1 librados: calling monclient init 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.767+0000 7f664b79f640 10 monclient: init 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.768+0000 7f664b79f640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.768+0000 7f664b79f640 10 monclient: _add_conns ranks=[7,3,6,5,0,1,8,4,2] 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.768+0000 7f664b79f640 10 monclient(hunting): picked mon.noname-i con 0x7f664406e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.768+0000 7f664b79f640 10 monclient(hunting): picked mon.noname-g con 0x7f66440a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.768+0000 7f664b79f640 10 monclient(hunting): picked mon.noname-d con 0x7f66440a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.768+0000 7f664b79f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6644138e10 con 0x7f66440a5920 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.768+0000 7f664b79f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66441382c0 con 0x7f66440a5d50 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.768+0000 7f664b79f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f664413b170 con 0x7f664406e7b0 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.768+0000 7f664b79f640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.768+0000 7f664b79f640 10 monclient(hunting): authenticate will time out at 851.855427s 2024-08-19T12:40:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.768+0000 7f6649d3c640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f66440a5d50 0x7f66441d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:44198/0 (socket says 172.21.15.26:44198) 2024-08-19T12:40:57.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.768+0000 7f6649d3c640 1 -- 172.21.15.26:0/3312184795 learned_addr learned my addr 172.21.15.26:0/3312184795 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:57.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.770+0000 7f6649d3c640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.26:6792/0 conn(0x7f66440a5d50 legacy=0x7f66441d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.770+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:40:57.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.770+0000 7f6648d3a640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.104:6790/0 conn(0x7f66440a5920 legacy=0x7f6644137bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.770+0000 7f664953b640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.104:6792/0 conn(0x7f664406e7b0 legacy=0x7f66441374a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 client.?.objecter ms_handle_connect 0x7f66440a5d50 2024-08-19T12:40:57.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 client.?.objecter ms_handle_connect 0x7f66440a5920 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 client.?.objecter ms_handle_connect 0x7f664406e7b0 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2594385216 0 0) 0x7f66441382c0 con 0x7f66440a5d50 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 monclient(hunting): my global_id is 25827 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6628004eb0 con 0x7f66440a5d50 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3262466715 0 0) 0x7f664413b170 con 0x7f664406e7b0 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 monclient(hunting): my global_id is 134332 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66441382c0 con 0x7f664406e7b0 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3533582214 0 0) 0x7f6644138e10 con 0x7f66440a5920 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 10 monclient(hunting): my global_id is 25725 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.772+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f664413b170 con 0x7f66440a5920 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 752490287 0 0) 0x7f6628004eb0 con 0x7f66440a5d50 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6644138e10 con 0x7f66440a5d50 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f663c002050 con 0x7f66440a5d50 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:57.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 10 monclient(hunting): dump: 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.786 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3453737699 0 0) 0x7f6644138e10 con 0x7f66440a5d50 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.104:6792/0 conn(0x7f664406e7b0 legacy=0x7f66441374a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.104:6790/0 conn(0x7f66440a5920 legacy=0x7f6644137bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 1 monclient: found mon.d 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f664413a8d0 con 0x7f66440a5d50 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 10 monclient: _finish_auth 0 2024-08-19T12:40:57.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:40:57.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:57.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f664b79f640 5 monclient: authenticate success, global_id 25827 2024-08-19T12:40:57.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f664b79f640 10 monclient: _renew_subs 2024-08-19T12:40:57.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f664b79f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:57.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f664b79f640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f664413b590 con 0x7f66440a5d50 2024-08-19T12:40:57.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f663c005470 con 0x7f66440a5d50 2024-08-19T12:40:57.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:57.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f663c005960 con 0x7f66440a5d50 2024-08-19T12:40:57.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f6639ffb640 10 monclient: dump: 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.789 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f663b7fe640 4 set_mon_vals no callback set 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f664b79f640 10 client.25827.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f664b79f640 10 monclient: _renew_subs 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f664b79f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f664b79f640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f66441428c0 con 0x7f66440a5d50 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.773+0000 7f664b79f640 1 librados: init done 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.774+0000 7f664b79f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.774+0000 7f664b79f640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f66440008d0 con 0x7f664402e340 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.774+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f663c011dc0 con 0x7f66440a5d50 2024-08-19T12:40:57.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.774+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(81..81 src has 1..81) ==== 6713+0+0 (unknown 3862053268 0 0) 0x7f663c04eaf0 con 0x7f66440a5d50 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.774+0000 7f6639ffb640 10 client.25827.objecter ms_dispatch 0x7f6644000b90 osd_map(81..81 src has 1..81) 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.774+0000 7f6639ffb640 3 client.25827.objecter handle_osd_map got epochs [81,81] > 0 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.774+0000 7f6639ffb640 3 client.25827.objecter handle_osd_map decoding full epoch 81 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.774+0000 7f6639ffb640 20 client.25827.objecter dump_active .. 0 homeless 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.774+0000 7f6648d3a640 10 monclient: get_auth_request con 0x7f6628041290 auth_method 0 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.776+0000 7f664953b640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.104:6791/0 conn(0x7f664402e340 legacy=0x7f6644063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.776+0000 7f6639ffb640 10 client.25827.objecter ms_handle_connect 0x7f664402e340 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.776+0000 7f6639ffb640 10 client.25827.objecter resend_mon_ops 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.776+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2610071966 0 0) 0x7f66440008d0 con 0x7f664402e340 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.776+0000 7f6639ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.776+0000 7f6639ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.776+0000 7f6639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.776+0000 7f6639ffb640 10 monclient(hunting): my global_id is 25556 2024-08-19T12:40:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.776+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6644138e10 con 0x7f664402e340 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4153969039 0 0) 0x7f6644138e10 con 0x7f664402e340 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66440008d0 con 0x7f664402e340 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6640004430 con 0x7f664402e340 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 10 monclient: dump: 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.792 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 10 client.25827.objecter ms_handle_connect 0x7f6628041290 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1214412863 0 0) 0x7f66440008d0 con 0x7f664402e340 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6644134620 con 0x7f664402e340 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6640006730 con 0x7f664402e340 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:57.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.104:6791/0 conn(0x7f664402e340 legacy=0x7f6644063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.777+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.104:6791/0 conn(0x7f664402e340 legacy=0x7f6644063a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:57.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.783+0000 7f664b79f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:57.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.783+0000 7f664b79f640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6644064190 con 0x7f66440a18d0 2024-08-19T12:40:57.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f664953b640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.104:6791/0 conn(0x7f66440a18d0 legacy=0x7f66441db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 10 client.25827.objecter ms_handle_connect 0x7f66440a18d0 2024-08-19T12:40:57.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 10 client.25827.objecter resend_mon_ops 2024-08-19T12:40:57.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1720829097 0 0) 0x7f6644064190 con 0x7f66440a18d0 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 10 monclient(hunting): my global_id is 25565 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66440008d0 con 0x7f66440a18d0 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3520306797 0 0) 0x7f66440008d0 con 0x7f66440a18d0 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6644064190 con 0x7f66440a18d0 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6640005fa0 con 0x7f66440a18d0 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 10 monclient: dump: 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.796 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.797 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3627391011 0 0) 0x7f6644064190 con 0x7f66440a18d0 2024-08-19T12:40:57.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.786+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f664402e720 con 0x7f66440a18d0 2024-08-19T12:40:57.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 1697072213) 0x7f6640006240 con 0x7f66440a18d0 2024-08-19T12:40:57.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f6639ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:57.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f6639ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:57.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.104:6791/0 conn(0x7f66440a18d0 legacy=0x7f66441db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f6639ffb640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.104:6791/0 conn(0x7f66440a18d0 legacy=0x7f66441db9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:57.798 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:57.798 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:57.798 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 14, 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 170809, 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.799 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.800 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.801 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:57.802 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.803 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:57.805 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.806 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.807 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:57.808 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.809 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.810 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:57.810 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.810 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.810 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:57.810 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:57.810 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.810 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.810 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.811 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:57.812 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.813 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:57.815 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.816 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:57.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.817 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.817 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.817 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:40:57.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.817 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.817 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:57.817 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:57.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 10 librados: watch_flush enter 2024-08-19T12:40:57.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 10 librados: watch_flush exit 2024-08-19T12:40:57.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 20 client.25827.objecter shutdown clearing up homeless session... 2024-08-19T12:40:57.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 10 client.25827.objecter successfully canceled tick 2024-08-19T12:40:57.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.104:6800/899887506 conn(0x7f6628041290 legacy=0x7f6628043750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 10 monclient: shutdown 2024-08-19T12:40:57.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:57.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 1 -- 172.21.15.26:0/3312184795 >> v1:172.21.15.26:6792/0 conn(0x7f66440a5d50 legacy=0x7f66441d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f6649d3c640 1 -- 172.21.15.26:0/3312184795 reap_dead start 2024-08-19T12:40:57.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f6649d3c640 1 -- 172.21.15.26:0/3312184795 reap_dead start 2024-08-19T12:40:57.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 1 -- 172.21.15.26:0/3312184795 shutdown_connections 2024-08-19T12:40:57.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 1 -- 172.21.15.26:0/3312184795 >> 172.21.15.26:0/3312184795 conn(0x7f66440a8740 msgr2=0x7f66440aa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:57.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 1 -- 172.21.15.26:0/3312184795 shutdown_connections 2024-08-19T12:40:57.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 1 -- 172.21.15.26:0/3312184795 wait complete. 2024-08-19T12:40:57.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.787+0000 7f664b79f640 1 librados: shutdown 2024-08-19T12:40:57.820 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:40:57.820 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-08-19T12:40:57.821 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:40:57.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.920+0000 7fc2d8fcf640 10 monclient: get_monmap_and_config 2024-08-19T12:40:57.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.920+0000 7fc2d8fcf640 10 monclient: build_initial_monmap 2024-08-19T12:40:57.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.920+0000 7fc2d8fcf640 10 monclient: monmap: 2024-08-19T12:40:57.921 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:57.921 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.922 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:40:57.921248+0000 2024-08-19T12:40:57.922 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:40:57.921248+0000 2024-08-19T12:40:57.922 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:57.922 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.922 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:57.922 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:57.922 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:57.922 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:57.922 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:57.922 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:57.922 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:57.922 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:57.923 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:57.923 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.920+0000 7fc2d8fcf640 1 Processor -- start 2024-08-19T12:40:57.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d8fcf640 1 -- start start 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d8fcf640 10 monclient: init 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d8fcf640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d8fcf640 10 monclient: _add_conns ranks=[4,8,6,7,3,2,0,5,1] 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d8fcf640 10 monclient(hunting): picked mon.noname-c con 0x7fc2d406e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d8fcf640 10 monclient(hunting): picked mon.noname-e con 0x7fc2d40a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d8fcf640 10 monclient(hunting): picked mon.noname-d con 0x7fc2d40a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d8fcf640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d4136790 con 0x7fc2d406e7b0 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d8fcf640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d409ddf0 con 0x7fc2d40a5d50 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d8fcf640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d409e0d0 con 0x7fc2d40a5920 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d8fcf640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d8fcf640 10 monclient(hunting): authenticate will time out at 852.008718s 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d259c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fc2d406e7b0 0x7fc2d4136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:55388/0 (socket says 172.21.15.26:55388) 2024-08-19T12:40:57.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d259c640 1 -- 172.21.15.26:0/3144401358 learned_addr learned my addr 172.21.15.26:0/3144401358 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d2d9d640 1 --1- 172.21.15.26:0/3144401358 >> v1:172.21.15.26:6792/0 conn(0x7fc2d40a5d50 0x7fc2d4141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:44200/0 (socket says 172.21.15.26:44200) 2024-08-19T12:40:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.921+0000 7fc2d1d9b640 1 --1- 172.21.15.26:0/3144401358 >> v1:172.21.15.26:6793/0 conn(0x7fc2d40a5920 0x7fc2d413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:45580/0 (socket says 172.21.15.26:45580) 2024-08-19T12:40:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d259c640 1 -- 172.21.15.26:0/3144401358 >> v1:172.21.15.26:6791/0 conn(0x7fc2d406e7b0 legacy=0x7fc2d4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d2d9d640 1 -- 172.21.15.26:0/3144401358 >> v1:172.21.15.26:6792/0 conn(0x7fc2d40a5d50 legacy=0x7fc2d4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d1d9b640 1 -- 172.21.15.26:0/3144401358 >> v1:172.21.15.26:6793/0 conn(0x7fc2d40a5920 legacy=0x7fc2d413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3978611862 0 0) 0x7fc2d4136790 con 0x7fc2d406e7b0 2024-08-19T12:40:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient(hunting): my global_id is 25789 2024-08-19T12:40:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2c0003520 con 0x7fc2d406e7b0 2024-08-19T12:40:57.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1317542158 0 0) 0x7fc2c4001380 con 0x7fc2d40a5d50 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient(hunting): my global_id is 25836 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d4136790 con 0x7fc2d40a5d50 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763776901 0 0) 0x7fc2d409e0d0 con 0x7fc2d40a5920 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient(hunting): my global_id is 25415 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2c4001380 con 0x7fc2d40a5920 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d2d9d640 1 --1- 172.21.15.26:0/3144401358 >> v1:172.21.15.26:6792/0 conn(0x7fc2d40a5d50 0x7fc2d4141a20 :-1 s=OPENED pgs=3044 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fc2c4001b40 mon_map magic: 0 2024-08-19T12:40:57.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3083089703 0 0) 0x7fc2c0003520 con 0x7fc2d406e7b0 2024-08-19T12:40:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 >> v1:172.21.15.26:6793/0 conn(0x7fc2d40a5920 legacy=0x7fc2d413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 >> v1:172.21.15.26:6792/0 conn(0x7fc2d40a5d50 legacy=0x7fc2d4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient: _finish_hunting 0 2024-08-19T12:40:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 monclient: found mon.noname-c 2024-08-19T12:40:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:40:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d409e3b0 con 0x7fc2d406e7b0 2024-08-19T12:40:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient: _finish_auth 0 2024-08-19T12:40:57.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient: _check_auth_tickets 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc2cc001e60 con 0x7fc2d406e7b0 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d8fcf640 5 monclient: authenticate success, global_id 25789 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d8fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient: dump: 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.933 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.934 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d8fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:40:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc2c4001b40 con 0x7fc2d40a5d50 2024-08-19T12:40:57.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc2cc004c20 con 0x7fc2d406e7b0 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 1 -- 172.21.15.26:0/3144401358 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc2cc005190 con 0x7fc2d406e7b0 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d8fcf640 10 monclient: get_monmap_and_config success 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d8fcf640 4 set_mon_vals no callback set 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.924+0000 7fc2d159a640 10 monclient: dump: 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.935 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.925+0000 7fc2d8fcf640 10 monclient: shutdown 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.925+0000 7fc2d8fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.925+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3144401358 >> v1:172.21.15.26:6791/0 conn(0x7fc2d406e7b0 legacy=0x7fc2d4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.925+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3144401358 shutdown_connections 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.925+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3144401358 >> 172.21.15.26:0/3144401358 conn(0x7fc2d40a8740 msgr2=0x7fc2d40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.925+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3144401358 shutdown_connections 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.925+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3144401358 wait complete. 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 10 monclient: build_initial_monmap 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 10 monclient: monmap: 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:40:57.937 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 1 librados: starting msgr at 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 1 librados: starting objecter 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 1 Processor -- start 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 1 -- start start 2024-08-19T12:40:57.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 1 librados: setting wanted keys 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 1 librados: calling monclient init 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 10 monclient: init 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 10 monclient: _reopen_session rank -1 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 10 monclient: _add_conns ranks=[6,2,5,8,7,4,1,0,3] 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 10 monclient(hunting): picked mon.noname-d con 0x7fc2d406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 10 monclient(hunting): picked mon.noname-b con 0x7fc2d40a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 10 monclient(hunting): picked mon.noname-h con 0x7fc2d40a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d413b390 con 0x7fc2d40a5920 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d4139030 con 0x7fc2d40a5d50 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d413aaf0 con 0x7fc2d406e7b0 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 10 monclient(hunting): _renew_subs 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d8fcf640 10 monclient(hunting): authenticate will time out at 852.013882s 2024-08-19T12:40:57.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d259c640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fc2d406e7b0 0x7fc2d4137950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:44204/0 (socket says 172.21.15.26:44204) 2024-08-19T12:40:57.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.926+0000 7fc2d259c640 1 -- 172.21.15.26:0/3577786788 learned_addr learned my addr 172.21.15.26:0/3577786788 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:40:57.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d259c640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.26:6792/0 conn(0x7fc2d406e7b0 legacy=0x7fc2d4137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 client.?.objecter ms_handle_connect 0x7fc2d406e7b0 2024-08-19T12:40:57.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 client.?.objecter ms_handle_connect 0x7fc2d40a5920 2024-08-19T12:40:57.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d1d9b640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.26:6790/0 conn(0x7fc2d40a5920 legacy=0x7fc2d4138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2975738561 0 0) 0x7fc2d413aaf0 con 0x7fc2d406e7b0 2024-08-19T12:40:57.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): my global_id is 25845 2024-08-19T12:40:57.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2b4004eb0 con 0x7fc2d406e7b0 2024-08-19T12:40:57.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2822000793 0 0) 0x7fc2d413b390 con 0x7fc2d40a5920 2024-08-19T12:40:57.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d2d9d640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.104:6791/0 conn(0x7fc2d40a5d50 legacy=0x7fc2d41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): my global_id is 25607 2024-08-19T12:40:57.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d413aaf0 con 0x7fc2d40a5920 2024-08-19T12:40:57.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 client.?.objecter ms_handle_connect 0x7fc2d40a5d50 2024-08-19T12:40:57.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:40:57.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2352180615 0 0) 0x7fc2d4139030 con 0x7fc2d40a5d50 2024-08-19T12:40:57.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): my global_id is 25574 2024-08-19T12:40:57.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d413b390 con 0x7fc2d40a5d50 2024-08-19T12:40:57.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3568681528 0 0) 0x7fc2b4004eb0 con 0x7fc2d406e7b0 2024-08-19T12:40:57.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2d4139030 con 0x7fc2d406e7b0 2024-08-19T12:40:57.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc2cc005110 con 0x7fc2d406e7b0 2024-08-19T12:40:57.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:57.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:40:57.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): dump: 2024-08-19T12:40:57.949 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.949 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.949 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.949 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:40:57.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4019756214 0 0) 0x7fc2d413aaf0 con 0x7fc2d40a5920 2024-08-19T12:40:57.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2b4004eb0 con 0x7fc2d40a5920 2024-08-19T12:40:57.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc2bc001320 con 0x7fc2d40a5920 2024-08-19T12:40:57.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:57.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:40:57.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): dump: 2024-08-19T12:40:57.951 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.951 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.951 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.951 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.951 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.951 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 843709485 0 0) 0x7fc2d413b390 con 0x7fc2d40a5d50 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2d413aaf0 con 0x7fc2d40a5d50 2024-08-19T12:40:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc2c4001940 con 0x7fc2d40a5d50 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient(hunting): dump: 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.953 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.954 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.954 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.954 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.954 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.954 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1357656390 0 0) 0x7fc2d4139030 con 0x7fc2d406e7b0 2024-08-19T12:40:57.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.104:6791/0 conn(0x7fc2d40a5d50 legacy=0x7fc2d41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.26:6790/0 conn(0x7fc2d40a5920 legacy=0x7fc2d4138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient: _finish_hunting 0 2024-08-19T12:40:57.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 monclient: found mon.d 2024-08-19T12:40:57.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:40:57.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d413add0 con 0x7fc2d406e7b0 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient: _finish_auth 0 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient: _check_auth_tickets 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d8fcf640 5 monclient: authenticate success, global_id 25845 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d8fcf640 10 monclient: _renew_subs 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d8fcf640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d4139b90 con 0x7fc2d406e7b0 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d8fcf640 10 client.25845.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d8fcf640 10 monclient: _renew_subs 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d8fcf640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fc2d4138770 con 0x7fc2d406e7b0 2024-08-19T12:40:57.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d8fcf640 1 librados: init done 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc2cc0020b0 con 0x7fc2d406e7b0 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc2cc004c20 con 0x7fc2d406e7b0 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2d0d99640 4 set_mon_vals no callback set 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.929+0000 7fc2caffd640 10 monclient: dump: 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.956 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.930+0000 7fc2d8fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.930+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc2b80065c0 con 0x7fc2b8002860 2024-08-19T12:40:57.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.930+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fc2cc011d80 con 0x7fc2d406e7b0 2024-08-19T12:40:57.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.931+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(81..81 src has 1..81) ==== 6713+0+0 (unknown 3862053268 0 0) 0x7fc2cc04ea90 con 0x7fc2d406e7b0 2024-08-19T12:40:57.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.931+0000 7fc2caffd640 10 client.25845.objecter ms_dispatch 0x7fc2d4000b90 osd_map(81..81 src has 1..81) 2024-08-19T12:40:57.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.931+0000 7fc2caffd640 3 client.25845.objecter handle_osd_map got epochs [81,81] > 0 2024-08-19T12:40:57.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.931+0000 7fc2caffd640 3 client.25845.objecter handle_osd_map decoding full epoch 81 2024-08-19T12:40:57.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.931+0000 7fc2caffd640 20 client.25845.objecter dump_active .. 0 homeless 2024-08-19T12:40:57.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.931+0000 7fc2d2d9d640 10 monclient: get_auth_request con 0x7fc2b4041c40 auth_method 0 2024-08-19T12:40:57.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 10 client.25845.objecter ms_handle_connect 0x7fc2b8002860 2024-08-19T12:40:57.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 10 client.25845.objecter resend_mon_ops 2024-08-19T12:40:57.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2d1d9b640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.104:6792/0 conn(0x7fc2b8002860 legacy=0x7fc2b8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1900147772 0 0) 0x7fc2b80065c0 con 0x7fc2b8002860 2024-08-19T12:40:57.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 10 monclient(hunting): my global_id is 134341 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d4139030 con 0x7fc2b8002860 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 10 client.25845.objecter ms_handle_connect 0x7fc2b4041c40 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3962351946 0 0) 0x7fc2d4139030 con 0x7fc2b8002860 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2b80065c0 con 0x7fc2b8002860 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc2bc005110 con 0x7fc2b8002860 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 10 monclient: dump: 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.959 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3011216931 0 0) 0x7fc2b80065c0 con 0x7fc2b8002860 2024-08-19T12:40:57.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2b800a160 con 0x7fc2b8002860 2024-08-19T12:40:57.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.933+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc2bc007410 con 0x7fc2b8002860 2024-08-19T12:40:57.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.934+0000 7fc2caffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:40:57.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.934+0000 7fc2caffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:40:57.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.934+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.104:6792/0 conn(0x7fc2b8002860 legacy=0x7fc2b8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.934+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.104:6792/0 conn(0x7fc2b8002860 legacy=0x7fc2b8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:57.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.940+0000 7fc2d8fcf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:40:57.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.940+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc2b8009e50 con 0x7fc2b8009110 2024-08-19T12:40:57.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.942+0000 7fc2d1d9b640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.104:6792/0 conn(0x7fc2b8009110 legacy=0x7fc2b80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:40:57.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.942+0000 7fc2caffd640 10 client.25845.objecter ms_handle_connect 0x7fc2b8009110 2024-08-19T12:40:57.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.942+0000 7fc2caffd640 10 client.25845.objecter resend_mon_ops 2024-08-19T12:40:57.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.942+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2610887089 0 0) 0x7fc2b8009e50 con 0x7fc2b8009110 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.942+0000 7fc2caffd640 10 monclient(hunting): _negotiate 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.942+0000 7fc2caffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.942+0000 7fc2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.942+0000 7fc2caffd640 10 monclient(hunting): my global_id is 134350 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.942+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2b80065c0 con 0x7fc2b8009110 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1259894573 0 0) 0x7fc2b80065c0 con 0x7fc2b8009110 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2b8009e50 con 0x7fc2b8009110 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc2bc005af0 con 0x7fc2b8009110 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 10 monclient: dump: 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:40:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:40:57.963 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:40:57.964 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:40:57.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2935414971 0 0) 0x7fc2b8009e50 con 0x7fc2b8009110 2024-08-19T12:40:57.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc2b800c3f0 con 0x7fc2b8009110 2024-08-19T12:40:57.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7532 (unknown 0 0 3365387378) 0x7fc2bc007060 con 0x7fc2b8009110 2024-08-19T12:40:57.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:40:57.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:40:57.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.104:6792/0 conn(0x7fc2b8009110 legacy=0x7fc2b80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2caffd640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.104:6792/0 conn(0x7fc2b8009110 legacy=0x7fc2b80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:40:57.964 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:40:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:40:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 14, 2024-08-19T12:40:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 28300, 2024-08-19T12:40:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:40:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:40:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:40:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:40:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:40:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:40:57.965 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:40:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:40:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:40:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:40:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:40:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:40:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.966 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.967 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:40:57.968 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:40:57.969 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:40:57.970 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:40:57.970 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:40:57.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:40:57.970 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:40:57.970 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:40:57.970 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:40:57.970 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:40:57.970 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:40:57.970 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:40:57.970 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:40:57.970 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.970 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.972 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.973 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:40:57.973 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.973 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.973 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.973 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:40:57.973 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:40:57.973 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.973 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.973 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.974 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:40:57.975 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:40:57.976 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.977 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.978 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:40:57.979 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.980 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.980 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.980 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:40:57.980 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:40:57.980 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.980 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.980 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.980 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:40:57.980 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.981 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.983 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:40:57.983 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:40:57.983 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:40:57.983 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:40:57.983 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:40:57.983 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.983 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.983 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:40:57.983 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:40:57.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.983 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:40:57.984 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:40:57.985 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:40:57.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2d8fcf640 10 librados: watch_flush enter 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2d8fcf640 10 librados: watch_flush exit 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2d8fcf640 20 client.25845.objecter shutdown clearing up homeless session... 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.943+0000 7fc2d8fcf640 10 client.25845.objecter successfully canceled tick 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.944+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.104:6800/899887506 conn(0x7fc2b4041c40 legacy=0x7fc2b4044100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.944+0000 7fc2d8fcf640 10 monclient: shutdown 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.944+0000 7fc2d8fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.944+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3577786788 >> v1:172.21.15.26:6792/0 conn(0x7fc2d406e7b0 legacy=0x7fc2d4137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.944+0000 7fc2d2d9d640 1 -- 172.21.15.26:0/3577786788 reap_dead start 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.944+0000 7fc2d2d9d640 1 -- 172.21.15.26:0/3577786788 reap_dead start 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.944+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3577786788 shutdown_connections 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.944+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3577786788 >> 172.21.15.26:0/3577786788 conn(0x7fc2d40a8740 msgr2=0x7fc2d406dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.944+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3577786788 shutdown_connections 2024-08-19T12:40:57.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.944+0000 7fc2d8fcf640 1 -- 172.21.15.26:0/3577786788 wait complete. 2024-08-19T12:40:57.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:40:57.944+0000 7fc2d8fcf640 1 librados: shutdown 2024-08-19T12:40:57.988 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:40:57.988 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T12:41:00.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-08-19T12:41:00.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-08-19T12:41:00.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-08-19T12:41:00.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-08-19T12:41:00.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-08-19T12:41:00.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-08-19T12:41:00.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: 2024-08-19T12:41:00.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-08-19T12:41:00.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: seed 24727 2024-08-19T12:41:00.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-08-19T12:41:00.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-08-19T12:41:00.048 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (322 ms) 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (9 ms) 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (6 ms) 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (9 ms) 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (1 ms) 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-08-19T12:41:00.049 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (4 ms) 2024-08-19T12:41:00.050 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-08-19T12:41:00.050 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (1 ms) 2024-08-19T12:41:00.050 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-08-19T12:41:00.050 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (8 ms) 2024-08-19T12:41:00.050 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-08-19T12:41:00.050 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (6 ms) 2024-08-19T12:41:00.050 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-08-19T12:41:00.050 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (4 ms) 2024-08-19T12:41:00.050 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-08-19T12:41:00.050 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (18 ms) 2024-08-19T12:41:00.050 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-08-19T12:41:00.050 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: osd_max_attr_size = 0 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1990 ms) 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (232 ms) 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: done waiting, doing copies 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: waiting for final deep scrubs... 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: done waiting 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60340 ms) 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (6 ms) 2024-08-19T12:41:00.051 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-08-19T12:41:00.052 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (4 ms) 2024-08-19T12:41:00.052 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-08-19T12:41:00.052 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (18042 ms) 2024-08-19T12:41:00.052 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-08-19T12:41:00.052 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-08-19T12:41:00.052 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-08-19T12:41:00.052 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-08-19T12:41:00.052 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-08-19T12:41:00.052 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-08-19T12:41:04.569 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (_list: : cursor()=9:62a1935d:::14:head expected=9:62a1935d:::14:head 2024-08-19T12:41:04.570 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:62a1935d:::14:head -> 14 2024-08-19T12:41:04.570 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=14 expected=14 2024-08-19T12:41:04.570 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2024-08-19T12:41:04.570 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:5c6b0b28:::7:head expected=9:5c6b0b28:::7:head 2024-08-19T12:41:04.570 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2024-08-19T12:41:04.570 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=7 expected=7 2024-08-19T12:41:04.570 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:566253c9:::13:head 2024-08-19T12:41:04.570 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:566253c9:::13:head expected=9:566253c9:::13:head 2024-08-19T12:41:04.570 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:566253c9:::13:head -> 13 2024-08-19T12:41:04.570 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=13 expected=13 2024-08-19T12:41:04.570 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:f905c69b:::2:head 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:f905c69b:::2:head expected=9:f905c69b:::2:head 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:f905c69b:::2:head -> 2 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=2 expected=2 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:e960b815:::9:head 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:e960b815:::9:head expected=9:e960b815:::9:head 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:e960b815:::9:head -> 9 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=9 expected=9 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:de5d7c5f:::12:head expected=9:de5d7c5f:::12:head 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=12 expected=12 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:d83876eb:::4:head 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:d83876eb:::4:head expected=9:d83876eb:::4:head 2024-08-19T12:41:04.571 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:d83876eb:::4:head -> 4 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=4 expected=4 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : seek to 9:02547ec2:::1:head 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : cursor()=9:02547ec2:::1:head expected=9:02547ec2:::1:head 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: > 9:02547ec2:::1:head -> 1 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: : entry=1 expected=1 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (200 ms) 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (56129 ms) 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: split 0/5 -> MIN 9:33333333::::head 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: split 1/5 -> 9:33333333::::head 9:66666666::::head 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: split 2/5 -> 9:66666666::::head 9:99999999::::head 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: split 3/5 -> 9:99999999::::head 9:cccccccc::::head 2024-08-19T12:41:04.572 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: split 4/5 -> 9:cccccccc::::head MAX 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (24219 ms) 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [----------] 7 tests from LibRadosList (81099 ms total) 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1056 ms) 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: myset foo1,foo2,foo3 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo1 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo2 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo3 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: myset foo1,foo4,foo5 2024-08-19T12:41:04.573 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo4 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo5 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo1 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: myset foo6,foo7 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo7 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: foo6 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: ns1:foo4 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: ns1:foo5 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: ns2:foo7 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: ns2:foo6 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: ns1:foo1 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: :foo1 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: :foo2 2024-08-19T12:41:04.574 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: :foo3 2024-08-19T12:41:04.575 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (36 ms) 2024-08-19T12:41:04.575 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-08-19T12:41:04.575 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 1 0 2024-08-19T12:41:04.575 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 10 0 2024-08-19T12:41:04.575 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 13 0 2024-08-19T12:41:05.471 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: api_snapshots_pp: Running main() from gmock_main.cc 2024-08-19T12:41:05.471 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-08-19T12:41:05.471 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-08-19T12:41:05.471 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-08-19T12:41:05.471 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: seed 24837 2024-08-19T12:41:05.472 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-08-19T12:41:05.472 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2075 ms) 2024-08-19T12:41:05.472 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-08-19T12:41:05.472 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2074 ms) 2024-08-19T12:41:05.472 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-08-19T12:41:05.472 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2012 ms) 2024-08-19T12:41:05.472 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-08-19T12:41:05.472 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2020 ms) 2024-08-19T12:41:05.472 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-08-19T12:41:05.472 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3835 ms) 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (12016 ms total) 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4100 ms) 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4627 ms) 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5292 ms) 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4068 ms) 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (1971 ms) 2024-08-19T12:41:05.473 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-08-19T12:41:05.474 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5847-g79766939/rpm/el9/BUILD/ceph-19.0.0-5847-g79766939/src/test/librados/snapshots_cxx.cc:460: Skipped 2024-08-19T12:41:05.474 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: 2024-08-19T12:41:05.474 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-08-19T12:41:05.474 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-08-19T12:41:05.474 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi104-24837-7 2024-08-19T12:41:05.474 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: waiting for snaps to purge 2024-08-19T12:41:05.474 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18043 ms) 2024-08-19T12:41:05.474 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (38101 ms total) 2024-08-19T12:41:05.474 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: 2024-08-19T12:41:05.474 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-08-19T12:41:05.474 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-08-19T12:41:05.474 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (14 ms) 2024-08-19T12:41:05.475 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-08-19T12:41:05.475 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (24321 ms) 2024-08-19T12:41:05.475 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (24335 ms total) 2024-08-19T12:41:05.475 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: 2024-08-19T12:41:05.475 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-08-19T12:41:05.475 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-08-19T12:41:05.475 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3407 ms) 2024-08-19T12:41:05.475 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-08-19T12:41:10.173 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSn7 0 2024-08-19T12:41:10.174 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 14 0 2024-08-19T12:41:10.174 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 0 0 2024-08-19T12:41:10.174 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 15 0 2024-08-19T12:41:10.174 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 11 0 2024-08-19T12:41:10.174 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 5 0 2024-08-19T12:41:10.174 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 8 0 2024-08-19T12:41:10.174 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 6 0 2024-08-19T12:41:10.174 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 3 0 2024-08-19T12:41:10.174 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 4 0 2024-08-19T12:41:10.174 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 12 0 2024-08-19T12:41:10.174 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 9 0 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 2 0 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (52 ms) 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [----------] 3 tests from LibRadosListEC (1144 ms total) 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3016 ms) 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [----------] 1 test from LibRadosListNP (3016 ms total) 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [----------] Global test environment tear-down 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [==========] 11 tests from 3 test suites ran. (94588 ms total) 2024-08-19T12:41:10.175 INFO:tasks.workunit.client.0.smithi104.stdout: api_list: [ PASSED ] 11 tests. 2024-08-19T12:41:11.182 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: Running main() from gmock_main.cc 2024-08-19T12:41:11.182 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-08-19T12:41:11.183 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [----------] Global test environment set-up. 2024-08-19T12:41:11.183 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-08-19T12:41:11.183 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-08-19T12:41:11.183 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (5186 ms) 2024-08-19T12:41:11.183 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-08-19T12:41:11.183 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5089 ms) 2024-08-19T12:41:11.183 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-08-19T12:41:11.183 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (3831 ms) 2024-08-19T12:41:11.183 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-08-19T12:41:11.183 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (4994 ms) 2024-08-19T12:41:11.183 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-08-19T12:41:11.183 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7008 ms) 2024-08-19T12:41:11.184 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (26108 ms total) 2024-08-19T12:41:11.184 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: 2024-08-19T12:41:11.184 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-08-19T12:41:11.184 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-08-19T12:41:11.184 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5446 ms) 2024-08-19T12:41:11.184 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-08-19T12:41:11.184 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6534 ms) 2024-08-19T12:41:11.184 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-08-19T12:41:11.185 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7968 ms) 2024-08-19T12:41:11.185 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-08-19T12:41:11.185 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6007 ms) 2024-08-19T12:41:11.185 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-08-19T12:41:11.185 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4786 ms) 2024-08-19T12:41:11.185 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-08-19T12:41:11.186 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi104-25862-11. 2024-08-19T12:41:11.186 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: Waiting for snaps to purge. 2024-08-19T12:41:11.186 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (38375 ms) 2024-08-19T12:41:11.186 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (69116 ms total) 2024-08-19T12:41:11.186 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: 2024-08-19T12:41:11.186 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [----------] Global test environment tear-down 2024-08-19T12:41:11.187 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (95224 ms total) 2024-08-19T12:41:11.187 INFO:tasks.workunit.client.0.smithi104.stdout: snapshots: [ PASSED ] 11 tests. 2024-08-19T12:41:27.044 INFO:tasks.workunit.client.0.smithi104.stdout:81005 ms total) 2024-08-19T12:41:27.044 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: 2024-08-19T12:41:27.044 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-08-19T12:41:27.044 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-08-19T12:41:27.044 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (109 ms) 2024-08-19T12:41:27.044 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (109 ms total) 2024-08-19T12:41:27.044 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: 2024-08-19T12:41:27.044 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (29 ms) 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (7 ms) 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (37 ms total) 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (33 ms) 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (6 ms) 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (40 ms total) 2024-08-19T12:41:27.045 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (19 ms) 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (7 ms) 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (26 ms total) 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1022 ms) 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1022 ms total) 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-08-19T12:41:27.046 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (111412 ms total) 2024-08-19T12:41:27.047 INFO:tasks.workunit.client.0.smithi104.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-08-19T12:41:31.703 INFO:tasks.workunit.client.0.smithi104.stdout:apshotsECPP.SnapRemovePP (2017 ms) 2024-08-19T12:41:31.703 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-08-19T12:41:31.704 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (1672 ms) 2024-08-19T12:41:31.704 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-08-19T12:41:31.704 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2009 ms) 2024-08-19T12:41:31.704 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9105 ms total) 2024-08-19T12:41:31.704 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: 2024-08-19T12:41:31.704 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-08-19T12:41:31.704 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-08-19T12:41:31.704 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (3553 ms) 2024-08-19T12:41:31.704 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-08-19T12:41:31.704 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4820 ms) 2024-08-19T12:41:31.704 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-08-19T12:41:31.704 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4752 ms) 2024-08-19T12:41:31.705 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (13125 ms total) 2024-08-19T12:41:31.705 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: 2024-08-19T12:41:31.705 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-08-19T12:41:31.705 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (116070 ms total) 2024-08-19T12:41:31.705 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-08-19T12:41:31.705 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-08-19T12:41:31.705 INFO:tasks.workunit.client.0.smithi104.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-08-19T12:41:38.888 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-08-19T12:41:38.888 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-08-19T12:41:38.889 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-08-19T12:41:38.889 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-08-19T12:41:38.889 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-08-19T12:41:38.889 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (3621 ms) 2024-08-19T12:41:38.889 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-08-19T12:41:38.889 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (20455 ms) 2024-08-19T12:41:38.889 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-08-19T12:41:38.889 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (5539 ms) 2024-08-19T12:41:38.889 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-08-19T12:41:38.889 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2924 ms) 2024-08-19T12:41:38.889 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-08-19T12:41:38.889 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3025 ms) 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2955 ms) 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2963 ms) 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3094 ms) 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2913 ms) 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3077 ms) 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2946 ms) 2024-08-19T12:41:38.890 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3755 ms) 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (17272 ms) 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (7996 ms) 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3009 ms) 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3395 ms) 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2687 ms) 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-08-19T12:41:38.891 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3024 ms) 2024-08-19T12:41:38.892 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-08-19T12:41:38.892 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3004 ms) 2024-08-19T12:41:38.892 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-08-19T12:41:38.892 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3395 ms) 2024-08-19T12:41:38.892 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-08-19T12:41:38.892 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2519 ms) 2024-08-19T12:41:38.892 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-08-19T12:41:38.892 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3834 ms) 2024-08-19T12:41:38.892 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-08-19T12:41:38.892 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3114 ms) 2024-08-19T12:41:38.893 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-08-19T12:41:38.893 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3660 ms) 2024-08-19T12:41:38.893 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-08-19T12:41:38.893 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2568 ms) 2024-08-19T12:41:38.893 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-08-19T12:41:38.893 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3618 ms) 2024-08-19T12:41:38.893 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-08-19T12:41:46.662 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAi api_aio: Running main() from gmock_main.cc 2024-08-19T12:41:46.662 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-08-19T12:41:46.663 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [----------] Global test environment set-up. 2024-08-19T12:41:46.663 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-08-19T12:41:46.663 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-08-19T12:41:46.663 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.TooBig (3627 ms) 2024-08-19T12:41:46.663 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-08-19T12:41:46.663 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3059 ms) 2024-08-19T12:41:46.663 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-08-19T12:41:46.663 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3017 ms) 2024-08-19T12:41:46.663 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-08-19T12:41:46.663 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2788 ms) 2024-08-19T12:41:46.663 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-08-19T12:41:46.663 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3052 ms) 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (2988 ms) 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3031 ms) 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3725 ms) 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2258 ms) 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (53984 ms) 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3005 ms) 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-08-19T12:41:46.664 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3411 ms) 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3010 ms) 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2686 ms) 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.Flush (3017 ms) 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3389 ms) 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2526 ms) 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3042 ms) 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-08-19T12:41:46.665 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3803 ms) 2024-08-19T12:41:46.666 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-08-19T12:41:46.666 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3751 ms) 2024-08-19T12:41:46.666 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-08-19T12:41:46.666 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3021 ms) 2024-08-19T12:41:46.666 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-08-19T12:41:46.666 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3170 ms) 2024-08-19T12:41:46.666 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-08-19T12:41:46.666 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3271 ms) 2024-08-19T12:41:46.666 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-08-19T12:41:46.666 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2751 ms) 2024-08-19T12:41:46.666 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-08-19T12:41:46.666 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2146 ms) 2024-08-19T12:41:46.667 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-08-19T12:41:46.667 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3617 ms) 2024-08-19T12:41:46.667 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [----------] 26 tests from LibRadosAio (131145 ms total) 2024-08-19T12:41:46.667 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: 2024-08-19T12:41:46.667 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-08-19T12:42:27.989 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-08-19T12:42:27.989 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-19T12:42:27.990 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-19T12:42:27.994 INFO:tasks.ceph.mon.a:Started 2024-08-19T12:42:27.994 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-08-19T12:42:27.994 INFO:tasks.ceph.mon.c:Sent signal 19 2024-08-19T12:42:27.994 INFO:tasks.mon_thrash:Sending STOP to mon h 2024-08-19T12:42:27.994 INFO:tasks.ceph.mon.h:Sent signal 19 2024-08-19T12:42:27.995 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:42:42.995 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-08-19T12:42:42.996 INFO:tasks.ceph.mon.c:Sent signal 18 2024-08-19T12:42:42.997 INFO:tasks.mon_thrash:Sending CONT to mon h 2024-08-19T12:42:42.997 INFO:tasks.ceph.mon.h:Sent signal 18 2024-08-19T12:42:42.997 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:42:42.997 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:42:43.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.113+0000 7fcbfc3f2640 10 monclient: get_monmap_and_config 2024-08-19T12:42:43.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.113+0000 7fcbfc3f2640 10 monclient: build_initial_monmap 2024-08-19T12:42:43.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.113+0000 7fcbfc3f2640 10 monclient: monmap: 2024-08-19T12:42:43.115 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:43.115 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:43.115 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:43.115224+0000 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:43.115224+0000 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:43.116 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.113+0000 7fcbfc3f2640 1 Processor -- start 2024-08-19T12:42:43.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfc3f2640 1 -- start start 2024-08-19T12:42:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfc3f2640 10 monclient: init 2024-08-19T12:42:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfc3f2640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfc3f2640 10 monclient: _add_conns ranks=[8,0,6,7,2,1,4,3,5] 2024-08-19T12:42:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfc3f2640 10 monclient(hunting): picked mon.noname-e con 0x7fcbf406e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfc3f2640 10 monclient(hunting): picked mon.noname-a con 0x7fcbf40a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfc3f2640 10 monclient(hunting): picked mon.noname-d con 0x7fcbf40a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfc3f2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf4136790 con 0x7fcbf40a5920 2024-08-19T12:42:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfc3f2640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf409ddf0 con 0x7fcbf40a5d50 2024-08-19T12:42:43.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfc3f2640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf409e0d0 con 0x7fcbf406e7b0 2024-08-19T12:42:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfc3f2640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfc3f2640 10 monclient(hunting): authenticate will time out at 957.202861s 2024-08-19T12:42:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfa18e640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fcbf406e7b0 0x7fcbf4136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:60034/0 (socket says 172.21.15.26:60034) 2024-08-19T12:42:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.114+0000 7fcbfa18e640 1 -- 172.21.15.26:0/2533696766 learned_addr learned my addr 172.21.15.26:0/2533696766 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.117+0000 7fcbfa18e640 1 -- 172.21.15.26:0/2533696766 >> v1:172.21.15.26:6793/0 conn(0x7fcbf406e7b0 legacy=0x7fcbf4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.117+0000 7fcbfa98f640 1 -- 172.21.15.26:0/2533696766 >> v1:172.21.15.26:6792/0 conn(0x7fcbf40a5d50 legacy=0x7fcbf4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:43.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:43.117+0000 7fcbf998d640 1 -- 172.21.15.26:0/2533696766 >> v1:172.21.15.26:6789/0 conn(0x7fcbf40a5920 legacy=0x7fcbf413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:46.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 10 monclient(hunting): tick 2024-08-19T12:42:46.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:42:46.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 1 monclient(hunting): continuing hunt 2024-08-19T12:42:46.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:42:46.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 1 -- 172.21.15.26:0/2533696766 >> v1:172.21.15.26:6793/0 conn(0x7fcbf406e7b0 legacy=0x7fcbf4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:46.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 1 -- 172.21.15.26:0/2533696766 >> v1:172.21.15.26:6792/0 conn(0x7fcbf40a5d50 legacy=0x7fcbf4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:46.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 1 -- 172.21.15.26:0/2533696766 >> v1:172.21.15.26:6789/0 conn(0x7fcbf40a5920 legacy=0x7fcbf413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:46.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 10 monclient: _add_conns ranks=[3,1,7,2,5,4] 2024-08-19T12:42:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 10 monclient(hunting): picked mon.noname-g con 0x7fcbd8003960 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 10 monclient(hunting): picked mon.noname-f con 0x7fcbd80075f0 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 10 monclient(hunting): picked mon.noname-i con 0x7fcbd800b380 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 1 -- 172.21.15.26:0/2533696766 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf409e3b0 con 0x7fcbd80075f0 2024-08-19T12:42:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 1 -- 172.21.15.26:0/2533696766 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbd80101a0 con 0x7fcbd8003960 2024-08-19T12:42:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 1 -- 172.21.15.26:0/2533696766 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbd8011420 con 0x7fcbd800b380 2024-08-19T12:42:46.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.115+0000 7fcbebfff640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:46.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.117+0000 7fcbf998d640 1 -- 172.21.15.26:0/2533696766 >> v1:172.21.15.104:6789/0 conn(0x7fcbd80075f0 legacy=0x7fcbd8009ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:46.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.117+0000 7fcbfa18e640 1 -- 172.21.15.26:0/2533696766 >> v1:172.21.15.104:6790/0 conn(0x7fcbd8003960 legacy=0x7fcbd8005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:46.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:46.117+0000 7fcbfa98f640 1 -- 172.21.15.26:0/2533696766 >> v1:172.21.15.104:6792/0 conn(0x7fcbd800b380 legacy=0x7fcbd800d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:48.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355274644 0 0) 0x7fcbf409e3b0 con 0x7fcbd80075f0 2024-08-19T12:42:48.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient(hunting): _negotiate 2024-08-19T12:42:48.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:48.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:48.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient(hunting): my global_id is 26569 2024-08-19T12:42:48.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbe0003520 con 0x7fcbd80075f0 2024-08-19T12:42:48.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2203950333 0 0) 0x7fcbd8011420 con 0x7fcbd800b380 2024-08-19T12:42:48.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient(hunting): _negotiate 2024-08-19T12:42:48.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:48.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:48.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient(hunting): my global_id is 134692 2024-08-19T12:42:48.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf409e3b0 con 0x7fcbd800b380 2024-08-19T12:42:48.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3300959478 0 0) 0x7fcbd80101a0 con 0x7fcbd8003960 2024-08-19T12:42:48.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient(hunting): _negotiate 2024-08-19T12:42:48.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:48.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:48.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient(hunting): my global_id is 26085 2024-08-19T12:42:48.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbd8011420 con 0x7fcbd8003960 2024-08-19T12:42:48.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 671891554 0 0) 0x7fcbe0003520 con 0x7fcbd80075f0 2024-08-19T12:42:48.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf998d640 1 --1- 172.21.15.26:0/2533696766 >> v1:172.21.15.104:6789/0 conn(0x7fcbd80075f0 0x7fcbd8009ab0 :-1 s=OPENED pgs=3273 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fcbe4001bb0 mon_map magic: 0 2024-08-19T12:42:48.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 >> v1:172.21.15.104:6792/0 conn(0x7fcbd800b380 legacy=0x7fcbd800d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:48.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 >> v1:172.21.15.104:6790/0 conn(0x7fcbd8003960 legacy=0x7fcbd8005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient: _finish_hunting 0 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 monclient: found mon.noname-f 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd8012780 con 0x7fcbd80075f0 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient: _finish_auth 0 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient: _check_auth_tickets 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbfa98f640 1 -- 172.21.15.26:0/2533696766 reap_dead start 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbfc3f2640 5 monclient: authenticate success, global_id 26569 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbfc3f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcbe4001bb0 con 0x7fcbd80075f0 2024-08-19T12:42:48.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:48.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:42:48.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient: dump: 2024-08-19T12:42:48.048 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:48.048 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:48.048 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:48.048 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:48.048 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:48.048 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:48.048 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:48.048 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:48.048 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:48.048 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:48.049 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:48.049 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:48.049 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:48.049 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:48.049 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:48.049 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:48.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcbe4003610 con 0x7fcbd80075f0 2024-08-19T12:42:48.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbfc3f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:48.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbf918c640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:48.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbfc3f2640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:48.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.042+0000 7fcbfc3f2640 4 set_mon_vals no callback set 2024-08-19T12:42:48.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.043+0000 7fcbfc3f2640 10 monclient: shutdown 2024-08-19T12:42:48.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.043+0000 7fcbfc3f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:48.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.043+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/2533696766 >> v1:172.21.15.104:6789/0 conn(0x7fcbd80075f0 legacy=0x7fcbd8009ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:48.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.043+0000 7fcbf918c640 1 -- 172.21.15.26:0/2533696766 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcbe4003b70 con 0x7fcbd80075f0 2024-08-19T12:42:48.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.043+0000 7fcbf918c640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:42:48.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.043+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/2533696766 shutdown_connections 2024-08-19T12:42:48.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.043+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/2533696766 >> 172.21.15.26:0/2533696766 conn(0x7fcbf40a8740 msgr2=0x7fcbf40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:48.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.043+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/2533696766 shutdown_connections 2024-08-19T12:42:48.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.043+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/2533696766 wait complete. 2024-08-19T12:42:48.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.044+0000 7fcbfc3f2640 10 monclient: build_initial_monmap 2024-08-19T12:42:48.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.044+0000 7fcbfc3f2640 10 monclient: monmap: 2024-08-19T12:42:48.053 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:48.054 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:48.054 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:48.054 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:48.054 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:48.054 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:48.054 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:48.054 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:48.054 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:48.054 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.044+0000 7fcbfc3f2640 1 librados: starting msgr at 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.044+0000 7fcbfc3f2640 1 librados: starting objecter 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.045+0000 7fcbfc3f2640 1 Processor -- start 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.045+0000 7fcbfc3f2640 1 -- start start 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.045+0000 7fcbfc3f2640 1 librados: setting wanted keys 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.045+0000 7fcbfc3f2640 1 librados: calling monclient init 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.045+0000 7fcbfc3f2640 10 monclient: init 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.045+0000 7fcbfc3f2640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:48.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.045+0000 7fcbfc3f2640 10 monclient: _add_conns ranks=[1,5,2,8,3,7,0,4,6] 2024-08-19T12:42:48.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.046+0000 7fcbfc3f2640 10 monclient(hunting): picked mon.noname-f con 0x7fcbd80075f0 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:48.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.046+0000 7fcbfc3f2640 10 monclient(hunting): picked mon.noname-h con 0x7fcbf41beea0 addr v1:172.21.15.104:6791/0 2024-08-19T12:42:48.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.046+0000 7fcbfc3f2640 10 monclient(hunting): picked mon.noname-b con 0x7fcbf41bd890 addr v1:172.21.15.26:6790/0 2024-08-19T12:42:48.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.046+0000 7fcbfc3f2640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf40aadf0 con 0x7fcbd80075f0 2024-08-19T12:42:48.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.046+0000 7fcbfa18e640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fcbd80075f0 0x7fcbf4132ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:57842/0 (socket says 172.21.15.26:57842) 2024-08-19T12:42:48.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.046+0000 7fcbfa18e640 1 -- 172.21.15.26:0/1469548351 learned_addr learned my addr 172.21.15.26:0/1469548351 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:48.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.046+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/1469548351 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf4134fc0 con 0x7fcbf41bd890 2024-08-19T12:42:48.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.046+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/1469548351 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf41be340 con 0x7fcbf41beea0 2024-08-19T12:42:48.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.046+0000 7fcbfc3f2640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:48.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.046+0000 7fcbfc3f2640 10 monclient(hunting): authenticate will time out at 962.134454s 2024-08-19T12:42:48.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.048+0000 7fcbfa18e640 1 -- 172.21.15.26:0/1469548351 >> v1:172.21.15.104:6789/0 conn(0x7fcbd80075f0 legacy=0x7fcbf4132ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:48.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.048+0000 7fcbeaffd640 10 client.?.objecter ms_handle_connect 0x7fcbd80075f0 2024-08-19T12:42:48.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.048+0000 7fcbeaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:48.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.048+0000 7fcbeaffd640 10 client.?.objecter ms_handle_connect 0x7fcbf41bd890 2024-08-19T12:42:48.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.048+0000 7fcbeaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:48.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.048+0000 7fcbeaffd640 10 client.?.objecter ms_handle_connect 0x7fcbf41beea0 2024-08-19T12:42:48.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.048+0000 7fcbeaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:48.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.048+0000 7fcbfa98f640 1 -- 172.21.15.26:0/1469548351 >> v1:172.21.15.26:6790/0 conn(0x7fcbf41bd890 legacy=0x7fcbf41d6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:48.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.048+0000 7fcbf998d640 1 -- 172.21.15.26:0/1469548351 >> v1:172.21.15.104:6791/0 conn(0x7fcbf41beea0 legacy=0x7fcbf41d30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:48.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.051+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2138744035 0 0) 0x7fcbf40aadf0 con 0x7fcbd80075f0 2024-08-19T12:42:48.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:48.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:48.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:48.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): my global_id is 26578 2024-08-19T12:42:48.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbd800f990 con 0x7fcbd80075f0 2024-08-19T12:42:48.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3894350749 0 0) 0x7fcbf4134fc0 con 0x7fcbf41bd890 2024-08-19T12:42:48.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:48.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:48.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:48.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): my global_id is 25958 2024-08-19T12:42:48.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf40aadf0 con 0x7fcbf41bd890 2024-08-19T12:42:48.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1040222223 0 0) 0x7fcbd800f990 con 0x7fcbd80075f0 2024-08-19T12:42:48.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbf4134fc0 con 0x7fcbd80075f0 2024-08-19T12:42:48.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcbf0001cf0 con 0x7fcbd80075f0 2024-08-19T12:42:48.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:48.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:42:48.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): dump: 2024-08-19T12:42:48.062 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:48.062 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:48.062 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:48.063 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1654955763 0 0) 0x7fcbf40aadf0 con 0x7fcbf41bd890 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcbd800f990 con 0x7fcbf41bd890 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcbec001fb0 con 0x7fcbf41bd890 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient(hunting): dump: 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:48.064 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2901987087 0 0) 0x7fcbf4134fc0 con 0x7fcbd80075f0 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 >> v1:172.21.15.104:6791/0 conn(0x7fcbf41beea0 legacy=0x7fcbf41d30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 >> v1:172.21.15.26:6790/0 conn(0x7fcbf41bd890 legacy=0x7fcbf41d6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:48.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient: _finish_hunting 0 2024-08-19T12:42:48.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 monclient: found mon.f 2024-08-19T12:42:48.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:48.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:42:48.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf41c06a0 con 0x7fcbd80075f0 2024-08-19T12:42:48.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient: _finish_auth 0 2024-08-19T12:42:48.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient: _check_auth_tickets 2024-08-19T12:42:48.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:48.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbfc3f2640 5 monclient: authenticate success, global_id 26578 2024-08-19T12:42:48.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbfc3f2640 10 monclient: _renew_subs 2024-08-19T12:42:48.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbfc3f2640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:42:48.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/1469548351 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf41bfe00 con 0x7fcbd80075f0 2024-08-19T12:42:48.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbfc3f2640 10 client.26578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbfc3f2640 10 monclient: _renew_subs 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbfc3f2640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/1469548351 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcbf41d7f20 con 0x7fcbd80075f0 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbfc3f2640 1 librados: init done 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcbf0004b40 con 0x7fcbd80075f0 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcbf00050b0 con 0x7fcbd80075f0 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbeaffd640 10 monclient: dump: 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:48.067 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:48.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.052+0000 7fcbf898b640 4 set_mon_vals no callback set 2024-08-19T12:42:48.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.053+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fcbf0005250 con 0x7fcbd80075f0 2024-08-19T12:42:48.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.054+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(178..178 src has 1..178) ==== 5692+0+0 (unknown 4028602477 0 0) 0x7fcbf004e270 con 0x7fcbd80075f0 2024-08-19T12:42:48.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.054+0000 7fcbeaffd640 10 client.26578.objecter ms_dispatch 0x7fcbf4000b90 osd_map(178..178 src has 1..178) 2024-08-19T12:42:48.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.054+0000 7fcbeaffd640 3 client.26578.objecter handle_osd_map got epochs [178,178] > 0 2024-08-19T12:42:48.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.054+0000 7fcbeaffd640 3 client.26578.objecter handle_osd_map decoding full epoch 178 2024-08-19T12:42:48.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.054+0000 7fcbf998d640 10 monclient: get_auth_request con 0x7fcbd8042b90 auth_method 0 2024-08-19T12:42:48.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.054+0000 7fcbfc3f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:48.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.054+0000 7fcbeaffd640 20 client.26578.objecter dump_active .. 0 homeless 2024-08-19T12:42:48.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.054+0000 7fcbfc3f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:48.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.054+0000 7fcbfc3f2640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:42:48.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.054+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/1469548351 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbdc005080 con 0x7fcbd80075f0 2024-08-19T12:42:48.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.056+0000 7fcbeaffd640 10 client.26578.objecter ms_handle_connect 0x7fcbd8042b90 2024-08-19T12:42:48.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.059+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.1 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fcbf00175a0 con 0x7fcbd80075f0 2024-08-19T12:42:48.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.059+0000 7fcbeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:48.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.059+0000 7fcbeaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:48.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.335+0000 7fcbfc3f2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:42:48.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.335+0000 7fcbfc3f2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:42:48.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.335+0000 7fcbfc3f2640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:42:48.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.335+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/1469548351 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcbdc005370 con 0x7fcbd80075f0 2024-08-19T12:42:48.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.336+0000 7fcbeaffd640 1 -- 172.21.15.26:0/1469548351 <== mon.1 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2723 (unknown 254746657 0 1975126266) 0x7fcbf0016ea0 con 0x7fcbd80075f0 2024-08-19T12:42:48.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.336+0000 7fcbeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:42:48.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.336+0000 7fcbeaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:48.338 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":20,"quorum":[0,1,2,3,5,6,7,8],"quorum_names":["a","f","b","g","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:42:48.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 10 librados: watch_flush enter 2024-08-19T12:42:48.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 10 librados: watch_flush exit 2024-08-19T12:42:48.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 20 client.26578.objecter shutdown clearing up homeless session... 2024-08-19T12:42:48.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 10 client.26578.objecter successfully canceled tick 2024-08-19T12:42:48.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/1469548351 >> v1:172.21.15.104:6800/899887506 conn(0x7fcbd8042b90 legacy=0x7fcbd8045050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:48.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 10 monclient: shutdown 2024-08-19T12:42:48.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:48.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/1469548351 >> v1:172.21.15.104:6789/0 conn(0x7fcbd80075f0 legacy=0x7fcbf4132ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:48.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/1469548351 shutdown_connections 2024-08-19T12:42:48.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/1469548351 >> 172.21.15.26:0/1469548351 conn(0x7fcbf40a8740 msgr2=0x7fcbf4133970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:48.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/1469548351 shutdown_connections 2024-08-19T12:42:48.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 1 -- 172.21.15.26:0/1469548351 wait complete. 2024-08-19T12:42:48.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:48.338+0000 7fcbfc3f2640 1 librados: shutdown 2024-08-19T12:42:51.351 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:42:51.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.460+0000 7f71fcfa1640 10 monclient: get_monmap_and_config 2024-08-19T12:42:51.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.460+0000 7f71fcfa1640 10 monclient: build_initial_monmap 2024-08-19T12:42:51.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.460+0000 7f71fcfa1640 10 monclient: monmap: 2024-08-19T12:42:51.462 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:51.462 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.462 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:51.461997+0000 2024-08-19T12:42:51.462 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:51.461997+0000 2024-08-19T12:42:51.462 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:51.463 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.463 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:51.463 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:51.463 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:51.463 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:51.463 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:51.463 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:51.463 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:51.463 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:51.463 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:51.463 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.460+0000 7f71fcfa1640 1 Processor -- start 2024-08-19T12:42:51.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.460+0000 7f71fcfa1640 1 -- start start 2024-08-19T12:42:51.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.460+0000 7f71fcfa1640 10 monclient: init 2024-08-19T12:42:51.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.461+0000 7f71fcfa1640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:51.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.461+0000 7f71fcfa1640 10 monclient: _add_conns ranks=[3,7,0,2,4,6,8,1,5] 2024-08-19T12:42:51.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.461+0000 7f71fcfa1640 10 monclient(hunting): picked mon.noname-g con 0x7f71f806e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:51.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.461+0000 7f71fcfa1640 10 monclient(hunting): picked mon.noname-i con 0x7f71f80a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:51.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.461+0000 7f71fcfa1640 10 monclient(hunting): picked mon.noname-a con 0x7f71f80a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:51.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.461+0000 7f71fcfa1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f8136790 con 0x7f71f80a5d50 2024-08-19T12:42:51.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.461+0000 7f71fcfa1640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f809ddf0 con 0x7f71f806e7b0 2024-08-19T12:42:51.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.461+0000 7f71fcfa1640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f809e0d0 con 0x7f71f80a5920 2024-08-19T12:42:51.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.461+0000 7f71fcfa1640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:51.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.461+0000 7f71fcfa1640 10 monclient(hunting): authenticate will time out at 965.549507s 2024-08-19T12:42:51.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.461+0000 7f71f6d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f71f80a5d50 0x7f71f8141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57846/0 (socket says 172.21.15.26:57846) 2024-08-19T12:42:51.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.461+0000 7f71f6d9d640 1 -- 172.21.15.26:0/4140211857 learned_addr learned my addr 172.21.15.26:0/4140211857 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:51.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.463+0000 7f71f6d9d640 1 -- 172.21.15.26:0/4140211857 >> v1:172.21.15.26:6789/0 conn(0x7f71f80a5d50 legacy=0x7f71f8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 1 -- 172.21.15.26:0/4140211857 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 697919181 0 0) 0x7f71f8136790 con 0x7f71f80a5d50 2024-08-19T12:42:51.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient(hunting): _negotiate 2024-08-19T12:42:51.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:51.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:51.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient(hunting): my global_id is 144108 2024-08-19T12:42:51.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 1 -- 172.21.15.26:0/4140211857 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71e0003520 con 0x7f71f80a5d50 2024-08-19T12:42:51.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f659c640 1 -- 172.21.15.26:0/4140211857 >> v1:172.21.15.104:6790/0 conn(0x7f71f806e7b0 legacy=0x7f71f8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f5d9b640 1 -- 172.21.15.26:0/4140211857 >> v1:172.21.15.104:6792/0 conn(0x7f71f80a5920 legacy=0x7f71f813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 1 -- 172.21.15.26:0/4140211857 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2751737633 0 0) 0x7f71e0003520 con 0x7f71f80a5d50 2024-08-19T12:42:51.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 1 -- 172.21.15.26:0/4140211857 >> v1:172.21.15.104:6792/0 conn(0x7f71f80a5920 legacy=0x7f71f813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 1 -- 172.21.15.26:0/4140211857 >> v1:172.21.15.104:6790/0 conn(0x7f71f806e7b0 legacy=0x7f71f8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient: _finish_hunting 0 2024-08-19T12:42:51.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 1 monclient: found mon.noname-a 2024-08-19T12:42:51.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:51.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:42:51.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 1 -- 172.21.15.26:0/4140211857 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f809e3b0 con 0x7f71f80a5d50 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient: _finish_auth 0 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient: _check_auth_tickets 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 1 -- 172.21.15.26:0/4140211857 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f71f0001ae0 con 0x7f71f80a5d50 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71fcfa1640 5 monclient: authenticate success, global_id 144108 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71fcfa1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient: dump: 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.471 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71fcfa1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 1 -- 172.21.15.26:0/4140211857 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71f00048a0 con 0x7f71f80a5d50 2024-08-19T12:42:51.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 1 -- 172.21.15.26:0/4140211857 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f71f0004dd0 con 0x7f71f80a5d50 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71fcfa1640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71fcfa1640 4 set_mon_vals no callback set 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71f559a640 10 monclient: dump: 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.473 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:51.474 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:51.474 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:51.474 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:51.474 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:51.474 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:51.474 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:51.474 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:51.474 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:51.474 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71fcfa1640 10 monclient: shutdown 2024-08-19T12:42:51.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71fcfa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:51.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.464+0000 7f71fcfa1640 1 -- 172.21.15.26:0/4140211857 >> v1:172.21.15.26:6789/0 conn(0x7f71f80a5d50 legacy=0x7f71f8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.465+0000 7f71fcfa1640 1 -- 172.21.15.26:0/4140211857 shutdown_connections 2024-08-19T12:42:51.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.465+0000 7f71fcfa1640 1 -- 172.21.15.26:0/4140211857 >> 172.21.15.26:0/4140211857 conn(0x7f71f80a8740 msgr2=0x7f71f80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:51.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.465+0000 7f71fcfa1640 1 -- 172.21.15.26:0/4140211857 shutdown_connections 2024-08-19T12:42:51.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.465+0000 7f71fcfa1640 1 -- 172.21.15.26:0/4140211857 wait complete. 2024-08-19T12:42:51.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.465+0000 7f71fcfa1640 10 monclient: build_initial_monmap 2024-08-19T12:42:51.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.465+0000 7f71fcfa1640 10 monclient: monmap: 2024-08-19T12:42:51.475 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:51.475 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:51.475 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 1 librados: starting msgr at 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 1 librados: starting objecter 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 1 Processor -- start 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 1 -- start start 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 1 librados: setting wanted keys 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 1 librados: calling monclient init 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 10 monclient: init 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 10 monclient: _add_conns ranks=[7,4,0,2,8,3,1,5,6] 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 10 monclient(hunting): picked mon.noname-i con 0x7f71f806e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 10 monclient(hunting): picked mon.noname-c con 0x7f71f80a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 10 monclient(hunting): picked mon.noname-a con 0x7f71f80a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f813b1e0 con 0x7f71f80a5d50 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f8138200 con 0x7f71f80a5920 2024-08-19T12:42:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f813af00 con 0x7f71f806e7b0 2024-08-19T12:42:51.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:51.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71fcfa1640 10 monclient(hunting): authenticate will time out at 965.554746s 2024-08-19T12:42:51.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71f6d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f71f80a5d50 0x7f71f81d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57852/0 (socket says 172.21.15.26:57852) 2024-08-19T12:42:51.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71f5d9b640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f71f80a5920 0x7f71f8137af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:54342/0 (socket says 172.21.15.26:54342) 2024-08-19T12:42:51.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.466+0000 7f71f6d9d640 1 -- 172.21.15.26:0/3209537574 learned_addr learned my addr 172.21.15.26:0/3209537574 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:51.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.468+0000 7f71f6d9d640 1 -- 172.21.15.26:0/3209537574 >> v1:172.21.15.26:6789/0 conn(0x7f71f80a5d50 legacy=0x7f71f81d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 client.?.objecter ms_handle_connect 0x7f71f80a5d50 2024-08-19T12:42:51.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:51.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 client.?.objecter ms_handle_connect 0x7f71f80a5920 2024-08-19T12:42:51.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:51.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71f5d9b640 1 -- 172.21.15.26:0/3209537574 >> v1:172.21.15.26:6791/0 conn(0x7f71f80a5920 legacy=0x7f71f8137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433304667 0 0) 0x7f71f813b1e0 con 0x7f71f80a5d50 2024-08-19T12:42:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): my global_id is 144117 2024-08-19T12:42:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71d8004eb0 con 0x7f71f80a5d50 2024-08-19T12:42:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1471826327 0 0) 0x7f71f8138200 con 0x7f71f80a5920 2024-08-19T12:42:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): my global_id is 26158 2024-08-19T12:42:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71f659c640 1 -- 172.21.15.26:0/3209537574 >> v1:172.21.15.104:6792/0 conn(0x7f71f806e7b0 legacy=0x7f71f81373e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71f813b1e0 con 0x7f71f80a5920 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 client.?.objecter ms_handle_connect 0x7f71f806e7b0 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2165311176 0 0) 0x7f71f813af00 con 0x7f71f806e7b0 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): my global_id is 134710 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71f8138200 con 0x7f71f806e7b0 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 57794504 0 0) 0x7f71d8004eb0 con 0x7f71f80a5d50 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71f813af00 con 0x7f71f80a5d50 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f71f0001d30 con 0x7f71f80a5d50 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:51.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): dump: 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:51.485 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 187371445 0 0) 0x7f71f813b1e0 con 0x7f71f80a5920 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71d8004eb0 con 0x7f71f80a5920 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f71ec003990 con 0x7f71f80a5920 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient(hunting): dump: 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.486 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2014111545 0 0) 0x7f71f813af00 con 0x7f71f80a5d50 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 >> v1:172.21.15.104:6792/0 conn(0x7f71f806e7b0 legacy=0x7f71f81373e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 >> v1:172.21.15.26:6791/0 conn(0x7f71f80a5920 legacy=0x7f71f8137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 monclient: found mon.a 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f8142950 con 0x7f71f80a5d50 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient: _finish_auth 0 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71fcfa1640 5 monclient: authenticate success, global_id 144117 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71fcfa1640 10 monclient: _renew_subs 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71fcfa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71fcfa1640 1 -- 172.21.15.26:0/3209537574 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71f8139c50 con 0x7f71f80a5d50 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71fcfa1640 10 client.144117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:51.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71fcfa1640 10 monclient: _renew_subs 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71fcfa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71fcfa1640 1 -- 172.21.15.26:0/3209537574 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f71f81d7b90 con 0x7f71f80a5d50 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71fcfa1640 1 librados: init done 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71f0005040 con 0x7f71f80a5d50 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f71f00048a0 con 0x7f71f80a5d50 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71f4d99640 4 set_mon_vals no callback set 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.469+0000 7f71e6ffd640 10 monclient: dump: 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.489 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.490 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.490 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:51.490 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.490 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:51.490 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:51.490 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:51.491 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:51.491 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:51.491 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:51.491 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:51.491 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:51.491 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:51.491 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.470+0000 7f71fcfa1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:51.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.470+0000 7f71fcfa1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:51.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.470+0000 7f71fcfa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:51.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.471+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f71f00117f0 con 0x7f71f80a5d50 2024-08-19T12:42:51.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.470+0000 7f71fcfa1640 1 -- 172.21.15.26:0/3209537574 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71dc005080 con 0x7f71f80a5d50 2024-08-19T12:42:51.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.471+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(181..181 src has 1..181) ==== 6473+0+0 (unknown 1282003511 0 0) 0x7f71f004e560 con 0x7f71f80a5d50 2024-08-19T12:42:51.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.471+0000 7f71e6ffd640 10 client.144117.objecter ms_dispatch 0x7f71f8000b90 osd_map(181..181 src has 1..181) 2024-08-19T12:42:51.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.471+0000 7f71e6ffd640 3 client.144117.objecter handle_osd_map got epochs [181,181] > 0 2024-08-19T12:42:51.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.471+0000 7f71e6ffd640 3 client.144117.objecter handle_osd_map decoding full epoch 181 2024-08-19T12:42:51.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.471+0000 7f71e6ffd640 20 client.144117.objecter dump_active .. 0 homeless 2024-08-19T12:42:51.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.471+0000 7f71f659c640 10 monclient: get_auth_request con 0x7f71d8041880 auth_method 0 2024-08-19T12:42:51.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.474+0000 7f71e6ffd640 10 client.144117.objecter ms_handle_connect 0x7f71d8041880 2024-08-19T12:42:51.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.475+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f71f0016eb0 con 0x7f71f80a5d50 2024-08-19T12:42:51.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.475+0000 7f71e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:51.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.475+0000 7f71e6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.731+0000 7f71fcfa1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:42:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.731+0000 7f71fcfa1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:42:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.731+0000 7f71fcfa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.731+0000 7f71fcfa1640 1 -- 172.21.15.26:0/3209537574 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f71dc005370 con 0x7f71f80a5d50 2024-08-19T12:42:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.732+0000 7f71e6ffd640 1 -- 172.21.15.26:0/3209537574 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 755502242) 0x7f71f001fdf0 con 0x7f71f80a5d50 2024-08-19T12:42:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.732+0000 7f71e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:42:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.732+0000 7f71e6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:51.734 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":24,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:42:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 10 librados: watch_flush enter 2024-08-19T12:42:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 10 librados: watch_flush exit 2024-08-19T12:42:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 20 client.144117.objecter shutdown clearing up homeless session... 2024-08-19T12:42:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 10 client.144117.objecter successfully canceled tick 2024-08-19T12:42:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 1 -- 172.21.15.26:0/3209537574 >> v1:172.21.15.104:6800/899887506 conn(0x7f71d8041880 legacy=0x7f71d8043d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 10 monclient: shutdown 2024-08-19T12:42:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 1 -- 172.21.15.26:0/3209537574 >> v1:172.21.15.26:6789/0 conn(0x7f71f80a5d50 legacy=0x7f71f81d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 1 -- 172.21.15.26:0/3209537574 shutdown_connections 2024-08-19T12:42:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 1 -- 172.21.15.26:0/3209537574 >> 172.21.15.26:0/3209537574 conn(0x7f71f80a8740 msgr2=0x7f71f80a51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 1 -- 172.21.15.26:0/3209537574 shutdown_connections 2024-08-19T12:42:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 1 -- 172.21.15.26:0/3209537574 wait complete. 2024-08-19T12:42:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.734+0000 7f71fcfa1640 1 librados: shutdown 2024-08-19T12:42:51.747 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:42:51.747 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:42:51.748 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:42:51.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.845+0000 7f9212fc9640 10 monclient: get_monmap_and_config 2024-08-19T12:42:51.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.845+0000 7f9212fc9640 10 monclient: build_initial_monmap 2024-08-19T12:42:51.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.845+0000 7f9212fc9640 10 monclient: monmap: 2024-08-19T12:42:51.847 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:51.848 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.848 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:51.846875+0000 2024-08-19T12:42:51.848 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:51.846875+0000 2024-08-19T12:42:51.848 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:51.848 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.849 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:51.849 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:51.849 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:51.849 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:51.849 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:51.850 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:51.850 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:51.850 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:51.850 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:51.850 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.845+0000 7f9212fc9640 1 Processor -- start 2024-08-19T12:42:51.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.845+0000 7f9212fc9640 1 -- start start 2024-08-19T12:42:51.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.845+0000 7f9212fc9640 10 monclient: init 2024-08-19T12:42:51.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9212fc9640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:51.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9212fc9640 10 monclient: _add_conns ranks=[8,3,6,4,1,5,2,0,7] 2024-08-19T12:42:51.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9212fc9640 10 monclient(hunting): picked mon.noname-e con 0x7f920c073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9212fc9640 10 monclient(hunting): picked mon.noname-g con 0x7f920c0a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9212fc9640 10 monclient(hunting): picked mon.noname-d con 0x7f920c13f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9212fc9640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f920c09dae0 con 0x7f920c0a57a0 2024-08-19T12:42:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9212fc9640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f920c136910 con 0x7f920c13f2e0 2024-08-19T12:42:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9212fc9640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f920c09dd50 con 0x7f920c073470 2024-08-19T12:42:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9212fc9640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9212fc9640 10 monclient(hunting): authenticate will time out at 965.934300s 2024-08-19T12:42:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9210d65640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f920c073470 0x7f920c136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:53384/0 (socket says 172.21.15.26:53384) 2024-08-19T12:42:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9211566640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f920c13f2e0 0x7f920c1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34486/0 (socket says 172.21.15.26:34486) 2024-08-19T12:42:51.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.846+0000 7f9210d65640 1 -- 172.21.15.26:0/3492388149 learned_addr learned my addr 172.21.15.26:0/3492388149 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f9211566640 1 -- 172.21.15.26:0/3492388149 >> v1:172.21.15.26:6792/0 conn(0x7f920c13f2e0 legacy=0x7f920c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f9210d65640 1 -- 172.21.15.26:0/3492388149 >> v1:172.21.15.26:6793/0 conn(0x7f920c073470 legacy=0x7f920c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 1 -- 172.21.15.26:0/3492388149 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1436979013 0 0) 0x7f920c136910 con 0x7f920c13f2e0 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 10 monclient(hunting): my global_id is 26187 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 1 -- 172.21.15.26:0/3492388149 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91f4003520 con 0x7f920c13f2e0 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 1 -- 172.21.15.26:0/3492388149 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 602100873 0 0) 0x7f920c09dd50 con 0x7f920c073470 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:51.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 10 monclient(hunting): my global_id is 25721 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 1 -- 172.21.15.26:0/3492388149 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f920c136910 con 0x7f920c073470 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920bfff640 1 -- 172.21.15.26:0/3492388149 >> v1:172.21.15.104:6790/0 conn(0x7f920c0a57a0 legacy=0x7f920c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.848+0000 7f920b7fe640 1 -- 172.21.15.26:0/3492388149 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1345276854 0 0) 0x7f91f4003520 con 0x7f920c13f2e0 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 1 -- 172.21.15.26:0/3492388149 >> v1:172.21.15.26:6793/0 conn(0x7f920c073470 legacy=0x7f920c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 1 -- 172.21.15.26:0/3492388149 >> v1:172.21.15.104:6790/0 conn(0x7f920c0a57a0 legacy=0x7f920c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 1 monclient: found mon.noname-d 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 1 -- 172.21.15.26:0/3492388149 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f920c09e030 con 0x7f920c13f2e0 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 10 monclient: _finish_auth 0 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:42:51.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f9212fc9640 5 monclient: authenticate success, global_id 26187 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 1 -- 172.21.15.26:0/3492388149 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9204001f20 con 0x7f920c13f2e0 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 10 monclient: dump: 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:51.879 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f9212fc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 1 -- 172.21.15.26:0/3492388149 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9204004b60 con 0x7f920c13f2e0 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 1 -- 172.21.15.26:0/3492388149 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f92040020c0 con 0x7f920c13f2e0 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f9212fc9640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f9212fc9640 4 set_mon_vals no callback set 2024-08-19T12:42:51.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f920b7fe640 10 monclient: dump: 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:51.881 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:51.882 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:51.882 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:51.882 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:51.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f9212fc9640 10 monclient: shutdown 2024-08-19T12:42:51.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f9212fc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:51.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f9212fc9640 1 -- 172.21.15.26:0/3492388149 >> v1:172.21.15.26:6792/0 conn(0x7f920c13f2e0 legacy=0x7f920c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f9212fc9640 1 -- 172.21.15.26:0/3492388149 shutdown_connections 2024-08-19T12:42:51.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f9212fc9640 1 -- 172.21.15.26:0/3492388149 >> 172.21.15.26:0/3492388149 conn(0x7f920c0abff0 msgr2=0x7f920c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:51.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f9212fc9640 1 -- 172.21.15.26:0/3492388149 shutdown_connections 2024-08-19T12:42:51.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.849+0000 7f9212fc9640 1 -- 172.21.15.26:0/3492388149 wait complete. 2024-08-19T12:42:51.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.850+0000 7f9212fc9640 10 monclient: build_initial_monmap 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.850+0000 7f9212fc9640 10 monclient: monmap: 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.850+0000 7f9212fc9640 1 librados: starting msgr at 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.850+0000 7f9212fc9640 1 librados: starting objecter 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.850+0000 7f9212fc9640 1 Processor -- start 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.850+0000 7f9212fc9640 1 -- start start 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.850+0000 7f9212fc9640 1 librados: setting wanted keys 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.850+0000 7f9212fc9640 1 librados: calling monclient init 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.850+0000 7f9212fc9640 10 monclient: init 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.851+0000 7f9212fc9640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.851+0000 7f9212fc9640 10 monclient: _add_conns ranks=[8,5,6,0,3,7,4,1,2] 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.851+0000 7f9212fc9640 10 monclient(hunting): picked mon.noname-e con 0x7f920c073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.851+0000 7f9212fc9640 10 monclient(hunting): picked mon.noname-h con 0x7f920c0a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.851+0000 7f9212fc9640 10 monclient(hunting): picked mon.noname-d con 0x7f920c13f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.851+0000 7f9212fc9640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f920c136d70 con 0x7f920c0a57a0 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.851+0000 7f9212fc9640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f920c137050 con 0x7f920c13f2e0 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.851+0000 7f9212fc9640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f920c0a4010 con 0x7f920c073470 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.851+0000 7f9212fc9640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.851+0000 7f9212fc9640 10 monclient(hunting): authenticate will time out at 965.939347s 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.851+0000 7f9210d65640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f920c073470 0x7f920c0a24e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:53396/0 (socket says 172.21.15.26:53396) 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.851+0000 7f9210d65640 1 -- 172.21.15.26:0/46447448 learned_addr learned my addr 172.21.15.26:0/46447448 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f9210d65640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.26:6793/0 conn(0x7f920c073470 legacy=0x7f920c0a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 client.?.objecter ms_handle_connect 0x7f920c073470 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f9211566640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.26:6792/0 conn(0x7f920c13f2e0 legacy=0x7f920c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 client.?.objecter ms_handle_connect 0x7f920c13f2e0 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4281496156 0 0) 0x7f920c0a4010 con 0x7f920c073470 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 monclient(hunting): my global_id is 25730 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f920bfff640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.104:6791/0 conn(0x7f920c0a57a0 legacy=0x7f920c0a2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91f8004eb0 con 0x7f920c073470 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 client.?.objecter ms_handle_connect 0x7f920c0a57a0 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2197871863 0 0) 0x7f920c137050 con 0x7f920c13f2e0 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 monclient(hunting): my global_id is 26196 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f920c0a4010 con 0x7f920c13f2e0 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1734373055 0 0) 0x7f920c0a4010 con 0x7f920c13f2e0 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f920c137050 con 0x7f920c13f2e0 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9204001ee0 con 0x7f920c13f2e0 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 monclient(hunting): dump: 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:51.887 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.853+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 289239451 0 0) 0x7f91f8004eb0 con 0x7f920c073470 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f920c0a4010 con 0x7f920c073470 2024-08-19T12:42:51.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3575538970 0 0) 0x7f920c136d70 con 0x7f920c0a57a0 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient(hunting): my global_id is 25907 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91f8004eb0 con 0x7f920c0a57a0 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f91fc0047e0 con 0x7f920c073470 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient(hunting): dump: 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.889 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2299461304 0 0) 0x7f920c137050 con 0x7f920c13f2e0 2024-08-19T12:42:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.26:6793/0 conn(0x7f920c073470 legacy=0x7f920c0a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.104:6791/0 conn(0x7f920c0a57a0 legacy=0x7f920c0a2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient: _finish_hunting 0 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 1 monclient: found mon.d 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f920c135270 con 0x7f920c13f2e0 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient: _finish_auth 0 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient: _check_auth_tickets 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f9212fc9640 5 monclient: authenticate success, global_id 26196 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f9212fc9640 10 monclient: _renew_subs 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f9212fc9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f9212fc9640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f920c0a3b70 con 0x7f920c13f2e0 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f9212fc9640 10 client.26196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f9212fc9640 10 monclient: _renew_subs 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f9212fc9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f9212fc9640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f920c0a3300 con 0x7f920c13f2e0 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f9212fc9640 1 librados: init done 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9204002080 con 0x7f920c13f2e0 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9204005730 con 0x7f920c13f2e0 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f920affd640 4 set_mon_vals no callback set 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.854+0000 7f92097fa640 10 monclient: dump: 2024-08-19T12:42:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.855+0000 7f9212fc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.855+0000 7f9212fc9640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f91ec0065c0 con 0x7f91ec002860 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.855+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f9204011b90 con 0x7f920c13f2e0 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.855+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(181..181 src has 1..181) ==== 6473+0+0 (unknown 1282003511 0 0) 0x7f920404e700 con 0x7f920c13f2e0 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.855+0000 7f92097fa640 10 client.26196.objecter ms_dispatch 0x7f920c000b90 osd_map(181..181 src has 1..181) 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.855+0000 7f92097fa640 3 client.26196.objecter handle_osd_map got epochs [181,181] > 0 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.855+0000 7f92097fa640 3 client.26196.objecter handle_osd_map decoding full epoch 181 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.855+0000 7f92097fa640 20 client.26196.objecter dump_active .. 0 homeless 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.855+0000 7f920bfff640 10 monclient: get_auth_request con 0x7f91f80417b0 auth_method 0 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f9210d65640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.26:6789/0 conn(0x7f91ec002860 legacy=0x7f91ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 10 client.26196.objecter ms_handle_connect 0x7f91ec002860 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 10 client.26196.objecter resend_mon_ops 2024-08-19T12:42:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2839980152 0 0) 0x7f91ec0065c0 con 0x7f91ec002860 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 10 monclient(hunting): my global_id is 144126 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f920c137050 con 0x7f91ec002860 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 295418510 0 0) 0x7f920c137050 con 0x7f91ec002860 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91ec0065c0 con 0x7f91ec002860 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f91fc004f70 con 0x7f91ec002860 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 10 monclient: dump: 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1214100947 0 0) 0x7f91ec0065c0 con 0x7f91ec002860 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.857+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91ec00a160 con 0x7f91ec002860 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.858+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f91fc006420 con 0x7f91ec002860 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.858+0000 7f92097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.858+0000 7f92097fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.858+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.26:6789/0 conn(0x7f91ec002860 legacy=0x7f91ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.858+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.26:6789/0 conn(0x7f91ec002860 legacy=0x7f91ec004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.858+0000 7f92097fa640 10 client.26196.objecter ms_handle_connect 0x7f91f80417b0 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.864+0000 7f9212fc9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.864+0000 7f9212fc9640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f91ec009e50 con 0x7f91ec009110 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f9210d65640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.26:6789/0 conn(0x7f91ec009110 legacy=0x7f91ec0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 10 client.26196.objecter ms_handle_connect 0x7f91ec009110 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 10 client.26196.objecter resend_mon_ops 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750948438 0 0) 0x7f91ec009e50 con 0x7f91ec009110 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 10 monclient(hunting): my global_id is 144135 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91ec0065c0 con 0x7f91ec009110 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3745406362 0 0) 0x7f91ec0065c0 con 0x7f91ec009110 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f91ec009e50 con 0x7f91ec009110 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f91fc0057c0 con 0x7f91ec009110 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 10 monclient: dump: 2024-08-19T12:42:51.899 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:51.899 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:51.899 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.899 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:51.899 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:51.899 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:51.899 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:51.899 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:51.899 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:51.899 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 505288546 0 0) 0x7f91ec009e50 con 0x7f91ec009110 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.866+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f91ec00c3a0 con 0x7f91ec009110 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7545 (unknown 0 0 1124904196) 0x7f91fc005b00 con 0x7f91ec009110 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f92097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f92097fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.26:6789/0 conn(0x7f91ec009110 legacy=0x7f91ec0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f92097fa640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.26:6789/0 conn(0x7f91ec009110 legacy=0x7f91ec0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:51.901 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:51.901 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:51.901 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:51.901 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:42:51.901 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:51.901 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 23785, 2024-08-19T12:42:51.901 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:51.901 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:51.901 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:51.903 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.904 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:51.905 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:51.906 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:51.907 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:51.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:51.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:51.911 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:51.912 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:51.913 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:51.914 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.916 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:51.917 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:51.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.919 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:51.920 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 10 librados: watch_flush enter 2024-08-19T12:42:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 10 librados: watch_flush exit 2024-08-19T12:42:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 20 client.26196.objecter shutdown clearing up homeless session... 2024-08-19T12:42:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 10 client.26196.objecter successfully canceled tick 2024-08-19T12:42:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.104:6800/899887506 conn(0x7f91f80417b0 legacy=0x7f91f8043c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 10 monclient: shutdown 2024-08-19T12:42:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 1 -- 172.21.15.26:0/46447448 >> v1:172.21.15.26:6792/0 conn(0x7f920c13f2e0 legacy=0x7f920c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9211566640 1 -- 172.21.15.26:0/46447448 reap_dead start 2024-08-19T12:42:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9211566640 1 -- 172.21.15.26:0/46447448 reap_dead start 2024-08-19T12:42:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 1 -- 172.21.15.26:0/46447448 shutdown_connections 2024-08-19T12:42:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 1 -- 172.21.15.26:0/46447448 >> 172.21.15.26:0/46447448 conn(0x7f920c0abff0 msgr2=0x7f920c1338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 1 -- 172.21.15.26:0/46447448 shutdown_connections 2024-08-19T12:42:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 1 -- 172.21.15.26:0/46447448 wait complete. 2024-08-19T12:42:51.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:51.867+0000 7f9212fc9640 1 librados: shutdown 2024-08-19T12:42:51.924 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:51.924 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:42:51.924 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:42:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.033+0000 7fb75602f640 10 monclient: get_monmap_and_config 2024-08-19T12:42:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.033+0000 7fb75602f640 10 monclient: build_initial_monmap 2024-08-19T12:42:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.033+0000 7fb75602f640 10 monclient: monmap: 2024-08-19T12:42:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:52.035 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:52.035491+0000 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:52.035491+0000 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:52.036 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 1 Processor -- start 2024-08-19T12:42:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 1 -- start start 2024-08-19T12:42:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 10 monclient: init 2024-08-19T12:42:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 10 monclient: _add_conns ranks=[4,6,7,2,1,8,0,3,5] 2024-08-19T12:42:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 10 monclient(hunting): picked mon.noname-c con 0x7fb75006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 10 monclient(hunting): picked mon.noname-d con 0x7fb7500a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 10 monclient(hunting): picked mon.noname-i con 0x7fb7500a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb750136790 con 0x7fb75006e7b0 2024-08-19T12:42:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb75009ddf0 con 0x7fb7500a5920 2024-08-19T12:42:52.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb75009e0d0 con 0x7fb7500a5d50 2024-08-19T12:42:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.034+0000 7fb75602f640 10 monclient(hunting): authenticate will time out at 966.122954s 2024-08-19T12:42:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.035+0000 7fb74effd640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fb7500a5920 0x7fb75013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34496/0 (socket says 172.21.15.26:34496) 2024-08-19T12:42:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.035+0000 7fb74f7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fb75006e7b0 0x7fb750136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:54356/0 (socket says 172.21.15.26:54356) 2024-08-19T12:42:52.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.035+0000 7fb74effd640 1 -- 172.21.15.26:0/2368948422 learned_addr learned my addr 172.21.15.26:0/2368948422 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74effd640 1 -- 172.21.15.26:0/2368948422 >> v1:172.21.15.26:6792/0 conn(0x7fb7500a5920 legacy=0x7fb75013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74f7fe640 1 -- 172.21.15.26:0/2368948422 >> v1:172.21.15.26:6791/0 conn(0x7fb75006e7b0 legacy=0x7fb750136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1846727851 0 0) 0x7fb744001450 con 0x7fb7500a5920 2024-08-19T12:42:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient(hunting): my global_id is 26205 2024-08-19T12:42:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb738003520 con 0x7fb7500a5920 2024-08-19T12:42:52.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2142589053 0 0) 0x7fb750136790 con 0x7fb75006e7b0 2024-08-19T12:42:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient(hunting): my global_id is 26167 2024-08-19T12:42:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74ffff640 1 -- 172.21.15.26:0/2368948422 >> v1:172.21.15.104:6792/0 conn(0x7fb7500a5d50 legacy=0x7fb750141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb744001450 con 0x7fb75006e7b0 2024-08-19T12:42:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3388116131 0 0) 0x7fb75009e0d0 con 0x7fb7500a5d50 2024-08-19T12:42:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient(hunting): my global_id is 134719 2024-08-19T12:42:52.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb750136790 con 0x7fb7500a5d50 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1117075803 0 0) 0x7fb738003520 con 0x7fb7500a5920 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 >> v1:172.21.15.104:6792/0 conn(0x7fb7500a5d50 legacy=0x7fb750141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 >> v1:172.21.15.26:6791/0 conn(0x7fb75006e7b0 legacy=0x7fb750136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 monclient: found mon.noname-d 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb75009e3b0 con 0x7fb7500a5920 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient: _finish_auth 0 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb75602f640 5 monclient: authenticate success, global_id 26205 2024-08-19T12:42:52.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb744001cb0 con 0x7fb7500a5920 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient: dump: 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.046 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.047 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb75602f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb744004950 con 0x7fb7500a5920 2024-08-19T12:42:52.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 1 -- 172.21.15.26:0/2368948422 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb744004ec0 con 0x7fb7500a5920 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb75602f640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb75602f640 4 set_mon_vals no callback set 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.037+0000 7fb74e7fc640 10 monclient: dump: 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.048 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.038+0000 7fb75602f640 10 monclient: shutdown 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.038+0000 7fb75602f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.038+0000 7fb75602f640 1 -- 172.21.15.26:0/2368948422 >> v1:172.21.15.26:6792/0 conn(0x7fb7500a5920 legacy=0x7fb75013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.038+0000 7fb75602f640 1 -- 172.21.15.26:0/2368948422 shutdown_connections 2024-08-19T12:42:52.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.038+0000 7fb75602f640 1 -- 172.21.15.26:0/2368948422 >> 172.21.15.26:0/2368948422 conn(0x7fb7500a8740 msgr2=0x7fb7500a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.038+0000 7fb75602f640 1 -- 172.21.15.26:0/2368948422 shutdown_connections 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.038+0000 7fb75602f640 1 -- 172.21.15.26:0/2368948422 wait complete. 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.038+0000 7fb75602f640 10 monclient: build_initial_monmap 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 10 monclient: monmap: 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:52.050 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 1 librados: starting msgr at 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 1 librados: starting objecter 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 1 Processor -- start 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 1 -- start start 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 1 librados: setting wanted keys 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 1 librados: calling monclient init 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 10 monclient: init 2024-08-19T12:42:52.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 10 monclient: _add_conns ranks=[1,0,5,3,7,4,2,8,6] 2024-08-19T12:42:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 10 monclient(hunting): picked mon.noname-f con 0x7fb75006e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 10 monclient(hunting): picked mon.noname-a con 0x7fb7500a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.039+0000 7fb75602f640 10 monclient(hunting): picked mon.noname-h con 0x7fb7500a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:42:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.040+0000 7fb75602f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7501345d0 con 0x7fb7500a5920 2024-08-19T12:42:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.040+0000 7fb75602f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb750070300 con 0x7fb75006e7b0 2024-08-19T12:42:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.040+0000 7fb75602f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7500705e0 con 0x7fb7500a5d50 2024-08-19T12:42:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.040+0000 7fb75602f640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.040+0000 7fb75602f640 10 monclient(hunting): authenticate will time out at 966.128038s 2024-08-19T12:42:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.040+0000 7fb74effd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb7500a5920 0x7fb750132d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57874/0 (socket says 172.21.15.26:57874) 2024-08-19T12:42:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.040+0000 7fb74effd640 1 -- 172.21.15.26:0/290111312 learned_addr learned my addr 172.21.15.26:0/290111312 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:52.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.040+0000 7fb74f7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb75006e7b0 0x7fb750132600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:38010/0 (socket says 172.21.15.26:38010) 2024-08-19T12:42:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb74effd640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.26:6789/0 conn(0x7fb7500a5920 legacy=0x7fb750132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 client.?.objecter ms_handle_connect 0x7fb7500a5920 2024-08-19T12:42:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb74f7fe640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.104:6789/0 conn(0x7fb75006e7b0 legacy=0x7fb750132600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 client.?.objecter ms_handle_connect 0x7fb75006e7b0 2024-08-19T12:42:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb74ffff640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.104:6791/0 conn(0x7fb7500a5d50 legacy=0x7fb7501da6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 client.?.objecter ms_handle_connect 0x7fb7500a5d50 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4001607236 0 0) 0x7fb75009e3b0 con 0x7fb7500a5920 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): my global_id is 144144 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb73c004eb0 con 0x7fb7500a5920 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4234645247 0 0) 0x7fb7500705e0 con 0x7fb7500a5d50 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): my global_id is 25916 2024-08-19T12:42:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb75009e3b0 con 0x7fb7500a5d50 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 724805383 0 0) 0x7fb73c004eb0 con 0x7fb7500a5920 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7500705e0 con 0x7fb7500a5920 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb744004b00 con 0x7fb7500a5920 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): dump: 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.061 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1626582458 0 0) 0x7fb750070300 con 0x7fb75006e7b0 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): my global_id is 26596 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb73c004eb0 con 0x7fb75006e7b0 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1035207969 0 0) 0x7fb75009e3b0 con 0x7fb7500a5d50 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb750070300 con 0x7fb7500a5d50 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb748001ff0 con 0x7fb7500a5d50 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient(hunting): dump: 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.063 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 475019349 0 0) 0x7fb7500705e0 con 0x7fb7500a5920 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.104:6791/0 conn(0x7fb7500a5d50 legacy=0x7fb7501da6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.104:6789/0 conn(0x7fb75006e7b0 legacy=0x7fb750132600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient: _finish_hunting 0 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 monclient: found mon.a 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb750142120 con 0x7fb7500a5920 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient: _finish_auth 0 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 10 monclient: _check_auth_tickets 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb737fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb75602f640 5 monclient: authenticate success, global_id 144144 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb75602f640 10 monclient: _renew_subs 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb75602f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.042+0000 7fb75602f640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7501415c0 con 0x7fb7500a5920 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb75602f640 10 client.144144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:52.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb75602f640 10 monclient: _renew_subs 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb75602f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb75602f640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb750140c50 con 0x7fb7500a5920 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb75602f640 1 librados: init done 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb744004950 con 0x7fb7500a5920 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb737fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb744001c50 con 0x7fb7500a5920 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb737fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb737fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb74dffb640 4 set_mon_vals no callback set 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb737fff640 10 monclient: dump: 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.066 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb75602f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb75602f640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb7300065c0 con 0x7fb730002860 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.043+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fb744011800 con 0x7fb7500a5920 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.044+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(182..182 src has 1..182) ==== 6196+0+0 (unknown 3389665571 0 0) 0x7fb74404e530 con 0x7fb7500a5920 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.044+0000 7fb737fff640 10 client.144144.objecter ms_dispatch 0x7fb750000b90 osd_map(182..182 src has 1..182) 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.044+0000 7fb737fff640 3 client.144144.objecter handle_osd_map got epochs [182,182] > 0 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.044+0000 7fb737fff640 3 client.144144.objecter handle_osd_map decoding full epoch 182 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.044+0000 7fb737fff640 20 client.144144.objecter dump_active .. 0 homeless 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.044+0000 7fb74ffff640 10 monclient: get_auth_request con 0x7fb73c041750 auth_method 0 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb74f7fe640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.26:6790/0 conn(0x7fb730002860 legacy=0x7fb730004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 10 client.144144.objecter ms_handle_connect 0x7fb730002860 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 10 client.144144.objecter resend_mon_ops 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3038666693 0 0) 0x7fb7300065c0 con 0x7fb730002860 2024-08-19T12:42:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 10 monclient(hunting): my global_id is 25967 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7500705e0 con 0x7fb730002860 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2522806752 0 0) 0x7fb7500705e0 con 0x7fb730002860 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7300065c0 con 0x7fb730002860 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb740001b30 con 0x7fb730002860 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 10 monclient: dump: 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2736405434 0 0) 0x7fb7300065c0 con 0x7fb730002860 2024-08-19T12:42:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb73000a160 con 0x7fb730002860 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 10 client.144144.objecter ms_handle_connect 0x7fb73c041750 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb740006920 con 0x7fb730002860 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.26:6790/0 conn(0x7fb730002860 legacy=0x7fb730004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.046+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.26:6790/0 conn(0x7fb730002860 legacy=0x7fb730004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.053+0000 7fb75602f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.053+0000 7fb75602f640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb730009e70 con 0x7fb730009130 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb74f7fe640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.26:6790/0 conn(0x7fb730009130 legacy=0x7fb7300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 10 client.144144.objecter ms_handle_connect 0x7fb730009130 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 10 client.144144.objecter resend_mon_ops 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 763930099 0 0) 0x7fb730009e70 con 0x7fb730009130 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 10 monclient(hunting): my global_id is 25976 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7300065c0 con 0x7fb730009130 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3280634540 0 0) 0x7fb7300065c0 con 0x7fb730009130 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb730009e70 con 0x7fb730009130 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb740005000 con 0x7fb730009130 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 10 monclient: dump: 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1369100300 0 0) 0x7fb730009e70 con 0x7fb730009130 2024-08-19T12:42:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.055+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb73000c3f0 con 0x7fb730009130 2024-08-19T12:42:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 15370340) 0x7fb740006590 con 0x7fb730009130 2024-08-19T12:42:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb737fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb737fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.26:6790/0 conn(0x7fb730009130 legacy=0x7fb7300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb737fff640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.26:6790/0 conn(0x7fb730009130 legacy=0x7fb7300095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:52.074 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 285110, 2024-08-19T12:42:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:52.095 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:52.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 10 librados: watch_flush enter 2024-08-19T12:42:52.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 10 librados: watch_flush exit 2024-08-19T12:42:52.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 20 client.144144.objecter shutdown clearing up homeless session... 2024-08-19T12:42:52.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 10 client.144144.objecter successfully canceled tick 2024-08-19T12:42:52.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.104:6800/899887506 conn(0x7fb73c041750 legacy=0x7fb73c043c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 10 monclient: shutdown 2024-08-19T12:42:52.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:52.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 1 -- 172.21.15.26:0/290111312 >> v1:172.21.15.26:6789/0 conn(0x7fb7500a5920 legacy=0x7fb750132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb74ffff640 1 -- 172.21.15.26:0/290111312 reap_dead start 2024-08-19T12:42:52.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb74ffff640 1 -- 172.21.15.26:0/290111312 reap_dead start 2024-08-19T12:42:52.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 1 -- 172.21.15.26:0/290111312 shutdown_connections 2024-08-19T12:42:52.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 1 -- 172.21.15.26:0/290111312 >> 172.21.15.26:0/290111312 conn(0x7fb7500a8740 msgr2=0x7fb7500aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:52.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 1 -- 172.21.15.26:0/290111312 shutdown_connections 2024-08-19T12:42:52.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 1 -- 172.21.15.26:0/290111312 wait complete. 2024-08-19T12:42:52.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.056+0000 7fb75602f640 1 librados: shutdown 2024-08-19T12:42:52.098 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:52.098 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:42:52.098 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:42:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.196+0000 7f22981b1640 10 monclient: get_monmap_and_config 2024-08-19T12:42:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.196+0000 7f22981b1640 10 monclient: build_initial_monmap 2024-08-19T12:42:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.196+0000 7f22981b1640 10 monclient: monmap: 2024-08-19T12:42:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:52.198 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:52.198393+0000 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:52.198393+0000 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:52.199 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 1 Processor -- start 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 1 -- start start 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 10 monclient: init 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 10 monclient: _add_conns ranks=[4,7,3,1,2,8,5,6,0] 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 10 monclient(hunting): picked mon.noname-c con 0x7f229006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 10 monclient(hunting): picked mon.noname-i con 0x7f22900a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 10 monclient(hunting): picked mon.noname-g con 0x7f22900a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2290136790 con 0x7f22900a5d50 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229009ddf0 con 0x7f229006e7b0 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229009e0d0 con 0x7f22900a5920 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.197+0000 7f22981b1640 10 monclient(hunting): authenticate will time out at 966.285959s 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.198+0000 7f2295f4d640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f229006e7b0 0x7f2290136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:54366/0 (socket says 172.21.15.26:54366) 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.198+0000 7f2295f4d640 1 -- 172.21.15.26:0/1555377845 learned_addr learned my addr 172.21.15.26:0/1555377845 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2295f4d640 1 -- 172.21.15.26:0/1555377845 >> v1:172.21.15.26:6791/0 conn(0x7f229006e7b0 legacy=0x7f2290136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 1 -- 172.21.15.26:0/1555377845 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2265162924 0 0) 0x7f228c001450 con 0x7f229006e7b0 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient(hunting): my global_id is 26176 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 1 -- 172.21.15.26:0/1555377845 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f227c003520 con 0x7f229006e7b0 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f229674e640 1 -- 172.21.15.26:0/1555377845 >> v1:172.21.15.104:6790/0 conn(0x7f22900a5d50 legacy=0x7f2290141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f229574c640 1 -- 172.21.15.26:0/1555377845 >> v1:172.21.15.104:6792/0 conn(0x7f22900a5920 legacy=0x7f229013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 1 -- 172.21.15.26:0/1555377845 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3995919457 0 0) 0x7f227c003520 con 0x7f229006e7b0 2024-08-19T12:42:52.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 1 -- 172.21.15.26:0/1555377845 >> v1:172.21.15.104:6792/0 conn(0x7f22900a5920 legacy=0x7f229013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 1 -- 172.21.15.26:0/1555377845 >> v1:172.21.15.104:6790/0 conn(0x7f22900a5d50 legacy=0x7f2290141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient: _finish_hunting 0 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 1 monclient: found mon.noname-c 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 1 -- 172.21.15.26:0/1555377845 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229009e3b0 con 0x7f229006e7b0 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient: _finish_auth 0 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient: _check_auth_tickets 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f22981b1640 5 monclient: authenticate success, global_id 26176 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f22981b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 1 -- 172.21.15.26:0/1555377845 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f228c001cb0 con 0x7f229006e7b0 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient: dump: 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.208 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f22981b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 1 -- 172.21.15.26:0/1555377845 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f228c004990 con 0x7f229006e7b0 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 1 -- 172.21.15.26:0/1555377845 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f228c004f00 con 0x7f229006e7b0 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f2294f4b640 10 monclient: dump: 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.209 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.210 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f22981b1640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.200+0000 7f22981b1640 4 set_mon_vals no callback set 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.201+0000 7f22981b1640 10 monclient: shutdown 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.201+0000 7f22981b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.201+0000 7f22981b1640 1 -- 172.21.15.26:0/1555377845 >> v1:172.21.15.26:6791/0 conn(0x7f229006e7b0 legacy=0x7f2290136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.201+0000 7f22981b1640 1 -- 172.21.15.26:0/1555377845 shutdown_connections 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.201+0000 7f22981b1640 1 -- 172.21.15.26:0/1555377845 >> 172.21.15.26:0/1555377845 conn(0x7f22900a8740 msgr2=0x7f22900a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.201+0000 7f22981b1640 1 -- 172.21.15.26:0/1555377845 shutdown_connections 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.201+0000 7f22981b1640 1 -- 172.21.15.26:0/1555377845 wait complete. 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.201+0000 7f22981b1640 10 monclient: build_initial_monmap 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.201+0000 7f22981b1640 10 monclient: monmap: 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:52.211 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:52.212 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 1 librados: starting msgr at 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 1 librados: starting objecter 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 1 Processor -- start 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 1 -- start start 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 1 librados: setting wanted keys 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 1 librados: calling monclient init 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 10 monclient: init 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 10 monclient: _add_conns ranks=[2,3,7,6,0,8,4,5,1] 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 10 monclient(hunting): picked mon.noname-b con 0x7f229006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 10 monclient(hunting): picked mon.noname-g con 0x7f22900a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 10 monclient(hunting): picked mon.noname-i con 0x7f22900a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22900a3530 con 0x7f229006e7b0 2024-08-19T12:42:52.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22901349f0 con 0x7f22900a5920 2024-08-19T12:42:52.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22900a11d0 con 0x7f22900a5d50 2024-08-19T12:42:52.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:52.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f22981b1640 10 monclient(hunting): authenticate will time out at 966.290799s 2024-08-19T12:42:52.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f2295f4d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f229006e7b0 0x7f2290132a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51682/0 (socket says 172.21.15.26:51682) 2024-08-19T12:42:52.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.202+0000 7f2295f4d640 1 -- 172.21.15.26:0/2041475550 learned_addr learned my addr 172.21.15.26:0/2041475550 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:52.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.204+0000 7f2295f4d640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.26:6790/0 conn(0x7f229006e7b0 legacy=0x7f2290132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.204+0000 7f22867fc640 10 client.?.objecter ms_handle_connect 0x7f229006e7b0 2024-08-19T12:42:52.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.204+0000 7f22867fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2815221327 0 0) 0x7f22900a3530 con 0x7f229006e7b0 2024-08-19T12:42:52.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): my global_id is 25985 2024-08-19T12:42:52.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f229674e640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.104:6792/0 conn(0x7f22900a5d50 legacy=0x7f22901d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2274004eb0 con 0x7f229006e7b0 2024-08-19T12:42:52.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f229574c640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.104:6790/0 conn(0x7f22900a5920 legacy=0x7f22900a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 client.?.objecter ms_handle_connect 0x7f22900a5d50 2024-08-19T12:42:52.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 client.?.objecter ms_handle_connect 0x7f22900a5920 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3479332237 0 0) 0x7f22901349f0 con 0x7f22900a5920 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): my global_id is 26121 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22900a3530 con 0x7f22900a5920 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 407311599 0 0) 0x7f2274004eb0 con 0x7f229006e7b0 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22901349f0 con 0x7f229006e7b0 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1030943216 0 0) 0x7f22900a11d0 con 0x7f22900a5d50 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): my global_id is 134737 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2274004eb0 con 0x7f22900a5d50 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f228c004e80 con 0x7f229006e7b0 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): dump: 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.221 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 830934683 0 0) 0x7f22901349f0 con 0x7f229006e7b0 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.104:6792/0 conn(0x7f22900a5d50 legacy=0x7f22901d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.104:6790/0 conn(0x7f22900a5920 legacy=0x7f22900a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient: _finish_hunting 0 2024-08-19T12:42:52.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 monclient: found mon.b 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22900a2c90 con 0x7f229006e7b0 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient: _finish_auth 0 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient: _check_auth_tickets 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22981b1640 5 monclient: authenticate success, global_id 25985 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22981b1640 10 monclient: _renew_subs 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22981b1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22981b1640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22900a2f70 con 0x7f229006e7b0 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22981b1640 10 client.25985.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22981b1640 10 monclient: _renew_subs 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22981b1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:42:52.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22981b1640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f22900a53a0 con 0x7f229006e7b0 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22981b1640 1 librados: init done 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f228c004990 con 0x7f229006e7b0 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f228c005580 con 0x7f229006e7b0 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f2287fff640 4 set_mon_vals no callback set 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.205+0000 7f22867fc640 10 monclient: dump: 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.224 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.206+0000 7f22981b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.206+0000 7f22981b1640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f22780065c0 con 0x7f2278002860 2024-08-19T12:42:52.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.206+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f228c0119e0 con 0x7f229006e7b0 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.207+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(182..182 src has 1..182) ==== 6196+0+0 (unknown 3389665571 0 0) 0x7f228c04e5e0 con 0x7f229006e7b0 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.207+0000 7f22867fc640 10 client.25985.objecter ms_dispatch 0x7f2290000b90 osd_map(182..182 src has 1..182) 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.207+0000 7f22867fc640 3 client.25985.objecter handle_osd_map got epochs [182,182] > 0 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.207+0000 7f22867fc640 3 client.25985.objecter handle_osd_map decoding full epoch 182 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.207+0000 7f22867fc640 20 client.25985.objecter dump_active .. 0 homeless 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.207+0000 7f229674e640 10 monclient: get_auth_request con 0x7f2274040f60 auth_method 0 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.208+0000 7f22867fc640 10 client.25985.objecter ms_handle_connect 0x7f2278002860 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.208+0000 7f229574c640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.26:6791/0 conn(0x7f2278002860 legacy=0x7f2278004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.208+0000 7f22867fc640 10 client.25985.objecter resend_mon_ops 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744622162 0 0) 0x7f22780065c0 con 0x7f2278002860 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 10 monclient(hunting): my global_id is 26185 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22901349f0 con 0x7f2278002860 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 10 client.25985.objecter ms_handle_connect 0x7f2274040f60 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 51551254 0 0) 0x7f22901349f0 con 0x7f2278002860 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22780065c0 con 0x7f2278002860 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2280003dd0 con 0x7f2278002860 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 10 monclient: dump: 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.227 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1780849619 0 0) 0x7f22780065c0 con 0x7f2278002860 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f227800a160 con 0x7f2278002860 2024-08-19T12:42:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2280004000 con 0x7f2278002860 2024-08-19T12:42:52.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:52.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:52.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.26:6791/0 conn(0x7f2278002860 legacy=0x7f2278004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.209+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.26:6791/0 conn(0x7f2278002860 legacy=0x7f2278004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:52.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.216+0000 7f22981b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:52.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.216+0000 7f22981b1640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2278009e50 con 0x7f2278009110 2024-08-19T12:42:52.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f229574c640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.26:6791/0 conn(0x7f2278009110 legacy=0x7f22780095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 10 client.25985.objecter ms_handle_connect 0x7f2278009110 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 10 client.25985.objecter resend_mon_ops 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1566764570 0 0) 0x7f2278009e50 con 0x7f2278009110 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 10 monclient(hunting): my global_id is 26194 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22780065c0 con 0x7f2278009110 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3552124383 0 0) 0x7f22780065c0 con 0x7f2278009110 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2278009e50 con 0x7f2278009110 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2280005b00 con 0x7f2278009110 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 10 monclient: dump: 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.232 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.233 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.233 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.233 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.233 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1731754512 0 0) 0x7f2278009e50 con 0x7f2278009110 2024-08-19T12:42:52.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.218+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f227800c3a0 con 0x7f2278009110 2024-08-19T12:42:52.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3141619341) 0x7f2280007e30 con 0x7f2278009110 2024-08-19T12:42:52.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22867fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:52.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22867fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:52.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.26:6791/0 conn(0x7f2278009110 legacy=0x7f22780095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22867fc640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.26:6791/0 conn(0x7f2278009110 legacy=0x7f22780095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:52.234 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:52.234 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:52.234 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:52.234 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:52.234 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:52.234 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 285248, 2024-08-19T12:42:52.234 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:52.234 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:52.234 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:52.234 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:52.234 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.235 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.236 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.236 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.236 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.236 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.236 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.236 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.236 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.236 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.236 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.236 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:52.236 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:52.236 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:52.237 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:52.238 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.239 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.240 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.244 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:52.247 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.248 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.248 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.248 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:52.248 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:52.248 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:52.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.252 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.253 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.254 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:52.254 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:52.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 10 librados: watch_flush enter 2024-08-19T12:42:52.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 10 librados: watch_flush exit 2024-08-19T12:42:52.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 20 client.25985.objecter shutdown clearing up homeless session... 2024-08-19T12:42:52.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 10 client.25985.objecter successfully canceled tick 2024-08-19T12:42:52.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.104:6800/899887506 conn(0x7f2274040f60 legacy=0x7f2274043420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 10 monclient: shutdown 2024-08-19T12:42:52.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:52.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 1 -- 172.21.15.26:0/2041475550 >> v1:172.21.15.26:6790/0 conn(0x7f229006e7b0 legacy=0x7f2290132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f229674e640 1 -- 172.21.15.26:0/2041475550 reap_dead start 2024-08-19T12:42:52.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f229674e640 1 -- 172.21.15.26:0/2041475550 reap_dead start 2024-08-19T12:42:52.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 1 -- 172.21.15.26:0/2041475550 shutdown_connections 2024-08-19T12:42:52.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 1 -- 172.21.15.26:0/2041475550 >> 172.21.15.26:0/2041475550 conn(0x7f22900a8740 msgr2=0x7f22900aa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:52.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 1 -- 172.21.15.26:0/2041475550 shutdown_connections 2024-08-19T12:42:52.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 1 -- 172.21.15.26:0/2041475550 wait complete. 2024-08-19T12:42:52.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.219+0000 7f22981b1640 1 librados: shutdown 2024-08-19T12:42:52.257 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:52.257 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:42:52.257 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:42:52.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.356+0000 7fee1b353640 10 monclient: get_monmap_and_config 2024-08-19T12:42:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.356+0000 7fee1b353640 10 monclient: build_initial_monmap 2024-08-19T12:42:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.356+0000 7fee1b353640 10 monclient: monmap: 2024-08-19T12:42:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:52.358012+0000 2024-08-19T12:42:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:52.358012+0000 2024-08-19T12:42:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:52.358 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:52.359 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:52.359 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:52.359 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:52.359 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:52.359 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:52.359 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:52.359 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.356+0000 7fee1b353640 1 Processor -- start 2024-08-19T12:42:52.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.356+0000 7fee1b353640 1 -- start start 2024-08-19T12:42:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.356+0000 7fee1b353640 10 monclient: init 2024-08-19T12:42:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee1b353640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee1b353640 10 monclient: _add_conns ranks=[6,0,4,8,7,5,2,3,1] 2024-08-19T12:42:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee1b353640 10 monclient(hunting): picked mon.noname-d con 0x7fee14073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee1b353640 10 monclient(hunting): picked mon.noname-a con 0x7fee140a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee1b353640 10 monclient(hunting): picked mon.noname-c con 0x7fee1413f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee1b353640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee1409dae0 con 0x7fee140a57a0 2024-08-19T12:42:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee1b353640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee14136910 con 0x7fee1413f2e0 2024-08-19T12:42:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee1b353640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee1409dd50 con 0x7fee14073470 2024-08-19T12:42:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee1b353640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee1b353640 10 monclient(hunting): authenticate will time out at 966.445494s 2024-08-19T12:42:52.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee198f0640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fee1413f2e0 0x7fee141416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:54384/0 (socket says 172.21.15.26:54384) 2024-08-19T12:42:52.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee190ef640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fee14073470 0x7fee14136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34498/0 (socket says 172.21.15.26:34498) 2024-08-19T12:42:52.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee198f0640 1 -- 172.21.15.26:0/4029573919 learned_addr learned my addr 172.21.15.26:0/4029573919 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:52.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.357+0000 7fee188ee640 1 --1- 172.21.15.26:0/4029573919 >> v1:172.21.15.26:6789/0 conn(0x7fee140a57a0 0x7fee140a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57878/0 (socket says 172.21.15.26:57878) 2024-08-19T12:42:52.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.359+0000 7fee188ee640 1 -- 172.21.15.26:0/4029573919 >> v1:172.21.15.26:6789/0 conn(0x7fee140a57a0 legacy=0x7fee140a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.359+0000 7fee190ef640 1 -- 172.21.15.26:0/4029573919 >> v1:172.21.15.26:6792/0 conn(0x7fee14073470 legacy=0x7fee14136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.359+0000 7fee198f0640 1 -- 172.21.15.26:0/4029573919 >> v1:172.21.15.26:6791/0 conn(0x7fee1413f2e0 legacy=0x7fee141416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.359+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 120136175 0 0) 0x7fee1409dae0 con 0x7fee140a57a0 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.359+0000 7fee0bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.359+0000 7fee0bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.359+0000 7fee0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.359+0000 7fee0bfff640 10 monclient(hunting): my global_id is 144153 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee00003520 con 0x7fee140a57a0 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2183517232 0 0) 0x7fee1409dd50 con 0x7fee14073470 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient(hunting): my global_id is 26214 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee1409dae0 con 0x7fee14073470 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3579750286 0 0) 0x7fee14136910 con 0x7fee1413f2e0 2024-08-19T12:42:52.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient(hunting): my global_id is 26203 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee1409dd50 con 0x7fee1413f2e0 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee190ef640 1 --1- 172.21.15.26:0/4029573919 >> v1:172.21.15.26:6792/0 conn(0x7fee14073470 0x7fee14136200 :-1 s=OPENED pgs=3130 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fee10001ac0 mon_map magic: 0 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2678795015 0 0) 0x7fee00003520 con 0x7fee140a57a0 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 >> v1:172.21.15.26:6792/0 conn(0x7fee14073470 legacy=0x7fee14136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 >> v1:172.21.15.26:6791/0 conn(0x7fee1413f2e0 legacy=0x7fee141416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 monclient: found mon.noname-a 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee1409e030 con 0x7fee140a57a0 2024-08-19T12:42:52.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: _finish_auth 0 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee04001ae0 con 0x7fee140a57a0 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee1b353640 5 monclient: authenticate success, global_id 144153 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee1b353640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: dump: 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.368 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee0c001d80 con 0x7fee1413f2e0 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee1b353640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee10001ac0 con 0x7fee14073470 2024-08-19T12:42:52.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee040047e0 con 0x7fee140a57a0 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee1b353640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 1 -- 172.21.15.26:0/4029573919 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee04004d50 con 0x7fee140a57a0 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee1b353640 4 set_mon_vals no callback set 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee0bfff640 10 monclient: dump: 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.370 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee1b353640 10 monclient: shutdown 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee1b353640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee1b353640 1 -- 172.21.15.26:0/4029573919 >> v1:172.21.15.26:6789/0 conn(0x7fee140a57a0 legacy=0x7fee140a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee1b353640 1 -- 172.21.15.26:0/4029573919 shutdown_connections 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.360+0000 7fee1b353640 1 -- 172.21.15.26:0/4029573919 >> 172.21.15.26:0/4029573919 conn(0x7fee140abff0 msgr2=0x7fee140ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.361+0000 7fee1b353640 1 -- 172.21.15.26:0/4029573919 shutdown_connections 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.361+0000 7fee1b353640 1 -- 172.21.15.26:0/4029573919 wait complete. 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.361+0000 7fee1b353640 10 monclient: build_initial_monmap 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.361+0000 7fee1b353640 10 monclient: monmap: 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:52.372 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.361+0000 7fee1b353640 1 librados: starting msgr at 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.361+0000 7fee1b353640 1 librados: starting objecter 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.361+0000 7fee1b353640 1 Processor -- start 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 1 -- start start 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 1 librados: setting wanted keys 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 1 librados: calling monclient init 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 10 monclient: init 2024-08-19T12:42:52.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:52.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 10 monclient: _add_conns ranks=[3,5,0,7,2,8,1,4,6] 2024-08-19T12:42:52.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 10 monclient(hunting): picked mon.noname-g con 0x7fee14073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:52.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 10 monclient(hunting): picked mon.noname-h con 0x7fee140a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:42:52.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 10 monclient(hunting): picked mon.noname-a con 0x7fee1413f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:52.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee1413dff0 con 0x7fee1413f2e0 2024-08-19T12:42:52.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee141cef20 con 0x7fee14073470 2024-08-19T12:42:52.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee14135040 con 0x7fee140a57a0 2024-08-19T12:42:52.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:52.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee1b353640 10 monclient(hunting): authenticate will time out at 966.450553s 2024-08-19T12:42:52.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee198f0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fee1413f2e0 0x7fee141d6360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57888/0 (socket says 172.21.15.26:57888) 2024-08-19T12:42:52.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.362+0000 7fee198f0640 1 -- 172.21.15.26:0/231674315 learned_addr learned my addr 172.21.15.26:0/231674315 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:52.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee198f0640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.26:6789/0 conn(0x7fee1413f2e0 legacy=0x7fee141d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee09ffb640 10 client.?.objecter ms_handle_connect 0x7fee1413f2e0 2024-08-19T12:42:52.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee09ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2339481535 0 0) 0x7fee1413dff0 con 0x7fee1413f2e0 2024-08-19T12:42:52.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee09ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee09ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee09ffb640 10 monclient(hunting): my global_id is 144162 2024-08-19T12:42:52.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedf8004eb0 con 0x7fee1413f2e0 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee190ef640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.104:6790/0 conn(0x7fee14073470 legacy=0x7fee141ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee09ffb640 10 client.?.objecter ms_handle_connect 0x7fee14073470 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee09ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee09ffb640 10 client.?.objecter ms_handle_connect 0x7fee140a57a0 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee09ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.364+0000 7fee188ee640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.104:6791/0 conn(0x7fee140a57a0 legacy=0x7fee141ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1022428826 0 0) 0x7fedf8004eb0 con 0x7fee1413f2e0 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee1413dff0 con 0x7fee1413f2e0 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee0c002030 con 0x7fee1413f2e0 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient(hunting): dump: 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.380 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.381 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3909554725 0 0) 0x7fee14135040 con 0x7fee140a57a0 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient(hunting): my global_id is 25925 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedf8004eb0 con 0x7fee140a57a0 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3346986352 0 0) 0x7fee141cef20 con 0x7fee14073470 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient(hunting): my global_id is 26130 2024-08-19T12:42:52.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee14135040 con 0x7fee14073470 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 844947311 0 0) 0x7fee1413dff0 con 0x7fee1413f2e0 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.104:6791/0 conn(0x7fee140a57a0 legacy=0x7fee141ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.104:6790/0 conn(0x7fee14073470 legacy=0x7fee141ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 monclient: found mon.a 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee140a3a40 con 0x7fee1413f2e0 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient: _finish_auth 0 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee1b353640 5 monclient: authenticate success, global_id 144162 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee1b353640 10 monclient: _renew_subs 2024-08-19T12:42:52.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee1b353640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee1b353640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fee1413e940 con 0x7fee1413f2e0 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee0c005150 con 0x7fee1413f2e0 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee0c005640 con 0x7fee1413f2e0 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee09ffb640 10 monclient: dump: 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.384 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee0b7fe640 4 set_mon_vals no callback set 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee1b353640 10 client.144162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee1b353640 10 monclient: _renew_subs 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee1b353640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:52.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee1b353640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fee14137680 con 0x7fee1413f2e0 2024-08-19T12:42:52.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.365+0000 7fee1b353640 1 librados: init done 2024-08-19T12:42:52.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.366+0000 7fee1b353640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:52.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.366+0000 7fee1b353640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fee141371c0 con 0x7fee14142df0 2024-08-19T12:42:52.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.366+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fee0c011b20 con 0x7fee1413f2e0 2024-08-19T12:42:52.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.366+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(182..182 src has 1..182) ==== 6196+0+0 (unknown 3389665571 0 0) 0x7fee0c04e920 con 0x7fee1413f2e0 2024-08-19T12:42:52.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.366+0000 7fee09ffb640 10 client.144162.objecter ms_dispatch 0x7fee14000b90 osd_map(182..182 src has 1..182) 2024-08-19T12:42:52.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.366+0000 7fee09ffb640 3 client.144162.objecter handle_osd_map got epochs [182,182] > 0 2024-08-19T12:42:52.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.366+0000 7fee09ffb640 3 client.144162.objecter handle_osd_map decoding full epoch 182 2024-08-19T12:42:52.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.367+0000 7fee09ffb640 20 client.144162.objecter dump_active .. 0 homeless 2024-08-19T12:42:52.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.367+0000 7fee188ee640 10 monclient: get_auth_request con 0x7fedf80411e0 auth_method 0 2024-08-19T12:42:52.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.368+0000 7fee190ef640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.26:6792/0 conn(0x7fee14142df0 legacy=0x7fee141416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.368+0000 7fee09ffb640 10 client.144162.objecter ms_handle_connect 0x7fee14142df0 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.368+0000 7fee09ffb640 10 client.144162.objecter resend_mon_ops 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 342221285 0 0) 0x7fee141371c0 con 0x7fee14142df0 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 10 monclient(hunting): my global_id is 26223 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee1413dff0 con 0x7fee14142df0 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2677202438 0 0) 0x7fee1413dff0 con 0x7fee14142df0 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee141371c0 con 0x7fee14142df0 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee10001af0 con 0x7fee14142df0 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 10 monclient: dump: 2024-08-19T12:42:52.387 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.388 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.389 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 10 client.144162.objecter ms_handle_connect 0x7fedf80411e0 2024-08-19T12:42:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2677276200 0 0) 0x7fee141371c0 con 0x7fee14142df0 2024-08-19T12:42:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee1413ee10 con 0x7fee14142df0 2024-08-19T12:42:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fee10006900 con 0x7fee14142df0 2024-08-19T12:42:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.26:6792/0 conn(0x7fee14142df0 legacy=0x7fee141416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.369+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.26:6792/0 conn(0x7fee14142df0 legacy=0x7fee141416d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.375+0000 7fee1b353640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:52.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.376+0000 7fee1b353640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fee1406c840 con 0x7fee1402e1d0 2024-08-19T12:42:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 10 client.144162.objecter ms_handle_connect 0x7fee1402e1d0 2024-08-19T12:42:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 10 client.144162.objecter resend_mon_ops 2024-08-19T12:42:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee190ef640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.26:6792/0 conn(0x7fee1402e1d0 legacy=0x7fee14141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2457991355 0 0) 0x7fee1406c840 con 0x7fee1402e1d0 2024-08-19T12:42:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 10 monclient(hunting): my global_id is 26232 2024-08-19T12:42:52.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee141371c0 con 0x7fee1402e1d0 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4159167808 0 0) 0x7fee141371c0 con 0x7fee1402e1d0 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee1406c840 con 0x7fee1402e1d0 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee100050f0 con 0x7fee1402e1d0 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 10 monclient: dump: 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.392 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4034591497 0 0) 0x7fee1406c840 con 0x7fee1402e1d0 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fee14135490 con 0x7fee1402e1d0 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 2796410012) 0x7fee10006550 con 0x7fee1402e1d0 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:52.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.26:6792/0 conn(0x7fee1402e1d0 legacy=0x7fee14141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.378+0000 7fee09ffb640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.26:6792/0 conn(0x7fee1402e1d0 legacy=0x7fee14141de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:52.394 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:52.394 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:52.394 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:52.394 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:52.394 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:52.394 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 285419, 2024-08-19T12:42:52.394 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:52.394 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:52.395 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:52.396 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.397 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:52.398 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.399 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.400 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.400 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.400 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:52.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.400 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:52.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.400 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:52.400 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:52.400 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.400 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.402 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:52.402 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:52.402 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.402 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:52.402 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.402 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.402 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:52.405 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:52.406 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.407 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:52.409 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:52.410 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.410 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.410 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.410 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:52.410 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:52.410 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.410 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.410 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.410 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:52.411 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.412 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.414 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:52.414 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:52.414 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.414 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.414 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.414 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:52.414 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:52.414 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.414 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.415 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.416 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.417 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.417 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.417 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:52.417 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 10 librados: watch_flush enter 2024-08-19T12:42:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 10 librados: watch_flush exit 2024-08-19T12:42:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 20 client.144162.objecter shutdown clearing up homeless session... 2024-08-19T12:42:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 10 client.144162.objecter successfully canceled tick 2024-08-19T12:42:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.104:6800/899887506 conn(0x7fedf80411e0 legacy=0x7fedf80436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 10 monclient: shutdown 2024-08-19T12:42:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 1 -- 172.21.15.26:0/231674315 >> v1:172.21.15.26:6789/0 conn(0x7fee1413f2e0 legacy=0x7fee141d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee198f0640 1 -- 172.21.15.26:0/231674315 reap_dead start 2024-08-19T12:42:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee198f0640 1 -- 172.21.15.26:0/231674315 reap_dead start 2024-08-19T12:42:52.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 1 -- 172.21.15.26:0/231674315 shutdown_connections 2024-08-19T12:42:52.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 1 -- 172.21.15.26:0/231674315 >> 172.21.15.26:0/231674315 conn(0x7fee140abff0 msgr2=0x7fee14134810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:52.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 1 -- 172.21.15.26:0/231674315 shutdown_connections 2024-08-19T12:42:52.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 1 -- 172.21.15.26:0/231674315 wait complete. 2024-08-19T12:42:52.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.379+0000 7fee1b353640 1 librados: shutdown 2024-08-19T12:42:52.420 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:52.420 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:42:52.420 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:42:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.518+0000 7fce9e838640 10 monclient: get_monmap_and_config 2024-08-19T12:42:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.518+0000 7fce9e838640 10 monclient: build_initial_monmap 2024-08-19T12:42:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.518+0000 7fce9e838640 10 monclient: monmap: 2024-08-19T12:42:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:52.520074+0000 2024-08-19T12:42:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:52.520074+0000 2024-08-19T12:42:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:52.521 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 1 Processor -- start 2024-08-19T12:42:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 1 -- start start 2024-08-19T12:42:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 10 monclient: init 2024-08-19T12:42:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 10 monclient: _add_conns ranks=[7,6,2,8,4,3,0,1,5] 2024-08-19T12:42:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 10 monclient(hunting): picked mon.noname-i con 0x7fce98073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 10 monclient(hunting): picked mon.noname-d con 0x7fce980a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 10 monclient(hunting): picked mon.noname-b con 0x7fce9813f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:42:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce9809dae0 con 0x7fce9813f2e0 2024-08-19T12:42:52.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce98136910 con 0x7fce980a57a0 2024-08-19T12:42:52.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce9809dd50 con 0x7fce98073470 2024-08-19T12:42:52.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:52.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce9e838640 10 monclient(hunting): authenticate will time out at 966.607733s 2024-08-19T12:42:52.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce8ffff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fce980a57a0 0x7fce980a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34516/0 (socket says 172.21.15.26:34516) 2024-08-19T12:42:52.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.519+0000 7fce8ffff640 1 -- 172.21.15.26:0/3621658516 learned_addr learned my addr 172.21.15.26:0/3621658516 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.521+0000 7fce8ffff640 1 -- 172.21.15.26:0/3621658516 >> v1:172.21.15.26:6792/0 conn(0x7fce980a57a0 legacy=0x7fce980a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.521+0000 7fce9cdd5640 1 -- 172.21.15.26:0/3621658516 >> v1:172.21.15.26:6790/0 conn(0x7fce9813f2e0 legacy=0x7fce981416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 1 -- 172.21.15.26:0/3621658516 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3538571799 0 0) 0x7fce98136910 con 0x7fce980a57a0 2024-08-19T12:42:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient(hunting): my global_id is 26241 2024-08-19T12:42:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 1 -- 172.21.15.26:0/3621658516 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce84003520 con 0x7fce980a57a0 2024-08-19T12:42:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 1 -- 172.21.15.26:0/3621658516 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1088134057 0 0) 0x7fce9809dae0 con 0x7fce9813f2e0 2024-08-19T12:42:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient(hunting): my global_id is 25994 2024-08-19T12:42:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 1 -- 172.21.15.26:0/3621658516 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce98136910 con 0x7fce9813f2e0 2024-08-19T12:42:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce97fff640 1 -- 172.21.15.26:0/3621658516 >> v1:172.21.15.104:6792/0 conn(0x7fce98073470 legacy=0x7fce98136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 1 -- 172.21.15.26:0/3621658516 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1417983141 0 0) 0x7fce98136910 con 0x7fce9813f2e0 2024-08-19T12:42:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 1 -- 172.21.15.26:0/3621658516 >> v1:172.21.15.104:6792/0 conn(0x7fce98073470 legacy=0x7fce98136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 1 -- 172.21.15.26:0/3621658516 >> v1:172.21.15.26:6792/0 conn(0x7fce980a57a0 legacy=0x7fce980a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient: _finish_hunting 0 2024-08-19T12:42:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 1 monclient: found mon.noname-b 2024-08-19T12:42:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:42:52.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 1 -- 172.21.15.26:0/3621658516 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce9809e030 con 0x7fce9813f2e0 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient: _finish_auth 0 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient: _check_auth_tickets 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 1 -- 172.21.15.26:0/3621658516 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fce90001dc0 con 0x7fce9813f2e0 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient: dump: 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.530 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce9e838640 5 monclient: authenticate success, global_id 25994 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce9e838640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 1 -- 172.21.15.26:0/3621658516 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fce90002090 con 0x7fce9813f2e0 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:52.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 1 -- 172.21.15.26:0/3621658516 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fce90003cd0 con 0x7fce9813f2e0 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce9e838640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce9e838640 4 set_mon_vals no callback set 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.522+0000 7fce977fe640 10 monclient: dump: 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.532 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.523+0000 7fce9e838640 10 monclient: shutdown 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.523+0000 7fce9e838640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.523+0000 7fce9e838640 1 -- 172.21.15.26:0/3621658516 >> v1:172.21.15.26:6790/0 conn(0x7fce9813f2e0 legacy=0x7fce981416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.523+0000 7fce9e838640 1 -- 172.21.15.26:0/3621658516 shutdown_connections 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.523+0000 7fce9e838640 1 -- 172.21.15.26:0/3621658516 >> 172.21.15.26:0/3621658516 conn(0x7fce980abff0 msgr2=0x7fce980ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.523+0000 7fce9e838640 1 -- 172.21.15.26:0/3621658516 shutdown_connections 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.523+0000 7fce9e838640 1 -- 172.21.15.26:0/3621658516 wait complete. 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.523+0000 7fce9e838640 10 monclient: build_initial_monmap 2024-08-19T12:42:52.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.523+0000 7fce9e838640 10 monclient: monmap: 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:52.534 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.523+0000 7fce9e838640 1 librados: starting msgr at 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.523+0000 7fce9e838640 1 librados: starting objecter 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 1 Processor -- start 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 1 -- start start 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 1 librados: setting wanted keys 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 1 librados: calling monclient init 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 10 monclient: init 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 10 monclient: _add_conns ranks=[1,4,7,5,2,8,0,6,3] 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 10 monclient(hunting): picked mon.noname-f con 0x7fce98073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:52.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 10 monclient(hunting): picked mon.noname-c con 0x7fce980a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 10 monclient(hunting): picked mon.noname-i con 0x7fce9813f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce98135040 con 0x7fce98073470 2024-08-19T12:42:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce980a3ad0 con 0x7fce980a57a0 2024-08-19T12:42:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce980a2f30 con 0x7fce9813f2e0 2024-08-19T12:42:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce9e838640 10 monclient(hunting): authenticate will time out at 966.612559s 2024-08-19T12:42:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce8ffff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fce980a57a0 0x7fce980a2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:54390/0 (socket says 172.21.15.26:54390) 2024-08-19T12:42:52.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.524+0000 7fce8ffff640 1 -- 172.21.15.26:0/2853863964 learned_addr learned my addr 172.21.15.26:0/2853863964 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.526+0000 7fce8ffff640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.26:6791/0 conn(0x7fce980a57a0 legacy=0x7fce980a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.526+0000 7fce957fa640 10 client.?.objecter ms_handle_connect 0x7fce980a57a0 2024-08-19T12:42:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.526+0000 7fce957fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.526+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3997392330 0 0) 0x7fce980a3ad0 con 0x7fce980a57a0 2024-08-19T12:42:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.526+0000 7fce957fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.526+0000 7fce957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.526+0000 7fce957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.526+0000 7fce957fa640 10 monclient(hunting): my global_id is 26212 2024-08-19T12:42:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.526+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce78004eb0 con 0x7fce980a57a0 2024-08-19T12:42:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 client.?.objecter ms_handle_connect 0x7fce9813f2e0 2024-08-19T12:42:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce9cdd5640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.104:6792/0 conn(0x7fce9813f2e0 legacy=0x7fce981d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce97fff640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.104:6789/0 conn(0x7fce98073470 legacy=0x7fce980a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 client.?.objecter ms_handle_connect 0x7fce98073470 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 110907958 0 0) 0x7fce78004eb0 con 0x7fce980a57a0 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce980a3ad0 con 0x7fce980a57a0 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fce800047e0 con 0x7fce980a57a0 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient(hunting): dump: 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.542 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 441683059 0 0) 0x7fce980a2f30 con 0x7fce9813f2e0 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient(hunting): my global_id is 134755 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce78004eb0 con 0x7fce9813f2e0 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3292786597 0 0) 0x7fce9809dd50 con 0x7fce98073470 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient(hunting): my global_id is 26605 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce980a2f30 con 0x7fce98073470 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4157196647 0 0) 0x7fce980a3ad0 con 0x7fce980a57a0 2024-08-19T12:42:52.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.104:6792/0 conn(0x7fce9813f2e0 legacy=0x7fce981d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.104:6789/0 conn(0x7fce98073470 legacy=0x7fce980a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient: _finish_hunting 0 2024-08-19T12:42:52.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 monclient: found mon.c 2024-08-19T12:42:52.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:42:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce981372f0 con 0x7fce980a57a0 2024-08-19T12:42:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient: _finish_auth 0 2024-08-19T12:42:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient: _check_auth_tickets 2024-08-19T12:42:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce9e838640 5 monclient: authenticate success, global_id 26212 2024-08-19T12:42:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce9e838640 10 monclient: _renew_subs 2024-08-19T12:42:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce9e838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:42:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce9e838640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fce980a4900 con 0x7fce980a57a0 2024-08-19T12:42:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce9e838640 10 client.26212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:52.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce9e838640 10 monclient: _renew_subs 2024-08-19T12:42:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce9e838640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:42:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce9e838640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fce98137710 con 0x7fce980a57a0 2024-08-19T12:42:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce9e838640 1 librados: init done 2024-08-19T12:42:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fce80004d20 con 0x7fce980a57a0 2024-08-19T12:42:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fce80005290 con 0x7fce980a57a0 2024-08-19T12:42:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce96ffd640 4 set_mon_vals no callback set 2024-08-19T12:42:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.527+0000 7fce957fa640 10 monclient: dump: 2024-08-19T12:42:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.547 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.548 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.549 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.528+0000 7fce9e838640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:52.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.528+0000 7fce9e838640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fce7c0065c0 con 0x7fce7c002860 2024-08-19T12:42:52.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.528+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fce80005430 con 0x7fce980a57a0 2024-08-19T12:42:52.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.528+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(182..182 src has 1..182) ==== 6196+0+0 (unknown 3389665571 0 0) 0x7fce8004e6a0 con 0x7fce980a57a0 2024-08-19T12:42:52.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.528+0000 7fce957fa640 10 client.26212.objecter ms_dispatch 0x7fce98000b90 osd_map(182..182 src has 1..182) 2024-08-19T12:42:52.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.528+0000 7fce957fa640 3 client.26212.objecter handle_osd_map got epochs [182,182] > 0 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.528+0000 7fce957fa640 3 client.26212.objecter handle_osd_map decoding full epoch 182 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.528+0000 7fce957fa640 20 client.26212.objecter dump_active .. 0 homeless 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.528+0000 7fce9cdd5640 10 monclient: get_auth_request con 0x7fce780411b0 auth_method 0 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce97fff640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.26:6793/0 conn(0x7fce7c002860 legacy=0x7fce7c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce957fa640 10 client.26212.objecter ms_handle_connect 0x7fce7c002860 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce957fa640 10 client.26212.objecter resend_mon_ops 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3563133801 0 0) 0x7fce7c0065c0 con 0x7fce7c002860 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce957fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce957fa640 10 monclient(hunting): my global_id is 25739 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce980a3ad0 con 0x7fce7c002860 2024-08-19T12:42:52.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 735834228 0 0) 0x7fce980a3ad0 con 0x7fce7c002860 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce7c0065c0 con 0x7fce7c002860 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fce88001cf0 con 0x7fce7c002860 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.530+0000 7fce957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.531+0000 7fce957fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.531+0000 7fce957fa640 10 monclient: dump: 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.551 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.531+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 713397236 0 0) 0x7fce7c0065c0 con 0x7fce7c002860 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.531+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce7c00a160 con 0x7fce7c002860 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.531+0000 7fce957fa640 10 client.26212.objecter ms_handle_connect 0x7fce780411b0 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.531+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fce880057c0 con 0x7fce7c002860 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.531+0000 7fce957fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.531+0000 7fce957fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.531+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.26:6793/0 conn(0x7fce7c002860 legacy=0x7fce7c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.531+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.26:6793/0 conn(0x7fce7c002860 legacy=0x7fce7c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:52.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.537+0000 7fce9e838640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:52.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.537+0000 7fce9e838640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fce7c009e50 con 0x7fce7c009110 2024-08-19T12:42:52.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.539+0000 7fce97fff640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.26:6793/0 conn(0x7fce7c009110 legacy=0x7fce7c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.539+0000 7fce957fa640 10 client.26212.objecter ms_handle_connect 0x7fce7c009110 2024-08-19T12:42:52.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.539+0000 7fce957fa640 10 client.26212.objecter resend_mon_ops 2024-08-19T12:42:52.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3694443986 0 0) 0x7fce7c009e50 con 0x7fce7c009110 2024-08-19T12:42:52.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 10 monclient(hunting): my global_id is 25748 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce7c0065c0 con 0x7fce7c009110 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1524438159 0 0) 0x7fce7c0065c0 con 0x7fce7c009110 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce7c009e50 con 0x7fce7c009110 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fce88003e30 con 0x7fce7c009110 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 10 monclient: dump: 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.555 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 119082107 0 0) 0x7fce7c009e50 con 0x7fce7c009110 2024-08-19T12:42:52.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fce7c00c3f0 con 0x7fce7c009110 2024-08-19T12:42:52.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 2016180825) 0x7fce880040f0 con 0x7fce7c009110 2024-08-19T12:42:52.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:52.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:52.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.26:6793/0 conn(0x7fce7c009110 legacy=0x7fce7c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce957fa640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.26:6793/0 conn(0x7fce7c009110 legacy=0x7fce7c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:52.557 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:52.557 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:52.557 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 285557, 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:52.558 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:52.559 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.559 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:42:52.559 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:52.559 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:52.559 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:52.559 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.559 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.559 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.559 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.559 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.559 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.559 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.560 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.560 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.560 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.560 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.560 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.560 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:52.560 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:52.560 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.560 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.560 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.560 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.560 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.561 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.561 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.561 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.561 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.561 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.561 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.561 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:52.561 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:52.561 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:52.561 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:52.562 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.563 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:52.565 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:52.565 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.565 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.565 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.565 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:52.565 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:52.565 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.566 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:52.566 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.566 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.566 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:52.566 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:52.566 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.566 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.566 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.567 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:52.567 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:52.567 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:52.568 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.569 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.570 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:52.572 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.573 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.575 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.577 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.578 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:52.578 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:52.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce9e838640 10 librados: watch_flush enter 2024-08-19T12:42:52.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce9e838640 10 librados: watch_flush exit 2024-08-19T12:42:52.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce9e838640 20 client.26212.objecter shutdown clearing up homeless session... 2024-08-19T12:42:52.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.540+0000 7fce9e838640 10 client.26212.objecter successfully canceled tick 2024-08-19T12:42:52.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.541+0000 7fce9e838640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.104:6800/899887506 conn(0x7fce780411b0 legacy=0x7fce78043670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.541+0000 7fce9e838640 10 monclient: shutdown 2024-08-19T12:42:52.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.541+0000 7fce9e838640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:52.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.541+0000 7fce9e838640 1 -- 172.21.15.26:0/2853863964 >> v1:172.21.15.26:6791/0 conn(0x7fce980a57a0 legacy=0x7fce980a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.541+0000 7fce9cdd5640 1 -- 172.21.15.26:0/2853863964 reap_dead start 2024-08-19T12:42:52.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.541+0000 7fce9cdd5640 1 -- 172.21.15.26:0/2853863964 reap_dead start 2024-08-19T12:42:52.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.541+0000 7fce9e838640 1 -- 172.21.15.26:0/2853863964 shutdown_connections 2024-08-19T12:42:52.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.541+0000 7fce9e838640 1 -- 172.21.15.26:0/2853863964 >> 172.21.15.26:0/2853863964 conn(0x7fce980abff0 msgr2=0x7fce98133960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:52.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.541+0000 7fce9e838640 1 -- 172.21.15.26:0/2853863964 shutdown_connections 2024-08-19T12:42:52.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.541+0000 7fce9e838640 1 -- 172.21.15.26:0/2853863964 wait complete. 2024-08-19T12:42:52.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.541+0000 7fce9e838640 1 librados: shutdown 2024-08-19T12:42:52.581 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:52.581 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:42:52.581 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:42:52.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.679+0000 7fdfcf68b640 10 monclient: get_monmap_and_config 2024-08-19T12:42:52.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.679+0000 7fdfcf68b640 10 monclient: build_initial_monmap 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.679+0000 7fdfcf68b640 10 monclient: monmap: 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:52.681471+0000 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:52.681471+0000 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:52.682 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:52.683 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:52.683 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:52.683 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:52.683 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 1 Processor -- start 2024-08-19T12:42:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 1 -- start start 2024-08-19T12:42:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 10 monclient: init 2024-08-19T12:42:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 10 monclient: _add_conns ranks=[7,4,3,2,8,5,6,1,0] 2024-08-19T12:42:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 10 monclient(hunting): picked mon.noname-i con 0x7fdfc806e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 10 monclient(hunting): picked mon.noname-c con 0x7fdfc80a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 10 monclient(hunting): picked mon.noname-g con 0x7fdfc80a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:52.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfc8136790 con 0x7fdfc80a5d50 2024-08-19T12:42:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfc809ddf0 con 0x7fdfc80a5920 2024-08-19T12:42:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfc809e0d0 con 0x7fdfc806e7b0 2024-08-19T12:42:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.680+0000 7fdfcf68b640 10 monclient(hunting): authenticate will time out at 966.768904s 2024-08-19T12:42:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.681+0000 7fdfccc26640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fdfc80a5920 0x7fdfc813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:54398/0 (socket says 172.21.15.26:54398) 2024-08-19T12:42:52.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.681+0000 7fdfccc26640 1 -- 172.21.15.26:0/1740948980 learned_addr learned my addr 172.21.15.26:0/1740948980 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfccc26640 1 -- 172.21.15.26:0/1740948980 >> v1:172.21.15.26:6791/0 conn(0x7fdfc80a5920 legacy=0x7fdfc813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 1 -- 172.21.15.26:0/1740948980 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 414235018 0 0) 0x7fdfb8001450 con 0x7fdfc80a5920 2024-08-19T12:42:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient(hunting): my global_id is 26221 2024-08-19T12:42:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 1 -- 172.21.15.26:0/1740948980 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfb4003520 con 0x7fdfc80a5920 2024-08-19T12:42:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfcd427640 1 -- 172.21.15.26:0/1740948980 >> v1:172.21.15.104:6792/0 conn(0x7fdfc806e7b0 legacy=0x7fdfc8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfcdc28640 1 -- 172.21.15.26:0/1740948980 >> v1:172.21.15.104:6790/0 conn(0x7fdfc80a5d50 legacy=0x7fdfc8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 1 -- 172.21.15.26:0/1740948980 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3128781201 0 0) 0x7fdfb4003520 con 0x7fdfc80a5920 2024-08-19T12:42:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 1 -- 172.21.15.26:0/1740948980 >> v1:172.21.15.104:6792/0 conn(0x7fdfc806e7b0 legacy=0x7fdfc8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 1 -- 172.21.15.26:0/1740948980 >> v1:172.21.15.104:6790/0 conn(0x7fdfc80a5d50 legacy=0x7fdfc8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient: _finish_hunting 0 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 1 monclient: found mon.noname-c 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 1 -- 172.21.15.26:0/1740948980 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc809e3b0 con 0x7fdfc80a5920 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient: _finish_auth 0 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient: _check_auth_tickets 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfcf68b640 5 monclient: authenticate success, global_id 26221 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 1 -- 172.21.15.26:0/1740948980 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdfb8001cb0 con 0x7fdfc80a5920 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfcf68b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:42:52.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient: dump: 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.691 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.692 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfcf68b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 1 -- 172.21.15.26:0/1740948980 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdfb8004990 con 0x7fdfc80a5920 2024-08-19T12:42:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 1 -- 172.21.15.26:0/1740948980 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdfb8004e80 con 0x7fdfc80a5920 2024-08-19T12:42:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfcf68b640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfcf68b640 4 set_mon_vals no callback set 2024-08-19T12:42:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.683+0000 7fdfbffff640 10 monclient: dump: 2024-08-19T12:42:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.693 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.694 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.684+0000 7fdfcf68b640 10 monclient: shutdown 2024-08-19T12:42:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.684+0000 7fdfcf68b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.684+0000 7fdfcf68b640 1 -- 172.21.15.26:0/1740948980 >> v1:172.21.15.26:6791/0 conn(0x7fdfc80a5920 legacy=0x7fdfc813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.684+0000 7fdfcf68b640 1 -- 172.21.15.26:0/1740948980 shutdown_connections 2024-08-19T12:42:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.684+0000 7fdfcf68b640 1 -- 172.21.15.26:0/1740948980 >> 172.21.15.26:0/1740948980 conn(0x7fdfc80a8740 msgr2=0x7fdfc80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.684+0000 7fdfcf68b640 1 -- 172.21.15.26:0/1740948980 shutdown_connections 2024-08-19T12:42:52.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.684+0000 7fdfcf68b640 1 -- 172.21.15.26:0/1740948980 wait complete. 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.684+0000 7fdfcf68b640 10 monclient: build_initial_monmap 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.684+0000 7fdfcf68b640 10 monclient: monmap: 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:52.695 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:52.696 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 1 librados: starting msgr at 2024-08-19T12:42:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 1 librados: starting objecter 2024-08-19T12:42:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 1 Processor -- start 2024-08-19T12:42:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 1 -- start start 2024-08-19T12:42:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 1 librados: setting wanted keys 2024-08-19T12:42:52.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 1 librados: calling monclient init 2024-08-19T12:42:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 10 monclient: init 2024-08-19T12:42:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 10 monclient: _add_conns ranks=[2,7,6,8,4,3,1,5,0] 2024-08-19T12:42:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 10 monclient(hunting): picked mon.noname-b con 0x7fdfc806e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:42:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 10 monclient(hunting): picked mon.noname-i con 0x7fdfc80a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 10 monclient(hunting): picked mon.noname-d con 0x7fdfc80a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfc80a1bb0 con 0x7fdfc806e7b0 2024-08-19T12:42:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfc8134990 con 0x7fdfc80a5d50 2024-08-19T12:42:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfc80a1050 con 0x7fdfc80a5920 2024-08-19T12:42:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcf68b640 10 monclient(hunting): authenticate will time out at 966.773707s 2024-08-19T12:42:52.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcd427640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fdfc806e7b0 0x7fdfc81ce550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51700/0 (socket says 172.21.15.26:51700) 2024-08-19T12:42:52.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.685+0000 7fdfcd427640 1 -- 172.21.15.26:0/959503896 learned_addr learned my addr 172.21.15.26:0/959503896 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:52.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.687+0000 7fdfcd427640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.26:6790/0 conn(0x7fdfc806e7b0 legacy=0x7fdfc81ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.687+0000 7fdfbdffb640 10 client.?.objecter ms_handle_connect 0x7fdfc806e7b0 2024-08-19T12:42:52.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.687+0000 7fdfbdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.687+0000 7fdfbdffb640 10 client.?.objecter ms_handle_connect 0x7fdfc80a5d50 2024-08-19T12:42:52.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.687+0000 7fdfbdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.687+0000 7fdfcdc28640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.26:6792/0 conn(0x7fdfc80a5d50 legacy=0x7fdfc81d6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763626678 0 0) 0x7fdfc8134990 con 0x7fdfc80a5d50 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): my global_id is 26250 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfac004eb0 con 0x7fdfc80a5d50 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 209286696 0 0) 0x7fdfc80a1bb0 con 0x7fdfc806e7b0 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): my global_id is 26003 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfc8134990 con 0x7fdfc806e7b0 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfccc26640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.104:6792/0 conn(0x7fdfc80a5920 legacy=0x7fdfc81d3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 client.?.objecter ms_handle_connect 0x7fdfc80a5920 2024-08-19T12:42:52.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 100330777 0 0) 0x7fdfac004eb0 con 0x7fdfc80a5d50 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdfc80a1bb0 con 0x7fdfc80a5d50 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdfc0002030 con 0x7fdfc80a5d50 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): dump: 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.706 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2276758755 0 0) 0x7fdfc8134990 con 0x7fdfc806e7b0 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdfac004eb0 con 0x7fdfc806e7b0 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdfc4003610 con 0x7fdfc806e7b0 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:42:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): dump: 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.708 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 95917454 0 0) 0x7fdfc80a1050 con 0x7fdfc80a5920 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient(hunting): my global_id is 134773 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfc8134990 con 0x7fdfc80a5920 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 974760181 0 0) 0x7fdfc80a1bb0 con 0x7fdfc80a5d50 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.104:6792/0 conn(0x7fdfc80a5920 legacy=0x7fdfc81d3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.26:6790/0 conn(0x7fdfc806e7b0 legacy=0x7fdfc81ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient: _finish_hunting 0 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 monclient: found mon.d 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:52.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc80a2df0 con 0x7fdfc80a5d50 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient: _finish_auth 0 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient: _check_auth_tickets 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfcf68b640 5 monclient: authenticate success, global_id 26250 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfcf68b640 10 monclient: _renew_subs 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfcf68b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfcf68b640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc80a3de0 con 0x7fdfc80a5d50 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdfc0003d50 con 0x7fdfc80a5d50 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdfc0004280 con 0x7fdfc80a5d50 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbdffb640 10 monclient: dump: 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.711 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfcf68b640 10 client.26250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfcf68b640 10 monclient: _renew_subs 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfbeffd640 4 set_mon_vals no callback set 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfcf68b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfcf68b640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fdfc80a0910 con 0x7fdfc80a5d50 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.688+0000 7fdfcf68b640 1 librados: init done 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.689+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fdfc0004420 con 0x7fdfc80a5d50 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.689+0000 7fdfcf68b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.689+0000 7fdfcf68b640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdfc80a2bd0 con 0x7fdfc81cec60 2024-08-19T12:42:52.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.689+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(182..182 src has 1..182) ==== 6196+0+0 (unknown 3389665571 0 0) 0x7fdfc004e620 con 0x7fdfc80a5d50 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.689+0000 7fdfbdffb640 10 client.26250.objecter ms_dispatch 0x7fdfc8000b90 osd_map(182..182 src has 1..182) 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.689+0000 7fdfbdffb640 3 client.26250.objecter handle_osd_map got epochs [182,182] > 0 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.689+0000 7fdfbdffb640 3 client.26250.objecter handle_osd_map decoding full epoch 182 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.689+0000 7fdfbdffb640 20 client.26250.objecter dump_active .. 0 homeless 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.689+0000 7fdfccc26640 10 monclient: get_auth_request con 0x7fdfac0416a0 auth_method 0 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfcd427640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.104:6789/0 conn(0x7fdfc81cec60 legacy=0x7fdfc81252b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 10 client.26250.objecter ms_handle_connect 0x7fdfc81cec60 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 10 client.26250.objecter resend_mon_ops 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2294874109 0 0) 0x7fdfc80a2bd0 con 0x7fdfc81cec60 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 10 monclient(hunting): my global_id is 26614 2024-08-19T12:42:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfc80a1bb0 con 0x7fdfc81cec60 2024-08-19T12:42:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 10 client.26250.objecter ms_handle_connect 0x7fdfac0416a0 2024-08-19T12:42:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 785831949 0 0) 0x7fdfc80a1bb0 con 0x7fdfc81cec60 2024-08-19T12:42:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdfc80a2bd0 con 0x7fdfc81cec60 2024-08-19T12:42:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdfc4004180 con 0x7fdfc81cec60 2024-08-19T12:42:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:42:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 10 monclient: dump: 2024-08-19T12:42:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.714 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2889073804 0 0) 0x7fdfc80a2bd0 con 0x7fdfc81cec60 2024-08-19T12:42:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.692+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdfc80a0c50 con 0x7fdfc81cec60 2024-08-19T12:42:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.693+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdfc4004440 con 0x7fdfc81cec60 2024-08-19T12:42:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.693+0000 7fdfbdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.693+0000 7fdfbdffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.693+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.104:6789/0 conn(0x7fdfc81cec60 legacy=0x7fdfc81252b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.693+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.104:6789/0 conn(0x7fdfc81cec60 legacy=0x7fdfc81252b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.699+0000 7fdfcf68b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.699+0000 7fdfcf68b640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdfc81cf040 con 0x7fdfc802e1d0 2024-08-19T12:42:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.701+0000 7fdfcd427640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.104:6789/0 conn(0x7fdfc802e1d0 legacy=0x7fdfc81259c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.701+0000 7fdfbdffb640 10 client.26250.objecter ms_handle_connect 0x7fdfc802e1d0 2024-08-19T12:42:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.701+0000 7fdfbdffb640 10 client.26250.objecter resend_mon_ops 2024-08-19T12:42:52.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.701+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3931066542 0 0) 0x7fdfc81cf040 con 0x7fdfc802e1d0 2024-08-19T12:42:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.701+0000 7fdfbdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.701+0000 7fdfbdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.701+0000 7fdfbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.701+0000 7fdfbdffb640 10 monclient(hunting): my global_id is 26623 2024-08-19T12:42:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfc80a2bd0 con 0x7fdfc802e1d0 2024-08-19T12:42:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 163711935 0 0) 0x7fdfc80a2bd0 con 0x7fdfc802e1d0 2024-08-19T12:42:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdfc81cf040 con 0x7fdfc802e1d0 2024-08-19T12:42:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdfc4005e40 con 0x7fdfc802e1d0 2024-08-19T12:42:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:42:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 10 monclient: dump: 2024-08-19T12:42:52.717 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.718 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.719 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.719 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2682504546 0 0) 0x7fdfc81cf040 con 0x7fdfc802e1d0 2024-08-19T12:42:52.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdfc8068df0 con 0x7fdfc802e1d0 2024-08-19T12:42:52.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1484048777) 0x7fdfc4008160 con 0x7fdfc802e1d0 2024-08-19T12:42:52.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:52.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:52.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.104:6789/0 conn(0x7fdfc802e1d0 legacy=0x7fdfc81259c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.702+0000 7fdfbdffb640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.104:6789/0 conn(0x7fdfc802e1d0 legacy=0x7fdfc81259c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:52.719 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 285707, 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:52.720 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.721 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.722 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:52.723 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.724 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:52.725 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.726 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:52.726 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.726 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:52.726 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:52.726 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.727 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.727 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.727 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:52.727 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:52.727 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.727 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:52.728 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.728 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.728 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:52.728 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:52.728 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.728 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.728 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.728 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:52.729 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:52.733 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.734 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.734 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.734 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:52.734 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:52.734 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.734 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.734 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.734 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:52.734 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:52.737 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.740 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:52.740 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:52.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 10 librados: watch_flush enter 2024-08-19T12:42:52.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 10 librados: watch_flush exit 2024-08-19T12:42:52.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 20 client.26250.objecter shutdown clearing up homeless session... 2024-08-19T12:42:52.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 10 client.26250.objecter successfully canceled tick 2024-08-19T12:42:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.104:6800/899887506 conn(0x7fdfac0416a0 legacy=0x7fdfac043b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 10 monclient: shutdown 2024-08-19T12:42:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 1 -- 172.21.15.26:0/959503896 >> v1:172.21.15.26:6792/0 conn(0x7fdfc80a5d50 legacy=0x7fdfc81d6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcdc28640 1 -- 172.21.15.26:0/959503896 reap_dead start 2024-08-19T12:42:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcdc28640 1 -- 172.21.15.26:0/959503896 reap_dead start 2024-08-19T12:42:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 1 -- 172.21.15.26:0/959503896 shutdown_connections 2024-08-19T12:42:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 1 -- 172.21.15.26:0/959503896 >> 172.21.15.26:0/959503896 conn(0x7fdfc80a8740 msgr2=0x7fdfc80aa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 1 -- 172.21.15.26:0/959503896 shutdown_connections 2024-08-19T12:42:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 1 -- 172.21.15.26:0/959503896 wait complete. 2024-08-19T12:42:52.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.703+0000 7fdfcf68b640 1 librados: shutdown 2024-08-19T12:42:52.743 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:52.743 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:42:52.743 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:42:52.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.841+0000 7f700529f640 10 monclient: get_monmap_and_config 2024-08-19T12:42:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.841+0000 7f700529f640 10 monclient: build_initial_monmap 2024-08-19T12:42:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.841+0000 7f700529f640 10 monclient: monmap: 2024-08-19T12:42:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:52.842943+0000 2024-08-19T12:42:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:52.842943+0000 2024-08-19T12:42:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:52.843 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:52.844 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:52.844 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.841+0000 7f700529f640 1 Processor -- start 2024-08-19T12:42:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.841+0000 7f700529f640 1 -- start start 2024-08-19T12:42:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.841+0000 7f700529f640 10 monclient: init 2024-08-19T12:42:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.842+0000 7f700529f640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.842+0000 7f700529f640 10 monclient: _add_conns ranks=[1,4,7,3,5,0,6,8,2] 2024-08-19T12:42:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.842+0000 7f700529f640 10 monclient(hunting): picked mon.noname-f con 0x7f7000073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.842+0000 7f700529f640 10 monclient(hunting): picked mon.noname-c con 0x7f70000a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.842+0000 7f700529f640 10 monclient(hunting): picked mon.noname-i con 0x7f700013f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.842+0000 7f700529f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f700009dae0 con 0x7f7000073470 2024-08-19T12:42:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.842+0000 7f700529f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7000136910 con 0x7f70000a57a0 2024-08-19T12:42:52.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.842+0000 7f700529f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f700009dd50 con 0x7f700013f2e0 2024-08-19T12:42:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.842+0000 7f700529f640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.842+0000 7f700529f640 10 monclient(hunting): authenticate will time out at 966.930373s 2024-08-19T12:42:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.842+0000 7f6ffe7fc640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f70000a57a0 0x7f70000a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:54410/0 (socket says 172.21.15.26:54410) 2024-08-19T12:42:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.842+0000 7f6ffe7fc640 1 -- 172.21.15.26:0/2977708672 learned_addr learned my addr 172.21.15.26:0/2977708672 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:52.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.844+0000 7f6ffe7fc640 1 -- 172.21.15.26:0/2977708672 >> v1:172.21.15.26:6791/0 conn(0x7f70000a57a0 legacy=0x7f70000a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.844+0000 7f6ffdffb640 1 -- 172.21.15.26:0/2977708672 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3275811978 0 0) 0x7f7000136910 con 0x7f70000a57a0 2024-08-19T12:42:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.844+0000 7f6ffdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.844+0000 7f6ffdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.844+0000 7f6ffdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.844+0000 7f6ffdffb640 10 monclient(hunting): my global_id is 26230 2024-08-19T12:42:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.844+0000 7f6ffeffd640 1 -- 172.21.15.26:0/2977708672 >> v1:172.21.15.104:6789/0 conn(0x7f7000073470 legacy=0x7f7000136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.844+0000 7f6ffdffb640 1 -- 172.21.15.26:0/2977708672 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fec003520 con 0x7f70000a57a0 2024-08-19T12:42:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.844+0000 7f6fff7fe640 1 -- 172.21.15.26:0/2977708672 >> v1:172.21.15.104:6792/0 conn(0x7f700013f2e0 legacy=0x7f70001416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.844+0000 7f6ffdffb640 1 -- 172.21.15.26:0/2977708672 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 680125949 0 0) 0x7f6fec003520 con 0x7f70000a57a0 2024-08-19T12:42:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 1 -- 172.21.15.26:0/2977708672 >> v1:172.21.15.104:6792/0 conn(0x7f700013f2e0 legacy=0x7f70001416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 1 -- 172.21.15.26:0/2977708672 >> v1:172.21.15.104:6789/0 conn(0x7f7000073470 legacy=0x7f7000136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 10 monclient: _finish_hunting 0 2024-08-19T12:42:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 1 monclient: found mon.noname-c 2024-08-19T12:42:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:52.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 1 -- 172.21.15.26:0/2977708672 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f700009e030 con 0x7f70000a57a0 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 10 monclient: _finish_auth 0 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 10 monclient: _check_auth_tickets 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 1 -- 172.21.15.26:0/2977708672 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6fe8001ae0 con 0x7f70000a57a0 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 10 monclient: dump: 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.852 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f700529f640 5 monclient: authenticate success, global_id 26230 2024-08-19T12:42:52.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f700529f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 1 -- 172.21.15.26:0/2977708672 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fe80048a0 con 0x7f70000a57a0 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 1 -- 172.21.15.26:0/2977708672 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6fe8004dd0 con 0x7f70000a57a0 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f700529f640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f700529f640 4 set_mon_vals no callback set 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f6ffdffb640 10 monclient: dump: 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.854 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f700529f640 10 monclient: shutdown 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f700529f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:52.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f700529f640 1 -- 172.21.15.26:0/2977708672 >> v1:172.21.15.26:6791/0 conn(0x7f70000a57a0 legacy=0x7f70000a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f700529f640 1 -- 172.21.15.26:0/2977708672 shutdown_connections 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f700529f640 1 -- 172.21.15.26:0/2977708672 >> 172.21.15.26:0/2977708672 conn(0x7f70000abff0 msgr2=0x7f70000ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f700529f640 1 -- 172.21.15.26:0/2977708672 shutdown_connections 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.845+0000 7f700529f640 1 -- 172.21.15.26:0/2977708672 wait complete. 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.846+0000 7f700529f640 10 monclient: build_initial_monmap 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.846+0000 7f700529f640 10 monclient: monmap: 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:52.856 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.846+0000 7f700529f640 1 librados: starting msgr at 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.846+0000 7f700529f640 1 librados: starting objecter 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.846+0000 7f700529f640 1 Processor -- start 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.846+0000 7f700529f640 1 -- start start 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.846+0000 7f700529f640 1 librados: setting wanted keys 2024-08-19T12:42:52.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.846+0000 7f700529f640 1 librados: calling monclient init 2024-08-19T12:42:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.846+0000 7f700529f640 10 monclient: init 2024-08-19T12:42:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.847+0000 7f700529f640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.847+0000 7f700529f640 10 monclient: _add_conns ranks=[3,5,4,6,2,8,7,0,1] 2024-08-19T12:42:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.847+0000 7f700529f640 10 monclient(hunting): picked mon.noname-g con 0x7f7000073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.847+0000 7f700529f640 10 monclient(hunting): picked mon.noname-h con 0x7f70000a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:42:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.847+0000 7f700529f640 10 monclient(hunting): picked mon.noname-c con 0x7f700013f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.847+0000 7f700529f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7000137280 con 0x7f7000073470 2024-08-19T12:42:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.847+0000 7f700529f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7000135420 con 0x7f700013f2e0 2024-08-19T12:42:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.847+0000 7f700529f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f700013f0b0 con 0x7f70000a57a0 2024-08-19T12:42:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.847+0000 7f700529f640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.847+0000 7f700529f640 10 monclient(hunting): authenticate will time out at 966.935327s 2024-08-19T12:42:52.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.847+0000 7f6fff7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f700013f2e0 0x7f70001d6a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:54422/0 (socket says 172.21.15.26:54422) 2024-08-19T12:42:52.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.847+0000 7f6fff7fe640 1 -- 172.21.15.26:0/2572354648 learned_addr learned my addr 172.21.15.26:0/2572354648 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:52.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6fff7fe640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.26:6791/0 conn(0x7f700013f2e0 legacy=0x7f70001d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 client.?.objecter ms_handle_connect 0x7f700013f2e0 2024-08-19T12:42:52.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 client.?.objecter ms_handle_connect 0x7f7000073470 2024-08-19T12:42:52.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ffeffd640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.104:6790/0 conn(0x7f7000073470 legacy=0x7f70001ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 client.?.objecter ms_handle_connect 0x7f70000a57a0 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ffe7fc640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.104:6791/0 conn(0x7f70000a57a0 legacy=0x7f70001d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440175139 0 0) 0x7f7000135420 con 0x7f700013f2e0 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): my global_id is 26239 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe0004eb0 con 0x7f700013f2e0 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1597500930 0 0) 0x7f7000137280 con 0x7f7000073470 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): my global_id is 26148 2024-08-19T12:42:52.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7000135420 con 0x7f7000073470 2024-08-19T12:42:52.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2054665534 0 0) 0x7f700013f0b0 con 0x7f70000a57a0 2024-08-19T12:42:52.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): my global_id is 25934 2024-08-19T12:42:52.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7000137280 con 0x7f70000a57a0 2024-08-19T12:42:52.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3733697984 0 0) 0x7f6fe0004eb0 con 0x7f700013f2e0 2024-08-19T12:42:52.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f700013f0b0 con 0x7f700013f2e0 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6ff0003610 con 0x7f700013f2e0 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): dump: 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.867 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.849+0000 7f6ff77fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 465135478 0 0) 0x7f7000135420 con 0x7f7000073470 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fe0004eb0 con 0x7f7000073470 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6ff8003990 con 0x7f7000073470 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient(hunting): dump: 2024-08-19T12:42:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.870 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 667810107 0 0) 0x7f7000137280 con 0x7f70000a57a0 2024-08-19T12:42:52.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7000135420 con 0x7f70000a57a0 2024-08-19T12:42:52.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6fe8004d90 con 0x7f70000a57a0 2024-08-19T12:42:52.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:52.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:42:52.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient(hunting): dump: 2024-08-19T12:42:52.870 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.870 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.870 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.870 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.870 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 817520669 0 0) 0x7f700013f0b0 con 0x7f700013f2e0 2024-08-19T12:42:52.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.104:6791/0 conn(0x7f70000a57a0 legacy=0x7f70001d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.104:6790/0 conn(0x7f7000073470 legacy=0x7f70001ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient: _finish_hunting 0 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 monclient: found mon.c 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7000142b60 con 0x7f700013f2e0 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient: _finish_auth 0 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient: _check_auth_tickets 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f700529f640 5 monclient: authenticate success, global_id 26239 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f700529f640 10 monclient: _renew_subs 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f700529f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f700529f640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7000136fa0 con 0x7f700013f2e0 2024-08-19T12:42:52.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f700529f640 10 client.26239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f700529f640 10 monclient: _renew_subs 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f700529f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f700529f640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f700013e9f0 con 0x7f700013f2e0 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f700529f640 1 librados: init done 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ff0003b10 con 0x7f700013f2e0 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6ff0004040 con 0x7f700013f2e0 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ffd7fa640 4 set_mon_vals no callback set 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.850+0000 7f6ff77fe640 10 monclient: dump: 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.873 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.874 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.851+0000 7f700529f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:52.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.851+0000 7f700529f640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fe40065c0 con 0x7f6fe4002860 2024-08-19T12:42:52.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.851+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6ff00041e0 con 0x7f700013f2e0 2024-08-19T12:42:52.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.851+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(182..182 src has 1..182) ==== 6196+0+0 (unknown 3389665571 0 0) 0x7f6ff004e430 con 0x7f700013f2e0 2024-08-19T12:42:52.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.851+0000 7f6ff77fe640 10 client.26239.objecter ms_dispatch 0x7f7000000b90 osd_map(182..182 src has 1..182) 2024-08-19T12:42:52.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.851+0000 7f6ff77fe640 3 client.26239.objecter handle_osd_map got epochs [182,182] > 0 2024-08-19T12:42:52.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.851+0000 7f6ff77fe640 3 client.26239.objecter handle_osd_map decoding full epoch 182 2024-08-19T12:42:52.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.851+0000 7f6ff77fe640 20 client.26239.objecter dump_active .. 0 homeless 2024-08-19T12:42:52.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.851+0000 7f6ffe7fc640 10 monclient: get_auth_request con 0x7f6fe00419a0 auth_method 0 2024-08-19T12:42:52.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ffeffd640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.104:6790/0 conn(0x7f6fe4002860 legacy=0x7f6fe4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 10 client.26239.objecter ms_handle_connect 0x7f6fe4002860 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 10 client.26239.objecter resend_mon_ops 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1662883567 0 0) 0x7f6fe40065c0 con 0x7f6fe4002860 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 10 monclient(hunting): my global_id is 26157 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f700013f0b0 con 0x7f6fe4002860 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3762508687 0 0) 0x7f700013f0b0 con 0x7f6fe4002860 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fe40065c0 con 0x7f6fe4002860 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6ff80041d0 con 0x7f6fe4002860 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:42:52.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.853+0000 7f6ff77fe640 10 monclient: dump: 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.877 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.878 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.878 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.854+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3013693702 0 0) 0x7f6fe40065c0 con 0x7f6fe4002860 2024-08-19T12:42:52.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.854+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fe400a160 con 0x7f6fe4002860 2024-08-19T12:42:52.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.854+0000 7f6ff77fe640 10 client.26239.objecter ms_handle_connect 0x7f6fe00419a0 2024-08-19T12:42:52.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.854+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6ff8005b40 con 0x7f6fe4002860 2024-08-19T12:42:52.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.854+0000 7f6ff77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:52.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.854+0000 7f6ff77fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:52.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.854+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.104:6790/0 conn(0x7f6fe4002860 legacy=0x7f6fe4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.854+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.104:6790/0 conn(0x7f6fe4002860 legacy=0x7f6fe4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:52.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.860+0000 7f700529f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:52.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.860+0000 7f700529f640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fe4009e70 con 0x7f6fe4009130 2024-08-19T12:42:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ffeffd640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.104:6790/0 conn(0x7f6fe4009130 legacy=0x7f6fe40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 10 client.26239.objecter ms_handle_connect 0x7f6fe4009130 2024-08-19T12:42:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 10 client.26239.objecter resend_mon_ops 2024-08-19T12:42:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2232654236 0 0) 0x7f6fe4009e70 con 0x7f6fe4009130 2024-08-19T12:42:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 10 monclient(hunting): my global_id is 26166 2024-08-19T12:42:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe40065c0 con 0x7f6fe4009130 2024-08-19T12:42:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2556680165 0 0) 0x7f6fe40065c0 con 0x7f6fe4009130 2024-08-19T12:42:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fe4009e70 con 0x7f6fe4009130 2024-08-19T12:42:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6ff8004770 con 0x7f6fe4009130 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 10 monclient: dump: 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:52.881 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2616055995 0 0) 0x7f6fe4009e70 con 0x7f6fe4009130 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.863+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6fe400c410 con 0x7f6fe4009130 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 951887776) 0x7f6ff8008150 con 0x7f6fe4009130 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f6ff77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f6ff77fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.104:6790/0 conn(0x7f6fe4009130 legacy=0x7f6fe40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f6ff77fe640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.104:6790/0 conn(0x7f6fe4009130 legacy=0x7f6fe40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:52.883 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:52.883 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:52.883 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:52.883 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:52.883 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:52.883 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 285877, 2024-08-19T12:42:52.883 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:52.883 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:52.883 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:52.883 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:52.884 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:52.884 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:52.884 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:52.884 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:52.884 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:52.884 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:52.884 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.884 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:42:52.884 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:52.884 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:52.884 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:52.884 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:52.885 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:52.887 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:52.888 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:52.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:52.890 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:52.892 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:52.893 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.894 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:52.894 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:52.894 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.894 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.894 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.894 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:52.894 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.895 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.896 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:52.896 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:52.896 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:52.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:52.898 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.898 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:52.898 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:52.898 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.898 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.899 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:52.899 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:52.899 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:52.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:52.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:52.900 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.900 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.900 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.900 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:52.900 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:52.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:52.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:52.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:42:52.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.903 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:52.903 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:52.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:52.903 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:52.903 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:52.903 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:42:52.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:52.903 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:52.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:52.903 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:52.903 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 10 librados: watch_flush enter 2024-08-19T12:42:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 10 librados: watch_flush exit 2024-08-19T12:42:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 20 client.26239.objecter shutdown clearing up homeless session... 2024-08-19T12:42:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 10 client.26239.objecter successfully canceled tick 2024-08-19T12:42:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.104:6800/899887506 conn(0x7f6fe00419a0 legacy=0x7f6fe0043e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 10 monclient: shutdown 2024-08-19T12:42:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 1 -- 172.21.15.26:0/2572354648 >> v1:172.21.15.26:6791/0 conn(0x7f700013f2e0 legacy=0x7f70001d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f6fff7fe640 1 -- 172.21.15.26:0/2572354648 reap_dead start 2024-08-19T12:42:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f6fff7fe640 1 -- 172.21.15.26:0/2572354648 reap_dead start 2024-08-19T12:42:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 1 -- 172.21.15.26:0/2572354648 shutdown_connections 2024-08-19T12:42:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 1 -- 172.21.15.26:0/2572354648 >> 172.21.15.26:0/2572354648 conn(0x7f70000abff0 msgr2=0x7f70001349a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 1 -- 172.21.15.26:0/2572354648 shutdown_connections 2024-08-19T12:42:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 1 -- 172.21.15.26:0/2572354648 wait complete. 2024-08-19T12:42:52.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:52.864+0000 7f700529f640 1 librados: shutdown 2024-08-19T12:42:52.906 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:52.907 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:42:52.907 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:42:53.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.006+0000 7fd3ed467640 10 monclient: get_monmap_and_config 2024-08-19T12:42:53.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.006+0000 7fd3ed467640 10 monclient: build_initial_monmap 2024-08-19T12:42:53.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.006+0000 7fd3ed467640 10 monclient: monmap: 2024-08-19T12:42:53.009 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:53.009 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.009 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:53.008551+0000 2024-08-19T12:42:53.009 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:53.008551+0000 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.007+0000 7fd3ed467640 1 Processor -- start 2024-08-19T12:42:53.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.007+0000 7fd3ed467640 1 -- start start 2024-08-19T12:42:53.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.007+0000 7fd3ed467640 10 monclient: init 2024-08-19T12:42:53.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3ed467640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:53.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3ed467640 10 monclient: _add_conns ranks=[5,8,0,4,1,2,3,7,6] 2024-08-19T12:42:53.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3ed467640 10 monclient(hunting): picked mon.noname-h con 0x7fd3e80a19f0 addr v1:172.21.15.104:6791/0 2024-08-19T12:42:53.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3ed467640 10 monclient(hunting): picked mon.noname-e con 0x7fd3e809e180 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:53.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3ed467640 10 monclient(hunting): picked mon.noname-a con 0x7fd3e809e5d0 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3ed467640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3e80a7300 con 0x7fd3e809e5d0 2024-08-19T12:42:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3ed467640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3e80a0970 con 0x7fd3e80a19f0 2024-08-19T12:42:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3ed467640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3e809fbd0 con 0x7fd3e809e180 2024-08-19T12:42:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3ed467640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3ed467640 10 monclient(hunting): authenticate will time out at 967.096331s 2024-08-19T12:42:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3dffff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fd3e809e180 0x7fd3e813b1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:53422/0 (socket says 172.21.15.26:53422) 2024-08-19T12:42:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3e77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd3e809e5d0 0x7fd3e813e900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57892/0 (socket says 172.21.15.26:57892) 2024-08-19T12:42:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.008+0000 7fd3dffff640 1 -- 172.21.15.26:0/1869508513 learned_addr learned my addr 172.21.15.26:0/1869508513 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.010+0000 7fd3e77fe640 1 -- 172.21.15.26:0/1869508513 >> v1:172.21.15.26:6789/0 conn(0x7fd3e809e5d0 legacy=0x7fd3e813e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.010+0000 7fd3dffff640 1 -- 172.21.15.26:0/1869508513 >> v1:172.21.15.26:6793/0 conn(0x7fd3e809e180 legacy=0x7fd3e813b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.010+0000 7fd3e6ffd640 1 -- 172.21.15.26:0/1869508513 >> v1:172.21.15.104:6791/0 conn(0x7fd3e80a19f0 legacy=0x7fd3e809da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.010+0000 7fd3e67fc640 1 -- 172.21.15.26:0/1869508513 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 463594133 0 0) 0x7fd3e80a7300 con 0x7fd3e809e5d0 2024-08-19T12:42:53.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.010+0000 7fd3e67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.010+0000 7fd3e67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.010+0000 7fd3e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.010+0000 7fd3e67fc640 10 monclient(hunting): my global_id is 144171 2024-08-19T12:42:53.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.010+0000 7fd3e67fc640 1 -- 172.21.15.26:0/1869508513 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3d4004630 con 0x7fd3e809e5d0 2024-08-19T12:42:53.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 1 -- 172.21.15.26:0/1869508513 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 430255046 0 0) 0x7fd3d4004630 con 0x7fd3e809e5d0 2024-08-19T12:42:53.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 1 -- 172.21.15.26:0/1869508513 >> v1:172.21.15.26:6793/0 conn(0x7fd3e809e180 legacy=0x7fd3e813b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 1 -- 172.21.15.26:0/1869508513 >> v1:172.21.15.104:6791/0 conn(0x7fd3e80a19f0 legacy=0x7fd3e809da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 10 monclient: _finish_hunting 0 2024-08-19T12:42:53.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 1 monclient: found mon.noname-a 2024-08-19T12:42:53.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:53.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:42:53.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 1 -- 172.21.15.26:0/1869508513 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e809feb0 con 0x7fd3e809e5d0 2024-08-19T12:42:53.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 10 monclient: _finish_auth 0 2024-08-19T12:42:53.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 10 monclient: _check_auth_tickets 2024-08-19T12:42:53.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:53.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 1 -- 172.21.15.26:0/1869508513 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3e0001e60 con 0x7fd3e809e5d0 2024-08-19T12:42:53.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:42:53.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 10 monclient: dump: 2024-08-19T12:42:53.022 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.022 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.022 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.022 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.022 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.022 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.023 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.023 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.023 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.023 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.023 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.023 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.023 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.023 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3ed467640 5 monclient: authenticate success, global_id 144171 2024-08-19T12:42:53.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3ed467640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:53.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 1 -- 172.21.15.26:0/1869508513 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd3e0004b60 con 0x7fd3e809e5d0 2024-08-19T12:42:53.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 1 -- 172.21.15.26:0/1869508513 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3e0005090 con 0x7fd3e809e5d0 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3e67fc640 10 monclient: dump: 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3ed467640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.014+0000 7fd3ed467640 4 set_mon_vals no callback set 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.015+0000 7fd3ed467640 10 monclient: shutdown 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.015+0000 7fd3ed467640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.015+0000 7fd3ed467640 1 -- 172.21.15.26:0/1869508513 >> v1:172.21.15.26:6789/0 conn(0x7fd3e809e5d0 legacy=0x7fd3e813e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.015+0000 7fd3ed467640 1 -- 172.21.15.26:0/1869508513 shutdown_connections 2024-08-19T12:42:53.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.015+0000 7fd3ed467640 1 -- 172.21.15.26:0/1869508513 >> 172.21.15.26:0/1869508513 conn(0x7fd3e80a44d0 msgr2=0x7fd3e80a48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.015+0000 7fd3ed467640 1 -- 172.21.15.26:0/1869508513 shutdown_connections 2024-08-19T12:42:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.015+0000 7fd3ed467640 1 -- 172.21.15.26:0/1869508513 wait complete. 2024-08-19T12:42:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.016+0000 7fd3ed467640 10 monclient: build_initial_monmap 2024-08-19T12:42:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.016+0000 7fd3ed467640 10 monclient: monmap: 2024-08-19T12:42:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:53.026 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.016+0000 7fd3ed467640 1 librados: starting msgr at 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.016+0000 7fd3ed467640 1 librados: starting objecter 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 1 Processor -- start 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 1 -- start start 2024-08-19T12:42:53.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 1 librados: setting wanted keys 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 1 librados: calling monclient init 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 10 monclient: init 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 10 monclient: _add_conns ranks=[1,4,8,5,7,6,3,2,0] 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 10 monclient(hunting): picked mon.noname-f con 0x7fd3e809e180 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 10 monclient(hunting): picked mon.noname-c con 0x7fd3e809e5d0 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 10 monclient(hunting): picked mon.noname-e con 0x7fd3e80a19f0 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3e81b2a50 con 0x7fd3e809e180 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3e81b1560 con 0x7fd3e809e5d0 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3e81b38c0 con 0x7fd3e80a19f0 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3ed467640 10 monclient(hunting): authenticate will time out at 967.105841s 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3e77fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fd3e80a19f0 0x7fd3e81daa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:53444/0 (socket says 172.21.15.26:53444) 2024-08-19T12:42:53.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3e77fe640 1 -- 172.21.15.26:0/96435979 learned_addr learned my addr 172.21.15.26:0/96435979 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:53.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.017+0000 7fd3dffff640 1 --1- 172.21.15.26:0/96435979 >> v1:172.21.15.26:6791/0 conn(0x7fd3e809e5d0 0x7fd3e81d7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:54434/0 (socket says 172.21.15.26:54434) 2024-08-19T12:42:53.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.020+0000 7fd3e6ffd640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.104:6789/0 conn(0x7fd3e809e180 legacy=0x7fd3e81cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.020+0000 7fd3e77fe640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.26:6793/0 conn(0x7fd3e80a19f0 legacy=0x7fd3e81daa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.020+0000 7fd3df7fe640 10 client.?.objecter ms_handle_connect 0x7fd3e809e180 2024-08-19T12:42:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.020+0000 7fd3df7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.020+0000 7fd3df7fe640 10 client.?.objecter ms_handle_connect 0x7fd3e80a19f0 2024-08-19T12:42:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.020+0000 7fd3df7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.020+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1329879492 0 0) 0x7fd3e81b2a50 con 0x7fd3e809e180 2024-08-19T12:42:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.020+0000 7fd3df7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.020+0000 7fd3df7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.021+0000 7fd3df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.021+0000 7fd3df7fe640 10 monclient(hunting): my global_id is 26641 2024-08-19T12:42:53.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.021+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3c8004eb0 con 0x7fd3e809e180 2024-08-19T12:42:53.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.021+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1676787966 0 0) 0x7fd3e81b38c0 con 0x7fd3e80a19f0 2024-08-19T12:42:53.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.021+0000 7fd3df7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.021+0000 7fd3df7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.021+0000 7fd3df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.021+0000 7fd3df7fe640 10 monclient(hunting): my global_id is 25757 2024-08-19T12:42:53.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.021+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3e81b2a50 con 0x7fd3e80a19f0 2024-08-19T12:42:53.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.021+0000 7fd3dffff640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.26:6791/0 conn(0x7fd3e809e5d0 legacy=0x7fd3e81d7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.021+0000 7fd3df7fe640 10 client.?.objecter ms_handle_connect 0x7fd3e809e5d0 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.022+0000 7fd3df7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.022+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559887410 0 0) 0x7fd3e81b1560 con 0x7fd3e809e5d0 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.022+0000 7fd3df7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.022+0000 7fd3df7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.022+0000 7fd3df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.022+0000 7fd3df7fe640 10 monclient(hunting): my global_id is 26248 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.022+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3e81b38c0 con 0x7fd3e809e5d0 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.022+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3658118852 0 0) 0x7fd3c8004eb0 con 0x7fd3e809e180 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.022+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3e81b1560 con 0x7fd3e809e180 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.022+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3353767659 0 0) 0x7fd3e81b2a50 con 0x7fd3e80a19f0 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.022+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3c8004eb0 con 0x7fd3e80a19f0 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.022+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3d8001cb0 con 0x7fd3e809e180 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.023+0000 7fd3df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:53.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.023+0000 7fd3df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:42:53.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.023+0000 7fd3df7fe640 10 monclient(hunting): dump: 2024-08-19T12:42:53.042 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.042 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.042 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.042 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.042 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.042 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.042 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.042 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.042 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.043 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.043 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.043 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.043 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.043 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.043 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.043 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.023+0000 7fd3df7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:53.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.023+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3e0005050 con 0x7fd3e80a19f0 2024-08-19T12:42:53.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.023+0000 7fd3df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:53.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.023+0000 7fd3df7fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:53.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.023+0000 7fd3df7fe640 10 monclient(hunting): dump: 2024-08-19T12:42:53.044 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.044 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.044 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.044 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.044 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.044 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.044 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.044 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.023+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 612260841 0 0) 0x7fd3e81b38c0 con 0x7fd3e809e5d0 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.023+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3e81b2a50 con 0x7fd3e809e5d0 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.023+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1913353320 0 0) 0x7fd3c8004eb0 con 0x7fd3e80a19f0 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.023+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.26:6791/0 conn(0x7fd3e809e5d0 legacy=0x7fd3e81d7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.104:6789/0 conn(0x7fd3e809e180 legacy=0x7fd3e81cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3df7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:42:53.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3df7fe640 1 monclient: found mon.e 2024-08-19T12:42:53.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:53.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3df7fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:53.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e81b3020 con 0x7fd3e80a19f0 2024-08-19T12:42:53.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3df7fe640 10 monclient: _finish_auth 0 2024-08-19T12:42:53.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3df7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:42:53.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:53.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3ed467640 5 monclient: authenticate success, global_id 25757 2024-08-19T12:42:53.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3ed467640 10 monclient: _renew_subs 2024-08-19T12:42:53.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3ed467640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:53.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3ed467640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e81cc570 con 0x7fd3e80a19f0 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3ed467640 10 client.25757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3ed467640 10 monclient: _renew_subs 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3ed467640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3ed467640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fd3e81d2690 con 0x7fd3e80a19f0 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.024+0000 7fd3ed467640 1 librados: init done 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.025+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd3e0004b60 con 0x7fd3e80a19f0 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.025+0000 7fd3df7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.025+0000 7fd3e5ffb640 4 set_mon_vals no callback set 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.025+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3e0001e80 con 0x7fd3e80a19f0 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.025+0000 7fd3df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.025+0000 7fd3df7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.025+0000 7fd3df7fe640 10 monclient: dump: 2024-08-19T12:42:53.047 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.048 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.049 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.026+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd3e0011c80 con 0x7fd3e80a19f0 2024-08-19T12:42:53.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.027+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(183..183 src has 1..183) ==== 6572+0+0 (unknown 3046874794 0 0) 0x7fd3e004e890 con 0x7fd3e80a19f0 2024-08-19T12:42:53.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.027+0000 7fd3df7fe640 10 client.25757.objecter ms_dispatch 0x7fd3e8000b90 osd_map(183..183 src has 1..183) 2024-08-19T12:42:53.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.027+0000 7fd3df7fe640 3 client.25757.objecter handle_osd_map got epochs [183,183] > 0 2024-08-19T12:42:53.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.027+0000 7fd3df7fe640 3 client.25757.objecter handle_osd_map decoding full epoch 183 2024-08-19T12:42:53.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.027+0000 7fd3df7fe640 20 client.25757.objecter dump_active .. 0 homeless 2024-08-19T12:42:53.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.028+0000 7fd3e6ffd640 10 monclient: get_auth_request con 0x7fd3c8041910 auth_method 0 2024-08-19T12:42:53.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.029+0000 7fd3ed467640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:53.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.029+0000 7fd3ed467640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd3cc0065c0 con 0x7fd3cc002860 2024-08-19T12:42:53.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.031+0000 7fd3df7fe640 10 client.25757.objecter ms_handle_connect 0x7fd3c8041910 2024-08-19T12:42:53.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.031+0000 7fd3df7fe640 10 client.25757.objecter ms_handle_connect 0x7fd3cc002860 2024-08-19T12:42:53.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.031+0000 7fd3df7fe640 10 client.25757.objecter resend_mon_ops 2024-08-19T12:42:53.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.031+0000 7fd3dffff640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.104:6791/0 conn(0x7fd3cc002860 legacy=0x7fd3cc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.031+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2689947703 0 0) 0x7fd3cc0065c0 con 0x7fd3cc002860 2024-08-19T12:42:53.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.031+0000 7fd3df7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.031+0000 7fd3df7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.031+0000 7fd3df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.031+0000 7fd3df7fe640 10 monclient(hunting): my global_id is 25943 2024-08-19T12:42:53.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3c8004eb0 con 0x7fd3cc002860 2024-08-19T12:42:53.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1197993730 0 0) 0x7fd3c8004eb0 con 0x7fd3cc002860 2024-08-19T12:42:53.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3cc0065c0 con 0x7fd3cc002860 2024-08-19T12:42:53.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3d0004fc0 con 0x7fd3cc002860 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 10 monclient: dump: 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.051 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.052 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.052 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.052 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.052 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.052 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.052 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3186189904 0 0) 0x7fd3cc0065c0 con 0x7fd3cc002860 2024-08-19T12:42:53.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3cc00a160 con 0x7fd3cc002860 2024-08-19T12:42:53.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd3d0007320 con 0x7fd3cc002860 2024-08-19T12:42:53.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:53.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:53.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.104:6791/0 conn(0x7fd3cc002860 legacy=0x7fd3cc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.032+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.104:6791/0 conn(0x7fd3cc002860 legacy=0x7fd3cc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:53.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.039+0000 7fd3ed467640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:53.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.039+0000 7fd3ed467640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd3cc009e70 con 0x7fd3cc009130 2024-08-19T12:42:53.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3dffff640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.104:6791/0 conn(0x7fd3cc009130 legacy=0x7fd3cc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 10 client.25757.objecter ms_handle_connect 0x7fd3cc009130 2024-08-19T12:42:53.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 10 client.25757.objecter resend_mon_ops 2024-08-19T12:42:53.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1905483346 0 0) 0x7fd3cc009e70 con 0x7fd3cc009130 2024-08-19T12:42:53.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 10 monclient(hunting): my global_id is 25952 2024-08-19T12:42:53.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3cc0065c0 con 0x7fd3cc009130 2024-08-19T12:42:53.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4178550460 0 0) 0x7fd3cc0065c0 con 0x7fd3cc009130 2024-08-19T12:42:53.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3cc009e70 con 0x7fd3cc009130 2024-08-19T12:42:53.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3d0006a10 con 0x7fd3cc009130 2024-08-19T12:42:53.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:42:53.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.041+0000 7fd3df7fe640 10 monclient: dump: 2024-08-19T12:42:53.056 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.057 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.057 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.057 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.057 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.057 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.057 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.057 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.057 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.057 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.057 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.057 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.058 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.058 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.058 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.058 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4055002068 0 0) 0x7fd3cc009e70 con 0x7fd3cc009130 2024-08-19T12:42:53.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd3cc00c410 con 0x7fd3cc009130 2024-08-19T12:42:53.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1183380363) 0x7fd3d0006e20 con 0x7fd3cc009130 2024-08-19T12:42:53.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3df7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:53.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3df7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:53.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.104:6791/0 conn(0x7fd3cc009130 legacy=0x7fd3cc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3df7fe640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.104:6791/0 conn(0x7fd3cc009130 legacy=0x7fd3cc0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:53.059 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:53.059 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 286066, 2024-08-19T12:42:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:53.060 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:53.061 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:42:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:53.062 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:53.063 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:53.064 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.065 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:53.066 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:53.067 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:53.068 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:53.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:53.070 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.072 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.073 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:53.074 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.078 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:53.078 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:53.078 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.078 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.078 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.078 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:53.079 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.080 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:53.083 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.084 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.085 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.086 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:53.086 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 10 librados: watch_flush enter 2024-08-19T12:42:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 10 librados: watch_flush exit 2024-08-19T12:42:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 20 client.25757.objecter shutdown clearing up homeless session... 2024-08-19T12:42:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 10 client.25757.objecter successfully canceled tick 2024-08-19T12:42:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.104:6800/899887506 conn(0x7fd3c8041910 legacy=0x7fd3c8043dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 10 monclient: shutdown 2024-08-19T12:42:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 1 -- 172.21.15.26:0/96435979 >> v1:172.21.15.26:6793/0 conn(0x7fd3e80a19f0 legacy=0x7fd3e81daa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3e77fe640 1 -- 172.21.15.26:0/96435979 reap_dead start 2024-08-19T12:42:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3e77fe640 1 -- 172.21.15.26:0/96435979 reap_dead start 2024-08-19T12:42:53.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 1 -- 172.21.15.26:0/96435979 shutdown_connections 2024-08-19T12:42:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 1 -- 172.21.15.26:0/96435979 >> 172.21.15.26:0/96435979 conn(0x7fd3e80a44d0 msgr2=0x7fd3e81a6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 1 -- 172.21.15.26:0/96435979 shutdown_connections 2024-08-19T12:42:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 1 -- 172.21.15.26:0/96435979 wait complete. 2024-08-19T12:42:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.042+0000 7fd3ed467640 1 librados: shutdown 2024-08-19T12:42:53.089 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:53.089 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:42:53.089 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:42:53.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.188+0000 7f3b94a8b640 10 monclient: get_monmap_and_config 2024-08-19T12:42:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.188+0000 7f3b94a8b640 10 monclient: build_initial_monmap 2024-08-19T12:42:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.188+0000 7f3b94a8b640 10 monclient: monmap: 2024-08-19T12:42:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:53.189940+0000 2024-08-19T12:42:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:53.189940+0000 2024-08-19T12:42:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:53.191 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:53.191 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:53.191 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:53.191 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:53.191 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:53.191 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:53.191 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.188+0000 7f3b94a8b640 1 Processor -- start 2024-08-19T12:42:53.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.188+0000 7f3b94a8b640 1 -- start start 2024-08-19T12:42:53.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.188+0000 7f3b94a8b640 10 monclient: init 2024-08-19T12:42:53.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b94a8b640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:53.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b94a8b640 10 monclient: _add_conns ranks=[8,1,0,7,4,2,5,6,3] 2024-08-19T12:42:53.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b94a8b640 10 monclient(hunting): picked mon.noname-e con 0x7f3b90073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:53.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b94a8b640 10 monclient(hunting): picked mon.noname-f con 0x7f3b900a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:53.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b94a8b640 10 monclient(hunting): picked mon.noname-a con 0x7f3b9013f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:53.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b94a8b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b9009dae0 con 0x7f3b9013f2e0 2024-08-19T12:42:53.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b94a8b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b90136910 con 0x7f3b900a57a0 2024-08-19T12:42:53.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b94a8b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b9009dd50 con 0x7f3b90073470 2024-08-19T12:42:53.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b94a8b640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:53.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b94a8b640 10 monclient(hunting): authenticate will time out at 967.277434s 2024-08-19T12:42:53.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b8ed9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3b9013f2e0 0x7f3b901416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57906/0 (socket says 172.21.15.26:57906) 2024-08-19T12:42:53.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b8ed9d640 1 -- 172.21.15.26:0/1925558371 learned_addr learned my addr 172.21.15.26:0/1925558371 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:53.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.189+0000 7f3b8e59c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f3b90073470 0x7f3b90136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:53452/0 (socket says 172.21.15.26:53452) 2024-08-19T12:42:53.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8ed9d640 1 -- 172.21.15.26:0/1925558371 >> v1:172.21.15.26:6789/0 conn(0x7f3b9013f2e0 legacy=0x7f3b901416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8e59c640 1 -- 172.21.15.26:0/1925558371 >> v1:172.21.15.26:6793/0 conn(0x7f3b90073470 legacy=0x7f3b90136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8d59a640 1 -- 172.21.15.26:0/1925558371 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1120620420 0 0) 0x7f3b9009dae0 con 0x7f3b9013f2e0 2024-08-19T12:42:53.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8d59a640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8d59a640 10 monclient(hunting): my global_id is 144180 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8d59a640 1 -- 172.21.15.26:0/1925558371 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b7c003520 con 0x7f3b9013f2e0 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8d59a640 1 -- 172.21.15.26:0/1925558371 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 343084539 0 0) 0x7f3b9009dd50 con 0x7f3b90073470 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8d59a640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8d59a640 10 monclient(hunting): my global_id is 25766 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8d59a640 1 -- 172.21.15.26:0/1925558371 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b9009dae0 con 0x7f3b90073470 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.191+0000 7f3b8dd9b640 1 -- 172.21.15.26:0/1925558371 >> v1:172.21.15.104:6789/0 conn(0x7f3b900a57a0 legacy=0x7f3b900a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 1 -- 172.21.15.26:0/1925558371 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2728845639 0 0) 0x7f3b7c003520 con 0x7f3b9013f2e0 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 1 -- 172.21.15.26:0/1925558371 >> v1:172.21.15.26:6793/0 conn(0x7f3b90073470 legacy=0x7f3b90136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 1 -- 172.21.15.26:0/1925558371 >> v1:172.21.15.104:6789/0 conn(0x7f3b900a57a0 legacy=0x7f3b900a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 10 monclient: _finish_hunting 0 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 1 monclient: found mon.noname-a 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 1 -- 172.21.15.26:0/1925558371 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b9009e030 con 0x7f3b9013f2e0 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 10 monclient: _finish_auth 0 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 10 monclient: _check_auth_tickets 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 1 -- 172.21.15.26:0/1925558371 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3b88001ae0 con 0x7f3b9013f2e0 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b94a8b640 5 monclient: authenticate success, global_id 144180 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b94a8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:42:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 10 monclient: dump: 2024-08-19T12:42:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.200 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b94a8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 1 -- 172.21.15.26:0/1925558371 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3b880048a0 con 0x7f3b9013f2e0 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 1 -- 172.21.15.26:0/1925558371 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3b88004e10 con 0x7f3b9013f2e0 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b94a8b640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b94a8b640 4 set_mon_vals no callback set 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:53.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b8d59a640 10 monclient: dump: 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.202 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.203 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.203 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.203 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b94a8b640 10 monclient: shutdown 2024-08-19T12:42:53.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b94a8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:53.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b94a8b640 1 -- 172.21.15.26:0/1925558371 >> v1:172.21.15.26:6789/0 conn(0x7f3b9013f2e0 legacy=0x7f3b901416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b94a8b640 1 -- 172.21.15.26:0/1925558371 shutdown_connections 2024-08-19T12:42:53.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b94a8b640 1 -- 172.21.15.26:0/1925558371 >> 172.21.15.26:0/1925558371 conn(0x7f3b900abff0 msgr2=0x7f3b900ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:53.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b94a8b640 1 -- 172.21.15.26:0/1925558371 shutdown_connections 2024-08-19T12:42:53.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.192+0000 7f3b94a8b640 1 -- 172.21.15.26:0/1925558371 wait complete. 2024-08-19T12:42:53.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.193+0000 7f3b94a8b640 10 monclient: build_initial_monmap 2024-08-19T12:42:53.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.193+0000 7f3b94a8b640 10 monclient: monmap: 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:53.204 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.193+0000 7f3b94a8b640 1 librados: starting msgr at 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.193+0000 7f3b94a8b640 1 librados: starting objecter 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.193+0000 7f3b94a8b640 1 Processor -- start 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 1 -- start start 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 1 librados: setting wanted keys 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 1 librados: calling monclient init 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 10 monclient: init 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 10 monclient: _add_conns ranks=[2,8,6,1,0,3,5,7,4] 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 10 monclient(hunting): picked mon.noname-b con 0x7f3b90073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:42:53.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 10 monclient(hunting): picked mon.noname-e con 0x7f3b900a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:53.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 10 monclient(hunting): picked mon.noname-d con 0x7f3b9013f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:53.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b9012fdd0 con 0x7f3b90073470 2024-08-19T12:42:53.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b901352b0 con 0x7f3b9013f2e0 2024-08-19T12:42:53.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b9013f0b0 con 0x7f3b900a57a0 2024-08-19T12:42:53.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:53.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b94a8b640 10 monclient(hunting): authenticate will time out at 967.282486s 2024-08-19T12:42:53.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b8e59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3b90073470 0x7f3b9012dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51716/0 (socket says 172.21.15.26:51716) 2024-08-19T12:42:53.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b8dd9b640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f3b900a57a0 0x7f3b9012e450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:53460/0 (socket says 172.21.15.26:53460) 2024-08-19T12:42:53.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.194+0000 7f3b8e59c640 1 -- 172.21.15.26:0/3846500546 learned_addr learned my addr 172.21.15.26:0/3846500546 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.196+0000 7f3b8dd9b640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.26:6793/0 conn(0x7f3b900a57a0 legacy=0x7f3b9012e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.196+0000 7f3b86ffd640 10 client.?.objecter ms_handle_connect 0x7f3b900a57a0 2024-08-19T12:42:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.196+0000 7f3b8e59c640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.26:6790/0 conn(0x7f3b90073470 legacy=0x7f3b9012dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.196+0000 7f3b86ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.196+0000 7f3b86ffd640 10 client.?.objecter ms_handle_connect 0x7f3b90073470 2024-08-19T12:42:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.196+0000 7f3b86ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.196+0000 7f3b8ed9d640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.26:6792/0 conn(0x7f3b9013f2e0 legacy=0x7f3b901d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.196+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:42:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 client.?.objecter ms_handle_connect 0x7f3b9013f2e0 2024-08-19T12:42:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:53.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 507005488 0 0) 0x7f3b90136910 con 0x7f3b900a57a0 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 monclient(hunting): my global_id is 25775 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b70004eb0 con 0x7f3b900a57a0 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1416331790 0 0) 0x7f3b9009e030 con 0x7f3b9013f2e0 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 monclient(hunting): my global_id is 26259 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b90136910 con 0x7f3b9013f2e0 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3041514823 0 0) 0x7f3b9012fdd0 con 0x7f3b90073470 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.199+0000 7f3b86ffd640 10 monclient(hunting): my global_id is 26012 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b9009e030 con 0x7f3b90073470 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2163857194 0 0) 0x7f3b90136910 con 0x7f3b9013f2e0 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b9012fdd0 con 0x7f3b9013f2e0 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3b88001d30 con 0x7f3b9013f2e0 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient(hunting): dump: 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.213 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2493453941 0 0) 0x7f3b70004eb0 con 0x7f3b900a57a0 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b90136910 con 0x7f3b900a57a0 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1904798867 0 0) 0x7f3b9009e030 con 0x7f3b90073470 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b70004eb0 con 0x7f3b90073470 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3b78003610 con 0x7f3b900a57a0 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient(hunting): dump: 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.215 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3b80004b60 con 0x7f3b90073470 2024-08-19T12:42:53.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient(hunting): dump: 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.217 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2132138948 0 0) 0x7f3b9012fdd0 con 0x7f3b9013f2e0 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.26:6793/0 conn(0x7f3b900a57a0 legacy=0x7f3b9012e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.26:6790/0 conn(0x7f3b90073470 legacy=0x7f3b9012dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 monclient: found mon.d 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:53.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b90131700 con 0x7f3b9013f2e0 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient: _finish_auth 0 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b94a8b640 5 monclient: authenticate success, global_id 26259 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b94a8b640 10 monclient: _renew_subs 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b94a8b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b94a8b640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3b901319e0 con 0x7f3b9013f2e0 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b94a8b640 10 client.26259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b94a8b640 10 monclient: _renew_subs 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b94a8b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b94a8b640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f3b9012eb60 con 0x7f3b9013f2e0 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b94a8b640 1 librados: init done 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3b880048a0 con 0x7f3b9013f2e0 2024-08-19T12:42:53.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3b88001b60 con 0x7f3b9013f2e0 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b8cd99640 4 set_mon_vals no callback set 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 monclient: dump: 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.220 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.221 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.221 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.221 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b94a8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b94a8b640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3b740065c0 con 0x7f3b74002860 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3b88011640 con 0x7f3b9013f2e0 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(183..183 src has 1..183) ==== 6572+0+0 (unknown 3046874794 0 0) 0x7f3b8804e340 con 0x7f3b9013f2e0 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 10 client.26259.objecter ms_dispatch 0x7f3b90000b90 osd_map(183..183 src has 1..183) 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 3 client.26259.objecter handle_osd_map got epochs [183,183] > 0 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 3 client.26259.objecter handle_osd_map decoding full epoch 183 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b86ffd640 20 client.26259.objecter dump_active .. 0 homeless 2024-08-19T12:42:53.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.200+0000 7f3b8dd9b640 10 monclient: get_auth_request con 0x7f3b70041ac0 auth_method 0 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.202+0000 7f3b86ffd640 10 client.26259.objecter ms_handle_connect 0x7f3b74002860 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.202+0000 7f3b8e59c640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.104:6792/0 conn(0x7f3b74002860 legacy=0x7f3b74004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.202+0000 7f3b86ffd640 10 client.26259.objecter resend_mon_ops 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2401837002 0 0) 0x7f3b740065c0 con 0x7f3b74002860 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 10 monclient(hunting): my global_id is 134791 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b9012fdd0 con 0x7f3b74002860 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 10 client.26259.objecter ms_handle_connect 0x7f3b70041ac0 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1565866316 0 0) 0x7f3b9012fdd0 con 0x7f3b74002860 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b740065c0 con 0x7f3b74002860 2024-08-19T12:42:53.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3b80005230 con 0x7f3b74002860 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 10 monclient: dump: 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.224 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.225 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1661025896 0 0) 0x7f3b740065c0 con 0x7f3b74002860 2024-08-19T12:42:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b7400a160 con 0x7f3b74002860 2024-08-19T12:42:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3b800074d0 con 0x7f3b74002860 2024-08-19T12:42:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.104:6792/0 conn(0x7f3b74002860 legacy=0x7f3b74004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.203+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.104:6792/0 conn(0x7f3b74002860 legacy=0x7f3b74004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:53.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.210+0000 7f3b94a8b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:53.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.210+0000 7f3b94a8b640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3b74009e70 con 0x7f3b74009130 2024-08-19T12:42:53.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.212+0000 7f3b8e59c640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.104:6792/0 conn(0x7f3b74009130 legacy=0x7f3b740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.212+0000 7f3b86ffd640 10 client.26259.objecter ms_handle_connect 0x7f3b74009130 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.212+0000 7f3b86ffd640 10 client.26259.objecter resend_mon_ops 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.212+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2480000991 0 0) 0x7f3b74009e70 con 0x7f3b74009130 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.212+0000 7f3b86ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.212+0000 7f3b86ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.212+0000 7f3b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.212+0000 7f3b86ffd640 10 monclient(hunting): my global_id is 134800 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.212+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b740065c0 con 0x7f3b74009130 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1818040565 0 0) 0x7f3b740065c0 con 0x7f3b74009130 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b74009e70 con 0x7f3b74009130 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3b80006d40 con 0x7f3b74009130 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:42:53.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 10 monclient: dump: 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.229 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.230 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2904225877 0 0) 0x7f3b74009e70 con 0x7f3b74009130 2024-08-19T12:42:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3b7400c3f0 con 0x7f3b74009130 2024-08-19T12:42:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1667335289) 0x7f3b80007020 con 0x7f3b74009130 2024-08-19T12:42:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.104:6792/0 conn(0x7f3b74009130 legacy=0x7f3b740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b86ffd640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.104:6792/0 conn(0x7f3b74009130 legacy=0x7f3b740095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:53.231 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:53.231 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:53.231 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:53.231 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:53.231 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:53.231 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 143572, 2024-08-19T12:42:53.231 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:53.231 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:53.231 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:53.231 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:53.231 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:53.231 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:53.232 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:53.233 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:53.233 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:53.233 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:53.233 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:53.233 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:53.233 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:53.233 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:53.233 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:53.233 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:53.233 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:53.233 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:53.233 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:53.234 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:53.234 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:53.234 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:53.234 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:53.234 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:53.234 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:53.234 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:53.234 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:53.234 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.234 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:53.234 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:53.235 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:53.236 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.238 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:53.238 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:53.238 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.238 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.238 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.238 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:53.238 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:53.238 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:53.242 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.243 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.244 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:53.247 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:53.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.249 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:53.250 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:53.251 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:53.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b94a8b640 10 librados: watch_flush enter 2024-08-19T12:42:53.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b94a8b640 10 librados: watch_flush exit 2024-08-19T12:42:53.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b94a8b640 20 client.26259.objecter shutdown clearing up homeless session... 2024-08-19T12:42:53.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b94a8b640 10 client.26259.objecter successfully canceled tick 2024-08-19T12:42:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b94a8b640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.104:6800/899887506 conn(0x7f3b70041ac0 legacy=0x7f3b70043f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b94a8b640 10 monclient: shutdown 2024-08-19T12:42:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b94a8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b94a8b640 1 -- 172.21.15.26:0/3846500546 >> v1:172.21.15.26:6792/0 conn(0x7f3b9013f2e0 legacy=0x7f3b901d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b8ed9d640 1 -- 172.21.15.26:0/3846500546 reap_dead start 2024-08-19T12:42:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b8ed9d640 1 -- 172.21.15.26:0/3846500546 reap_dead start 2024-08-19T12:42:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b94a8b640 1 -- 172.21.15.26:0/3846500546 shutdown_connections 2024-08-19T12:42:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.213+0000 7f3b94a8b640 1 -- 172.21.15.26:0/3846500546 >> 172.21.15.26:0/3846500546 conn(0x7f3b900abff0 msgr2=0x7f3b900a9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.214+0000 7f3b94a8b640 1 -- 172.21.15.26:0/3846500546 shutdown_connections 2024-08-19T12:42:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.214+0000 7f3b94a8b640 1 -- 172.21.15.26:0/3846500546 wait complete. 2024-08-19T12:42:53.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.214+0000 7f3b94a8b640 1 librados: shutdown 2024-08-19T12:42:53.254 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:53.254 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T12:42:53.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T12:42:53.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.350+0000 7f7a0e07e640 10 monclient: get_monmap_and_config 2024-08-19T12:42:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.350+0000 7f7a0e07e640 10 monclient: build_initial_monmap 2024-08-19T12:42:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.351+0000 7f7a0e07e640 10 monclient: monmap: 2024-08-19T12:42:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:53.352811+0000 2024-08-19T12:42:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:53.352811+0000 2024-08-19T12:42:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:53.353 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:53.354 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:53.354 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:53.354 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:53.354 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:53.354 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:53.354 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.351+0000 7f7a0e07e640 1 Processor -- start 2024-08-19T12:42:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.351+0000 7f7a0e07e640 1 -- start start 2024-08-19T12:42:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.351+0000 7f7a0e07e640 10 monclient: init 2024-08-19T12:42:53.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.352+0000 7f7a0e07e640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.352+0000 7f7a0e07e640 10 monclient: _add_conns ranks=[5,0,8,2,3,4,6,7,1] 2024-08-19T12:42:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.352+0000 7f7a0e07e640 10 monclient(hunting): picked mon.noname-h con 0x7f7a080a19f0 addr v1:172.21.15.104:6791/0 2024-08-19T12:42:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.352+0000 7f7a0e07e640 10 monclient(hunting): picked mon.noname-a con 0x7f7a0809e180 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.352+0000 7f7a0e07e640 10 monclient(hunting): picked mon.noname-e con 0x7f7a0809e5d0 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.352+0000 7f7a0e07e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a080a7300 con 0x7f7a0809e180 2024-08-19T12:42:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.352+0000 7f7a0e07e640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a080a0970 con 0x7f7a080a19f0 2024-08-19T12:42:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.352+0000 7f7a0e07e640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a0809fbd0 con 0x7f7a0809e5d0 2024-08-19T12:42:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.352+0000 7f7a0e07e640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.352+0000 7f7a0e07e640 10 monclient(hunting): authenticate will time out at 967.440394s 2024-08-19T12:42:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.352+0000 7f7a07fff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f7a0809e5d0 0x7f7a0813e900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:53470/0 (socket says 172.21.15.26:53470) 2024-08-19T12:42:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.352+0000 7f7a07fff640 1 -- 172.21.15.26:0/790300587 learned_addr learned my addr 172.21.15.26:0/790300587 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:53.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a07fff640 1 -- 172.21.15.26:0/790300587 >> v1:172.21.15.26:6793/0 conn(0x7f7a0809e5d0 legacy=0x7f7a0813e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a06ffd640 1 -- 172.21.15.26:0/790300587 >> v1:172.21.15.26:6789/0 conn(0x7f7a0809e180 legacy=0x7f7a0813b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a077fe640 1 -- 172.21.15.26:0/790300587 >> v1:172.21.15.104:6791/0 conn(0x7f7a080a19f0 legacy=0x7f7a0809da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2536369360 0 0) 0x7f7a0809fbd0 con 0x7f7a0809e5d0 2024-08-19T12:42:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 10 monclient(hunting): my global_id is 25784 2024-08-19T12:42:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79f0003520 con 0x7f7a0809e5d0 2024-08-19T12:42:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3223372942 0 0) 0x7f7a080a7300 con 0x7f7a0809e180 2024-08-19T12:42:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 10 monclient(hunting): my global_id is 144189 2024-08-19T12:42:53.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a0809fbd0 con 0x7f7a0809e180 2024-08-19T12:42:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.354+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2434482783 0 0) 0x7f7a080a0970 con 0x7f7a080a19f0 2024-08-19T12:42:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient(hunting): my global_id is 25961 2024-08-19T12:42:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a080a7300 con 0x7f7a080a19f0 2024-08-19T12:42:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3168308001 0 0) 0x7f79f0003520 con 0x7f7a0809e5d0 2024-08-19T12:42:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 >> v1:172.21.15.104:6791/0 conn(0x7f7a080a19f0 legacy=0x7f7a0809da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 >> v1:172.21.15.26:6789/0 conn(0x7f7a0809e180 legacy=0x7f7a0813b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient: _finish_hunting 0 2024-08-19T12:42:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 1 monclient: found mon.noname-e 2024-08-19T12:42:53.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a0809feb0 con 0x7f7a0809e5d0 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient: _finish_auth 0 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient: _check_auth_tickets 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7a00001e60 con 0x7f7a0809e5d0 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a0e07e640 5 monclient: authenticate success, global_id 25784 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient: dump: 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.363 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a0e07e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7a00004c20 con 0x7f7a0809e5d0 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 1 -- 172.21.15.26:0/790300587 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7a00005190 con 0x7f7a0809e5d0 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a0e07e640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a0e07e640 4 set_mon_vals no callback set 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a067fc640 10 monclient: dump: 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.365 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a0e07e640 10 monclient: shutdown 2024-08-19T12:42:53.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a0e07e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a0e07e640 1 -- 172.21.15.26:0/790300587 >> v1:172.21.15.26:6793/0 conn(0x7f7a0809e5d0 legacy=0x7f7a0813e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a0e07e640 1 -- 172.21.15.26:0/790300587 shutdown_connections 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a0e07e640 1 -- 172.21.15.26:0/790300587 >> 172.21.15.26:0/790300587 conn(0x7f7a080a44d0 msgr2=0x7f7a080a48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a0e07e640 1 -- 172.21.15.26:0/790300587 shutdown_connections 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.355+0000 7f7a0e07e640 1 -- 172.21.15.26:0/790300587 wait complete. 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.356+0000 7f7a0e07e640 10 monclient: build_initial_monmap 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.356+0000 7f7a0e07e640 10 monclient: monmap: 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.367 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.356+0000 7f7a0e07e640 1 librados: starting msgr at 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.356+0000 7f7a0e07e640 1 librados: starting objecter 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.356+0000 7f7a0e07e640 1 Processor -- start 2024-08-19T12:42:53.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.356+0000 7f7a0e07e640 1 -- start start 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.356+0000 7f7a0e07e640 1 librados: setting wanted keys 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.356+0000 7f7a0e07e640 1 librados: calling monclient init 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.356+0000 7f7a0e07e640 10 monclient: init 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.357+0000 7f7a0e07e640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.357+0000 7f7a0e07e640 10 monclient: _add_conns ranks=[6,4,7,8,5,3,1,2,0] 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.357+0000 7f7a0e07e640 10 monclient(hunting): picked mon.noname-d con 0x7f7a0809e180 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.357+0000 7f7a0e07e640 10 monclient(hunting): picked mon.noname-c con 0x7f7a0809e5d0 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.357+0000 7f7a0e07e640 10 monclient(hunting): picked mon.noname-i con 0x7f7a080a19f0 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.357+0000 7f7a0e07e640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a080a4b20 con 0x7f7a0809e5d0 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.357+0000 7f7a0e07e640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a081a9bd0 con 0x7f7a0809e180 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.357+0000 7f7a0e07e640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a080734e0 con 0x7f7a080a19f0 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.357+0000 7f7a0e07e640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.357+0000 7f7a0e07e640 10 monclient(hunting): authenticate will time out at 967.445361s 2024-08-19T12:42:53.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.357+0000 7f7a077fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f7a0809e180 0x7f7a081ba1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34546/0 (socket says 172.21.15.26:34546) 2024-08-19T12:42:53.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.357+0000 7f7a077fe640 1 -- 172.21.15.26:0/343271502 learned_addr learned my addr 172.21.15.26:0/343271502 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:53.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f7a077fe640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.26:6792/0 conn(0x7f7a0809e180 legacy=0x7f7a081ba1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 client.?.objecter ms_handle_connect 0x7f7a0809e180 2024-08-19T12:42:53.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:53.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 client.?.objecter ms_handle_connect 0x7f7a0809e5d0 2024-08-19T12:42:53.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:53.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f7a06ffd640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.26:6791/0 conn(0x7f7a0809e5d0 legacy=0x7f7a081d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2624582350 0 0) 0x7f7a081a9bd0 con 0x7f7a0809e180 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f7a07fff640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.104:6792/0 conn(0x7f7a080a19f0 legacy=0x7f7a081d4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 monclient(hunting): my global_id is 26268 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79f4004eb0 con 0x7f7a0809e180 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 client.?.objecter ms_handle_connect 0x7f7a080a19f0 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3556088910 0 0) 0x7f79fc001500 con 0x7f7a0809e5d0 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 monclient(hunting): my global_id is 26257 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a081a9bd0 con 0x7f7a0809e5d0 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 466183945 0 0) 0x7f7a080734e0 con 0x7f7a080a19f0 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 10 monclient(hunting): my global_id is 134809 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.359+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79fc001500 con 0x7f7a080a19f0 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3879712065 0 0) 0x7f7a081a9bd0 con 0x7f7a0809e5d0 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a080734e0 con 0x7f7a0809e5d0 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79fc003950 con 0x7f7a0809e5d0 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): dump: 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.378 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 539363577 0 0) 0x7f79f4004eb0 con 0x7f7a0809e180 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a081a9bd0 con 0x7f7a0809e180 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79f8003610 con 0x7f7a0809e180 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): dump: 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.379 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1342677411 0 0) 0x7f79fc001500 con 0x7f7a080a19f0 2024-08-19T12:42:53.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79f4004eb0 con 0x7f7a080a19f0 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7a00005110 con 0x7f7a080a19f0 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): dump: 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.381 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1937581113 0 0) 0x7f7a080734e0 con 0x7f7a0809e5d0 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.104:6792/0 conn(0x7f7a080a19f0 legacy=0x7f7a081d4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.26:6792/0 conn(0x7f7a0809e180 legacy=0x7f7a081ba1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient: _finish_hunting 0 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 monclient: found mon.c 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:53.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a081a7870 con 0x7f7a0809e5d0 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient: _finish_auth 0 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient: _check_auth_tickets 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a0e07e640 5 monclient: authenticate success, global_id 26257 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a0e07e640 10 monclient: _renew_subs 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a0e07e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a0e07e640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7a081a98f0 con 0x7f7a0809e5d0 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a0e07e640 10 client.26257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a0e07e640 10 monclient: _renew_subs 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a06ffd640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.26:6791/0 conn(0x7f7a0809e5d0 legacy=0x7f7a081d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a0e07e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:42:53.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a06ffd640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.26:6791/0 conn(0x7f7a0809e5d0 legacy=0x7f7a081d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a0e07e640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f7a081baae0 con 0x7f7a0809e5d0 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a06ffd640 1 --1- 172.21.15.26:0/343271502 >> v1:172.21.15.26:6791/0 conn(0x7f7a0809e5d0 0x7f7a081d0b40 :-1 s=OPENED pgs=3200 cs=1 l=1).handle_message read tag failed 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a0e07e640 1 librados: init done 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f7a06ffd640 1 --1- 172.21.15.26:0/343271502 >> v1:172.21.15.26:6791/0 conn(0x7f7a0809e5d0 0x7f7a081d0b40 :-1 s=OPENED pgs=3200 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient: ms_handle_reset current mon v1:172.21.15.26:6791/0 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.26:6791/0 conn(0x7f7a0809e5d0 legacy=0x7f7a081d0b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient: _add_conns ranks=[8,7,1,3,0,6,4,2,5] 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): picked mon.e con 0x7f79f4007b20 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): picked mon.i con 0x7f7a080a19f0 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): picked mon.f con 0x7f79f400f0d0 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7a080734e0 con 0x7f79f400f0d0 2024-08-19T12:42:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7a081a9bd0 con 0x7f7a080a19f0 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f79fc001500 con 0x7f79f4007b20 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.360+0000 7f79effff640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.361+0000 7f7a0e07e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.361+0000 7f7a0e07e640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f7a077fe640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.26:6793/0 conn(0x7f79f4007b20 legacy=0x7f79f4009f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f79effff640 10 client.26257.objecter ms_handle_connect 0x7f79f4007b20 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f79effff640 10 client.26257.objecter resend_mon_ops 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f79effff640 10 client.26257.objecter ms_handle_connect 0x7f79f400f0d0 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f7a07fff640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.104:6789/0 conn(0x7f79f400f0d0 legacy=0x7f79f4011520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f79effff640 10 client.26257.objecter resend_mon_ops 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2929449425 0 0) 0x7f79fc001500 con 0x7f79f4007b20 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f79effff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f79effff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f79effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f7a06ffd640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.104:6792/0 conn(0x7f7a080a19f0 legacy=0x7f79f400d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6793/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f79f40130d0 con 0x7f79f4007b20 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f79effff640 10 client.26257.objecter ms_handle_connect 0x7f7a080a19f0 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.362+0000 7f79effff640 10 client.26257.objecter resend_mon_ops 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1818450491 0 0) 0x7f7a080734e0 con 0x7f79f400f0d0 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.104:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f79fc001500 con 0x7f79f400f0d0 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4161781623 0 0) 0x7f7a081a9bd0 con 0x7f7a080a19f0 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.104:6792/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f7a080734e0 con 0x7f7a080a19f0 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 2687319084 0 0) 0x7f79f40130d0 con 0x7f79f4007b20 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.104:6792/0 conn(0x7f7a080a19f0 legacy=0x7f79f400d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.104:6789/0 conn(0x7f79f400f0d0 legacy=0x7f79f4011520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient: _finish_hunting 0 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 monclient: found mon.e 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f79f4016440 con 0x7f79f4007b20 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a080008d0 con 0x7f79f4007b20 2024-08-19T12:42:53.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a081a9bd0 con 0x7f79f4007b20 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient: _finish_auth 0 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient: _check_auth_tickets 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79f8003f90 con 0x7f79f4007b20 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient: dump: 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.388 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79f80042b0 con 0x7f79f4007b20 2024-08-19T12:42:53.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f7a05ffb640 4 set_mon_vals no callback set 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.363+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f79f8011a80 con 0x7f79f4007b20 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.364+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.8 v1:172.21.15.26:6793/0 6 ==== osd_map(183..183 src has 1..183) ==== 6572+0+0 (unknown 3046874794 0 0) 0x7f79f804e860 con 0x7f79f4007b20 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.364+0000 7f79effff640 10 client.26257.objecter ms_dispatch 0x7f7a08000b90 osd_map(183..183 src has 1..183) 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.364+0000 7f79effff640 3 client.26257.objecter handle_osd_map got epochs [183,183] > 0 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.364+0000 7f79effff640 3 client.26257.objecter handle_osd_map decoding full epoch 183 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.364+0000 7f79effff640 20 client.26257.objecter dump_active .. 0 homeless 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.364+0000 7f7a06ffd640 10 monclient: get_auth_request con 0x7f79f404d8d0 auth_method 0 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.366+0000 7f79effff640 10 client.26257.objecter ms_handle_connect 0x7f79f404d8d0 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.368+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f79f80177a0 con 0x7f79f4007b20 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.368+0000 7f79effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.368+0000 7f79effff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:53.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.372+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.8 v1:172.21.15.26:6793/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f79f80170a0 con 0x7f79f4007b20 2024-08-19T12:42:53.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.372+0000 7f79effff640 10 monclient: handle_mon_command_ack 1 not found 2024-08-19T12:42:53.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.628+0000 7f7a0e07e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T12:42:53.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.628+0000 7f7a0e07e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T12:42:53.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.628+0000 7f7a0e07e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:53.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.628+0000 7f7a0e07e640 1 -- 172.21.15.26:0/343271502 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f7a081cb800 con 0x7f79f4007b20 2024-08-19T12:42:53.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.630+0000 7f79effff640 1 -- 172.21.15.26:0/343271502 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f79f8020090 con 0x7f79f4007b20 2024-08-19T12:42:53.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.630+0000 7f79effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T12:42:53.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.630+0000 7f79effff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.632+0000 7f7a0e07e640 10 librados: watch_flush enter 2024-08-19T12:42:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.632+0000 7f7a0e07e640 10 librados: watch_flush exit 2024-08-19T12:42:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.632+0000 7f7a0e07e640 20 client.26257.objecter shutdown clearing up homeless session... 2024-08-19T12:42:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.632+0000 7f7a0e07e640 10 client.26257.objecter successfully canceled tick 2024-08-19T12:42:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.632+0000 7f7a0e07e640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.104:6800/899887506 conn(0x7f79f404d8d0 legacy=0x7f79f404fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.632+0000 7f7a0e07e640 10 monclient: shutdown 2024-08-19T12:42:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.632+0000 7f7a0e07e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.632+0000 7f7a0e07e640 1 -- 172.21.15.26:0/343271502 >> v1:172.21.15.26:6793/0 conn(0x7f79f4007b20 legacy=0x7f79f4009f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.632+0000 7f7a07fff640 1 -- 172.21.15.26:0/343271502 reap_dead start 2024-08-19T12:42:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.632+0000 7f7a07fff640 1 -- 172.21.15.26:0/343271502 reap_dead start 2024-08-19T12:42:53.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.632+0000 7f7a0e07e640 1 -- 172.21.15.26:0/343271502 shutdown_connections 2024-08-19T12:42:53.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.632+0000 7f7a0e07e640 1 -- 172.21.15.26:0/343271502 >> 172.21.15.26:0/343271502 conn(0x7f7a080a44d0 msgr2=0x7f7a081bd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:53.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.633+0000 7f7a0e07e640 1 -- 172.21.15.26:0/343271502 shutdown_connections 2024-08-19T12:42:53.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.633+0000 7f7a0e07e640 1 -- 172.21.15.26:0/343271502 wait complete. 2024-08-19T12:42:53.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:53.633+0000 7f7a0e07e640 1 librados: shutdown 2024-08-19T12:42:53.645 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T12:42:53.921 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.921 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=13,logm=52} crc {auth=4282241650,config=1097430493,health=335027296,kv=1344821656,logm=2799196406}) 2024-08-19T12:42:53.921 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=11,kv=13,logm=54} crc {auth=1826875612,config=1097430493,health=233713815,kv=1344821656,logm=3953336300}) 2024-08-19T12:42:53.921 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.921 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=13,logm=52} crc {auth=4282241650,config=1097430493,health=335027296,kv=1344821656,logm=2799196406}) 2024-08-19T12:42:53.921 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=11,kv=13,logm=53} crc {auth=4282241650,config=1097430493,health=2111563348,kv=1344821656,logm=104093694}) 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=13,logm=52} crc {auth=4282241650,config=1097430493,health=335027296,kv=1344821656,logm=2799196406}) 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=4282241650,config=1097430493,health=3468539608,kv=3639533066,logm=3018132416}) 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=13,logm=52} crc {auth=4282241650,config=1097430493,health=335027296,kv=1344821656,logm=2799196406}) 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=3744888429,config=1097430493,health=1237534684,kv=2462598728,logm=3682269141}) 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=13,logm=52} crc {auth=4282241650,config=1097430493,health=335027296,kv=1344821656,logm=2799196406}) 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=20,config=2,health=11,kv=13,logm=54} crc {auth=2352740828,config=1097430493,health=980778987,kv=1344821656,logm=658494544}) 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=13,logm=52} crc {auth=4282241650,config=1097430493,health=335027296,kv=1344821656,logm=2799196406}) 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=20,config=2,health=11,kv=13,logm=54} crc {auth=4213053847,config=1097430493,health=980778987,kv=2631711969,logm=3469234239}) 2024-08-19T12:42:53.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=13,logm=52} crc {auth=4282241650,config=1097430493,health=335027296,kv=1344821656,logm=2799196406}) 2024-08-19T12:42:53.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=4282241650,config=1097430493,health=33587189,kv=1344821656,logm=2199540895}) 2024-08-19T12:42:53.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=13,logm=52} crc {auth=4282241650,config=1097430493,health=335027296,kv=1344821656,logm=2799196406}) 2024-08-19T12:42:53.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=20,config=1,health=11,kv=13,logm=55} crc {auth=2712247375,config=1228700967,health=780005338,kv=1344821656,logm=58275020}) 2024-08-19T12:42:53.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848496951}) 2024-08-19T12:42:53.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2121369039}) 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848496951}) 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615355725}) 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848496951}) 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3494618732}) 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848496951}) 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2617704569}) 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848496951}) 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2128688222}) 2024-08-19T12:42:53.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848496951}) 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=809024914}) 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848496951}) 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1882451165}) 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848496951}) 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.920+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2171755019}) 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172689080}) 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1041128575}) 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172689080}) 2024-08-19T12:42:53.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45664207}) 2024-08-19T12:42:53.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172689080}) 2024-08-19T12:42:53.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3863321886}) 2024-08-19T12:42:53.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172689080}) 2024-08-19T12:42:53.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2455796843}) 2024-08-19T12:42:53.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172689080}) 2024-08-19T12:42:53.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3292360917}) 2024-08-19T12:42:53.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172689080}) 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3421755197}) 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172689080}) 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=131967839}) 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172689080}) 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.922+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=871200840}) 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178746507}) 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1665979397}) 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178746507}) 2024-08-19T12:42:53.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2205171038}) 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178746507}) 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1589058656}) 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178746507}) 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=231504896}) 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178746507}) 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1324567004}) 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178746507}) 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=523172584}) 2024-08-19T12:42:53.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178746507}) 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2429549348}) 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178746507}) 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.923+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4077370422}) 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.924+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410174025}) 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3000517597}) 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410174025}) 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1450415509}) 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410174025}) 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442401755}) 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410174025}) 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3206298451}) 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410174025}) 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2895843037}) 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410174025}) 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1277290499}) 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410174025}) 2024-08-19T12:42:53.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4089746776}) 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410174025}) 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.925+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=42839597}) 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476338501}) 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669863071}) 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476338501}) 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=197561311}) 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476338501}) 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3144520045}) 2024-08-19T12:42:53.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476338501}) 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4229920967}) 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476338501}) 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2235642300}) 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476338501}) 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1176864734}) 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476338501}) 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1809814129}) 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476338501}) 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.926+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2343626112}) 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330686812}) 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2932687274}) 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330686812}) 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1445254819}) 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330686812}) 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1499392560}) 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330686812}) 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3907092795}) 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330686812}) 2024-08-19T12:42:53.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2872119992}) 2024-08-19T12:42:53.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330686812}) 2024-08-19T12:42:53.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1049148927}) 2024-08-19T12:42:53.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330686812}) 2024-08-19T12:42:53.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2948043353}) 2024-08-19T12:42:53.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330686812}) 2024-08-19T12:42:53.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.927+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=983099288}) 2024-08-19T12:42:53.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743585656}) 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3952745140}) 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743585656}) 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=769012883}) 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743585656}) 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=803810574}) 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743585656}) 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3561146049}) 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743585656}) 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1280921923}) 2024-08-19T12:42:53.971 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743585656}) 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=222618348}) 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743585656}) 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3427418697}) 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743585656}) 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4004865195}) 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414809287}) 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=985067430}) 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.972 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414809287}) 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=219076775}) 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414809287}) 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1207830991}) 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414809287}) 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1353533024}) 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414809287}) 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=318373706}) 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414809287}) 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2768432048}) 2024-08-19T12:42:53.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414809287}) 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1952487109}) 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414809287}) 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2715059292}) 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.931+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.931+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802384261}) 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.931+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=7049993}) 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.931+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.931+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802384261}) 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.931+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3674382320}) 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.931+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.974 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.931+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802384261}) 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.931+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=719747439}) 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.931+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.931+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802384261}) 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.931+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3046800426}) 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802384261}) 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=146216455}) 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802384261}) 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1010539389}) 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802384261}) 2024-08-19T12:42:53.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3308842594}) 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802384261}) 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3608472127}) 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751884329}) 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3157871156}) 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751884329}) 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=14432926}) 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751884329}) 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2423620186}) 2024-08-19T12:42:53.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751884329}) 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4096483173}) 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751884329}) 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3268891081}) 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751884329}) 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3982765223}) 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751884329}) 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2650212446}) 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751884329}) 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2558559825}) 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251677298}) 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1610211567}) 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251677298}) 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=933743686}) 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251677298}) 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1191976198}) 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251677298}) 2024-08-19T12:42:53.978 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=528112496}) 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251677298}) 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2213752290}) 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251677298}) 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=259874229}) 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251677298}) 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2729001402}) 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251677298}) 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.934+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3475599159}) 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.935+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.979 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.935+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011778457}) 2024-08-19T12:42:53.980 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.935+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2160473076}) 2024-08-19T12:42:53.980 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.935+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.980 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.935+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011778457}) 2024-08-19T12:42:53.980 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.935+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4013105469}) 2024-08-19T12:42:53.980 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.935+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.980 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011778457}) 2024-08-19T12:42:53.980 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3793746016}) 2024-08-19T12:42:53.980 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.980 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011778457}) 2024-08-19T12:42:53.980 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2548627968}) 2024-08-19T12:42:53.980 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.980 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011778457}) 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2076535229}) 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011778457}) 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3503405569}) 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011778457}) 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=735777545}) 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011778457}) 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.936+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1453917805}) 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381420931}) 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2509700527}) 2024-08-19T12:42:53.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381420931}) 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1812916935}) 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381420931}) 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3722407420}) 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381420931}) 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1163367758}) 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381420931}) 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3515685387}) 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381420931}) 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4157691959}) 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381420931}) 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2246338931}) 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381420931}) 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.937+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3287395438}) 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.938+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.938+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627669002}) 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.938+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3049065789}) 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.938+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.938+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627669002}) 2024-08-19T12:42:53.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.938+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3305713625}) 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.938+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.938+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627669002}) 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.938+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2002229719}) 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.938+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.938+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627669002}) 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.938+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1203961010}) 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.939+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.939+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627669002}) 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.939+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2015656833}) 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.939+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.939+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627669002}) 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.939+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=21465201}) 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.939+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.939+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627669002}) 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.939+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2730988979}) 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.939+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.939+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3627669002}) 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.939+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=579543051}) 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286323101}) 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4136513384}) 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286323101}) 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2297895122}) 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286323101}) 2024-08-19T12:42:53.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1836180157}) 2024-08-19T12:42:53.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286323101}) 2024-08-19T12:42:53.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4142639928}) 2024-08-19T12:42:53.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286323101}) 2024-08-19T12:42:53.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1503931939}) 2024-08-19T12:42:53.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286323101}) 2024-08-19T12:42:53.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1257390426}) 2024-08-19T12:42:53.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286323101}) 2024-08-19T12:42:53.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=844470826}) 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286323101}) 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.940+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1350101875}) 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.941+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.941+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209671028}) 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.941+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2025146619}) 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.941+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.941+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209671028}) 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.941+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2746072611}) 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.941+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209671028}) 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3088362601}) 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209671028}) 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2572380593}) 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209671028}) 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3722334650}) 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209671028}) 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1546279781}) 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209671028}) 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1293643805}) 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209671028}) 2024-08-19T12:42:53.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.942+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1829304438}) 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971669679}) 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4058115205}) 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971669679}) 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3395245217}) 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971669679}) 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1850829630}) 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971669679}) 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3918716281}) 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971669679}) 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1609906069}) 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971669679}) 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1425113703}) 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971669679}) 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=836043356}) 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971669679}) 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.943+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1732105933}) 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544574021}) 2024-08-19T12:42:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=751935360}) 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544574021}) 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2876464942}) 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544574021}) 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=737673287}) 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544574021}) 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=976632634}) 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.944+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.945+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544574021}) 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.945+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1686843028}) 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.945+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.945+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544574021}) 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.945+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3262515825}) 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.945+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.945+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544574021}) 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.945+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3636780609}) 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.945+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.945+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544574021}) 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.945+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1689425674}) 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025304811}) 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4019711283}) 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025304811}) 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4114234533}) 2024-08-19T12:42:53.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025304811}) 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4208736880}) 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025304811}) 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1351503705}) 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025304811}) 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=731398479}) 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025304811}) 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1820532664}) 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025304811}) 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=138300673}) 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025304811}) 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4017212168}) 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3213640711}) 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2605975271}) 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3213640711}) 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2289667851}) 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3213640711}) 2024-08-19T12:42:53.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2880429611}) 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3213640711}) 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2933003522}) 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3213640711}) 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2040785596}) 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3213640711}) 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1414330791}) 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3213640711}) 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4163566548}) 2024-08-19T12:42:53.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:53.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3213640711}) 2024-08-19T12:42:53.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1633118073}) 2024-08-19T12:42:54.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548386486}) 2024-08-19T12:42:54.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3579672062}) 2024-08-19T12:42:54.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548386486}) 2024-08-19T12:42:54.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798367436}) 2024-08-19T12:42:54.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548386486}) 2024-08-19T12:42:54.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=514372906}) 2024-08-19T12:42:54.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548386486}) 2024-08-19T12:42:54.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1604205}) 2024-08-19T12:42:54.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548386486}) 2024-08-19T12:42:54.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2055751169}) 2024-08-19T12:42:54.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548386486}) 2024-08-19T12:42:54.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2133029064}) 2024-08-19T12:42:54.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548386486}) 2024-08-19T12:42:54.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3461819773}) 2024-08-19T12:42:54.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548386486}) 2024-08-19T12:42:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2629202700}) 2024-08-19T12:42:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470016214}) 2024-08-19T12:42:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1275990583}) 2024-08-19T12:42:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470016214}) 2024-08-19T12:42:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1295076821}) 2024-08-19T12:42:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470016214}) 2024-08-19T12:42:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3141911200}) 2024-08-19T12:42:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470016214}) 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1197876910}) 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470016214}) 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2638332829}) 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470016214}) 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3239719835}) 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470016214}) 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2091051846}) 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470016214}) 2024-08-19T12:42:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1506327907}) 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1980313855}) 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3030397887}) 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1980313855}) 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1598097433}) 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1980313855}) 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3981285249}) 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1980313855}) 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2184499324}) 2024-08-19T12:42:54.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1980313855}) 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4174886428}) 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1980313855}) 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1002861983}) 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1980313855}) 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=812649710}) 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1980313855}) 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=595851089}) 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=410123994}) 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=198669922}) 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=410123994}) 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2737774195}) 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=410123994}) 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=291674529}) 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=410123994}) 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=973438121}) 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=410123994}) 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1078362299}) 2024-08-19T12:42:54.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=410123994}) 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3591862778}) 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=410123994}) 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1180441604}) 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=410123994}) 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2830422149}) 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469127501}) 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3336783005}) 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469127501}) 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=632904285}) 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469127501}) 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1466652970}) 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469127501}) 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2927781934}) 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469127501}) 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2815992462}) 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469127501}) 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2296008468}) 2024-08-19T12:42:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469127501}) 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1526041478}) 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469127501}) 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=106757741}) 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909566743}) 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2768506191}) 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909566743}) 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219038498}) 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909566743}) 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1915024501}) 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909566743}) 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=739103225}) 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909566743}) 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3372696166}) 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909566743}) 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2024647047}) 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909566743}) 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3787840177}) 2024-08-19T12:42:54.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909566743}) 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.976+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1974434573}) 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788148660}) 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4043254341}) 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788148660}) 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1215451554}) 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788148660}) 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=549324508}) 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788148660}) 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=535437243}) 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788148660}) 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1904238058}) 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788148660}) 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3098651005}) 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788148660}) 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=810102125}) 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788148660}) 2024-08-19T12:42:54.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.977+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1526260506}) 2024-08-19T12:42:54.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=3337156212,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=2369108238,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=316129630}) 2024-08-19T12:42:54.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14} crc {logm=1754416574,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3394760992,mon_config_key=2171442413,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=4070223017}) 2024-08-19T12:42:54.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=3337156212,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=2369108238,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=316129630}) 2024-08-19T12:42:54.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=35,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13} crc {logm=1840252804,mdsmap=3707151070,mgr=3044422214,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=3689853235,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=2600734958}) 2024-08-19T12:42:54.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=3337156212,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=2369108238,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=316129630}) 2024-08-19T12:42:54.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=34,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14} crc {logm=1272832763,mdsmap=3707151070,mgr=3834367735,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=3222896933,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=3842065507}) 2024-08-19T12:42:54.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=3337156212,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=2369108238,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=316129630}) 2024-08-19T12:42:54.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=35,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=12} crc {logm=1840252804,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=906312777,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=2104687725}) 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=3337156212,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=2369108238,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=316129630}) 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=32,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=3714102909,mdsmap=182599002,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=2171442413,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=316129630}) 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=3337156212,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=2369108238,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=316129630}) 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=35,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=12} crc {logm=1840252804,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=1637405881,monmap=2050960908,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=3364198534}) 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=3337156212,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=2369108238,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=316129630}) 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=959749164,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2288842234,mon_config_key=2171442413,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=2767435882}) 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=3337156212,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=3383379413,mon_config_key=2369108238,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=316129630}) 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.979+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=2773577136,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=890086295,mon_config_key=1649291736,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=2901145124,osd_snap=1647661043}) 2024-08-19T12:42:54.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=1042027395,osdmap=4264272522}) 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=2228480912,osdmap=3476238676}) 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=1042027395,osdmap=4264272522}) 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1011625424,osdmap=2646303582}) 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=1042027395,osdmap=4264272522}) 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1011625424,osdmap=2527762273}) 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=1042027395,osdmap=4264272522}) 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1011625424,osdmap=3607698980}) 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=1042027395,osdmap=4264272522}) 2024-08-19T12:42:54.032 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1011625424,osdmap=2168410287}) 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=1042027395,osdmap=4264272522}) 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1011625424,osdmap=3433963124}) 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=1042027395,osdmap=4264272522}) 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1011625424,osdmap=861984666}) 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=1042027395,osdmap=4264272522}) 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.981+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=1011625424,osdmap=2501969632}) 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3183342722}) 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3998981705}) 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.033 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3183342722}) 2024-08-19T12:42:54.034 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3026036899}) 2024-08-19T12:42:54.034 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.034 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3183342722}) 2024-08-19T12:42:54.034 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2258348274}) 2024-08-19T12:42:54.034 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.034 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3183342722}) 2024-08-19T12:42:54.034 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3418108428}) 2024-08-19T12:42:54.034 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.034 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3183342722}) 2024-08-19T12:42:54.034 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3624803495}) 2024-08-19T12:42:54.034 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.034 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3183342722}) 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=181737024}) 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3183342722}) 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3537940450}) 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3183342722}) 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.983+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2395150782}) 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=50002233}) 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=82} crc {osdmap=1843994174}) 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=50002233}) 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=81} crc {osdmap=2852325336}) 2024-08-19T12:42:54.035 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.036 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=50002233}) 2024-08-19T12:42:54.036 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=82} crc {osdmap=2604139692}) 2024-08-19T12:42:54.036 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.036 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=50002233}) 2024-08-19T12:42:54.036 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=82} crc {osdmap=383454051}) 2024-08-19T12:42:54.036 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.036 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=50002233}) 2024-08-19T12:42:54.036 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=83} crc {osdmap=3257328970}) 2024-08-19T12:42:54.037 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=50002233}) 2024-08-19T12:42:54.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=78} crc {osdmap=2054583647}) 2024-08-19T12:42:54.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=50002233}) 2024-08-19T12:42:54.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=85} crc {osdmap=3185018481}) 2024-08-19T12:42:54.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:42:54.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=50002233}) 2024-08-19T12:42:54.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:42:53.986+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=84} crc {osdmap=3439513759}) 2024-08-19T12:42:54.645 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:42:54.646 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:42:54.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.754+0000 7ff40380e640 10 monclient: get_monmap_and_config 2024-08-19T12:42:54.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.754+0000 7ff40380e640 10 monclient: build_initial_monmap 2024-08-19T12:42:54.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.754+0000 7ff40380e640 10 monclient: monmap: 2024-08-19T12:42:54.756 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:54.756 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:54.756 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:54.755936+0000 2024-08-19T12:42:54.756 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:54.755936+0000 2024-08-19T12:42:54.756 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:54.756 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:54.756 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:54.757 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:54.757 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:54.757 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:54.757 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:54.757 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:54.757 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:54.757 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:54.757 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:54.757 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:54.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.754+0000 7ff40380e640 1 Processor -- start 2024-08-19T12:42:54.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.754+0000 7ff40380e640 1 -- start start 2024-08-19T12:42:54.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.754+0000 7ff40380e640 10 monclient: init 2024-08-19T12:42:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.755+0000 7ff40380e640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.755+0000 7ff40380e640 10 monclient: _add_conns ranks=[8,7,3,0,4,6,5,1,2] 2024-08-19T12:42:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.755+0000 7ff40380e640 10 monclient(hunting): picked mon.noname-e con 0x7ff3fc073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.755+0000 7ff40380e640 10 monclient(hunting): picked mon.noname-i con 0x7ff3fc0a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.755+0000 7ff40380e640 10 monclient(hunting): picked mon.noname-g con 0x7ff3fc13f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.755+0000 7ff40380e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3fc09dae0 con 0x7ff3fc13f2e0 2024-08-19T12:42:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.755+0000 7ff40380e640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3fc136910 con 0x7ff3fc0a57a0 2024-08-19T12:42:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.755+0000 7ff40380e640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3fc09dd50 con 0x7ff3fc073470 2024-08-19T12:42:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.755+0000 7ff40380e640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.755+0000 7ff40380e640 10 monclient(hunting): authenticate will time out at 968.843436s 2024-08-19T12:42:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.755+0000 7ff4015aa640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7ff3fc073470 0x7ff3fc136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:53478/0 (socket says 172.21.15.26:53478) 2024-08-19T12:42:54.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.755+0000 7ff4015aa640 1 -- 172.21.15.26:0/1532721643 learned_addr learned my addr 172.21.15.26:0/1532721643 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:54.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.758+0000 7ff4015aa640 1 -- 172.21.15.26:0/1532721643 >> v1:172.21.15.26:6793/0 conn(0x7ff3fc073470 legacy=0x7ff3fc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff400da9640 1 -- 172.21.15.26:0/1532721643 >> v1:172.21.15.104:6792/0 conn(0x7ff3fc0a57a0 legacy=0x7ff3fc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff401dab640 1 -- 172.21.15.26:0/1532721643 >> v1:172.21.15.104:6790/0 conn(0x7ff3fc13f2e0 legacy=0x7ff3fc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1815985807 0 0) 0x7ff3fc09dd50 con 0x7ff3fc073470 2024-08-19T12:42:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient(hunting): my global_id is 25793 2024-08-19T12:42:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3e4003520 con 0x7ff3fc073470 2024-08-19T12:42:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 895717207 0 0) 0x7ff3fc136910 con 0x7ff3fc0a57a0 2024-08-19T12:42:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:54.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient(hunting): my global_id is 134818 2024-08-19T12:42:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3fc09dd50 con 0x7ff3fc0a57a0 2024-08-19T12:42:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2429901157 0 0) 0x7ff3fc09dae0 con 0x7ff3fc13f2e0 2024-08-19T12:42:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient(hunting): my global_id is 26175 2024-08-19T12:42:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3fc136910 con 0x7ff3fc13f2e0 2024-08-19T12:42:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 757956574 0 0) 0x7ff3e4003520 con 0x7ff3fc073470 2024-08-19T12:42:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 >> v1:172.21.15.104:6792/0 conn(0x7ff3fc0a57a0 legacy=0x7ff3fc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 >> v1:172.21.15.104:6790/0 conn(0x7ff3fc13f2e0 legacy=0x7ff3fc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:54.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient: _finish_hunting 0 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 monclient: found mon.noname-e 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3fc09e030 con 0x7ff3fc073470 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient: _finish_auth 0 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient: _check_auth_tickets 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff3ec001e60 con 0x7ff3fc073470 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 5 monclient: authenticate success, global_id 25793 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient: dump: 2024-08-19T12:42:54.766 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:54.767 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff3ec004c20 con 0x7ff3fc073470 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 1 -- 172.21.15.26:0/1532721643 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff3ec005190 con 0x7ff3fc073470 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 4 set_mon_vals no callback set 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff3fbfff640 10 monclient: dump: 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:54.768 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:54.769 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:54.770 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 10 monclient: shutdown 2024-08-19T12:42:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 1 -- 172.21.15.26:0/1532721643 >> v1:172.21.15.26:6793/0 conn(0x7ff3fc073470 legacy=0x7ff3fc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 1 -- 172.21.15.26:0/1532721643 shutdown_connections 2024-08-19T12:42:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 1 -- 172.21.15.26:0/1532721643 >> 172.21.15.26:0/1532721643 conn(0x7ff3fc0abff0 msgr2=0x7ff3fc0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 1 -- 172.21.15.26:0/1532721643 shutdown_connections 2024-08-19T12:42:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 1 -- 172.21.15.26:0/1532721643 wait complete. 2024-08-19T12:42:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 10 monclient: build_initial_monmap 2024-08-19T12:42:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.759+0000 7ff40380e640 10 monclient: monmap: 2024-08-19T12:42:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:54.770 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:54.771 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 1 librados: starting msgr at 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 1 librados: starting objecter 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 1 Processor -- start 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 1 -- start start 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 1 librados: setting wanted keys 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 1 librados: calling monclient init 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 10 monclient: init 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 10 monclient: _add_conns ranks=[8,4,1,0,7,5,6,3,2] 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 10 monclient(hunting): picked mon.noname-e con 0x7ff3fc073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 10 monclient(hunting): picked mon.noname-c con 0x7ff3fc0a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 10 monclient(hunting): picked mon.noname-f con 0x7ff3fc13f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3fc1370c0 con 0x7ff3fc13f2e0 2024-08-19T12:42:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3fc1373a0 con 0x7ff3fc0a57a0 2024-08-19T12:42:54.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3fc0a46e0 con 0x7ff3fc073470 2024-08-19T12:42:54.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:54.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff40380e640 10 monclient(hunting): authenticate will time out at 968.848794s 2024-08-19T12:42:54.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff4015aa640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7ff3fc073470 0x7ff3fc1ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:53486/0 (socket says 172.21.15.26:53486) 2024-08-19T12:42:54.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.760+0000 7ff4015aa640 1 -- 172.21.15.26:0/249728914 learned_addr learned my addr 172.21.15.26:0/249728914 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:54.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.762+0000 7ff4015aa640 1 -- 172.21.15.26:0/249728914 >> v1:172.21.15.26:6793/0 conn(0x7ff3fc073470 legacy=0x7ff3fc1ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:54.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.762+0000 7ff3f9ffb640 10 client.?.objecter ms_handle_connect 0x7ff3fc073470 2024-08-19T12:42:54.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.762+0000 7ff3f9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:54.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.762+0000 7ff3f9ffb640 10 client.?.objecter ms_handle_connect 0x7ff3fc0a57a0 2024-08-19T12:42:54.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.762+0000 7ff3f9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:54.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.762+0000 7ff400da9640 1 -- 172.21.15.26:0/249728914 >> v1:172.21.15.26:6791/0 conn(0x7ff3fc0a57a0 legacy=0x7ff3fc1d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:54.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3731154994 0 0) 0x7ff3fc0a46e0 con 0x7ff3fc073470 2024-08-19T12:42:54.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:54.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:54.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:54.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): my global_id is 25802 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3e8004eb0 con 0x7ff3fc073470 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 168044698 0 0) 0x7ff3fc1373a0 con 0x7ff3fc0a57a0 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): my global_id is 26266 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3fc0a46e0 con 0x7ff3fc0a57a0 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 client.?.objecter ms_handle_connect 0x7ff3fc13f2e0 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff401dab640 1 -- 172.21.15.26:0/249728914 >> v1:172.21.15.104:6789/0 conn(0x7ff3fc13f2e0 legacy=0x7ff3fc1d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1763198622 0 0) 0x7ff3e8004eb0 con 0x7ff3fc073470 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3fc1373a0 con 0x7ff3fc073470 2024-08-19T12:42:54.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff3ec005110 con 0x7ff3fc073470 2024-08-19T12:42:54.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:54.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:42:54.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): dump: 2024-08-19T12:42:54.779 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:54.779 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:54.779 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:54.779 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:54.779 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:54.779 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:54.779 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:54.779 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:54.779 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 735632644 0 0) 0x7ff3fc0a46e0 con 0x7ff3fc0a57a0 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3e8004eb0 con 0x7ff3fc0a57a0 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff3f0003610 con 0x7ff3fc0a57a0 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:54.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): dump: 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:54.781 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2329048706 0 0) 0x7ff3fc1370c0 con 0x7ff3fc13f2e0 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient(hunting): my global_id is 26659 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3fc0a46e0 con 0x7ff3fc13f2e0 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4190509636 0 0) 0x7ff3fc1373a0 con 0x7ff3fc073470 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 >> v1:172.21.15.26:6791/0 conn(0x7ff3fc0a57a0 legacy=0x7ff3fc1d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 >> v1:172.21.15.104:6789/0 conn(0x7ff3fc13f2e0 legacy=0x7ff3fc1d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:54.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 monclient: found mon.e 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3fc0a5380 con 0x7ff3fc073470 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient: _finish_auth 0 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff40380e640 5 monclient: authenticate success, global_id 25802 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff40380e640 10 monclient: _renew_subs 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff40380e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff40380e640 1 -- 172.21.15.26:0/249728914 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff3fc136b20 con 0x7ff3fc073470 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff3ec0020b0 con 0x7ff3fc073470 2024-08-19T12:42:54.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff3ec001e80 con 0x7ff3fc073470 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3f9ffb640 10 monclient: dump: 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:54.784 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:54.785 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:54.785 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:54.785 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:54.785 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:54.785 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:54.785 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:54.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff40380e640 10 client.25802.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:54.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff40380e640 10 monclient: _renew_subs 2024-08-19T12:42:54.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff40380e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:54.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff40380e640 1 -- 172.21.15.26:0/249728914 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7ff3fc137da0 con 0x7ff3fc073470 2024-08-19T12:42:54.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff3fb7fe640 4 set_mon_vals no callback set 2024-08-19T12:42:54.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.763+0000 7ff40380e640 1 librados: init done 2024-08-19T12:42:54.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.764+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff3ec011d80 con 0x7ff3fc073470 2024-08-19T12:42:54.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.764+0000 7ff40380e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:54.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.764+0000 7ff40380e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:54.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.764+0000 7ff40380e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:54.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.764+0000 7ff40380e640 1 -- 172.21.15.26:0/249728914 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3fc06c510 con 0x7ff3fc073470 2024-08-19T12:42:54.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.764+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(184..184 src has 1..184) ==== 6259+0+0 (unknown 2211655891 0 0) 0x7ff3ec04e7f0 con 0x7ff3fc073470 2024-08-19T12:42:54.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.764+0000 7ff3f9ffb640 10 client.25802.objecter ms_dispatch 0x7ff3fc000b90 osd_map(184..184 src has 1..184) 2024-08-19T12:42:54.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.764+0000 7ff3f9ffb640 3 client.25802.objecter handle_osd_map got epochs [184,184] > 0 2024-08-19T12:42:54.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.764+0000 7ff3f9ffb640 3 client.25802.objecter handle_osd_map decoding full epoch 184 2024-08-19T12:42:54.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.765+0000 7ff3f9ffb640 20 client.25802.objecter dump_active .. 0 homeless 2024-08-19T12:42:54.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.765+0000 7ff400da9640 10 monclient: get_auth_request con 0x7ff3e80416e0 auth_method 0 2024-08-19T12:42:54.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.767+0000 7ff3f9ffb640 10 client.25802.objecter ms_handle_connect 0x7ff3e80416e0 2024-08-19T12:42:54.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.770+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7ff3ec017980 con 0x7ff3fc073470 2024-08-19T12:42:54.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.770+0000 7ff3f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:54.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:54.770+0000 7ff3f9ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:55.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.026+0000 7ff40380e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:42:55.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.026+0000 7ff40380e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:42:55.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.026+0000 7ff40380e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:55.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.026+0000 7ff40380e640 1 -- 172.21.15.26:0/249728914 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff3fc06c840 con 0x7ff3fc073470 2024-08-19T12:42:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.029+0000 7ff3f9ffb640 1 -- 172.21.15.26:0/249728914 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 1899483475) 0x7ff3ec017280 con 0x7ff3fc073470 2024-08-19T12:42:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.029+0000 7ff3f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:42:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.029+0000 7ff3f9ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:55.030 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":24,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:42:55.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.034+0000 7ff40380e640 10 librados: watch_flush enter 2024-08-19T12:42:55.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.034+0000 7ff40380e640 10 librados: watch_flush exit 2024-08-19T12:42:55.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.035+0000 7ff40380e640 20 client.25802.objecter shutdown clearing up homeless session... 2024-08-19T12:42:55.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.035+0000 7ff40380e640 10 client.25802.objecter successfully canceled tick 2024-08-19T12:42:55.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.035+0000 7ff40380e640 1 -- 172.21.15.26:0/249728914 >> v1:172.21.15.104:6800/899887506 conn(0x7ff3e80416e0 legacy=0x7ff3e8043ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.035+0000 7ff40380e640 10 monclient: shutdown 2024-08-19T12:42:55.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.035+0000 7ff40380e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:55.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.035+0000 7ff40380e640 1 -- 172.21.15.26:0/249728914 >> v1:172.21.15.26:6793/0 conn(0x7ff3fc073470 legacy=0x7ff3fc1ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.035+0000 7ff40380e640 1 -- 172.21.15.26:0/249728914 shutdown_connections 2024-08-19T12:42:55.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.035+0000 7ff40380e640 1 -- 172.21.15.26:0/249728914 >> 172.21.15.26:0/249728914 conn(0x7ff3fc0abff0 msgr2=0x7ff3fc076880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:55.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.035+0000 7ff40380e640 1 -- 172.21.15.26:0/249728914 shutdown_connections 2024-08-19T12:42:55.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.035+0000 7ff40380e640 1 -- 172.21.15.26:0/249728914 wait complete. 2024-08-19T12:42:55.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.035+0000 7ff40380e640 1 librados: shutdown 2024-08-19T12:42:55.047 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:42:55.047 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T12:42:55.047 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:42:55.047 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:42:55.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.147+0000 7fa597e20640 10 monclient: get_monmap_and_config 2024-08-19T12:42:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.147+0000 7fa597e20640 10 monclient: build_initial_monmap 2024-08-19T12:42:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.147+0000 7fa597e20640 10 monclient: monmap: 2024-08-19T12:42:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:55.147968+0000 2024-08-19T12:42:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:55.147968+0000 2024-08-19T12:42:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:55.148 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:55.149 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:55.149 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:55.149 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:55.149 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:55.149 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:55.149 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.147+0000 7fa597e20640 1 Processor -- start 2024-08-19T12:42:55.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.147+0000 7fa597e20640 1 -- start start 2024-08-19T12:42:55.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.147+0000 7fa597e20640 10 monclient: init 2024-08-19T12:42:55.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa597e20640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:55.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa597e20640 10 monclient: _add_conns ranks=[7,3,1,4,0,6,5,2,8] 2024-08-19T12:42:55.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa597e20640 10 monclient(hunting): picked mon.noname-i con 0x7fa590073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:55.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa597e20640 10 monclient(hunting): picked mon.noname-g con 0x7fa5900a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:55.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa597e20640 10 monclient(hunting): picked mon.noname-f con 0x7fa59013f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:55.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa597e20640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa59009dae0 con 0x7fa59013f2e0 2024-08-19T12:42:55.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa597e20640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa590136910 con 0x7fa5900a57a0 2024-08-19T12:42:55.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa597e20640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa59009dd50 con 0x7fa590073470 2024-08-19T12:42:55.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa597e20640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:55.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa597e20640 10 monclient(hunting): authenticate will time out at 969.235418s 2024-08-19T12:42:55.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa5963bd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa59013f2e0 0x7fa5901416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:38098/0 (socket says 172.21.15.26:38098) 2024-08-19T12:42:55.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa5963bd640 1 -- 172.21.15.26:0/177973892 learned_addr learned my addr 172.21.15.26:0/177973892 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:55.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.148+0000 7fa595bbc640 1 --1- 172.21.15.26:0/177973892 >> v1:172.21.15.104:6792/0 conn(0x7fa590073470 0x7fa590136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:55578/0 (socket says 172.21.15.26:55578) 2024-08-19T12:42:55.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.150+0000 7fa5953bb640 1 -- 172.21.15.26:0/177973892 >> v1:172.21.15.104:6790/0 conn(0x7fa5900a57a0 legacy=0x7fa5900a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.150+0000 7fa595bbc640 1 -- 172.21.15.26:0/177973892 >> v1:172.21.15.104:6792/0 conn(0x7fa590073470 legacy=0x7fa590136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.150+0000 7fa5963bd640 1 -- 172.21.15.26:0/177973892 >> v1:172.21.15.104:6789/0 conn(0x7fa59013f2e0 legacy=0x7fa5901416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3286735968 0 0) 0x7fa59009dd50 con 0x7fa590073470 2024-08-19T12:42:55.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 10 monclient(hunting): my global_id is 134827 2024-08-19T12:42:55.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa578003520 con 0x7fa590073470 2024-08-19T12:42:55.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1227521375 0 0) 0x7fa59009dae0 con 0x7fa59013f2e0 2024-08-19T12:42:55.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 10 monclient(hunting): my global_id is 26668 2024-08-19T12:42:55.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa59009dd50 con 0x7fa59013f2e0 2024-08-19T12:42:55.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1746526167 0 0) 0x7fa590136910 con 0x7fa5900a57a0 2024-08-19T12:42:55.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 10 monclient(hunting): my global_id is 26184 2024-08-19T12:42:55.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.166+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa59009dae0 con 0x7fa5900a57a0 2024-08-19T12:42:55.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 249230229 0 0) 0x7fa59009dd50 con 0x7fa59013f2e0 2024-08-19T12:42:55.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 >> v1:172.21.15.104:6792/0 conn(0x7fa590073470 legacy=0x7fa590136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 >> v1:172.21.15.104:6790/0 conn(0x7fa5900a57a0 legacy=0x7fa5900a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 10 monclient: _finish_hunting 0 2024-08-19T12:42:55.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 1 monclient: found mon.noname-f 2024-08-19T12:42:55.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59009e030 con 0x7fa59013f2e0 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 10 monclient: _finish_auth 0 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 10 monclient: _check_auth_tickets 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa580001d70 con 0x7fa59013f2e0 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 10 monclient: dump: 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.176 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa597e20640 5 monclient: authenticate success, global_id 26668 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa597e20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa580002040 con 0x7fa59013f2e0 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 1 -- 172.21.15.26:0/177973892 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa580004f60 con 0x7fa59013f2e0 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa594bba640 10 monclient: dump: 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.178 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa597e20640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.172+0000 7fa597e20640 4 set_mon_vals no callback set 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.173+0000 7fa597e20640 10 monclient: shutdown 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.173+0000 7fa597e20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.173+0000 7fa597e20640 1 -- 172.21.15.26:0/177973892 >> v1:172.21.15.104:6789/0 conn(0x7fa59013f2e0 legacy=0x7fa5901416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.174+0000 7fa597e20640 1 -- 172.21.15.26:0/177973892 shutdown_connections 2024-08-19T12:42:55.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.174+0000 7fa597e20640 1 -- 172.21.15.26:0/177973892 >> 172.21.15.26:0/177973892 conn(0x7fa5900abff0 msgr2=0x7fa5900ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.174+0000 7fa597e20640 1 -- 172.21.15.26:0/177973892 shutdown_connections 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.174+0000 7fa597e20640 1 -- 172.21.15.26:0/177973892 wait complete. 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.174+0000 7fa597e20640 10 monclient: build_initial_monmap 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.175+0000 7fa597e20640 10 monclient: monmap: 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:55.180 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:55.181 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:55.181 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:55.181 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:55.181 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:55.181 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:55.181 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.175+0000 7fa597e20640 1 librados: starting msgr at 2024-08-19T12:42:55.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.175+0000 7fa597e20640 1 librados: starting objecter 2024-08-19T12:42:55.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.175+0000 7fa597e20640 1 Processor -- start 2024-08-19T12:42:55.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.175+0000 7fa597e20640 1 -- start start 2024-08-19T12:42:55.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.175+0000 7fa597e20640 1 librados: setting wanted keys 2024-08-19T12:42:55.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.175+0000 7fa597e20640 1 librados: calling monclient init 2024-08-19T12:42:55.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.175+0000 7fa597e20640 10 monclient: init 2024-08-19T12:42:55.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.175+0000 7fa597e20640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.175+0000 7fa597e20640 10 monclient: _add_conns ranks=[1,6,7,2,5,8,4,3,0] 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.175+0000 7fa597e20640 10 monclient(hunting): picked mon.noname-f con 0x7fa590073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.176+0000 7fa597e20640 10 monclient(hunting): picked mon.noname-d con 0x7fa5900a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.176+0000 7fa597e20640 10 monclient(hunting): picked mon.noname-i con 0x7fa59013f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.176+0000 7fa597e20640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa590131810 con 0x7fa590073470 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.176+0000 7fa597e20640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa59013f070 con 0x7fa5900a57a0 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.176+0000 7fa597e20640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa59012ed50 con 0x7fa59013f2e0 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.176+0000 7fa597e20640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.176+0000 7fa597e20640 10 monclient(hunting): authenticate will time out at 969.263208s 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.176+0000 7fa5953bb640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fa5900a57a0 0x7fa59012e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34550/0 (socket says 172.21.15.26:34550) 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.176+0000 7fa5953bb640 1 -- 172.21.15.26:0/491639404 learned_addr learned my addr 172.21.15.26:0/491639404 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa5953bb640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.26:6792/0 conn(0x7fa5900a57a0 legacy=0x7fa59012e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa595bbc640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.104:6789/0 conn(0x7fa590073470 legacy=0x7fa59012df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 client.?.objecter ms_handle_connect 0x7fa5900a57a0 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 client.?.objecter ms_handle_connect 0x7fa590073470 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa5963bd640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.104:6792/0 conn(0x7fa59013f2e0 legacy=0x7fa5901d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 client.?.objecter ms_handle_connect 0x7fa59013f2e0 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009880560 0 0) 0x7fa590131810 con 0x7fa590073470 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 monclient(hunting): my global_id is 26677 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa57c004eb0 con 0x7fa590073470 2024-08-19T12:42:55.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2953728107 0 0) 0x7fa59013f070 con 0x7fa5900a57a0 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 monclient(hunting): my global_id is 26277 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa590131810 con 0x7fa5900a57a0 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2634858660 0 0) 0x7fa59012ed50 con 0x7fa59013f2e0 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 10 monclient(hunting): my global_id is 134836 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.178+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa59013f070 con 0x7fa59013f2e0 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1252147 0 0) 0x7fa590131810 con 0x7fa5900a57a0 2024-08-19T12:42:55.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa59012ed50 con 0x7fa5900a57a0 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa584003610 con 0x7fa5900a57a0 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient(hunting): dump: 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.191 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1574688515 0 0) 0x7fa57c004eb0 con 0x7fa590073470 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa590131810 con 0x7fa590073470 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa5880047e0 con 0x7fa590073470 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient(hunting): dump: 2024-08-19T12:42:55.192 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.193 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1405878824 0 0) 0x7fa59013f070 con 0x7fa59013f2e0 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa57c004eb0 con 0x7fa59013f2e0 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa580004ea0 con 0x7fa59013f2e0 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient(hunting): dump: 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.194 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 189845546 0 0) 0x7fa59012ed50 con 0x7fa5900a57a0 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.104:6792/0 conn(0x7fa59013f2e0 legacy=0x7fa5901d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.104:6789/0 conn(0x7fa590073470 legacy=0x7fa59012df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 monclient: found mon.d 2024-08-19T12:42:55.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa590131530 con 0x7fa5900a57a0 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient: _finish_auth 0 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa58e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa597e20640 5 monclient: authenticate success, global_id 26277 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa597e20640 10 monclient: _renew_subs 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa597e20640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa597e20640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa59012fe70 con 0x7fa5900a57a0 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa597e20640 10 client.26277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa597e20640 10 monclient: _renew_subs 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa597e20640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:55.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa597e20640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa5901321d0 con 0x7fa5900a57a0 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.179+0000 7fa597e20640 1 librados: init done 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.180+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa584003ae0 con 0x7fa5900a57a0 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.180+0000 7fa58e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.180+0000 7fa58ffff640 4 set_mon_vals no callback set 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.181+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa584003fd0 con 0x7fa5900a57a0 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.181+0000 7fa58e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.181+0000 7fa58e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.181+0000 7fa58e7fc640 10 monclient: dump: 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.181+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa584004170 con 0x7fa5900a57a0 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.181+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(185..185 src has 1..185) ==== 5746+0+0 (unknown 2474800286 0 0) 0x7fa58404e3f0 con 0x7fa5900a57a0 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.181+0000 7fa58e7fc640 10 client.26277.objecter ms_dispatch 0x7fa590000b90 osd_map(185..185 src has 1..185) 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.181+0000 7fa58e7fc640 3 client.26277.objecter handle_osd_map got epochs [185,185] > 0 2024-08-19T12:42:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.181+0000 7fa58e7fc640 3 client.26277.objecter handle_osd_map decoding full epoch 185 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.181+0000 7fa58e7fc640 20 client.26277.objecter dump_active .. 0 homeless 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.181+0000 7fa595bbc640 10 monclient: get_auth_request con 0x7fa57c041a30 auth_method 0 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.182+0000 7fa597e20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.182+0000 7fa597e20640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa5700065c0 con 0x7fa570002860 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.184+0000 7fa58e7fc640 10 client.26277.objecter ms_handle_connect 0x7fa57c041a30 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.184+0000 7fa5963bd640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.26:6789/0 conn(0x7fa570002860 legacy=0x7fa570004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.184+0000 7fa58e7fc640 10 client.26277.objecter ms_handle_connect 0x7fa570002860 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.184+0000 7fa58e7fc640 10 client.26277.objecter resend_mon_ops 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.184+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3519213959 0 0) 0x7fa5700065c0 con 0x7fa570002860 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.184+0000 7fa58e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.184+0000 7fa58e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.184+0000 7fa58e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.184+0000 7fa58e7fc640 10 monclient(hunting): my global_id is 144198 2024-08-19T12:42:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.184+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa59012ed50 con 0x7fa570002860 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1086400256 0 0) 0x7fa59012ed50 con 0x7fa570002860 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5700065c0 con 0x7fa570002860 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa580001fc0 con 0x7fa570002860 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 10 monclient: dump: 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3644716552 0 0) 0x7fa5700065c0 con 0x7fa570002860 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa57000a160 con 0x7fa570002860 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa5800073e0 con 0x7fa570002860 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.26:6789/0 conn(0x7fa570002860 legacy=0x7fa570004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.185+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.26:6789/0 conn(0x7fa570002860 legacy=0x7fa570004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:55.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.192+0000 7fa597e20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:55.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.192+0000 7fa597e20640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa570009e70 con 0x7fa570009130 2024-08-19T12:42:55.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa5963bd640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.26:6789/0 conn(0x7fa570009130 legacy=0x7fa5700095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 10 client.26277.objecter ms_handle_connect 0x7fa570009130 2024-08-19T12:42:55.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 10 client.26277.objecter resend_mon_ops 2024-08-19T12:42:55.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3619480593 0 0) 0x7fa570009e70 con 0x7fa570009130 2024-08-19T12:42:55.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 10 monclient(hunting): my global_id is 144207 2024-08-19T12:42:55.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5700065c0 con 0x7fa570009130 2024-08-19T12:42:55.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3106237754 0 0) 0x7fa5700065c0 con 0x7fa570009130 2024-08-19T12:42:55.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa570009e70 con 0x7fa570009130 2024-08-19T12:42:55.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa580005b20 con 0x7fa570009130 2024-08-19T12:42:55.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:55.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 10 monclient: dump: 2024-08-19T12:42:55.204 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.206 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1366659460 0 0) 0x7fa570009e70 con 0x7fa570009130 2024-08-19T12:42:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.194+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa57000c410 con 0x7fa570009130 2024-08-19T12:42:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7545 (unknown 0 0 3517511063) 0x7fa580009d90 con 0x7fa570009130 2024-08-19T12:42:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa58e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa58e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.26:6789/0 conn(0x7fa570009130 legacy=0x7fa5700095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa58e7fc640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.26:6789/0 conn(0x7fa570009130 legacy=0x7fa5700095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:55.206 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:55.206 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 27112, 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:55.207 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.208 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.209 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:55.210 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:55.211 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.212 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.213 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:55.214 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.215 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.216 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.218 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:55.221 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.222 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:55.225 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.226 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.227 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:55.227 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 10 librados: watch_flush enter 2024-08-19T12:42:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 10 librados: watch_flush exit 2024-08-19T12:42:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 20 client.26277.objecter shutdown clearing up homeless session... 2024-08-19T12:42:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 10 client.26277.objecter successfully canceled tick 2024-08-19T12:42:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.104:6800/899887506 conn(0x7fa57c041a30 legacy=0x7fa57c043ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 10 monclient: shutdown 2024-08-19T12:42:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 1 -- 172.21.15.26:0/491639404 >> v1:172.21.15.26:6792/0 conn(0x7fa5900a57a0 legacy=0x7fa59012e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa5963bd640 1 -- 172.21.15.26:0/491639404 reap_dead start 2024-08-19T12:42:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa5963bd640 1 -- 172.21.15.26:0/491639404 reap_dead start 2024-08-19T12:42:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 1 -- 172.21.15.26:0/491639404 shutdown_connections 2024-08-19T12:42:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 1 -- 172.21.15.26:0/491639404 >> 172.21.15.26:0/491639404 conn(0x7fa5900abff0 msgr2=0x7fa59012c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 1 -- 172.21.15.26:0/491639404 shutdown_connections 2024-08-19T12:42:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 1 -- 172.21.15.26:0/491639404 wait complete. 2024-08-19T12:42:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.195+0000 7fa597e20640 1 librados: shutdown 2024-08-19T12:42:55.230 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:55.230 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:42:55.230 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:42:55.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.329+0000 7fe1d7636640 10 monclient: get_monmap_and_config 2024-08-19T12:42:55.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.329+0000 7fe1d7636640 10 monclient: build_initial_monmap 2024-08-19T12:42:55.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.329+0000 7fe1d7636640 10 monclient: monmap: 2024-08-19T12:42:55.330 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:55.330 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.330 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:55.330324+0000 2024-08-19T12:42:55.330 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:55.330324+0000 2024-08-19T12:42:55.330 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:55.331 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.331 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:55.331 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:55.331 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:55.331 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:55.331 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:55.331 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:55.331 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:55.331 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:55.331 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:55.331 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 1 Processor -- start 2024-08-19T12:42:55.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 1 -- start start 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 10 monclient: init 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 10 monclient: _add_conns ranks=[1,6,7,8,2,0,3,5,4] 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 10 monclient(hunting): picked mon.noname-f con 0x7fe1d006e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 10 monclient(hunting): picked mon.noname-d con 0x7fe1d00a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 10 monclient(hunting): picked mon.noname-i con 0x7fe1d00a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d0136790 con 0x7fe1d006e7b0 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d009ddf0 con 0x7fe1d00a5920 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d009e0d0 con 0x7fe1d00a5d50 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d7636640 10 monclient(hunting): authenticate will time out at 969.417832s 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d4bd1640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fe1d00a5920 0x7fe1d013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34564/0 (socket says 172.21.15.26:34564) 2024-08-19T12:42:55.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.330+0000 7fe1d4bd1640 1 -- 172.21.15.26:0/3298186717 learned_addr learned my addr 172.21.15.26:0/3298186717 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:55.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1d4bd1640 1 -- 172.21.15.26:0/3298186717 >> v1:172.21.15.26:6792/0 conn(0x7fe1d00a5920 legacy=0x7fe1d013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1d5bd3640 1 -- 172.21.15.26:0/3298186717 >> v1:172.21.15.104:6792/0 conn(0x7fe1d00a5d50 legacy=0x7fe1d0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1d53d2640 1 -- 172.21.15.26:0/3298186717 >> v1:172.21.15.104:6789/0 conn(0x7fe1d006e7b0 legacy=0x7fe1d0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3378575251 0 0) 0x7fe1c4001450 con 0x7fe1d00a5920 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient(hunting): my global_id is 26295 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1b8003520 con 0x7fe1d00a5920 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2090343520 0 0) 0x7fe1d009e0d0 con 0x7fe1d00a5d50 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient(hunting): my global_id is 134854 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1c4001450 con 0x7fe1d00a5d50 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1283336871 0 0) 0x7fe1d0136790 con 0x7fe1d006e7b0 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient(hunting): my global_id is 26695 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1d009e0d0 con 0x7fe1d006e7b0 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3951899850 0 0) 0x7fe1b8003520 con 0x7fe1d00a5920 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 >> v1:172.21.15.104:6792/0 conn(0x7fe1d00a5d50 legacy=0x7fe1d0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 >> v1:172.21.15.104:6789/0 conn(0x7fe1d006e7b0 legacy=0x7fe1d0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient: _finish_hunting 0 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 monclient: found mon.noname-d 2024-08-19T12:42:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d009e3b0 con 0x7fe1d00a5920 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient: _finish_auth 0 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient: _check_auth_tickets 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1d7636640 5 monclient: authenticate success, global_id 26295 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1c4001cb0 con 0x7fe1d00a5920 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1d7636640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient: dump: 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.346 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1d7636640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1c4004950 con 0x7fe1d00a5920 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 1 -- 172.21.15.26:0/3298186717 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1c4004e80 con 0x7fe1d00a5920 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1d7636640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1d7636640 4 set_mon_vals no callback set 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.333+0000 7fe1cffff640 10 monclient: dump: 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.348 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.334+0000 7fe1d7636640 10 monclient: shutdown 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.334+0000 7fe1d7636640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:55.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.334+0000 7fe1d7636640 1 -- 172.21.15.26:0/3298186717 >> v1:172.21.15.26:6792/0 conn(0x7fe1d00a5920 legacy=0x7fe1d013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.334+0000 7fe1d7636640 1 -- 172.21.15.26:0/3298186717 shutdown_connections 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.334+0000 7fe1d7636640 1 -- 172.21.15.26:0/3298186717 >> 172.21.15.26:0/3298186717 conn(0x7fe1d00a8740 msgr2=0x7fe1d00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.334+0000 7fe1d7636640 1 -- 172.21.15.26:0/3298186717 shutdown_connections 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.334+0000 7fe1d7636640 1 -- 172.21.15.26:0/3298186717 wait complete. 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.334+0000 7fe1d7636640 10 monclient: build_initial_monmap 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.334+0000 7fe1d7636640 10 monclient: monmap: 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.350 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 1 librados: starting msgr at 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 1 librados: starting objecter 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 1 Processor -- start 2024-08-19T12:42:55.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 1 -- start start 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 1 librados: setting wanted keys 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 1 librados: calling monclient init 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 10 monclient: init 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 10 monclient: _add_conns ranks=[1,6,5,3,8,7,2,4,0] 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 10 monclient(hunting): picked mon.noname-f con 0x7fe1d006e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 10 monclient(hunting): picked mon.noname-d con 0x7fe1d00a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 10 monclient(hunting): picked mon.noname-h con 0x7fe1d00a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d0125170 con 0x7fe1d006e7b0 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d0125450 con 0x7fe1d00a5d50 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d0125730 con 0x7fe1d00a5920 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:55.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d7636640 10 monclient(hunting): authenticate will time out at 969.422659s 2024-08-19T12:42:55.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d4bd1640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fe1d00a5920 0x7fe1d0122c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34576/0 (socket says 172.21.15.26:34576) 2024-08-19T12:42:55.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.335+0000 7fe1d4bd1640 1 -- 172.21.15.26:0/1436508984 learned_addr learned my addr 172.21.15.26:0/1436508984 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:55.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.337+0000 7fe1d4bd1640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.26:6792/0 conn(0x7fe1d00a5920 legacy=0x7fe1d0122c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.337+0000 7fe1cdffb640 10 client.?.objecter ms_handle_connect 0x7fe1d00a5920 2024-08-19T12:42:55.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.337+0000 7fe1cdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.337+0000 7fe1cdffb640 10 client.?.objecter ms_handle_connect 0x7fe1d006e7b0 2024-08-19T12:42:55.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.337+0000 7fe1cdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.337+0000 7fe1d53d2640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.104:6789/0 conn(0x7fe1d006e7b0 legacy=0x7fe1d0122500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.337+0000 7fe1cdffb640 10 client.?.objecter ms_handle_connect 0x7fe1d00a5d50 2024-08-19T12:42:55.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.337+0000 7fe1cdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.337+0000 7fe1d5bd3640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.104:6791/0 conn(0x7fe1d00a5d50 legacy=0x7fe1d01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2584450549 0 0) 0x7fe1d0125730 con 0x7fe1d00a5920 2024-08-19T12:42:55.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): my global_id is 26304 2024-08-19T12:42:55.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1bc004eb0 con 0x7fe1d00a5920 2024-08-19T12:42:55.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3623420943 0 0) 0x7fe1d0125170 con 0x7fe1d006e7b0 2024-08-19T12:42:55.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): my global_id is 26704 2024-08-19T12:42:55.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1d0125730 con 0x7fe1d006e7b0 2024-08-19T12:42:55.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3834662262 0 0) 0x7fe1d0125450 con 0x7fe1d00a5d50 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): my global_id is 25988 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1d0125170 con 0x7fe1d00a5d50 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1476121235 0 0) 0x7fe1bc004eb0 con 0x7fe1d00a5920 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1d0125450 con 0x7fe1d00a5920 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1c4004e40 con 0x7fe1d00a5920 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): dump: 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.355 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2381764146 0 0) 0x7fe1d0125730 con 0x7fe1d006e7b0 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1bc004eb0 con 0x7fe1d006e7b0 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1c8003610 con 0x7fe1d006e7b0 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): dump: 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.357 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2669130531 0 0) 0x7fe1d0125170 con 0x7fe1d00a5d50 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1d0125730 con 0x7fe1d00a5d50 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1c0003990 con 0x7fe1d00a5d50 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:42:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient(hunting): dump: 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.359 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3225596865 0 0) 0x7fe1d0125450 con 0x7fe1d00a5920 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.104:6791/0 conn(0x7fe1d00a5d50 legacy=0x7fe1d01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.104:6789/0 conn(0x7fe1d006e7b0 legacy=0x7fe1d0122500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient: _finish_hunting 0 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 monclient: found mon.d 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d0124210 con 0x7fe1d00a5920 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient: _finish_auth 0 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient: _check_auth_tickets 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1d7636640 5 monclient: authenticate success, global_id 26304 2024-08-19T12:42:55.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1d7636640 10 monclient: _renew_subs 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1d7636640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1d7636640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d0124cb0 con 0x7fe1d00a5920 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1c4001c50 con 0x7fe1d00a5920 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1c40057b0 con 0x7fe1d00a5920 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cdffb640 10 monclient: dump: 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.361 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1d7636640 10 client.26304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:55.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1d7636640 10 monclient: _renew_subs 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1d7636640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1d7636640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1d0123500 con 0x7fe1d00a5920 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1cf7fe640 4 set_mon_vals no callback set 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.338+0000 7fe1d7636640 1 librados: init done 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.339+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe1c4011c10 con 0x7fe1d00a5920 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.339+0000 7fe1d7636640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.339+0000 7fe1d7636640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1d00008d0 con 0x7fe1d002e340 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.339+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(185..185 src has 1..185) ==== 5746+0+0 (unknown 2474800286 0 0) 0x7fe1c404e510 con 0x7fe1d00a5920 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.339+0000 7fe1cdffb640 10 client.26304.objecter ms_dispatch 0x7fe1d0000b90 osd_map(185..185 src has 1..185) 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.339+0000 7fe1cdffb640 3 client.26304.objecter handle_osd_map got epochs [185,185] > 0 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.339+0000 7fe1cdffb640 3 client.26304.objecter handle_osd_map decoding full epoch 185 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.339+0000 7fe1cdffb640 20 client.26304.objecter dump_active .. 0 homeless 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.339+0000 7fe1d5bd3640 10 monclient: get_auth_request con 0x7fe1bc0419e0 auth_method 0 2024-08-19T12:42:55.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.341+0000 7fe1cdffb640 10 client.26304.objecter ms_handle_connect 0x7fe1d002e340 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.341+0000 7fe1cdffb640 10 client.26304.objecter resend_mon_ops 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.341+0000 7fe1d53d2640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.26:6790/0 conn(0x7fe1d002e340 legacy=0x7fe1d0063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.341+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4099123149 0 0) 0x7fe1d00008d0 con 0x7fe1d002e340 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.341+0000 7fe1cdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.341+0000 7fe1cdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.341+0000 7fe1cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.341+0000 7fe1cdffb640 10 monclient(hunting): my global_id is 26021 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1d0125450 con 0x7fe1d002e340 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2149808968 0 0) 0x7fe1d0125450 con 0x7fe1d002e340 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1d00008d0 con 0x7fe1d002e340 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1c8003e10 con 0x7fe1d002e340 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 10 monclient: dump: 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.365 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.366 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.366 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.366 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 10 client.26304.objecter ms_handle_connect 0x7fe1bc0419e0 2024-08-19T12:42:55.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2495018104 0 0) 0x7fe1d00008d0 con 0x7fe1d002e340 2024-08-19T12:42:55.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe1d002e720 con 0x7fe1d002e340 2024-08-19T12:42:55.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe1c8004100 con 0x7fe1d002e340 2024-08-19T12:42:55.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:55.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:55.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.26:6790/0 conn(0x7fe1d002e340 legacy=0x7fe1d0063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.342+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.26:6790/0 conn(0x7fe1d002e340 legacy=0x7fe1d0063a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:55.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.348+0000 7fe1d7636640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:55.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.348+0000 7fe1d7636640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1d0064190 con 0x7fe1d00a18d0 2024-08-19T12:42:55.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1d53d2640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.26:6790/0 conn(0x7fe1d00a18d0 legacy=0x7fe1d01db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 10 client.26304.objecter ms_handle_connect 0x7fe1d00a18d0 2024-08-19T12:42:55.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 10 client.26304.objecter resend_mon_ops 2024-08-19T12:42:55.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3144009535 0 0) 0x7fe1d0064190 con 0x7fe1d00a18d0 2024-08-19T12:42:55.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 10 monclient(hunting): my global_id is 26030 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1d00008d0 con 0x7fe1d00a18d0 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 970845246 0 0) 0x7fe1d00008d0 con 0x7fe1d00a18d0 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1d0064190 con 0x7fe1d00a18d0 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1c8005b00 con 0x7fe1d00a18d0 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 10 monclient: dump: 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.369 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3075573482 0 0) 0x7fe1d0064190 con 0x7fe1d00a18d0 2024-08-19T12:42:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.351+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe1d0126410 con 0x7fe1d00a18d0 2024-08-19T12:42:55.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 3067224244) 0x7fe1c8006f50 con 0x7fe1d00a18d0 2024-08-19T12:42:55.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1cdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:55.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1cdffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:55.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.26:6790/0 conn(0x7fe1d00a18d0 legacy=0x7fe1d01db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1cdffb640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.26:6790/0 conn(0x7fe1d00a18d0 legacy=0x7fe1d01db9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:55.371 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:55.371 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:55.371 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:55.371 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 288406, 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:55.372 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.373 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.374 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.374 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.374 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.374 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:55.374 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:55.374 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.374 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.374 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.374 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.374 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.374 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.374 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:55.375 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.376 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:55.377 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.378 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.379 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:55.380 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.381 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.381 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.381 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:55.381 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:55.382 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.382 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.382 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.382 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:55.382 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:55.382 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.382 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:55.383 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.384 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.386 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.387 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:55.388 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.389 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.390 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:55.390 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:55.390 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.390 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.390 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.390 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.390 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.390 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:55.390 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:55.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.391 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:55.392 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:55.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 10 librados: watch_flush enter 2024-08-19T12:42:55.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 10 librados: watch_flush exit 2024-08-19T12:42:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 20 client.26304.objecter shutdown clearing up homeless session... 2024-08-19T12:42:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 10 client.26304.objecter successfully canceled tick 2024-08-19T12:42:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.104:6800/899887506 conn(0x7fe1bc0419e0 legacy=0x7fe1bc043ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 10 monclient: shutdown 2024-08-19T12:42:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 1 -- 172.21.15.26:0/1436508984 >> v1:172.21.15.26:6792/0 conn(0x7fe1d00a5920 legacy=0x7fe1d0122c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d5bd3640 1 -- 172.21.15.26:0/1436508984 reap_dead start 2024-08-19T12:42:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d5bd3640 1 -- 172.21.15.26:0/1436508984 reap_dead start 2024-08-19T12:42:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 1 -- 172.21.15.26:0/1436508984 shutdown_connections 2024-08-19T12:42:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 1 -- 172.21.15.26:0/1436508984 >> 172.21.15.26:0/1436508984 conn(0x7fe1d00a8740 msgr2=0x7fe1d0135150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 1 -- 172.21.15.26:0/1436508984 shutdown_connections 2024-08-19T12:42:55.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 1 -- 172.21.15.26:0/1436508984 wait complete. 2024-08-19T12:42:55.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.352+0000 7fe1d7636640 1 librados: shutdown 2024-08-19T12:42:55.395 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:55.395 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:42:55.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:42:55.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.493+0000 7f4cb8d5a640 10 monclient: get_monmap_and_config 2024-08-19T12:42:55.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.493+0000 7f4cb8d5a640 10 monclient: build_initial_monmap 2024-08-19T12:42:55.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.493+0000 7f4cb8d5a640 10 monclient: monmap: 2024-08-19T12:42:55.495 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:55.495 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.495 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:55.494779+0000 2024-08-19T12:42:55.495 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:55.494779+0000 2024-08-19T12:42:55.495 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:55.495 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.495 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:55.495 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:55.495 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:55.495 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:55.496 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:55.496 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:55.496 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:55.496 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:55.496 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:55.496 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.494+0000 7f4cb8d5a640 1 Processor -- start 2024-08-19T12:42:55.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.494+0000 7f4cb8d5a640 1 -- start start 2024-08-19T12:42:55.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.494+0000 7f4cb8d5a640 10 monclient: init 2024-08-19T12:42:55.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb8d5a640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:55.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb8d5a640 10 monclient: _add_conns ranks=[5,0,2,3,7,1,8,6,4] 2024-08-19T12:42:55.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb8d5a640 10 monclient(hunting): picked mon.noname-h con 0x7f4cb406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:42:55.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb8d5a640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb40a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:55.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb8d5a640 10 monclient(hunting): picked mon.noname-b con 0x7f4cb40a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:42:55.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb8d5a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cb4136790 con 0x7f4cb40a5920 2024-08-19T12:42:55.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb8d5a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cb409ddf0 con 0x7f4cb40a5d50 2024-08-19T12:42:55.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb8d5a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cb409e0d0 con 0x7f4cb406e7b0 2024-08-19T12:42:55.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb8d5a640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:55.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb8d5a640 10 monclient(hunting): authenticate will time out at 969.582246s 2024-08-19T12:42:55.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb1d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4cb40a5920 0x7f4cb413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57938/0 (socket says 172.21.15.26:57938) 2024-08-19T12:42:55.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb2d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4cb40a5d50 0x7f4cb4141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51742/0 (socket says 172.21.15.26:51742) 2024-08-19T12:42:55.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.495+0000 7f4cb1d9b640 1 -- 172.21.15.26:0/1552779911 learned_addr learned my addr 172.21.15.26:0/1552779911 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:55.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb2d9d640 1 -- 172.21.15.26:0/1552779911 >> v1:172.21.15.26:6790/0 conn(0x7f4cb40a5d50 legacy=0x7f4cb4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb1d9b640 1 -- 172.21.15.26:0/1552779911 >> v1:172.21.15.26:6789/0 conn(0x7f4cb40a5920 legacy=0x7f4cb413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb259c640 1 -- 172.21.15.26:0/1552779911 >> v1:172.21.15.104:6791/0 conn(0x7f4cb406e7b0 legacy=0x7f4cb4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 980124969 0 0) 0x7f4cb4136790 con 0x7f4cb40a5920 2024-08-19T12:42:55.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 10 monclient(hunting): my global_id is 144216 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ca0003520 con 0x7f4cb40a5920 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2306397988 0 0) 0x7f4ca4001450 con 0x7f4cb40a5d50 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 10 monclient(hunting): my global_id is 26039 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cb4136790 con 0x7f4cb40a5d50 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2348385696 0 0) 0x7f4cb409e0d0 con 0x7f4cb406e7b0 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 10 monclient(hunting): my global_id is 25997 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ca4001450 con 0x7f4cb406e7b0 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 718250798 0 0) 0x7f4ca0003520 con 0x7f4cb40a5920 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.497+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 >> v1:172.21.15.104:6791/0 conn(0x7f4cb406e7b0 legacy=0x7f4cb4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 >> v1:172.21.15.26:6790/0 conn(0x7f4cb40a5d50 legacy=0x7f4cb4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 10 monclient: _finish_hunting 0 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 1 monclient: found mon.noname-a 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb409e3b0 con 0x7f4cb40a5920 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 10 monclient: _finish_auth 0 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 10 monclient: _check_auth_tickets 2024-08-19T12:42:55.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb8d5a640 5 monclient: authenticate success, global_id 144216 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c9c001a10 con 0x7f4cb40a5920 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb8d5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 10 monclient: dump: 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.505 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.506 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb8d5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4c9c001ce0 con 0x7f4cb40a5920 2024-08-19T12:42:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 1 -- 172.21.15.26:0/1552779911 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c9c004be0 con 0x7f4cb40a5920 2024-08-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb8d5a640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb8d5a640 4 set_mon_vals no callback set 2024-08-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb159a640 10 monclient: dump: 2024-08-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb8d5a640 10 monclient: shutdown 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb8d5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/1552779911 >> v1:172.21.15.26:6789/0 conn(0x7f4cb40a5920 legacy=0x7f4cb413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/1552779911 shutdown_connections 2024-08-19T12:42:55.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/1552779911 >> 172.21.15.26:0/1552779911 conn(0x7f4cb40a8740 msgr2=0x7f4cb40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/1552779911 shutdown_connections 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.498+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/1552779911 wait complete. 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.499+0000 7f4cb8d5a640 10 monclient: build_initial_monmap 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.499+0000 7f4cb8d5a640 10 monclient: monmap: 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:55.509 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.499+0000 7f4cb8d5a640 1 librados: starting msgr at 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.499+0000 7f4cb8d5a640 1 librados: starting objecter 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.499+0000 7f4cb8d5a640 1 Processor -- start 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.499+0000 7f4cb8d5a640 1 -- start start 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.499+0000 7f4cb8d5a640 1 librados: setting wanted keys 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.499+0000 7f4cb8d5a640 1 librados: calling monclient init 2024-08-19T12:42:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.499+0000 7f4cb8d5a640 10 monclient: init 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb8d5a640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb8d5a640 10 monclient: _add_conns ranks=[4,8,0,1,5,7,3,2,6] 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb8d5a640 10 monclient(hunting): picked mon.noname-c con 0x7f4cb406e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb8d5a640 10 monclient(hunting): picked mon.noname-e con 0x7f4cb40a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb8d5a640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb40a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb8d5a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cb40a09c0 con 0x7f4cb40a5d50 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb8d5a640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cb409f6c0 con 0x7f4cb406e7b0 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb8d5a640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cb40a06e0 con 0x7f4cb40a5920 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb8d5a640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb8d5a640 10 monclient(hunting): authenticate will time out at 969.587357s 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb259c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f4cb406e7b0 0x7f4cb4131e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:54476/0 (socket says 172.21.15.26:54476) 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb2d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4cb40a5d50 0x7f4cb4132c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57942/0 (socket says 172.21.15.26:57942) 2024-08-19T12:42:55.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.500+0000 7f4cb259c640 1 -- 172.21.15.26:0/795976649 learned_addr learned my addr 172.21.15.26:0/795976649 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:55.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4cb1d9b640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.26:6793/0 conn(0x7f4cb40a5920 legacy=0x7f4cb4132540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 client.?.objecter ms_handle_connect 0x7f4cb40a5920 2024-08-19T12:42:55.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4cb2d9d640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.26:6789/0 conn(0x7f4cb40a5d50 legacy=0x7f4cb4132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4cb259c640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.26:6791/0 conn(0x7f4cb406e7b0 legacy=0x7f4cb4131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 client.?.objecter ms_handle_connect 0x7f4cb40a5d50 2024-08-19T12:42:55.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 client.?.objecter ms_handle_connect 0x7f4cb406e7b0 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2553271614 0 0) 0x7f4cb409f6c0 con 0x7f4cb406e7b0 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 monclient(hunting): my global_id is 26275 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c94004eb0 con 0x7f4cb406e7b0 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3618199151 0 0) 0x7f4cb40a06e0 con 0x7f4cb40a5920 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 monclient(hunting): my global_id is 25811 2024-08-19T12:42:55.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cb409f6c0 con 0x7f4cb40a5920 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2212683012 0 0) 0x7f4cb40a09c0 con 0x7f4cb40a5d50 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 10 monclient(hunting): my global_id is 144225 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.502+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cb40a06e0 con 0x7f4cb40a5d50 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 464408355 0 0) 0x7f4c94004eb0 con 0x7f4cb406e7b0 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cb40a09c0 con 0x7f4cb406e7b0 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4cac003990 con 0x7f4cb406e7b0 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient(hunting): dump: 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.520 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.521 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1651432190 0 0) 0x7f4cb409f6c0 con 0x7f4cb40a5920 2024-08-19T12:42:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c94004eb0 con 0x7f4cb40a5920 2024-08-19T12:42:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2786950812 0 0) 0x7f4cb40a06e0 con 0x7f4cb40a5d50 2024-08-19T12:42:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cb409f6c0 con 0x7f4cb40a5d50 2024-08-19T12:42:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c9c004b20 con 0x7f4cb40a5920 2024-08-19T12:42:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient(hunting): dump: 2024-08-19T12:42:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.522 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.523 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4ca4001380 con 0x7f4cb40a5d50 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient(hunting): dump: 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.524 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2764851386 0 0) 0x7f4c94004eb0 con 0x7f4cb40a5920 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.26:6791/0 conn(0x7f4cb406e7b0 legacy=0x7f4cb4131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.26:6789/0 conn(0x7f4cb40a5d50 legacy=0x7f4cb4132c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient: _finish_hunting 0 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 monclient: found mon.e 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:55.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb409f020 con 0x7f4cb40a5920 2024-08-19T12:42:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient: _finish_auth 0 2024-08-19T12:42:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient: _check_auth_tickets 2024-08-19T12:42:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4cb8d5a640 5 monclient: authenticate success, global_id 25811 2024-08-19T12:42:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4cb8d5a640 10 monclient: _renew_subs 2024-08-19T12:42:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4cb8d5a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb41d72a0 con 0x7f4cb40a5920 2024-08-19T12:42:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4cb8d5a640 10 client.25811.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4cb8d5a640 10 monclient: _renew_subs 2024-08-19T12:42:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4cb8d5a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f4cb41d7650 con 0x7f4cb40a5920 2024-08-19T12:42:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4c9c004cc0 con 0x7f4cb40a5920 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4cb8d5a640 1 librados: init done 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c9c0019b0 con 0x7f4cb40a5920 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4caaffd640 10 monclient: dump: 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.527 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.503+0000 7f4cb0d99640 4 set_mon_vals no callback set 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.504+0000 7f4cb8d5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.504+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4cb40008d0 con 0x7f4cb412db30 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.504+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f4c9c011850 con 0x7f4cb40a5920 2024-08-19T12:42:55.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.504+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(185..185 src has 1..185) ==== 5746+0+0 (unknown 2474800286 0 0) 0x7f4c9c04e0d0 con 0x7f4cb40a5920 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.504+0000 7f4caaffd640 10 client.25811.objecter ms_dispatch 0x7f4cb4000b90 osd_map(185..185 src has 1..185) 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.504+0000 7f4caaffd640 3 client.25811.objecter handle_osd_map got epochs [185,185] > 0 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.504+0000 7f4caaffd640 3 client.25811.objecter handle_osd_map decoding full epoch 185 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.504+0000 7f4caaffd640 20 client.25811.objecter dump_active .. 0 homeless 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.504+0000 7f4cb2d9d640 10 monclient: get_auth_request con 0x7f4c94041960 auth_method 0 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 10 client.25811.objecter ms_handle_connect 0x7f4cb412db30 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 10 client.25811.objecter resend_mon_ops 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4cb259c640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.26:6791/0 conn(0x7f4cb412db30 legacy=0x7f4cb412df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2938280995 0 0) 0x7f4cb40008d0 con 0x7f4cb412db30 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 10 monclient(hunting): my global_id is 26284 2024-08-19T12:42:55.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c94004eb0 con 0x7f4cb412db30 2024-08-19T12:42:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 196539097 0 0) 0x7f4c94004eb0 con 0x7f4cb412db30 2024-08-19T12:42:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cb40008d0 con 0x7f4cb412db30 2024-08-19T12:42:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4cac0044a0 con 0x7f4cb412db30 2024-08-19T12:42:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.506+0000 7f4caaffd640 10 monclient: dump: 2024-08-19T12:42:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.530 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.507+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 42104787 0 0) 0x7f4cb40008d0 con 0x7f4cb412db30 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.507+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4cb4134e10 con 0x7f4cb412db30 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.507+0000 7f4caaffd640 10 client.25811.objecter ms_handle_connect 0x7f4c94041960 2024-08-19T12:42:55.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.507+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4cac004500 con 0x7f4cb412db30 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.507+0000 7f4caaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.507+0000 7f4caaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.507+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.26:6791/0 conn(0x7f4cb412db30 legacy=0x7f4cb412df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.507+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.26:6791/0 conn(0x7f4cb412db30 legacy=0x7f4cb412df10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.513+0000 7f4cb8d5a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.513+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4cb412ea00 con 0x7f4cb412e620 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.515+0000 7f4cb259c640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.26:6791/0 conn(0x7f4cb412e620 legacy=0x7f4cb41db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.515+0000 7f4caaffd640 10 client.25811.objecter ms_handle_connect 0x7f4cb412e620 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.515+0000 7f4caaffd640 10 client.25811.objecter resend_mon_ops 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1002348009 0 0) 0x7f4cb412ea00 con 0x7f4cb412e620 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 10 monclient(hunting): my global_id is 26293 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cb40008d0 con 0x7f4cb412e620 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2429523298 0 0) 0x7f4cb40008d0 con 0x7f4cb412e620 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cb412ea00 con 0x7f4cb412e620 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4cac0060f0 con 0x7f4cb412e620 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 10 monclient: dump: 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2676870158 0 0) 0x7f4cb412ea00 con 0x7f4cb412e620 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4cb41430e0 con 0x7f4cb412e620 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3286905999) 0x7f4cac008410 con 0x7f4cb412e620 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:55.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:55.535 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:55.535 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:55.535 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:55.535 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:55.535 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:55.535 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 288546, 2024-08-19T12:42:55.535 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:55.535 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:55.536 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.537 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:55.538 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:55.538 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.538 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.538 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.538 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.538 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.538 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.538 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.538 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.538 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.538 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.538 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:55.539 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.540 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.541 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.541 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.541 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.541 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:55.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.541 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:55.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.541 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:55.541 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:55.541 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.541 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:55.543 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.544 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.545 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:55.548 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:55.549 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:55.550 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.550 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:55.550 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:55.550 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.550 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.551 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.552 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:55.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.555 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.555 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:42:55.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.555 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:55.555 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:55.555 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:55.555 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.555 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.555 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:55.555 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:55.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.26:6791/0 conn(0x7f4cb412e620 legacy=0x7f4cb41db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.516+0000 7f4caaffd640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.26:6791/0 conn(0x7f4cb412e620 legacy=0x7f4cb41db840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:55.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 10 librados: watch_flush enter 2024-08-19T12:42:55.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 10 librados: watch_flush exit 2024-08-19T12:42:55.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 20 client.25811.objecter shutdown clearing up homeless session... 2024-08-19T12:42:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 10 client.25811.objecter successfully canceled tick 2024-08-19T12:42:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.104:6800/899887506 conn(0x7f4c94041960 legacy=0x7f4c94043e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 10 monclient: shutdown 2024-08-19T12:42:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/795976649 >> v1:172.21.15.26:6793/0 conn(0x7f4cb40a5920 legacy=0x7f4cb4132540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb2d9d640 1 -- 172.21.15.26:0/795976649 reap_dead start 2024-08-19T12:42:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb2d9d640 1 -- 172.21.15.26:0/795976649 reap_dead start 2024-08-19T12:42:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/795976649 shutdown_connections 2024-08-19T12:42:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/795976649 >> 172.21.15.26:0/795976649 conn(0x7f4cb40a8740 msgr2=0x7f4cb40a9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/795976649 shutdown_connections 2024-08-19T12:42:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 1 -- 172.21.15.26:0/795976649 wait complete. 2024-08-19T12:42:55.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.517+0000 7f4cb8d5a640 1 librados: shutdown 2024-08-19T12:42:55.559 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:55.559 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:42:55.559 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:42:55.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.658+0000 7f5435353640 10 monclient: get_monmap_and_config 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.658+0000 7f5435353640 10 monclient: build_initial_monmap 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.658+0000 7f5435353640 10 monclient: monmap: 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:55.658863+0000 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:55.658863+0000 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:55.659 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:55.660 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:55.660 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:55.660 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:55.660 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:55.660 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.658+0000 7f5435353640 1 Processor -- start 2024-08-19T12:42:55.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.658+0000 7f5435353640 1 -- start start 2024-08-19T12:42:55.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.658+0000 7f5435353640 10 monclient: init 2024-08-19T12:42:55.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.659+0000 7f5435353640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:55.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.659+0000 7f5435353640 10 monclient: _add_conns ranks=[6,3,4,5,8,7,2,1,0] 2024-08-19T12:42:55.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.659+0000 7f5435353640 10 monclient(hunting): picked mon.noname-d con 0x7f543006e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:55.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.659+0000 7f5435353640 10 monclient(hunting): picked mon.noname-g con 0x7f54300a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:55.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.659+0000 7f5435353640 10 monclient(hunting): picked mon.noname-c con 0x7f54300a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:55.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.659+0000 7f5435353640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5430136790 con 0x7f54300a5920 2024-08-19T12:42:55.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.659+0000 7f5435353640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f543009ddf0 con 0x7f54300a5d50 2024-08-19T12:42:55.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.659+0000 7f5435353640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f543009e0d0 con 0x7f543006e7b0 2024-08-19T12:42:55.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.659+0000 7f5435353640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:55.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.659+0000 7f5435353640 10 monclient(hunting): authenticate will time out at 969.746331s 2024-08-19T12:42:55.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.659+0000 7f542effd640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f543006e7b0 0x7f5430136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34592/0 (socket says 172.21.15.26:34592) 2024-08-19T12:42:55.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.659+0000 7f542effd640 1 -- 172.21.15.26:0/4046011263 learned_addr learned my addr 172.21.15.26:0/4046011263 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542effd640 1 -- 172.21.15.26:0/4046011263 >> v1:172.21.15.26:6792/0 conn(0x7f543006e7b0 legacy=0x7f5430136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542f7fe640 1 -- 172.21.15.26:0/4046011263 >> v1:172.21.15.26:6791/0 conn(0x7f54300a5d50 legacy=0x7f5430141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 1 -- 172.21.15.26:0/4046011263 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2815242521 0 0) 0x7f543009e0d0 con 0x7f543006e7b0 2024-08-19T12:42:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient(hunting): my global_id is 26313 2024-08-19T12:42:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 1 -- 172.21.15.26:0/4046011263 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f541c003520 con 0x7f543006e7b0 2024-08-19T12:42:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 1 -- 172.21.15.26:0/4046011263 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2670672536 0 0) 0x7f5420001820 con 0x7f54300a5d50 2024-08-19T12:42:55.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient(hunting): my global_id is 26302 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 1 -- 172.21.15.26:0/4046011263 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f543009e0d0 con 0x7f54300a5d50 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542e7fc640 1 -- 172.21.15.26:0/4046011263 >> v1:172.21.15.104:6790/0 conn(0x7f54300a5920 legacy=0x7f543013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 1 -- 172.21.15.26:0/4046011263 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3352570370 0 0) 0x7f541c003520 con 0x7f543006e7b0 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 1 -- 172.21.15.26:0/4046011263 >> v1:172.21.15.26:6791/0 conn(0x7f54300a5d50 legacy=0x7f5430141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 1 -- 172.21.15.26:0/4046011263 >> v1:172.21.15.104:6790/0 conn(0x7f54300a5920 legacy=0x7f543013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 1 monclient: found mon.noname-d 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:42:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 1 -- 172.21.15.26:0/4046011263 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543009e3b0 con 0x7f543006e7b0 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient: _finish_auth 0 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 1 -- 172.21.15.26:0/4046011263 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5428001ae0 con 0x7f543006e7b0 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f5435353640 5 monclient: authenticate success, global_id 26313 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f5435353640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.661+0000 7f542dffb640 10 monclient: dump: 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f5435353640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f542dffb640 1 -- 172.21.15.26:0/4046011263 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f54280048a0 con 0x7f543006e7b0 2024-08-19T12:42:55.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f542dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f542dffb640 1 -- 172.21.15.26:0/4046011263 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5428004e10 con 0x7f543006e7b0 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f5435353640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f5435353640 4 set_mon_vals no callback set 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f542dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f542dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f542dffb640 10 monclient: dump: 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.670 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.671 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.671 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.671 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.671 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.671 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.671 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.671 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.671 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f5435353640 10 monclient: shutdown 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f5435353640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f5435353640 1 -- 172.21.15.26:0/4046011263 >> v1:172.21.15.26:6792/0 conn(0x7f543006e7b0 legacy=0x7f5430136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f5435353640 1 -- 172.21.15.26:0/4046011263 shutdown_connections 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f5435353640 1 -- 172.21.15.26:0/4046011263 >> 172.21.15.26:0/4046011263 conn(0x7f54300a8740 msgr2=0x7f54300a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f5435353640 1 -- 172.21.15.26:0/4046011263 shutdown_connections 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.662+0000 7f5435353640 1 -- 172.21.15.26:0/4046011263 wait complete. 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.663+0000 7f5435353640 10 monclient: build_initial_monmap 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.663+0000 7f5435353640 10 monclient: monmap: 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:55.672 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.663+0000 7f5435353640 1 librados: starting msgr at 2024-08-19T12:42:55.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.663+0000 7f5435353640 1 librados: starting objecter 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.663+0000 7f5435353640 1 Processor -- start 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.663+0000 7f5435353640 1 -- start start 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.663+0000 7f5435353640 1 librados: setting wanted keys 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.663+0000 7f5435353640 1 librados: calling monclient init 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.663+0000 7f5435353640 10 monclient: init 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.664+0000 7f5435353640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.664+0000 7f5435353640 10 monclient: _add_conns ranks=[2,3,8,7,1,5,6,4,0] 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.664+0000 7f5435353640 10 monclient(hunting): picked mon.noname-b con 0x7f543006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.664+0000 7f5435353640 10 monclient(hunting): picked mon.noname-g con 0x7f54300a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.664+0000 7f5435353640 10 monclient(hunting): picked mon.noname-e con 0x7f54300a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.664+0000 7f5435353640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54301222b0 con 0x7f543006e7b0 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.664+0000 7f5435353640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5430123750 con 0x7f54300a5920 2024-08-19T12:42:55.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.664+0000 7f5435353640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5430142ec0 con 0x7f54300a5d50 2024-08-19T12:42:55.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.664+0000 7f5435353640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:55.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.664+0000 7f5435353640 10 monclient(hunting): authenticate will time out at 969.751235s 2024-08-19T12:42:55.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.664+0000 7f542effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f543006e7b0 0x7f5430130d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51754/0 (socket says 172.21.15.26:51754) 2024-08-19T12:42:55.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.664+0000 7f542effd640 1 -- 172.21.15.26:0/2016499491 learned_addr learned my addr 172.21.15.26:0/2016499491 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:55.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f542effd640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.26:6790/0 conn(0x7f543006e7b0 legacy=0x7f5430130d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f542f7fe640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.26:6793/0 conn(0x7f54300a5d50 legacy=0x7f5430131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 client.?.objecter ms_handle_connect 0x7f543006e7b0 2024-08-19T12:42:55.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 client.?.objecter ms_handle_connect 0x7f54300a5d50 2024-08-19T12:42:55.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2473704985 0 0) 0x7f54301222b0 con 0x7f543006e7b0 2024-08-19T12:42:55.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): my global_id is 26048 2024-08-19T12:42:55.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5410004eb0 con 0x7f543006e7b0 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962755864 0 0) 0x7f5430142ec0 con 0x7f54300a5d50 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): my global_id is 25820 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54301222b0 con 0x7f54300a5d50 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 client.?.objecter ms_handle_connect 0x7f54300a5920 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f542e7fc640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.104:6790/0 conn(0x7f54300a5920 legacy=0x7f5430131480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2336737492 0 0) 0x7f5410004eb0 con 0x7f543006e7b0 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5430142ec0 con 0x7f543006e7b0 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5428004d90 con 0x7f543006e7b0 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:55.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): dump: 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3572884433 0 0) 0x7f54301222b0 con 0x7f54300a5d50 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5410004eb0 con 0x7f54300a5d50 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5420001800 con 0x7f54300a5d50 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): dump: 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.687 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4133701018 0 0) 0x7f5430123750 con 0x7f54300a5920 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 10 monclient(hunting): my global_id is 26202 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.666+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54301222b0 con 0x7f54300a5920 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4000347333 0 0) 0x7f5430142ec0 con 0x7f543006e7b0 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.26:6793/0 conn(0x7f54300a5d50 legacy=0x7f5430131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.104:6790/0 conn(0x7f54300a5920 legacy=0x7f5430131480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 10 monclient: _finish_hunting 0 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 1 monclient: found mon.b 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5430132670 con 0x7f543006e7b0 2024-08-19T12:42:55.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 10 monclient: _finish_auth 0 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 10 monclient: _check_auth_tickets 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f5435353640 5 monclient: authenticate success, global_id 26048 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f5435353640 10 monclient: _renew_subs 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f5435353640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f5435353640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5430123470 con 0x7f543006e7b0 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f5435353640 10 client.26048.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f5435353640 10 monclient: _renew_subs 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f5435353640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f5435353640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f54301d7480 con 0x7f543006e7b0 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f5435353640 1 librados: init done 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5428001d30 con 0x7f543006e7b0 2024-08-19T12:42:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5428001b00 con 0x7f543006e7b0 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f542d7fa640 4 set_mon_vals no callback set 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.667+0000 7f54277fe640 10 monclient: dump: 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.691 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.668+0000 7f5435353640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.668+0000 7f5435353640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f54140065c0 con 0x7f5414002860 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.668+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5428011a00 con 0x7f543006e7b0 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.668+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(185..185 src has 1..185) ==== 5746+0+0 (unknown 2474800286 0 0) 0x7f542804e300 con 0x7f543006e7b0 2024-08-19T12:42:55.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.668+0000 7f54277fe640 10 client.26048.objecter ms_dispatch 0x7f5430000b90 osd_map(185..185 src has 1..185) 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.668+0000 7f54277fe640 3 client.26048.objecter handle_osd_map got epochs [185,185] > 0 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.668+0000 7f54277fe640 3 client.26048.objecter handle_osd_map decoding full epoch 185 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.668+0000 7f54277fe640 20 client.26048.objecter dump_active .. 0 homeless 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.668+0000 7f542f7fe640 10 monclient: get_auth_request con 0x7f54100416c0 auth_method 0 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f542e7fc640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.26:6792/0 conn(0x7f5414002860 legacy=0x7f5414004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 10 client.26048.objecter ms_handle_connect 0x7f5414002860 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 10 client.26048.objecter resend_mon_ops 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2707835944 0 0) 0x7f54140065c0 con 0x7f5414002860 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 10 monclient(hunting): my global_id is 26322 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5430142ec0 con 0x7f5414002860 2024-08-19T12:42:55.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3885996239 0 0) 0x7f5430142ec0 con 0x7f5414002860 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54140065c0 con 0x7f5414002860 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5418001c30 con 0x7f5414002860 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 10 monclient: dump: 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3915188305 0 0) 0x7f54140065c0 con 0x7f5414002860 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.670+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f541400a160 con 0x7f5414002860 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.671+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f54180057c0 con 0x7f5414002860 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.671+0000 7f54277fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.671+0000 7f54277fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:55.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.671+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.26:6792/0 conn(0x7f5414002860 legacy=0x7f5414004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.671+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.26:6792/0 conn(0x7f5414002860 legacy=0x7f5414004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.671+0000 7f54277fe640 10 client.26048.objecter ms_handle_connect 0x7f54100416c0 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.677+0000 7f5435353640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.677+0000 7f5435353640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5414009e50 con 0x7f5414009110 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.679+0000 7f542e7fc640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.26:6792/0 conn(0x7f5414009110 legacy=0x7f54140095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.679+0000 7f54277fe640 10 client.26048.objecter ms_handle_connect 0x7f5414009110 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.679+0000 7f54277fe640 10 client.26048.objecter resend_mon_ops 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.679+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2845745751 0 0) 0x7f5414009e50 con 0x7f5414009110 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.679+0000 7f54277fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.679+0000 7f54277fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.679+0000 7f54277fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.679+0000 7f54277fe640 10 monclient(hunting): my global_id is 26331 2024-08-19T12:42:55.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.679+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54140065c0 con 0x7f5414009110 2024-08-19T12:42:55.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.679+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2625700808 0 0) 0x7f54140065c0 con 0x7f5414009110 2024-08-19T12:42:55.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5414009e50 con 0x7f5414009110 2024-08-19T12:42:55.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5418003d00 con 0x7f5414009110 2024-08-19T12:42:55.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f54277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f54277fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:55.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f54277fe640 10 monclient: dump: 2024-08-19T12:42:55.697 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.697 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.697 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.697 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.697 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.697 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1101385606 0 0) 0x7f5414009e50 con 0x7f5414009110 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f541400c3f0 con 0x7f5414009110 2024-08-19T12:42:55.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 3440103018) 0x7f5418005970 con 0x7f5414009110 2024-08-19T12:42:55.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f54277fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:55.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f54277fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:55.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.26:6792/0 conn(0x7f5414009110 legacy=0x7f54140095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f54277fe640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.26:6792/0 conn(0x7f5414009110 legacy=0x7f54140095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:55.699 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:55.699 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:55.699 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:55.699 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 288721, 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.700 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.701 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.702 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:55.703 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.704 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:55.707 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:55.708 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.709 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:55.711 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.713 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:55.713 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:55.713 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.713 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.713 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.714 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:55.714 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:55.714 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.714 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:55.714 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.714 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.714 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:55.715 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:55.718 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:55.719 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:55.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.719 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:55.719 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:55.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 10 librados: watch_flush enter 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 10 librados: watch_flush exit 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 20 client.26048.objecter shutdown clearing up homeless session... 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 10 client.26048.objecter successfully canceled tick 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.104:6800/899887506 conn(0x7f54100416c0 legacy=0x7f5410043b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 10 monclient: shutdown 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 1 -- 172.21.15.26:0/2016499491 >> v1:172.21.15.26:6790/0 conn(0x7f543006e7b0 legacy=0x7f5430130d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f542f7fe640 1 -- 172.21.15.26:0/2016499491 reap_dead start 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f542f7fe640 1 -- 172.21.15.26:0/2016499491 reap_dead start 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 1 -- 172.21.15.26:0/2016499491 shutdown_connections 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 1 -- 172.21.15.26:0/2016499491 >> 172.21.15.26:0/2016499491 conn(0x7f54300a8740 msgr2=0x7f543006d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 1 -- 172.21.15.26:0/2016499491 shutdown_connections 2024-08-19T12:42:55.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 1 -- 172.21.15.26:0/2016499491 wait complete. 2024-08-19T12:42:55.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.680+0000 7f5435353640 1 librados: shutdown 2024-08-19T12:42:55.722 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:55.722 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:42:55.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:42:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.821+0000 7fb7470fd640 10 monclient: get_monmap_and_config 2024-08-19T12:42:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.821+0000 7fb7470fd640 10 monclient: build_initial_monmap 2024-08-19T12:42:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.821+0000 7fb7470fd640 10 monclient: monmap: 2024-08-19T12:42:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:55.821947+0000 2024-08-19T12:42:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:55.821947+0000 2024-08-19T12:42:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:55.823 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:55.823 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:55.823 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:55.823 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:55.823 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:55.823 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:55.823 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:55.823 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.821+0000 7fb7470fd640 1 Processor -- start 2024-08-19T12:42:55.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.821+0000 7fb7470fd640 1 -- start start 2024-08-19T12:42:55.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.821+0000 7fb7470fd640 10 monclient: init 2024-08-19T12:42:55.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.822+0000 7fb7470fd640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:55.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.822+0000 7fb7470fd640 10 monclient: _add_conns ranks=[3,0,1,6,2,5,4,7,8] 2024-08-19T12:42:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.822+0000 7fb7470fd640 10 monclient(hunting): picked mon.noname-g con 0x7fb740073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.822+0000 7fb7470fd640 10 monclient(hunting): picked mon.noname-a con 0x7fb7400a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.822+0000 7fb7470fd640 10 monclient(hunting): picked mon.noname-f con 0x7fb74013f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.822+0000 7fb7470fd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb74009dae0 con 0x7fb7400a57a0 2024-08-19T12:42:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.822+0000 7fb7470fd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb740136910 con 0x7fb74013f2e0 2024-08-19T12:42:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.822+0000 7fb7470fd640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb74009dd50 con 0x7fb740073470 2024-08-19T12:42:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.822+0000 7fb7470fd640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.822+0000 7fb7470fd640 10 monclient(hunting): authenticate will time out at 969.909423s 2024-08-19T12:42:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.822+0000 7fb73ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb7400a57a0 0x7fb7400a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57954/0 (socket says 172.21.15.26:57954) 2024-08-19T12:42:55.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.822+0000 7fb73ffff640 1 -- 172.21.15.26:0/3792851046 learned_addr learned my addr 172.21.15.26:0/3792851046 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:55.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.824+0000 7fb73ffff640 1 -- 172.21.15.26:0/3792851046 >> v1:172.21.15.26:6789/0 conn(0x7fb7400a57a0 legacy=0x7fb7400a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.824+0000 7fb744e99640 1 -- 172.21.15.26:0/3792851046 >> v1:172.21.15.104:6790/0 conn(0x7fb740073470 legacy=0x7fb740136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.824+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2298819933 0 0) 0x7fb74009dae0 con 0x7fb7400a57a0 2024-08-19T12:42:55.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.824+0000 7fb73f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.824+0000 7fb73f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.824+0000 7fb73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.824+0000 7fb73f7fe640 10 monclient(hunting): my global_id is 144234 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.824+0000 7fb74569a640 1 -- 172.21.15.26:0/3792851046 >> v1:172.21.15.104:6789/0 conn(0x7fb74013f2e0 legacy=0x7fb7401416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.824+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb728003520 con 0x7fb7400a57a0 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2247443419 0 0) 0x7fb74009dd50 con 0x7fb740073470 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient(hunting): my global_id is 26211 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb74009dae0 con 0x7fb740073470 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2973087435 0 0) 0x7fb740136910 con 0x7fb74013f2e0 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient(hunting): my global_id is 26713 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb74009dd50 con 0x7fb74013f2e0 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2709164945 0 0) 0x7fb728003520 con 0x7fb7400a57a0 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 >> v1:172.21.15.104:6790/0 conn(0x7fb740073470 legacy=0x7fb740136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 >> v1:172.21.15.104:6789/0 conn(0x7fb74013f2e0 legacy=0x7fb7401416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 1 monclient: found mon.noname-a 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74009e030 con 0x7fb7400a57a0 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:42:55.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb734001ae0 con 0x7fb7400a57a0 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb7470fd640 5 monclient: authenticate success, global_id 144234 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb7470fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient: dump: 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.832 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb7470fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb7340048a0 con 0x7fb7400a57a0 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb7470fd640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 1 -- 172.21.15.26:0/3792851046 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb734004e10 con 0x7fb7400a57a0 2024-08-19T12:42:55.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb7470fd640 4 set_mon_vals no callback set 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb73f7fe640 10 monclient: dump: 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.834 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb7470fd640 10 monclient: shutdown 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb7470fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb7470fd640 1 -- 172.21.15.26:0/3792851046 >> v1:172.21.15.26:6789/0 conn(0x7fb7400a57a0 legacy=0x7fb7400a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb7470fd640 1 -- 172.21.15.26:0/3792851046 shutdown_connections 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb7470fd640 1 -- 172.21.15.26:0/3792851046 >> 172.21.15.26:0/3792851046 conn(0x7fb7400abff0 msgr2=0x7fb7400ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb7470fd640 1 -- 172.21.15.26:0/3792851046 shutdown_connections 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.825+0000 7fb7470fd640 1 -- 172.21.15.26:0/3792851046 wait complete. 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.826+0000 7fb7470fd640 10 monclient: build_initial_monmap 2024-08-19T12:42:55.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.826+0000 7fb7470fd640 10 monclient: monmap: 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:55.836 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.826+0000 7fb7470fd640 1 librados: starting msgr at 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.826+0000 7fb7470fd640 1 librados: starting objecter 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.826+0000 7fb7470fd640 1 Processor -- start 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.826+0000 7fb7470fd640 1 -- start start 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.826+0000 7fb7470fd640 1 librados: setting wanted keys 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.826+0000 7fb7470fd640 1 librados: calling monclient init 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.826+0000 7fb7470fd640 10 monclient: init 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.827+0000 7fb7470fd640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.827+0000 7fb7470fd640 10 monclient: _add_conns ranks=[1,0,5,7,6,8,3,2,4] 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.827+0000 7fb7470fd640 10 monclient(hunting): picked mon.noname-f con 0x7fb740073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:55.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.827+0000 7fb7470fd640 10 monclient(hunting): picked mon.noname-a con 0x7fb7400a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.827+0000 7fb7470fd640 10 monclient(hunting): picked mon.noname-h con 0x7fb74013f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:42:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.827+0000 7fb7470fd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb740136f30 con 0x7fb7400a57a0 2024-08-19T12:42:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.827+0000 7fb7470fd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb740137210 con 0x7fb740073470 2024-08-19T12:42:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.827+0000 7fb7470fd640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7401374f0 con 0x7fb74013f2e0 2024-08-19T12:42:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.827+0000 7fb7470fd640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.827+0000 7fb7470fd640 10 monclient(hunting): authenticate will time out at 969.914391s 2024-08-19T12:42:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.827+0000 7fb73ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb7400a57a0 0x7fb7401d3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57964/0 (socket says 172.21.15.26:57964) 2024-08-19T12:42:55.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.827+0000 7fb73ffff640 1 -- 172.21.15.26:0/727868653 learned_addr learned my addr 172.21.15.26:0/727868653 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73ffff640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.26:6789/0 conn(0x7fb7400a57a0 legacy=0x7fb7401d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 client.?.objecter ms_handle_connect 0x7fb7400a57a0 2024-08-19T12:42:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 client.?.objecter ms_handle_connect 0x7fb740073470 2024-08-19T12:42:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb744e99640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.104:6789/0 conn(0x7fb740073470 legacy=0x7fb7401ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3006612192 0 0) 0x7fb740136f30 con 0x7fb7400a57a0 2024-08-19T12:42:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 monclient(hunting): my global_id is 144243 2024-08-19T12:42:55.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb72c004eb0 con 0x7fb7400a57a0 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 client.?.objecter ms_handle_connect 0x7fb74013f2e0 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb74569a640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.104:6791/0 conn(0x7fb74013f2e0 legacy=0x7fb7401d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2282371684 0 0) 0x7fb72c004eb0 con 0x7fb7400a57a0 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb740136f30 con 0x7fb7400a57a0 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb734004d90 con 0x7fb7400a57a0 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 monclient(hunting): dump: 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.844 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:55.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.829+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3235474827 0 0) 0x7fb740137210 con 0x7fb740073470 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient(hunting): my global_id is 26722 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb72c004eb0 con 0x7fb740073470 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 225949151 0 0) 0x7fb7401374f0 con 0x7fb74013f2e0 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient(hunting): my global_id is 26006 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb740137210 con 0x7fb74013f2e0 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3079339355 0 0) 0x7fb740136f30 con 0x7fb7400a57a0 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.104:6791/0 conn(0x7fb74013f2e0 legacy=0x7fb7401d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.104:6789/0 conn(0x7fb740073470 legacy=0x7fb7401ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient: _finish_hunting 0 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 1 monclient: found mon.a 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb740136c50 con 0x7fb7400a57a0 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient: _finish_auth 0 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient: _check_auth_tickets 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb7470fd640 5 monclient: authenticate success, global_id 144243 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb7470fd640 10 monclient: _renew_subs 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb7470fd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb7470fd640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7400a32b0 con 0x7fb7400a57a0 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb7470fd640 10 client.144243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:55.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb7470fd640 10 monclient: _renew_subs 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb7470fd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb7470fd640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb7401ceab0 con 0x7fb7400a57a0 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb7470fd640 1 librados: init done 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb734001d30 con 0x7fb7400a57a0 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb734001a80 con 0x7fb7400a57a0 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73effd640 4 set_mon_vals no callback set 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.830+0000 7fb73d7fa640 10 monclient: dump: 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.848 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.831+0000 7fb7470fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.831+0000 7fb7470fd640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb7200065c0 con 0x7fb720002860 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.831+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fb734011a00 con 0x7fb7400a57a0 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.831+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(185..185 src has 1..185) ==== 5746+0+0 (unknown 2474800286 0 0) 0x7fb73404e3b0 con 0x7fb7400a57a0 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.831+0000 7fb73d7fa640 10 client.144243.objecter ms_dispatch 0x7fb740000b90 osd_map(185..185 src has 1..185) 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.831+0000 7fb73d7fa640 3 client.144243.objecter handle_osd_map got epochs [185,185] > 0 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.831+0000 7fb73d7fa640 3 client.144243.objecter handle_osd_map decoding full epoch 185 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.831+0000 7fb73d7fa640 20 client.144243.objecter dump_active .. 0 homeless 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.831+0000 7fb74569a640 10 monclient: get_auth_request con 0x7fb72c041170 auth_method 0 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb744e99640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.26:6793/0 conn(0x7fb720002860 legacy=0x7fb720004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 10 client.144243.objecter ms_handle_connect 0x7fb720002860 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 10 client.144243.objecter resend_mon_ops 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1596834740 0 0) 0x7fb7200065c0 con 0x7fb720002860 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 10 monclient(hunting): my global_id is 25829 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb740136f30 con 0x7fb720002860 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3000872457 0 0) 0x7fb740136f30 con 0x7fb720002860 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7200065c0 con 0x7fb720002860 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb738002060 con 0x7fb720002860 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 10 monclient: dump: 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.851 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 10 client.144243.objecter ms_handle_connect 0x7fb72c041170 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2502304387 0 0) 0x7fb7200065c0 con 0x7fb720002860 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.833+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb72000a160 con 0x7fb720002860 2024-08-19T12:42:55.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.834+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb738005b40 con 0x7fb720002860 2024-08-19T12:42:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.834+0000 7fb73d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.834+0000 7fb73d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.834+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.26:6793/0 conn(0x7fb720002860 legacy=0x7fb720004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.834+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.26:6793/0 conn(0x7fb720002860 legacy=0x7fb720004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.840+0000 7fb7470fd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:55.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.840+0000 7fb7470fd640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb720009e50 con 0x7fb720009110 2024-08-19T12:42:55.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.842+0000 7fb744e99640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.26:6793/0 conn(0x7fb720009110 legacy=0x7fb7200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.842+0000 7fb73d7fa640 10 client.144243.objecter ms_handle_connect 0x7fb720009110 2024-08-19T12:42:55.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.842+0000 7fb73d7fa640 10 client.144243.objecter resend_mon_ops 2024-08-19T12:42:55.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.842+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3653730674 0 0) 0x7fb720009e50 con 0x7fb720009110 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.842+0000 7fb73d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.842+0000 7fb73d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.842+0000 7fb73d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.842+0000 7fb73d7fa640 10 monclient(hunting): my global_id is 25838 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.842+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7200065c0 con 0x7fb720009110 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2898035571 0 0) 0x7fb7200065c0 con 0x7fb720009110 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb720009e50 con 0x7fb720009110 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb738004190 con 0x7fb720009110 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 10 monclient: dump: 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.855 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2321091524 0 0) 0x7fb720009e50 con 0x7fb720009110 2024-08-19T12:42:55.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb72000c3f0 con 0x7fb720009110 2024-08-19T12:42:55.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1286042694) 0x7fb738004520 con 0x7fb720009110 2024-08-19T12:42:55.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:55.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:55.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.26:6793/0 conn(0x7fb720009110 legacy=0x7fb7200095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb73d7fa640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.26:6793/0 conn(0x7fb720009110 legacy=0x7fb7200095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:55.857 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:55.857 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 288860, 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:55.858 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.859 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.860 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:55.861 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:55.862 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.865 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.869 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:55.870 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:55.872 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:55.875 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:55.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:55.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:55.878 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:55.878 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb7470fd640 10 librados: watch_flush enter 2024-08-19T12:42:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb7470fd640 10 librados: watch_flush exit 2024-08-19T12:42:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb7470fd640 20 client.144243.objecter shutdown clearing up homeless session... 2024-08-19T12:42:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb7470fd640 10 client.144243.objecter successfully canceled tick 2024-08-19T12:42:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb7470fd640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.104:6800/899887506 conn(0x7fb72c041170 legacy=0x7fb72c043630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb7470fd640 10 monclient: shutdown 2024-08-19T12:42:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb7470fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:55.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb7470fd640 1 -- 172.21.15.26:0/727868653 >> v1:172.21.15.26:6789/0 conn(0x7fb7400a57a0 legacy=0x7fb7401d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb74569a640 1 -- 172.21.15.26:0/727868653 reap_dead start 2024-08-19T12:42:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.843+0000 7fb74569a640 1 -- 172.21.15.26:0/727868653 reap_dead start 2024-08-19T12:42:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.844+0000 7fb7470fd640 1 -- 172.21.15.26:0/727868653 shutdown_connections 2024-08-19T12:42:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.844+0000 7fb7470fd640 1 -- 172.21.15.26:0/727868653 >> 172.21.15.26:0/727868653 conn(0x7fb7400abff0 msgr2=0x7fb740134aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.844+0000 7fb7470fd640 1 -- 172.21.15.26:0/727868653 shutdown_connections 2024-08-19T12:42:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.844+0000 7fb7470fd640 1 -- 172.21.15.26:0/727868653 wait complete. 2024-08-19T12:42:55.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.844+0000 7fb7470fd640 1 librados: shutdown 2024-08-19T12:42:55.881 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:55.881 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:42:55.882 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:42:55.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.983+0000 7fd90b17b640 10 monclient: get_monmap_and_config 2024-08-19T12:42:55.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.983+0000 7fd90b17b640 10 monclient: build_initial_monmap 2024-08-19T12:42:55.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.983+0000 7fd90b17b640 10 monclient: monmap: 2024-08-19T12:42:55.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:55.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:55.984278+0000 2024-08-19T12:42:55.984 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:55.984278+0000 2024-08-19T12:42:55.984 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:55.985 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:55.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 1 Processor -- start 2024-08-19T12:42:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 1 -- start start 2024-08-19T12:42:55.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 10 monclient: init 2024-08-19T12:42:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 10 monclient: _add_conns ranks=[0,3,5,6,2,4,8,7,1] 2024-08-19T12:42:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 10 monclient(hunting): picked mon.noname-a con 0x7fd90406e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 10 monclient(hunting): picked mon.noname-g con 0x7fd9040a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 10 monclient(hunting): picked mon.noname-h con 0x7fd9040a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:42:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd904136790 con 0x7fd90406e7b0 2024-08-19T12:42:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd90409ddf0 con 0x7fd9040a5920 2024-08-19T12:42:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd90409e0d0 con 0x7fd9040a5d50 2024-08-19T12:42:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd90b17b640 10 monclient(hunting): authenticate will time out at 970.071689s 2024-08-19T12:42:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd908f17640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd90406e7b0 0x7fd904136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57966/0 (socket says 172.21.15.26:57966) 2024-08-19T12:42:55.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.984+0000 7fd908f17640 1 -- 172.21.15.26:0/2102135071 learned_addr learned my addr 172.21.15.26:0/2102135071 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.986+0000 7fd908f17640 1 -- 172.21.15.26:0/2102135071 >> v1:172.21.15.26:6789/0 conn(0x7fd90406e7b0 legacy=0x7fd904136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 1 -- 172.21.15.26:0/2102135071 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4179768309 0 0) 0x7fd904136790 con 0x7fd90406e7b0 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient(hunting): my global_id is 144252 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 1 -- 172.21.15.26:0/2102135071 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8f4003520 con 0x7fd90406e7b0 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd903fff640 1 -- 172.21.15.26:0/2102135071 >> v1:172.21.15.104:6790/0 conn(0x7fd9040a5920 legacy=0x7fd90413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd909718640 1 -- 172.21.15.26:0/2102135071 >> v1:172.21.15.104:6791/0 conn(0x7fd9040a5d50 legacy=0x7fd904141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 1 -- 172.21.15.26:0/2102135071 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 686717854 0 0) 0x7fd8f4003520 con 0x7fd90406e7b0 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 1 -- 172.21.15.26:0/2102135071 >> v1:172.21.15.104:6791/0 conn(0x7fd9040a5d50 legacy=0x7fd904141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 1 -- 172.21.15.26:0/2102135071 >> v1:172.21.15.104:6790/0 conn(0x7fd9040a5920 legacy=0x7fd90413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient: _finish_hunting 0 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 1 monclient: found mon.noname-a 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 1 -- 172.21.15.26:0/2102135071 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd90409e3b0 con 0x7fd90406e7b0 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient: _finish_auth 0 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient: _check_auth_tickets 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 1 -- 172.21.15.26:0/2102135071 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd8f8001ae0 con 0x7fd90406e7b0 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd90b17b640 5 monclient: authenticate success, global_id 144252 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd90b17b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient: dump: 2024-08-19T12:42:55.992 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.993 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.994 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd90b17b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:55.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 1 -- 172.21.15.26:0/2102135071 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8f80048a0 con 0x7fd90406e7b0 2024-08-19T12:42:55.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:55.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 1 -- 172.21.15.26:0/2102135071 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd8f8004e10 con 0x7fd90406e7b0 2024-08-19T12:42:55.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:55.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:55.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd9037fe640 10 monclient: dump: 2024-08-19T12:42:55.994 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:55.994 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:55.994 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.994 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd90b17b640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.987+0000 7fd90b17b640 4 set_mon_vals no callback set 2024-08-19T12:42:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.988+0000 7fd90b17b640 10 monclient: shutdown 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.988+0000 7fd90b17b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.988+0000 7fd90b17b640 1 -- 172.21.15.26:0/2102135071 >> v1:172.21.15.26:6789/0 conn(0x7fd90406e7b0 legacy=0x7fd904136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.988+0000 7fd90b17b640 1 -- 172.21.15.26:0/2102135071 shutdown_connections 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.988+0000 7fd90b17b640 1 -- 172.21.15.26:0/2102135071 >> 172.21.15.26:0/2102135071 conn(0x7fd9040a8740 msgr2=0x7fd9040a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.988+0000 7fd90b17b640 1 -- 172.21.15.26:0/2102135071 shutdown_connections 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.988+0000 7fd90b17b640 1 -- 172.21.15.26:0/2102135071 wait complete. 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.988+0000 7fd90b17b640 10 monclient: build_initial_monmap 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.988+0000 7fd90b17b640 10 monclient: monmap: 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:55.996 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.988+0000 7fd90b17b640 1 librados: starting msgr at 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.988+0000 7fd90b17b640 1 librados: starting objecter 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 1 Processor -- start 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 1 -- start start 2024-08-19T12:42:55.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 1 librados: setting wanted keys 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 1 librados: calling monclient init 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 10 monclient: init 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 10 monclient: _add_conns ranks=[0,8,5,7,2,4,6,3,1] 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 10 monclient(hunting): picked mon.noname-a con 0x7fd90406e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 10 monclient(hunting): picked mon.noname-e con 0x7fd9040a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 10 monclient(hunting): picked mon.noname-h con 0x7fd9040a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd904142000 con 0x7fd90406e7b0 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd904138f10 con 0x7fd9040a5d50 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9041383e0 con 0x7fd9040a5920 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd908f17640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd90406e7b0 0x7fd9041375c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57980/0 (socket says 172.21.15.26:57980) 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:55.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd908f17640 1 -- 172.21.15.26:0/711239703 learned_addr learned my addr 172.21.15.26:0/711239703 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:55.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.989+0000 7fd90b17b640 10 monclient(hunting): authenticate will time out at 970.076769s 2024-08-19T12:42:56.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.991+0000 7fd908f17640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.26:6789/0 conn(0x7fd90406e7b0 legacy=0x7fd9041375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.991+0000 7fd903fff640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.26:6793/0 conn(0x7fd9040a5920 legacy=0x7fd904137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.991+0000 7fd9017fa640 10 client.?.objecter ms_handle_connect 0x7fd90406e7b0 2024-08-19T12:42:56.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.991+0000 7fd9017fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:56.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.991+0000 7fd9017fa640 10 client.?.objecter ms_handle_connect 0x7fd9040a5920 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.991+0000 7fd9017fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046284333 0 0) 0x7fd904142000 con 0x7fd90406e7b0 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): my global_id is 144261 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8e8004eb0 con 0x7fd90406e7b0 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 309977591 0 0) 0x7fd9041383e0 con 0x7fd9040a5920 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): my global_id is 25847 2024-08-19T12:42:56.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd909718640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.104:6791/0 conn(0x7fd9040a5d50 legacy=0x7fd9041d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd904142000 con 0x7fd9040a5920 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 client.?.objecter ms_handle_connect 0x7fd9040a5d50 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3694495350 0 0) 0x7fd904142000 con 0x7fd9040a5920 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9041383e0 con 0x7fd9040a5920 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd8f0003610 con 0x7fd9040a5920 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): dump: 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.005 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2354526974 0 0) 0x7fd8e8004eb0 con 0x7fd90406e7b0 2024-08-19T12:42:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd904142000 con 0x7fd90406e7b0 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd8f8004d90 con 0x7fd90406e7b0 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): dump: 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.007 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.008 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.008 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.008 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.008 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.008 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.008 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2510331576 0 0) 0x7fd904138f10 con 0x7fd9040a5d50 2024-08-19T12:42:56.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient(hunting): my global_id is 26024 2024-08-19T12:42:56.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8e8004eb0 con 0x7fd9040a5d50 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1366099890 0 0) 0x7fd9041383e0 con 0x7fd9040a5920 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.104:6791/0 conn(0x7fd9040a5d50 legacy=0x7fd9041d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.26:6789/0 conn(0x7fd90406e7b0 legacy=0x7fd9041375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient: _finish_hunting 0 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 monclient: found mon.e 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd90413a9d0 con 0x7fd9040a5920 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient: _finish_auth 0 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient: _check_auth_tickets 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd90b17b640 5 monclient: authenticate success, global_id 25847 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd90b17b640 10 monclient: _renew_subs 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd90b17b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:56.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd90b17b640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd904139d10 con 0x7fd9040a5920 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd90b17b640 10 client.25847.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd90b17b640 10 monclient: _renew_subs 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd90b17b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd90b17b640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fd90413b690 con 0x7fd9040a5920 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd90b17b640 1 librados: init done 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8f0003b90 con 0x7fd9040a5920 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd8f0004080 con 0x7fd9040a5920 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd902ffd640 4 set_mon_vals no callback set 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.992+0000 7fd9017fa640 10 monclient: dump: 2024-08-19T12:42:56.010 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.011 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.993+0000 7fd90b17b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.993+0000 7fd90b17b640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd8e40065c0 con 0x7fd8e4002860 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.993+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd8f0004220 con 0x7fd9040a5920 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.993+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(185..185 src has 1..185) ==== 5746+0+0 (unknown 2474800286 0 0) 0x7fd8f004e260 con 0x7fd9040a5920 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.993+0000 7fd9017fa640 10 client.25847.objecter ms_dispatch 0x7fd904000b90 osd_map(185..185 src has 1..185) 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.993+0000 7fd9017fa640 3 client.25847.objecter handle_osd_map got epochs [185,185] > 0 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.993+0000 7fd9017fa640 3 client.25847.objecter handle_osd_map decoding full epoch 185 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.993+0000 7fd9017fa640 20 client.25847.objecter dump_active .. 0 homeless 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.994+0000 7fd909718640 10 monclient: get_auth_request con 0x7fd8e8041620 auth_method 0 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.995+0000 7fd9017fa640 10 client.25847.objecter ms_handle_connect 0x7fd8e4002860 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.995+0000 7fd908f17640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.104:6789/0 conn(0x7fd8e4002860 legacy=0x7fd8e4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.995+0000 7fd9017fa640 10 client.25847.objecter resend_mon_ops 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1981726010 0 0) 0x7fd8e40065c0 con 0x7fd8e4002860 2024-08-19T12:42:56.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 10 monclient(hunting): my global_id is 26731 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9041383e0 con 0x7fd8e4002860 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 10 client.25847.objecter ms_handle_connect 0x7fd8e8041620 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4111456453 0 0) 0x7fd9041383e0 con 0x7fd8e4002860 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8e40065c0 con 0x7fd8e4002860 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd8f8001d30 con 0x7fd8e4002860 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 10 monclient: dump: 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.013 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1845345641 0 0) 0x7fd8e40065c0 con 0x7fd8e4002860 2024-08-19T12:42:56.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd8e400a160 con 0x7fd8e4002860 2024-08-19T12:42:56.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd8f8005490 con 0x7fd8e4002860 2024-08-19T12:42:56.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:56.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:56.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.104:6789/0 conn(0x7fd8e4002860 legacy=0x7fd8e4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:55.996+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.104:6789/0 conn(0x7fd8e4002860 legacy=0x7fd8e4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:56.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.003+0000 7fd90b17b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:56.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.003+0000 7fd90b17b640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd8e4009e70 con 0x7fd8e4009130 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.005+0000 7fd9017fa640 10 client.25847.objecter ms_handle_connect 0x7fd8e4009130 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.005+0000 7fd9017fa640 10 client.25847.objecter resend_mon_ops 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.005+0000 7fd908f17640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.104:6789/0 conn(0x7fd8e4009130 legacy=0x7fd8e40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3322555279 0 0) 0x7fd8e4009e70 con 0x7fd8e4009130 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 10 monclient(hunting): my global_id is 26740 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8e40065c0 con 0x7fd8e4009130 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3898226987 0 0) 0x7fd8e40065c0 con 0x7fd8e4009130 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8e4009e70 con 0x7fd8e4009130 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd8f8005bf0 con 0x7fd8e4009130 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 10 monclient: dump: 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.018 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.019 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.019 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.019 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 221962663 0 0) 0x7fd8e4009e70 con 0x7fd8e4009130 2024-08-19T12:42:56.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.006+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd8e400c410 con 0x7fd8e4009130 2024-08-19T12:42:56.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.007+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3613047223) 0x7fd8f8005ff0 con 0x7fd8e4009130 2024-08-19T12:42:56.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.007+0000 7fd9017fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:56.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.007+0000 7fd9017fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:56.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.007+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.104:6789/0 conn(0x7fd8e4009130 legacy=0x7fd8e40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.007+0000 7fd9017fa640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.104:6789/0 conn(0x7fd8e4009130 legacy=0x7fd8e40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:56.019 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 289011, 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:56.020 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:56.021 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:56.022 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:56.023 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:56.024 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:56.025 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.027 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:56.028 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:56.031 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.032 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.033 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:56.034 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:56.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-08-19T12:42:56.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.039 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.039 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:56.039 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.007+0000 7fd90b17b640 10 librados: watch_flush enter 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd90b17b640 10 librados: watch_flush exit 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd90b17b640 20 client.25847.objecter shutdown clearing up homeless session... 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd90b17b640 10 client.25847.objecter successfully canceled tick 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd90b17b640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.104:6800/899887506 conn(0x7fd8e8041620 legacy=0x7fd8e8043ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd90b17b640 10 monclient: shutdown 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd90b17b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd90b17b640 1 -- 172.21.15.26:0/711239703 >> v1:172.21.15.26:6793/0 conn(0x7fd9040a5920 legacy=0x7fd904137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd909718640 1 -- 172.21.15.26:0/711239703 reap_dead start 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd909718640 1 -- 172.21.15.26:0/711239703 reap_dead start 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd90b17b640 1 -- 172.21.15.26:0/711239703 shutdown_connections 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd90b17b640 1 -- 172.21.15.26:0/711239703 >> 172.21.15.26:0/711239703 conn(0x7fd9040a8740 msgr2=0x7fd90406dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd90b17b640 1 -- 172.21.15.26:0/711239703 shutdown_connections 2024-08-19T12:42:56.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd90b17b640 1 -- 172.21.15.26:0/711239703 wait complete. 2024-08-19T12:42:56.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.008+0000 7fd90b17b640 1 librados: shutdown 2024-08-19T12:42:56.042 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:56.042 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:42:56.042 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:42:56.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.142+0000 7fa305f14640 10 monclient: get_monmap_and_config 2024-08-19T12:42:56.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.142+0000 7fa305f14640 10 monclient: build_initial_monmap 2024-08-19T12:42:56.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.142+0000 7fa305f14640 10 monclient: monmap: 2024-08-19T12:42:56.144 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:56.144 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.144 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:56.143701+0000 2024-08-19T12:42:56.144 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:56.143701+0000 2024-08-19T12:42:56.144 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:56.144 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.144 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:56.145 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:56.145 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:56.145 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:56.145 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:56.145 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:56.145 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:56.145 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:56.145 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:56.145 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.143+0000 7fa305f14640 1 Processor -- start 2024-08-19T12:42:56.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.143+0000 7fa305f14640 1 -- start start 2024-08-19T12:42:56.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.143+0000 7fa305f14640 10 monclient: init 2024-08-19T12:42:56.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa305f14640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:56.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa305f14640 10 monclient: _add_conns ranks=[1,4,0,5,6,7,2,3,8] 2024-08-19T12:42:56.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa305f14640 10 monclient(hunting): picked mon.noname-f con 0x7fa30006e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:56.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa305f14640 10 monclient(hunting): picked mon.noname-c con 0x7fa3000a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:56.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa305f14640 10 monclient(hunting): picked mon.noname-a con 0x7fa3000a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:56.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa305f14640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa300136790 con 0x7fa3000a5d50 2024-08-19T12:42:56.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa305f14640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa30009ddf0 con 0x7fa30006e7b0 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa305f14640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa30009e0d0 con 0x7fa3000a5920 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa305f14640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa305f14640 10 monclient(hunting): authenticate will time out at 970.231160s 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa2feffd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fa3000a5920 0x7fa30013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:54518/0 (socket says 172.21.15.26:54518) 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa2fffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa3000a5d50 0x7fa300141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57986/0 (socket says 172.21.15.26:57986) 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.144+0000 7fa2feffd640 1 -- 172.21.15.26:0/394660421 learned_addr learned my addr 172.21.15.26:0/394660421 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fffff640 1 -- 172.21.15.26:0/394660421 >> v1:172.21.15.26:6789/0 conn(0x7fa3000a5d50 legacy=0x7fa300141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2feffd640 1 -- 172.21.15.26:0/394660421 >> v1:172.21.15.26:6791/0 conn(0x7fa3000a5920 legacy=0x7fa30013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2ff7fe640 1 -- 172.21.15.26:0/394660421 >> v1:172.21.15.104:6789/0 conn(0x7fa30006e7b0 legacy=0x7fa300136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 663779526 0 0) 0x7fa300136790 con 0x7fa3000a5d50 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient(hunting): my global_id is 144270 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2e8003520 con 0x7fa3000a5d50 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 344084570 0 0) 0x7fa30009e0d0 con 0x7fa3000a5920 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient(hunting): my global_id is 26311 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa300136790 con 0x7fa3000a5920 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884979800 0 0) 0x7fa2f00014a0 con 0x7fa30006e7b0 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient(hunting): my global_id is 26749 2024-08-19T12:42:56.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa30009e0d0 con 0x7fa30006e7b0 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1882583612 0 0) 0x7fa300136790 con 0x7fa3000a5920 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 >> v1:172.21.15.104:6789/0 conn(0x7fa30006e7b0 legacy=0x7fa300136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 >> v1:172.21.15.26:6789/0 conn(0x7fa3000a5d50 legacy=0x7fa300141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 1 monclient: found mon.noname-c 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30009e3b0 con 0x7fa3000a5920 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient: _finish_auth 0 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa2f4001ac0 con 0x7fa3000a5920 2024-08-19T12:42:56.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.146+0000 7fa2fe7fc640 10 monclient: dump: 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.153 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa305f14640 5 monclient: authenticate success, global_id 26311 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa305f14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa2f40048a0 con 0x7fa3000a5920 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa2fe7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa2fe7fc640 1 -- 172.21.15.26:0/394660421 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa2f4004d90 con 0x7fa3000a5920 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa2fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa2fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa2fe7fc640 10 monclient: dump: 2024-08-19T12:42:56.154 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.155 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa305f14640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa305f14640 4 set_mon_vals no callback set 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa305f14640 10 monclient: shutdown 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa305f14640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa305f14640 1 -- 172.21.15.26:0/394660421 >> v1:172.21.15.26:6791/0 conn(0x7fa3000a5920 legacy=0x7fa30013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa305f14640 1 -- 172.21.15.26:0/394660421 shutdown_connections 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa305f14640 1 -- 172.21.15.26:0/394660421 >> 172.21.15.26:0/394660421 conn(0x7fa3000a8740 msgr2=0x7fa3000a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa305f14640 1 -- 172.21.15.26:0/394660421 shutdown_connections 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.147+0000 7fa305f14640 1 -- 172.21.15.26:0/394660421 wait complete. 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.148+0000 7fa305f14640 10 monclient: build_initial_monmap 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.148+0000 7fa305f14640 10 monclient: monmap: 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:56.156 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.148+0000 7fa305f14640 1 librados: starting msgr at 2024-08-19T12:42:56.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.148+0000 7fa305f14640 1 librados: starting objecter 2024-08-19T12:42:56.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.148+0000 7fa305f14640 1 Processor -- start 2024-08-19T12:42:56.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.148+0000 7fa305f14640 1 -- start start 2024-08-19T12:42:56.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.148+0000 7fa305f14640 1 librados: setting wanted keys 2024-08-19T12:42:56.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.148+0000 7fa305f14640 1 librados: calling monclient init 2024-08-19T12:42:56.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.148+0000 7fa305f14640 10 monclient: init 2024-08-19T12:42:56.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa305f14640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:56.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa305f14640 10 monclient: _add_conns ranks=[2,1,0,3,7,5,8,4,6] 2024-08-19T12:42:56.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa305f14640 10 monclient(hunting): picked mon.noname-b con 0x7fa30006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:42:56.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa305f14640 10 monclient(hunting): picked mon.noname-f con 0x7fa3000a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:56.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa305f14640 10 monclient(hunting): picked mon.noname-a con 0x7fa3000a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:56.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa305f14640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa300125350 con 0x7fa3000a5d50 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa305f14640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa300125630 con 0x7fa3000a5920 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa305f14640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa300125910 con 0x7fa30006e7b0 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa305f14640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa305f14640 10 monclient(hunting): authenticate will time out at 970.236190s 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa2fffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa3000a5d50 0x7fa3001d6bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:57988/0 (socket says 172.21.15.26:57988) 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa2fffff640 1 -- 172.21.15.26:0/2219854983 learned_addr learned my addr 172.21.15.26:0/2219854983 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.149+0000 7fa2ff7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa30006e7b0 0x7fa3001ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51756/0 (socket says 172.21.15.26:51756) 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2ff7fe640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.26:6790/0 conn(0x7fa30006e7b0 legacy=0x7fa3001ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 client.?.objecter ms_handle_connect 0x7fa30006e7b0 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 client.?.objecter ms_handle_connect 0x7fa3000a5d50 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:56.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 client.?.objecter ms_handle_connect 0x7fa3000a5920 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2fffff640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.26:6789/0 conn(0x7fa3000a5d50 legacy=0x7fa3001d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2feffd640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.104:6789/0 conn(0x7fa3000a5920 legacy=0x7fa3001d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3349938226 0 0) 0x7fa300125630 con 0x7fa3000a5920 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 monclient(hunting): my global_id is 26758 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2ec004eb0 con 0x7fa3000a5920 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 992098681 0 0) 0x7fa300125910 con 0x7fa30006e7b0 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 monclient(hunting): my global_id is 26057 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa300125630 con 0x7fa30006e7b0 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1756953140 0 0) 0x7fa300125350 con 0x7fa3000a5d50 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 10 monclient(hunting): my global_id is 144279 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.151+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa300125910 con 0x7fa3000a5d50 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4161475001 0 0) 0x7fa2ec004eb0 con 0x7fa3000a5920 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa300125350 con 0x7fa3000a5920 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa2f4004d90 con 0x7fa3000a5920 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:42:56.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient(hunting): dump: 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.167 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2343925371 0 0) 0x7fa300125910 con 0x7fa3000a5d50 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa2ec004eb0 con 0x7fa3000a5d50 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3176322501 0 0) 0x7fa300125630 con 0x7fa30006e7b0 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa300125910 con 0x7fa30006e7b0 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa2f80020b0 con 0x7fa3000a5d50 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient(hunting): dump: 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.168 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa2f0003950 con 0x7fa30006e7b0 2024-08-19T12:42:56.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient(hunting): dump: 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.170 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3189405410 0 0) 0x7fa300125350 con 0x7fa3000a5920 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.26:6790/0 conn(0x7fa30006e7b0 legacy=0x7fa3001ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.26:6789/0 conn(0x7fa3000a5d50 legacy=0x7fa3001d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient: _finish_hunting 0 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 monclient: found mon.f 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa300125bf0 con 0x7fa3000a5920 2024-08-19T12:42:56.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient: _finish_auth 0 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient: _check_auth_tickets 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa305f14640 5 monclient: authenticate success, global_id 26758 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa305f14640 10 monclient: _renew_subs 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa305f14640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa305f14640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa3001265c0 con 0x7fa3000a5920 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa305f14640 10 client.26758.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa305f14640 10 monclient: _renew_subs 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa305f14640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa305f14640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa3001238d0 con 0x7fa3000a5920 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa305f14640 1 librados: init done 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa2f40048a0 con 0x7fa3000a5920 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:56.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa2f4001d10 con 0x7fa3000a5920 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2fdffb640 4 set_mon_vals no callback set 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.152+0000 7fa2e7fff640 10 monclient: dump: 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.173 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.153+0000 7fa305f14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.153+0000 7fa305f14640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa2e00065c0 con 0x7fa2e0002860 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.153+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa2f4011980 con 0x7fa3000a5920 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.153+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(185..185 src has 1..185) ==== 5746+0+0 (unknown 2474800286 0 0) 0x7fa2f404e350 con 0x7fa3000a5920 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.153+0000 7fa2e7fff640 10 client.26758.objecter ms_dispatch 0x7fa300000b90 osd_map(185..185 src has 1..185) 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.153+0000 7fa2e7fff640 3 client.26758.objecter handle_osd_map got epochs [185,185] > 0 2024-08-19T12:42:56.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.153+0000 7fa2e7fff640 3 client.26758.objecter handle_osd_map decoding full epoch 185 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.153+0000 7fa2e7fff640 20 client.26758.objecter dump_active .. 0 homeless 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.153+0000 7fa2fffff640 10 monclient: get_auth_request con 0x7fa2ec041ad0 auth_method 0 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.155+0000 7fa2ff7fe640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.104:6790/0 conn(0x7fa2e0002860 legacy=0x7fa2e0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.155+0000 7fa2e7fff640 10 client.26758.objecter ms_handle_connect 0x7fa2e0002860 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.155+0000 7fa2e7fff640 10 client.26758.objecter resend_mon_ops 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.155+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 850820163 0 0) 0x7fa2e00065c0 con 0x7fa2e0002860 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.155+0000 7fa2e7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.155+0000 7fa2e7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.155+0000 7fa2e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.155+0000 7fa2e7fff640 10 monclient(hunting): my global_id is 26229 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.155+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa300125350 con 0x7fa2e0002860 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.155+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 703218851 0 0) 0x7fa300125350 con 0x7fa2e0002860 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa2e00065c0 con 0x7fa2e0002860 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa2f0003ee0 con 0x7fa2e0002860 2024-08-19T12:42:56.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 10 monclient: dump: 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.176 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 10 client.26758.objecter ms_handle_connect 0x7fa2ec041ad0 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3324309770 0 0) 0x7fa2e00065c0 con 0x7fa2e0002860 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2e000a160 con 0x7fa2e0002860 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa2f0003fd0 con 0x7fa2e0002860 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.104:6790/0 conn(0x7fa2e0002860 legacy=0x7fa2e0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.156+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.104:6790/0 conn(0x7fa2e0002860 legacy=0x7fa2e0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:56.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.162+0000 7fa305f14640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:56.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.162+0000 7fa305f14640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa2e0009e50 con 0x7fa2e0009110 2024-08-19T12:42:56.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.165+0000 7fa2ff7fe640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.104:6790/0 conn(0x7fa2e0009110 legacy=0x7fa2e00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.165+0000 7fa2e7fff640 10 client.26758.objecter ms_handle_connect 0x7fa2e0009110 2024-08-19T12:42:56.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.165+0000 7fa2e7fff640 10 client.26758.objecter resend_mon_ops 2024-08-19T12:42:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.181+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 493349543 0 0) 0x7fa2e0009e50 con 0x7fa2e0009110 2024-08-19T12:42:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.181+0000 7fa2e7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.181+0000 7fa2e7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.181+0000 7fa2e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.181+0000 7fa2e7fff640 10 monclient(hunting): my global_id is 26238 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.181+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2e00065c0 con 0x7fa2e0009110 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.181+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1273881026 0 0) 0x7fa2e00065c0 con 0x7fa2e0009110 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.181+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa2e0009e50 con 0x7fa2e0009110 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.181+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa2f0005b00 con 0x7fa2e0009110 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.181+0000 7fa2e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.181+0000 7fa2e7fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.181+0000 7fa2e7fff640 10 monclient: dump: 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.183 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.184 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.184 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.184 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.184 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.184 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.184 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.184 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.184 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.184 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 789344606 0 0) 0x7fa2e0009e50 con 0x7fa2e0009110 2024-08-19T12:42:56.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa2e000c3a0 con 0x7fa2e0009110 2024-08-19T12:42:56.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1725051386) 0x7fa2f0007e30 con 0x7fa2e0009110 2024-08-19T12:42:56.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa2e7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:56.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa2e7fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:56.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.104:6790/0 conn(0x7fa2e0009110 legacy=0x7fa2e00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa2e7fff640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.104:6790/0 conn(0x7fa2e0009110 legacy=0x7fa2e00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:56.185 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 289195, 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:56.186 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 7, 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:56.187 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:56.188 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:56.189 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:56.190 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:56.191 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.192 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.193 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:56.194 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.195 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:56.197 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.198 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.200 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.201 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:56.203 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:56.204 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:42:56.205 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.205 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.205 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:56.205 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:56.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa305f14640 10 librados: watch_flush enter 2024-08-19T12:42:56.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa305f14640 10 librados: watch_flush exit 2024-08-19T12:42:56.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa305f14640 20 client.26758.objecter shutdown clearing up homeless session... 2024-08-19T12:42:56.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa305f14640 10 client.26758.objecter successfully canceled tick 2024-08-19T12:42:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa305f14640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.104:6800/899887506 conn(0x7fa2ec041ad0 legacy=0x7fa2ec043f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa305f14640 10 monclient: shutdown 2024-08-19T12:42:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa305f14640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa305f14640 1 -- 172.21.15.26:0/2219854983 >> v1:172.21.15.104:6789/0 conn(0x7fa3000a5920 legacy=0x7fa3001d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa2fffff640 1 -- 172.21.15.26:0/2219854983 reap_dead start 2024-08-19T12:42:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa2fffff640 1 -- 172.21.15.26:0/2219854983 reap_dead start 2024-08-19T12:42:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa305f14640 1 -- 172.21.15.26:0/2219854983 shutdown_connections 2024-08-19T12:42:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.182+0000 7fa305f14640 1 -- 172.21.15.26:0/2219854983 >> 172.21.15.26:0/2219854983 conn(0x7fa3000a8740 msgr2=0x7fa3000aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.183+0000 7fa305f14640 1 -- 172.21.15.26:0/2219854983 shutdown_connections 2024-08-19T12:42:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.183+0000 7fa305f14640 1 -- 172.21.15.26:0/2219854983 wait complete. 2024-08-19T12:42:56.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.183+0000 7fa305f14640 1 librados: shutdown 2024-08-19T12:42:56.208 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:56.208 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:42:56.208 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:42:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.307+0000 7f615a258640 10 monclient: get_monmap_and_config 2024-08-19T12:42:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.307+0000 7f615a258640 10 monclient: build_initial_monmap 2024-08-19T12:42:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.307+0000 7f615a258640 10 monclient: monmap: 2024-08-19T12:42:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:56.308549+0000 2024-08-19T12:42:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:56.308549+0000 2024-08-19T12:42:56.309 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:56.310 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:56.311 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:56.311 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.308+0000 7f615a258640 1 Processor -- start 2024-08-19T12:42:56.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.308+0000 7f615a258640 1 -- start start 2024-08-19T12:42:56.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.308+0000 7f615a258640 10 monclient: init 2024-08-19T12:42:56.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f615a258640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:56.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f615a258640 10 monclient: _add_conns ranks=[4,7,2,0,5,3,8,1,6] 2024-08-19T12:42:56.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f615a258640 10 monclient(hunting): picked mon.noname-c con 0x7f615406e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:42:56.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f615a258640 10 monclient(hunting): picked mon.noname-i con 0x7f61540a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:56.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f615a258640 10 monclient(hunting): picked mon.noname-b con 0x7f61540a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:42:56.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f615a258640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6154136790 con 0x7f61540a5d50 2024-08-19T12:42:56.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f615a258640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615409ddf0 con 0x7f615406e7b0 2024-08-19T12:42:56.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f615a258640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615409e0d0 con 0x7f61540a5920 2024-08-19T12:42:56.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f615a258640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:56.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f615a258640 10 monclient(hunting): authenticate will time out at 970.396115s 2024-08-19T12:42:56.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f61537fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f615406e7b0 0x7f6154136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:54532/0 (socket says 172.21.15.26:54532) 2024-08-19T12:42:56.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f61537fe640 1 -- 172.21.15.26:0/2235650731 learned_addr learned my addr 172.21.15.26:0/2235650731 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:56.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.309+0000 7f6153fff640 1 --1- 172.21.15.26:0/2235650731 >> v1:172.21.15.26:6790/0 conn(0x7f61540a5d50 0x7f6154141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51776/0 (socket says 172.21.15.26:51776) 2024-08-19T12:42:56.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f6153fff640 1 -- 172.21.15.26:0/2235650731 >> v1:172.21.15.26:6790/0 conn(0x7f61540a5d50 legacy=0x7f6154141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61537fe640 1 -- 172.21.15.26:0/2235650731 >> v1:172.21.15.26:6791/0 conn(0x7f615406e7b0 legacy=0x7f6154136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 1 -- 172.21.15.26:0/2235650731 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2940679762 0 0) 0x7f6154136790 con 0x7f61540a5d50 2024-08-19T12:42:56.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient(hunting): my global_id is 26066 2024-08-19T12:42:56.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 1 -- 172.21.15.26:0/2235650731 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f613c003520 con 0x7f61540a5d50 2024-08-19T12:42:56.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 1 -- 172.21.15.26:0/2235650731 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1665574432 0 0) 0x7f6144001450 con 0x7f615406e7b0 2024-08-19T12:42:56.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient(hunting): my global_id is 26320 2024-08-19T12:42:56.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 1 -- 172.21.15.26:0/2235650731 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6154136790 con 0x7f615406e7b0 2024-08-19T12:42:56.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f6152ffd640 1 -- 172.21.15.26:0/2235650731 >> v1:172.21.15.104:6792/0 conn(0x7f61540a5920 legacy=0x7f615413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 1 -- 172.21.15.26:0/2235650731 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2703740338 0 0) 0x7f6154136790 con 0x7f615406e7b0 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 1 -- 172.21.15.26:0/2235650731 >> v1:172.21.15.104:6792/0 conn(0x7f61540a5920 legacy=0x7f615413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 1 -- 172.21.15.26:0/2235650731 >> v1:172.21.15.26:6790/0 conn(0x7f61540a5d50 legacy=0x7f6154141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient: _finish_hunting 0 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 1 monclient: found mon.noname-c 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 1 -- 172.21.15.26:0/2235650731 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615409e3b0 con 0x7f615406e7b0 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient: _finish_auth 0 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient: _check_auth_tickets 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 1 -- 172.21.15.26:0/2235650731 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6144001c90 con 0x7f615406e7b0 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f615a258640 5 monclient: authenticate success, global_id 26320 2024-08-19T12:42:56.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 10 monclient: dump: 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f615a258640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.311+0000 7f61527fc640 1 -- 172.21.15.26:0/2235650731 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6144004920 con 0x7f615406e7b0 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f61527fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f615a258640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f61527fc640 1 -- 172.21.15.26:0/2235650731 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6144004e90 con 0x7f615406e7b0 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f615a258640 4 set_mon_vals no callback set 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f61527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f61527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:42:56.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f61527fc640 10 monclient: dump: 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.330 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f615a258640 10 monclient: shutdown 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f615a258640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f615a258640 1 -- 172.21.15.26:0/2235650731 >> v1:172.21.15.26:6791/0 conn(0x7f615406e7b0 legacy=0x7f6154136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f615a258640 1 -- 172.21.15.26:0/2235650731 shutdown_connections 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f615a258640 1 -- 172.21.15.26:0/2235650731 >> 172.21.15.26:0/2235650731 conn(0x7f61540a8740 msgr2=0x7f61540a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f615a258640 1 -- 172.21.15.26:0/2235650731 shutdown_connections 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.312+0000 7f615a258640 1 -- 172.21.15.26:0/2235650731 wait complete. 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.313+0000 7f615a258640 10 monclient: build_initial_monmap 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.313+0000 7f615a258640 10 monclient: monmap: 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:56.332 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.313+0000 7f615a258640 1 librados: starting msgr at 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.313+0000 7f615a258640 1 librados: starting objecter 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.313+0000 7f615a258640 1 Processor -- start 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.313+0000 7f615a258640 1 -- start start 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.313+0000 7f615a258640 1 librados: setting wanted keys 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.313+0000 7f615a258640 1 librados: calling monclient init 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.313+0000 7f615a258640 10 monclient: init 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.314+0000 7f615a258640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.314+0000 7f615a258640 10 monclient: _add_conns ranks=[1,2,0,4,6,7,3,8,5] 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.314+0000 7f615a258640 10 monclient(hunting): picked mon.noname-f con 0x7f615406e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.314+0000 7f615a258640 10 monclient(hunting): picked mon.noname-b con 0x7f61540a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.314+0000 7f615a258640 10 monclient(hunting): picked mon.noname-a con 0x7f61540a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:56.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.314+0000 7f615a258640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615413b490 con 0x7f61540a5d50 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.314+0000 7f615a258640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6154139130 con 0x7f615406e7b0 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.314+0000 7f615a258640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615413abf0 con 0x7f61540a5920 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.314+0000 7f615a258640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.314+0000 7f615a258640 10 monclient(hunting): authenticate will time out at 970.401166s 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.314+0000 7f6152ffd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f61540a5920 0x7f61541d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:51788/0 (socket says 172.21.15.26:51788) 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.314+0000 7f6152ffd640 1 -- 172.21.15.26:0/717994095 learned_addr learned my addr 172.21.15.26:0/717994095 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f6152ffd640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.26:6790/0 conn(0x7f61540a5920 legacy=0x7f61541d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 client.?.objecter ms_handle_connect 0x7f61540a5920 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 client.?.objecter ms_handle_connect 0x7f61540a5d50 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f6153fff640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.26:6789/0 conn(0x7f61540a5d50 legacy=0x7f61541d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3045145144 0 0) 0x7f615413abf0 con 0x7f61540a5920 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): my global_id is 26075 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6140004eb0 con 0x7f61540a5920 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875588270 0 0) 0x7f615413b490 con 0x7f61540a5d50 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): my global_id is 144288 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f615413abf0 con 0x7f61540a5d50 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 client.?.objecter ms_handle_connect 0x7f615406e7b0 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:56.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f61537fe640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.104:6789/0 conn(0x7f615406e7b0 legacy=0x7f61541ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3245500865 0 0) 0x7f6140004eb0 con 0x7f61540a5920 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f615413b490 con 0x7f61540a5920 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6148003610 con 0x7f61540a5920 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): dump: 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.336 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1842523622 0 0) 0x7f615413abf0 con 0x7f61540a5d50 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6140004eb0 con 0x7f61540a5d50 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228639535 0 0) 0x7f6154139130 con 0x7f615406e7b0 2024-08-19T12:42:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): my global_id is 26767 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f615413abf0 con 0x7f615406e7b0 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f614c0020b0 con 0x7f61540a5d50 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.316+0000 7f613bfff640 10 monclient(hunting): dump: 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.338 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3702218255 0 0) 0x7f615413b490 con 0x7f61540a5920 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.104:6789/0 conn(0x7f615406e7b0 legacy=0x7f61541ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.26:6789/0 conn(0x7f61540a5d50 legacy=0x7f61541d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 1 monclient: found mon.b 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615413aed0 con 0x7f61540a5920 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 10 monclient: _finish_auth 0 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f615a258640 5 monclient: authenticate success, global_id 26075 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f615a258640 10 monclient: _renew_subs 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f615a258640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f615a258640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f615413a8a0 con 0x7f61540a5920 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f615a258640 10 client.26075.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:56.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f615a258640 10 monclient: _renew_subs 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f615a258640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f615a258640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f61541386a0 con 0x7f61540a5920 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f615a258640 1 librados: init done 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6148003b90 con 0x7f61540a5920 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f61480040c0 con 0x7f61540a5920 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f6151ffb640 4 set_mon_vals no callback set 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f613bfff640 10 monclient: dump: 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.341 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f615a258640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.317+0000 7f615a258640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f61340065c0 con 0x7f6134002860 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.318+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6148004260 con 0x7f61540a5920 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.318+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(186..186 src has 1..186) ==== 5738+0+0 (unknown 3584490421 0 0) 0x7f614804e400 con 0x7f61540a5920 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.318+0000 7f613bfff640 10 client.26075.objecter ms_dispatch 0x7f6154000b90 osd_map(186..186 src has 1..186) 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.318+0000 7f613bfff640 3 client.26075.objecter handle_osd_map got epochs [186,186] > 0 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.318+0000 7f613bfff640 3 client.26075.objecter handle_osd_map decoding full epoch 186 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.318+0000 7f613bfff640 20 client.26075.objecter dump_active .. 0 homeless 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.318+0000 7f6153fff640 10 monclient: get_auth_request con 0x7f6140041930 auth_method 0 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f61537fe640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.104:6791/0 conn(0x7f6134002860 legacy=0x7f6134004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 10 client.26075.objecter ms_handle_connect 0x7f6134002860 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 10 client.26075.objecter resend_mon_ops 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2124498303 0 0) 0x7f61340065c0 con 0x7f6134002860 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 10 monclient(hunting): my global_id is 26033 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f615413b490 con 0x7f6134002860 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2319127700 0 0) 0x7f615413b490 con 0x7f6134002860 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61340065c0 con 0x7f6134002860 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 10 client.26075.objecter ms_handle_connect 0x7f6140041930 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6144001a90 con 0x7f6134002860 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 10 monclient: dump: 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.344 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3673669551 0 0) 0x7f61340065c0 con 0x7f6134002860 2024-08-19T12:42:56.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.320+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f613400a160 con 0x7f6134002860 2024-08-19T12:42:56.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.321+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6144004870 con 0x7f6134002860 2024-08-19T12:42:56.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.321+0000 7f613bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:56.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.321+0000 7f613bfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:56.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.321+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.104:6791/0 conn(0x7f6134002860 legacy=0x7f6134004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.321+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.104:6791/0 conn(0x7f6134002860 legacy=0x7f6134004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:56.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.327+0000 7f615a258640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:56.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.327+0000 7f615a258640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6134009e50 con 0x7f6134009110 2024-08-19T12:42:56.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f61537fe640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.104:6791/0 conn(0x7f6134009110 legacy=0x7f61340095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 10 client.26075.objecter ms_handle_connect 0x7f6134009110 2024-08-19T12:42:56.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 10 client.26075.objecter resend_mon_ops 2024-08-19T12:42:56.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3018842842 0 0) 0x7f6134009e50 con 0x7f6134009110 2024-08-19T12:42:56.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 10 monclient(hunting): my global_id is 26042 2024-08-19T12:42:56.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61340065c0 con 0x7f6134009110 2024-08-19T12:42:56.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3976399446 0 0) 0x7f61340065c0 con 0x7f6134009110 2024-08-19T12:42:56.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6134009e50 con 0x7f6134009110 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f61440051d0 con 0x7f6134009110 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 10 monclient: dump: 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.349 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2523889239 0 0) 0x7f6134009e50 con 0x7f6134009110 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.330+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f613400c3f0 con 0x7f6134009110 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 899278222) 0x7f6144007e30 con 0x7f6134009110 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f613bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f613bfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.104:6791/0 conn(0x7f6134009110 legacy=0x7f61340095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f613bfff640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.104:6791/0 conn(0x7f6134009110 legacy=0x7f61340095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:56.351 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:56.351 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:56.351 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:56.351 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:56.351 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:56.351 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 289353, 2024-08-19T12:42:56.351 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:56.351 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:56.351 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:56.351 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 7, 2024-08-19T12:42:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:56.352 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:56.353 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:56.354 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:56.355 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:56.356 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.357 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.358 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:56.359 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:56.363 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.364 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.366 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.367 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:56.369 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:56.370 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:56.371 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:42:56.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.371 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.371 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:56.371 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:56.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 10 librados: watch_flush enter 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 10 librados: watch_flush exit 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 20 client.26075.objecter shutdown clearing up homeless session... 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 10 client.26075.objecter successfully canceled tick 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.104:6800/899887506 conn(0x7f6140041930 legacy=0x7f6140043df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 10 monclient: shutdown 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 1 -- 172.21.15.26:0/717994095 >> v1:172.21.15.26:6790/0 conn(0x7f61540a5920 legacy=0x7f61541d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f6153fff640 1 -- 172.21.15.26:0/717994095 reap_dead start 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f6153fff640 1 -- 172.21.15.26:0/717994095 reap_dead start 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 1 -- 172.21.15.26:0/717994095 shutdown_connections 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 1 -- 172.21.15.26:0/717994095 >> 172.21.15.26:0/717994095 conn(0x7f61540a8740 msgr2=0x7f61540aa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 1 -- 172.21.15.26:0/717994095 shutdown_connections 2024-08-19T12:42:56.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 1 -- 172.21.15.26:0/717994095 wait complete. 2024-08-19T12:42:56.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.331+0000 7f615a258640 1 librados: shutdown 2024-08-19T12:42:56.374 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:56.374 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:42:56.374 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:42:56.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.472+0000 7f2795098640 10 monclient: get_monmap_and_config 2024-08-19T12:42:56.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.472+0000 7f2795098640 10 monclient: build_initial_monmap 2024-08-19T12:42:56.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.472+0000 7f2795098640 10 monclient: monmap: 2024-08-19T12:42:56.474 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:56.474 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.474 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:42:56.473714+0000 2024-08-19T12:42:56.474 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:42:56.473714+0000 2024-08-19T12:42:56.474 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:56.474 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.474 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:56.474 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:56.474 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:56.474 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:56.474 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:56.475 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:56.475 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:56.475 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:56.475 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:56.475 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.473+0000 7f2795098640 1 Processor -- start 2024-08-19T12:42:56.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.473+0000 7f2795098640 1 -- start start 2024-08-19T12:42:56.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.473+0000 7f2795098640 10 monclient: init 2024-08-19T12:42:56.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.473+0000 7f2795098640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:56.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.473+0000 7f2795098640 10 monclient: _add_conns ranks=[0,3,6,8,2,4,1,5,7] 2024-08-19T12:42:56.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.473+0000 7f2795098640 10 monclient(hunting): picked mon.noname-a con 0x7f279006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:42:56.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.473+0000 7f2795098640 10 monclient(hunting): picked mon.noname-g con 0x7f27900a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:42:56.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.474+0000 7f2795098640 10 monclient(hunting): picked mon.noname-d con 0x7f27900a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:42:56.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.474+0000 7f2795098640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2790136790 con 0x7f279006e7b0 2024-08-19T12:42:56.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.474+0000 7f2795098640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279009ddf0 con 0x7f27900a5920 2024-08-19T12:42:56.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.474+0000 7f2795098640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279009e0d0 con 0x7f27900a5d50 2024-08-19T12:42:56.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.474+0000 7f2795098640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:56.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.474+0000 7f2795098640 10 monclient(hunting): authenticate will time out at 970.561068s 2024-08-19T12:42:56.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.474+0000 7f278f59e640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f27900a5d50 0x7f2790141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34622/0 (socket says 172.21.15.26:34622) 2024-08-19T12:42:56.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.474+0000 7f278f59e640 1 -- 172.21.15.26:0/2451134677 learned_addr learned my addr 172.21.15.26:0/2451134677 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:56.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278f59e640 1 -- 172.21.15.26:0/2451134677 >> v1:172.21.15.26:6792/0 conn(0x7f27900a5d50 legacy=0x7f2790141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278ed9d640 1 -- 172.21.15.26:0/2451134677 >> v1:172.21.15.26:6789/0 conn(0x7f279006e7b0 legacy=0x7f2790136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 1 -- 172.21.15.26:0/2451134677 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4179999462 0 0) 0x7f279009e0d0 con 0x7f27900a5d50 2024-08-19T12:42:56.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient(hunting): my global_id is 26340 2024-08-19T12:42:56.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 1 -- 172.21.15.26:0/2451134677 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f277c003520 con 0x7f27900a5d50 2024-08-19T12:42:56.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 1 -- 172.21.15.26:0/2451134677 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 156233385 0 0) 0x7f2790136790 con 0x7f279006e7b0 2024-08-19T12:42:56.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient(hunting): my global_id is 144297 2024-08-19T12:42:56.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 1 -- 172.21.15.26:0/2451134677 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f279009e0d0 con 0x7f279006e7b0 2024-08-19T12:42:56.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278e59c640 1 -- 172.21.15.26:0/2451134677 >> v1:172.21.15.104:6790/0 conn(0x7f27900a5920 legacy=0x7f279013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 1 -- 172.21.15.26:0/2451134677 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 592521855 0 0) 0x7f277c003520 con 0x7f27900a5d50 2024-08-19T12:42:56.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 1 -- 172.21.15.26:0/2451134677 >> v1:172.21.15.104:6790/0 conn(0x7f27900a5920 legacy=0x7f279013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 1 -- 172.21.15.26:0/2451134677 >> v1:172.21.15.26:6789/0 conn(0x7f279006e7b0 legacy=0x7f2790136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient: _finish_hunting 0 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 1 monclient: found mon.noname-d 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 1 -- 172.21.15.26:0/2451134677 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279009e3b0 con 0x7f27900a5d50 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient: _finish_auth 0 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient: _check_auth_tickets 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f2795098640 5 monclient: authenticate success, global_id 26340 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 1 -- 172.21.15.26:0/2451134677 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2788001ae0 con 0x7f27900a5d50 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f2795098640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient: dump: 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.484 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f2795098640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 1 -- 172.21.15.26:0/2451134677 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f27880048a0 con 0x7f27900a5d50 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 1 -- 172.21.15.26:0/2451134677 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2788004e50 con 0x7f27900a5d50 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f2795098640 10 monclient: get_monmap_and_config success 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f2795098640 4 set_mon_vals no callback set 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:42:56.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.476+0000 7f278dd9b640 10 monclient: dump: 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.486 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.477+0000 7f2795098640 10 monclient: shutdown 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.477+0000 7f2795098640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.477+0000 7f2795098640 1 -- 172.21.15.26:0/2451134677 >> v1:172.21.15.26:6792/0 conn(0x7f27900a5d50 legacy=0x7f2790141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.477+0000 7f2795098640 1 -- 172.21.15.26:0/2451134677 shutdown_connections 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.477+0000 7f2795098640 1 -- 172.21.15.26:0/2451134677 >> 172.21.15.26:0/2451134677 conn(0x7f27900a8740 msgr2=0x7f27900a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.477+0000 7f2795098640 1 -- 172.21.15.26:0/2451134677 shutdown_connections 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.477+0000 7f2795098640 1 -- 172.21.15.26:0/2451134677 wait complete. 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.478+0000 7f2795098640 10 monclient: build_initial_monmap 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.478+0000 7f2795098640 10 monclient: monmap: 2024-08-19T12:42:56.487 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:42:56.488 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.478+0000 7f2795098640 1 librados: starting msgr at 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.478+0000 7f2795098640 1 librados: starting objecter 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.478+0000 7f2795098640 1 Processor -- start 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.478+0000 7f2795098640 1 -- start start 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.478+0000 7f2795098640 1 librados: setting wanted keys 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.478+0000 7f2795098640 1 librados: calling monclient init 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.478+0000 7f2795098640 10 monclient: init 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.479+0000 7f2795098640 10 monclient: _reopen_session rank -1 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.479+0000 7f2795098640 10 monclient: _add_conns ranks=[7,8,1,4,5,2,3,0,6] 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.479+0000 7f2795098640 10 monclient(hunting): picked mon.noname-i con 0x7f279006e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.479+0000 7f2795098640 10 monclient(hunting): picked mon.noname-e con 0x7f27900a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:42:56.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.479+0000 7f2795098640 10 monclient(hunting): picked mon.noname-f con 0x7f27900a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:42:56.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.479+0000 7f2795098640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2790142ec0 con 0x7f27900a5d50 2024-08-19T12:42:56.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.479+0000 7f2795098640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279013b120 con 0x7f279006e7b0 2024-08-19T12:42:56.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.479+0000 7f2795098640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279013b400 con 0x7f27900a5920 2024-08-19T12:42:56.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.479+0000 7f2795098640 10 monclient(hunting): _renew_subs 2024-08-19T12:42:56.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.479+0000 7f2795098640 10 monclient(hunting): authenticate will time out at 970.566140s 2024-08-19T12:42:56.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.479+0000 7f278e59c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f27900a5920 0x7f27901d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:53534/0 (socket says 172.21.15.26:53534) 2024-08-19T12:42:56.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.479+0000 7f278e59c640 1 -- 172.21.15.26:0/1722175548 learned_addr learned my addr 172.21.15.26:0/1722175548 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:42:56.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f278e59c640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.26:6793/0 conn(0x7f27900a5920 legacy=0x7f27901d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 client.?.objecter ms_handle_connect 0x7f27900a5920 2024-08-19T12:42:56.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:56.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f278f59e640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.104:6789/0 conn(0x7f27900a5d50 legacy=0x7f27901d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 client.?.objecter ms_handle_connect 0x7f27900a5d50 2024-08-19T12:42:56.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1564277373 0 0) 0x7f279013b400 con 0x7f27900a5920 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): my global_id is 25856 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2770004eb0 con 0x7f27900a5920 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 client.?.objecter ms_handle_connect 0x7f279006e7b0 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f278ed9d640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.104:6792/0 conn(0x7f279006e7b0 legacy=0x7f27901ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4217347627 0 0) 0x7f2770004eb0 con 0x7f27900a5920 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f279013b400 con 0x7f27900a5920 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2778003610 con 0x7f27900a5920 2024-08-19T12:42:56.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): dump: 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.502 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567817129 0 0) 0x7f279009e3b0 con 0x7f27900a5d50 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): my global_id is 26776 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2770004eb0 con 0x7f27900a5d50 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3056002592 0 0) 0x7f279013b120 con 0x7f279006e7b0 2024-08-19T12:42:56.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient(hunting): my global_id is 134872 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f279009e3b0 con 0x7f279006e7b0 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2217806345 0 0) 0x7f279013b400 con 0x7f27900a5920 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.104:6792/0 conn(0x7f279006e7b0 legacy=0x7f27901ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.104:6789/0 conn(0x7f27900a5d50 legacy=0x7f27901d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient: _finish_hunting 0 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 monclient: found mon.e 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279013a8a0 con 0x7f27900a5920 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient: _finish_auth 0 2024-08-19T12:42:56.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 10 monclient: _check_auth_tickets 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f27877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.481+0000 7f2795098640 5 monclient: authenticate success, global_id 25856 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f2795098640 10 monclient: _renew_subs 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f2795098640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f2795098640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f27901390a0 con 0x7f27900a5920 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f2795098640 10 client.25856.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f2795098640 10 monclient: _renew_subs 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f2795098640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f2795098640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f27901385f0 con 0x7f27900a5920 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f2795098640 1 librados: init done 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2778003bd0 con 0x7f27900a5920 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f27877fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:42:56.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2778004100 con 0x7f27900a5920 2024-08-19T12:42:56.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f27877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f278d59a640 4 set_mon_vals no callback set 2024-08-19T12:42:56.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f27877fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:42:56.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f27877fe640 10 monclient: dump: 2024-08-19T12:42:56.506 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.506 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.506 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.506 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.506 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.506 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.506 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.507 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.507 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.507 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.507 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.507 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.507 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.507 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.507 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.507 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f2795098640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:56.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f2795098640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f27740065c0 con 0x7f2774002860 2024-08-19T12:42:56.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.482+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f27780042a0 con 0x7f27900a5920 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.483+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(186..186 src has 1..186) ==== 5738+0+0 (unknown 3584490421 0 0) 0x7f277804e470 con 0x7f27900a5920 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.483+0000 7f27877fe640 10 client.25856.objecter ms_dispatch 0x7f2790000b90 osd_map(186..186 src has 1..186) 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.483+0000 7f27877fe640 3 client.25856.objecter handle_osd_map got epochs [186,186] > 0 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.483+0000 7f27877fe640 3 client.25856.objecter handle_osd_map decoding full epoch 186 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.483+0000 7f27877fe640 20 client.25856.objecter dump_active .. 0 homeless 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.483+0000 7f278f59e640 10 monclient: get_auth_request con 0x7f27700411b0 auth_method 0 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f278ed9d640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.104:6792/0 conn(0x7f2774002860 legacy=0x7f2774004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 10 client.25856.objecter ms_handle_connect 0x7f2774002860 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 10 client.25856.objecter resend_mon_ops 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 211620383 0 0) 0x7f27740065c0 con 0x7f2774002860 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 10 monclient(hunting): my global_id is 134881 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f279013b400 con 0x7f2774002860 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2040556796 0 0) 0x7f279013b400 con 0x7f2774002860 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27740065c0 con 0x7f2774002860 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2780001600 con 0x7f2774002860 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 10 monclient: dump: 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 10 client.25856.objecter ms_handle_connect 0x7f27700411b0 2024-08-19T12:42:56.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 983640704 0 0) 0x7f27740065c0 con 0x7f2774002860 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.485+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f277400a160 con 0x7f2774002860 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.486+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2780005b40 con 0x7f2774002860 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.486+0000 7f27877fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.486+0000 7f27877fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.486+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.104:6792/0 conn(0x7f2774002860 legacy=0x7f2774004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.486+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.104:6792/0 conn(0x7f2774002860 legacy=0x7f2774004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.492+0000 7f2795098640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.492+0000 7f2795098640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2774009e70 con 0x7f2774009130 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.494+0000 7f278ed9d640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.104:6792/0 conn(0x7f2774009130 legacy=0x7f27740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.494+0000 7f27877fe640 10 client.25856.objecter ms_handle_connect 0x7f2774009130 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.494+0000 7f27877fe640 10 client.25856.objecter resend_mon_ops 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.494+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2098813839 0 0) 0x7f2774009e70 con 0x7f2774009130 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.494+0000 7f27877fe640 10 monclient(hunting): _negotiate 2024-08-19T12:42:56.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.494+0000 7f27877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.494+0000 7f27877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.494+0000 7f27877fe640 10 monclient(hunting): my global_id is 134890 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27740065c0 con 0x7f2774009130 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1986333267 0 0) 0x7f27740065c0 con 0x7f2774009130 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2774009e70 con 0x7f2774009130 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f27800040c0 con 0x7f2774009130 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 10 monclient: dump: 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.512 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 686156258 0 0) 0x7f2774009e70 con 0x7f2774009130 2024-08-19T12:42:56.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f277400c410 con 0x7f2774009130 2024-08-19T12:42:56.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1664018733) 0x7f2780004280 con 0x7f2774009130 2024-08-19T12:42:56.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:42:56.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:42:56.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.104:6792/0 conn(0x7f2774009130 legacy=0x7f27740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.495+0000 7f27877fe640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.104:6792/0 conn(0x7f2774009130 legacy=0x7f27740095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:42:56.514 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:42:56.514 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:56.514 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:56.514 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 24, 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 146853, 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:42:56.515 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 7, 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:56.516 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:56.517 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:56.518 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:56.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.518 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:42:56.518 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:42:56.518 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:42:56.518 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:42:56.518 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:42:56.518 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:42:56.518 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:56.518 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:42:56.518 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:42:56.519 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:42:56.520 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:42:56.520 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:42:56.520 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:42:56.520 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:42:56.520 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:42:56.520 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:42:56.520 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.520 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:42:56.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.520 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:42:56.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.520 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:42:56.521 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.522 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.522 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.522 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:42:56.522 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:42:56.522 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:42:56.523 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.524 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.525 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.527 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.528 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:42:56.531 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:42:56.532 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:42:56.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:42:56.534 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:42:56.534 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:42:56.534 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:42:56.534 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:42:56.534 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:42:56.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:42:56.534 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:42:56.534 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:42:56.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 10 librados: watch_flush enter 2024-08-19T12:42:56.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 10 librados: watch_flush exit 2024-08-19T12:42:56.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 20 client.25856.objecter shutdown clearing up homeless session... 2024-08-19T12:42:56.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 10 client.25856.objecter successfully canceled tick 2024-08-19T12:42:56.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.104:6800/899887506 conn(0x7f27700411b0 legacy=0x7f2770043670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 10 monclient: shutdown 2024-08-19T12:42:56.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:42:56.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 1 -- 172.21.15.26:0/1722175548 >> v1:172.21.15.26:6793/0 conn(0x7f27900a5920 legacy=0x7f27901d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:42:56.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f278f59e640 1 -- 172.21.15.26:0/1722175548 reap_dead start 2024-08-19T12:42:56.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f278f59e640 1 -- 172.21.15.26:0/1722175548 reap_dead start 2024-08-19T12:42:56.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 1 -- 172.21.15.26:0/1722175548 shutdown_connections 2024-08-19T12:42:56.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 1 -- 172.21.15.26:0/1722175548 >> 172.21.15.26:0/1722175548 conn(0x7f27900a8740 msgr2=0x7f27900aa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:42:56.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 1 -- 172.21.15.26:0/1722175548 shutdown_connections 2024-08-19T12:42:56.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 1 -- 172.21.15.26:0/1722175548 wait complete. 2024-08-19T12:42:56.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:42:56.496+0000 7f2795098640 1 librados: shutdown 2024-08-19T12:42:56.537 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:42:56.537 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['g'] 2024-08-19T12:42:56.537 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-08-19T12:42:56.538 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-08-19T12:42:56.538 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-08-19T12:42:56.538 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-08-19T12:42:56.538 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:42:56.555 INFO:tasks.ceph.mon.g:Stopped 2024-08-19T12:42:56.555 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-08-19T12:42:56.555 INFO:tasks.ceph.mon.a:Sent signal 19 2024-08-19T12:42:56.555 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:43:11.556 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-08-19T12:43:11.556 INFO:tasks.ceph.mon.a:Sent signal 18 2024-08-19T12:43:11.556 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-08-19T12:43:11.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:43:11.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.667+0000 7f2bb0656640 10 monclient: get_monmap_and_config 2024-08-19T12:43:11.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.667+0000 7f2bb0656640 10 monclient: build_initial_monmap 2024-08-19T12:43:11.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.667+0000 7f2bb0656640 10 monclient: monmap: 2024-08-19T12:43:11.668 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:11.668 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:43:11.668406+0000 2024-08-19T12:43:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:43:11.668406+0000 2024-08-19T12:43:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:11.670 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:11.670 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:11.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.668+0000 7f2bb0656640 1 Processor -- start 2024-08-19T12:43:11.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.668+0000 7f2bb0656640 1 -- start start 2024-08-19T12:43:11.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.668+0000 7f2bb0656640 10 monclient: init 2024-08-19T12:43:11.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.668+0000 7f2bb0656640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:11.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.668+0000 7f2bb0656640 10 monclient: _add_conns ranks=[0,2,8,5,3,4,7,6,1] 2024-08-19T12:43:11.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.668+0000 7f2bb0656640 10 monclient(hunting): picked mon.noname-a con 0x7f2ba806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:43:11.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.668+0000 7f2bb0656640 10 monclient(hunting): picked mon.noname-b con 0x7f2ba80a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:43:11.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.669+0000 7f2bb0656640 10 monclient(hunting): picked mon.noname-e con 0x7f2ba80a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:43:11.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.669+0000 7f2bb0656640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ba8136790 con 0x7f2ba806e7b0 2024-08-19T12:43:11.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.669+0000 7f2bb0656640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ba809ddf0 con 0x7f2ba80a5920 2024-08-19T12:43:11.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.669+0000 7f2bb0656640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ba809e0d0 con 0x7f2ba80a5d50 2024-08-19T12:43:11.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.669+0000 7f2bb0656640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:11.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.669+0000 7f2bb0656640 10 monclient(hunting): authenticate will time out at 985.756042s 2024-08-19T12:43:11.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.669+0000 7f2badbf1640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2ba80a5920 0x7f2ba813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48786/0 (socket says 172.21.15.26:48786) 2024-08-19T12:43:11.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.669+0000 7f2baebf3640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f2ba80a5d50 0x7f2ba8141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:58536/0 (socket says 172.21.15.26:58536) 2024-08-19T12:43:11.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.669+0000 7f2badbf1640 1 -- 172.21.15.26:0/1822623434 learned_addr learned my addr 172.21.15.26:0/1822623434 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:11.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.669+0000 7f2bae3f2640 1 --1- 172.21.15.26:0/1822623434 >> v1:172.21.15.26:6789/0 conn(0x7f2ba806e7b0 0x7f2ba8136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58096/0 (socket says 172.21.15.26:58096) 2024-08-19T12:43:11.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.671+0000 7f2badbf1640 1 -- 172.21.15.26:0/1822623434 >> v1:172.21.15.26:6790/0 conn(0x7f2ba80a5920 legacy=0x7f2ba813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:11.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.671+0000 7f2bae3f2640 1 -- 172.21.15.26:0/1822623434 >> v1:172.21.15.26:6789/0 conn(0x7f2ba806e7b0 legacy=0x7f2ba8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:11.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:11.671+0000 7f2baebf3640 1 -- 172.21.15.26:0/1822623434 >> v1:172.21.15.26:6793/0 conn(0x7f2ba80a5d50 legacy=0x7f2ba8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:14.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 10 monclient(hunting): tick 2024-08-19T12:43:14.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:43:14.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 1 monclient(hunting): continuing hunt 2024-08-19T12:43:14.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:43:14.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 1 -- 172.21.15.26:0/1822623434 >> v1:172.21.15.26:6793/0 conn(0x7f2ba80a5d50 legacy=0x7f2ba8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:14.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 1 -- 172.21.15.26:0/1822623434 >> v1:172.21.15.26:6790/0 conn(0x7f2ba80a5920 legacy=0x7f2ba813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 1 -- 172.21.15.26:0/1822623434 >> v1:172.21.15.26:6789/0 conn(0x7f2ba806e7b0 legacy=0x7f2ba8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 10 monclient: _add_conns ranks=[4,5,7,3,1,6] 2024-08-19T12:43:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f2b8c003960 addr v1:172.21.15.26:6791/0 2024-08-19T12:43:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 10 monclient(hunting): picked mon.noname-h con 0x7f2b8c0075f0 addr v1:172.21.15.104:6791/0 2024-08-19T12:43:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 10 monclient(hunting): picked mon.noname-i con 0x7f2b8c00b380 addr v1:172.21.15.104:6792/0 2024-08-19T12:43:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 1 -- 172.21.15.26:0/1822623434 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ba809e3b0 con 0x7f2b8c003960 2024-08-19T12:43:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 1 -- 172.21.15.26:0/1822623434 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b8c0101a0 con 0x7f2b8c0075f0 2024-08-19T12:43:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 1 -- 172.21.15.26:0/1822623434 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b8c011420 con 0x7f2b8c00b380 2024-08-19T12:43:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.669+0000 7f2b9ffff640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:14.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.671+0000 7f2bae3f2640 1 -- 172.21.15.26:0/1822623434 >> v1:172.21.15.26:6791/0 conn(0x7f2b8c003960 legacy=0x7f2b8c005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:14.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.672+0000 7f2baebf3640 1 -- 172.21.15.26:0/1822623434 >> v1:172.21.15.104:6792/0 conn(0x7f2b8c00b380 legacy=0x7f2b8c00d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:14.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:14.672+0000 7f2badbf1640 1 -- 172.21.15.26:0/1822623434 >> v1:172.21.15.104:6791/0 conn(0x7f2b8c0075f0 legacy=0x7f2b8c009ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:16.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 1 -- 172.21.15.26:0/1822623434 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1884186991 0 0) 0x7f2b8c011420 con 0x7f2b8c00b380 2024-08-19T12:43:16.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 10 monclient(hunting): _negotiate 2024-08-19T12:43:16.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:16.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:16.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 10 monclient(hunting): my global_id is 134908 2024-08-19T12:43:16.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 1 -- 172.21.15.26:0/1822623434 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b94003520 con 0x7f2b8c00b380 2024-08-19T12:43:16.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 1 -- 172.21.15.26:0/1822623434 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2225452163 0 0) 0x7f2b8c0101a0 con 0x7f2b8c0075f0 2024-08-19T12:43:16.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 10 monclient(hunting): _negotiate 2024-08-19T12:43:16.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:16.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:16.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 10 monclient(hunting): my global_id is 26060 2024-08-19T12:43:16.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 1 -- 172.21.15.26:0/1822623434 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b8c011420 con 0x7f2b8c0075f0 2024-08-19T12:43:16.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 1 -- 172.21.15.26:0/1822623434 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1760059622 0 0) 0x7f2b8c011420 con 0x7f2b8c0075f0 2024-08-19T12:43:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 1 -- 172.21.15.26:0/1822623434 >> v1:172.21.15.104:6792/0 conn(0x7f2b8c00b380 legacy=0x7f2b8c00d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 1 -- 172.21.15.26:0/1822623434 >> v1:172.21.15.26:6791/0 conn(0x7f2b8c003960 legacy=0x7f2b8c005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 10 monclient: _finish_hunting 0 2024-08-19T12:43:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 1 monclient: found mon.noname-h 2024-08-19T12:43:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2bad3f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:43:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.568+0000 7f2baebf3640 1 -- 172.21.15.26:0/1822623434 reap_dead start 2024-08-19T12:43:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 1 -- 172.21.15.26:0/1822623434 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8c012760 con 0x7f2b8c0075f0 2024-08-19T12:43:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 10 monclient: _finish_auth 0 2024-08-19T12:43:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 10 monclient: _check_auth_tickets 2024-08-19T12:43:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 1 -- 172.21.15.26:0/1822623434 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2b98001a60 con 0x7f2b8c0075f0 2024-08-19T12:43:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:43:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 10 monclient: dump: 2024-08-19T12:43:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 1 -- 172.21.15.26:0/1822623434 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ba0001e20 con 0x7f2b8c00b380 2024-08-19T12:43:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bb0656640 5 monclient: authenticate success, global_id 26060 2024-08-19T12:43:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bb0656640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:43:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 1 -- 172.21.15.26:0/1822623434 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b98001cb0 con 0x7f2b8c0075f0 2024-08-19T12:43:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 1 -- 172.21.15.26:0/1822623434 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2b98004be0 con 0x7f2b8c0075f0 2024-08-19T12:43:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:43:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bad3f0640 10 monclient: dump: 2024-08-19T12:43:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.583 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bb0656640 10 monclient: get_monmap_and_config success 2024-08-19T12:43:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.569+0000 7f2bb0656640 4 set_mon_vals no callback set 2024-08-19T12:43:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.570+0000 7f2bb0656640 10 monclient: shutdown 2024-08-19T12:43:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.570+0000 7f2bb0656640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.570+0000 7f2bb0656640 1 -- 172.21.15.26:0/1822623434 >> v1:172.21.15.104:6791/0 conn(0x7f2b8c0075f0 legacy=0x7f2b8c009ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.570+0000 7f2bb0656640 1 -- 172.21.15.26:0/1822623434 shutdown_connections 2024-08-19T12:43:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.570+0000 7f2bb0656640 1 -- 172.21.15.26:0/1822623434 >> 172.21.15.26:0/1822623434 conn(0x7f2ba80a8740 msgr2=0x7f2ba80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.570+0000 7f2bb0656640 1 -- 172.21.15.26:0/1822623434 shutdown_connections 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.570+0000 7f2bb0656640 1 -- 172.21.15.26:0/1822623434 wait complete. 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.571+0000 7f2bb0656640 10 monclient: build_initial_monmap 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.571+0000 7f2bb0656640 10 monclient: monmap: 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:16.585 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.571+0000 7f2bb0656640 1 librados: starting msgr at 2024-08-19T12:43:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.571+0000 7f2bb0656640 1 librados: starting objecter 2024-08-19T12:43:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.571+0000 7f2bb0656640 1 Processor -- start 2024-08-19T12:43:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 1 -- start start 2024-08-19T12:43:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 1 librados: setting wanted keys 2024-08-19T12:43:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 1 librados: calling monclient init 2024-08-19T12:43:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 10 monclient: init 2024-08-19T12:43:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 10 monclient: _add_conns ranks=[5,7,8,3,2,6,4,0,1] 2024-08-19T12:43:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 10 monclient(hunting): picked mon.noname-h con 0x7f2b8c0075f0 addr v1:172.21.15.104:6791/0 2024-08-19T12:43:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 10 monclient(hunting): picked mon.noname-i con 0x7f2ba809f7a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:43:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 10 monclient(hunting): picked mon.noname-e con 0x7f2ba81ceac0 addr v1:172.21.15.26:6793/0 2024-08-19T12:43:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ba8134e50 con 0x7f2b8c0075f0 2024-08-19T12:43:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ba80a2040 con 0x7f2ba809f7a0 2024-08-19T12:43:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ba81cef30 con 0x7f2ba81ceac0 2024-08-19T12:43:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.572+0000 7f2bb0656640 10 monclient(hunting): authenticate will time out at 990.659930s 2024-08-19T12:43:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.573+0000 7f2bae3f2640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f2b8c0075f0 0x7f2ba81ce3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:43338/0 (socket says 172.21.15.26:43338) 2024-08-19T12:43:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.573+0000 7f2bae3f2640 1 -- 172.21.15.26:0/97690055 learned_addr learned my addr 172.21.15.26:0/97690055 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:16.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2baebf3640 1 -- 172.21.15.26:0/97690055 >> v1:172.21.15.26:6793/0 conn(0x7f2ba81ceac0 legacy=0x7f2ba81d6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2badbf1640 1 -- 172.21.15.26:0/97690055 >> v1:172.21.15.104:6792/0 conn(0x7f2ba809f7a0 legacy=0x7f2ba81d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 client.?.objecter ms_handle_connect 0x7f2ba81ceac0 2024-08-19T12:43:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 client.?.objecter ms_handle_connect 0x7f2ba809f7a0 2024-08-19T12:43:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 client.?.objecter ms_handle_connect 0x7f2b8c0075f0 2024-08-19T12:43:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2bae3f2640 1 -- 172.21.15.26:0/97690055 >> v1:172.21.15.104:6791/0 conn(0x7f2b8c0075f0 legacy=0x7f2ba81ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 895806549 0 0) 0x7f2ba8136790 con 0x7f2b8c0075f0 2024-08-19T12:43:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 monclient(hunting): _negotiate 2024-08-19T12:43:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 monclient(hunting): my global_id is 26069 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b8c00b320 con 0x7f2b8c0075f0 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3391560192 0 0) 0x7f2ba81cef30 con 0x7f2ba81ceac0 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 monclient(hunting): _negotiate 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 monclient(hunting): my global_id is 25883 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ba8136790 con 0x7f2ba81ceac0 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1131343412 0 0) 0x7f2ba80a2040 con 0x7f2ba809f7a0 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 monclient(hunting): _negotiate 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 10 monclient(hunting): my global_id is 134917 2024-08-19T12:43:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ba81cef30 con 0x7f2ba809f7a0 2024-08-19T12:43:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.575+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 876855924 0 0) 0x7f2b8c00b320 con 0x7f2b8c0075f0 2024-08-19T12:43:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ba80a2040 con 0x7f2b8c0075f0 2024-08-19T12:43:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ba4001cc0 con 0x7f2b8c0075f0 2024-08-19T12:43:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:43:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient(hunting): dump: 2024-08-19T12:43:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.595 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:43:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 901011597 0 0) 0x7f2ba81cef30 con 0x7f2ba809f7a0 2024-08-19T12:43:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b8c00b320 con 0x7f2ba809f7a0 2024-08-19T12:43:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2366061792 0 0) 0x7f2ba8136790 con 0x7f2ba81ceac0 2024-08-19T12:43:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ba81cef30 con 0x7f2ba81ceac0 2024-08-19T12:43:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2b98004b20 con 0x7f2ba809f7a0 2024-08-19T12:43:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:43:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient(hunting): dump: 2024-08-19T12:43:16.595 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.596 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.596 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.597 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.598 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ba00020b0 con 0x7f2ba81ceac0 2024-08-19T12:43:16.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient(hunting): dump: 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.599 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.600 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1026098649 0 0) 0x7f2ba80a2040 con 0x7f2b8c0075f0 2024-08-19T12:43:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 >> v1:172.21.15.26:6793/0 conn(0x7f2ba81ceac0 legacy=0x7f2ba81d6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 >> v1:172.21.15.104:6792/0 conn(0x7f2ba809f7a0 legacy=0x7f2ba81d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient: _finish_hunting 0 2024-08-19T12:43:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 monclient: found mon.h 2024-08-19T12:43:16.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba80a0120 con 0x7f2b8c0075f0 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient: _finish_auth 0 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient: _check_auth_tickets 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2bb0656640 5 monclient: authenticate success, global_id 26069 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2bb0656640 10 monclient: _renew_subs 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2bb0656640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2bb0656640 1 -- 172.21.15.26:0/97690055 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba80a1d60 con 0x7f2b8c0075f0 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2bb0656640 10 client.26069.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2bb0656640 10 monclient: _renew_subs 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2bb0656640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2bb0656640 1 -- 172.21.15.26:0/97690055 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f2ba81d8240 con 0x7f2b8c0075f0 2024-08-19T12:43:16.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2bb0656640 1 librados: init done 2024-08-19T12:43:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ba4004a90 con 0x7f2b8c0075f0 2024-08-19T12:43:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ba4005000 con 0x7f2b8c0075f0 2024-08-19T12:43:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:43:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2bacbef640 4 set_mon_vals no callback set 2024-08-19T12:43:16.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.576+0000 7f2b9effd640 10 monclient: dump: 2024-08-19T12:43:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.577+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2ba4011460 con 0x7f2b8c0075f0 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.577+0000 7f2bb0656640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.577+0000 7f2bb0656640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.577+0000 7f2bb0656640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.577+0000 7f2bb0656640 1 -- 172.21.15.26:0/97690055 --> v1:172.21.15.104:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b90005080 con 0x7f2b8c0075f0 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.578+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(187..187 src has 1..187) ==== 5730+0+0 (unknown 4122717534 0 0) 0x7f2ba404e1c0 con 0x7f2b8c0075f0 2024-08-19T12:43:16.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.578+0000 7f2b9effd640 10 client.26069.objecter ms_dispatch 0x7f2ba8000b90 osd_map(187..187 src has 1..187) 2024-08-19T12:43:16.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.578+0000 7f2b9effd640 3 client.26069.objecter handle_osd_map got epochs [187,187] > 0 2024-08-19T12:43:16.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.578+0000 7f2b9effd640 3 client.26069.objecter handle_osd_map decoding full epoch 187 2024-08-19T12:43:16.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.578+0000 7f2badbf1640 10 monclient: get_auth_request con 0x7f2b8c042da0 auth_method 0 2024-08-19T12:43:16.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.578+0000 7f2b9effd640 20 client.26069.objecter dump_active .. 0 homeless 2024-08-19T12:43:16.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.580+0000 7f2b9effd640 10 client.26069.objecter ms_handle_connect 0x7f2b8c042da0 2024-08-19T12:43:16.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.583+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.5 v1:172.21.15.104:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f2ba8134e50 con 0x7f2b8c0075f0 2024-08-19T12:43:16.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.583+0000 7f2b9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:16.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.583+0000 7f2b9effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:43:16.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.843+0000 7f2bb0656640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:43:16.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.843+0000 7f2bb0656640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:43:16.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.843+0000 7f2bb0656640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:43:16.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.843+0000 7f2bb0656640 1 -- 172.21.15.26:0/97690055 --> v1:172.21.15.104:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2b90005370 con 0x7f2b8c0075f0 2024-08-19T12:43:16.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.844+0000 7f2b9effd640 1 -- 172.21.15.26:0/97690055 <== mon.5 v1:172.21.15.104:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2723 (unknown 254746657 0 892428615) 0x7f2ba40173e0 con 0x7f2b8c0075f0 2024-08-19T12:43:16.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.844+0000 7f2b9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:43:16.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.844+0000 7f2b9effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:43:16.845 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":28,"quorum":[0,1,2,4,5,6,7,8],"quorum_names":["a","f","b","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:43:16.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 10 librados: watch_flush enter 2024-08-19T12:43:16.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 10 librados: watch_flush exit 2024-08-19T12:43:16.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 20 client.26069.objecter shutdown clearing up homeless session... 2024-08-19T12:43:16.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 10 client.26069.objecter successfully canceled tick 2024-08-19T12:43:16.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 1 -- 172.21.15.26:0/97690055 >> v1:172.21.15.104:6800/899887506 conn(0x7f2b8c042da0 legacy=0x7f2b8c045260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:16.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 10 monclient: shutdown 2024-08-19T12:43:16.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:16.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 1 -- 172.21.15.26:0/97690055 >> v1:172.21.15.104:6791/0 conn(0x7f2b8c0075f0 legacy=0x7f2ba81ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:16.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 1 -- 172.21.15.26:0/97690055 shutdown_connections 2024-08-19T12:43:16.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 1 -- 172.21.15.26:0/97690055 >> 172.21.15.26:0/97690055 conn(0x7f2ba80a8740 msgr2=0x7f2ba8132cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:16.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 1 -- 172.21.15.26:0/97690055 shutdown_connections 2024-08-19T12:43:16.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 1 -- 172.21.15.26:0/97690055 wait complete. 2024-08-19T12:43:16.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.846+0000 7f2bb0656640 1 librados: shutdown 2024-08-19T12:43:16.857 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-08-19T12:43:16.858 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-08-19T12:43:16.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:43:16.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.956+0000 7fb081122640 10 monclient: get_monmap_and_config 2024-08-19T12:43:16.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.956+0000 7fb081122640 10 monclient: build_initial_monmap 2024-08-19T12:43:16.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.956+0000 7fb081122640 10 monclient: monmap: 2024-08-19T12:43:16.957 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:16.957 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:43:16.957351+0000 2024-08-19T12:43:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:43:16.957351+0000 2024-08-19T12:43:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:16.958 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:16.959 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:16.959 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 1 Processor -- start 2024-08-19T12:43:16.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 1 -- start start 2024-08-19T12:43:16.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 10 monclient: init 2024-08-19T12:43:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 10 monclient: _add_conns ranks=[6,8,3,0,5,2,4,7,1] 2024-08-19T12:43:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 10 monclient(hunting): picked mon.noname-d con 0x7fb07c06e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:43:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 10 monclient(hunting): picked mon.noname-e con 0x7fb07c0a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:43:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 10 monclient(hunting): picked mon.noname-g con 0x7fb07c0a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:43:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07c136790 con 0x7fb07c0a5d50 2024-08-19T12:43:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07c09ddf0 con 0x7fb07c06e7b0 2024-08-19T12:43:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07c09e0d0 con 0x7fb07c0a5920 2024-08-19T12:43:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:16.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb081122640 10 monclient(hunting): authenticate will time out at 991.044647s 2024-08-19T12:43:16.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb07a59c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fb07c0a5920 0x7fb07c13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:58560/0 (socket says 172.21.15.26:58560) 2024-08-19T12:43:16.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb07a59c640 1 -- 172.21.15.26:0/4227802680 learned_addr learned my addr 172.21.15.26:0/4227802680 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:16.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb07ad9d640 1 --1- 172.21.15.26:0/4227802680 >> v1:172.21.15.26:6792/0 conn(0x7fb07c06e7b0 0x7fb07c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49582/0 (socket says 172.21.15.26:49582) 2024-08-19T12:43:16.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.957+0000 7fb07b59e640 1 -- 172.21.15.26:0/4227802680 >> v1:172.21.15.104:6790/0 conn(0x7fb07c0a5d50 legacy=0x7fb07c141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:43:16.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.959+0000 7fb07a59c640 1 -- 172.21.15.26:0/4227802680 >> v1:172.21.15.26:6793/0 conn(0x7fb07c0a5920 legacy=0x7fb07c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:16.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.959+0000 7fb07ad9d640 1 -- 172.21.15.26:0/4227802680 >> v1:172.21.15.26:6792/0 conn(0x7fb07c06e7b0 legacy=0x7fb07c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:16.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 1 -- 172.21.15.26:0/4227802680 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298813766 0 0) 0x7fb07c09e0d0 con 0x7fb07c0a5920 2024-08-19T12:43:16.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:43:16.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:16.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:16.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient(hunting): my global_id is 25892 2024-08-19T12:43:16.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 1 -- 172.21.15.26:0/4227802680 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb068003520 con 0x7fb07c0a5920 2024-08-19T12:43:16.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 1 -- 172.21.15.26:0/4227802680 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3827002617 0 0) 0x7fb06c0017d0 con 0x7fb07c06e7b0 2024-08-19T12:43:16.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:43:16.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:16.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:16.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient(hunting): my global_id is 26349 2024-08-19T12:43:16.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 1 -- 172.21.15.26:0/4227802680 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb07c09e0d0 con 0x7fb07c06e7b0 2024-08-19T12:43:16.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 1 -- 172.21.15.26:0/4227802680 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1903240662 0 0) 0x7fb068003520 con 0x7fb07c0a5920 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 1 -- 172.21.15.26:0/4227802680 >> v1:172.21.15.26:6792/0 conn(0x7fb07c06e7b0 legacy=0x7fb07c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 1 -- 172.21.15.26:0/4227802680 >> v1:172.21.15.104:6790/0 conn(0x7fb07c0a5d50 legacy=0x7fb07c141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 1 monclient: found mon.noname-e 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 1 -- 172.21.15.26:0/4227802680 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07c09e3b0 con 0x7fb07c0a5920 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient: _finish_auth 0 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb081122640 5 monclient: authenticate success, global_id 25892 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 1 -- 172.21.15.26:0/4227802680 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb064001ae0 con 0x7fb07c0a5920 2024-08-19T12:43:16.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:43:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient: dump: 2024-08-19T12:43:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.970 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.970 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.970 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.970 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.970 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.970 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.970 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb081122640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:16.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 1 -- 172.21.15.26:0/4227802680 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb0640048a0 con 0x7fb07c0a5920 2024-08-19T12:43:16.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:16.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 1 -- 172.21.15.26:0/4227802680 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb064004e10 con 0x7fb07c0a5920 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb081122640 10 monclient: get_monmap_and_config success 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb081122640 4 set_mon_vals no callback set 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb079d9b640 10 monclient: dump: 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.971 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.972 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.972 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.972 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.972 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.972 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.972 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb081122640 10 monclient: shutdown 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb081122640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.960+0000 7fb081122640 1 -- 172.21.15.26:0/4227802680 >> v1:172.21.15.26:6793/0 conn(0x7fb07c0a5920 legacy=0x7fb07c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.961+0000 7fb081122640 1 -- 172.21.15.26:0/4227802680 shutdown_connections 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.961+0000 7fb081122640 1 -- 172.21.15.26:0/4227802680 >> 172.21.15.26:0/4227802680 conn(0x7fb07c0a8740 msgr2=0x7fb07c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.961+0000 7fb081122640 1 -- 172.21.15.26:0/4227802680 shutdown_connections 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.961+0000 7fb081122640 1 -- 172.21.15.26:0/4227802680 wait complete. 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.961+0000 7fb081122640 10 monclient: build_initial_monmap 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.961+0000 7fb081122640 10 monclient: monmap: 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:43:16.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:16.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:16.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:16.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:16.974 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:16.974 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:16.974 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:16.974 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:16.974 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:16.974 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:16.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.961+0000 7fb081122640 1 librados: starting msgr at 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.961+0000 7fb081122640 1 librados: starting objecter 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 1 Processor -- start 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 1 -- start start 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 1 librados: setting wanted keys 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 1 librados: calling monclient init 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 10 monclient: init 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 10 monclient: _add_conns ranks=[7,3,1,2,0,6,4,5,8] 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 10 monclient(hunting): picked mon.noname-i con 0x7fb07c06e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 10 monclient(hunting): picked mon.noname-g con 0x7fb07c0a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 10 monclient(hunting): picked mon.noname-f con 0x7fb07c0a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07c1345d0 con 0x7fb07c0a5d50 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07c125bd0 con 0x7fb07c0a5920 2024-08-19T12:43:16.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07c125330 con 0x7fb07c06e7b0 2024-08-19T12:43:16.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:16.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb081122640 10 monclient(hunting): authenticate will time out at 991.049828s 2024-08-19T12:43:16.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.962+0000 7fb07a59c640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7fb07c0a5920 legacy=0x7fb07c122c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:43:16.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.963+0000 7fb07b59e640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb07c0a5d50 0x7fb07c1d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:53842/0 (socket says 172.21.15.26:53842) 2024-08-19T12:43:16.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.963+0000 7fb07ad9d640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7fb07c06e7b0 0x7fb07c122500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:53044/0 (socket says 172.21.15.26:53044) 2024-08-19T12:43:16.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.963+0000 7fb07b59e640 1 -- 172.21.15.26:0/2381000611 learned_addr learned my addr 172.21.15.26:0/2381000611 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:16.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb07ad9d640 1 -- 172.21.15.26:0/2381000611 >> v1:172.21.15.104:6792/0 conn(0x7fb07c06e7b0 legacy=0x7fb07c122500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:16.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 10 client.?.objecter ms_handle_connect 0x7fb07c06e7b0 2024-08-19T12:43:16.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:16.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 10 client.?.objecter ms_handle_connect 0x7fb07c0a5d50 2024-08-19T12:43:16.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:16.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb07b59e640 1 -- 172.21.15.26:0/2381000611 >> v1:172.21.15.104:6789/0 conn(0x7fb07c0a5d50 legacy=0x7fb07c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:16.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056847695 0 0) 0x7fb07c125330 con 0x7fb07c06e7b0 2024-08-19T12:43:16.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 10 monclient(hunting): _negotiate 2024-08-19T12:43:16.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 10 monclient(hunting): my global_id is 134926 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb05c004eb0 con 0x7fb07c06e7b0 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2506916590 0 0) 0x7fb074003d30 con 0x7fb07c0a5d50 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 10 monclient(hunting): _negotiate 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 10 monclient(hunting): my global_id is 26785 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.965+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb07c125330 con 0x7fb07c0a5d50 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2120600145 0 0) 0x7fb05c004eb0 con 0x7fb07c06e7b0 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb074003d30 con 0x7fb07c06e7b0 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb06c0017f0 con 0x7fb07c06e7b0 2024-08-19T12:43:16.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:16.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:43:16.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient(hunting): dump: 2024-08-19T12:43:16.985 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.985 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.985 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.986 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.986 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.986 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.986 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.986 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.986 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.986 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:43:16.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 432037258 0 0) 0x7fb07c125330 con 0x7fb07c0a5d50 2024-08-19T12:43:16.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb05c004eb0 con 0x7fb07c0a5d50 2024-08-19T12:43:16.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb074004710 con 0x7fb07c0a5d50 2024-08-19T12:43:16.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient(hunting): dump: 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.987 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1648016249 0 0) 0x7fb074003d30 con 0x7fb07c06e7b0 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 >> v1:172.21.15.104:6790/0 conn(0x7fb07c0a5920 legacy=0x7fb07c122c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 >> v1:172.21.15.104:6789/0 conn(0x7fb07c0a5d50 legacy=0x7fb07c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient: _finish_hunting 0 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 monclient: found mon.i 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07c125610 con 0x7fb07c06e7b0 2024-08-19T12:43:16.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient: _finish_auth 0 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient: _check_auth_tickets 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb081122640 5 monclient: authenticate success, global_id 134926 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb081122640 10 monclient: _renew_subs 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb081122640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb081122640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb07c124f90 con 0x7fb07c06e7b0 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb081122640 10 client.134926.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb081122640 10 monclient: _renew_subs 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb081122640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb081122640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fb07c123320 con 0x7fb07c06e7b0 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb081122640 1 librados: init done 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb06c004f30 con 0x7fb07c06e7b0 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb06c0054a0 con 0x7fb07c06e7b0 2024-08-19T12:43:16.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:16.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:43:16.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb07959a640 4 set_mon_vals no callback set 2024-08-19T12:43:16.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.966+0000 7fb0737fe640 10 monclient: dump: 2024-08-19T12:43:16.990 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.990 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.990 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.990 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.990 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.990 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.990 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.990 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.990 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.991 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.991 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.991 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.991 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.991 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.991 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.991 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.967+0000 7fb081122640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:16.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.967+0000 7fb081122640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb0600065c0 con 0x7fb060002860 2024-08-19T12:43:16.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.967+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fb06c005640 con 0x7fb07c06e7b0 2024-08-19T12:43:16.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.967+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(187..187 src has 1..187) ==== 5730+0+0 (unknown 4122717534 0 0) 0x7fb06c04e750 con 0x7fb07c06e7b0 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.967+0000 7fb0737fe640 10 client.134926.objecter ms_dispatch 0x7fb07c000b90 osd_map(187..187 src has 1..187) 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.967+0000 7fb0737fe640 3 client.134926.objecter handle_osd_map got epochs [187,187] > 0 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.967+0000 7fb0737fe640 3 client.134926.objecter handle_osd_map decoding full epoch 187 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.967+0000 7fb0737fe640 20 client.134926.objecter dump_active .. 0 homeless 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.967+0000 7fb07b59e640 10 monclient: get_auth_request con 0x7fb05c041470 auth_method 0 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 10 client.134926.objecter ms_handle_connect 0x7fb060002860 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 10 client.134926.objecter resend_mon_ops 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb07a59c640 1 -- 172.21.15.26:0/2381000611 >> v1:172.21.15.26:6789/0 conn(0x7fb060002860 legacy=0x7fb060004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 462643090 0 0) 0x7fb0600065c0 con 0x7fb060002860 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 10 monclient(hunting): _negotiate 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 10 monclient(hunting): my global_id is 144306 2024-08-19T12:43:16.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb074003d30 con 0x7fb060002860 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3552496884 0 0) 0x7fb074003d30 con 0x7fb060002860 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0600065c0 con 0x7fb060002860 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb064004dd0 con 0x7fb060002860 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 10 monclient: dump: 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.993 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.994 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.994 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.994 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.994 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.994 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.994 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.994 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.994 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.994 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2730918546 0 0) 0x7fb0600065c0 con 0x7fb060002860 2024-08-19T12:43:16.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.969+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb06000a160 con 0x7fb060002860 2024-08-19T12:43:16.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.970+0000 7fb0737fe640 10 client.134926.objecter ms_handle_connect 0x7fb05c041470 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.970+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb064004dd0 con 0x7fb060002860 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.970+0000 7fb0737fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.970+0000 7fb0737fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.970+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 >> v1:172.21.15.26:6789/0 conn(0x7fb060002860 legacy=0x7fb060004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.970+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 >> v1:172.21.15.26:6789/0 conn(0x7fb060002860 legacy=0x7fb060004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.976+0000 7fb081122640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.976+0000 7fb081122640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb060009e50 con 0x7fb060009110 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.978+0000 7fb07a59c640 1 -- 172.21.15.26:0/2381000611 >> v1:172.21.15.26:6789/0 conn(0x7fb060009110 legacy=0x7fb0600095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.978+0000 7fb0737fe640 10 client.134926.objecter ms_handle_connect 0x7fb060009110 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.978+0000 7fb0737fe640 10 client.134926.objecter resend_mon_ops 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2898345054 0 0) 0x7fb060009e50 con 0x7fb060009110 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 10 monclient(hunting): _negotiate 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:16.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 10 monclient(hunting): my global_id is 144315 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0600065c0 con 0x7fb060009110 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2700253996 0 0) 0x7fb0600065c0 con 0x7fb060009110 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb060009e50 con 0x7fb060009110 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb064005150 con 0x7fb060009110 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 10 monclient: dump: 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:16.996 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4241195210 0 0) 0x7fb060009e50 con 0x7fb060009110 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb06000c3f0 con 0x7fb060009110 2024-08-19T12:43:16.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7701 (unknown 0 0 2237630917) 0x7fb064007e20 con 0x7fb060009110 2024-08-19T12:43:16.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:43:16.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:43:16.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 >> v1:172.21.15.26:6789/0 conn(0x7fb060009110 legacy=0x7fb0600095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:16.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb0737fe640 1 -- 172.21.15.26:0/2381000611 >> v1:172.21.15.26:6789/0 conn(0x7fb060009110 legacy=0x7fb0600095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:16.998 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:43:16.998 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:43:16.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:43:16.998 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:43:16.998 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 28, 2024-08-19T12:43:16.998 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 48897, 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:16.999 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:43:17.000 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:43:17.000 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.000 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.000 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.000 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.000 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.000 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.000 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.000 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.000 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.000 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.000 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.001 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.001 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:43:17.001 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:43:17.001 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.001 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.001 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.001 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.001 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.001 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.001 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.001 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.001 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.002 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:43:17.004 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:43:17.005 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:43:17.005 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:43:17.005 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:43:17.005 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.007 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:43:17.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:43:17.007 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.007 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.007 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:43:17.007 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:43:17.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.008 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:43:17.009 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:43:17.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:43:17.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:43:17.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:43:17.015 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:43:17.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:43:17.015 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.015 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.016 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:43:17.016 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:43:17.016 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:43:17.016 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.017 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:43:17.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:43:17.017 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.017 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.017 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.017 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.018 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:43:17.018 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:43:17.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.018 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.018 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.018 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.018 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.018 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.018 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:43:17.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.018 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:43:17.019 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:43:17.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.979+0000 7fb081122640 10 librados: watch_flush enter 2024-08-19T12:43:17.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb081122640 10 librados: watch_flush exit 2024-08-19T12:43:17.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb081122640 20 client.134926.objecter shutdown clearing up homeless session... 2024-08-19T12:43:17.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb081122640 10 client.134926.objecter successfully canceled tick 2024-08-19T12:43:17.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb081122640 1 -- 172.21.15.26:0/2381000611 >> v1:172.21.15.104:6800/899887506 conn(0x7fb05c041470 legacy=0x7fb05c043930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb081122640 10 monclient: shutdown 2024-08-19T12:43:17.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb081122640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:17.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb081122640 1 -- 172.21.15.26:0/2381000611 >> v1:172.21.15.104:6792/0 conn(0x7fb07c06e7b0 legacy=0x7fb07c122500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb07b59e640 1 -- 172.21.15.26:0/2381000611 reap_dead start 2024-08-19T12:43:17.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb07b59e640 1 -- 172.21.15.26:0/2381000611 reap_dead start 2024-08-19T12:43:17.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb081122640 1 -- 172.21.15.26:0/2381000611 shutdown_connections 2024-08-19T12:43:17.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb081122640 1 -- 172.21.15.26:0/2381000611 >> 172.21.15.26:0/2381000611 conn(0x7fb07c0a8740 msgr2=0x7fb07c132e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:17.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb081122640 1 -- 172.21.15.26:0/2381000611 shutdown_connections 2024-08-19T12:43:17.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb081122640 1 -- 172.21.15.26:0/2381000611 wait complete. 2024-08-19T12:43:17.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:16.980+0000 7fb081122640 1 librados: shutdown 2024-08-19T12:43:17.023 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:43:17.023 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-08-19T12:43:17.023 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:43:17.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.122+0000 7f9de8a15640 10 monclient: get_monmap_and_config 2024-08-19T12:43:17.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.122+0000 7f9de8a15640 10 monclient: build_initial_monmap 2024-08-19T12:43:17.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.122+0000 7f9de8a15640 10 monclient: monmap: 2024-08-19T12:43:17.123 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:17.123 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.123 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:43:17.123109+0000 2024-08-19T12:43:17.123 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:43:17.123109+0000 2024-08-19T12:43:17.123 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:17.123 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.124 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:17.124 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:17.124 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:17.124 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:17.124 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:17.124 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:17.124 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:17.124 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:17.124 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:17.124 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.122+0000 7f9de8a15640 1 Processor -- start 2024-08-19T12:43:17.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.122+0000 7f9de8a15640 1 -- start start 2024-08-19T12:43:17.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.122+0000 7f9de8a15640 10 monclient: init 2024-08-19T12:43:17.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de8a15640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:17.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de8a15640 10 monclient: _add_conns ranks=[0,6,7,1,3,8,5,2,4] 2024-08-19T12:43:17.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de8a15640 10 monclient(hunting): picked mon.noname-a con 0x7f9de406e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:43:17.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de8a15640 10 monclient(hunting): picked mon.noname-d con 0x7f9de40a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:43:17.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de8a15640 10 monclient(hunting): picked mon.noname-i con 0x7f9de40a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:43:17.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de8a15640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de4136790 con 0x7f9de406e7b0 2024-08-19T12:43:17.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de8a15640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de409ddf0 con 0x7f9de40a5920 2024-08-19T12:43:17.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de8a15640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de409e0d0 con 0x7f9de40a5d50 2024-08-19T12:43:17.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de8a15640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de8a15640 10 monclient(hunting): authenticate will time out at 991.210551s 2024-08-19T12:43:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de1d9b640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f9de40a5920 0x7f9de413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49584/0 (socket says 172.21.15.26:49584) 2024-08-19T12:43:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9de406e7b0 0x7f9de4136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58116/0 (socket says 172.21.15.26:58116) 2024-08-19T12:43:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.123+0000 7f9de1d9b640 1 -- 172.21.15.26:0/1528227006 learned_addr learned my addr 172.21.15.26:0/1528227006 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.125+0000 7f9de259c640 1 -- 172.21.15.26:0/1528227006 >> v1:172.21.15.26:6789/0 conn(0x7f9de406e7b0 legacy=0x7f9de4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.125+0000 7f9de1d9b640 1 -- 172.21.15.26:0/1528227006 >> v1:172.21.15.26:6792/0 conn(0x7f9de40a5920 legacy=0x7f9de413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 1 -- 172.21.15.26:0/1528227006 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 572191890 0 0) 0x7f9dd8001450 con 0x7f9de40a5920 2024-08-19T12:43:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient(hunting): my global_id is 26358 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 1 -- 172.21.15.26:0/1528227006 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dcc003520 con 0x7f9de40a5920 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 1 -- 172.21.15.26:0/1528227006 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2656919265 0 0) 0x7f9de4136790 con 0x7f9de406e7b0 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de2d9d640 1 -- 172.21.15.26:0/1528227006 >> v1:172.21.15.104:6792/0 conn(0x7f9de40a5d50 legacy=0x7f9de4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient(hunting): my global_id is 144324 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 1 -- 172.21.15.26:0/1528227006 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dd8001450 con 0x7f9de406e7b0 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 1 -- 172.21.15.26:0/1528227006 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1504865826 0 0) 0x7f9dd8001450 con 0x7f9de406e7b0 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 1 -- 172.21.15.26:0/1528227006 >> v1:172.21.15.104:6792/0 conn(0x7f9de40a5d50 legacy=0x7f9de4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 1 -- 172.21.15.26:0/1528227006 >> v1:172.21.15.26:6792/0 conn(0x7f9de40a5920 legacy=0x7f9de413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient: _finish_hunting 0 2024-08-19T12:43:17.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 1 monclient: found mon.noname-a 2024-08-19T12:43:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:43:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 1 -- 172.21.15.26:0/1528227006 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de409e3b0 con 0x7f9de406e7b0 2024-08-19T12:43:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient: _finish_auth 0 2024-08-19T12:43:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient: _check_auth_tickets 2024-08-19T12:43:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 1 -- 172.21.15.26:0/1528227006 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9dd4001ac0 con 0x7f9de406e7b0 2024-08-19T12:43:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:43:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient: dump: 2024-08-19T12:43:17.134 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.135 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.135 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.135 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.135 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.135 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.135 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.135 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.135 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.135 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.135 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.135 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.135 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de8a15640 5 monclient: authenticate success, global_id 144324 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de8a15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 1 -- 172.21.15.26:0/1528227006 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9dd40047e0 con 0x7f9de406e7b0 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de8a15640 10 monclient: get_monmap_and_config success 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 1 -- 172.21.15.26:0/1528227006 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9dd4004d50 con 0x7f9de406e7b0 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de8a15640 4 set_mon_vals no callback set 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de159a640 10 monclient: dump: 2024-08-19T12:43:17.136 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.137 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de8a15640 10 monclient: shutdown 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de8a15640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.126+0000 7f9de8a15640 1 -- 172.21.15.26:0/1528227006 >> v1:172.21.15.26:6789/0 conn(0x7f9de406e7b0 legacy=0x7f9de4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.127+0000 7f9de8a15640 1 -- 172.21.15.26:0/1528227006 shutdown_connections 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.127+0000 7f9de8a15640 1 -- 172.21.15.26:0/1528227006 >> 172.21.15.26:0/1528227006 conn(0x7f9de40a8740 msgr2=0x7f9de40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.127+0000 7f9de8a15640 1 -- 172.21.15.26:0/1528227006 shutdown_connections 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.127+0000 7f9de8a15640 1 -- 172.21.15.26:0/1528227006 wait complete. 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.127+0000 7f9de8a15640 10 monclient: build_initial_monmap 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.127+0000 7f9de8a15640 10 monclient: monmap: 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:43:17.138 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.127+0000 7f9de8a15640 1 librados: starting msgr at 2024-08-19T12:43:17.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.127+0000 7f9de8a15640 1 librados: starting objecter 2024-08-19T12:43:17.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 1 Processor -- start 2024-08-19T12:43:17.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 1 -- start start 2024-08-19T12:43:17.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 1 librados: setting wanted keys 2024-08-19T12:43:17.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 1 librados: calling monclient init 2024-08-19T12:43:17.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 10 monclient: init 2024-08-19T12:43:17.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:17.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 10 monclient: _add_conns ranks=[0,2,5,3,4,1,6,7,8] 2024-08-19T12:43:17.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 10 monclient(hunting): picked mon.noname-a con 0x7f9de406e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:43:17.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 10 monclient(hunting): picked mon.noname-b con 0x7f9de40a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:43:17.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 10 monclient(hunting): picked mon.noname-h con 0x7f9de40a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:43:17.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de4142310 con 0x7f9de406e7b0 2024-08-19T12:43:17.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de41cf060 con 0x7f9de40a5920 2024-08-19T12:43:17.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de41349a0 con 0x7f9de40a5d50 2024-08-19T12:43:17.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:17.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de8a15640 10 monclient(hunting): authenticate will time out at 991.215664s 2024-08-19T12:43:17.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9de406e7b0 0x7f9de41ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58130/0 (socket says 172.21.15.26:58130) 2024-08-19T12:43:17.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.128+0000 7f9de259c640 1 -- 172.21.15.26:0/1431403583 learned_addr learned my addr 172.21.15.26:0/1431403583 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:17.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.130+0000 7f9de259c640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.26:6789/0 conn(0x7f9de406e7b0 legacy=0x7f9de41ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.130+0000 7f9de1d9b640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.26:6790/0 conn(0x7f9de40a5920 legacy=0x7f9de41ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.130+0000 7f9dd2ffd640 10 client.?.objecter ms_handle_connect 0x7f9de406e7b0 2024-08-19T12:43:17.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.130+0000 7f9dd2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.130+0000 7f9dd2ffd640 10 client.?.objecter ms_handle_connect 0x7f9de40a5920 2024-08-19T12:43:17.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.130+0000 7f9dd2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.130+0000 7f9dd2ffd640 10 client.?.objecter ms_handle_connect 0x7f9de40a5d50 2024-08-19T12:43:17.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.130+0000 7f9de2d9d640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.104:6791/0 conn(0x7f9de40a5d50 legacy=0x7f9de41d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.130+0000 7f9dd2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.130+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 795757353 0 0) 0x7f9de4142310 con 0x7f9de406e7b0 2024-08-19T12:43:17.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): my global_id is 144333 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dc4004eb0 con 0x7f9de406e7b0 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286896959 0 0) 0x7f9de41cf060 con 0x7f9de40a5920 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): my global_id is 26084 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9de4142310 con 0x7f9de40a5920 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1461277489 0 0) 0x7f9de41349a0 con 0x7f9de40a5d50 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): my global_id is 26078 2024-08-19T12:43:17.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9de41cf060 con 0x7f9de40a5d50 2024-08-19T12:43:17.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2348608034 0 0) 0x7f9dc4004eb0 con 0x7f9de406e7b0 2024-08-19T12:43:17.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9de41349a0 con 0x7f9de406e7b0 2024-08-19T12:43:17.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9dd4004cd0 con 0x7f9de406e7b0 2024-08-19T12:43:17.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:17.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:43:17.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): dump: 2024-08-19T12:43:17.150 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.150 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.150 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 830955389 0 0) 0x7f9de4142310 con 0x7f9de40a5920 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9dc4004eb0 con 0x7f9de40a5920 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9dd8001b10 con 0x7f9de40a5920 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient(hunting): dump: 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.152 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3099297921 0 0) 0x7f9de41349a0 con 0x7f9de406e7b0 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.104:6791/0 conn(0x7f9de40a5d50 legacy=0x7f9de41d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.26:6790/0 conn(0x7f9de40a5920 legacy=0x7f9de41ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 monclient: found mon.a 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de413b3b0 con 0x7f9de406e7b0 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient: _finish_auth 0 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9de8a15640 5 monclient: authenticate success, global_id 144333 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9de8a15640 10 monclient: _renew_subs 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9de8a15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9de8a15640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9de413ab10 con 0x7f9de406e7b0 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9de8a15640 10 client.144333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:43:17.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9de8a15640 10 monclient: _renew_subs 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9de8a15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9de8a15640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9de4138bc0 con 0x7f9de406e7b0 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9de8a15640 1 librados: init done 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9dd4001d10 con 0x7f9de406e7b0 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9dd40047e0 con 0x7f9de406e7b0 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9de0d99640 4 set_mon_vals no callback set 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.131+0000 7f9dd2ffd640 10 monclient: dump: 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.155 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.156 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.132+0000 7f9de8a15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.132+0000 7f9de8a15640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9dc80065c0 con 0x7f9dc8002860 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.132+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f9dd4011940 con 0x7f9de406e7b0 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.132+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(187..187 src has 1..187) ==== 5730+0+0 (unknown 4122717534 0 0) 0x7f9dd404e3e0 con 0x7f9de406e7b0 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.132+0000 7f9dd2ffd640 10 client.144333.objecter ms_dispatch 0x7f9de4000b90 osd_map(187..187 src has 1..187) 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.132+0000 7f9dd2ffd640 3 client.144333.objecter handle_osd_map got epochs [187,187] > 0 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.132+0000 7f9dd2ffd640 3 client.144333.objecter handle_osd_map decoding full epoch 187 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.132+0000 7f9dd2ffd640 20 client.144333.objecter dump_active .. 0 homeless 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.132+0000 7f9de2d9d640 10 monclient: get_auth_request con 0x7f9dc4041860 auth_method 0 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 10 client.144333.objecter ms_handle_connect 0x7f9dc8002860 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9de1d9b640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.26:6790/0 conn(0x7f9dc8002860 legacy=0x7f9dc8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 10 client.144333.objecter resend_mon_ops 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1761135222 0 0) 0x7f9dc80065c0 con 0x7f9dc8002860 2024-08-19T12:43:17.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 10 monclient(hunting): my global_id is 26093 2024-08-19T12:43:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9de41349a0 con 0x7f9dc8002860 2024-08-19T12:43:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1618849152 0 0) 0x7f9de41349a0 con 0x7f9dc8002860 2024-08-19T12:43:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9dc80065c0 con 0x7f9dc8002860 2024-08-19T12:43:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9dd8005210 con 0x7f9dc8002860 2024-08-19T12:43:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:43:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.134+0000 7f9dd2ffd640 10 monclient: dump: 2024-08-19T12:43:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.159 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.135+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1988665152 0 0) 0x7f9dc80065c0 con 0x7f9dc8002860 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.135+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9dc800a160 con 0x7f9dc8002860 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.135+0000 7f9dd2ffd640 10 client.144333.objecter ms_handle_connect 0x7f9dc4041860 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.135+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9dd8005410 con 0x7f9dc8002860 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.135+0000 7f9dd2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.135+0000 7f9dd2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.135+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.26:6790/0 conn(0x7f9dc8002860 legacy=0x7f9dc8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.135+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.26:6790/0 conn(0x7f9dc8002860 legacy=0x7f9dc8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.141+0000 7f9de8a15640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.141+0000 7f9de8a15640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9dc8009e70 con 0x7f9dc8009130 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.143+0000 7f9de1d9b640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.26:6790/0 conn(0x7f9dc8009130 legacy=0x7f9dc80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.143+0000 7f9dd2ffd640 10 client.144333.objecter ms_handle_connect 0x7f9dc8009130 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.143+0000 7f9dd2ffd640 10 client.144333.objecter resend_mon_ops 2024-08-19T12:43:17.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1520294514 0 0) 0x7f9dc8009e70 con 0x7f9dc8009130 2024-08-19T12:43:17.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 10 monclient(hunting): my global_id is 26102 2024-08-19T12:43:17.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dc80065c0 con 0x7f9dc8009130 2024-08-19T12:43:17.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3625273483 0 0) 0x7f9dc80065c0 con 0x7f9dc8009130 2024-08-19T12:43:17.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9dc8009e70 con 0x7f9dc8009130 2024-08-19T12:43:17.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9dd8005c40 con 0x7f9dc8009130 2024-08-19T12:43:17.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:43:17.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 10 monclient: dump: 2024-08-19T12:43:17.161 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.162 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.163 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.163 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 247162623 0 0) 0x7f9dc8009e70 con 0x7f9dc8009130 2024-08-19T12:43:17.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9dc800c410 con 0x7f9dc8009130 2024-08-19T12:43:17.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 1331727335) 0x7f9dd8006050 con 0x7f9dc8009130 2024-08-19T12:43:17.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:43:17.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:43:17.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.26:6790/0 conn(0x7f9dc8009130 legacy=0x7f9dc80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9dd2ffd640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.26:6790/0 conn(0x7f9dc8009130 legacy=0x7f9dc80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:17.163 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 28, 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 310199, 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:43:17.164 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.165 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.166 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.166 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.166 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.166 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.166 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.166 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:43:17.166 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:43:17.166 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.166 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.166 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.166 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.166 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:43:17.167 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:43:17.168 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:43:17.168 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.168 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.168 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:43:17.168 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:43:17.168 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:43:17.168 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:43:17.168 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:43:17.168 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:43:17.168 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:43:17.168 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.168 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.169 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:43:17.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.170 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:43:17.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.170 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:43:17.170 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:43:17.170 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:43:17.170 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:43:17.172 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.172 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:43:17.172 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:43:17.172 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.172 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.172 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:43:17.175 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.176 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:43:17.179 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.180 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.181 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.181 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:43:17.181 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:43:17.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.181 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:43:17.181 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:43:17.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.183 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:43:17.184 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:43:17.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9de8a15640 10 librados: watch_flush enter 2024-08-19T12:43:17.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9de8a15640 10 librados: watch_flush exit 2024-08-19T12:43:17.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9de8a15640 20 client.144333.objecter shutdown clearing up homeless session... 2024-08-19T12:43:17.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.144+0000 7f9de8a15640 10 client.144333.objecter successfully canceled tick 2024-08-19T12:43:17.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.145+0000 7f9de8a15640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.104:6800/899887506 conn(0x7f9dc4041860 legacy=0x7f9dc4043d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.145+0000 7f9de8a15640 10 monclient: shutdown 2024-08-19T12:43:17.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.145+0000 7f9de8a15640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:17.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.145+0000 7f9de8a15640 1 -- 172.21.15.26:0/1431403583 >> v1:172.21.15.26:6789/0 conn(0x7f9de406e7b0 legacy=0x7f9de41ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.145+0000 7f9de2d9d640 1 -- 172.21.15.26:0/1431403583 reap_dead start 2024-08-19T12:43:17.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.145+0000 7f9de2d9d640 1 -- 172.21.15.26:0/1431403583 reap_dead start 2024-08-19T12:43:17.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.145+0000 7f9de8a15640 1 -- 172.21.15.26:0/1431403583 shutdown_connections 2024-08-19T12:43:17.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.145+0000 7f9de8a15640 1 -- 172.21.15.26:0/1431403583 >> 172.21.15.26:0/1431403583 conn(0x7f9de40a8740 msgr2=0x7f9de406e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:17.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.145+0000 7f9de8a15640 1 -- 172.21.15.26:0/1431403583 shutdown_connections 2024-08-19T12:43:17.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.145+0000 7f9de8a15640 1 -- 172.21.15.26:0/1431403583 wait complete. 2024-08-19T12:43:17.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.145+0000 7f9de8a15640 1 librados: shutdown 2024-08-19T12:43:17.187 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:43:17.187 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-08-19T12:43:17.187 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:43:17.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.286+0000 7f49a3b3c640 10 monclient: get_monmap_and_config 2024-08-19T12:43:17.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.286+0000 7f49a3b3c640 10 monclient: build_initial_monmap 2024-08-19T12:43:17.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.286+0000 7f49a3b3c640 10 monclient: monmap: 2024-08-19T12:43:17.287 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:17.287 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.287 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:43:17.287247+0000 2024-08-19T12:43:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:43:17.287247+0000 2024-08-19T12:43:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:17.288 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:17.289 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.286+0000 7f49a3b3c640 1 Processor -- start 2024-08-19T12:43:17.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a3b3c640 1 -- start start 2024-08-19T12:43:17.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a3b3c640 10 monclient: init 2024-08-19T12:43:17.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a3b3c640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:17.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a3b3c640 10 monclient: _add_conns ranks=[5,6,8,7,2,3,0,4,1] 2024-08-19T12:43:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a3b3c640 10 monclient(hunting): picked mon.noname-h con 0x7f499c06e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:43:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a3b3c640 10 monclient(hunting): picked mon.noname-d con 0x7f499c0a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:43:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a3b3c640 10 monclient(hunting): picked mon.noname-e con 0x7f499c0a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:43:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a3b3c640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499c136790 con 0x7f499c06e7b0 2024-08-19T12:43:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a3b3c640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499c09ddf0 con 0x7f499c0a5920 2024-08-19T12:43:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a3b3c640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499c09e0d0 con 0x7f499c0a5d50 2024-08-19T12:43:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a3b3c640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a3b3c640 10 monclient(hunting): authenticate will time out at 991.374806s 2024-08-19T12:43:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a10d7640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f499c0a5920 0x7f499c13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49590/0 (socket says 172.21.15.26:49590) 2024-08-19T12:43:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a20d9640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f499c0a5d50 0x7f499c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:58570/0 (socket says 172.21.15.26:58570) 2024-08-19T12:43:17.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.287+0000 7f49a10d7640 1 -- 172.21.15.26:0/1546087570 learned_addr learned my addr 172.21.15.26:0/1546087570 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:17.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a10d7640 1 -- 172.21.15.26:0/1546087570 >> v1:172.21.15.26:6792/0 conn(0x7f499c0a5920 legacy=0x7f499c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a20d9640 1 -- 172.21.15.26:0/1546087570 >> v1:172.21.15.26:6793/0 conn(0x7f499c0a5d50 legacy=0x7f499c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 1 -- 172.21.15.26:0/1546087570 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1976992846 0 0) 0x7f499c09e0d0 con 0x7f499c0a5d50 2024-08-19T12:43:17.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient(hunting): my global_id is 25901 2024-08-19T12:43:17.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 1 -- 172.21.15.26:0/1546087570 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4988003520 con 0x7f499c0a5d50 2024-08-19T12:43:17.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 1 -- 172.21.15.26:0/1546087570 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 897950719 0 0) 0x7f498c001450 con 0x7f499c0a5920 2024-08-19T12:43:17.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient(hunting): my global_id is 26367 2024-08-19T12:43:17.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 1 -- 172.21.15.26:0/1546087570 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f499c09e0d0 con 0x7f499c0a5920 2024-08-19T12:43:17.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a18d8640 1 -- 172.21.15.26:0/1546087570 >> v1:172.21.15.104:6791/0 conn(0x7f499c06e7b0 legacy=0x7f499c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 1 -- 172.21.15.26:0/1546087570 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3078607995 0 0) 0x7f4988003520 con 0x7f499c0a5d50 2024-08-19T12:43:17.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 1 -- 172.21.15.26:0/1546087570 >> v1:172.21.15.26:6792/0 conn(0x7f499c0a5920 legacy=0x7f499c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 1 -- 172.21.15.26:0/1546087570 >> v1:172.21.15.104:6791/0 conn(0x7f499c06e7b0 legacy=0x7f499c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient: _finish_hunting 0 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 1 monclient: found mon.noname-e 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 1 -- 172.21.15.26:0/1546087570 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499c09e3b0 con 0x7f499c0a5d50 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient: _finish_auth 0 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient: _check_auth_tickets 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 1 -- 172.21.15.26:0/1546087570 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4994001e60 con 0x7f499c0a5d50 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient: dump: 2024-08-19T12:43:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.304 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a3b3c640 5 monclient: authenticate success, global_id 25901 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a3b3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 1 -- 172.21.15.26:0/1546087570 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4994004b60 con 0x7f499c0a5d50 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 1 -- 172.21.15.26:0/1546087570 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f49940050d0 con 0x7f499c0a5d50 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a3b3c640 10 monclient: get_monmap_and_config success 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a3b3c640 4 set_mon_vals no callback set 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.290+0000 7f49a08d6640 10 monclient: dump: 2024-08-19T12:43:17.305 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.306 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.306 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.306 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.306 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.306 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.306 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.306 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.306 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.306 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.306 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.306 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.306 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.307 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.307 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.307 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.291+0000 7f49a3b3c640 10 monclient: shutdown 2024-08-19T12:43:17.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.291+0000 7f49a3b3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:17.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.291+0000 7f49a3b3c640 1 -- 172.21.15.26:0/1546087570 >> v1:172.21.15.26:6793/0 conn(0x7f499c0a5d50 legacy=0x7f499c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.291+0000 7f49a3b3c640 1 -- 172.21.15.26:0/1546087570 shutdown_connections 2024-08-19T12:43:17.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.291+0000 7f49a3b3c640 1 -- 172.21.15.26:0/1546087570 >> 172.21.15.26:0/1546087570 conn(0x7f499c0a8740 msgr2=0x7f499c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:17.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.291+0000 7f49a3b3c640 1 -- 172.21.15.26:0/1546087570 shutdown_connections 2024-08-19T12:43:17.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.291+0000 7f49a3b3c640 1 -- 172.21.15.26:0/1546087570 wait complete. 2024-08-19T12:43:17.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.291+0000 7f49a3b3c640 10 monclient: build_initial_monmap 2024-08-19T12:43:17.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.291+0000 7f49a3b3c640 10 monclient: monmap: 2024-08-19T12:43:17.308 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:17.308 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:43:17.308 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:43:17.308 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:43:17.308 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:17.308 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.308 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:17.308 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:17.308 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:17.308 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:17.308 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:17.308 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:17.309 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:17.309 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:17.309 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:17.309 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 1 librados: starting msgr at 2024-08-19T12:43:17.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 1 librados: starting objecter 2024-08-19T12:43:17.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 1 Processor -- start 2024-08-19T12:43:17.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 1 -- start start 2024-08-19T12:43:17.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 1 librados: setting wanted keys 2024-08-19T12:43:17.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 1 librados: calling monclient init 2024-08-19T12:43:17.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 10 monclient: init 2024-08-19T12:43:17.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:17.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 10 monclient: _add_conns ranks=[0,5,3,8,2,4,7,6,1] 2024-08-19T12:43:17.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 10 monclient(hunting): picked mon.noname-a con 0x7f499c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:43:17.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 10 monclient(hunting): picked mon.noname-h con 0x7f499c0a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:43:17.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 10 monclient(hunting): picked mon.noname-g con 0x7f499c0a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:43:17.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499c141fd0 con 0x7f499c06e7b0 2024-08-19T12:43:17.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499c138ee0 con 0x7f499c0a5d50 2024-08-19T12:43:17.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499c1383e0 con 0x7f499c0a5920 2024-08-19T12:43:17.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:17.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a3b3c640 10 monclient(hunting): authenticate will time out at 991.379615s 2024-08-19T12:43:17.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a18d8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f499c06e7b0 0x7f499c1375c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:58132/0 (socket says 172.21.15.26:58132) 2024-08-19T12:43:17.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a18d8640 1 -- 172.21.15.26:0/2539074572 learned_addr learned my addr 172.21.15.26:0/2539074572 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.292+0000 7f49a20d9640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.104:6790/0 conn(0x7f499c0a5d50 legacy=0x7f499c1d6290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.294+0000 7f49a18d8640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.26:6789/0 conn(0x7f499c06e7b0 legacy=0x7f499c1375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.294+0000 7f49927fc640 10 client.?.objecter ms_handle_connect 0x7f499c06e7b0 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.294+0000 7f49927fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 client.?.objecter ms_handle_connect 0x7f499c0a5920 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49a10d7640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.104:6791/0 conn(0x7f499c0a5920 legacy=0x7f499c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3957912441 0 0) 0x7f499c141fd0 con 0x7f499c06e7b0 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): my global_id is 144342 2024-08-19T12:43:17.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4980004eb0 con 0x7f499c06e7b0 2024-08-19T12:43:17.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 883241644 0 0) 0x7f499c1383e0 con 0x7f499c0a5920 2024-08-19T12:43:17.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): my global_id is 26096 2024-08-19T12:43:17.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f499c141fd0 con 0x7f499c0a5920 2024-08-19T12:43:17.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3826961335 0 0) 0x7f4980004eb0 con 0x7f499c06e7b0 2024-08-19T12:43:17.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f499c1383e0 con 0x7f499c06e7b0 2024-08-19T12:43:17.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4998003610 con 0x7f499c06e7b0 2024-08-19T12:43:17.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:17.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:43:17.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): dump: 2024-08-19T12:43:17.313 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.313 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.313 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.313 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.313 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.313 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.313 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.313 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.313 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.313 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.313 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.313 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.314 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.314 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.314 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.314 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:43:17.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1156404121 0 0) 0x7f499c141fd0 con 0x7f499c0a5920 2024-08-19T12:43:17.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4980004eb0 con 0x7f499c0a5920 2024-08-19T12:43:17.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f498c001a90 con 0x7f499c0a5920 2024-08-19T12:43:17.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:17.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:43:17.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient(hunting): dump: 2024-08-19T12:43:17.315 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.315 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.315 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.315 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.315 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.315 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.315 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.315 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.315 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.315 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.315 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.315 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4037872997 0 0) 0x7f499c1383e0 con 0x7f499c06e7b0 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.104:6791/0 conn(0x7f499c0a5920 legacy=0x7f499c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.104:6790/0 conn(0x7f499c0a5d50 legacy=0x7f499c1d6290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient: _finish_hunting 0 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 monclient: found mon.a 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499c13a9a0 con 0x7f499c06e7b0 2024-08-19T12:43:17.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient: _finish_auth 0 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient: _check_auth_tickets 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49a3b3c640 5 monclient: authenticate success, global_id 144342 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49a3b3c640 10 monclient: _renew_subs 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49a3b3c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49a3b3c640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f499c139d10 con 0x7f499c06e7b0 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49a3b3c640 10 client.144342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49a3b3c640 10 monclient: _renew_subs 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49a3b3c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49a3b3c640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f499c13b660 con 0x7f499c06e7b0 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49a3b3c640 1 librados: init done 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4998003b90 con 0x7f499c06e7b0 2024-08-19T12:43:17.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f4993fff640 4 set_mon_vals no callback set 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4998004100 con 0x7f499c06e7b0 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.295+0000 7f49927fc640 10 monclient: dump: 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.318 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.319 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.319 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.319 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.319 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.319 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.319 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.319 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.319 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.296+0000 7f49a3b3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:17.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.296+0000 7f49a3b3c640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f49840065c0 con 0x7f4984002860 2024-08-19T12:43:17.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.296+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f49980042a0 con 0x7f499c06e7b0 2024-08-19T12:43:17.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.297+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(187..187 src has 1..187) ==== 5730+0+0 (unknown 4122717534 0 0) 0x7f499804e400 con 0x7f499c06e7b0 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.297+0000 7f49927fc640 10 client.144342.objecter ms_dispatch 0x7f499c000b90 osd_map(187..187 src has 1..187) 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.297+0000 7f49927fc640 3 client.144342.objecter handle_osd_map got epochs [187,187] > 0 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.297+0000 7f49927fc640 3 client.144342.objecter handle_osd_map decoding full epoch 187 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.297+0000 7f49927fc640 20 client.144342.objecter dump_active .. 0 homeless 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.297+0000 7f49a20d9640 10 monclient: get_auth_request con 0x7f49800414f0 auth_method 0 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.298+0000 7f49a10d7640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.26:6791/0 conn(0x7f4984002860 legacy=0x7f4984004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.298+0000 7f49927fc640 10 client.144342.objecter ms_handle_connect 0x7f4984002860 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.298+0000 7f49927fc640 10 client.144342.objecter resend_mon_ops 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.298+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3635228372 0 0) 0x7f49840065c0 con 0x7f4984002860 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.298+0000 7f49927fc640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.298+0000 7f49927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.298+0000 7f49927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.298+0000 7f49927fc640 10 monclient(hunting): my global_id is 26347 2024-08-19T12:43:17.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.298+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f499c1383e0 con 0x7f4984002860 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3260147501 0 0) 0x7f499c1383e0 con 0x7f4984002860 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49840065c0 con 0x7f4984002860 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f498c004e70 con 0x7f4984002860 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 10 monclient: dump: 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.321 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.322 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.322 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.322 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.322 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.322 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.322 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.322 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.322 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.322 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3241335839 0 0) 0x7f49840065c0 con 0x7f4984002860 2024-08-19T12:43:17.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f498400a160 con 0x7f4984002860 2024-08-19T12:43:17.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 10 client.144342.objecter ms_handle_connect 0x7f49800414f0 2024-08-19T12:43:17.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f498c006370 con 0x7f4984002860 2024-08-19T12:43:17.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:17.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:43:17.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.26:6791/0 conn(0x7f4984002860 legacy=0x7f4984004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.299+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.26:6791/0 conn(0x7f4984002860 legacy=0x7f4984004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:17.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.305+0000 7f49a3b3c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:43:17.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.305+0000 7f49a3b3c640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4984009e70 con 0x7f4984009130 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 10 client.144342.objecter ms_handle_connect 0x7f4984009130 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49a10d7640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.26:6791/0 conn(0x7f4984009130 legacy=0x7f49840095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 10 client.144342.objecter resend_mon_ops 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3140570235 0 0) 0x7f4984009e70 con 0x7f4984009130 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 10 monclient(hunting): my global_id is 26356 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49840065c0 con 0x7f4984009130 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4287385216 0 0) 0x7f49840065c0 con 0x7f4984009130 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4984009e70 con 0x7f4984009130 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f498c005420 con 0x7f4984009130 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:43:17.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 10 monclient: dump: 2024-08-19T12:43:17.326 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.326 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.326 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.326 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.326 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.326 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.326 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.326 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.326 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.326 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.327 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.327 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.327 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.327 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.327 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.327 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1428424895 0 0) 0x7f4984009e70 con 0x7f4984009130 2024-08-19T12:43:17.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.308+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f498400c3f0 con 0x7f4984009130 2024-08-19T12:43:17.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7867 (unknown 0 0 3566888801) 0x7f498c005870 con 0x7f4984009130 2024-08-19T12:43:17.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49927fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:43:17.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49927fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:43:17.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.26:6791/0 conn(0x7f4984009130 legacy=0x7f49840095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49927fc640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.26:6791/0 conn(0x7f4984009130 legacy=0x7f49840095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:17.328 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:43:17.328 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:43:17.328 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:43:17.328 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:43:17.328 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 28, 2024-08-19T12:43:17.328 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 310338, 2024-08-19T12:43:17.328 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:43:17.329 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:43:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.330 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:43:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:43:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.331 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:43:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:43:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:43:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:43:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:43:17.332 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:43:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:43:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:43:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:43:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:43:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:43:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:43:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:43:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:43:17.333 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.334 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.335 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:43:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:43:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:43:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:43:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:43:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:43:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:43:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.337 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:43:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:43:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:43:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.338 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.339 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:43:17.341 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:43:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:43:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:43:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.342 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.343 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:43:17.344 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:43:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:43:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:43:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:43:17.346 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:43:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:43:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:43:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.347 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:43:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:43:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.348 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:43:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:43:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.349 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:43:17.350 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:43:17.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 10 librados: watch_flush enter 2024-08-19T12:43:17.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 10 librados: watch_flush exit 2024-08-19T12:43:17.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 20 client.144342.objecter shutdown clearing up homeless session... 2024-08-19T12:43:17.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 10 client.144342.objecter successfully canceled tick 2024-08-19T12:43:17.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.104:6800/899887506 conn(0x7f49800414f0 legacy=0x7f49800439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 10 monclient: shutdown 2024-08-19T12:43:17.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:17.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 1 -- 172.21.15.26:0/2539074572 >> v1:172.21.15.26:6789/0 conn(0x7f499c06e7b0 legacy=0x7f499c1375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a20d9640 1 -- 172.21.15.26:0/2539074572 reap_dead start 2024-08-19T12:43:17.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a20d9640 1 -- 172.21.15.26:0/2539074572 reap_dead start 2024-08-19T12:43:17.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 1 -- 172.21.15.26:0/2539074572 shutdown_connections 2024-08-19T12:43:17.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 1 -- 172.21.15.26:0/2539074572 >> 172.21.15.26:0/2539074572 conn(0x7f499c0a8740 msgr2=0x7f499c1350a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:17.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 1 -- 172.21.15.26:0/2539074572 shutdown_connections 2024-08-19T12:43:17.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 1 -- 172.21.15.26:0/2539074572 wait complete. 2024-08-19T12:43:17.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.309+0000 7f49a3b3c640 1 librados: shutdown 2024-08-19T12:43:17.353 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:43:17.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon d 2024-08-19T12:43:17.354 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:43:17.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.452+0000 7f5a6f060640 10 monclient: get_monmap_and_config 2024-08-19T12:43:17.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.452+0000 7f5a6f060640 10 monclient: build_initial_monmap 2024-08-19T12:43:17.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.452+0000 7f5a6f060640 10 monclient: monmap: 2024-08-19T12:43:17.454 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:17.454 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.454 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:43:17.453507+0000 2024-08-19T12:43:17.454 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:43:17.453507+0000 2024-08-19T12:43:17.454 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:17.454 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.453+0000 7f5a6f060640 1 Processor -- start 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.453+0000 7f5a6f060640 1 -- start start 2024-08-19T12:43:17.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.453+0000 7f5a6f060640 10 monclient: init 2024-08-19T12:43:17.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.453+0000 7f5a6f060640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:17.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.453+0000 7f5a6f060640 10 monclient: _add_conns ranks=[5,1,8,2,3,0,7,4,6] 2024-08-19T12:43:17.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.453+0000 7f5a6f060640 10 monclient(hunting): picked mon.noname-h con 0x7f5a6806e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:43:17.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.453+0000 7f5a6f060640 10 monclient(hunting): picked mon.noname-f con 0x7f5a680a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:43:17.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.453+0000 7f5a6f060640 10 monclient(hunting): picked mon.noname-e con 0x7f5a680a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:43:17.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.453+0000 7f5a6f060640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a68136790 con 0x7f5a680a5920 2024-08-19T12:43:17.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.453+0000 7f5a6f060640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a6809ddf0 con 0x7f5a6806e7b0 2024-08-19T12:43:17.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.454+0000 7f5a6f060640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a6809e0d0 con 0x7f5a680a5d50 2024-08-19T12:43:17.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.454+0000 7f5a6f060640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:17.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.454+0000 7f5a6f060640 10 monclient(hunting): authenticate will time out at 991.541020s 2024-08-19T12:43:17.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.454+0000 7f5a6d5fd640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f5a680a5d50 0x7f5a68141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:58586/0 (socket says 172.21.15.26:58586) 2024-08-19T12:43:17.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.454+0000 7f5a6d5fd640 1 -- 172.21.15.26:0/70687374 learned_addr learned my addr 172.21.15.26:0/70687374 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:17.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a6d5fd640 1 -- 172.21.15.26:0/70687374 >> v1:172.21.15.26:6793/0 conn(0x7f5a680a5d50 legacy=0x7f5a68141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 1 -- 172.21.15.26:0/70687374 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234209087 0 0) 0x7f5a6809e0d0 con 0x7f5a680a5d50 2024-08-19T12:43:17.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient(hunting): my global_id is 25910 2024-08-19T12:43:17.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 1 -- 172.21.15.26:0/70687374 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a58003520 con 0x7f5a680a5d50 2024-08-19T12:43:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a6cdfc640 1 -- 172.21.15.26:0/70687374 >> v1:172.21.15.104:6791/0 conn(0x7f5a6806e7b0 legacy=0x7f5a68136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a67fff640 1 -- 172.21.15.26:0/70687374 >> v1:172.21.15.104:6789/0 conn(0x7f5a680a5920 legacy=0x7f5a6813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 1 -- 172.21.15.26:0/70687374 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1123628443 0 0) 0x7f5a58003520 con 0x7f5a680a5d50 2024-08-19T12:43:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 1 -- 172.21.15.26:0/70687374 >> v1:172.21.15.104:6791/0 conn(0x7f5a6806e7b0 legacy=0x7f5a68136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 1 -- 172.21.15.26:0/70687374 >> v1:172.21.15.104:6789/0 conn(0x7f5a680a5920 legacy=0x7f5a6813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient: _finish_hunting 0 2024-08-19T12:43:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 1 monclient: found mon.noname-e 2024-08-19T12:43:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:43:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 1 -- 172.21.15.26:0/70687374 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6809e3b0 con 0x7f5a680a5d50 2024-08-19T12:43:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient: _finish_auth 0 2024-08-19T12:43:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient: _check_auth_tickets 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 1 -- 172.21.15.26:0/70687374 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5a60001e60 con 0x7f5a680a5d50 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a6f060640 5 monclient: authenticate success, global_id 25910 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient: dump: 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.464 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.465 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a6f060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 1 -- 172.21.15.26:0/70687374 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a60004c20 con 0x7f5a680a5d50 2024-08-19T12:43:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a6f060640 10 monclient: get_monmap_and_config success 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 1 -- 172.21.15.26:0/70687374 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5a60005190 con 0x7f5a680a5d50 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a6f060640 4 set_mon_vals no callback set 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.456+0000 7f5a677fe640 10 monclient: dump: 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.466 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.457+0000 7f5a6f060640 10 monclient: shutdown 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.457+0000 7f5a6f060640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.457+0000 7f5a6f060640 1 -- 172.21.15.26:0/70687374 >> v1:172.21.15.26:6793/0 conn(0x7f5a680a5d50 legacy=0x7f5a68141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.457+0000 7f5a6f060640 1 -- 172.21.15.26:0/70687374 shutdown_connections 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.457+0000 7f5a6f060640 1 -- 172.21.15.26:0/70687374 >> 172.21.15.26:0/70687374 conn(0x7f5a680a8740 msgr2=0x7f5a680a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.457+0000 7f5a6f060640 1 -- 172.21.15.26:0/70687374 shutdown_connections 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.457+0000 7f5a6f060640 1 -- 172.21.15.26:0/70687374 wait complete. 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.457+0000 7f5a6f060640 10 monclient: build_initial_monmap 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 10 monclient: monmap: 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:17.468 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 1 librados: starting msgr at 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 1 librados: starting objecter 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 1 Processor -- start 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 1 -- start start 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 1 librados: setting wanted keys 2024-08-19T12:43:17.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 1 librados: calling monclient init 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 10 monclient: init 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 10 monclient: _add_conns ranks=[7,4,5,0,8,3,2,1,6] 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 10 monclient(hunting): picked mon.noname-i con 0x7f5a6806e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 10 monclient(hunting): picked mon.noname-c con 0x7f5a680a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 10 monclient(hunting): picked mon.noname-h con 0x7f5a680a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a68138e40 con 0x7f5a680a5920 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a68138250 con 0x7f5a680a5d50 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a6813a900 con 0x7f5a6806e7b0 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.458+0000 7f5a6f060640 10 monclient(hunting): authenticate will time out at 991.545992s 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.459+0000 7f5a67fff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f5a680a5920 0x7f5a68137b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:35456/0 (socket says 172.21.15.26:35456) 2024-08-19T12:43:17.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.459+0000 7f5a67fff640 1 -- 172.21.15.26:0/490408394 learned_addr learned my addr 172.21.15.26:0/490408394 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:17.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a67fff640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.26:6791/0 conn(0x7f5a680a5920 legacy=0x7f5a68137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 client.?.objecter ms_handle_connect 0x7f5a680a5920 2024-08-19T12:43:17.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1361945026 0 0) 0x7f5a68138e40 con 0x7f5a680a5920 2024-08-19T12:43:17.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): my global_id is 26365 2024-08-19T12:43:17.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a4c004eb0 con 0x7f5a680a5920 2024-08-19T12:43:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a6d5fd640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.104:6791/0 conn(0x7f5a680a5d50 legacy=0x7f5a681d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 client.?.objecter ms_handle_connect 0x7f5a680a5d50 2024-08-19T12:43:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 client.?.objecter ms_handle_connect 0x7f5a6806e7b0 2024-08-19T12:43:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a6cdfc640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.104:6792/0 conn(0x7f5a6806e7b0 legacy=0x7f5a68137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2716004915 0 0) 0x7f5a4c004eb0 con 0x7f5a680a5920 2024-08-19T12:43:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a68138e40 con 0x7f5a680a5920 2024-08-19T12:43:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5a54003610 con 0x7f5a680a5920 2024-08-19T12:43:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:43:17.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): dump: 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.479 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.480 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.480 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.480 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:43:17.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3272481772 0 0) 0x7f5a6813a900 con 0x7f5a6806e7b0 2024-08-19T12:43:17.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): my global_id is 134944 2024-08-19T12:43:17.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a4c004eb0 con 0x7f5a6806e7b0 2024-08-19T12:43:17.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3924471541 0 0) 0x7f5a68138250 con 0x7f5a680a5d50 2024-08-19T12:43:17.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient(hunting): my global_id is 26114 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a6813a900 con 0x7f5a680a5d50 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2154324761 0 0) 0x7f5a68138e40 con 0x7f5a680a5920 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.104:6792/0 conn(0x7f5a6806e7b0 legacy=0x7f5a68137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.104:6791/0 conn(0x7f5a680a5d50 legacy=0x7f5a681d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient: _finish_hunting 0 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 monclient: found mon.c 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6813b5c0 con 0x7f5a680a5920 2024-08-19T12:43:17.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient: _finish_auth 0 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient: _check_auth_tickets 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a6f060640 5 monclient: authenticate success, global_id 26365 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a6f060640 10 monclient: _renew_subs 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a6f060640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a6f060640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5a68142880 con 0x7f5a680a5920 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a6f060640 10 client.26365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a6f060640 10 monclient: _renew_subs 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a6f060640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a6f060640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f5a68139b80 con 0x7f5a680a5920 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a6f060640 1 librados: init done 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a54003bd0 con 0x7f5a680a5920 2024-08-19T12:43:17.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:17.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5a54004100 con 0x7f5a680a5920 2024-08-19T12:43:17.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a66ffd640 4 set_mon_vals no callback set 2024-08-19T12:43:17.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:43:17.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.461+0000 7f5a657fa640 10 monclient: dump: 2024-08-19T12:43:17.483 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.483 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.483 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.483 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.483 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.483 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.483 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.462+0000 7f5a6f060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.462+0000 7f5a6f060640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5a480065c0 con 0x7f5a48002860 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.462+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5a540042a0 con 0x7f5a680a5920 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.463+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(187..187 src has 1..187) ==== 5730+0+0 (unknown 4122717534 0 0) 0x7f5a5404e470 con 0x7f5a680a5920 2024-08-19T12:43:17.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.463+0000 7f5a657fa640 10 client.26365.objecter ms_dispatch 0x7f5a68000b90 osd_map(187..187 src has 1..187) 2024-08-19T12:43:17.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.463+0000 7f5a657fa640 3 client.26365.objecter handle_osd_map got epochs [187,187] > 0 2024-08-19T12:43:17.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.463+0000 7f5a657fa640 3 client.26365.objecter handle_osd_map decoding full epoch 187 2024-08-19T12:43:17.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.463+0000 7f5a657fa640 20 client.26365.objecter dump_active .. 0 homeless 2024-08-19T12:43:17.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.463+0000 7f5a6d5fd640 10 monclient: get_auth_request con 0x7f5a4c0411b0 auth_method 0 2024-08-19T12:43:17.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.464+0000 7f5a6cdfc640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.26:6792/0 conn(0x7f5a48002860 legacy=0x7f5a48004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.464+0000 7f5a657fa640 10 client.26365.objecter ms_handle_connect 0x7f5a48002860 2024-08-19T12:43:17.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.464+0000 7f5a657fa640 10 client.26365.objecter resend_mon_ops 2024-08-19T12:43:17.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 550315323 0 0) 0x7f5a480065c0 con 0x7f5a48002860 2024-08-19T12:43:17.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 10 monclient(hunting): my global_id is 26376 2024-08-19T12:43:17.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a68138e40 con 0x7f5a48002860 2024-08-19T12:43:17.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 174521181 0 0) 0x7f5a68138e40 con 0x7f5a48002860 2024-08-19T12:43:17.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a480065c0 con 0x7f5a48002860 2024-08-19T12:43:17.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5a5c001b40 con 0x7f5a48002860 2024-08-19T12:43:17.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:43:17.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 10 monclient: dump: 2024-08-19T12:43:17.486 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.486 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.486 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.487 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 10 client.26365.objecter ms_handle_connect 0x7f5a4c0411b0 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2669161799 0 0) 0x7f5a480065c0 con 0x7f5a48002860 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a4800a160 con 0x7f5a48002860 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5a5c0057c0 con 0x7f5a48002860 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.26:6792/0 conn(0x7f5a48002860 legacy=0x7f5a48004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.465+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.26:6792/0 conn(0x7f5a48002860 legacy=0x7f5a48004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.472+0000 7f5a6f060640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.472+0000 7f5a6f060640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5a48009e50 con 0x7f5a48009110 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a6cdfc640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.26:6792/0 conn(0x7f5a48009110 legacy=0x7f5a480095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 10 client.26365.objecter ms_handle_connect 0x7f5a48009110 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 10 client.26365.objecter resend_mon_ops 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1897753725 0 0) 0x7f5a48009e50 con 0x7f5a48009110 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 10 monclient(hunting): my global_id is 26385 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a480065c0 con 0x7f5a48009110 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3293932739 0 0) 0x7f5a480065c0 con 0x7f5a48009110 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a48009e50 con 0x7f5a48009110 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5a5c003d40 con 0x7f5a48009110 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:43:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 10 monclient: dump: 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.491 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.491 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1535002177 0 0) 0x7f5a48009e50 con 0x7f5a48009110 2024-08-19T12:43:17.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.474+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a4800c3f0 con 0x7f5a48009110 2024-08-19T12:43:17.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 1896624311) 0x7f5a5c005a70 con 0x7f5a48009110 2024-08-19T12:43:17.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:43:17.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a657fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:43:17.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.26:6792/0 conn(0x7f5a48009110 legacy=0x7f5a480095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a657fa640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.26:6792/0 conn(0x7f5a48009110 legacy=0x7f5a480095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:17.491 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 28, 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 310515, 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:43:17.492 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:43:17.493 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:43:17.493 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.493 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:43:17.493 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.493 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:43:17.493 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:43:17.493 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.493 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.493 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.493 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.493 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.493 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.494 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.494 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.494 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.494 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.494 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.494 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.494 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:43:17.494 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:43:17.494 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.494 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.494 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.494 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:43:17.495 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:43:17.496 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:43:17.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:43:17.499 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:43:17.499 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.499 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.499 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.499 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:43:17.499 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:43:17.499 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:43:17.502 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:43:17.502 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.502 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.502 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.502 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:43:17.502 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:43:17.502 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.502 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.502 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.503 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:43:17.504 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:43:17.504 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.504 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:43:17.504 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:43:17.504 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:43:17.505 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.506 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.506 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:43:17.506 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:43:17.506 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.506 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.506 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.506 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:43:17.506 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.507 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.508 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:43:17.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:43:17.510 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:43:17.510 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.510 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.510 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.511 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:43:17.511 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:43:17.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.511 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.511 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.511 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.511 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.511 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:43:17.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.512 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.512 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.512 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.512 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.512 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.512 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:43:17.512 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:43:17.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 10 librados: watch_flush enter 2024-08-19T12:43:17.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 10 librados: watch_flush exit 2024-08-19T12:43:17.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 20 client.26365.objecter shutdown clearing up homeless session... 2024-08-19T12:43:17.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 10 client.26365.objecter successfully canceled tick 2024-08-19T12:43:17.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.104:6800/899887506 conn(0x7f5a4c0411b0 legacy=0x7f5a4c043670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 10 monclient: shutdown 2024-08-19T12:43:17.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:17.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 1 -- 172.21.15.26:0/490408394 >> v1:172.21.15.26:6791/0 conn(0x7f5a680a5920 legacy=0x7f5a68137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6d5fd640 1 -- 172.21.15.26:0/490408394 reap_dead start 2024-08-19T12:43:17.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6d5fd640 1 -- 172.21.15.26:0/490408394 reap_dead start 2024-08-19T12:43:17.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 1 -- 172.21.15.26:0/490408394 shutdown_connections 2024-08-19T12:43:17.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 1 -- 172.21.15.26:0/490408394 >> 172.21.15.26:0/490408394 conn(0x7f5a680a8740 msgr2=0x7f5a680ab340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:17.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 1 -- 172.21.15.26:0/490408394 shutdown_connections 2024-08-19T12:43:17.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 1 -- 172.21.15.26:0/490408394 wait complete. 2024-08-19T12:43:17.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.475+0000 7f5a6f060640 1 librados: shutdown 2024-08-19T12:43:17.515 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:43:17.515 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-08-19T12:43:17.516 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:43:17.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.620+0000 7f643f98a640 10 monclient: get_monmap_and_config 2024-08-19T12:43:17.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.620+0000 7f643f98a640 10 monclient: build_initial_monmap 2024-08-19T12:43:17.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.620+0000 7f643f98a640 10 monclient: monmap: 2024-08-19T12:43:17.621 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:17.621 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.621 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:43:17.621259+0000 2024-08-19T12:43:17.621 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:43:17.621259+0000 2024-08-19T12:43:17.622 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:17.622 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.622 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:17.622 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:17.622 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:17.622 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:17.622 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:17.622 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:17.622 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:17.622 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:17.622 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:17.622 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.620+0000 7f643f98a640 1 Processor -- start 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643f98a640 1 -- start start 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643f98a640 10 monclient: init 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643f98a640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643f98a640 10 monclient: _add_conns ranks=[3,4,5,6,7,2,1,8,0] 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643f98a640 10 monclient(hunting): picked mon.noname-g con 0x7f643806e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643f98a640 10 monclient(hunting): picked mon.noname-c con 0x7f64380a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643f98a640 10 monclient(hunting): picked mon.noname-h con 0x7f64380a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643f98a640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6438136790 con 0x7f643806e7b0 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643f98a640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f643809ddf0 con 0x7f64380a5920 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643f98a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f643809e0d0 con 0x7f64380a5d50 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643f98a640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643f98a640 10 monclient(hunting): authenticate will time out at 991.708716s 2024-08-19T12:43:17.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643cf25640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f64380a5920 0x7f643813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:35472/0 (socket says 172.21.15.26:35472) 2024-08-19T12:43:17.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643cf25640 1 -- 172.21.15.26:0/3761802104 learned_addr learned my addr 172.21.15.26:0/3761802104 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:17.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.621+0000 7f643d726640 1 -- 172.21.15.26:0/3761802104 >> v1:172.21.15.104:6790/0 conn(0x7f643806e7b0 legacy=0x7f6438136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:43:17.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.624+0000 7f643cf25640 1 -- 172.21.15.26:0/3761802104 >> v1:172.21.15.26:6791/0 conn(0x7f64380a5920 legacy=0x7f643813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.624+0000 7f643df27640 1 -- 172.21.15.26:0/3761802104 >> v1:172.21.15.104:6791/0 conn(0x7f64380a5d50 legacy=0x7f6438141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.627+0000 7f642bfff640 1 -- 172.21.15.26:0/3761802104 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2373501754 0 0) 0x7f642c001450 con 0x7f64380a5920 2024-08-19T12:43:17.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.627+0000 7f642bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.627+0000 7f642bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.627+0000 7f642bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.627+0000 7f642bfff640 10 monclient(hunting): my global_id is 26374 2024-08-19T12:43:17.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.627+0000 7f642bfff640 1 -- 172.21.15.26:0/3761802104 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6424003520 con 0x7f64380a5920 2024-08-19T12:43:17.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 1 -- 172.21.15.26:0/3761802104 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 476531617 0 0) 0x7f6424003520 con 0x7f64380a5920 2024-08-19T12:43:17.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 1 -- 172.21.15.26:0/3761802104 >> v1:172.21.15.104:6791/0 conn(0x7f64380a5d50 legacy=0x7f6438141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 1 -- 172.21.15.26:0/3761802104 >> v1:172.21.15.104:6790/0 conn(0x7f643806e7b0 legacy=0x7f6438136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:43:17.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:43:17.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 1 monclient: found mon.noname-c 2024-08-19T12:43:17.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 1 -- 172.21.15.26:0/3761802104 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643809e3b0 con 0x7f64380a5920 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 10 monclient: _finish_auth 0 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 1 -- 172.21.15.26:0/3761802104 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f642c001cb0 con 0x7f64380a5920 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f643f98a640 5 monclient: authenticate success, global_id 26374 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f643f98a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 10 monclient: dump: 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.636 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.637 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f643f98a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 1 -- 172.21.15.26:0/3761802104 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f642c004990 con 0x7f64380a5920 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 1 -- 172.21.15.26:0/3761802104 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f642c004f00 con 0x7f64380a5920 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f643f98a640 10 monclient: get_monmap_and_config success 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f643f98a640 4 set_mon_vals no callback set 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.633+0000 7f642bfff640 10 monclient: dump: 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.638 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.639 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.639 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.639 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.639 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.639 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.639 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.639 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.639 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.639 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.639 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.640 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.640 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.634+0000 7f643f98a640 10 monclient: shutdown 2024-08-19T12:43:17.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.634+0000 7f643f98a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:17.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.634+0000 7f643f98a640 1 -- 172.21.15.26:0/3761802104 >> v1:172.21.15.26:6791/0 conn(0x7f64380a5920 legacy=0x7f643813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.634+0000 7f643f98a640 1 -- 172.21.15.26:0/3761802104 shutdown_connections 2024-08-19T12:43:17.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.634+0000 7f643f98a640 1 -- 172.21.15.26:0/3761802104 >> 172.21.15.26:0/3761802104 conn(0x7f64380a8740 msgr2=0x7f64380a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:17.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.634+0000 7f643f98a640 1 -- 172.21.15.26:0/3761802104 shutdown_connections 2024-08-19T12:43:17.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.634+0000 7f643f98a640 1 -- 172.21.15.26:0/3761802104 wait complete. 2024-08-19T12:43:17.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.635+0000 7f643f98a640 10 monclient: build_initial_monmap 2024-08-19T12:43:17.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.635+0000 7f643f98a640 10 monclient: monmap: 2024-08-19T12:43:17.646 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:17.646 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:43:17.646 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:43:17.646 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:43:17.646 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:17.647 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.647 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:17.647 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:17.647 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:17.647 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:17.647 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:17.647 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:17.647 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.635+0000 7f643f98a640 1 librados: starting msgr at 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.635+0000 7f643f98a640 1 librados: starting objecter 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.635+0000 7f643f98a640 1 Processor -- start 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.635+0000 7f643f98a640 1 -- start start 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.635+0000 7f643f98a640 1 librados: setting wanted keys 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.635+0000 7f643f98a640 1 librados: calling monclient init 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.635+0000 7f643f98a640 10 monclient: init 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.636+0000 7f643f98a640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.636+0000 7f643f98a640 10 monclient: _add_conns ranks=[7,2,6,8,0,4,5,1,3] 2024-08-19T12:43:17.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.636+0000 7f643f98a640 10 monclient(hunting): picked mon.noname-i con 0x7f643806e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.636+0000 7f643f98a640 10 monclient(hunting): picked mon.noname-b con 0x7f64380a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.636+0000 7f643f98a640 10 monclient(hunting): picked mon.noname-d con 0x7f64380a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.636+0000 7f643f98a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f643813b390 con 0x7f64380a5920 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.636+0000 7f643f98a640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6438139030 con 0x7f64380a5d50 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.636+0000 7f643f98a640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f643813aaf0 con 0x7f643806e7b0 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.636+0000 7f643f98a640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.636+0000 7f643f98a640 10 monclient(hunting): authenticate will time out at 991.723283s 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.636+0000 7f643df27640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f64380a5d50 0x7f64381d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49628/0 (socket says 172.21.15.26:49628) 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.636+0000 7f643df27640 1 -- 172.21.15.26:0/3875032473 learned_addr learned my addr 172.21.15.26:0/3875032473 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f643df27640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.26:6792/0 conn(0x7f64380a5d50 legacy=0x7f64381d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 10 client.?.objecter ms_handle_connect 0x7f64380a5d50 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 10 client.?.objecter ms_handle_connect 0x7f64380a5920 2024-08-19T12:43:17.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f643cf25640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.26:6790/0 conn(0x7f64380a5920 legacy=0x7f6438137ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f643d726640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.104:6792/0 conn(0x7f643806e7b0 legacy=0x7f64381377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 10 client.?.objecter ms_handle_connect 0x7f643806e7b0 2024-08-19T12:43:17.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 729086743 0 0) 0x7f643813b390 con 0x7f64380a5920 2024-08-19T12:43:17.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 10 monclient(hunting): my global_id is 26111 2024-08-19T12:43:17.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f641c004eb0 con 0x7f64380a5920 2024-08-19T12:43:17.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.638+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3959852574 0 0) 0x7f6438139030 con 0x7f64380a5d50 2024-08-19T12:43:17.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): my global_id is 26394 2024-08-19T12:43:17.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f643813b390 con 0x7f64380a5d50 2024-08-19T12:43:17.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3299393420 0 0) 0x7f643813aaf0 con 0x7f643806e7b0 2024-08-19T12:43:17.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): my global_id is 134953 2024-08-19T12:43:17.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6438139030 con 0x7f643806e7b0 2024-08-19T12:43:17.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 458941475 0 0) 0x7f641c004eb0 con 0x7f64380a5920 2024-08-19T12:43:17.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f643813aaf0 con 0x7f64380a5920 2024-08-19T12:43:17.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f642c001cf0 con 0x7f64380a5920 2024-08-19T12:43:17.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:17.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:43:17.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): dump: 2024-08-19T12:43:17.652 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.652 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.652 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.652 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.652 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.652 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.652 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4128549906 0 0) 0x7f643813b390 con 0x7f64380a5d50 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f641c004eb0 con 0x7f64380a5d50 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6430002050 con 0x7f64380a5d50 2024-08-19T12:43:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient(hunting): dump: 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.654 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3998952332 0 0) 0x7f643813aaf0 con 0x7f64380a5920 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.104:6792/0 conn(0x7f643806e7b0 legacy=0x7f64381377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.26:6792/0 conn(0x7f64380a5d50 legacy=0x7f64381d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 monclient: found mon.b 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:43:17.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643813add0 con 0x7f64380a5920 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient: _finish_auth 0 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f643f98a640 5 monclient: authenticate success, global_id 26111 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f642c004e50 con 0x7f64380a5920 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f642c005000 con 0x7f64380a5920 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f643f98a640 10 monclient: _renew_subs 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f643f98a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f643f98a640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6438139b90 con 0x7f64380a5920 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.639+0000 7f642b7fe640 4 set_mon_vals no callback set 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.640+0000 7f643f98a640 10 client.26111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.640+0000 7f643f98a640 10 monclient: _renew_subs 2024-08-19T12:43:17.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.640+0000 7f643f98a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.640+0000 7f643f98a640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6438138600 con 0x7f64380a5920 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.640+0000 7f643f98a640 1 librados: init done 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.640+0000 7f6429ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.640+0000 7f6429ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.640+0000 7f6429ffb640 10 monclient: dump: 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.657 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.658 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.658 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.658 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.658 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.658 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.658 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.658 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.641+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f642c0118c0 con 0x7f64380a5920 2024-08-19T12:43:17.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.641+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(188..188 src has 1..188) ==== 6457+0+0 (unknown 600343228 0 0) 0x7f642c04e4d0 con 0x7f64380a5920 2024-08-19T12:43:17.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.641+0000 7f6429ffb640 10 client.26111.objecter ms_dispatch 0x7f6438000b90 osd_map(188..188 src has 1..188) 2024-08-19T12:43:17.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.641+0000 7f6429ffb640 3 client.26111.objecter handle_osd_map got epochs [188,188] > 0 2024-08-19T12:43:17.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.641+0000 7f6429ffb640 3 client.26111.objecter handle_osd_map decoding full epoch 188 2024-08-19T12:43:17.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.641+0000 7f643f98a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:17.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.641+0000 7f643f98a640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f64380008d0 con 0x7f643802e340 2024-08-19T12:43:17.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.641+0000 7f6429ffb640 20 client.26111.objecter dump_active .. 0 homeless 2024-08-19T12:43:17.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.641+0000 7f643d726640 10 monclient: get_auth_request con 0x7f641c041540 auth_method 0 2024-08-19T12:43:17.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.643+0000 7f643df27640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.26:6793/0 conn(0x7f643802e340 legacy=0x7f6438063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.643+0000 7f6429ffb640 10 client.26111.objecter ms_handle_connect 0x7f643802e340 2024-08-19T12:43:17.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.643+0000 7f6429ffb640 10 client.26111.objecter resend_mon_ops 2024-08-19T12:43:17.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.643+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2498718788 0 0) 0x7f64380008d0 con 0x7f643802e340 2024-08-19T12:43:17.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.643+0000 7f6429ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.643+0000 7f6429ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.643+0000 7f6429ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.643+0000 7f6429ffb640 10 monclient(hunting): my global_id is 25919 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.643+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f643813aaf0 con 0x7f643802e340 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 10 client.26111.objecter ms_handle_connect 0x7f641c041540 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3506493541 0 0) 0x7f643813aaf0 con 0x7f643802e340 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64380008d0 con 0x7f643802e340 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6430004070 con 0x7f643802e340 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 10 monclient: dump: 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.660 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4125022412 0 0) 0x7f64380008d0 con 0x7f643802e340 2024-08-19T12:43:17.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64381345d0 con 0x7f643802e340 2024-08-19T12:43:17.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6430005b40 con 0x7f643802e340 2024-08-19T12:43:17.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:17.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:43:17.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.26:6793/0 conn(0x7f643802e340 legacy=0x7f6438063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.644+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.26:6793/0 conn(0x7f643802e340 legacy=0x7f6438063a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:17.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.651+0000 7f643f98a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:43:17.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.651+0000 7f643f98a640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6438064190 con 0x7f64380a18d0 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.653+0000 7f6429ffb640 10 client.26111.objecter ms_handle_connect 0x7f64380a18d0 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.653+0000 7f643df27640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.26:6793/0 conn(0x7f64380a18d0 legacy=0x7f64381db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.653+0000 7f6429ffb640 10 client.26111.objecter resend_mon_ops 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.653+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1072973670 0 0) 0x7f6438064190 con 0x7f64380a18d0 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.653+0000 7f6429ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.653+0000 7f6429ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.653+0000 7f6429ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.653+0000 7f6429ffb640 10 monclient(hunting): my global_id is 25928 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.653+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64380008d0 con 0x7f64380a18d0 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3466451794 0 0) 0x7f64380008d0 con 0x7f64380a18d0 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6438064190 con 0x7f64380a18d0 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6430005cf0 con 0x7f64380a18d0 2024-08-19T12:43:17.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:43:17.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 10 monclient: dump: 2024-08-19T12:43:17.665 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.665 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.665 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.665 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.665 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.665 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.665 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.665 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.665 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.666 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.666 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.666 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.666 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.666 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.666 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.666 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3102848018 0 0) 0x7f6438064190 con 0x7f64380a18d0 2024-08-19T12:43:17.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f643802e720 con 0x7f64380a18d0 2024-08-19T12:43:17.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7700 (unknown 0 0 3659979645) 0x7f64300060e0 con 0x7f64380a18d0 2024-08-19T12:43:17.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:43:17.667 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:43:17.667 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:43:17.667 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:43:17.667 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:43:17.667 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 28, 2024-08-19T12:43:17.667 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 310671, 2024-08-19T12:43:17.667 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:43:17.668 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:43:17.668 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:43:17.668 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:43:17.668 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:43:17.668 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:43:17.668 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:43:17.668 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:43:17.668 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:43:17.668 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.668 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:43:17.668 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.668 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.669 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.670 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:43:17.670 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:43:17.670 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.670 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.670 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.670 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.670 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.670 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.670 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:43:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:43:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:43:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:43:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:43:17.671 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:43:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:43:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:43:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:43:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:43:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:43:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:43:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:43:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:43:17.672 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.673 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:43:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:43:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:43:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:43:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:43:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:43:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:43:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:43:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.676 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.677 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:43:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:43:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:43:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:43:17.681 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:43:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:43:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:43:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:43:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:43:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:43:17.685 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:43:17.686 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:43:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:43:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:43:17.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:43:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.688 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:43:17.688 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:43:17.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:43:17.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.26:6793/0 conn(0x7f64380a18d0 legacy=0x7f64381db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f6429ffb640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.26:6793/0 conn(0x7f64380a18d0 legacy=0x7f64381db9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:17.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f643f98a640 10 librados: watch_flush enter 2024-08-19T12:43:17.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f643f98a640 10 librados: watch_flush exit 2024-08-19T12:43:17.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f643f98a640 20 client.26111.objecter shutdown clearing up homeless session... 2024-08-19T12:43:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.654+0000 7f643f98a640 10 client.26111.objecter successfully canceled tick 2024-08-19T12:43:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.655+0000 7f643f98a640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.104:6800/899887506 conn(0x7f641c041540 legacy=0x7f641c043a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.655+0000 7f643f98a640 10 monclient: shutdown 2024-08-19T12:43:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.655+0000 7f643f98a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.655+0000 7f643f98a640 1 -- 172.21.15.26:0/3875032473 >> v1:172.21.15.26:6790/0 conn(0x7f64380a5920 legacy=0x7f6438137ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.655+0000 7f643df27640 1 -- 172.21.15.26:0/3875032473 reap_dead start 2024-08-19T12:43:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.655+0000 7f643df27640 1 -- 172.21.15.26:0/3875032473 reap_dead start 2024-08-19T12:43:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.655+0000 7f643f98a640 1 -- 172.21.15.26:0/3875032473 shutdown_connections 2024-08-19T12:43:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.655+0000 7f643f98a640 1 -- 172.21.15.26:0/3875032473 >> 172.21.15.26:0/3875032473 conn(0x7f64380a8740 msgr2=0x7f643806dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.655+0000 7f643f98a640 1 -- 172.21.15.26:0/3875032473 shutdown_connections 2024-08-19T12:43:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.655+0000 7f643f98a640 1 -- 172.21.15.26:0/3875032473 wait complete. 2024-08-19T12:43:17.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.655+0000 7f643f98a640 1 librados: shutdown 2024-08-19T12:43:17.692 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:43:17.692 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-08-19T12:43:17.692 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:43:17.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.790+0000 7f86fb820640 10 monclient: get_monmap_and_config 2024-08-19T12:43:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.790+0000 7f86fb820640 10 monclient: build_initial_monmap 2024-08-19T12:43:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.791+0000 7f86fb820640 10 monclient: monmap: 2024-08-19T12:43:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:43:17.791846+0000 2024-08-19T12:43:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:43:17.791846+0000 2024-08-19T12:43:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:17.792 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:17.793 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:17.793 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:17.793 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:17.793 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:17.793 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:17.793 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.791+0000 7f86fb820640 1 Processor -- start 2024-08-19T12:43:17.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.791+0000 7f86fb820640 1 -- start start 2024-08-19T12:43:17.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.791+0000 7f86fb820640 10 monclient: init 2024-08-19T12:43:17.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.792+0000 7f86fb820640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.792+0000 7f86fb820640 10 monclient: _add_conns ranks=[6,7,8,4,2,1,5,0,3] 2024-08-19T12:43:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.792+0000 7f86fb820640 10 monclient(hunting): picked mon.noname-d con 0x7f86f4073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:43:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.792+0000 7f86fb820640 10 monclient(hunting): picked mon.noname-i con 0x7f86f40a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:43:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.792+0000 7f86fb820640 10 monclient(hunting): picked mon.noname-e con 0x7f86f413f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:43:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.792+0000 7f86fb820640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f409dae0 con 0x7f86f4073470 2024-08-19T12:43:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.792+0000 7f86fb820640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f4136910 con 0x7f86f40a57a0 2024-08-19T12:43:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.792+0000 7f86fb820640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f409dd50 con 0x7f86f413f2e0 2024-08-19T12:43:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.792+0000 7f86fb820640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.792+0000 7f86fb820640 10 monclient(hunting): authenticate will time out at 991.879348s 2024-08-19T12:43:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.792+0000 7f86f9dbd640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f86f413f2e0 0x7f86f41416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:58600/0 (socket says 172.21.15.26:58600) 2024-08-19T12:43:17.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.792+0000 7f86f9dbd640 1 -- 172.21.15.26:0/1681281369 learned_addr learned my addr 172.21.15.26:0/1681281369 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:17.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86f95bc640 1 -- 172.21.15.26:0/1681281369 >> v1:172.21.15.26:6792/0 conn(0x7f86f4073470 legacy=0x7f86f4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86f9dbd640 1 -- 172.21.15.26:0/1681281369 >> v1:172.21.15.26:6793/0 conn(0x7f86f413f2e0 legacy=0x7f86f41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86f8dbb640 1 -- 172.21.15.26:0/1681281369 >> v1:172.21.15.104:6792/0 conn(0x7f86f40a57a0 legacy=0x7f86f40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1376887653 0 0) 0x7f86f409dae0 con 0x7f86f4073470 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 10 monclient(hunting): my global_id is 26403 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86e0003520 con 0x7f86f4073470 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2442398901 0 0) 0x7f86f409dd50 con 0x7f86f413f2e0 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 10 monclient(hunting): my global_id is 25937 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86f409dae0 con 0x7f86f413f2e0 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3973832101 0 0) 0x7f86f4136910 con 0x7f86f40a57a0 2024-08-19T12:43:17.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 10 monclient(hunting): my global_id is 134962 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.794+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86f409dd50 con 0x7f86f40a57a0 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3249007576 0 0) 0x7f86e0003520 con 0x7f86f4073470 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 >> v1:172.21.15.26:6793/0 conn(0x7f86f413f2e0 legacy=0x7f86f41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 >> v1:172.21.15.104:6792/0 conn(0x7f86f40a57a0 legacy=0x7f86f40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 10 monclient: _finish_hunting 0 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 1 monclient: found mon.noname-d 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f409e030 con 0x7f86f4073470 2024-08-19T12:43:17.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 10 monclient: _finish_auth 0 2024-08-19T12:43:17.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 10 monclient: _check_auth_tickets 2024-08-19T12:43:17.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:17.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f86f0001b30 con 0x7f86f4073470 2024-08-19T12:43:17.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86fb820640 5 monclient: authenticate success, global_id 26403 2024-08-19T12:43:17.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86fb820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:17.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:43:17.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 10 monclient: dump: 2024-08-19T12:43:17.802 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.802 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.802 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.802 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86fb820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86f0003610 con 0x7f86f4073470 2024-08-19T12:43:17.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 1 -- 172.21.15.26:0/1681281369 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f86f0003b80 con 0x7f86f4073470 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86ebfff640 10 monclient: dump: 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.804 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86fb820640 10 monclient: get_monmap_and_config success 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86fb820640 4 set_mon_vals no callback set 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86fb820640 10 monclient: shutdown 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86fb820640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86fb820640 1 -- 172.21.15.26:0/1681281369 >> v1:172.21.15.26:6792/0 conn(0x7f86f4073470 legacy=0x7f86f4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86fb820640 1 -- 172.21.15.26:0/1681281369 shutdown_connections 2024-08-19T12:43:17.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86fb820640 1 -- 172.21.15.26:0/1681281369 >> 172.21.15.26:0/1681281369 conn(0x7f86f40abff0 msgr2=0x7f86f40ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86fb820640 1 -- 172.21.15.26:0/1681281369 shutdown_connections 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.795+0000 7f86fb820640 1 -- 172.21.15.26:0/1681281369 wait complete. 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.796+0000 7f86fb820640 10 monclient: build_initial_monmap 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.796+0000 7f86fb820640 10 monclient: monmap: 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.796+0000 7f86fb820640 1 librados: starting msgr at 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.796+0000 7f86fb820640 1 librados: starting objecter 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.796+0000 7f86fb820640 1 Processor -- start 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.796+0000 7f86fb820640 1 -- start start 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.796+0000 7f86fb820640 1 librados: setting wanted keys 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.796+0000 7f86fb820640 1 librados: calling monclient init 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.796+0000 7f86fb820640 10 monclient: init 2024-08-19T12:43:17.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86fb820640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:17.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86fb820640 10 monclient: _add_conns ranks=[7,6,3,2,0,1,8,5,4] 2024-08-19T12:43:17.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86fb820640 10 monclient(hunting): picked mon.noname-i con 0x7f86f4073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:43:17.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86fb820640 10 monclient(hunting): picked mon.noname-d con 0x7f86f40a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:43:17.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86fb820640 10 monclient(hunting): picked mon.noname-g con 0x7f86f413f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:43:17.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86fb820640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f412f810 con 0x7f86f413f2e0 2024-08-19T12:43:17.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86fb820640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f4142ba0 con 0x7f86f40a57a0 2024-08-19T12:43:17.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86fb820640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f4131890 con 0x7f86f4073470 2024-08-19T12:43:17.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86fb820640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:17.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86fb820640 10 monclient(hunting): authenticate will time out at 991.884334s 2024-08-19T12:43:17.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86f9dbd640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f86f413f2e0 legacy=0x7f86f41d6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:43:17.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86f8dbb640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f86f40a57a0 0x7f86f412e620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49650/0 (socket says 172.21.15.26:49650) 2024-08-19T12:43:17.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.797+0000 7f86f8dbb640 1 -- 172.21.15.26:0/2033683149 learned_addr learned my addr 172.21.15.26:0/2033683149 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:17.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86f8dbb640 1 -- 172.21.15.26:0/2033683149 >> v1:172.21.15.26:6792/0 conn(0x7f86f40a57a0 legacy=0x7f86f412e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 client.?.objecter ms_handle_connect 0x7f86f40a57a0 2024-08-19T12:43:17.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 client.?.objecter ms_handle_connect 0x7f86f4073470 2024-08-19T12:43:17.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86f95bc640 1 -- 172.21.15.26:0/2033683149 >> v1:172.21.15.104:6792/0 conn(0x7f86f4073470 legacy=0x7f86f412df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4150402760 0 0) 0x7f86f409dd50 con 0x7f86f40a57a0 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 monclient(hunting): my global_id is 26412 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86d8004eb0 con 0x7f86f40a57a0 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546229873 0 0) 0x7f86f4131890 con 0x7f86f4073470 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 monclient(hunting): my global_id is 134971 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86f409dd50 con 0x7f86f4073470 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 999013476 0 0) 0x7f86d8004eb0 con 0x7f86f40a57a0 2024-08-19T12:43:17.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86f4131890 con 0x7f86f40a57a0 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f86e4003610 con 0x7f86f40a57a0 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 monclient(hunting): dump: 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.815 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.799+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1275954088 0 0) 0x7f86f409dd50 con 0x7f86f4073470 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86d8004eb0 con 0x7f86f4073470 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f86f0003b00 con 0x7f86f4073470 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 10 monclient(hunting): dump: 2024-08-19T12:43:17.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.817 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 511253341 0 0) 0x7f86f4131890 con 0x7f86f40a57a0 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 >> v1:172.21.15.104:6792/0 conn(0x7f86f4073470 legacy=0x7f86f412df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 >> v1:172.21.15.104:6790/0 conn(0x7f86f413f2e0 legacy=0x7f86f41d6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 1 monclient: found mon.d 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f4131b70 con 0x7f86f40a57a0 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 10 monclient: _finish_auth 0 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86fb820640 5 monclient: authenticate success, global_id 26412 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86fb820640 10 monclient: _renew_subs 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86fb820640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86fb820640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f86f4135270 con 0x7f86f40a57a0 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86e4003bd0 con 0x7f86f40a57a0 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f86e4004140 con 0x7f86f40a57a0 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86e9ffb640 10 monclient: dump: 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86eb7fe640 4 set_mon_vals no callback set 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86fb820640 10 client.26412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:43:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86fb820640 10 monclient: _renew_subs 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86fb820640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86fb820640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f86f4132330 con 0x7f86f40a57a0 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.800+0000 7f86fb820640 1 librados: init done 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.801+0000 7f86fb820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.801+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f86e40042e0 con 0x7f86f40a57a0 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.801+0000 7f86fb820640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f86f40008d0 con 0x7f86f4142df0 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.801+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(188..188 src has 1..188) ==== 6457+0+0 (unknown 600343228 0 0) 0x7f86e404e610 con 0x7f86f40a57a0 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.801+0000 7f86e9ffb640 10 client.26412.objecter ms_dispatch 0x7f86f4000b90 osd_map(188..188 src has 1..188) 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.801+0000 7f86e9ffb640 3 client.26412.objecter handle_osd_map got epochs [188,188] > 0 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.801+0000 7f86e9ffb640 3 client.26412.objecter handle_osd_map decoding full epoch 188 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.801+0000 7f86e9ffb640 20 client.26412.objecter dump_active .. 0 homeless 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.801+0000 7f86f9dbd640 10 monclient: get_auth_request con 0x7f86d80414e0 auth_method 0 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86f95bc640 1 -- 172.21.15.26:0/2033683149 >> v1:172.21.15.104:6789/0 conn(0x7f86f4142df0 legacy=0x7f86f4133e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 10 client.26412.objecter ms_handle_connect 0x7f86f4142df0 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 10 client.26412.objecter resend_mon_ops 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543680430 0 0) 0x7f86f40008d0 con 0x7f86f4142df0 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 10 monclient(hunting): my global_id is 26794 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86f4131890 con 0x7f86f4142df0 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 10 client.26412.objecter ms_handle_connect 0x7f86d80414e0 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3998675271 0 0) 0x7f86f4131890 con 0x7f86f4142df0 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86f40008d0 con 0x7f86f4142df0 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f86f00015d0 con 0x7f86f4142df0 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.803+0000 7f86e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.804+0000 7f86e9ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.804+0000 7f86e9ffb640 10 monclient: dump: 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.823 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.824 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.824 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.824 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.804+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 735881940 0 0) 0x7f86f40008d0 con 0x7f86f4142df0 2024-08-19T12:43:17.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.804+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86f4131330 con 0x7f86f4142df0 2024-08-19T12:43:17.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.804+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f86f0003610 con 0x7f86f4142df0 2024-08-19T12:43:17.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.804+0000 7f86e9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:17.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.804+0000 7f86e9ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:43:17.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.804+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 >> v1:172.21.15.104:6789/0 conn(0x7f86f4142df0 legacy=0x7f86f4133e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.804+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 >> v1:172.21.15.104:6789/0 conn(0x7f86f4142df0 legacy=0x7f86f4133e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:17.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.810+0000 7f86fb820640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:43:17.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.810+0000 7f86fb820640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f86f402e5b0 con 0x7f86f402e1d0 2024-08-19T12:43:17.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86f95bc640 1 -- 172.21.15.26:0/2033683149 >> v1:172.21.15.104:6789/0 conn(0x7f86f402e1d0 legacy=0x7f86f4134570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 10 client.26412.objecter ms_handle_connect 0x7f86f402e1d0 2024-08-19T12:43:17.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 10 client.26412.objecter resend_mon_ops 2024-08-19T12:43:17.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3599294830 0 0) 0x7f86f402e5b0 con 0x7f86f402e1d0 2024-08-19T12:43:17.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 10 monclient(hunting): my global_id is 26803 2024-08-19T12:43:17.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86f40008d0 con 0x7f86f402e1d0 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 250033894 0 0) 0x7f86f40008d0 con 0x7f86f402e1d0 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86f402e5b0 con 0x7f86f402e1d0 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f86f0005b00 con 0x7f86f402e1d0 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 10 monclient: dump: 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2345376141 0 0) 0x7f86f402e5b0 con 0x7f86f402e1d0 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.813+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f86f406c840 con 0x7f86f402e1d0 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7700 (unknown 0 0 2925722225) 0x7f86f0006f80 con 0x7f86f402e1d0 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86e9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:43:17.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86e9ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:43:17.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 >> v1:172.21.15.104:6789/0 conn(0x7f86f402e1d0 legacy=0x7f86f4134570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86e9ffb640 1 -- 172.21.15.26:0/2033683149 >> v1:172.21.15.104:6789/0 conn(0x7f86f402e1d0 legacy=0x7f86f4134570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:17.829 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:43:17.830 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:43:17.830 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:43:17.830 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:43:17.830 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 28, 2024-08-19T12:43:17.830 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 310817, 2024-08-19T12:43:17.830 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:43:17.830 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:43:17.830 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:43:17.830 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:43:17.830 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:43:17.831 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:43:17.831 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:43:17.831 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:43:17.831 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:43:17.831 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.831 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:43:17.831 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.831 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:43:17.831 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:43:17.831 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.831 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.831 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.832 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:43:17.833 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:43:17.834 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:43:17.834 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:43:17.834 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:43:17.834 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.834 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.834 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:43:17.834 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:43:17.834 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:43:17.834 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:43:17.834 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:43:17.834 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:43:17.834 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:43:17.835 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.835 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:43:17.835 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.835 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.835 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.835 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.835 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.835 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.835 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.835 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.835 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:43:17.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:43:17.837 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:43:17.837 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.837 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.837 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.837 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:43:17.837 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:43:17.838 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.838 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:43:17.838 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:43:17.838 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:43:17.838 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.842 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:43:17.842 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:43:17.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:43:17.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:43:17.843 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:43:17.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:43:17.847 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:43:17.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.849 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:43:17.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:17.850 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:17.850 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:17.850 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:43:17.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:17.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.850 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:43:17.850 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:43:17.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 10 librados: watch_flush enter 2024-08-19T12:43:17.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 10 librados: watch_flush exit 2024-08-19T12:43:17.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 20 client.26412.objecter shutdown clearing up homeless session... 2024-08-19T12:43:17.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 10 client.26412.objecter successfully canceled tick 2024-08-19T12:43:17.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 1 -- 172.21.15.26:0/2033683149 >> v1:172.21.15.104:6800/899887506 conn(0x7f86d80414e0 legacy=0x7f86d80439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 10 monclient: shutdown 2024-08-19T12:43:17.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:17.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 1 -- 172.21.15.26:0/2033683149 >> v1:172.21.15.26:6792/0 conn(0x7f86f40a57a0 legacy=0x7f86f412e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86f9dbd640 1 -- 172.21.15.26:0/2033683149 reap_dead start 2024-08-19T12:43:17.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86f9dbd640 1 -- 172.21.15.26:0/2033683149 reap_dead start 2024-08-19T12:43:17.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 1 -- 172.21.15.26:0/2033683149 shutdown_connections 2024-08-19T12:43:17.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 1 -- 172.21.15.26:0/2033683149 >> 172.21.15.26:0/2033683149 conn(0x7f86f40abff0 msgr2=0x7f86f412c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:17.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 1 -- 172.21.15.26:0/2033683149 shutdown_connections 2024-08-19T12:43:17.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 1 -- 172.21.15.26:0/2033683149 wait complete. 2024-08-19T12:43:17.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.814+0000 7f86fb820640 1 librados: shutdown 2024-08-19T12:43:17.853 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:43:17.853 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-08-19T12:43:17.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:43:17.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.952+0000 7ff506c48640 10 monclient: get_monmap_and_config 2024-08-19T12:43:17.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.952+0000 7ff506c48640 10 monclient: build_initial_monmap 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.952+0000 7ff506c48640 10 monclient: monmap: 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:43:17.953740+0000 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:43:17.953740+0000 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:17.954 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:17.955 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:17.955 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:17.955 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:17.955 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.953+0000 7ff506c48640 1 Processor -- start 2024-08-19T12:43:17.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.953+0000 7ff506c48640 1 -- start start 2024-08-19T12:43:17.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.953+0000 7ff506c48640 10 monclient: init 2024-08-19T12:43:17.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff506c48640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:17.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff506c48640 10 monclient: _add_conns ranks=[3,7,5,2,4,6,0,8,1] 2024-08-19T12:43:17.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff506c48640 10 monclient(hunting): picked mon.noname-g con 0x7ff50006e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:43:17.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff506c48640 10 monclient(hunting): picked mon.noname-i con 0x7ff5000a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:43:17.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff506c48640 10 monclient(hunting): picked mon.noname-h con 0x7ff5000a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:43:17.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff506c48640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff500136790 con 0x7ff50006e7b0 2024-08-19T12:43:17.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff506c48640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff50009ddf0 con 0x7ff5000a5d50 2024-08-19T12:43:17.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff506c48640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff50009e0d0 con 0x7ff5000a5920 2024-08-19T12:43:17.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff506c48640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:17.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff506c48640 10 monclient(hunting): authenticate will time out at 992.041240s 2024-08-19T12:43:17.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff5049e4640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7ff50006e7b0 legacy=0x7ff500136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:43:17.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff5051e5640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7ff5000a5d50 0x7ff500141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:43412/0 (socket says 172.21.15.26:43412) 2024-08-19T12:43:17.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.954+0000 7ff5051e5640 1 -- 172.21.15.26:0/2307803442 learned_addr learned my addr 172.21.15.26:0/2307803442 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:17.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.956+0000 7ff4fffff640 1 -- 172.21.15.26:0/2307803442 >> v1:172.21.15.104:6792/0 conn(0x7ff5000a5920 legacy=0x7ff50013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.956+0000 7ff5051e5640 1 -- 172.21.15.26:0/2307803442 >> v1:172.21.15.104:6791/0 conn(0x7ff5000a5d50 legacy=0x7ff500141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2307803442 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1603240851 0 0) 0x7ff50009e0d0 con 0x7ff5000a5920 2024-08-19T12:43:17.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient(hunting): my global_id is 134980 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2307803442 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4ec003520 con 0x7ff5000a5920 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2307803442 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4028885225 0 0) 0x7ff4f8001380 con 0x7ff5000a5d50 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient(hunting): my global_id is 26123 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2307803442 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff50009e0d0 con 0x7ff5000a5d50 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2307803442 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 827510754 0 0) 0x7ff4ec003520 con 0x7ff5000a5920 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2307803442 >> v1:172.21.15.104:6791/0 conn(0x7ff5000a5d50 legacy=0x7ff500141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2307803442 >> v1:172.21.15.104:6790/0 conn(0x7ff50006e7b0 legacy=0x7ff500136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 1 monclient: found mon.noname-i 2024-08-19T12:43:17.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2307803442 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff50009e3b0 con 0x7ff5000a5920 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient: _finish_auth 0 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff506c48640 5 monclient: authenticate success, global_id 134980 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2307803442 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4e8001b30 con 0x7ff5000a5920 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff506c48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient: dump: 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.963 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.964 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff506c48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2307803442 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff4e8003610 con 0x7ff5000a5920 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff506c48640 10 monclient: get_monmap_and_config success 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 1 -- 172.21.15.26:0/2307803442 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4e8003b80 con 0x7ff5000a5920 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff506c48640 4 set_mon_vals no callback set 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.957+0000 7ff4ff7fe640 10 monclient: dump: 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.965 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.958+0000 7ff506c48640 10 monclient: shutdown 2024-08-19T12:43:17.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.958+0000 7ff506c48640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.958+0000 7ff506c48640 1 -- 172.21.15.26:0/2307803442 >> v1:172.21.15.104:6792/0 conn(0x7ff5000a5920 legacy=0x7ff50013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.958+0000 7ff506c48640 1 -- 172.21.15.26:0/2307803442 shutdown_connections 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.958+0000 7ff506c48640 1 -- 172.21.15.26:0/2307803442 >> 172.21.15.26:0/2307803442 conn(0x7ff5000a8740 msgr2=0x7ff5000a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.958+0000 7ff506c48640 1 -- 172.21.15.26:0/2307803442 shutdown_connections 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.958+0000 7ff506c48640 1 -- 172.21.15.26:0/2307803442 wait complete. 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.958+0000 7ff506c48640 10 monclient: build_initial_monmap 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.958+0000 7ff506c48640 10 monclient: monmap: 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 1 librados: starting msgr at 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 1 librados: starting objecter 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 1 Processor -- start 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 1 -- start start 2024-08-19T12:43:17.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 1 librados: setting wanted keys 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 1 librados: calling monclient init 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 10 monclient: init 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 10 monclient: _add_conns ranks=[7,2,1,6,3,0,5,4,8] 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 10 monclient(hunting): picked mon.noname-i con 0x7ff50006e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 10 monclient(hunting): picked mon.noname-b con 0x7ff5000a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 10 monclient(hunting): picked mon.noname-f con 0x7ff5000a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff500139260 con 0x7ff5000a5d50 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff50013b5c0 con 0x7ff5000a5920 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff50013ad20 con 0x7ff50006e7b0 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff506c48640 10 monclient(hunting): authenticate will time out at 992.046878s 2024-08-19T12:43:17.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff4fffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff5000a5920 0x7ff5001d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48832/0 (socket says 172.21.15.26:48832) 2024-08-19T12:43:17.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.959+0000 7ff4fffff640 1 -- 172.21.15.26:0/1812489162 learned_addr learned my addr 172.21.15.26:0/1812489162 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:17.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fffff640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.26:6790/0 conn(0x7ff5000a5920 legacy=0x7ff5001d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 client.?.objecter ms_handle_connect 0x7ff5000a5920 2024-08-19T12:43:17.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff5051e5640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.104:6789/0 conn(0x7ff5000a5d50 legacy=0x7ff5001d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 client.?.objecter ms_handle_connect 0x7ff5000a5d50 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 client.?.objecter ms_handle_connect 0x7ff50006e7b0 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff5049e4640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.104:6792/0 conn(0x7ff50006e7b0 legacy=0x7ff5001ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2230190299 0 0) 0x7ff50013b5c0 con 0x7ff5000a5920 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): my global_id is 26120 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4e0004eb0 con 0x7ff5000a5920 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2754456456 0 0) 0x7ff500139260 con 0x7ff5000a5d50 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): my global_id is 26812 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff50013b5c0 con 0x7ff5000a5d50 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 767968492 0 0) 0x7ff50013ad20 con 0x7ff50006e7b0 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): my global_id is 134989 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff500139260 con 0x7ff50006e7b0 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 264700895 0 0) 0x7ff4e0004eb0 con 0x7ff5000a5920 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff50013ad20 con 0x7ff5000a5920 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4e8003b00 con 0x7ff5000a5920 2024-08-19T12:43:17.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): dump: 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.978 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1593319542 0 0) 0x7ff500139260 con 0x7ff50006e7b0 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4e0004eb0 con 0x7ff50006e7b0 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4f00048e0 con 0x7ff50006e7b0 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): dump: 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.979 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.980 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1331534343 0 0) 0x7ff50013b5c0 con 0x7ff5000a5d50 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff500139260 con 0x7ff5000a5d50 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4f80012b0 con 0x7ff5000a5d50 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient(hunting): dump: 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.981 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3257307684 0 0) 0x7ff50013ad20 con 0x7ff5000a5920 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.104:6792/0 conn(0x7ff50006e7b0 legacy=0x7ff5001ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.104:6789/0 conn(0x7ff5000a5d50 legacy=0x7ff5001d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient: _finish_hunting 0 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 monclient: found mon.b 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff500139dc0 con 0x7ff5000a5920 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient: _finish_auth 0 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient: _check_auth_tickets 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff506c48640 5 monclient: authenticate success, global_id 26120 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff506c48640 10 monclient: _renew_subs 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff506c48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff506c48640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff50013be40 con 0x7ff5000a5920 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff506c48640 10 client.26120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:43:17.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff506c48640 10 monclient: _renew_subs 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff506c48640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff506c48640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff500138a40 con 0x7ff5000a5920 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff506c48640 1 librados: init done 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff4e8001ad0 con 0x7ff5000a5920 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4e80057c0 con 0x7ff5000a5920 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4feffd640 4 set_mon_vals no callback set 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.962+0000 7ff4fd7fa640 10 monclient: dump: 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.963+0000 7ff506c48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.963+0000 7ff506c48640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff4e40065c0 con 0x7ff4e4002860 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.963+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff4e8011c20 con 0x7ff5000a5920 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.964+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(188..188 src has 1..188) ==== 6457+0+0 (unknown 600343228 0 0) 0x7ff4e804e610 con 0x7ff5000a5920 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.964+0000 7ff4fd7fa640 10 client.26120.objecter ms_dispatch 0x7ff500000b90 osd_map(188..188 src has 1..188) 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.964+0000 7ff4fd7fa640 3 client.26120.objecter handle_osd_map got epochs [188,188] > 0 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.964+0000 7ff4fd7fa640 3 client.26120.objecter handle_osd_map decoding full epoch 188 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.964+0000 7ff4fd7fa640 20 client.26120.objecter dump_active .. 0 homeless 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.964+0000 7ff5051e5640 10 monclient: get_auth_request con 0x7ff4e00419d0 auth_method 0 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff5049e4640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.104:6791/0 conn(0x7ff4e4002860 legacy=0x7ff4e4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 10 client.26120.objecter ms_handle_connect 0x7ff4e4002860 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 10 client.26120.objecter resend_mon_ops 2024-08-19T12:43:17.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3317384489 0 0) 0x7ff4e40065c0 con 0x7ff4e4002860 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 10 monclient(hunting): my global_id is 26132 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff50013ad20 con 0x7ff4e4002860 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 10 client.26120.objecter ms_handle_connect 0x7ff4e00419d0 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3404456179 0 0) 0x7ff50013ad20 con 0x7ff4e4002860 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4e40065c0 con 0x7ff4e4002860 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4f0005140 con 0x7ff4e4002860 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 10 monclient: dump: 2024-08-19T12:43:17.987 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.988 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.989 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3754997065 0 0) 0x7ff4e40065c0 con 0x7ff4e4002860 2024-08-19T12:43:17.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.966+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4e400a160 con 0x7ff4e4002860 2024-08-19T12:43:17.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.967+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff4f0007400 con 0x7ff4e4002860 2024-08-19T12:43:17.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.967+0000 7ff4fd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:17.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.967+0000 7ff4fd7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:43:17.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.967+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.104:6791/0 conn(0x7ff4e4002860 legacy=0x7ff4e4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.967+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.104:6791/0 conn(0x7ff4e4002860 legacy=0x7ff4e4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:17.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.973+0000 7ff506c48640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:43:17.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.973+0000 7ff506c48640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff4e4009e70 con 0x7ff4e4009130 2024-08-19T12:43:17.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.975+0000 7ff5049e4640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.104:6791/0 conn(0x7ff4e4009130 legacy=0x7ff4e40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:17.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.975+0000 7ff4fd7fa640 10 client.26120.objecter ms_handle_connect 0x7ff4e4009130 2024-08-19T12:43:17.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.975+0000 7ff4fd7fa640 10 client.26120.objecter resend_mon_ops 2024-08-19T12:43:17.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 822132020 0 0) 0x7ff4e4009e70 con 0x7ff4e4009130 2024-08-19T12:43:17.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:43:17.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:17.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:17.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 10 monclient(hunting): my global_id is 26141 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4e40065c0 con 0x7ff4e4009130 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3642064474 0 0) 0x7ff4e40065c0 con 0x7ff4e4009130 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4e4009e70 con 0x7ff4e4009130 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4f0006c70 con 0x7ff4e4009130 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 10 monclient: dump: 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:17.992 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3769914602 0 0) 0x7ff4e4009e70 con 0x7ff4e4009130 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff4e400c3f0 con 0x7ff4e4009130 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7700 (unknown 0 0 586477613) 0x7ff4f0006e90 con 0x7ff4e4009130 2024-08-19T12:43:17.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:43:17.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:43:17.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.104:6791/0 conn(0x7ff4e4009130 legacy=0x7ff4e40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:17.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.976+0000 7ff4fd7fa640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.104:6791/0 conn(0x7ff4e4009130 legacy=0x7ff4e40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:17.994 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:43:17.994 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:43:17.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:43:17.994 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:43:17.994 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 28, 2024-08-19T12:43:17.994 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 310999, 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.995 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:17.996 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:17.997 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:17.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:17.998 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:43:17.998 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:43:17.998 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:43:17.998 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:43:17.998 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:43:17.998 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:43:17.998 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.998 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:17.998 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:43:17.998 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:43:17.998 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:17.999 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:43:18.002 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:43:18.002 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.002 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:43:18.002 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.002 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:43:18.002 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:43:18.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:43:18.004 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.005 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:43:18.008 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:43:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-08-19T12:43:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:43:18.014 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:43:18.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 10 librados: watch_flush enter 2024-08-19T12:43:18.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 10 librados: watch_flush exit 2024-08-19T12:43:18.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 20 client.26120.objecter shutdown clearing up homeless session... 2024-08-19T12:43:18.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 10 client.26120.objecter successfully canceled tick 2024-08-19T12:43:18.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.104:6800/899887506 conn(0x7ff4e00419d0 legacy=0x7ff4e0043e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:18.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 10 monclient: shutdown 2024-08-19T12:43:18.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:18.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 1 -- 172.21.15.26:0/1812489162 >> v1:172.21.15.26:6790/0 conn(0x7ff5000a5920 legacy=0x7ff5001d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:18.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff5051e5640 1 -- 172.21.15.26:0/1812489162 reap_dead start 2024-08-19T12:43:18.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff5051e5640 1 -- 172.21.15.26:0/1812489162 reap_dead start 2024-08-19T12:43:18.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 1 -- 172.21.15.26:0/1812489162 shutdown_connections 2024-08-19T12:43:18.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 1 -- 172.21.15.26:0/1812489162 >> 172.21.15.26:0/1812489162 conn(0x7ff5000a8740 msgr2=0x7ff5000aa640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:18.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 1 -- 172.21.15.26:0/1812489162 shutdown_connections 2024-08-19T12:43:18.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 1 -- 172.21.15.26:0/1812489162 wait complete. 2024-08-19T12:43:18.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:17.977+0000 7ff506c48640 1 librados: shutdown 2024-08-19T12:43:18.017 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:43:18.018 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-08-19T12:43:18.018 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:43:18.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.118+0000 7fc6da721640 10 monclient: get_monmap_and_config 2024-08-19T12:43:18.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.118+0000 7fc6da721640 10 monclient: build_initial_monmap 2024-08-19T12:43:18.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.118+0000 7fc6da721640 10 monclient: monmap: 2024-08-19T12:43:18.119 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:18.119 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:18.119 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:43:18.119025+0000 2024-08-19T12:43:18.119 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:43:18.119025+0000 2024-08-19T12:43:18.119 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:18.119 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:18.119 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:18.119 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:18.119 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:18.120 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:18.120 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:18.120 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:18.120 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:18.120 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:18.120 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:18.120 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:18.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.118+0000 7fc6da721640 1 Processor -- start 2024-08-19T12:43:18.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.118+0000 7fc6da721640 1 -- start start 2024-08-19T12:43:18.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.118+0000 7fc6da721640 10 monclient: init 2024-08-19T12:43:18.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6da721640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:18.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6da721640 10 monclient: _add_conns ranks=[2,4,3,6,0,8,5,7,1] 2024-08-19T12:43:18.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6da721640 10 monclient(hunting): picked mon.noname-b con 0x7fc6d4073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:43:18.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6da721640 10 monclient(hunting): picked mon.noname-c con 0x7fc6d40a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:43:18.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6da721640 10 monclient(hunting): picked mon.noname-g con 0x7fc6d413f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:43:18.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6da721640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d409dae0 con 0x7fc6d4073470 2024-08-19T12:43:18.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6da721640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d4136910 con 0x7fc6d413f2e0 2024-08-19T12:43:18.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6da721640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d409dd50 con 0x7fc6d40a57a0 2024-08-19T12:43:18.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6da721640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:18.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6da721640 10 monclient(hunting): authenticate will time out at 992.206453s 2024-08-19T12:43:18.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6d37fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fc6d40a57a0 0x7fc6d40a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:35488/0 (socket says 172.21.15.26:35488) 2024-08-19T12:43:18.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6d37fe640 1 -- 172.21.15.26:0/605777998 learned_addr learned my addr 172.21.15.26:0/605777998 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:18.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6d3fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc6d4073470 0x7fc6d4136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:48846/0 (socket says 172.21.15.26:48846) 2024-08-19T12:43:18.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.119+0000 7fc6d8cbe640 1 -- 172.21.15.26:0/605777998 >> v1:172.21.15.104:6790/0 conn(0x7fc6d413f2e0 legacy=0x7fc6d41416d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:43:18.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.121+0000 7fc6d37fe640 1 -- 172.21.15.26:0/605777998 >> v1:172.21.15.26:6791/0 conn(0x7fc6d40a57a0 legacy=0x7fc6d40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.121+0000 7fc6d3fff640 1 -- 172.21.15.26:0/605777998 >> v1:172.21.15.26:6790/0 conn(0x7fc6d4073470 legacy=0x7fc6d4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 1 -- 172.21.15.26:0/605777998 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2955006997 0 0) 0x7fc6d409dae0 con 0x7fc6d4073470 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient(hunting): my global_id is 26129 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 1 -- 172.21.15.26:0/605777998 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6bc003520 con 0x7fc6d4073470 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 1 -- 172.21.15.26:0/605777998 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2888876768 0 0) 0x7fc6d409dd50 con 0x7fc6d40a57a0 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient(hunting): my global_id is 26383 2024-08-19T12:43:18.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 1 -- 172.21.15.26:0/605777998 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6d409dae0 con 0x7fc6d40a57a0 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 1 -- 172.21.15.26:0/605777998 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1812715860 0 0) 0x7fc6d409dae0 con 0x7fc6d40a57a0 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 1 -- 172.21.15.26:0/605777998 >> v1:172.21.15.104:6790/0 conn(0x7fc6d413f2e0 legacy=0x7fc6d41416d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 1 -- 172.21.15.26:0/605777998 >> v1:172.21.15.26:6790/0 conn(0x7fc6d4073470 legacy=0x7fc6d4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 1 monclient: found mon.noname-c 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 1 -- 172.21.15.26:0/605777998 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d409e030 con 0x7fc6d40a57a0 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient: _finish_auth 0 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 1 -- 172.21.15.26:0/605777998 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc6c8001ac0 con 0x7fc6d40a57a0 2024-08-19T12:43:18.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6da721640 5 monclient: authenticate success, global_id 26383 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6da721640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient: dump: 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:18.128 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6da721640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 1 -- 172.21.15.26:0/605777998 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc6c80048a0 con 0x7fc6d40a57a0 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6da721640 10 monclient: get_monmap_and_config success 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 1 -- 172.21.15.26:0/605777998 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc6c8004e10 con 0x7fc6d40a57a0 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6da721640 4 set_mon_vals no callback set 2024-08-19T12:43:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.122+0000 7fc6d2ffd640 10 monclient: dump: 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:18.130 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.123+0000 7fc6da721640 10 monclient: shutdown 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.123+0000 7fc6da721640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.123+0000 7fc6da721640 1 -- 172.21.15.26:0/605777998 >> v1:172.21.15.26:6791/0 conn(0x7fc6d40a57a0 legacy=0x7fc6d40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.123+0000 7fc6da721640 1 -- 172.21.15.26:0/605777998 shutdown_connections 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.123+0000 7fc6da721640 1 -- 172.21.15.26:0/605777998 >> 172.21.15.26:0/605777998 conn(0x7fc6d40abff0 msgr2=0x7fc6d40ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.123+0000 7fc6da721640 1 -- 172.21.15.26:0/605777998 shutdown_connections 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.123+0000 7fc6da721640 1 -- 172.21.15.26:0/605777998 wait complete. 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.123+0000 7fc6da721640 10 monclient: build_initial_monmap 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.123+0000 7fc6da721640 10 monclient: monmap: 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:43:18.131 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.123+0000 7fc6da721640 1 librados: starting msgr at 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.123+0000 7fc6da721640 1 librados: starting objecter 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.123+0000 7fc6da721640 1 Processor -- start 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 1 -- start start 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 1 librados: setting wanted keys 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 1 librados: calling monclient init 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 10 monclient: init 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 10 monclient: _reopen_session rank -1 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 10 monclient: _add_conns ranks=[6,4,7,2,3,1,8,5,0] 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 10 monclient(hunting): picked mon.noname-d con 0x7fc6d4073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 10 monclient(hunting): picked mon.noname-c con 0x7fc6d40a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 10 monclient(hunting): picked mon.noname-i con 0x7fc6d413f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d4131850 con 0x7fc6d40a57a0 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d40aa1f0 con 0x7fc6d4073470 2024-08-19T12:43:18.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d412e8f0 con 0x7fc6d413f2e0 2024-08-19T12:43:18.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 10 monclient(hunting): _renew_subs 2024-08-19T12:43:18.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6da721640 10 monclient(hunting): authenticate will time out at 992.211662s 2024-08-19T12:43:18.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6d37fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fc6d40a57a0 0x7fc6d412e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:35498/0 (socket says 172.21.15.26:35498) 2024-08-19T12:43:18.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.124+0000 7fc6d37fe640 1 -- 172.21.15.26:0/1196157616 learned_addr learned my addr 172.21.15.26:0/1196157616 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:43:18.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.126+0000 7fc6d3fff640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.26:6792/0 conn(0x7fc6d4073470 legacy=0x7fc6d412dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.126+0000 7fc6d37fe640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.26:6791/0 conn(0x7fc6d40a57a0 legacy=0x7fc6d412e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.126+0000 7fc6d0ff9640 10 client.?.objecter ms_handle_connect 0x7fc6d4073470 2024-08-19T12:43:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.126+0000 7fc6d0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.126+0000 7fc6d0ff9640 10 client.?.objecter ms_handle_connect 0x7fc6d40a57a0 2024-08-19T12:43:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.126+0000 7fc6d0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1899152016 0 0) 0x7fc6d4131850 con 0x7fc6d40a57a0 2024-08-19T12:43:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:43:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): my global_id is 26392 2024-08-19T12:43:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6c0004eb0 con 0x7fc6d40a57a0 2024-08-19T12:43:18.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 781796100 0 0) 0x7fc6d40aa1f0 con 0x7fc6d4073470 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): my global_id is 26421 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6d4131850 con 0x7fc6d4073470 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 client.?.objecter ms_handle_connect 0x7fc6d413f2e0 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d8cbe640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.104:6792/0 conn(0x7fc6d413f2e0 legacy=0x7fc6d41d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1488156491 0 0) 0x7fc6d412e8f0 con 0x7fc6d413f2e0 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): my global_id is 134998 2024-08-19T12:43:18.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6d40aa1f0 con 0x7fc6d413f2e0 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4140824736 0 0) 0x7fc6c0004eb0 con 0x7fc6d40a57a0 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6d412e8f0 con 0x7fc6d40a57a0 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 498362641 0 0) 0x7fc6d4131850 con 0x7fc6d4073470 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6c0004eb0 con 0x7fc6d4073470 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc6c8004d90 con 0x7fc6d40a57a0 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): dump: 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.141 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc6c40047e0 con 0x7fc6d4073470 2024-08-19T12:43:18.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient(hunting): dump: 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:18.143 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 969346809 0 0) 0x7fc6d40aa1f0 con 0x7fc6d413f2e0 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6d4131850 con 0x7fc6d413f2e0 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 908958183 0 0) 0x7fc6d412e8f0 con 0x7fc6d40a57a0 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.104:6792/0 conn(0x7fc6d413f2e0 legacy=0x7fc6d41d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.26:6792/0 conn(0x7fc6d4073470 legacy=0x7fc6d412dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 monclient: found mon.c 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d4130320 con 0x7fc6d40a57a0 2024-08-19T12:43:18.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient: _finish_auth 0 2024-08-19T12:43:18.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:43:18.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:43:18.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6da721640 5 monclient: authenticate success, global_id 26392 2024-08-19T12:43:18.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6da721640 10 monclient: _renew_subs 2024-08-19T12:43:18.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6da721640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:43:18.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6da721640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d413f050 con 0x7fc6d40a57a0 2024-08-19T12:43:18.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6da721640 10 client.26392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:43:18.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6da721640 10 monclient: _renew_subs 2024-08-19T12:43:18.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6da721640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:43:18.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6da721640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fc6d41330e0 con 0x7fc6d40a57a0 2024-08-19T12:43:18.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6da721640 1 librados: init done 2024-08-19T12:43:18.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc6c8001ae0 con 0x7fc6d40a57a0 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc6c80048a0 con 0x7fc6d40a57a0 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d27fc640 4 set_mon_vals no callback set 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.127+0000 7fc6d0ff9640 10 monclient: dump: 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:18.146 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.128+0000 7fc6da721640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.128+0000 7fc6da721640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc6b40065c0 con 0x7fc6b4002860 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.128+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fc6c8011a00 con 0x7fc6d40a57a0 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.128+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(188..188 src has 1..188) ==== 6457+0+0 (unknown 600343228 0 0) 0x7fc6c804e570 con 0x7fc6d40a57a0 2024-08-19T12:43:18.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.128+0000 7fc6d0ff9640 10 client.26392.objecter ms_dispatch 0x7fc6d4000b90 osd_map(188..188 src has 1..188) 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.128+0000 7fc6d0ff9640 3 client.26392.objecter handle_osd_map got epochs [188,188] > 0 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.128+0000 7fc6d0ff9640 3 client.26392.objecter handle_osd_map decoding full epoch 188 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.128+0000 7fc6d0ff9640 20 client.26392.objecter dump_active .. 0 homeless 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.129+0000 7fc6d8cbe640 10 monclient: get_auth_request con 0x7fc6c0041990 auth_method 0 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d3fff640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.104:6792/0 conn(0x7fc6b4002860 legacy=0x7fc6b4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 10 client.26392.objecter ms_handle_connect 0x7fc6b4002860 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 10 client.26392.objecter resend_mon_ops 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4191504129 0 0) 0x7fc6b40065c0 con 0x7fc6b4002860 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 10 monclient(hunting): my global_id is 135007 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6d412e8f0 con 0x7fc6b4002860 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 10 client.26392.objecter ms_handle_connect 0x7fc6c0041990 2024-08-19T12:43:18.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3433998958 0 0) 0x7fc6d412e8f0 con 0x7fc6b4002860 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6b40065c0 con 0x7fc6b4002860 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc6c40051a0 con 0x7fc6b4002860 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 10 monclient: dump: 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:18.149 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2970894467 0 0) 0x7fc6b40065c0 con 0x7fc6b4002860 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.131+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6b400a160 con 0x7fc6b4002860 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.132+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc6c4007440 con 0x7fc6b4002860 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.132+0000 7fc6d0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.132+0000 7fc6d0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:43:18.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.132+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.104:6792/0 conn(0x7fc6b4002860 legacy=0x7fc6b4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:18.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.132+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.104:6792/0 conn(0x7fc6b4002860 legacy=0x7fc6b4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:18.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.138+0000 7fc6da721640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:43:18.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.138+0000 7fc6da721640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc6b4009e70 con 0x7fc6b4009130 2024-08-19T12:43:18.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.140+0000 7fc6d3fff640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.104:6792/0 conn(0x7fc6b4009130 legacy=0x7fc6b40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:43:18.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.140+0000 7fc6d0ff9640 10 client.26392.objecter ms_handle_connect 0x7fc6b4009130 2024-08-19T12:43:18.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.140+0000 7fc6d0ff9640 10 client.26392.objecter resend_mon_ops 2024-08-19T12:43:18.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.140+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3019394493 0 0) 0x7fc6b4009e70 con 0x7fc6b4009130 2024-08-19T12:43:18.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.140+0000 7fc6d0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:43:18.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.140+0000 7fc6d0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.140+0000 7fc6d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.140+0000 7fc6d0ff9640 10 monclient(hunting): my global_id is 135016 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.140+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6b40065c0 con 0x7fc6b4009130 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4122408735 0 0) 0x7fc6b40065c0 con 0x7fc6b4009130 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6b4009e70 con 0x7fc6b4009130 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc6c4006cb0 con 0x7fc6b4009130 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 10 monclient: dump: 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:43:18.153 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1355869277 0 0) 0x7fc6b4009e70 con 0x7fc6b4009130 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc6b400c410 con 0x7fc6b4009130 2024-08-19T12:43:18.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 2325092353) 0x7fc6c4006f10 con 0x7fc6b4009130 2024-08-19T12:43:18.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:43:18.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:43:18.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.104:6792/0 conn(0x7fc6b4009130 legacy=0x7fc6b40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:18.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.141+0000 7fc6d0ff9640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.104:6792/0 conn(0x7fc6b4009130 legacy=0x7fc6b40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:43:18.155 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:43:18.155 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:43:18.155 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:43:18.155 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:43:18.155 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 28, 2024-08-19T12:43:18.155 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 168499, 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:18.156 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:18.157 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:18.158 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:43:18.159 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:43:18.160 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.161 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:43:18.162 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.163 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.165 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:43:18.166 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.167 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:43:18.170 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.171 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:18.173 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:43:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:43:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:43:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:43:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:43:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:43:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:43:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:43:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:43:18.174 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:43:18.174 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:43:18.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 10 librados: watch_flush enter 2024-08-19T12:43:18.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 10 librados: watch_flush exit 2024-08-19T12:43:18.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 20 client.26392.objecter shutdown clearing up homeless session... 2024-08-19T12:43:18.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 10 client.26392.objecter successfully canceled tick 2024-08-19T12:43:18.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.104:6800/899887506 conn(0x7fc6c0041990 legacy=0x7fc6c0043e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:18.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 10 monclient: shutdown 2024-08-19T12:43:18.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:43:18.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 1 -- 172.21.15.26:0/1196157616 >> v1:172.21.15.26:6791/0 conn(0x7fc6d40a57a0 legacy=0x7fc6d412e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:43:18.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6d8cbe640 1 -- 172.21.15.26:0/1196157616 reap_dead start 2024-08-19T12:43:18.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6d8cbe640 1 -- 172.21.15.26:0/1196157616 reap_dead start 2024-08-19T12:43:18.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 1 -- 172.21.15.26:0/1196157616 shutdown_connections 2024-08-19T12:43:18.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 1 -- 172.21.15.26:0/1196157616 >> 172.21.15.26:0/1196157616 conn(0x7fc6d40abff0 msgr2=0x7fc6d412c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:43:18.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 1 -- 172.21.15.26:0/1196157616 shutdown_connections 2024-08-19T12:43:18.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 1 -- 172.21.15.26:0/1196157616 wait complete. 2024-08-19T12:43:18.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:43:18.142+0000 7fc6da721640 1 librados: shutdown 2024-08-19T12:43:18.177 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:43:18.178 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T12:43:42.594 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-08-19T12:43:42.595 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-08-19T12:43:42.595 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-08-19T12:43:42.595 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: seed 24748 2024-08-19T12:43:42.595 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-08-19T12:43:42.595 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (233 ms) 2024-08-19T12:43:42.595 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-08-19T12:43:42.595 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10774 ms) 2024-08-19T12:43:42.595 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-08-19T12:43:42.595 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (27 ms) 2024-08-19T12:43:42.595 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (11034 ms total) 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7584 ms) 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7440 ms) 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10190 ms) 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: my_snaps [3] 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: my_snaps [4,3] 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: my_snaps [5,4,3] 2024-08-19T12:43:42.596 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: promoting some heads 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: promoting from clones for snap 6 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: promoting from clones for snap 5 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: promoting from clones for snap 4 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: promoting from clones for snap 3 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: waiting for scrubs... 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: done waiting 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (47245 ms) 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9722 ms) 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8732 ms) 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-08-19T12:43:42.597 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8786 ms) 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8191 ms) 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9802 ms) 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9341 ms) 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10027 ms) 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13421 ms) 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (12999 ms) 2024-08-19T12:43:42.598 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-08-19T12:43:42.599 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (16538 ms) 2024-08-19T12:43:42.599 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-08-19T12:43:42.599 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (32747 ms) 2024-08-19T12:43:42.599 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-08-19T12:43:42.599 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (9328 ms) 2024-08-19T12:43:42.599 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-08-19T12:43:42.599 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8502 ms) 2024-08-19T12:43:57.895 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-08-19T12:43:57.895 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: hmm, no HitSet yet 2024-08-19T12:43:57.895 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: ok, hit_set contains 261:602f83fe:::foo:head 2024-08-19T12:43:57.895 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8992 ms) 2024-08-19T12:43:57.895 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-08-19T12:43:57.895 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg_num = 32 2024-08-19T12:43:57.895 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 0 ls 1724071438,0 2024-08-19T12:43:57.895 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 1 ls 1724071438,0 2024-08-19T12:43:57.895 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 2 ls 1724071438,0 2024-08-19T12:43:57.895 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 3 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 4 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 5 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 6 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 7 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 8 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 9 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 10 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 11 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 12 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 13 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 14 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 15 ls 1724071438,0 2024-08-19T12:43:57.896 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 16 ls 1724071438,0 2024-08-19T12:43:57.897 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 17 ls 1724071438,0 2024-08-19T12:43:57.897 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 18 ls 1724071438,0 2024-08-19T12:43:57.897 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 19 ls 1724071438,0 2024-08-19T12:43:57.897 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 20 ls 1724071438,0 2024-08-19T12:43:57.897 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 21 ls 1724071438,0 2024-08-19T12:43:57.897 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 22 ls 1724071438,0 2024-08-19T12:43:57.897 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 23 ls 1724071438,0 2024-08-19T12:43:57.897 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 24 ls 1724071438,0 2024-08-19T12:43:57.897 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 25 ls 1724071438,0 2024-08-19T12:43:57.897 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 26 ls 1724071438,0 2024-08-19T12:43:57.897 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 27 ls 1724071438,0 2024-08-19T12:43:57.898 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 28 ls 1724071438,0 2024-08-19T12:43:57.898 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 29 ls 1724071438,0 2024-08-19T12:43:57.898 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 30 ls 1724071438,0 2024-08-19T12:43:57.898 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg 31 ls 1724071438,0 2024-08-19T12:43:57.898 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: pg_num = 32 2024-08-19T12:43:57.898 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:6cac518f:::0:head 2024-08-19T12:43:57.898 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:02547ec2:::1:head 2024-08-19T12:43:57.898 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:f905c69b:::2:head 2024-08-19T12:43:57.898 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:cfc208b3:::3:head 2024-08-19T12:43:57.898 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:d83876eb:::4:head 2024-08-19T12:43:57.898 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:b29083e3:::5:head 2024-08-19T12:43:57.899 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:c4fdafeb:::6:head 2024-08-19T12:43:57.899 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:5c6b0b28:::7:head 2024-08-19T12:43:57.899 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:bd63b0f1:::8:head 2024-08-19T12:43:57.899 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:e960b815:::9:head 2024-08-19T12:43:57.899 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:52ea6a34:::10:head 2024-08-19T12:43:57.899 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:89d3ae78:::11:head 2024-08-19T12:43:57.899 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:de5d7c5f:::12:head 2024-08-19T12:43:57.899 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:566253c9:::13:head 2024-08-19T12:43:57.899 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:62a1935d:::14:head 2024-08-19T12:43:57.899 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:863748b0:::15:head 2024-08-19T12:43:57.899 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:3958e169:::16:head 2024-08-19T12:43:57.899 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:4d4dabf9:::17:head 2024-08-19T12:43:57.900 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:8391935d:::18:head 2024-08-19T12:43:57.900 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:28883081:::19:head 2024-08-19T12:43:57.900 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:69259c59:::20:head 2024-08-19T12:43:57.900 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:4bdb80b7:::21:head 2024-08-19T12:43:57.900 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:a11c5d71:::22:head 2024-08-19T12:43:57.900 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:271af37b:::23:head 2024-08-19T12:43:57.900 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:95b121be:::24:head 2024-08-19T12:43:57.900 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:58d1031b:::25:head 2024-08-19T12:43:57.900 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:0a050783:::26:head 2024-08-19T12:43:57.900 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:c709704c:::27:head 2024-08-19T12:43:57.900 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:cbe56eaf:::28:head 2024-08-19T12:43:57.900 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:86b4b162:::29:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:70d89383:::30:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:dd450c7c:::31:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:6d5729b1:::32:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:c388f3fb:::33:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:56cfea31:::34:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:9dbc1bf7:::35:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:40b74ccd:::36:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:4d5aaf42:::37:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:920f362c:::38:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:6cc53222:::39:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:9cad833f:::40:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:1ea84d41:::41:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:c4480ef6:::42:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:a694361e:::43:head 2024-08-19T12:43:57.901 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:d1bd33e9:::44:head 2024-08-19T12:43:57.902 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:ddc2cd5d:::45:head 2024-08-19T12:43:57.902 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:2b782207:::46:head 2024-08-19T12:43:57.902 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:7b187fca:::47:head 2024-08-19T12:43:57.902 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:90ecdf6f:::48:head 2024-08-19T12:43:57.902 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:a5ed95fe:::49:head 2024-08-19T12:43:57.902 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:ea0eaa55:::50:head 2024-08-19T12:43:57.902 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:f33ef17b:::51:head 2024-08-19T12:43:57.902 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:a0d1b2f6:::52:head 2024-08-19T12:43:57.902 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:60c5229e:::53:head 2024-08-19T12:43:57.902 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:edcbc575:::54:head 2024-08-19T12:43:57.902 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:102cf253:::55:head 2024-08-19T12:43:57.902 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:efb7fb0b:::56:head 2024-08-19T12:43:57.903 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:50d0a326:::57:head 2024-08-19T12:43:57.903 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:d4dc5daf:::58:head 2024-08-19T12:43:57.903 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:3a130462:::59:head 2024-08-19T12:43:57.903 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:ec87ed71:::60:head 2024-08-19T12:43:57.903 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:d5bc9454:::61:head 2024-08-19T12:43:57.903 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:3ddfe313:::62:head 2024-08-19T12:43:57.903 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:7c2816b9:::63:head 2024-08-19T12:43:57.903 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:47e00e4d:::64:head 2024-08-19T12:43:57.903 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:c6410c18:::65:head 2024-08-19T12:43:57.903 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:b48ed237:::66:head 2024-08-19T12:43:57.903 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:cd63ad31:::67:head 2024-08-19T12:43:57.903 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:b179e92b:::68:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:0d9f741a:::69:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:6d3352ae:::70:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:c6d5c19e:::71:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:bc4729c3:::72:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:77e930b9:::73:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:0abeecfd:::74:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:b7c37e15:::75:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:b6378398:::76:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:02bd68de:::77:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:cc795d2d:::78:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:630d4fea:::79:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:e0d29ef5:::80:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:fd6f13d2:::81:head 2024-08-19T12:43:57.904 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:606461d5:::82:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:eadbdc43:::83:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:8761d0bb:::84:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:9ef0186f:::85:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:e0d41294:::86:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:961de695:::87:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:1423148f:::88:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:633a8fa2:::89:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:a8653809:::90:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:3dac8b33:::91:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:35aad435:::92:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:f6dcc343:::93:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:dbbdad87:::94:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:1cb48ce0:::95:head 2024-08-19T12:43:57.905 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:03cd461c:::96:head 2024-08-19T12:43:57.906 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:17a4ea99:::97:head 2024-08-19T12:43:57.906 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:9993c9a7:::98:head 2024-08-19T12:43:57.906 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:6394211c:::99:head 2024-08-19T12:43:57.906 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:94c7ae57:::100:head 2024-08-19T12:43:57.906 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:6fdee5bb:::101:head 2024-08-19T12:43:57.906 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:9a477fd1:::102:head 2024-08-19T12:43:57.906 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:eb850916:::103:head 2024-08-19T12:43:57.906 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:affc56b9:::104:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:b42dc814:::105:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:f319f8f0:::106:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:9a40b9de:::107:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:8b524f28:::108:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:e3de589f:::109:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:90f90a5b:::110:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:a7b4f1d7:::111:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:af51766e:::112:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:b6f90bd1:::113:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:e0261208:::114:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:c9569ef7:::115:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:61bebe50:::116:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:fe93412b:::117:head 2024-08-19T12:43:57.907 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:d3d38bee:::118:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:3100ba0c:::119:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:d0560ada:::120:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:f0ea8b35:::121:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:766f231a:::122:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:a07a2582:::123:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:bd7c6b3a:::124:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:fb2ddaff:::125:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:4408e1fe:::126:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:ee1df7a7:::127:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:c3002909:::128:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:4f48ffa9:::129:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:edf38733:::130:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:c08425c0:::131:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:5f902d98:::132:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:41ea2c93:::133:head 2024-08-19T12:43:57.908 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:813cee13:::134:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:0131818d:::135:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:26ba5a85:::136:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:381b8a5a:::137:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:28797e47:::138:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:bfca7f22:::139:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:36807075:::140:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:80b03975:::141:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:5c15709b:::142:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:f39ea15e:::143:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:ea992956:::144:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:48887b1c:::145:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:9f24a9dd:::146:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:987f100b:::147:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:d2dd3581:::148:head 2024-08-19T12:43:57.909 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:7fed1808:::149:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:c80b70e9:::150:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:85ed90f9:::151:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:36428b24:::152:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:d044c34a:::153:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:7c18bf58:::154:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:d1c21232:::155:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:a7a3c575:::156:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:87da0633:::157:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:d5ac3822:::158:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:3f20522d:::159:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:6ca26563:::160:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:532ce135:::161:head 2024-08-19T12:43:57.910 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:c78863e6:::162:head 2024-08-19T12:44:15.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:7462ddf6:: 2024-08-19T12:44:15.324 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6403 ms) 2024-08-19T12:44:15.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-08-19T12:44:15.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (8007 ms) 2024-08-19T12:44:15.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-08-19T12:44:15.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6986 ms) 2024-08-19T12:44:15.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-08-19T12:44:15.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (8019 ms) 2024-08-19T12:44:15.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-08-19T12:44:15.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7400 ms) 2024-08-19T12:44:15.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-08-19T12:44:15.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (12003 ms) 2024-08-19T12:44:15.325 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-08-19T12:44:15.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7035 ms) 2024-08-19T12:44:15.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-08-19T12:44:15.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (12655 ms) 2024-08-19T12:44:15.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-08-19T12:44:15.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (26430 ms) 2024-08-19T12:44:15.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-08-19T12:44:15.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7438 ms) 2024-08-19T12:44:15.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-08-19T12:44:15.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (8080 ms) 2024-08-19T12:44:15.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-08-19T12:44:15.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7480 ms) 2024-08-19T12:44:15.326 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-08-19T12:44:15.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6991 ms) 2024-08-19T12:44:15.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-08-19T12:44:15.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (8929 ms) 2024-08-19T12:44:15.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-08-19T12:44:15.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7569 ms) 2024-08-19T12:44:15.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-08-19T12:44:15.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7237 ms) 2024-08-19T12:44:15.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (148663 ms total) 2024-08-19T12:44:15.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: 2024-08-19T12:44:15.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [----------] Global test environment tear-down 2024-08-19T12:44:15.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (279808 ms total) 2024-08-19T12:44:15.327 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio: [ PASSED ] 42 tests. 2024-08-19T12:44:48.178 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-08-19T12:44:48.179 INFO:tasks.ceph.mon.g:Restarting daemon 2024-08-19T12:44:48.179 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-08-19T12:44:48.183 INFO:tasks.ceph.mon.g:Started 2024-08-19T12:44:48.183 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-08-19T12:44:48.183 INFO:tasks.ceph.mon.a:Sent signal 19 2024-08-19T12:44:48.183 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:44:56.137 INFO:tasks.workunit.client.0.smithi104.stdout::163:head 2024-08-19T12:44:56.137 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:5d165639:::164:head 2024-08-19T12:44:56.137 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:f43765fc:::165:head 2024-08-19T12:44:56.137 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:b4c720e9:::166:head 2024-08-19T12:44:56.137 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:e694b040:::167:head 2024-08-19T12:44:56.137 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:afa38db2:::168:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:77ba9f53:::169:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:87495034:::170:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:7c96bf0e:::171:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:dbe346cc:::172:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:e943ec24:::173:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:f97a9c0c:::174:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:6f26e74d:::175:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:4f95e106:::176:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:0e6f2f8f:::177:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:05db05f1:::178:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:38a78d66:::179:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:d095610b:::180:head 2024-08-19T12:44:56.138 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:a1a9d709:::181:head 2024-08-19T12:44:56.139 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:1e5d39db:::182:head 2024-08-19T12:44:56.139 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:f7df4fb9:::183:head 2024-08-19T12:44:56.139 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:03a7f161:::184:head 2024-08-19T12:44:56.139 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:ba70721e:::185:head 2024-08-19T12:44:56.139 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:28e5662d:::186:head 2024-08-19T12:44:56.139 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:973d52de:::187:head 2024-08-19T12:44:56.139 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:4303eb1c:::188:head 2024-08-19T12:44:56.139 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:b990b48e:::189:head 2024-08-19T12:44:56.139 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:29b8165b:::190:head 2024-08-19T12:44:56.139 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:3547f197:::191:head 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:7e260936:::192:head 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:1abec7b1:::193:head 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:10fdda93:::194:head 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:15817eea:::195:head 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:770bab57:::196:head 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:ed9e13e7:::197:head 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:71471a8f:::198:head 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: checking for 264:10fb1d02:::199:head 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8936 ms) 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071449,0 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: first is 1724071449 2024-08-19T12:44:56.140 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071449,0 2024-08-19T12:44:56.141 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071449,0 2024-08-19T12:44:56.141 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071449,1724071451,1724071452,0 2024-08-19T12:44:56.141 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071449,1724071451,1724071452,0 2024-08-19T12:44:56.141 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071449,1724071451,1724071452,0 2024-08-19T12:44:56.141 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071449,1724071451,1724071452,1724071454,1724071455,0 2024-08-19T12:44:56.141 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071449,1724071451,1724071452,1724071454,1724071455,0 2024-08-19T12:44:56.141 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071449,1724071451,1724071452,1724071454,1724071455,0 2024-08-19T12:44:56.141 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071449,1724071451,1724071452,1724071454,1724071455,1724071457,1724071458,0 2024-08-19T12:44:56.141 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071449,1724071451,1724071452,1724071454,1724071455,1724071457,1724071458,0 2024-08-19T12:44:56.142 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071449,1724071451,1724071452,1724071454,1724071455,1724071457,1724071458,0 2024-08-19T12:44:56.142 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724071452,1724071454,1724071455,1724071457,1724071458,1724071460,1724071461,0 2024-08-19T12:44:56.142 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: first now 1724071452, trimmed 2024-08-19T12:44:56.142 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21062 ms) 2024-08-19T12:44:56.142 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-08-19T12:44:56.142 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: foo0 2024-08-19T12:44:56.142 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-08-19T12:44:56.142 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14971 ms) 2024-08-19T12:44:56.143 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-08-19T12:45:03.184 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-08-19T12:45:03.185 INFO:tasks.ceph.mon.a:Sent signal 18 2024-08-19T12:45:03.185 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:45:03.185 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:45:03.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.296+0000 7faac9007640 10 monclient: get_monmap_and_config 2024-08-19T12:45:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.296+0000 7faac9007640 10 monclient: build_initial_monmap 2024-08-19T12:45:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.296+0000 7faac9007640 10 monclient: monmap: 2024-08-19T12:45:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:03.297910+0000 2024-08-19T12:45:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:03.297910+0000 2024-08-19T12:45:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:03.299 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.296+0000 7faac9007640 1 Processor -- start 2024-08-19T12:45:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.296+0000 7faac9007640 1 -- start start 2024-08-19T12:45:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.296+0000 7faac9007640 10 monclient: init 2024-08-19T12:45:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac9007640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac9007640 10 monclient: _add_conns ranks=[1,5,7,4,2,0,6,8,3] 2024-08-19T12:45:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac9007640 10 monclient(hunting): picked mon.noname-f con 0x7faac4073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac9007640 10 monclient(hunting): picked mon.noname-h con 0x7faac40a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac9007640 10 monclient(hunting): picked mon.noname-i con 0x7faac413f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac9007640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac409dae0 con 0x7faac4073470 2024-08-19T12:45:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac9007640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac4136910 con 0x7faac40a57a0 2024-08-19T12:45:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac9007640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac409dd50 con 0x7faac413f2e0 2024-08-19T12:45:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac9007640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac9007640 10 monclient(hunting): authenticate will time out at 1097.385388s 2024-08-19T12:45:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac2d9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7faac4073470 0x7faac4136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:52096/0 (socket says 172.21.15.26:52096) 2024-08-19T12:45:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac2d9d640 1 -- 172.21.15.26:0/2074103055 learned_addr learned my addr 172.21.15.26:0/2074103055 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.297+0000 7faac359e640 1 --1- 172.21.15.26:0/2074103055 >> v1:172.21.15.104:6792/0 conn(0x7faac413f2e0 0x7faac41416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:33700/0 (socket says 172.21.15.26:33700) 2024-08-19T12:45:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.299+0000 7faac359e640 1 -- 172.21.15.26:0/2074103055 >> v1:172.21.15.104:6792/0 conn(0x7faac413f2e0 legacy=0x7faac41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.299+0000 7faac2d9d640 1 -- 172.21.15.26:0/2074103055 >> v1:172.21.15.104:6789/0 conn(0x7faac4073470 legacy=0x7faac4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:03.299+0000 7faaba59c640 1 -- 172.21.15.26:0/2074103055 >> v1:172.21.15.104:6791/0 conn(0x7faac40a57a0 legacy=0x7faac40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:04.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 1 -- 172.21.15.26:0/2074103055 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 957770189 0 0) 0x7faac409dd50 con 0x7faac413f2e0 2024-08-19T12:45:04.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient(hunting): _negotiate 2024-08-19T12:45:04.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:04.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:04.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient(hunting): my global_id is 135115 2024-08-19T12:45:04.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 1 -- 172.21.15.26:0/2074103055 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faab0003520 con 0x7faac413f2e0 2024-08-19T12:45:04.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 1 -- 172.21.15.26:0/2074103055 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2692070155 0 0) 0x7faac409dae0 con 0x7faac4073470 2024-08-19T12:45:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient(hunting): _negotiate 2024-08-19T12:45:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient(hunting): my global_id is 26929 2024-08-19T12:45:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 1 -- 172.21.15.26:0/2074103055 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac409dd50 con 0x7faac4073470 2024-08-19T12:45:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 1 -- 172.21.15.26:0/2074103055 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3207792425 0 0) 0x7faac409dd50 con 0x7faac4073470 2024-08-19T12:45:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 1 -- 172.21.15.26:0/2074103055 >> v1:172.21.15.104:6792/0 conn(0x7faac413f2e0 legacy=0x7faac41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 1 -- 172.21.15.26:0/2074103055 >> v1:172.21.15.104:6791/0 conn(0x7faac40a57a0 legacy=0x7faac40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient: _finish_hunting 0 2024-08-19T12:45:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 1 monclient: found mon.noname-f 2024-08-19T12:45:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:04.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 1 -- 172.21.15.26:0/2074103055 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac409e030 con 0x7faac4073470 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient: _finish_auth 0 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient: _check_auth_tickets 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 1 -- 172.21.15.26:0/2074103055 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faab4001970 con 0x7faac4073470 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac9007640 5 monclient: authenticate success, global_id 26929 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac9007640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient: dump: 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:04.040 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac9007640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:04.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 1 -- 172.21.15.26:0/2074103055 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faab4001c40 con 0x7faac4073470 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 1 -- 172.21.15.26:0/2074103055 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faab4004be0 con 0x7faac4073470 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac259c640 10 monclient: dump: 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:04.042 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:04.043 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac9007640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:04.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.035+0000 7faac9007640 4 set_mon_vals no callback set 2024-08-19T12:45:04.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.036+0000 7faac9007640 10 monclient: shutdown 2024-08-19T12:45:04.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.036+0000 7faac9007640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.036+0000 7faac9007640 1 -- 172.21.15.26:0/2074103055 >> v1:172.21.15.104:6789/0 conn(0x7faac4073470 legacy=0x7faac4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.036+0000 7faac9007640 1 -- 172.21.15.26:0/2074103055 shutdown_connections 2024-08-19T12:45:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.036+0000 7faac9007640 1 -- 172.21.15.26:0/2074103055 >> 172.21.15.26:0/2074103055 conn(0x7faac40abff0 msgr2=0x7faac40ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.036+0000 7faac9007640 1 -- 172.21.15.26:0/2074103055 shutdown_connections 2024-08-19T12:45:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.036+0000 7faac9007640 1 -- 172.21.15.26:0/2074103055 wait complete. 2024-08-19T12:45:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.037+0000 7faac9007640 10 monclient: build_initial_monmap 2024-08-19T12:45:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.037+0000 7faac9007640 10 monclient: monmap: 2024-08-19T12:45:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:04.049 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.038+0000 7faac9007640 1 librados: starting msgr at 2024-08-19T12:45:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.038+0000 7faac9007640 1 librados: starting objecter 2024-08-19T12:45:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.038+0000 7faac9007640 1 Processor -- start 2024-08-19T12:45:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.038+0000 7faac9007640 1 -- start start 2024-08-19T12:45:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.038+0000 7faac9007640 1 librados: setting wanted keys 2024-08-19T12:45:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.038+0000 7faac9007640 1 librados: calling monclient init 2024-08-19T12:45:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.038+0000 7faac9007640 10 monclient: init 2024-08-19T12:45:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.039+0000 7faac9007640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.039+0000 7faac9007640 10 monclient: _add_conns ranks=[4,2,3,1,8,0,5,6,7] 2024-08-19T12:45:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.039+0000 7faac9007640 10 monclient(hunting): picked mon.noname-c con 0x7faac4073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.039+0000 7faac9007640 10 monclient(hunting): picked mon.noname-b con 0x7faac40a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.039+0000 7faac9007640 10 monclient(hunting): picked mon.noname-g con 0x7faac413f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:04.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.039+0000 7faac9007640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac406ec30 con 0x7faac40a57a0 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.039+0000 7faac9007640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac40a4390 con 0x7faac413f2e0 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.039+0000 7faac9007640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac406ef10 con 0x7faac4073470 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.039+0000 7faac9007640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.039+0000 7faac9007640 10 monclient(hunting): authenticate will time out at 1098.127527s 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.039+0000 7faac2d9d640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7faac4073470 0x7faac41d23e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:42762/0 (socket says 172.21.15.26:42762) 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.039+0000 7faac2d9d640 1 -- 172.21.15.26:0/843570448 learned_addr learned my addr 172.21.15.26:0/843570448 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.041+0000 7faaba59c640 1 -- 172.21.15.26:0/843570448 >> v1:172.21.15.26:6790/0 conn(0x7faac40a57a0 legacy=0x7faac41d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.041+0000 7faac2d9d640 1 -- 172.21.15.26:0/843570448 >> v1:172.21.15.26:6791/0 conn(0x7faac4073470 legacy=0x7faac41d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.041+0000 7faabbfff640 10 client.?.objecter ms_handle_connect 0x7faac40a57a0 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.041+0000 7faabbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.041+0000 7faac359e640 1 -- 172.21.15.26:0/843570448 >> v1:172.21.15.104:6790/0 conn(0x7faac413f2e0 legacy=0x7faac41daab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.041+0000 7faabbfff640 10 client.?.objecter ms_handle_connect 0x7faac4073470 2024-08-19T12:45:04.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.041+0000 7faabbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.041+0000 7faabbfff640 10 client.?.objecter ms_handle_connect 0x7faac413f2e0 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.042+0000 7faabbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3250650220 0 0) 0x7faac406ef10 con 0x7faac4073470 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient(hunting): my global_id is 26536 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faaa4004eb0 con 0x7faac4073470 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2709336022 0 0) 0x7faac406ec30 con 0x7faac40a57a0 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient(hunting): my global_id is 26210 2024-08-19T12:45:04.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac406ef10 con 0x7faac40a57a0 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 459538859 0 0) 0x7faaa4004eb0 con 0x7faac4073470 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faac406ec30 con 0x7faac4073470 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faab4004b20 con 0x7faac4073470 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient(hunting): dump: 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:04.054 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1990502021 0 0) 0x7faac406ef10 con 0x7faac40a57a0 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faaa4004eb0 con 0x7faac40a57a0 2024-08-19T12:45:04.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4254287653 0 0) 0x7faac406ec30 con 0x7faac4073470 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 >> v1:172.21.15.104:6790/0 conn(0x7faac413f2e0 legacy=0x7faac41daab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 >> v1:172.21.15.26:6790/0 conn(0x7faac40a57a0 legacy=0x7faac41d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient: _finish_hunting 0 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 monclient: found mon.c 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac40a2ef0 con 0x7faac4073470 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient: _finish_auth 0 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient: _check_auth_tickets 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faac9007640 5 monclient: authenticate success, global_id 26536 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faac9007640 10 monclient: _renew_subs 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faac9007640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:04.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faac9007640 1 -- 172.21.15.26:0/843570448 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faac40a3180 con 0x7faac4073470 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faab4001250 con 0x7faac4073470 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faab4005510 con 0x7faac4073470 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faac9007640 10 client.26536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faac9007640 10 monclient: _renew_subs 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faac9007640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faac1d9b640 4 set_mon_vals no callback set 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faac9007640 1 -- 172.21.15.26:0/843570448 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7faac41d2cd0 con 0x7faac4073470 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faac9007640 1 librados: init done 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.044+0000 7faabbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.045+0000 7faabbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.045+0000 7faabbfff640 10 monclient: dump: 2024-08-19T12:45:04.057 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:04.058 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:04.059 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.045+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7faab4011970 con 0x7faac4073470 2024-08-19T12:45:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.046+0000 7faac9007640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.046+0000 7faac9007640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.046+0000 7faac9007640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.046+0000 7faac9007640 1 -- 172.21.15.26:0/843570448 --> v1:172.21.15.26:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faac4135270 con 0x7faac4073470 2024-08-19T12:45:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.046+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(277..277 src has 1..277) ==== 5338+0+0 (unknown 1397763768 0 0) 0x7faab404e080 con 0x7faac4073470 2024-08-19T12:45:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.046+0000 7faabbfff640 10 client.26536.objecter ms_dispatch 0x7faac4000b90 osd_map(277..277 src has 1..277) 2024-08-19T12:45:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.046+0000 7faabbfff640 3 client.26536.objecter handle_osd_map got epochs [277,277] > 0 2024-08-19T12:45:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.046+0000 7faabbfff640 3 client.26536.objecter handle_osd_map decoding full epoch 277 2024-08-19T12:45:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.046+0000 7faabbfff640 20 client.26536.objecter dump_active .. 0 homeless 2024-08-19T12:45:04.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.046+0000 7faaba59c640 10 monclient: get_auth_request con 0x7faaa4041240 auth_method 0 2024-08-19T12:45:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.048+0000 7faabbfff640 10 client.26536.objecter ms_handle_connect 0x7faaa4041240 2024-08-19T12:45:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.051+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 <== mon.4 v1:172.21.15.26:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7faab40174f0 con 0x7faac4073470 2024-08-19T12:45:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.051+0000 7faabbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.051+0000 7faabbfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.312+0000 7faac9007640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:45:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.312+0000 7faac9007640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:45:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.312+0000 7faac9007640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.312+0000 7faac9007640 1 -- 172.21.15.26:0/843570448 --> v1:172.21.15.26:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faac406c840 con 0x7faac4073470 2024-08-19T12:45:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.312+0000 7faabbfff640 1 -- 172.21.15.26:0/843570448 <== mon.4 v1:172.21.15.26:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 2652922226) 0x7faab4016df0 con 0x7faac4073470 2024-08-19T12:45:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.312+0000 7faabbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:45:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.312+0000 7faabbfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:04.315 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":42,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:45:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.314+0000 7faac9007640 10 librados: watch_flush enter 2024-08-19T12:45:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.314+0000 7faac9007640 10 librados: watch_flush exit 2024-08-19T12:45:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.314+0000 7faac9007640 20 client.26536.objecter shutdown clearing up homeless session... 2024-08-19T12:45:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.314+0000 7faac9007640 10 client.26536.objecter successfully canceled tick 2024-08-19T12:45:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.314+0000 7faac9007640 1 -- 172.21.15.26:0/843570448 >> v1:172.21.15.104:6800/899887506 conn(0x7faaa4041240 legacy=0x7faaa4043700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.314+0000 7faac9007640 10 monclient: shutdown 2024-08-19T12:45:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.314+0000 7faac9007640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.314+0000 7faac9007640 1 -- 172.21.15.26:0/843570448 >> v1:172.21.15.26:6791/0 conn(0x7faac4073470 legacy=0x7faac41d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.314+0000 7faac9007640 1 -- 172.21.15.26:0/843570448 shutdown_connections 2024-08-19T12:45:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.314+0000 7faac9007640 1 -- 172.21.15.26:0/843570448 >> 172.21.15.26:0/843570448 conn(0x7faac40abff0 msgr2=0x7faac41338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.314+0000 7faac9007640 1 -- 172.21.15.26:0/843570448 shutdown_connections 2024-08-19T12:45:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.314+0000 7faac9007640 1 -- 172.21.15.26:0/843570448 wait complete. 2024-08-19T12:45:04.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.315+0000 7faac9007640 1 librados: shutdown 2024-08-19T12:45:04.327 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:45:04.327 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:45:04.327 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:45:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.425+0000 7f0e9531b640 10 monclient: get_monmap_and_config 2024-08-19T12:45:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.425+0000 7f0e9531b640 10 monclient: build_initial_monmap 2024-08-19T12:45:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.425+0000 7f0e9531b640 10 monclient: monmap: 2024-08-19T12:45:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:04.427304+0000 2024-08-19T12:45:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:04.427304+0000 2024-08-19T12:45:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:04.428 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 1 Processor -- start 2024-08-19T12:45:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 1 -- start start 2024-08-19T12:45:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 10 monclient: init 2024-08-19T12:45:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 10 monclient: _add_conns ranks=[2,8,7,6,3,5,1,4,0] 2024-08-19T12:45:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 10 monclient(hunting): picked mon.noname-b con 0x7f0e9006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 10 monclient(hunting): picked mon.noname-e con 0x7f0e900a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 10 monclient(hunting): picked mon.noname-i con 0x7f0e900a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e90136790 con 0x7f0e9006e7b0 2024-08-19T12:45:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e9009ddf0 con 0x7f0e900a5d50 2024-08-19T12:45:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e9009e0d0 con 0x7f0e900a5920 2024-08-19T12:45:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e9531b640 10 monclient(hunting): authenticate will time out at 1098.514583s 2024-08-19T12:45:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e8e7fc640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f0e900a5920 0x7f0e9013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34214/0 (socket says 172.21.15.26:34214) 2024-08-19T12:45:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e8effd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0e9006e7b0 0x7f0e90136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52372/0 (socket says 172.21.15.26:52372) 2024-08-19T12:45:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.426+0000 7f0e8e7fc640 1 -- 172.21.15.26:0/3704207128 learned_addr learned my addr 172.21.15.26:0/3704207128 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.428+0000 7f0e8e7fc640 1 -- 172.21.15.26:0/3704207128 >> v1:172.21.15.26:6793/0 conn(0x7f0e900a5920 legacy=0x7f0e9013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.428+0000 7f0e8effd640 1 -- 172.21.15.26:0/3704207128 >> v1:172.21.15.26:6790/0 conn(0x7f0e9006e7b0 legacy=0x7f0e90136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:04.428+0000 7f0e8f7fe640 1 -- 172.21.15.26:0/3704207128 >> v1:172.21.15.104:6792/0 conn(0x7f0e900a5d50 legacy=0x7f0e90141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.103+0000 7f0e8dffb640 1 -- 172.21.15.26:0/3704207128 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3358799689 0 0) 0x7f0e90136790 con 0x7f0e9006e7b0 2024-08-19T12:45:05.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.103+0000 7f0e8dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.103+0000 7f0e8dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.103+0000 7f0e8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.103+0000 7f0e8dffb640 10 monclient(hunting): my global_id is 26219 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.103+0000 7f0e8dffb640 1 -- 172.21.15.26:0/3704207128 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e7c003520 con 0x7f0e9006e7b0 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.103+0000 7f0e8dffb640 1 -- 172.21.15.26:0/3704207128 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2037375010 0 0) 0x7f0e9009e0d0 con 0x7f0e900a5920 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.103+0000 7f0e8dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.103+0000 7f0e8dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.103+0000 7f0e8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.103+0000 7f0e8dffb640 10 monclient(hunting): my global_id is 26072 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.103+0000 7f0e8dffb640 1 -- 172.21.15.26:0/3704207128 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e90136790 con 0x7f0e900a5920 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 1 -- 172.21.15.26:0/3704207128 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2084694514 0 0) 0x7f0e90136790 con 0x7f0e900a5920 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 1 -- 172.21.15.26:0/3704207128 >> v1:172.21.15.104:6792/0 conn(0x7f0e900a5d50 legacy=0x7f0e90141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 1 -- 172.21.15.26:0/3704207128 >> v1:172.21.15.26:6790/0 conn(0x7f0e9006e7b0 legacy=0x7f0e90136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 1 monclient: found mon.noname-e 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 1 -- 172.21.15.26:0/3704207128 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9009e3b0 con 0x7f0e900a5920 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 10 monclient: _finish_auth 0 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 1 -- 172.21.15.26:0/3704207128 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e78001ac0 con 0x7f0e900a5920 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 10 monclient: dump: 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.109 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e9531b640 5 monclient: authenticate success, global_id 26072 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e9531b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 1 -- 172.21.15.26:0/3704207128 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0e780048a0 con 0x7f0e900a5920 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 1 -- 172.21.15.26:0/3704207128 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e78004d90 con 0x7f0e900a5920 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e9531b640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e9531b640 4 set_mon_vals no callback set 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.104+0000 7f0e8dffb640 10 monclient: dump: 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.111 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.111 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.105+0000 7f0e9531b640 10 monclient: shutdown 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.105+0000 7f0e9531b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.105+0000 7f0e9531b640 1 -- 172.21.15.26:0/3704207128 >> v1:172.21.15.26:6793/0 conn(0x7f0e900a5920 legacy=0x7f0e9013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.105+0000 7f0e9531b640 1 -- 172.21.15.26:0/3704207128 shutdown_connections 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.105+0000 7f0e9531b640 1 -- 172.21.15.26:0/3704207128 >> 172.21.15.26:0/3704207128 conn(0x7f0e900a8740 msgr2=0x7f0e900a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.105+0000 7f0e9531b640 1 -- 172.21.15.26:0/3704207128 shutdown_connections 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.105+0000 7f0e9531b640 1 -- 172.21.15.26:0/3704207128 wait complete. 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.106+0000 7f0e9531b640 10 monclient: build_initial_monmap 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.106+0000 7f0e9531b640 10 monclient: monmap: 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:05.118 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:05.119 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.106+0000 7f0e9531b640 1 librados: starting msgr at 2024-08-19T12:45:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.106+0000 7f0e9531b640 1 librados: starting objecter 2024-08-19T12:45:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.107+0000 7f0e9531b640 1 Processor -- start 2024-08-19T12:45:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.107+0000 7f0e9531b640 1 -- start start 2024-08-19T12:45:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.107+0000 7f0e9531b640 1 librados: setting wanted keys 2024-08-19T12:45:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.107+0000 7f0e9531b640 1 librados: calling monclient init 2024-08-19T12:45:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.107+0000 7f0e9531b640 10 monclient: init 2024-08-19T12:45:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.107+0000 7f0e9531b640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.107+0000 7f0e9531b640 10 monclient: _add_conns ranks=[1,7,4,3,0,5,2,6,8] 2024-08-19T12:45:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.107+0000 7f0e9531b640 10 monclient(hunting): picked mon.noname-f con 0x7f0e9006e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:05.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.107+0000 7f0e9531b640 10 monclient(hunting): picked mon.noname-i con 0x7f0e900a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.108+0000 7f0e9531b640 10 monclient(hunting): picked mon.noname-c con 0x7f0e900a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.108+0000 7f0e9531b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e9013b450 con 0x7f0e9006e7b0 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.108+0000 7f0e9531b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e901390f0 con 0x7f0e900a5d50 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.108+0000 7f0e9531b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e90139c50 con 0x7f0e900a5920 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.108+0000 7f0e9531b640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.108+0000 7f0e9531b640 10 monclient(hunting): authenticate will time out at 1099.196153s 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.108+0000 7f0e8f7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f0e900a5d50 0x7f0e901d6bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:42764/0 (socket says 172.21.15.26:42764) 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.108+0000 7f0e8f7fe640 1 -- 172.21.15.26:0/3841255089 learned_addr learned my addr 172.21.15.26:0/3841255089 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e8f7fe640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.26:6791/0 conn(0x7f0e900a5d50 legacy=0x7f0e901d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e877fe640 10 client.?.objecter ms_handle_connect 0x7f0e900a5d50 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e877fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1170872173 0 0) 0x7f0e901390f0 con 0x7f0e900a5d50 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e877fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e877fe640 10 monclient(hunting): my global_id is 26545 2024-08-19T12:45:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e70004eb0 con 0x7f0e900a5d50 2024-08-19T12:45:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e8e7fc640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.104:6792/0 conn(0x7f0e900a5920 legacy=0x7f0e901d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e877fe640 10 client.?.objecter ms_handle_connect 0x7f0e900a5920 2024-08-19T12:45:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e877fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e877fe640 10 client.?.objecter ms_handle_connect 0x7f0e9006e7b0 2024-08-19T12:45:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e877fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.110+0000 7f0e8effd640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.104:6789/0 conn(0x7f0e9006e7b0 legacy=0x7f0e901ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 425760687 0 0) 0x7f0e70004eb0 con 0x7f0e900a5d50 2024-08-19T12:45:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e901390f0 con 0x7f0e900a5d50 2024-08-19T12:45:05.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e80003990 con 0x7f0e900a5d50 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient(hunting): dump: 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.123 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.124 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2601847873 0 0) 0x7f0e90139c50 con 0x7f0e900a5920 2024-08-19T12:45:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient(hunting): my global_id is 135124 2024-08-19T12:45:05.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e70004eb0 con 0x7f0e900a5920 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2540015492 0 0) 0x7f0e9013b450 con 0x7f0e9006e7b0 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient(hunting): my global_id is 26938 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e90139c50 con 0x7f0e9006e7b0 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4070811558 0 0) 0x7f0e901390f0 con 0x7f0e900a5d50 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.104:6792/0 conn(0x7f0e900a5920 legacy=0x7f0e901d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.104:6789/0 conn(0x7f0e9006e7b0 legacy=0x7f0e901ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient: _finish_hunting 0 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 monclient: found mon.c 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:05.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9013abb0 con 0x7f0e900a5d50 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient: _finish_auth 0 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient: _check_auth_tickets 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e9531b640 5 monclient: authenticate success, global_id 26545 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e9531b640 10 monclient: _renew_subs 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e9531b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e9531b640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0e9013ae90 con 0x7f0e900a5d50 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e9531b640 10 client.26545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e9531b640 10 monclient: _renew_subs 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e9531b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e9531b640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f0e90138690 con 0x7f0e900a5d50 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e9531b640 1 librados: init done 2024-08-19T12:45:05.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0e80003e60 con 0x7f0e900a5d50 2024-08-19T12:45:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e80004390 con 0x7f0e900a5d50 2024-08-19T12:45:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e8cff9640 4 set_mon_vals no callback set 2024-08-19T12:45:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.111+0000 7f0e877fe640 10 monclient: dump: 2024-08-19T12:45:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.127 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.112+0000 7f0e9531b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.112+0000 7f0e9531b640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0e740065c0 con 0x7f0e74002860 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.112+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f0e80004530 con 0x7f0e900a5d50 2024-08-19T12:45:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.113+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(278..278 src has 1..278) ==== 5338+0+0 (unknown 991588785 0 0) 0x7f0e8004e480 con 0x7f0e900a5d50 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.113+0000 7f0e877fe640 10 client.26545.objecter ms_dispatch 0x7f0e90000b90 osd_map(278..278 src has 1..278) 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.113+0000 7f0e877fe640 3 client.26545.objecter handle_osd_map got epochs [278,278] > 0 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.113+0000 7f0e877fe640 3 client.26545.objecter handle_osd_map decoding full epoch 278 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.113+0000 7f0e877fe640 20 client.26545.objecter dump_active .. 0 homeless 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.113+0000 7f0e8e7fc640 10 monclient: get_auth_request con 0x7f0e70041180 auth_method 0 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.114+0000 7f0e877fe640 10 client.26545.objecter ms_handle_connect 0x7f0e74002860 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.114+0000 7f0e877fe640 10 client.26545.objecter resend_mon_ops 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.114+0000 7f0e8effd640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.26:6789/0 conn(0x7f0e74002860 legacy=0x7f0e74004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3005188286 0 0) 0x7f0e740065c0 con 0x7f0e74002860 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 10 monclient(hunting): my global_id is 144441 2024-08-19T12:45:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e901390f0 con 0x7f0e74002860 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2381562655 0 0) 0x7f0e901390f0 con 0x7f0e74002860 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e740065c0 con 0x7f0e74002860 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e88001250 con 0x7f0e74002860 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 10 monclient: dump: 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.130 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1029287367 0 0) 0x7f0e740065c0 con 0x7f0e74002860 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e7400a160 con 0x7f0e74002860 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 10 client.26545.objecter ms_handle_connect 0x7f0e70041180 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0e88006840 con 0x7f0e74002860 2024-08-19T12:45:05.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.26:6789/0 conn(0x7f0e74002860 legacy=0x7f0e74004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.115+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.26:6789/0 conn(0x7f0e74002860 legacy=0x7f0e74004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.122+0000 7f0e9531b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.122+0000 7f0e9531b640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0e74009e70 con 0x7f0e74009130 2024-08-19T12:45:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e8effd640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.26:6789/0 conn(0x7f0e74009130 legacy=0x7f0e740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 10 client.26545.objecter ms_handle_connect 0x7f0e74009130 2024-08-19T12:45:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 10 client.26545.objecter resend_mon_ops 2024-08-19T12:45:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1705376981 0 0) 0x7f0e74009e70 con 0x7f0e74009130 2024-08-19T12:45:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 10 monclient(hunting): my global_id is 144450 2024-08-19T12:45:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e740065c0 con 0x7f0e74009130 2024-08-19T12:45:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2139689565 0 0) 0x7f0e740065c0 con 0x7f0e74009130 2024-08-19T12:45:05.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e74009e70 con 0x7f0e74009130 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e88005040 con 0x7f0e74009130 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 10 monclient: dump: 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.135 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 553168887 0 0) 0x7f0e74009e70 con 0x7f0e74009130 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.125+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0e7400c410 con 0x7f0e74009130 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7713 (unknown 0 0 2619249407) 0x7f0e88006670 con 0x7f0e74009130 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e877fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e877fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.26:6789/0 conn(0x7f0e74009130 legacy=0x7f0e740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e877fe640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.26:6789/0 conn(0x7f0e74009130 legacy=0x7f0e740095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.137 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:05.137 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:05.137 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:05.137 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:45:05.137 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:05.137 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 157043, 2024-08-19T12:45:05.137 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:05.137 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:05.137 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:05.138 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:05.139 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:05.141 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.142 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:05.143 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:05.144 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:05.145 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.146 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:05.147 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.148 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:05.151 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:05.153 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:05.154 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.154 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.154 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.154 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:05.154 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:05.154 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.155 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.155 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.155 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:05.155 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.155 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.155 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:05.155 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:05.155 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.155 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.156 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.158 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:05.158 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:05.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 10 librados: watch_flush enter 2024-08-19T12:45:05.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 10 librados: watch_flush exit 2024-08-19T12:45:05.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 20 client.26545.objecter shutdown clearing up homeless session... 2024-08-19T12:45:05.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 10 client.26545.objecter successfully canceled tick 2024-08-19T12:45:05.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.104:6800/899887506 conn(0x7f0e70041180 legacy=0x7f0e70043640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 10 monclient: shutdown 2024-08-19T12:45:05.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:05.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 1 -- 172.21.15.26:0/3841255089 >> v1:172.21.15.26:6791/0 conn(0x7f0e900a5d50 legacy=0x7f0e901d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e8f7fe640 1 -- 172.21.15.26:0/3841255089 reap_dead start 2024-08-19T12:45:05.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 1 -- 172.21.15.26:0/3841255089 shutdown_connections 2024-08-19T12:45:05.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 1 -- 172.21.15.26:0/3841255089 >> 172.21.15.26:0/3841255089 conn(0x7f0e900a8740 msgr2=0x7f0e900ab120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:05.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 1 -- 172.21.15.26:0/3841255089 shutdown_connections 2024-08-19T12:45:05.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 1 -- 172.21.15.26:0/3841255089 wait complete. 2024-08-19T12:45:05.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.126+0000 7f0e9531b640 1 librados: shutdown 2024-08-19T12:45:05.161 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:05.161 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:45:05.161 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:45:05.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.264+0000 7f355fbc5640 10 monclient: get_monmap_and_config 2024-08-19T12:45:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.264+0000 7f355fbc5640 10 monclient: build_initial_monmap 2024-08-19T12:45:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.264+0000 7f355fbc5640 10 monclient: monmap: 2024-08-19T12:45:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:05.265967+0000 2024-08-19T12:45:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:05.265967+0000 2024-08-19T12:45:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:05.267 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:05.267 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.264+0000 7f355fbc5640 1 Processor -- start 2024-08-19T12:45:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.264+0000 7f355fbc5640 1 -- start start 2024-08-19T12:45:05.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.264+0000 7f355fbc5640 10 monclient: init 2024-08-19T12:45:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.265+0000 7f355fbc5640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.265+0000 7f355fbc5640 10 monclient: _add_conns ranks=[7,8,1,2,4,3,5,6,0] 2024-08-19T12:45:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.265+0000 7f355fbc5640 10 monclient(hunting): picked mon.noname-i con 0x7f3558073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.265+0000 7f355fbc5640 10 monclient(hunting): picked mon.noname-e con 0x7f35580a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.265+0000 7f355fbc5640 10 monclient(hunting): picked mon.noname-f con 0x7f355813f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.265+0000 7f355fbc5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355809dae0 con 0x7f355813f2e0 2024-08-19T12:45:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.265+0000 7f355fbc5640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3558136910 con 0x7f3558073470 2024-08-19T12:45:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.265+0000 7f355fbc5640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355809dd50 con 0x7f35580a57a0 2024-08-19T12:45:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.265+0000 7f355fbc5640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.265+0000 7f355fbc5640 10 monclient(hunting): authenticate will time out at 1099.353534s 2024-08-19T12:45:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.265+0000 7f355d160640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f35580a57a0 0x7f35580a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34222/0 (socket says 172.21.15.26:34222) 2024-08-19T12:45:05.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.265+0000 7f355d160640 1 -- 172.21.15.26:0/1793639763 learned_addr learned my addr 172.21.15.26:0/1793639763 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.267+0000 7f355d160640 1 -- 172.21.15.26:0/1793639763 >> v1:172.21.15.26:6793/0 conn(0x7f35580a57a0 legacy=0x7f35580a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.267+0000 7f355d961640 1 -- 172.21.15.26:0/1793639763 >> v1:172.21.15.104:6792/0 conn(0x7f3558073470 legacy=0x7f3558136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.267+0000 7f355e162640 1 -- 172.21.15.26:0/1793639763 >> v1:172.21.15.104:6789/0 conn(0x7f355813f2e0 legacy=0x7f35581416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4063899940 0 0) 0x7f355809dd50 con 0x7f35580a57a0 2024-08-19T12:45:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient(hunting): my global_id is 26081 2024-08-19T12:45:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3540003520 con 0x7f35580a57a0 2024-08-19T12:45:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2145941859 0 0) 0x7f355809dae0 con 0x7f355813f2e0 2024-08-19T12:45:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient(hunting): my global_id is 26947 2024-08-19T12:45:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f355809dd50 con 0x7f355813f2e0 2024-08-19T12:45:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3252585164 0 0) 0x7f3558136910 con 0x7f3558073470 2024-08-19T12:45:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient(hunting): my global_id is 135133 2024-08-19T12:45:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f355809dae0 con 0x7f3558073470 2024-08-19T12:45:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1840080596 0 0) 0x7f3540003520 con 0x7f35580a57a0 2024-08-19T12:45:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 >> v1:172.21.15.104:6792/0 conn(0x7f3558073470 legacy=0x7f3558136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 >> v1:172.21.15.104:6789/0 conn(0x7f355813f2e0 legacy=0x7f35581416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient: _finish_hunting 0 2024-08-19T12:45:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 monclient: found mon.noname-e 2024-08-19T12:45:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:45:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355809e030 con 0x7f35580a57a0 2024-08-19T12:45:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient: _finish_auth 0 2024-08-19T12:45:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient: _check_auth_tickets 2024-08-19T12:45:05.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355fbc5640 5 monclient: authenticate success, global_id 26081 2024-08-19T12:45:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355fbc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f354c001ae0 con 0x7f35580a57a0 2024-08-19T12:45:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient: dump: 2024-08-19T12:45:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.282 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355fbc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f354c0048a0 con 0x7f35580a57a0 2024-08-19T12:45:05.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355fbc5640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355fbc5640 4 set_mon_vals no callback set 2024-08-19T12:45:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 1 -- 172.21.15.26:0/1793639763 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f354c004e10 con 0x7f35580a57a0 2024-08-19T12:45:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:05.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355c95f640 10 monclient: dump: 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.285 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.286 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355fbc5640 10 monclient: shutdown 2024-08-19T12:45:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355fbc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355fbc5640 1 -- 172.21.15.26:0/1793639763 >> v1:172.21.15.26:6793/0 conn(0x7f35580a57a0 legacy=0x7f35580a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355fbc5640 1 -- 172.21.15.26:0/1793639763 shutdown_connections 2024-08-19T12:45:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.268+0000 7f355fbc5640 1 -- 172.21.15.26:0/1793639763 >> 172.21.15.26:0/1793639763 conn(0x7f35580abff0 msgr2=0x7f35580ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.269+0000 7f355fbc5640 1 -- 172.21.15.26:0/1793639763 shutdown_connections 2024-08-19T12:45:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.269+0000 7f355fbc5640 1 -- 172.21.15.26:0/1793639763 wait complete. 2024-08-19T12:45:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.269+0000 7f355fbc5640 10 monclient: build_initial_monmap 2024-08-19T12:45:05.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.269+0000 7f355fbc5640 10 monclient: monmap: 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:05.287 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.269+0000 7f355fbc5640 1 librados: starting msgr at 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.269+0000 7f355fbc5640 1 librados: starting objecter 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.269+0000 7f355fbc5640 1 Processor -- start 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.269+0000 7f355fbc5640 1 -- start start 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.269+0000 7f355fbc5640 1 librados: setting wanted keys 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.269+0000 7f355fbc5640 1 librados: calling monclient init 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.269+0000 7f355fbc5640 10 monclient: init 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355fbc5640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355fbc5640 10 monclient: _add_conns ranks=[7,4,6,5,3,1,0,8,2] 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355fbc5640 10 monclient(hunting): picked mon.noname-i con 0x7f3558073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:05.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355fbc5640 10 monclient(hunting): picked mon.noname-c con 0x7f35580a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355fbc5640 10 monclient(hunting): picked mon.noname-d con 0x7f355813f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355fbc5640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3558131d10 con 0x7f35580a57a0 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355fbc5640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35581cef20 con 0x7f355813f2e0 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355fbc5640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355813f0b0 con 0x7f3558073470 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355fbc5640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355fbc5640 10 monclient(hunting): authenticate will time out at 1099.358405s 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355d160640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f35580a57a0 0x7f35581ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:42768/0 (socket says 172.21.15.26:42768) 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355d160640 1 -- 172.21.15.26:0/3676018471 learned_addr learned my addr 172.21.15.26:0/3676018471 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.270+0000 7f355e162640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f355813f2e0 0x7f35581d6360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49798/0 (socket says 172.21.15.26:49798) 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f355e162640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.26:6792/0 conn(0x7f355813f2e0 legacy=0x7f35581d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f35567fc640 10 client.?.objecter ms_handle_connect 0x7f355813f2e0 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f35567fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f35567fc640 10 client.?.objecter ms_handle_connect 0x7f35580a57a0 2024-08-19T12:45:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f35567fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f355d160640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.26:6791/0 conn(0x7f35580a57a0 legacy=0x7f35581ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f355d961640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.104:6792/0 conn(0x7f3558073470 legacy=0x7f35581ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f35567fc640 10 client.?.objecter ms_handle_connect 0x7f3558073470 2024-08-19T12:45:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f35567fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 455468391 0 0) 0x7f35581cef20 con 0x7f355813f2e0 2024-08-19T12:45:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f35567fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f35567fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f35567fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f35567fc640 10 monclient(hunting): my global_id is 26547 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.272+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3544004eb0 con 0x7f355813f2e0 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3144434362 0 0) 0x7f3558131d10 con 0x7f35580a57a0 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): my global_id is 26554 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35581cef20 con 0x7f35580a57a0 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4114960717 0 0) 0x7f355813f0b0 con 0x7f3558073470 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): my global_id is 135142 2024-08-19T12:45:05.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3558131d10 con 0x7f3558073470 2024-08-19T12:45:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1278930086 0 0) 0x7f35581cef20 con 0x7f35580a57a0 2024-08-19T12:45:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f355813f0b0 con 0x7f35580a57a0 2024-08-19T12:45:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1198888690 0 0) 0x7f3544004eb0 con 0x7f355813f2e0 2024-08-19T12:45:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35581cef20 con 0x7f355813f2e0 2024-08-19T12:45:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f354c004d90 con 0x7f35580a57a0 2024-08-19T12:45:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:45:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): dump: 2024-08-19T12:45:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.292 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.293 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.293 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.293 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.293 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.293 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.293 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.293 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.293 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.293 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.293 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.293 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.293 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f35480020b0 con 0x7f355813f2e0 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): dump: 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.294 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.295 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.295 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.295 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.295 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.295 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.295 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.295 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.295 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 660906630 0 0) 0x7f3558131d10 con 0x7f3558073470 2024-08-19T12:45:05.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3544004eb0 con 0x7f3558073470 2024-08-19T12:45:05.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3550001d10 con 0x7f3558073470 2024-08-19T12:45:05.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:05.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:45:05.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient(hunting): dump: 2024-08-19T12:45:05.296 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.296 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.296 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.296 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.296 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.296 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.296 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.296 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.296 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.297 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.297 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.297 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.297 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.297 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.297 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.297 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2942636296 0 0) 0x7f35581cef20 con 0x7f355813f2e0 2024-08-19T12:45:05.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.104:6792/0 conn(0x7f3558073470 legacy=0x7f35581ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.26:6791/0 conn(0x7f35580a57a0 legacy=0x7f35581ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient: _finish_hunting 0 2024-08-19T12:45:05.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 monclient: found mon.d 2024-08-19T12:45:05.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:05.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:05.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355812f9b0 con 0x7f355813f2e0 2024-08-19T12:45:05.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient: _finish_auth 0 2024-08-19T12:45:05.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient: _check_auth_tickets 2024-08-19T12:45:05.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f355fbc5640 5 monclient: authenticate success, global_id 26547 2024-08-19T12:45:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f355fbc5640 10 monclient: _renew_subs 2024-08-19T12:45:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f355fbc5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f355fbc5640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3558131a30 con 0x7f355813f2e0 2024-08-19T12:45:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f355fbc5640 10 client.26547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f355fbc5640 10 monclient: _renew_subs 2024-08-19T12:45:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f355fbc5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f355fbc5640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f35581324a0 con 0x7f355813f2e0 2024-08-19T12:45:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f355fbc5640 1 librados: init done 2024-08-19T12:45:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3548003e30 con 0x7f355813f2e0 2024-08-19T12:45:05.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3548004360 con 0x7f355813f2e0 2024-08-19T12:45:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f3557fff640 4 set_mon_vals no callback set 2024-08-19T12:45:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.273+0000 7f35567fc640 10 monclient: dump: 2024-08-19T12:45:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.300 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.301 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.301 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.301 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.301 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.301 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.301 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.301 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.301 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.301 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.301 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.302 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.274+0000 7f355fbc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:05.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.274+0000 7f355fbc5640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f35380065c0 con 0x7f3538002860 2024-08-19T12:45:05.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.274+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3548004500 con 0x7f355813f2e0 2024-08-19T12:45:05.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.274+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(278..278 src has 1..278) ==== 5338+0+0 (unknown 991588785 0 0) 0x7f354804e300 con 0x7f355813f2e0 2024-08-19T12:45:05.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.274+0000 7f35567fc640 10 client.26547.objecter ms_dispatch 0x7f3558000b90 osd_map(278..278 src has 1..278) 2024-08-19T12:45:05.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.274+0000 7f35567fc640 3 client.26547.objecter handle_osd_map got epochs [278,278] > 0 2024-08-19T12:45:05.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.274+0000 7f35567fc640 3 client.26547.objecter handle_osd_map decoding full epoch 278 2024-08-19T12:45:05.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.275+0000 7f35567fc640 20 client.26547.objecter dump_active .. 0 homeless 2024-08-19T12:45:05.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.275+0000 7f355d160640 10 monclient: get_auth_request con 0x7f3544041ad0 auth_method 0 2024-08-19T12:45:05.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.276+0000 7f355d961640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.26:6790/0 conn(0x7f3538002860 legacy=0x7f3538004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.276+0000 7f35567fc640 10 client.26547.objecter ms_handle_connect 0x7f3538002860 2024-08-19T12:45:05.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.276+0000 7f35567fc640 10 client.26547.objecter resend_mon_ops 2024-08-19T12:45:05.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.276+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3512499045 0 0) 0x7f35380065c0 con 0x7f3538002860 2024-08-19T12:45:05.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.276+0000 7f35567fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.276+0000 7f35567fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.276+0000 7f35567fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.276+0000 7f35567fc640 10 monclient(hunting): my global_id is 26228 2024-08-19T12:45:05.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.276+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35581cef20 con 0x7f3538002860 2024-08-19T12:45:05.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1585608428 0 0) 0x7f35581cef20 con 0x7f3538002860 2024-08-19T12:45:05.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35380065c0 con 0x7f3538002860 2024-08-19T12:45:05.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3550003bc0 con 0x7f3538002860 2024-08-19T12:45:05.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:05.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 10 monclient: dump: 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.305 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.306 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3922341649 0 0) 0x7f35380065c0 con 0x7f3538002860 2024-08-19T12:45:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f353800a160 con 0x7f3538002860 2024-08-19T12:45:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 10 client.26547.objecter ms_handle_connect 0x7f3544041ad0 2024-08-19T12:45:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3550003e70 con 0x7f3538002860 2024-08-19T12:45:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.26:6790/0 conn(0x7f3538002860 legacy=0x7f3538004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.277+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.26:6790/0 conn(0x7f3538002860 legacy=0x7f3538004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.283+0000 7f355fbc5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:05.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.283+0000 7f355fbc5640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3538009e50 con 0x7f3538009110 2024-08-19T12:45:05.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.285+0000 7f355d961640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.26:6790/0 conn(0x7f3538009110 legacy=0x7f35380095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 10 client.26547.objecter ms_handle_connect 0x7f3538009110 2024-08-19T12:45:05.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 10 client.26547.objecter resend_mon_ops 2024-08-19T12:45:05.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3058766479 0 0) 0x7f3538009e50 con 0x7f3538009110 2024-08-19T12:45:05.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 10 monclient(hunting): my global_id is 26237 2024-08-19T12:45:05.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35380065c0 con 0x7f3538009110 2024-08-19T12:45:05.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1089049019 0 0) 0x7f35380065c0 con 0x7f3538009110 2024-08-19T12:45:05.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3538009e50 con 0x7f3538009110 2024-08-19T12:45:05.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3550004330 con 0x7f3538009110 2024-08-19T12:45:05.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:05.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 10 monclient: dump: 2024-08-19T12:45:05.312 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.312 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.312 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.312 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.313 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.313 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.313 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.313 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.313 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.313 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.313 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.313 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.313 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.313 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.313 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.313 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2509809324 0 0) 0x7f3538009e50 con 0x7f3538009110 2024-08-19T12:45:05.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f353800c3f0 con 0x7f3538009110 2024-08-19T12:45:05.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 3772439263) 0x7f3550007b80 con 0x7f3538009110 2024-08-19T12:45:05.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:05.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:05.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.26:6790/0 conn(0x7f3538009110 legacy=0x7f35380095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f35567fc640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.26:6790/0 conn(0x7f3538009110 legacy=0x7f35380095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.314 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:05.315 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:05.315 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:05.315 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:05.315 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:05.315 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 418341, 2024-08-19T12:45:05.315 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:05.315 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:05.315 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:05.315 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:05.315 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:05.316 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:05.316 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:05.316 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:05.316 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:05.316 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:05.316 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.316 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:45:05.316 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:05.316 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.317 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.318 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:05.319 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:05.320 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.321 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.321 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.321 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.321 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.321 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.321 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.321 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.321 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.321 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.321 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.321 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.321 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.322 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.323 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:05.323 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:05.323 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.323 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.323 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.323 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:05.323 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:05.324 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.324 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:05.324 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.324 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.324 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:05.325 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:05.325 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.325 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.325 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.325 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:05.325 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:05.325 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.325 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.326 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:05.326 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.326 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.326 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.326 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:05.326 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:05.326 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.326 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.327 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.327 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:05.327 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:05.327 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.327 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.327 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.327 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:05.327 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:05.328 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.329 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.329 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.329 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:05.329 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.329 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.329 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.329 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:05.330 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:05.330 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.330 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.330 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.330 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:05.330 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:05.330 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.330 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:05.331 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.331 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.331 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.331 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:05.331 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:05.331 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.331 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.331 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.332 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:05.332 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:05.332 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.332 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.332 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.332 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:05.332 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:05.335 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:05.336 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.336 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.336 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.336 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.336 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.336 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:05.336 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:05.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.336 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.337 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.337 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:05.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.337 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.337 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:05.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.337 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.337 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.337 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:05.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.338 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:05.338 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:05.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.286+0000 7f355fbc5640 10 librados: watch_flush enter 2024-08-19T12:45:05.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355fbc5640 10 librados: watch_flush exit 2024-08-19T12:45:05.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355fbc5640 20 client.26547.objecter shutdown clearing up homeless session... 2024-08-19T12:45:05.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355fbc5640 10 client.26547.objecter successfully canceled tick 2024-08-19T12:45:05.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355fbc5640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.104:6800/899887506 conn(0x7f3544041ad0 legacy=0x7f3544043f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355fbc5640 10 monclient: shutdown 2024-08-19T12:45:05.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355fbc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:05.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355fbc5640 1 -- 172.21.15.26:0/3676018471 >> v1:172.21.15.26:6792/0 conn(0x7f355813f2e0 legacy=0x7f35581d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355e162640 1 -- 172.21.15.26:0/3676018471 reap_dead start 2024-08-19T12:45:05.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355e162640 1 -- 172.21.15.26:0/3676018471 reap_dead start 2024-08-19T12:45:05.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355fbc5640 1 -- 172.21.15.26:0/3676018471 shutdown_connections 2024-08-19T12:45:05.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355fbc5640 1 -- 172.21.15.26:0/3676018471 >> 172.21.15.26:0/3676018471 conn(0x7f35580abff0 msgr2=0x7f3558076880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:05.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355fbc5640 1 -- 172.21.15.26:0/3676018471 shutdown_connections 2024-08-19T12:45:05.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355fbc5640 1 -- 172.21.15.26:0/3676018471 wait complete. 2024-08-19T12:45:05.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.287+0000 7f355fbc5640 1 librados: shutdown 2024-08-19T12:45:05.341 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:05.341 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:45:05.341 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:45:05.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.438+0000 7f66f0898640 10 monclient: get_monmap_and_config 2024-08-19T12:45:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.438+0000 7f66f0898640 10 monclient: build_initial_monmap 2024-08-19T12:45:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.438+0000 7f66f0898640 10 monclient: monmap: 2024-08-19T12:45:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:05.440633+0000 2024-08-19T12:45:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:05.440633+0000 2024-08-19T12:45:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:05.441 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:05.442 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:05.442 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.439+0000 7f66f0898640 1 Processor -- start 2024-08-19T12:45:05.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.439+0000 7f66f0898640 1 -- start start 2024-08-19T12:45:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.439+0000 7f66f0898640 10 monclient: init 2024-08-19T12:45:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66f0898640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66f0898640 10 monclient: _add_conns ranks=[6,2,3,4,1,0,7,8,5] 2024-08-19T12:45:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66f0898640 10 monclient(hunting): picked mon.noname-d con 0x7f66ec06e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66f0898640 10 monclient(hunting): picked mon.noname-b con 0x7f66ec0a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66f0898640 10 monclient(hunting): picked mon.noname-g con 0x7f66ec0a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66f0898640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec136790 con 0x7f66ec0a5920 2024-08-19T12:45:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66f0898640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec09ddf0 con 0x7f66ec0a5d50 2024-08-19T12:45:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66f0898640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec09e0d0 con 0x7f66ec06e7b0 2024-08-19T12:45:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66f0898640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66f0898640 10 monclient(hunting): authenticate will time out at 1099.528217s 2024-08-19T12:45:05.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66e9d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f66ec0a5920 0x7f66ec13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52392/0 (socket says 172.21.15.26:52392) 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66ea59c640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f66ec06e7b0 0x7f66ec136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49802/0 (socket says 172.21.15.26:49802) 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.440+0000 7f66e9d9b640 1 -- 172.21.15.26:0/2169169263 learned_addr learned my addr 172.21.15.26:0/2169169263 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66ea59c640 1 -- 172.21.15.26:0/2169169263 >> v1:172.21.15.26:6792/0 conn(0x7f66ec06e7b0 legacy=0x7f66ec136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e9d9b640 1 -- 172.21.15.26:0/2169169263 >> v1:172.21.15.26:6790/0 conn(0x7f66ec0a5920 legacy=0x7f66ec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 1 -- 172.21.15.26:0/2169169263 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728313114 0 0) 0x7f66ec09e0d0 con 0x7f66ec06e7b0 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient(hunting): my global_id is 26556 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 1 -- 172.21.15.26:0/2169169263 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66d8003520 con 0x7f66ec06e7b0 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 1 -- 172.21.15.26:0/2169169263 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2825771205 0 0) 0x7f66ec136790 con 0x7f66ec0a5920 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient(hunting): my global_id is 26246 2024-08-19T12:45:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 1 -- 172.21.15.26:0/2169169263 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66ec09e0d0 con 0x7f66ec0a5920 2024-08-19T12:45:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66ead9d640 1 -- 172.21.15.26:0/2169169263 >> v1:172.21.15.104:6790/0 conn(0x7f66ec0a5d50 legacy=0x7f66ec141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 1 -- 172.21.15.26:0/2169169263 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3306352252 0 0) 0x7f66d8003520 con 0x7f66ec06e7b0 2024-08-19T12:45:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 1 -- 172.21.15.26:0/2169169263 >> v1:172.21.15.104:6790/0 conn(0x7f66ec0a5d50 legacy=0x7f66ec141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 1 -- 172.21.15.26:0/2169169263 >> v1:172.21.15.26:6790/0 conn(0x7f66ec0a5920 legacy=0x7f66ec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient: _finish_hunting 0 2024-08-19T12:45:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 1 monclient: found mon.noname-d 2024-08-19T12:45:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:05.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:45:05.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 1 -- 172.21.15.26:0/2169169263 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec09e3b0 con 0x7f66ec06e7b0 2024-08-19T12:45:05.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient: _finish_auth 0 2024-08-19T12:45:05.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient: _check_auth_tickets 2024-08-19T12:45:05.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:05.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 1 -- 172.21.15.26:0/2169169263 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f66e4001e60 con 0x7f66ec06e7b0 2024-08-19T12:45:05.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:45:05.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66e959a640 10 monclient: dump: 2024-08-19T12:45:05.451 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.451 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.452 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66f0898640 5 monclient: authenticate success, global_id 26556 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.442+0000 7f66f0898640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66e959a640 1 -- 172.21.15.26:0/2169169263 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f66e4004b60 con 0x7f66ec06e7b0 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66e959a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66e959a640 1 -- 172.21.15.26:0/2169169263 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f66e40050d0 con 0x7f66ec06e7b0 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66f0898640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66f0898640 4 set_mon_vals no callback set 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66e959a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66e959a640 10 monclient: dump: 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.453 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66f0898640 10 monclient: shutdown 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66f0898640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66f0898640 1 -- 172.21.15.26:0/2169169263 >> v1:172.21.15.26:6792/0 conn(0x7f66ec06e7b0 legacy=0x7f66ec136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66f0898640 1 -- 172.21.15.26:0/2169169263 shutdown_connections 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66f0898640 1 -- 172.21.15.26:0/2169169263 >> 172.21.15.26:0/2169169263 conn(0x7f66ec0a8740 msgr2=0x7f66ec0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66f0898640 1 -- 172.21.15.26:0/2169169263 shutdown_connections 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.443+0000 7f66f0898640 1 -- 172.21.15.26:0/2169169263 wait complete. 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.444+0000 7f66f0898640 10 monclient: build_initial_monmap 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.444+0000 7f66f0898640 10 monclient: monmap: 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:05.455 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.444+0000 7f66f0898640 1 librados: starting msgr at 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.444+0000 7f66f0898640 1 librados: starting objecter 2024-08-19T12:45:05.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.444+0000 7f66f0898640 1 Processor -- start 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.444+0000 7f66f0898640 1 -- start start 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.444+0000 7f66f0898640 1 librados: setting wanted keys 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.444+0000 7f66f0898640 1 librados: calling monclient init 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.444+0000 7f66f0898640 10 monclient: init 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.445+0000 7f66f0898640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.445+0000 7f66f0898640 10 monclient: _add_conns ranks=[7,1,0,5,4,2,6,8,3] 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.445+0000 7f66f0898640 10 monclient(hunting): picked mon.noname-i con 0x7f66ec06e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.445+0000 7f66f0898640 10 monclient(hunting): picked mon.noname-f con 0x7f66ec0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.445+0000 7f66f0898640 10 monclient(hunting): picked mon.noname-a con 0x7f66ec0a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.445+0000 7f66f0898640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec0a0cc0 con 0x7f66ec0a5d50 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.445+0000 7f66f0898640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec0a0fa0 con 0x7f66ec0a5920 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.445+0000 7f66f0898640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec133110 con 0x7f66ec06e7b0 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.445+0000 7f66f0898640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:05.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.445+0000 7f66f0898640 10 monclient(hunting): authenticate will time out at 1099.533205s 2024-08-19T12:45:05.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.445+0000 7f66ead9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f66ec0a5d50 0x7f66ec1d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45244/0 (socket says 172.21.15.26:45244) 2024-08-19T12:45:05.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.445+0000 7f66ead9d640 1 -- 172.21.15.26:0/1235749099 learned_addr learned my addr 172.21.15.26:0/1235749099 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:05.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66ead9d640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.26:6789/0 conn(0x7f66ec0a5d50 legacy=0x7f66ec1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 client.?.objecter ms_handle_connect 0x7f66ec0a5d50 2024-08-19T12:45:05.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66ea59c640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.104:6792/0 conn(0x7f66ec06e7b0 legacy=0x7f66ec09e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 client.?.objecter ms_handle_connect 0x7f66ec06e7b0 2024-08-19T12:45:05.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 client.?.objecter ms_handle_connect 0x7f66ec0a5920 2024-08-19T12:45:05.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e9d9b640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.104:6789/0 conn(0x7f66ec0a5920 legacy=0x7f66ec09efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085722015 0 0) 0x7f66ec0a0cc0 con 0x7f66ec0a5d50 2024-08-19T12:45:05.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): my global_id is 144459 2024-08-19T12:45:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66cc004eb0 con 0x7f66ec0a5d50 2024-08-19T12:45:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3745560503 0 0) 0x7f66ec133110 con 0x7f66ec06e7b0 2024-08-19T12:45:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): my global_id is 135151 2024-08-19T12:45:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66ec0a0cc0 con 0x7f66ec06e7b0 2024-08-19T12:45:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 713123495 0 0) 0x7f66ec0a0fa0 con 0x7f66ec0a5920 2024-08-19T12:45:05.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): my global_id is 26956 2024-08-19T12:45:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66ec133110 con 0x7f66ec0a5920 2024-08-19T12:45:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3736959214 0 0) 0x7f66cc004eb0 con 0x7f66ec0a5d50 2024-08-19T12:45:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66ec0a0fa0 con 0x7f66ec0a5d50 2024-08-19T12:45:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f66dc003610 con 0x7f66ec0a5d50 2024-08-19T12:45:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:45:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): dump: 2024-08-19T12:45:05.467 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.468 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.469 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.469 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.447+0000 7f66e2ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:05.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2896752284 0 0) 0x7f66ec0a0fa0 con 0x7f66ec0a5d50 2024-08-19T12:45:05.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.104:6792/0 conn(0x7f66ec06e7b0 legacy=0x7f66ec09e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.104:6789/0 conn(0x7f66ec0a5920 legacy=0x7f66ec09efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:05.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 1 monclient: found mon.a 2024-08-19T12:45:05.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:05.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:05.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec0a09e0 con 0x7f66ec0a5d50 2024-08-19T12:45:05.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 10 monclient: _finish_auth 0 2024-08-19T12:45:05.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:05.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:05.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66f0898640 5 monclient: authenticate success, global_id 144459 2024-08-19T12:45:05.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66f0898640 10 monclient: _renew_subs 2024-08-19T12:45:05.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66f0898640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:05.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66f0898640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f66ec1321e0 con 0x7f66ec0a5d50 2024-08-19T12:45:05.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66f0898640 10 client.144459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:05.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66f0898640 10 monclient: _renew_subs 2024-08-19T12:45:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66f0898640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66f0898640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f66ec09f8b0 con 0x7f66ec0a5d50 2024-08-19T12:45:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66f0898640 1 librados: init done 2024-08-19T12:45:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f66dc003a60 con 0x7f66ec0a5d50 2024-08-19T12:45:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f66dc003f90 con 0x7f66ec0a5d50 2024-08-19T12:45:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e8d99640 4 set_mon_vals no callback set 2024-08-19T12:45:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66e2ffd640 10 monclient: dump: 2024-08-19T12:45:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.471 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.472 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.472 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.472 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.472 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.472 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.473 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.473 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.473 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.473 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.473 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.473 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.473 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.473 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.473 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66f0898640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:05.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.448+0000 7f66f0898640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f66d00065c0 con 0x7f66d0002860 2024-08-19T12:45:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.449+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f66dc0041e0 con 0x7f66ec0a5d50 2024-08-19T12:45:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.449+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(278..278 src has 1..278) ==== 5338+0+0 (unknown 991588785 0 0) 0x7f66dc04df80 con 0x7f66ec0a5d50 2024-08-19T12:45:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.449+0000 7f66e2ffd640 10 client.144459.objecter ms_dispatch 0x7f66ec000b90 osd_map(278..278 src has 1..278) 2024-08-19T12:45:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.449+0000 7f66e2ffd640 3 client.144459.objecter handle_osd_map got epochs [278,278] > 0 2024-08-19T12:45:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.449+0000 7f66e2ffd640 3 client.144459.objecter handle_osd_map decoding full epoch 278 2024-08-19T12:45:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.449+0000 7f66e2ffd640 20 client.144459.objecter dump_active .. 0 homeless 2024-08-19T12:45:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.449+0000 7f66e9d9b640 10 monclient: get_auth_request con 0x7f66cc040fd0 auth_method 0 2024-08-19T12:45:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66ea59c640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.26:6791/0 conn(0x7f66d0002860 legacy=0x7f66d0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 10 client.144459.objecter ms_handle_connect 0x7f66d0002860 2024-08-19T12:45:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 10 client.144459.objecter resend_mon_ops 2024-08-19T12:45:05.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292320162 0 0) 0x7f66d00065c0 con 0x7f66d0002860 2024-08-19T12:45:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 10 monclient(hunting): my global_id is 26563 2024-08-19T12:45:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66ec0a0fa0 con 0x7f66d0002860 2024-08-19T12:45:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4057928221 0 0) 0x7f66ec0a0fa0 con 0x7f66d0002860 2024-08-19T12:45:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66d00065c0 con 0x7f66d0002860 2024-08-19T12:45:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f66e40020b0 con 0x7f66d0002860 2024-08-19T12:45:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:05.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 10 monclient: dump: 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.476 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.477 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3176694750 0 0) 0x7f66d00065c0 con 0x7f66d0002860 2024-08-19T12:45:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f66d000a160 con 0x7f66d0002860 2024-08-19T12:45:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.451+0000 7f66e2ffd640 10 client.144459.objecter ms_handle_connect 0x7f66cc040fd0 2024-08-19T12:45:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.452+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f66e4005750 con 0x7f66d0002860 2024-08-19T12:45:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.452+0000 7f66e2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.452+0000 7f66e2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.452+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.26:6791/0 conn(0x7f66d0002860 legacy=0x7f66d0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.452+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.26:6791/0 conn(0x7f66d0002860 legacy=0x7f66d0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.458+0000 7f66f0898640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.458+0000 7f66f0898640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f66d0009e70 con 0x7f66d0009130 2024-08-19T12:45:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66ea59c640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.26:6791/0 conn(0x7f66d0009130 legacy=0x7f66d00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 10 client.144459.objecter ms_handle_connect 0x7f66d0009130 2024-08-19T12:45:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 10 client.144459.objecter resend_mon_ops 2024-08-19T12:45:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3962192191 0 0) 0x7f66d0009e70 con 0x7f66d0009130 2024-08-19T12:45:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 10 monclient(hunting): my global_id is 26572 2024-08-19T12:45:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66d00065c0 con 0x7f66d0009130 2024-08-19T12:45:05.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2360962362 0 0) 0x7f66d00065c0 con 0x7f66d0009130 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66d0009e70 con 0x7f66d0009130 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f66e4005e70 con 0x7f66d0009130 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 10 monclient: dump: 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.479 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.480 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.480 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.480 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.480 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.480 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.480 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.480 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1242919667 0 0) 0x7f66d0009e70 con 0x7f66d0009130 2024-08-19T12:45:05.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.460+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f66d000c3f0 con 0x7f66d0009130 2024-08-19T12:45:05.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7878 (unknown 0 0 576921645) 0x7f66e40081d0 con 0x7f66d0009130 2024-08-19T12:45:05.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66e2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:05.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66e2ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:05.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.26:6791/0 conn(0x7f66d0009130 legacy=0x7f66d00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66e2ffd640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.26:6791/0 conn(0x7f66d0009130 legacy=0x7f66d00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.481 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:05.481 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:05.481 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:05.481 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 418491, 2024-08-19T12:45:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.483 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.484 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.485 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.485 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.485 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.485 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.485 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.485 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.485 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.486 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:05.486 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:05.486 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:05.486 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:05.486 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:05.486 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:05.486 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:05.486 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:05.486 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:05.486 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:05.486 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.487 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.489 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:05.490 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.491 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.493 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:05.494 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.495 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.496 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.497 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:05.498 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.500 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.501 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.502 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:05.503 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:05.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 10 librados: watch_flush enter 2024-08-19T12:45:05.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 10 librados: watch_flush exit 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 20 client.144459.objecter shutdown clearing up homeless session... 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 10 client.144459.objecter successfully canceled tick 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.104:6800/899887506 conn(0x7f66cc040fd0 legacy=0x7f66cc043490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 10 monclient: shutdown 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 1 -- 172.21.15.26:0/1235749099 >> v1:172.21.15.26:6789/0 conn(0x7f66ec0a5d50 legacy=0x7f66ec1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66ead9d640 1 -- 172.21.15.26:0/1235749099 reap_dead start 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66ead9d640 1 -- 172.21.15.26:0/1235749099 reap_dead start 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 1 -- 172.21.15.26:0/1235749099 shutdown_connections 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 1 -- 172.21.15.26:0/1235749099 >> 172.21.15.26:0/1235749099 conn(0x7f66ec0a8740 msgr2=0x7f66ec06d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 1 -- 172.21.15.26:0/1235749099 shutdown_connections 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 1 -- 172.21.15.26:0/1235749099 wait complete. 2024-08-19T12:45:05.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.461+0000 7f66f0898640 1 librados: shutdown 2024-08-19T12:45:05.506 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:05.506 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:45:05.506 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:45:05.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.611+0000 7f222ca20640 10 monclient: get_monmap_and_config 2024-08-19T12:45:05.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.611+0000 7f222ca20640 10 monclient: build_initial_monmap 2024-08-19T12:45:05.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.611+0000 7f222ca20640 10 monclient: monmap: 2024-08-19T12:45:05.613 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:05.613 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.613 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:05.613151+0000 2024-08-19T12:45:05.613 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:05.613151+0000 2024-08-19T12:45:05.613 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:05.614 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:05.614 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 1 Processor -- start 2024-08-19T12:45:05.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 1 -- start start 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 10 monclient: init 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 10 monclient: _add_conns ranks=[3,0,7,8,1,2,6,4,5] 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 10 monclient(hunting): picked mon.noname-g con 0x7f222806e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 10 monclient(hunting): picked mon.noname-a con 0x7f22280a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 10 monclient(hunting): picked mon.noname-i con 0x7f22280a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2228136790 con 0x7f22280a5920 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f222809ddf0 con 0x7f222806e7b0 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f222809e0d0 con 0x7f22280a5d50 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f222ca20640 10 monclient(hunting): authenticate will time out at 1099.700580s 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f221ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f22280a5920 0x7f222813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45252/0 (socket says 172.21.15.26:45252) 2024-08-19T12:45:05.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.612+0000 7f221ffff640 1 -- 172.21.15.26:0/2927691793 learned_addr learned my addr 172.21.15.26:0/2927691793 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.614+0000 7f221ffff640 1 -- 172.21.15.26:0/2927691793 >> v1:172.21.15.26:6789/0 conn(0x7f22280a5920 legacy=0x7f222813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2226d9d640 1 -- 172.21.15.26:0/2927691793 >> v1:172.21.15.104:6792/0 conn(0x7f22280a5d50 legacy=0x7f2228141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f222659c640 1 -- 172.21.15.26:0/2927691793 >> v1:172.21.15.104:6790/0 conn(0x7f222806e7b0 legacy=0x7f2228136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3847785944 0 0) 0x7f2228136790 con 0x7f22280a5920 2024-08-19T12:45:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient(hunting): my global_id is 144468 2024-08-19T12:45:05.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2214003520 con 0x7f22280a5920 2024-08-19T12:45:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 817464123 0 0) 0x7f22180013d0 con 0x7f222806e7b0 2024-08-19T12:45:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient(hunting): my global_id is 154110 2024-08-19T12:45:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2228136790 con 0x7f222806e7b0 2024-08-19T12:45:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2962581726 0 0) 0x7f222809e0d0 con 0x7f22280a5d50 2024-08-19T12:45:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient(hunting): my global_id is 135160 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22180013d0 con 0x7f22280a5d50 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1277218248 0 0) 0x7f2214003520 con 0x7f22280a5920 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 >> v1:172.21.15.104:6792/0 conn(0x7f22280a5d50 legacy=0x7f2228141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 >> v1:172.21.15.104:6790/0 conn(0x7f222806e7b0 legacy=0x7f2228136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 monclient: found mon.noname-a 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222809e3b0 con 0x7f22280a5920 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient: _finish_auth 0 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f222ca20640 5 monclient: authenticate success, global_id 144468 2024-08-19T12:45:05.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2210001ae0 con 0x7f22280a5920 2024-08-19T12:45:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:45:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient: dump: 2024-08-19T12:45:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.628 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.629 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.629 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.629 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.629 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.629 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.629 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.629 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f222ca20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:05.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f22100048a0 con 0x7f22280a5920 2024-08-19T12:45:05.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:05.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f222ca20640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:05.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 1 -- 172.21.15.26:0/2927691793 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2210004e10 con 0x7f22280a5920 2024-08-19T12:45:05.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f222ca20640 4 set_mon_vals no callback set 2024-08-19T12:45:05.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:05.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.615+0000 7f2225d9b640 10 monclient: dump: 2024-08-19T12:45:05.630 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.630 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.630 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.630 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.630 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.630 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.630 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.630 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.616+0000 7f222ca20640 10 monclient: shutdown 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.616+0000 7f222ca20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.616+0000 7f222ca20640 1 -- 172.21.15.26:0/2927691793 >> v1:172.21.15.26:6789/0 conn(0x7f22280a5920 legacy=0x7f222813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.616+0000 7f222ca20640 1 -- 172.21.15.26:0/2927691793 shutdown_connections 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.616+0000 7f222ca20640 1 -- 172.21.15.26:0/2927691793 >> 172.21.15.26:0/2927691793 conn(0x7f22280a8740 msgr2=0x7f22280a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.616+0000 7f222ca20640 1 -- 172.21.15.26:0/2927691793 shutdown_connections 2024-08-19T12:45:05.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.616+0000 7f222ca20640 1 -- 172.21.15.26:0/2927691793 wait complete. 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.616+0000 7f222ca20640 10 monclient: build_initial_monmap 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.616+0000 7f222ca20640 10 monclient: monmap: 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:05.632 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:05.633 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:05.633 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:05.633 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:05.633 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:05.633 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 1 librados: starting msgr at 2024-08-19T12:45:05.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 1 librados: starting objecter 2024-08-19T12:45:05.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 1 Processor -- start 2024-08-19T12:45:05.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 1 -- start start 2024-08-19T12:45:05.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 1 librados: setting wanted keys 2024-08-19T12:45:05.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 1 librados: calling monclient init 2024-08-19T12:45:05.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 10 monclient: init 2024-08-19T12:45:05.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:05.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 10 monclient: _add_conns ranks=[6,8,2,5,3,0,1,4,7] 2024-08-19T12:45:05.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 10 monclient(hunting): picked mon.noname-d con 0x7f222806e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:05.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 10 monclient(hunting): picked mon.noname-e con 0x7f22280a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:05.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 10 monclient(hunting): picked mon.noname-b con 0x7f22280a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:05.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2228138fd0 con 0x7f22280a5d50 2024-08-19T12:45:05.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2228139b30 con 0x7f222806e7b0 2024-08-19T12:45:05.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f222813aa90 con 0x7f22280a5920 2024-08-19T12:45:05.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:05.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222ca20640 10 monclient(hunting): authenticate will time out at 1099.705849s 2024-08-19T12:45:05.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222659c640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f222806e7b0 0x7f2228137920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49804/0 (socket says 172.21.15.26:49804) 2024-08-19T12:45:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f222659c640 1 -- 172.21.15.26:0/620091256 learned_addr learned my addr 172.21.15.26:0/620091256 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.617+0000 7f221ffff640 1 --1- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6793/0 conn(0x7f22280a5920 0x7f2228138030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34224/0 (socket says 172.21.15.26:34224) 2024-08-19T12:45:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f2226d9d640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6790/0 conn(0x7f22280a5d50 legacy=0x7f22281d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f222659c640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6792/0 conn(0x7f222806e7b0 legacy=0x7f2228137920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221ffff640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6793/0 conn(0x7f22280a5920 legacy=0x7f2228138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 client.?.objecter ms_handle_connect 0x7f22280a5d50 2024-08-19T12:45:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 client.?.objecter ms_handle_connect 0x7f222806e7b0 2024-08-19T12:45:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 client.?.objecter ms_handle_connect 0x7f22280a5920 2024-08-19T12:45:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1455595817 0 0) 0x7f2228139b30 con 0x7f222806e7b0 2024-08-19T12:45:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): my global_id is 26565 2024-08-19T12:45:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2208004eb0 con 0x7f222806e7b0 2024-08-19T12:45:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 913103137 0 0) 0x7f222813aa90 con 0x7f22280a5920 2024-08-19T12:45:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): my global_id is 26090 2024-08-19T12:45:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2228139b30 con 0x7f22280a5920 2024-08-19T12:45:05.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 332592476 0 0) 0x7f2228138fd0 con 0x7f22280a5d50 2024-08-19T12:45:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): my global_id is 26255 2024-08-19T12:45:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f222813aa90 con 0x7f22280a5d50 2024-08-19T12:45:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 275501653 0 0) 0x7f2208004eb0 con 0x7f222806e7b0 2024-08-19T12:45:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2228138fd0 con 0x7f222806e7b0 2024-08-19T12:45:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2218003950 con 0x7f222806e7b0 2024-08-19T12:45:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:45:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): dump: 2024-08-19T12:45:05.637 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.638 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.639 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 94155318 0 0) 0x7f2228139b30 con 0x7f22280a5920 2024-08-19T12:45:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2208004eb0 con 0x7f22280a5920 2024-08-19T12:45:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2210004d90 con 0x7f22280a5920 2024-08-19T12:45:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): dump: 2024-08-19T12:45:05.639 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.640 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.641 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1473392781 0 0) 0x7f222813aa90 con 0x7f22280a5d50 2024-08-19T12:45:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2228139b30 con 0x7f22280a5d50 2024-08-19T12:45:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2220002070 con 0x7f22280a5d50 2024-08-19T12:45:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient(hunting): dump: 2024-08-19T12:45:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.641 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.642 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2774064144 0 0) 0x7f2228138fd0 con 0x7f222806e7b0 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6793/0 conn(0x7f22280a5920 legacy=0x7f2228138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6790/0 conn(0x7f22280a5d50 legacy=0x7f22281d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 monclient: found mon.d 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222813ad70 con 0x7f222806e7b0 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient: _finish_auth 0 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:05.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f222ca20640 5 monclient: authenticate success, global_id 26565 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f222ca20640 10 monclient: _renew_subs 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f222ca20640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f222ca20640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f222813a6f0 con 0x7f222806e7b0 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f222ca20640 10 client.26565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f222ca20640 10 monclient: _renew_subs 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f222ca20640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f222ca20640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f2228138740 con 0x7f222806e7b0 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f222ca20640 1 librados: init done 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2218003e60 con 0x7f222806e7b0 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2218004390 con 0x7f222806e7b0 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f222559a640 4 set_mon_vals no callback set 2024-08-19T12:45:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:05.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.620+0000 7f221effd640 10 monclient: dump: 2024-08-19T12:45:05.645 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.645 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.645 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.645 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.645 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.645 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.645 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.645 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.645 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.645 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.645 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.621+0000 7f222ca20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.621+0000 7f222ca20640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f220c0065c0 con 0x7f220c002860 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.621+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2218011a80 con 0x7f222806e7b0 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.622+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(278..278 src has 1..278) ==== 5338+0+0 (unknown 991588785 0 0) 0x7f221804e3b0 con 0x7f222806e7b0 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.622+0000 7f221effd640 10 client.26565.objecter ms_dispatch 0x7f2228000b90 osd_map(278..278 src has 1..278) 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.622+0000 7f221effd640 3 client.26565.objecter handle_osd_map got epochs [278,278] > 0 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.622+0000 7f221effd640 3 client.26565.objecter handle_osd_map decoding full epoch 278 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.622+0000 7f221effd640 20 client.26565.objecter dump_active .. 0 homeless 2024-08-19T12:45:05.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.622+0000 7f2226d9d640 10 monclient: get_auth_request con 0x7f22080418a0 auth_method 0 2024-08-19T12:45:05.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.623+0000 7f221effd640 10 client.26565.objecter ms_handle_connect 0x7f220c002860 2024-08-19T12:45:05.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.623+0000 7f221effd640 10 client.26565.objecter resend_mon_ops 2024-08-19T12:45:05.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.623+0000 7f221ffff640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6792/0 conn(0x7f220c002860 legacy=0x7f220c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771548667 0 0) 0x7f220c0065c0 con 0x7f220c002860 2024-08-19T12:45:05.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 10 monclient(hunting): my global_id is 26574 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2228138fd0 con 0x7f220c002860 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 192629686 0 0) 0x7f2228138fd0 con 0x7f220c002860 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f220c0065c0 con 0x7f220c002860 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2210001580 con 0x7f220c002860 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 10 monclient: dump: 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.650 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2766420932 0 0) 0x7f220c0065c0 con 0x7f220c002860 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f220c00a160 con 0x7f220c002860 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 10 client.26565.objecter ms_handle_connect 0x7f22080418a0 2024-08-19T12:45:05.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f22100048a0 con 0x7f220c002860 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6792/0 conn(0x7f220c002860 legacy=0x7f220c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.624+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6792/0 conn(0x7f220c002860 legacy=0x7f220c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.630+0000 7f222ca20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.630+0000 7f222ca20640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f220c009e70 con 0x7f220c009130 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 10 client.26565.objecter ms_handle_connect 0x7f220c009130 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 10 client.26565.objecter resend_mon_ops 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221ffff640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6792/0 conn(0x7f220c009130 legacy=0x7f220c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3927866397 0 0) 0x7f220c009e70 con 0x7f220c009130 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 10 monclient(hunting): my global_id is 26583 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f220c0065c0 con 0x7f220c009130 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2751041522 0 0) 0x7f220c0065c0 con 0x7f220c009130 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f220c009e70 con 0x7f220c009130 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2210005cb0 con 0x7f220c009130 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 10 monclient: dump: 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.653 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1765446543 0 0) 0x7f220c009e70 con 0x7f220c009130 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f220c00c3f0 con 0x7f220c009130 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 2212673968) 0x7f2210001d30 con 0x7f220c009130 2024-08-19T12:45:05.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:05.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:05.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6792/0 conn(0x7f220c009130 legacy=0x7f220c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.633+0000 7f221effd640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6792/0 conn(0x7f220c009130 legacy=0x7f220c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.655 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 418674, 2024-08-19T12:45:05.655 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:05.656 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:45:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.657 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.658 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.659 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:05.660 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.661 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:05.662 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:05.663 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.664 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.668 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:05.674 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.675 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:05.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.677 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.678 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:05.678 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:05.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 10 librados: watch_flush enter 2024-08-19T12:45:05.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 10 librados: watch_flush exit 2024-08-19T12:45:05.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 20 client.26565.objecter shutdown clearing up homeless session... 2024-08-19T12:45:05.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 10 client.26565.objecter successfully canceled tick 2024-08-19T12:45:05.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.104:6800/899887506 conn(0x7f22080418a0 legacy=0x7f2208043d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 10 monclient: shutdown 2024-08-19T12:45:05.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:05.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 1 -- 172.21.15.26:0/620091256 >> v1:172.21.15.26:6792/0 conn(0x7f222806e7b0 legacy=0x7f2228137920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f2226d9d640 1 -- 172.21.15.26:0/620091256 reap_dead start 2024-08-19T12:45:05.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f2226d9d640 1 -- 172.21.15.26:0/620091256 reap_dead start 2024-08-19T12:45:05.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 1 -- 172.21.15.26:0/620091256 shutdown_connections 2024-08-19T12:45:05.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 1 -- 172.21.15.26:0/620091256 >> 172.21.15.26:0/620091256 conn(0x7f22280a8740 msgr2=0x7f22280aa000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:05.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 1 -- 172.21.15.26:0/620091256 shutdown_connections 2024-08-19T12:45:05.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 1 -- 172.21.15.26:0/620091256 wait complete. 2024-08-19T12:45:05.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.634+0000 7f222ca20640 1 librados: shutdown 2024-08-19T12:45:05.681 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:05.681 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:45:05.682 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:45:05.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.778+0000 7f2ae64f6640 10 monclient: get_monmap_and_config 2024-08-19T12:45:05.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.778+0000 7f2ae64f6640 10 monclient: build_initial_monmap 2024-08-19T12:45:05.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.778+0000 7f2ae64f6640 10 monclient: monmap: 2024-08-19T12:45:05.781 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:05.781 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.781 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:05.780780+0000 2024-08-19T12:45:05.781 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:05.780780+0000 2024-08-19T12:45:05.781 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:05.781 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.781 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:05.782 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:05.782 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:05.782 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:05.782 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:05.782 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:05.782 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:05.782 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:05.782 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:05.782 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.779+0000 7f2ae64f6640 1 Processor -- start 2024-08-19T12:45:05.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.779+0000 7f2ae64f6640 1 -- start start 2024-08-19T12:45:05.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.779+0000 7f2ae64f6640 10 monclient: init 2024-08-19T12:45:05.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2ae64f6640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:05.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2ae64f6640 10 monclient: _add_conns ranks=[3,5,1,2,7,6,8,0,4] 2024-08-19T12:45:05.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2ae64f6640 10 monclient(hunting): picked mon.noname-g con 0x7f2ae006e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:05.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2ae64f6640 10 monclient(hunting): picked mon.noname-h con 0x7f2ae00a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:05.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2ae64f6640 10 monclient(hunting): picked mon.noname-f con 0x7f2ae00a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:05.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2ae64f6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae0136790 con 0x7f2ae00a5d50 2024-08-19T12:45:05.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2ae64f6640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae009ddf0 con 0x7f2ae006e7b0 2024-08-19T12:45:05.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2ae64f6640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae009e0d0 con 0x7f2ae00a5920 2024-08-19T12:45:05.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2ae64f6640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:05.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2ae64f6640 10 monclient(hunting): authenticate will time out at 1099.868212s 2024-08-19T12:45:05.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2adffff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f2ae006e7b0 0x7f2ae0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:44302/0 (socket says 172.21.15.26:44302) 2024-08-19T12:45:05.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2ae4a93640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2ae00a5d50 0x7f2ae0141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:52122/0 (socket says 172.21.15.26:52122) 2024-08-19T12:45:05.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.780+0000 7f2adffff640 1 -- 172.21.15.26:0/2190182415 learned_addr learned my addr 172.21.15.26:0/2190182415 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:05.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adffff640 1 -- 172.21.15.26:0/2190182415 >> v1:172.21.15.104:6790/0 conn(0x7f2ae006e7b0 legacy=0x7f2ae0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2ae4a93640 1 -- 172.21.15.26:0/2190182415 >> v1:172.21.15.104:6789/0 conn(0x7f2ae00a5d50 legacy=0x7f2ae0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adf7fe640 1 -- 172.21.15.26:0/2190182415 >> v1:172.21.15.104:6791/0 conn(0x7f2ae00a5920 legacy=0x7f2ae013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4291394541 0 0) 0x7f2ad0001450 con 0x7f2ae006e7b0 2024-08-19T12:45:05.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 10 monclient(hunting): my global_id is 154119 2024-08-19T12:45:05.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ac8003520 con 0x7f2ae006e7b0 2024-08-19T12:45:05.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1493453968 0 0) 0x7f2ae0136790 con 0x7f2ae00a5d50 2024-08-19T12:45:05.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 10 monclient(hunting): my global_id is 26965 2024-08-19T12:45:05.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ad0001450 con 0x7f2ae00a5d50 2024-08-19T12:45:05.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 522922076 0 0) 0x7f2ae009e0d0 con 0x7f2ae00a5920 2024-08-19T12:45:05.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 10 monclient(hunting): my global_id is 26222 2024-08-19T12:45:05.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.782+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ae0136790 con 0x7f2ae00a5920 2024-08-19T12:45:05.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2925512408 0 0) 0x7f2ac8003520 con 0x7f2ae006e7b0 2024-08-19T12:45:05.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 >> v1:172.21.15.104:6791/0 conn(0x7f2ae00a5920 legacy=0x7f2ae013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 >> v1:172.21.15.104:6789/0 conn(0x7f2ae00a5d50 legacy=0x7f2ae0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:05.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 1 monclient: found mon.noname-g 2024-08-19T12:45:05.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:05.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae009e3b0 con 0x7f2ae006e7b0 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 10 monclient: _finish_auth 0 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ad0001cb0 con 0x7f2ae006e7b0 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2ae64f6640 5 monclient: authenticate success, global_id 154119 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2ae64f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 10 monclient: dump: 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.795 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.796 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.796 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.796 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.796 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.796 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.796 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.796 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.796 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.796 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.796 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.796 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.796 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2ae64f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:05.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ad0004950 con 0x7f2ae006e7b0 2024-08-19T12:45:05.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:05.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 1 -- 172.21.15.26:0/2190182415 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ad0004f00 con 0x7f2ae006e7b0 2024-08-19T12:45:05.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2ae64f6640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:05.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2ae64f6640 4 set_mon_vals no callback set 2024-08-19T12:45:05.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:05.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2adeffd640 10 monclient: dump: 2024-08-19T12:45:05.797 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.797 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.798 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.798 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.798 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.798 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.798 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.798 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.798 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.798 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.798 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.798 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.798 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2ae64f6640 10 monclient: shutdown 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2ae64f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2ae64f6640 1 -- 172.21.15.26:0/2190182415 >> v1:172.21.15.104:6790/0 conn(0x7f2ae006e7b0 legacy=0x7f2ae0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2ae64f6640 1 -- 172.21.15.26:0/2190182415 shutdown_connections 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2ae64f6640 1 -- 172.21.15.26:0/2190182415 >> 172.21.15.26:0/2190182415 conn(0x7f2ae00a8740 msgr2=0x7f2ae00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2ae64f6640 1 -- 172.21.15.26:0/2190182415 shutdown_connections 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.783+0000 7f2ae64f6640 1 -- 172.21.15.26:0/2190182415 wait complete. 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.784+0000 7f2ae64f6640 10 monclient: build_initial_monmap 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.784+0000 7f2ae64f6640 10 monclient: monmap: 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:05.800 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:05.800 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:05.800 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:05.800 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.800 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:05.800 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:05.800 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:05.800 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:05.800 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:05.800 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:05.800 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:05.800 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:05.801 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:05.801 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.784+0000 7f2ae64f6640 1 librados: starting msgr at 2024-08-19T12:45:05.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.784+0000 7f2ae64f6640 1 librados: starting objecter 2024-08-19T12:45:05.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.784+0000 7f2ae64f6640 1 Processor -- start 2024-08-19T12:45:05.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.784+0000 7f2ae64f6640 1 -- start start 2024-08-19T12:45:05.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.784+0000 7f2ae64f6640 1 librados: setting wanted keys 2024-08-19T12:45:05.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.784+0000 7f2ae64f6640 1 librados: calling monclient init 2024-08-19T12:45:05.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.784+0000 7f2ae64f6640 10 monclient: init 2024-08-19T12:45:05.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.785+0000 7f2ae64f6640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:05.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.785+0000 7f2ae64f6640 10 monclient: _add_conns ranks=[0,8,1,3,6,5,2,7,4] 2024-08-19T12:45:05.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.785+0000 7f2ae64f6640 10 monclient(hunting): picked mon.noname-a con 0x7f2ae006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:05.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.785+0000 7f2ae64f6640 10 monclient(hunting): picked mon.noname-e con 0x7f2ae00a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:05.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.785+0000 7f2ae64f6640 10 monclient(hunting): picked mon.noname-f con 0x7f2ae00a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:05.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.785+0000 7f2ae64f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae0141d00 con 0x7f2ae006e7b0 2024-08-19T12:45:05.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.785+0000 7f2ae64f6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae01346f0 con 0x7f2ae00a5d50 2024-08-19T12:45:05.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.785+0000 7f2ae64f6640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae0142ea0 con 0x7f2ae00a5920 2024-08-19T12:45:05.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.785+0000 7f2ae64f6640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:05.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.785+0000 7f2ae64f6640 10 monclient(hunting): authenticate will time out at 1099.873414s 2024-08-19T12:45:05.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.785+0000 7f2adffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2ae006e7b0 0x7f2ae01374c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45254/0 (socket says 172.21.15.26:45254) 2024-08-19T12:45:05.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.785+0000 7f2adffff640 1 -- 172.21.15.26:0/3904853363 learned_addr learned my addr 172.21.15.26:0/3904853363 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:05.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adf7fe640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.26:6793/0 conn(0x7f2ae00a5920 legacy=0x7f2ae0137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 client.?.objecter ms_handle_connect 0x7f2ae00a5920 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 client.?.objecter ms_handle_connect 0x7f2ae006e7b0 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adffff640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.26:6789/0 conn(0x7f2ae006e7b0 legacy=0x7f2ae01374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2ae4a93640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.104:6789/0 conn(0x7f2ae00a5d50 legacy=0x7f2ae01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 client.?.objecter ms_handle_connect 0x7f2ae00a5d50 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3953948160 0 0) 0x7f2ad40015d0 con 0x7f2ae00a5920 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 monclient(hunting): my global_id is 26099 2024-08-19T12:45:05.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2acc004eb0 con 0x7f2ae00a5920 2024-08-19T12:45:05.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2338488939 0 0) 0x7f2ae0141d00 con 0x7f2ae006e7b0 2024-08-19T12:45:05.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 10 monclient(hunting): my global_id is 144477 2024-08-19T12:45:05.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.787+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ad40015d0 con 0x7f2ae006e7b0 2024-08-19T12:45:05.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659061917 0 0) 0x7f2ad8002090 con 0x7f2ae00a5d50 2024-08-19T12:45:05.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient(hunting): my global_id is 26974 2024-08-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ae0141d00 con 0x7f2ae00a5d50 2024-08-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1710931711 0 0) 0x7f2acc004eb0 con 0x7f2ae00a5920 2024-08-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ad8002090 con 0x7f2ae00a5920 2024-08-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ad4003950 con 0x7f2ae00a5920 2024-08-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient(hunting): dump: 2024-08-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.806 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.807 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:05.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3720323676 0 0) 0x7f2ad40015d0 con 0x7f2ae006e7b0 2024-08-19T12:45:05.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2acc004eb0 con 0x7f2ae006e7b0 2024-08-19T12:45:05.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ad0001cb0 con 0x7f2ae006e7b0 2024-08-19T12:45:05.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:05.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:05.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient(hunting): dump: 2024-08-19T12:45:05.807 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.807 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.807 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.808 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4287397187 0 0) 0x7f2ad8002090 con 0x7f2ae00a5920 2024-08-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.104:6789/0 conn(0x7f2ae00a5d50 legacy=0x7f2ae01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.26:6789/0 conn(0x7f2ae006e7b0 legacy=0x7f2ae01374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient: _finish_hunting 0 2024-08-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 monclient: found mon.e 2024-08-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae01382e0 con 0x7f2ae00a5920 2024-08-19T12:45:05.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient: _finish_auth 0 2024-08-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient: _check_auth_tickets 2024-08-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2ae64f6640 5 monclient: authenticate success, global_id 26099 2024-08-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2ae64f6640 10 monclient: _renew_subs 2024-08-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2ae64f6640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2ae64f6640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae013ad90 con 0x7f2ae00a5920 2024-08-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2ae64f6640 10 client.26099.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2ae64f6640 10 monclient: _renew_subs 2024-08-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2ae64f6640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2ae64f6640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f2ae01427e0 con 0x7f2ae00a5920 2024-08-19T12:45:05.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2ae64f6640 1 librados: init done 2024-08-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ad4003da0 con 0x7f2ae00a5920 2024-08-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ad40042d0 con 0x7f2ae00a5920 2024-08-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2ade7fc640 4 set_mon_vals no callback set 2024-08-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.788+0000 7f2adcff9640 10 monclient: dump: 2024-08-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.811 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.789+0000 7f2ae64f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:05.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.789+0000 7f2ae64f6640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ac00065c0 con 0x7f2ac0002860 2024-08-19T12:45:05.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.789+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2ad4011a80 con 0x7f2ae00a5920 2024-08-19T12:45:05.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.789+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(278..278 src has 1..278) ==== 5338+0+0 (unknown 991588785 0 0) 0x7f2ad404e360 con 0x7f2ae00a5920 2024-08-19T12:45:05.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.789+0000 7f2adcff9640 10 client.26099.objecter ms_dispatch 0x7f2ae0000b90 osd_map(278..278 src has 1..278) 2024-08-19T12:45:05.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.789+0000 7f2adcff9640 3 client.26099.objecter handle_osd_map got epochs [278,278] > 0 2024-08-19T12:45:05.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.789+0000 7f2adcff9640 3 client.26099.objecter handle_osd_map decoding full epoch 278 2024-08-19T12:45:05.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.789+0000 7f2adcff9640 20 client.26099.objecter dump_active .. 0 homeless 2024-08-19T12:45:05.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.789+0000 7f2ae4a93640 10 monclient: get_auth_request con 0x7f2acc041610 auth_method 0 2024-08-19T12:45:05.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 10 client.26099.objecter ms_handle_connect 0x7f2ac0002860 2024-08-19T12:45:05.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adffff640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.26:6793/0 conn(0x7f2ac0002860 legacy=0x7f2ac0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 10 client.26099.objecter resend_mon_ops 2024-08-19T12:45:05.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3996775158 0 0) 0x7f2ac00065c0 con 0x7f2ac0002860 2024-08-19T12:45:05.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 10 monclient(hunting): my global_id is 26108 2024-08-19T12:45:05.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ad8002090 con 0x7f2ac0002860 2024-08-19T12:45:05.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1971620431 0 0) 0x7f2ad8002090 con 0x7f2ac0002860 2024-08-19T12:45:05.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ac00065c0 con 0x7f2ac0002860 2024-08-19T12:45:05.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ad0004910 con 0x7f2ac0002860 2024-08-19T12:45:05.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:05.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 10 monclient: dump: 2024-08-19T12:45:05.817 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.817 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.818 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.818 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.818 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.818 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.818 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.818 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.818 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.818 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.819 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.819 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.819 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1402316565 0 0) 0x7f2ac00065c0 con 0x7f2ac0002860 2024-08-19T12:45:05.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.791+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ac000a160 con 0x7f2ac0002860 2024-08-19T12:45:05.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.792+0000 7f2adcff9640 10 client.26099.objecter ms_handle_connect 0x7f2acc041610 2024-08-19T12:45:05.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.792+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2ad0005500 con 0x7f2ac0002860 2024-08-19T12:45:05.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.792+0000 7f2adcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:05.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.792+0000 7f2adcff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:05.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.792+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.26:6793/0 conn(0x7f2ac0002860 legacy=0x7f2ac0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.792+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.26:6793/0 conn(0x7f2ac0002860 legacy=0x7f2ac0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.798+0000 7f2ae64f6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:05.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.798+0000 7f2ae64f6640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ac0009e70 con 0x7f2ac0009130 2024-08-19T12:45:05.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.800+0000 7f2adffff640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.26:6793/0 conn(0x7f2ac0009130 legacy=0x7f2ac00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.800+0000 7f2adcff9640 10 client.26099.objecter ms_handle_connect 0x7f2ac0009130 2024-08-19T12:45:05.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.800+0000 7f2adcff9640 10 client.26099.objecter resend_mon_ops 2024-08-19T12:45:05.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.800+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2761994114 0 0) 0x7f2ac0009e70 con 0x7f2ac0009130 2024-08-19T12:45:05.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.800+0000 7f2adcff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.800+0000 7f2adcff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.800+0000 7f2adcff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.800+0000 7f2adcff9640 10 monclient(hunting): my global_id is 26117 2024-08-19T12:45:05.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.800+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ac00065c0 con 0x7f2ac0009130 2024-08-19T12:45:05.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3785716958 0 0) 0x7f2ac00065c0 con 0x7f2ac0009130 2024-08-19T12:45:05.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ac0009e70 con 0x7f2ac0009130 2024-08-19T12:45:05.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ad0005c60 con 0x7f2ac0009130 2024-08-19T12:45:05.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:05.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 10 monclient: dump: 2024-08-19T12:45:05.821 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.821 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.821 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 478602980 0 0) 0x7f2ac0009e70 con 0x7f2ac0009130 2024-08-19T12:45:05.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ac000c3f0 con 0x7f2ac0009130 2024-08-19T12:45:05.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1531484435) 0x7f2ad0005ff0 con 0x7f2ac0009130 2024-08-19T12:45:05.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:05.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:05.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.26:6793/0 conn(0x7f2ac0009130 legacy=0x7f2ac00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2adcff9640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.26:6793/0 conn(0x7f2ac0009130 legacy=0x7f2ac00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.823 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:05.823 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:05.823 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:05.823 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:05.823 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:05.824 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 418818, 2024-08-19T12:45:05.824 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:05.824 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:05.824 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:05.824 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:05.824 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:05.824 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:05.824 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:05.824 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:05.824 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:05.824 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:05.824 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.825 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:45:05.825 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:05.825 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:05.825 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:05.825 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.825 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.825 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.825 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.825 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.825 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.825 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.825 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.826 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:05.827 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:05.828 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:05.828 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:05.828 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:05.828 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:05.828 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:05.828 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:05.828 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:05.828 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:05.828 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:05.828 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.828 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.828 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.829 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.829 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.829 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.829 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.829 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.829 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.829 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.829 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.829 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.829 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:05.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.829 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:05.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:05.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:05.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:05.837 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:05.837 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.837 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.838 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.838 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:05.838 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:05.838 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.838 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:05.838 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:05.839 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:05.839 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.839 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.839 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.839 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:05.839 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:05.839 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:05.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:05.840 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:05.840 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.841 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.841 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.841 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:05.841 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:05.841 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:05.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:05.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:05.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:05.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:05.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:05.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:05.842 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:05.842 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:05.842 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:05.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.843 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:05.843 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:05.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.843 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.843 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.843 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:05.843 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.843 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.843 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:05.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:05.843 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:05.844 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:05.844 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:05.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:05.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.844 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:05.844 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:05.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2ae64f6640 10 librados: watch_flush enter 2024-08-19T12:45:05.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2ae64f6640 10 librados: watch_flush exit 2024-08-19T12:45:05.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2ae64f6640 20 client.26099.objecter shutdown clearing up homeless session... 2024-08-19T12:45:05.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2ae64f6640 10 client.26099.objecter successfully canceled tick 2024-08-19T12:45:05.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2ae64f6640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.104:6800/899887506 conn(0x7f2acc041610 legacy=0x7f2acc043ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2ae64f6640 10 monclient: shutdown 2024-08-19T12:45:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2ae64f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2ae64f6640 1 -- 172.21.15.26:0/3904853363 >> v1:172.21.15.26:6793/0 conn(0x7f2ae00a5920 legacy=0x7f2ae0137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2ae4a93640 1 -- 172.21.15.26:0/3904853363 reap_dead start 2024-08-19T12:45:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2ae4a93640 1 -- 172.21.15.26:0/3904853363 reap_dead start 2024-08-19T12:45:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2ae64f6640 1 -- 172.21.15.26:0/3904853363 shutdown_connections 2024-08-19T12:45:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.801+0000 7f2ae64f6640 1 -- 172.21.15.26:0/3904853363 >> 172.21.15.26:0/3904853363 conn(0x7f2ae00a8740 msgr2=0x7f2ae00aaf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.802+0000 7f2ae64f6640 1 -- 172.21.15.26:0/3904853363 shutdown_connections 2024-08-19T12:45:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.802+0000 7f2ae64f6640 1 -- 172.21.15.26:0/3904853363 wait complete. 2024-08-19T12:45:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.802+0000 7f2ae64f6640 1 librados: shutdown 2024-08-19T12:45:05.848 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:05.848 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:45:05.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:45:05.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.946+0000 7fa4c92f6640 10 monclient: get_monmap_and_config 2024-08-19T12:45:05.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.946+0000 7fa4c92f6640 10 monclient: build_initial_monmap 2024-08-19T12:45:05.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.946+0000 7fa4c92f6640 10 monclient: monmap: 2024-08-19T12:45:05.948 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:05.948 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.948 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:05.948335+0000 2024-08-19T12:45:05.948 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:05.948335+0000 2024-08-19T12:45:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:05.949 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:05.949 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 1 Processor -- start 2024-08-19T12:45:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 1 -- start start 2024-08-19T12:45:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 10 monclient: init 2024-08-19T12:45:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 10 monclient: _add_conns ranks=[6,5,8,4,3,2,7,1,0] 2024-08-19T12:45:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 10 monclient(hunting): picked mon.noname-d con 0x7fa4c4139690 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 10 monclient(hunting): picked mon.noname-h con 0x7fa4c406eb10 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 10 monclient(hunting): picked mon.noname-e con 0x7fa4c413fc10 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c40a5c10 con 0x7fa4c406eb10 2024-08-19T12:45:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c40a6090 con 0x7fa4c4139690 2024-08-19T12:45:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c40a5e10 con 0x7fa4c413fc10 2024-08-19T12:45:05.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c92f6640 10 monclient(hunting): authenticate will time out at 1100.035609s 2024-08-19T12:45:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c2ffd640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fa4c4139690 0x7fa4c4139a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49834/0 (socket says 172.21.15.26:49834) 2024-08-19T12:45:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c2ffd640 1 -- 172.21.15.26:0/4158177456 learned_addr learned my addr 172.21.15.26:0/4158177456 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:05.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.947+0000 7fa4c37fe640 1 --1- 172.21.15.26:0/4158177456 >> v1:172.21.15.26:6793/0 conn(0x7fa4c413fc10 0x7fa4c4142000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34256/0 (socket says 172.21.15.26:34256) 2024-08-19T12:45:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.949+0000 7fa4c37fe640 1 -- 172.21.15.26:0/4158177456 >> v1:172.21.15.26:6793/0 conn(0x7fa4c413fc10 legacy=0x7fa4c4142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.949+0000 7fa4c2ffd640 1 -- 172.21.15.26:0/4158177456 >> v1:172.21.15.26:6792/0 conn(0x7fa4c4139690 legacy=0x7fa4c4139a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c27fc640 1 -- 172.21.15.26:0/4158177456 >> v1:172.21.15.104:6791/0 conn(0x7fa4c406eb10 legacy=0x7fa4c413e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1739679578 0 0) 0x7fa4c40a5e10 con 0x7fa4c413fc10 2024-08-19T12:45:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient(hunting): my global_id is 26126 2024-08-19T12:45:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4b0003520 con 0x7fa4c413fc10 2024-08-19T12:45:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1086642866 0 0) 0x7fa4c40a6090 con 0x7fa4c4139690 2024-08-19T12:45:05.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient(hunting): my global_id is 26592 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4c40a5e10 con 0x7fa4c4139690 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 36140382 0 0) 0x7fa4c40a5c10 con 0x7fa4c406eb10 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient(hunting): my global_id is 26231 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4c40a6090 con 0x7fa4c406eb10 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2685966556 0 0) 0x7fa4b0003520 con 0x7fa4c413fc10 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 >> v1:172.21.15.26:6792/0 conn(0x7fa4c4139690 legacy=0x7fa4c4139a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 >> v1:172.21.15.104:6791/0 conn(0x7fa4c406eb10 legacy=0x7fa4c413e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 monclient: found mon.noname-e 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c406f590 con 0x7fa4c413fc10 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient: _finish_auth 0 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa4bc001ae0 con 0x7fa4c413fc10 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c92f6640 5 monclient: authenticate success, global_id 26126 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c92f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient: dump: 2024-08-19T12:45:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.960 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.960 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.960 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.960 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.960 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c92f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:05.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4bc0047e0 con 0x7fa4c413fc10 2024-08-19T12:45:05.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:05.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 1 -- 172.21.15.26:0/4158177456 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa4bc004d50 con 0x7fa4c413fc10 2024-08-19T12:45:05.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c92f6640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:05.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c92f6640 4 set_mon_vals no callback set 2024-08-19T12:45:05.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.950+0000 7fa4c1ffb640 10 monclient: dump: 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.961 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.962 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.951+0000 7fa4c92f6640 10 monclient: shutdown 2024-08-19T12:45:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.951+0000 7fa4c92f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.951+0000 7fa4c92f6640 1 -- 172.21.15.26:0/4158177456 >> v1:172.21.15.26:6793/0 conn(0x7fa4c413fc10 legacy=0x7fa4c4142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.951+0000 7fa4c92f6640 1 -- 172.21.15.26:0/4158177456 shutdown_connections 2024-08-19T12:45:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.951+0000 7fa4c92f6640 1 -- 172.21.15.26:0/4158177456 >> 172.21.15.26:0/4158177456 conn(0x7fa4c40a65d0 msgr2=0x7fa4c40a69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.951+0000 7fa4c92f6640 1 -- 172.21.15.26:0/4158177456 shutdown_connections 2024-08-19T12:45:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.951+0000 7fa4c92f6640 1 -- 172.21.15.26:0/4158177456 wait complete. 2024-08-19T12:45:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.951+0000 7fa4c92f6640 10 monclient: build_initial_monmap 2024-08-19T12:45:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.951+0000 7fa4c92f6640 10 monclient: monmap: 2024-08-19T12:45:05.963 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:05.963 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:05.963 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:05.963 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:05.963 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:05.963 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.963 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:05.963 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:05.963 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:05.963 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:05.963 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:05.964 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:05.964 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:05.964 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:05.964 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:05.964 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.951+0000 7fa4c92f6640 1 librados: starting msgr at 2024-08-19T12:45:05.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.951+0000 7fa4c92f6640 1 librados: starting objecter 2024-08-19T12:45:05.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 1 Processor -- start 2024-08-19T12:45:05.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 1 -- start start 2024-08-19T12:45:05.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 1 librados: setting wanted keys 2024-08-19T12:45:05.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 1 librados: calling monclient init 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 10 monclient: init 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 10 monclient: _add_conns ranks=[0,7,6,8,2,1,4,3,5] 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 10 monclient(hunting): picked mon.noname-a con 0x7fa4c406eb10 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 10 monclient(hunting): picked mon.noname-i con 0x7fa4c4139690 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 10 monclient(hunting): picked mon.noname-d con 0x7fa4c413fc10 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c4139b10 con 0x7fa4c406eb10 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c4139df0 con 0x7fa4c413fc10 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c4143350 con 0x7fa4c4139690 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c92f6640 10 monclient(hunting): authenticate will time out at 1100.040634s 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c2ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa4c406eb10 0x7fa4c41358c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45258/0 (socket says 172.21.15.26:45258) 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c37fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fa4c413fc10 0x7fa4c41d67e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49838/0 (socket says 172.21.15.26:49838) 2024-08-19T12:45:05.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.952+0000 7fa4c2ffd640 1 -- 172.21.15.26:0/2304990967 learned_addr learned my addr 172.21.15.26:0/2304990967 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:05.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.954+0000 7fa4c37fe640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.26:6792/0 conn(0x7fa4c413fc10 legacy=0x7fa4c41d67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.954+0000 7fa4bb7fe640 10 client.?.objecter ms_handle_connect 0x7fa4c413fc10 2024-08-19T12:45:05.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.954+0000 7fa4bb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.954+0000 7fa4bb7fe640 10 client.?.objecter ms_handle_connect 0x7fa4c406eb10 2024-08-19T12:45:05.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.954+0000 7fa4bb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.954+0000 7fa4c2ffd640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.26:6789/0 conn(0x7fa4c406eb10 legacy=0x7fa4c41358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 209112815 0 0) 0x7fa4c4139df0 con 0x7fa4c413fc10 2024-08-19T12:45:05.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): my global_id is 26601 2024-08-19T12:45:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4a4004eb0 con 0x7fa4c413fc10 2024-08-19T12:45:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4c27fc640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6792/0 conn(0x7fa4c4139690 legacy=0x7fa4c4135fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 client.?.objecter ms_handle_connect 0x7fa4c4139690 2024-08-19T12:45:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1393273101 0 0) 0x7fa4c4139b10 con 0x7fa4c406eb10 2024-08-19T12:45:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): my global_id is 144486 2024-08-19T12:45:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4c4139df0 con 0x7fa4c406eb10 2024-08-19T12:45:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2362655441 0 0) 0x7fa4c4143350 con 0x7fa4c4139690 2024-08-19T12:45:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): my global_id is 135169 2024-08-19T12:45:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4c4139b10 con 0x7fa4c4139690 2024-08-19T12:45:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2563424388 0 0) 0x7fa4a4004eb0 con 0x7fa4c413fc10 2024-08-19T12:45:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4c4143350 con 0x7fa4c413fc10 2024-08-19T12:45:05.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa4bc001d30 con 0x7fa4c413fc10 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): dump: 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.975 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.976 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3210917701 0 0) 0x7fa4c4139df0 con 0x7fa4c406eb10 2024-08-19T12:45:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4a4004eb0 con 0x7fa4c406eb10 2024-08-19T12:45:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa4b4004b60 con 0x7fa4c406eb10 2024-08-19T12:45:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:05.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient(hunting): dump: 2024-08-19T12:45:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.978 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1596701482 0 0) 0x7fa4c4143350 con 0x7fa4c413fc10 2024-08-19T12:45:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6792/0 conn(0x7fa4c4139690 legacy=0x7fa4c4135fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.26:6789/0 conn(0x7fa4c406eb10 legacy=0x7fa4c41358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:45:05.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 monclient: found mon.d 2024-08-19T12:45:05.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:05.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:05.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c4143020 con 0x7fa4c413fc10 2024-08-19T12:45:05.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient: _finish_auth 0 2024-08-19T12:45:05.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:45:05.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:05.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4c92f6640 5 monclient: authenticate success, global_id 26601 2024-08-19T12:45:05.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4c92f6640 10 monclient: _renew_subs 2024-08-19T12:45:05.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4c92f6640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:05.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4c92f6640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c41366e0 con 0x7fa4c413fc10 2024-08-19T12:45:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4c92f6640 10 client.26601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4c92f6640 10 monclient: _renew_subs 2024-08-19T12:45:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4c92f6640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4c92f6640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa4c41380f0 con 0x7fa4c413fc10 2024-08-19T12:45:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4c92f6640 1 librados: init done 2024-08-19T12:45:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4bc004f80 con 0x7fa4c413fc10 2024-08-19T12:45:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa4bc0047e0 con 0x7fa4c413fc10 2024-08-19T12:45:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4c17fa640 4 set_mon_vals no callback set 2024-08-19T12:45:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.955+0000 7fa4bb7fe640 10 monclient: dump: 2024-08-19T12:45:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.982 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.956+0000 7fa4c92f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.956+0000 7fa4c92f6640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa4a80065c0 con 0x7fa4a8002860 2024-08-19T12:45:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.956+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa4bc011730 con 0x7fa4c413fc10 2024-08-19T12:45:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.956+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(278..278 src has 1..278) ==== 5338+0+0 (unknown 991588785 0 0) 0x7fa4bc04df60 con 0x7fa4c413fc10 2024-08-19T12:45:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.956+0000 7fa4bb7fe640 10 client.26601.objecter ms_dispatch 0x7fa4c4000b90 osd_map(278..278 src has 1..278) 2024-08-19T12:45:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.956+0000 7fa4bb7fe640 3 client.26601.objecter handle_osd_map got epochs [278,278] > 0 2024-08-19T12:45:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.956+0000 7fa4bb7fe640 3 client.26601.objecter handle_osd_map decoding full epoch 278 2024-08-19T12:45:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.956+0000 7fa4bb7fe640 20 client.26601.objecter dump_active .. 0 homeless 2024-08-19T12:45:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.956+0000 7fa4c27fc640 10 monclient: get_auth_request con 0x7fa4a40417e0 auth_method 0 2024-08-19T12:45:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.958+0000 7fa4bb7fe640 10 client.26601.objecter ms_handle_connect 0x7fa4a8002860 2024-08-19T12:45:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.958+0000 7fa4c2ffd640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a8002860 legacy=0x7fa4a8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.958+0000 7fa4bb7fe640 10 client.26601.objecter resend_mon_ops 2024-08-19T12:45:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.958+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 348947843 0 0) 0x7fa4a80065c0 con 0x7fa4a8002860 2024-08-19T12:45:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.958+0000 7fa4bb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.958+0000 7fa4bb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.958+0000 7fa4bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.958+0000 7fa4bb7fe640 10 monclient(hunting): my global_id is 26983 2024-08-19T12:45:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.958+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4c4143350 con 0x7fa4a8002860 2024-08-19T12:45:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 108606682 0 0) 0x7fa4c4143350 con 0x7fa4a8002860 2024-08-19T12:45:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4a80065c0 con 0x7fa4a8002860 2024-08-19T12:45:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa4b40052b0 con 0x7fa4a8002860 2024-08-19T12:45:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:45:05.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 10 monclient: dump: 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.985 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 10 client.26601.objecter ms_handle_connect 0x7fa4a40417e0 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 753709688 0 0) 0x7fa4a80065c0 con 0x7fa4a8002860 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4a800a160 con 0x7fa4a8002860 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa4b40075d0 con 0x7fa4a8002860 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a8002860 legacy=0x7fa4a8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.959+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a8002860 legacy=0x7fa4a8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.965+0000 7fa4c92f6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:05.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.965+0000 7fa4c92f6640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa4a800c4f0 con 0x7fa4a8009110 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.968+0000 7fa4c2ffd640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a8009110 legacy=0x7fa4a8009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.968+0000 7fa4bb7fe640 10 client.26601.objecter ms_handle_connect 0x7fa4a8009110 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.968+0000 7fa4bb7fe640 10 client.26601.objecter resend_mon_ops 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.968+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2173569049 0 0) 0x7fa4a800c4f0 con 0x7fa4a8009110 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.968+0000 7fa4bb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.968+0000 7fa4bb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.968+0000 7fa4bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.968+0000 7fa4bb7fe640 10 monclient(hunting): my global_id is 26992 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.968+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4a80065c0 con 0x7fa4a8009110 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2361306004 0 0) 0x7fa4a80065c0 con 0x7fa4a8009110 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4a800c4f0 con 0x7fa4a8009110 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa4b4005c90 con 0x7fa4a8009110 2024-08-19T12:45:05.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4bb7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4bb7fe640 10 monclient: dump: 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.988 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4c2ffd640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a8009110 legacy=0x7fa4a8009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4c2ffd640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a8009110 legacy=0x7fa4a8009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4c2ffd640 1 --1- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a8009110 0x7fa4a8009580 :-1 s=OPENED pgs=3373 cs=1 l=1).handle_message read tag failed 2024-08-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4c2ffd640 1 --1- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a8009110 0x7fa4a8009580 :-1 s=OPENED pgs=3373 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a8009110 legacy=0x7fa4a8009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a8009110 legacy=0x7fa4a8009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.969+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa4a80065c0 con 0x7fa4a404c8d0 2024-08-19T12:45:05.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 10 client.26601.objecter ms_handle_connect 0x7fa4a404c8d0 2024-08-19T12:45:05.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4c2ffd640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a404c8d0 legacy=0x7fa4a404af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:05.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 10 client.26601.objecter resend_mon_ops 2024-08-19T12:45:05.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3293635144 0 0) 0x7fa4a80065c0 con 0x7fa4a404c8d0 2024-08-19T12:45:05.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:05.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:05.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:05.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 10 monclient(hunting): my global_id is 27001 2024-08-19T12:45:05.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4c4143350 con 0x7fa4a404c8d0 2024-08-19T12:45:05.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2389559431 0 0) 0x7fa4c4143350 con 0x7fa4a404c8d0 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4a80065c0 con 0x7fa4a404c8d0 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa4b4005c90 con 0x7fa4a404c8d0 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.971+0000 7fa4bb7fe640 10 monclient: dump: 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:05.992 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 95815657 0 0) 0x7fa4a80065c0 con 0x7fa4a404c8d0 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa4b40075d0 con 0x7fa4a404c8d0 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 620468508) 0x7fa4b4007780 con 0x7fa4a404c8d0 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4bb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4bb7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a404c8d0 legacy=0x7fa4a404af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4bb7fe640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6789/0 conn(0x7fa4a404c8d0 legacy=0x7fa4a404af70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:05.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c37fe640 1 -- 172.21.15.26:0/2304990967 reap_dead start 2024-08-19T12:45:05.994 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:05.994 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:05.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:05.994 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:05.994 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:05.994 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 418977, 2024-08-19T12:45:05.994 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:05.994 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:05.994 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:05.995 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:05.996 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:05.997 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:05.998 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:05.999 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:06.000 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.001 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:06.001 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:06.001 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.001 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.001 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:06.002 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:06.002 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.002 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:06.002 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.002 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:06.002 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.004 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:06.004 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:06.004 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.004 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.004 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:06.004 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:06.005 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:06.005 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:06.005 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:06.006 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:06.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:06.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:06.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:06.008 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.009 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.009 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:06.009 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:06.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:06.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:06.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:06.010 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.010 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.010 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:06.010 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:06.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:06.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:06.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:06.011 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.012 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.012 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:06.012 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:06.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:06.013 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:06.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:06.013 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.013 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.013 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.014 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:06.014 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:06.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.014 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:06.014 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:06.014 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:06.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.014 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.015 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:06.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.015 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:06.015 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:06.015 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:45:06.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.015 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:06.015 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 10 librados: watch_flush enter 2024-08-19T12:45:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 10 librados: watch_flush exit 2024-08-19T12:45:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 20 client.26601.objecter shutdown clearing up homeless session... 2024-08-19T12:45:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 10 client.26601.objecter successfully canceled tick 2024-08-19T12:45:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.104:6800/899887506 conn(0x7fa4a40417e0 legacy=0x7fa4a4043ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 10 monclient: shutdown 2024-08-19T12:45:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 1 -- 172.21.15.26:0/2304990967 >> v1:172.21.15.26:6792/0 conn(0x7fa4c413fc10 legacy=0x7fa4c41d67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 1 -- 172.21.15.26:0/2304990967 shutdown_connections 2024-08-19T12:45:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 1 -- 172.21.15.26:0/2304990967 >> 172.21.15.26:0/2304990967 conn(0x7fa4c40a65d0 msgr2=0x7fa4c4072cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 1 -- 172.21.15.26:0/2304990967 shutdown_connections 2024-08-19T12:45:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 1 -- 172.21.15.26:0/2304990967 wait complete. 2024-08-19T12:45:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:05.972+0000 7fa4c92f6640 1 librados: shutdown 2024-08-19T12:45:06.019 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:06.019 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:45:06.019 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:45:06.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.118+0000 7f3820743640 10 monclient: get_monmap_and_config 2024-08-19T12:45:06.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.118+0000 7f3820743640 10 monclient: build_initial_monmap 2024-08-19T12:45:06.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.118+0000 7f3820743640 10 monclient: monmap: 2024-08-19T12:45:06.120 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:06.120 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.120 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:06.120200+0000 2024-08-19T12:45:06.120 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:06.120200+0000 2024-08-19T12:45:06.120 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:06.120 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.120 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:06.121 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:06.121 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:06.121 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:06.121 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:06.121 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:06.121 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:06.121 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:06.121 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:06.121 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.118+0000 7f3820743640 1 Processor -- start 2024-08-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f3820743640 1 -- start start 2024-08-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f3820743640 10 monclient: init 2024-08-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f3820743640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f3820743640 10 monclient: _add_conns ranks=[1,3,5,4,6,7,8,2,0] 2024-08-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f3820743640 10 monclient(hunting): picked mon.noname-f con 0x7f3818073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f3820743640 10 monclient(hunting): picked mon.noname-g con 0x7f38180a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f3820743640 10 monclient(hunting): picked mon.noname-h con 0x7f381813f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f3820743640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f381809dae0 con 0x7f3818073470 2024-08-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f3820743640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3818136910 con 0x7f38180a57a0 2024-08-19T12:45:06.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f3820743640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f381809dd50 con 0x7f381813f2e0 2024-08-19T12:45:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f3820743640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f3820743640 10 monclient(hunting): authenticate will time out at 1100.207740s 2024-08-19T12:45:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f381e4df640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3818073470 0x7f3818136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:52160/0 (socket says 172.21.15.26:52160) 2024-08-19T12:45:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f381dcde640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f38180a57a0 0x7f38180a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:44310/0 (socket says 172.21.15.26:44310) 2024-08-19T12:45:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.119+0000 7f381e4df640 1 -- 172.21.15.26:0/3435793636 learned_addr learned my addr 172.21.15.26:0/3435793636 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.122+0000 7f381dcde640 1 -- 172.21.15.26:0/3435793636 >> v1:172.21.15.104:6790/0 conn(0x7f38180a57a0 legacy=0x7f38180a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.122+0000 7f381e4df640 1 -- 172.21.15.26:0/3435793636 >> v1:172.21.15.104:6789/0 conn(0x7f3818073470 legacy=0x7f3818136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.122+0000 7f381ece0640 1 -- 172.21.15.26:0/3435793636 >> v1:172.21.15.104:6791/0 conn(0x7f381813f2e0 legacy=0x7f38181416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.189+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1788700442 0 0) 0x7f381809dd50 con 0x7f381813f2e0 2024-08-19T12:45:06.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.189+0000 7f381d4dd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.189+0000 7f381d4dd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.189+0000 7f381d4dd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.189+0000 7f381d4dd640 10 monclient(hunting): my global_id is 26240 2024-08-19T12:45:06.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.189+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3804003520 con 0x7f381813f2e0 2024-08-19T12:45:06.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 879049335 0 0) 0x7f381809dae0 con 0x7f3818073470 2024-08-19T12:45:06.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient(hunting): my global_id is 27010 2024-08-19T12:45:06.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f381809dd50 con 0x7f3818073470 2024-08-19T12:45:06.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3058448513 0 0) 0x7f3818136910 con 0x7f38180a57a0 2024-08-19T12:45:06.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient(hunting): my global_id is 154128 2024-08-19T12:45:06.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f381809dae0 con 0x7f38180a57a0 2024-08-19T12:45:06.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2035427986 0 0) 0x7f3804003520 con 0x7f381813f2e0 2024-08-19T12:45:06.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 >> v1:172.21.15.104:6790/0 conn(0x7f38180a57a0 legacy=0x7f38180a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 >> v1:172.21.15.104:6789/0 conn(0x7f3818073470 legacy=0x7f3818136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 1 monclient: found mon.noname-h 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381809e030 con 0x7f381813f2e0 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient: _finish_auth 0 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3810001eb0 con 0x7f381813f2e0 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f3820743640 5 monclient: authenticate success, global_id 26240 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f3820743640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient: dump: 2024-08-19T12:45:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.196 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f3820743640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3810003990 con 0x7f381813f2e0 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 1 -- 172.21.15.26:0/3435793636 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3810003f00 con 0x7f381813f2e0 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f3820743640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f3820743640 4 set_mon_vals no callback set 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.190+0000 7f381d4dd640 10 monclient: dump: 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.197 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.191+0000 7f3820743640 10 monclient: shutdown 2024-08-19T12:45:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.191+0000 7f3820743640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.191+0000 7f3820743640 1 -- 172.21.15.26:0/3435793636 >> v1:172.21.15.104:6791/0 conn(0x7f381813f2e0 legacy=0x7f38181416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.191+0000 7f3820743640 1 -- 172.21.15.26:0/3435793636 shutdown_connections 2024-08-19T12:45:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.191+0000 7f3820743640 1 -- 172.21.15.26:0/3435793636 >> 172.21.15.26:0/3435793636 conn(0x7f38180abff0 msgr2=0x7f38180ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.191+0000 7f3820743640 1 -- 172.21.15.26:0/3435793636 shutdown_connections 2024-08-19T12:45:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.191+0000 7f3820743640 1 -- 172.21.15.26:0/3435793636 wait complete. 2024-08-19T12:45:06.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.191+0000 7f3820743640 10 monclient: build_initial_monmap 2024-08-19T12:45:06.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.191+0000 7f3820743640 10 monclient: monmap: 2024-08-19T12:45:06.203 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:06.203 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:06.203 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 1 librados: starting msgr at 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 1 librados: starting objecter 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 1 Processor -- start 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 1 -- start start 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 1 librados: setting wanted keys 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 1 librados: calling monclient init 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 10 monclient: init 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 10 monclient: _add_conns ranks=[2,7,4,0,1,3,6,5,8] 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 10 monclient(hunting): picked mon.noname-b con 0x7f3818073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 10 monclient(hunting): picked mon.noname-i con 0x7f38180a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 10 monclient(hunting): picked mon.noname-c con 0x7f381813f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38181370e0 con 0x7f3818073470 2024-08-19T12:45:06.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3818142b60 con 0x7f381813f2e0 2024-08-19T12:45:06.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38180a4830 con 0x7f38180a57a0 2024-08-19T12:45:06.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:06.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f3820743640 10 monclient(hunting): authenticate will time out at 1100.280838s 2024-08-19T12:45:06.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f381e4df640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f3818073470 0x7f38180a2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52414/0 (socket says 172.21.15.26:52414) 2024-08-19T12:45:06.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.192+0000 7f381e4df640 1 -- 172.21.15.26:0/2234803464 learned_addr learned my addr 172.21.15.26:0/2234803464 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:06.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f381e4df640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.26:6790/0 conn(0x7f3818073470 legacy=0x7f38180a2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f381ece0640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.26:6791/0 conn(0x7f381813f2e0 legacy=0x7f38181d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 client.?.objecter ms_handle_connect 0x7f3818073470 2024-08-19T12:45:06.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:06.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 client.?.objecter ms_handle_connect 0x7f381813f2e0 2024-08-19T12:45:06.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:06.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432345610 0 0) 0x7f3818142b60 con 0x7f381813f2e0 2024-08-19T12:45:06.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient(hunting): my global_id is 26581 2024-08-19T12:45:06.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f381dcde640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.104:6792/0 conn(0x7f38180a57a0 legacy=0x7f38180a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37fc004eb0 con 0x7f381813f2e0 2024-08-19T12:45:06.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 client.?.objecter ms_handle_connect 0x7f38180a57a0 2024-08-19T12:45:06.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:06.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3116745879 0 0) 0x7f37fc004eb0 con 0x7f381813f2e0 2024-08-19T12:45:06.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3818142b60 con 0x7f381813f2e0 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3810003e80 con 0x7f381813f2e0 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient(hunting): dump: 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.209 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.209 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.209 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.209 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.209 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.209 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.209 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:06.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4090867495 0 0) 0x7f3818142b60 con 0x7f381813f2e0 2024-08-19T12:45:06.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.104:6792/0 conn(0x7f38180a57a0 legacy=0x7f38180a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.26:6790/0 conn(0x7f3818073470 legacy=0x7f38180a2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:06.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 1 monclient: found mon.c 2024-08-19T12:45:06.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:06.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:06.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38180a5350 con 0x7f381813f2e0 2024-08-19T12:45:06.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient: _finish_auth 0 2024-08-19T12:45:06.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:06.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:06.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f3820743640 5 monclient: authenticate success, global_id 26581 2024-08-19T12:45:06.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f3820743640 10 monclient: _renew_subs 2024-08-19T12:45:06.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f3820743640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:06.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f3820743640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3818136b80 con 0x7f381813f2e0 2024-08-19T12:45:06.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f3820743640 10 client.26581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:06.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f3820743640 10 monclient: _renew_subs 2024-08-19T12:45:06.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f3820743640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:06.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f3820743640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f38180a3910 con 0x7f381813f2e0 2024-08-19T12:45:06.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f3820743640 1 librados: init done 2024-08-19T12:45:06.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3810003990 con 0x7f381813f2e0 2024-08-19T12:45:06.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:06.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3810001950 con 0x7f381813f2e0 2024-08-19T12:45:06.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:06.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f381ccdc640 4 set_mon_vals no callback set 2024-08-19T12:45:06.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.195+0000 7f380effd640 10 monclient: dump: 2024-08-19T12:45:06.212 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.212 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.212 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.212 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.212 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.212 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.212 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.212 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.212 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.212 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.212 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.213 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.213 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.213 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.213 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.213 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.196+0000 7f3820743640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:06.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.196+0000 7f3820743640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f38000065c0 con 0x7f3800002860 2024-08-19T12:45:06.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.196+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f38100046d0 con 0x7f381813f2e0 2024-08-19T12:45:06.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.197+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(278..278 src has 1..278) ==== 5338+0+0 (unknown 991588785 0 0) 0x7f381004e340 con 0x7f381813f2e0 2024-08-19T12:45:06.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.197+0000 7f380effd640 10 client.26581.objecter ms_dispatch 0x7f3818000b90 osd_map(278..278 src has 1..278) 2024-08-19T12:45:06.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.197+0000 7f380effd640 3 client.26581.objecter handle_osd_map got epochs [278,278] > 0 2024-08-19T12:45:06.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.197+0000 7f380effd640 3 client.26581.objecter handle_osd_map decoding full epoch 278 2024-08-19T12:45:06.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.197+0000 7f380effd640 20 client.26581.objecter dump_active .. 0 homeless 2024-08-19T12:45:06.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.197+0000 7f381dcde640 10 monclient: get_auth_request con 0x7f37fc040f80 auth_method 0 2024-08-19T12:45:06.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.198+0000 7f381e4df640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.104:6790/0 conn(0x7f3800002860 legacy=0x7f3800004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.198+0000 7f380effd640 10 client.26581.objecter ms_handle_connect 0x7f3800002860 2024-08-19T12:45:06.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.198+0000 7f380effd640 10 client.26581.objecter resend_mon_ops 2024-08-19T12:45:06.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.199+0000 7f380effd640 10 client.26581.objecter ms_handle_connect 0x7f37fc040f80 2024-08-19T12:45:06.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.335+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923464673 0 0) 0x7f38000065c0 con 0x7f3800002860 2024-08-19T12:45:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.335+0000 7f380effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.335+0000 7f380effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.335+0000 7f380effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.335+0000 7f380effd640 10 monclient(hunting): my global_id is 154137 2024-08-19T12:45:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.335+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3818142b60 con 0x7f3800002860 2024-08-19T12:45:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.336+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 284641142 0 0) 0x7f3818142b60 con 0x7f3800002860 2024-08-19T12:45:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.336+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38000065c0 con 0x7f3800002860 2024-08-19T12:45:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.336+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3814001b80 con 0x7f3800002860 2024-08-19T12:45:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.336+0000 7f380effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.336+0000 7f380effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.336+0000 7f380effd640 10 monclient: dump: 2024-08-19T12:45:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.336+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1338250687 0 0) 0x7f38000065c0 con 0x7f3800002860 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.336+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f380000a160 con 0x7f3800002860 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.337+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f38140047e0 con 0x7f3800002860 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.337+0000 7f380effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.337+0000 7f380effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.337+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.104:6790/0 conn(0x7f3800002860 legacy=0x7f3800004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.337+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.104:6790/0 conn(0x7f3800002860 legacy=0x7f3800004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.343+0000 7f3820743640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:06.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.343+0000 7f3820743640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3800009e70 con 0x7f3800009130 2024-08-19T12:45:06.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.345+0000 7f381e4df640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.104:6790/0 conn(0x7f3800009130 legacy=0x7f38000095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.345+0000 7f380effd640 10 client.26581.objecter ms_handle_connect 0x7f3800009130 2024-08-19T12:45:06.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.345+0000 7f380effd640 10 client.26581.objecter resend_mon_ops 2024-08-19T12:45:06.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2460570954 0 0) 0x7f3800009e70 con 0x7f3800009130 2024-08-19T12:45:06.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 10 monclient(hunting): my global_id is 154146 2024-08-19T12:45:06.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38000065c0 con 0x7f3800009130 2024-08-19T12:45:06.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4294558257 0 0) 0x7f38000065c0 con 0x7f3800009130 2024-08-19T12:45:06.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3800009e70 con 0x7f3800009130 2024-08-19T12:45:06.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3814004f80 con 0x7f3800009130 2024-08-19T12:45:06.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:06.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 10 monclient: dump: 2024-08-19T12:45:06.350 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.350 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.350 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.350 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.350 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.350 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.350 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.350 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1555782137 0 0) 0x7f3800009e70 con 0x7f3800009130 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f380000c410 con 0x7f3800009130 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 2685767629) 0x7f3814007320 con 0x7f3800009130 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:06.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:06.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.104:6790/0 conn(0x7f3800009130 legacy=0x7f38000095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.346+0000 7f380effd640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.104:6790/0 conn(0x7f3800009130 legacy=0x7f38000095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:06.352 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 18076, 2024-08-19T12:45:06.352 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:45:06.353 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:06.354 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:06.355 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:06.356 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:06.357 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.358 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:06.359 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:06.360 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:06.361 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.362 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.362 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.362 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:06.362 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.362 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.362 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:06.362 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:06.362 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.362 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:06.364 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:06.364 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.364 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.364 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.364 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:06.364 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:06.364 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.365 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:06.365 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.365 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.365 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.365 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:06.365 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:06.365 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.365 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.365 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.367 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:06.367 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:06.367 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.367 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.367 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.367 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:06.368 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:06.368 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.368 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.368 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.368 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:06.368 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.369 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:06.369 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:06.369 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.369 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.369 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.369 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:06.369 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:06.370 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:06.370 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.370 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:06.371 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:06.371 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.371 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.371 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.371 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:06.371 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:06.371 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.372 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:06.372 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.372 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.372 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:06.372 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:06.372 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.372 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.372 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.373 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.373 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:06.373 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:06.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.373 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:06.373 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:06.373 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:06.373 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.373 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.373 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:06.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.374 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:06.374 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:06.374 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:06.374 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.374 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.374 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:06.374 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:06.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 10 librados: watch_flush enter 2024-08-19T12:45:06.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 10 librados: watch_flush exit 2024-08-19T12:45:06.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 20 client.26581.objecter shutdown clearing up homeless session... 2024-08-19T12:45:06.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 10 client.26581.objecter successfully canceled tick 2024-08-19T12:45:06.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.104:6800/899887506 conn(0x7f37fc040f80 legacy=0x7f37fc043440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 10 monclient: shutdown 2024-08-19T12:45:06.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:06.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 1 -- 172.21.15.26:0/2234803464 >> v1:172.21.15.26:6791/0 conn(0x7f381813f2e0 legacy=0x7f38181d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f381ece0640 1 -- 172.21.15.26:0/2234803464 reap_dead start 2024-08-19T12:45:06.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f381ece0640 1 -- 172.21.15.26:0/2234803464 reap_dead start 2024-08-19T12:45:06.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 1 -- 172.21.15.26:0/2234803464 shutdown_connections 2024-08-19T12:45:06.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 1 -- 172.21.15.26:0/2234803464 >> 172.21.15.26:0/2234803464 conn(0x7f38180abff0 msgr2=0x7f38181338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:06.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 1 -- 172.21.15.26:0/2234803464 shutdown_connections 2024-08-19T12:45:06.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 1 -- 172.21.15.26:0/2234803464 wait complete. 2024-08-19T12:45:06.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.347+0000 7f3820743640 1 librados: shutdown 2024-08-19T12:45:06.378 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:06.378 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:45:06.379 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:45:06.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.476+0000 7f0137f3a640 10 monclient: get_monmap_and_config 2024-08-19T12:45:06.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.476+0000 7f0137f3a640 10 monclient: build_initial_monmap 2024-08-19T12:45:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.476+0000 7f0137f3a640 10 monclient: monmap: 2024-08-19T12:45:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:06.478231+0000 2024-08-19T12:45:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:06.478231+0000 2024-08-19T12:45:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:06.480 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:06.480 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:06.480 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:06.480 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:06.480 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:06.480 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:06.480 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:06.480 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.476+0000 7f0137f3a640 1 Processor -- start 2024-08-19T12:45:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f0137f3a640 1 -- start start 2024-08-19T12:45:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f0137f3a640 10 monclient: init 2024-08-19T12:45:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f0137f3a640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f0137f3a640 10 monclient: _add_conns ranks=[7,2,5,1,0,6,4,8,3] 2024-08-19T12:45:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f0137f3a640 10 monclient(hunting): picked mon.noname-i con 0x7f013006e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f0137f3a640 10 monclient(hunting): picked mon.noname-b con 0x7f01300a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f0137f3a640 10 monclient(hunting): picked mon.noname-h con 0x7f01300a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:06.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f0137f3a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0130136790 con 0x7f01300a5920 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f0137f3a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f013009ddf0 con 0x7f01300a5d50 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f0137f3a640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f013009e0d0 con 0x7f013006e7b0 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f0137f3a640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f0137f3a640 10 monclient(hunting): authenticate will time out at 1100.565753s 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f01354d5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f01300a5920 0x7f013013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:52420/0 (socket says 172.21.15.26:52420) 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.477+0000 7f01354d5640 1 -- 172.21.15.26:0/202871423 learned_addr learned my addr 172.21.15.26:0/202871423 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f01354d5640 1 -- 172.21.15.26:0/202871423 >> v1:172.21.15.26:6790/0 conn(0x7f01300a5920 legacy=0x7f013013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 1 -- 172.21.15.26:0/202871423 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4077435138 0 0) 0x7f0130136790 con 0x7f01300a5920 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient(hunting): my global_id is 26264 2024-08-19T12:45:06.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 1 -- 172.21.15.26:0/202871423 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f011c003520 con 0x7f01300a5920 2024-08-19T12:45:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f01364d7640 1 -- 172.21.15.26:0/202871423 >> v1:172.21.15.104:6791/0 conn(0x7f01300a5d50 legacy=0x7f0130141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0135cd6640 1 -- 172.21.15.26:0/202871423 >> v1:172.21.15.104:6792/0 conn(0x7f013006e7b0 legacy=0x7f0130136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 1 -- 172.21.15.26:0/202871423 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2064486619 0 0) 0x7f011c003520 con 0x7f01300a5920 2024-08-19T12:45:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 1 -- 172.21.15.26:0/202871423 >> v1:172.21.15.104:6792/0 conn(0x7f013006e7b0 legacy=0x7f0130136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 1 -- 172.21.15.26:0/202871423 >> v1:172.21.15.104:6791/0 conn(0x7f01300a5d50 legacy=0x7f0130141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient: _finish_hunting 0 2024-08-19T12:45:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 1 monclient: found mon.noname-b 2024-08-19T12:45:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:45:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 1 -- 172.21.15.26:0/202871423 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013009e3b0 con 0x7f01300a5920 2024-08-19T12:45:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient: _finish_auth 0 2024-08-19T12:45:06.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient: _check_auth_tickets 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 1 -- 172.21.15.26:0/202871423 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0120001ae0 con 0x7f01300a5920 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0137f3a640 5 monclient: authenticate success, global_id 26264 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0137f3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient: dump: 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.488 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0137f3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 1 -- 172.21.15.26:0/202871423 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01200048a0 con 0x7f01300a5920 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:06.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 1 -- 172.21.15.26:0/202871423 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0120004e10 con 0x7f01300a5920 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0137f3a640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0137f3a640 4 set_mon_vals no callback set 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.480+0000 7f0134cd4640 10 monclient: dump: 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.490 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.491 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.491 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.491 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.491 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.491 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.491 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.491 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.481+0000 7f0137f3a640 10 monclient: shutdown 2024-08-19T12:45:06.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.481+0000 7f0137f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:06.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.481+0000 7f0137f3a640 1 -- 172.21.15.26:0/202871423 >> v1:172.21.15.26:6790/0 conn(0x7f01300a5920 legacy=0x7f013013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.481+0000 7f0137f3a640 1 -- 172.21.15.26:0/202871423 shutdown_connections 2024-08-19T12:45:06.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.481+0000 7f0137f3a640 1 -- 172.21.15.26:0/202871423 >> 172.21.15.26:0/202871423 conn(0x7f01300a8740 msgr2=0x7f01300a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.481+0000 7f0137f3a640 1 -- 172.21.15.26:0/202871423 shutdown_connections 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.481+0000 7f0137f3a640 1 -- 172.21.15.26:0/202871423 wait complete. 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.481+0000 7f0137f3a640 10 monclient: build_initial_monmap 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.481+0000 7f0137f3a640 10 monclient: monmap: 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:06.492 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:06.493 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:06.493 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:06.493 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:06.493 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:06.493 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:06.493 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.481+0000 7f0137f3a640 1 librados: starting msgr at 2024-08-19T12:45:06.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.481+0000 7f0137f3a640 1 librados: starting objecter 2024-08-19T12:45:06.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 1 Processor -- start 2024-08-19T12:45:06.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 1 -- start start 2024-08-19T12:45:06.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 1 librados: setting wanted keys 2024-08-19T12:45:06.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 1 librados: calling monclient init 2024-08-19T12:45:06.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 10 monclient: init 2024-08-19T12:45:06.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:06.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 10 monclient: _add_conns ranks=[0,4,8,3,1,7,6,5,2] 2024-08-19T12:45:06.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 10 monclient(hunting): picked mon.noname-a con 0x7f013006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:06.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 10 monclient(hunting): picked mon.noname-c con 0x7f01300a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:06.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 10 monclient(hunting): picked mon.noname-e con 0x7f01300a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:06.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f013013b3c0 con 0x7f013006e7b0 2024-08-19T12:45:06.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0130139060 con 0x7f01300a5920 2024-08-19T12:45:06.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f013013ab20 con 0x7f01300a5d50 2024-08-19T12:45:06.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:06.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0137f3a640 10 monclient(hunting): authenticate will time out at 1100.570680s 2024-08-19T12:45:06.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0135cd6640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f013006e7b0 0x7f01301ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45268/0 (socket says 172.21.15.26:45268) 2024-08-19T12:45:06.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.482+0000 7f0135cd6640 1 -- 172.21.15.26:0/776541612 learned_addr learned my addr 172.21.15.26:0/776541612 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.484+0000 7f01354d5640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.26:6791/0 conn(0x7f01300a5920 legacy=0x7f01301d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.484+0000 7f01267fc640 10 client.?.objecter ms_handle_connect 0x7f01300a5920 2024-08-19T12:45:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.484+0000 7f01267fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.484+0000 7f0135cd6640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.26:6789/0 conn(0x7f013006e7b0 legacy=0x7f01301ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.484+0000 7f01267fc640 10 client.?.objecter ms_handle_connect 0x7f013006e7b0 2024-08-19T12:45:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.484+0000 7f01267fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.484+0000 7f01267fc640 10 client.?.objecter ms_handle_connect 0x7f01300a5d50 2024-08-19T12:45:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.484+0000 7f01267fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.484+0000 7f01364d7640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.26:6793/0 conn(0x7f01300a5d50 legacy=0x7f01301d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2019955436 0 0) 0x7f0130139060 con 0x7f01300a5920 2024-08-19T12:45:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): my global_id is 26590 2024-08-19T12:45:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0114004eb0 con 0x7f01300a5920 2024-08-19T12:45:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2317613811 0 0) 0x7f013013b3c0 con 0x7f013006e7b0 2024-08-19T12:45:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): my global_id is 144495 2024-08-19T12:45:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0130139060 con 0x7f013006e7b0 2024-08-19T12:45:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3058939359 0 0) 0x7f013013ab20 con 0x7f01300a5d50 2024-08-19T12:45:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): my global_id is 26135 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f013013b3c0 con 0x7f01300a5d50 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 368347530 0 0) 0x7f013013b3c0 con 0x7f01300a5d50 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f013013ab20 con 0x7f01300a5d50 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0128001fb0 con 0x7f01300a5d50 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): dump: 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.504 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.505 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.506 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1457499808 0 0) 0x7f0114004eb0 con 0x7f01300a5920 2024-08-19T12:45:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f013013b3c0 con 0x7f01300a5920 2024-08-19T12:45:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1755690056 0 0) 0x7f0130139060 con 0x7f013006e7b0 2024-08-19T12:45:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0114004eb0 con 0x7f013006e7b0 2024-08-19T12:45:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0120001a60 con 0x7f01300a5920 2024-08-19T12:45:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:06.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): dump: 2024-08-19T12:45:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.507 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f012c003610 con 0x7f013006e7b0 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient(hunting): dump: 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.508 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2787098095 0 0) 0x7f013013ab20 con 0x7f01300a5d50 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.26:6791/0 conn(0x7f01300a5920 legacy=0x7f01301d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.26:6789/0 conn(0x7f013006e7b0 legacy=0x7f01301ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient: _finish_hunting 0 2024-08-19T12:45:06.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 monclient: found mon.e 2024-08-19T12:45:06.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:06.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:06.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013013ae00 con 0x7f01300a5d50 2024-08-19T12:45:06.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient: _finish_auth 0 2024-08-19T12:45:06.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient: _check_auth_tickets 2024-08-19T12:45:06.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:06.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f0137f3a640 5 monclient: authenticate success, global_id 26135 2024-08-19T12:45:06.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f0137f3a640 10 monclient: _renew_subs 2024-08-19T12:45:06.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f0137f3a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:06.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f0137f3a640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f013013a7f0 con 0x7f01300a5d50 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f0137f3a640 10 client.26135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f0137f3a640 10 monclient: _renew_subs 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f0137f3a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f0137f3a640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f0130138650 con 0x7f01300a5d50 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f0137f3a640 1 librados: init done 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0128003c40 con 0x7f01300a5d50 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0128004130 con 0x7f01300a5d50 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f0127fff640 4 set_mon_vals no callback set 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.485+0000 7f01267fc640 10 monclient: dump: 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.511 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.512 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.486+0000 7f0137f3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.486+0000 7f0137f3a640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f01180065c0 con 0x7f0118002860 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.486+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f01280042d0 con 0x7f01300a5d50 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.487+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(279..279 src has 1..279) ==== 5338+0+0 (unknown 3765987609 0 0) 0x7f012804e140 con 0x7f01300a5d50 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.487+0000 7f01267fc640 10 client.26135.objecter ms_dispatch 0x7f0130000b90 osd_map(279..279 src has 1..279) 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.487+0000 7f01267fc640 3 client.26135.objecter handle_osd_map got epochs [279,279] > 0 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.487+0000 7f01267fc640 3 client.26135.objecter handle_osd_map decoding full epoch 279 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.487+0000 7f01267fc640 20 client.26135.objecter dump_active .. 0 homeless 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.487+0000 7f01354d5640 10 monclient: get_auth_request con 0x7f0114041980 auth_method 0 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f0135cd6640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.104:6791/0 conn(0x7f0118002860 legacy=0x7f0118004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 10 client.26135.objecter ms_handle_connect 0x7f0118002860 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 10 client.26135.objecter resend_mon_ops 2024-08-19T12:45:06.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3408462810 0 0) 0x7f01180065c0 con 0x7f0118002860 2024-08-19T12:45:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 10 monclient(hunting): my global_id is 26258 2024-08-19T12:45:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f013013ab20 con 0x7f0118002860 2024-08-19T12:45:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 10 client.26135.objecter ms_handle_connect 0x7f0114041980 2024-08-19T12:45:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 75624314 0 0) 0x7f013013ab20 con 0x7f0118002860 2024-08-19T12:45:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01180065c0 con 0x7f0118002860 2024-08-19T12:45:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f012c003e10 con 0x7f0118002860 2024-08-19T12:45:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 10 monclient: dump: 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.515 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2328624608 0 0) 0x7f01180065c0 con 0x7f0118002860 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.489+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f011800a160 con 0x7f0118002860 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.490+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f012c004100 con 0x7f0118002860 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.490+0000 7f01267fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.490+0000 7f01267fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.490+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.104:6791/0 conn(0x7f0118002860 legacy=0x7f0118004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.490+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.104:6791/0 conn(0x7f0118002860 legacy=0x7f0118004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.496+0000 7f0137f3a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.496+0000 7f0137f3a640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0118009e70 con 0x7f0118009130 2024-08-19T12:45:06.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.498+0000 7f0135cd6640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.104:6791/0 conn(0x7f0118009130 legacy=0x7f01180095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.498+0000 7f01267fc640 10 client.26135.objecter ms_handle_connect 0x7f0118009130 2024-08-19T12:45:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.498+0000 7f01267fc640 10 client.26135.objecter resend_mon_ops 2024-08-19T12:45:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 195566623 0 0) 0x7f0118009e70 con 0x7f0118009130 2024-08-19T12:45:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 10 monclient(hunting): my global_id is 26267 2024-08-19T12:45:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01180065c0 con 0x7f0118009130 2024-08-19T12:45:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1928617721 0 0) 0x7f01180065c0 con 0x7f0118009130 2024-08-19T12:45:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0118009e70 con 0x7f0118009130 2024-08-19T12:45:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f012c005b00 con 0x7f0118009130 2024-08-19T12:45:06.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:45:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 10 monclient: dump: 2024-08-19T12:45:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.518 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.519 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.519 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.519 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.519 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.519 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.519 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.519 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4050417995 0 0) 0x7f0118009e70 con 0x7f0118009130 2024-08-19T12:45:06.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.499+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f011800c410 con 0x7f0118009130 2024-08-19T12:45:06.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1483243355) 0x7f012c007e30 con 0x7f0118009130 2024-08-19T12:45:06.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f01267fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:06.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f01267fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:06.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.104:6791/0 conn(0x7f0118009130 legacy=0x7f01180095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f01267fc640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.104:6791/0 conn(0x7f0118009130 legacy=0x7f01180095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:06.520 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:06.520 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 419523, 2024-08-19T12:45:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.521 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:45:06.522 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:06.522 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:06.522 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:06.522 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:06.522 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:06.522 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:06.522 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:06.522 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:06.522 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:06.522 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:06.522 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:06.522 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:06.523 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:06.524 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:06.525 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:06.526 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:06.526 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:06.526 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:06.526 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:06.526 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:06.526 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:06.526 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:06.526 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.527 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:06.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.527 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:06.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.527 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:06.527 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:06.527 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.527 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.527 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.527 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:06.528 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.528 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.528 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.528 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:06.528 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:06.528 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.528 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.528 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.528 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:06.529 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.530 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.531 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.532 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.532 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:06.532 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.532 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.532 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:06.532 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:06.532 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.532 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.534 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.535 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:06.536 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.537 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.539 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:06.540 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:06.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.541 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.541 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:06.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.541 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:06.541 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:06.541 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:45:06.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.541 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.541 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:06.541 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:06.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 10 librados: watch_flush enter 2024-08-19T12:45:06.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 10 librados: watch_flush exit 2024-08-19T12:45:06.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 20 client.26135.objecter shutdown clearing up homeless session... 2024-08-19T12:45:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 10 client.26135.objecter successfully canceled tick 2024-08-19T12:45:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.104:6800/899887506 conn(0x7f0114041980 legacy=0x7f0114043e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 10 monclient: shutdown 2024-08-19T12:45:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 1 -- 172.21.15.26:0/776541612 >> v1:172.21.15.26:6793/0 conn(0x7f01300a5d50 legacy=0x7f01301d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f01364d7640 1 -- 172.21.15.26:0/776541612 reap_dead start 2024-08-19T12:45:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f01364d7640 1 -- 172.21.15.26:0/776541612 reap_dead start 2024-08-19T12:45:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 1 -- 172.21.15.26:0/776541612 shutdown_connections 2024-08-19T12:45:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 1 -- 172.21.15.26:0/776541612 >> 172.21.15.26:0/776541612 conn(0x7f01300a8740 msgr2=0x7f01300ab380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 1 -- 172.21.15.26:0/776541612 shutdown_connections 2024-08-19T12:45:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 1 -- 172.21.15.26:0/776541612 wait complete. 2024-08-19T12:45:06.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.500+0000 7f0137f3a640 1 librados: shutdown 2024-08-19T12:45:06.545 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:06.545 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:45:06.545 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:45:06.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.643+0000 7fdc81e15640 10 monclient: get_monmap_and_config 2024-08-19T12:45:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.643+0000 7fdc81e15640 10 monclient: build_initial_monmap 2024-08-19T12:45:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.643+0000 7fdc81e15640 10 monclient: monmap: 2024-08-19T12:45:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:06.645800+0000 2024-08-19T12:45:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:06.645800+0000 2024-08-19T12:45:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:06.646 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:06.647 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:06.647 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.644+0000 7fdc81e15640 1 Processor -- start 2024-08-19T12:45:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.644+0000 7fdc81e15640 1 -- start start 2024-08-19T12:45:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.644+0000 7fdc81e15640 10 monclient: init 2024-08-19T12:45:06.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc81e15640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc81e15640 10 monclient: _add_conns ranks=[1,7,5,3,4,8,6,0,2] 2024-08-19T12:45:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc81e15640 10 monclient(hunting): picked mon.noname-f con 0x7fdc7c0a5cd0 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc81e15640 10 monclient(hunting): picked mon.noname-i con 0x7fdc7c0a1e80 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc81e15640 10 monclient(hunting): picked mon.noname-h con 0x7fdc7c139aa0 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc81e15640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7c0a26b0 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc81e15640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7c0a2920 con 0x7fdc7c139aa0 2024-08-19T12:45:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc81e15640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7c0a2c00 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc81e15640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc81e15640 10 monclient(hunting): authenticate will time out at 1100.733497s 2024-08-19T12:45:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc73fff640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7fdc7c0a1e80 0x7fdc7c13e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:33774/0 (socket says 172.21.15.26:33774) 2024-08-19T12:45:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc7b7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fdc7c0a5cd0 0x7fdc7c139390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:52176/0 (socket says 172.21.15.26:52176) 2024-08-19T12:45:06.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.645+0000 7fdc73fff640 1 -- 172.21.15.26:0/3670901180 learned_addr learned my addr 172.21.15.26:0/3670901180 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:06.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.647+0000 7fdc7b7fe640 1 -- 172.21.15.26:0/3670901180 >> v1:172.21.15.104:6789/0 conn(0x7fdc7c0a5cd0 legacy=0x7fdc7c139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.647+0000 7fdc73fff640 1 -- 172.21.15.26:0/3670901180 >> v1:172.21.15.104:6792/0 conn(0x7fdc7c0a1e80 legacy=0x7fdc7c13e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.647+0000 7fdc7bfff640 1 -- 172.21.15.26:0/3670901180 >> v1:172.21.15.104:6791/0 conn(0x7fdc7c139aa0 legacy=0x7fdc7c141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2694160231 0 0) 0x7fdc7c0a2c00 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient(hunting): my global_id is 135187 2024-08-19T12:45:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc68003520 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 40263550 0 0) 0x7fdc7c0a26b0 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient(hunting): my global_id is 27019 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc7c0a2c00 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 685282717 0 0) 0x7fdc7c0a2920 con 0x7fdc7c139aa0 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient(hunting): my global_id is 26276 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc7c0a26b0 con 0x7fdc7c139aa0 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4202075553 0 0) 0x7fdc68003520 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 >> v1:172.21.15.104:6791/0 conn(0x7fdc7c139aa0 legacy=0x7fdc7c141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 >> v1:172.21.15.104:6789/0 conn(0x7fdc7c0a5cd0 legacy=0x7fdc7c139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 monclient: found mon.noname-i 2024-08-19T12:45:06.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7c0a2ee0 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: _finish_auth 0 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdc64001ae0 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: dump: 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.656 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc81e15640 5 monclient: authenticate success, global_id 135187 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc81e15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2196750664 0 0) 0x7fdc7c0a2c00 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdc6c001ac0 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdc640048a0 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 1 -- 172.21.15.26:0/3670901180 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdc64004e50 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc81e15640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc81e15640 4 set_mon_vals no callback set 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc7affd640 10 monclient: dump: 2024-08-19T12:45:06.658 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.659 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.660 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.648+0000 7fdc81e15640 10 monclient: shutdown 2024-08-19T12:45:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.649+0000 7fdc81e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.649+0000 7fdc81e15640 1 -- 172.21.15.26:0/3670901180 >> v1:172.21.15.104:6792/0 conn(0x7fdc7c0a1e80 legacy=0x7fdc7c13e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.649+0000 7fdc81e15640 1 -- 172.21.15.26:0/3670901180 shutdown_connections 2024-08-19T12:45:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.649+0000 7fdc81e15640 1 -- 172.21.15.26:0/3670901180 >> 172.21.15.26:0/3670901180 conn(0x7fdc7c06ce70 msgr2=0x7fdc7c0a55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.649+0000 7fdc81e15640 1 -- 172.21.15.26:0/3670901180 shutdown_connections 2024-08-19T12:45:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.649+0000 7fdc81e15640 1 -- 172.21.15.26:0/3670901180 wait complete. 2024-08-19T12:45:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.649+0000 7fdc81e15640 10 monclient: build_initial_monmap 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.649+0000 7fdc81e15640 10 monclient: monmap: 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:06.661 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:06.662 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.649+0000 7fdc81e15640 1 librados: starting msgr at 2024-08-19T12:45:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.649+0000 7fdc81e15640 1 librados: starting objecter 2024-08-19T12:45:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 1 Processor -- start 2024-08-19T12:45:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 1 -- start start 2024-08-19T12:45:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 1 librados: setting wanted keys 2024-08-19T12:45:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 1 librados: calling monclient init 2024-08-19T12:45:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 10 monclient: init 2024-08-19T12:45:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 10 monclient: _add_conns ranks=[1,4,6,2,5,0,8,3,7] 2024-08-19T12:45:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 10 monclient(hunting): picked mon.noname-f con 0x7fdc7c0a1e80 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c0a5cd0 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 10 monclient(hunting): picked mon.noname-d con 0x7fdc7c139aa0 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7c0721d0 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7c12de30 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7c1387d0 con 0x7fdc7c139aa0 2024-08-19T12:45:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc81e15640 10 monclient(hunting): authenticate will time out at 1100.738574s 2024-08-19T12:45:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc73fff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fdc7c0a5cd0 0x7fdc7c0708b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:42834/0 (socket says 172.21.15.26:42834) 2024-08-19T12:45:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc73fff640 1 -- 172.21.15.26:0/1273912421 learned_addr learned my addr 172.21.15.26:0/1273912421 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:06.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.650+0000 7fdc7bfff640 1 --1- 172.21.15.26:0/1273912421 >> v1:172.21.15.26:6792/0 conn(0x7fdc7c139aa0 0x7fdc7c1d6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49852/0 (socket says 172.21.15.26:49852) 2024-08-19T12:45:06.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc73fff640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.26:6791/0 conn(0x7fdc7c0a5cd0 legacy=0x7fdc7c0708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc78ff9640 10 client.?.objecter ms_handle_connect 0x7fdc7c0a5cd0 2024-08-19T12:45:06.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc78ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:06.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc78ff9640 10 client.?.objecter ms_handle_connect 0x7fdc7c0a1e80 2024-08-19T12:45:06.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc7b7fe640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.104:6789/0 conn(0x7fdc7c0a1e80 legacy=0x7fdc7c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc78ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc78ff9640 10 client.?.objecter ms_handle_connect 0x7fdc7c139aa0 2024-08-19T12:45:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc78ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc7bfff640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.26:6792/0 conn(0x7fdc7c139aa0 legacy=0x7fdc7c1d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 102394293 0 0) 0x7fdc7c0a2ee0 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc78ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc78ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.652+0000 7fdc78ff9640 10 monclient(hunting): my global_id is 26599 2024-08-19T12:45:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc5c004eb0 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4225343198 0 0) 0x7fdc7c0a26b0 con 0x7fdc7c139aa0 2024-08-19T12:45:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): my global_id is 26610 2024-08-19T12:45:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc7c0a2ee0 con 0x7fdc7c139aa0 2024-08-19T12:45:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 322831595 0 0) 0x7fdc7c0721d0 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): my global_id is 27028 2024-08-19T12:45:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc7c0a26b0 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1839529581 0 0) 0x7fdc7c0a2ee0 con 0x7fdc7c139aa0 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc7c0721d0 con 0x7fdc7c139aa0 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdc740020b0 con 0x7fdc7c139aa0 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): dump: 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.673 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.674 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.674 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.674 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.674 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.674 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.674 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.674 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.674 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.674 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:06.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 894418704 0 0) 0x7fdc5c004eb0 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc7c0a2ee0 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdc64004d90 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): dump: 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.675 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.676 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.676 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.676 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.676 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.676 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.676 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.676 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1593648857 0 0) 0x7fdc7c0a26b0 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc5c004eb0 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdc6c0019e0 con 0x7fdc7c0a1e80 2024-08-19T12:45:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:45:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient(hunting): dump: 2024-08-19T12:45:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.677 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.678 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.678 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.678 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.678 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.678 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.678 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.678 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 668046671 0 0) 0x7fdc7c0a2ee0 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.26:6792/0 conn(0x7fdc7c139aa0 legacy=0x7fdc7c1d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.104:6789/0 conn(0x7fdc7c0a1e80 legacy=0x7fdc7c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:45:06.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 monclient: found mon.c 2024-08-19T12:45:06.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:06.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:06.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7c073b70 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient: _finish_auth 0 2024-08-19T12:45:06.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:45:06.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:06.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc81e15640 5 monclient: authenticate success, global_id 26599 2024-08-19T12:45:06.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc81e15640 10 monclient: _renew_subs 2024-08-19T12:45:06.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc81e15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:06.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc81e15640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdc7c1382a0 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc81e15640 10 client.26599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:06.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc81e15640 10 monclient: _renew_subs 2024-08-19T12:45:06.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc81e15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:06.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc81e15640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fdc7c073890 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc81e15640 1 librados: init done 2024-08-19T12:45:06.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdc64001d30 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:06.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdc64001b00 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:06.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc7a7fc640 4 set_mon_vals no callback set 2024-08-19T12:45:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.653+0000 7fdc78ff9640 10 monclient: dump: 2024-08-19T12:45:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.681 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.654+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fdc640119c0 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.654+0000 7fdc81e15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.654+0000 7fdc81e15640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdc600065c0 con 0x7fdc60002860 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.654+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(279..279 src has 1..279) ==== 5338+0+0 (unknown 3765987609 0 0) 0x7fdc6404e140 con 0x7fdc7c0a5cd0 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.654+0000 7fdc78ff9640 10 client.26599.objecter ms_dispatch 0x7fdc7c000b90 osd_map(279..279 src has 1..279) 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.654+0000 7fdc78ff9640 3 client.26599.objecter handle_osd_map got epochs [279,279] > 0 2024-08-19T12:45:06.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.654+0000 7fdc78ff9640 3 client.26599.objecter handle_osd_map decoding full epoch 279 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.654+0000 7fdc78ff9640 20 client.26599.objecter dump_active .. 0 homeless 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.655+0000 7fdc7bfff640 10 monclient: get_auth_request con 0x7fdc5c041a60 auth_method 0 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 10 client.26599.objecter ms_handle_connect 0x7fdc60002860 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 10 client.26599.objecter resend_mon_ops 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc7b7fe640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.104:6792/0 conn(0x7fdc60002860 legacy=0x7fdc60004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 663875281 0 0) 0x7fdc600065c0 con 0x7fdc60002860 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 10 monclient(hunting): my global_id is 135196 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc7c0a2ee0 con 0x7fdc60002860 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 10 client.26599.objecter ms_handle_connect 0x7fdc5c041a60 2024-08-19T12:45:06.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3980689737 0 0) 0x7fdc7c0a2ee0 con 0x7fdc60002860 2024-08-19T12:45:06.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc600065c0 con 0x7fdc60002860 2024-08-19T12:45:06.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdc6c005050 con 0x7fdc60002860 2024-08-19T12:45:06.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:45:06.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 10 monclient: dump: 2024-08-19T12:45:06.684 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.684 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.684 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.684 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.684 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.684 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.684 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.685 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.685 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.685 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.685 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.685 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.685 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.685 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.685 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.685 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 443840043 0 0) 0x7fdc600065c0 con 0x7fdc60002860 2024-08-19T12:45:06.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.657+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc6000a160 con 0x7fdc60002860 2024-08-19T12:45:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.658+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdc6c0051f0 con 0x7fdc60002860 2024-08-19T12:45:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.658+0000 7fdc78ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.658+0000 7fdc78ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.658+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.104:6792/0 conn(0x7fdc60002860 legacy=0x7fdc60004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.658+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.104:6792/0 conn(0x7fdc60002860 legacy=0x7fdc60004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.664+0000 7fdc81e15640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.664+0000 7fdc81e15640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdc60009e70 con 0x7fdc60009130 2024-08-19T12:45:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.666+0000 7fdc7b7fe640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.104:6792/0 conn(0x7fdc60009130 legacy=0x7fdc600095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.666+0000 7fdc78ff9640 10 client.26599.objecter ms_handle_connect 0x7fdc60009130 2024-08-19T12:45:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.666+0000 7fdc78ff9640 10 client.26599.objecter resend_mon_ops 2024-08-19T12:45:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.666+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2684675152 0 0) 0x7fdc60009e70 con 0x7fdc60009130 2024-08-19T12:45:06.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.666+0000 7fdc78ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.666+0000 7fdc78ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.666+0000 7fdc78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.666+0000 7fdc78ff9640 10 monclient(hunting): my global_id is 135205 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.666+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc600065c0 con 0x7fdc60009130 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 319569839 0 0) 0x7fdc600065c0 con 0x7fdc60009130 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc60009e70 con 0x7fdc60009130 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fdc6c005a20 con 0x7fdc60009130 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 10 monclient: dump: 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.687 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3423290822 0 0) 0x7fdc60009e70 con 0x7fdc60009130 2024-08-19T12:45:06.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdc6000c3f0 con 0x7fdc60009130 2024-08-19T12:45:06.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1237317673) 0x7fdc6c008000 con 0x7fdc60009130 2024-08-19T12:45:06.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:06.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:06.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.104:6792/0 conn(0x7fdc60009130 legacy=0x7fdc600095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc78ff9640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.104:6792/0 conn(0x7fdc60009130 legacy=0x7fdc600095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:06.689 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:06.689 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:06.689 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 277026, 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:06.690 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:06.691 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:06.692 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:06.693 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:06.694 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.695 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.696 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:06.699 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.700 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.702 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.703 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:06.704 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.704 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:06.704 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:06.704 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.704 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.704 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.704 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.705 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.706 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.706 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:06.706 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:06.706 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:06.706 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.706 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:06.707 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:06.707 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.707 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.707 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.707 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:06.707 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:06.708 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:06.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:06.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.708 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:06.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:06.708 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:06.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:06.708 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:06.709 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:06.709 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:06.709 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:06.709 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.709 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.709 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:06.709 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:06.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.709 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:06.709 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:06.709 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:06.711 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:06.711 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:06.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc81e15640 10 librados: watch_flush enter 2024-08-19T12:45:06.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc81e15640 10 librados: watch_flush exit 2024-08-19T12:45:06.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc81e15640 20 client.26599.objecter shutdown clearing up homeless session... 2024-08-19T12:45:06.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc81e15640 10 client.26599.objecter successfully canceled tick 2024-08-19T12:45:06.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc81e15640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.104:6800/899887506 conn(0x7fdc5c041a60 legacy=0x7fdc5c043f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc81e15640 10 monclient: shutdown 2024-08-19T12:45:06.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc81e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:06.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.667+0000 7fdc81e15640 1 -- 172.21.15.26:0/1273912421 >> v1:172.21.15.26:6791/0 conn(0x7fdc7c0a5cd0 legacy=0x7fdc7c0708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.668+0000 7fdc7bfff640 1 -- 172.21.15.26:0/1273912421 reap_dead start 2024-08-19T12:45:06.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.668+0000 7fdc7bfff640 1 -- 172.21.15.26:0/1273912421 reap_dead start 2024-08-19T12:45:06.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.668+0000 7fdc81e15640 1 -- 172.21.15.26:0/1273912421 shutdown_connections 2024-08-19T12:45:06.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.668+0000 7fdc81e15640 1 -- 172.21.15.26:0/1273912421 >> 172.21.15.26:0/1273912421 conn(0x7fdc7c06ce70 msgr2=0x7fdc7c0a3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:06.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.668+0000 7fdc81e15640 1 -- 172.21.15.26:0/1273912421 shutdown_connections 2024-08-19T12:45:06.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.668+0000 7fdc81e15640 1 -- 172.21.15.26:0/1273912421 wait complete. 2024-08-19T12:45:06.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.668+0000 7fdc81e15640 1 librados: shutdown 2024-08-19T12:45:06.714 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:06.714 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T12:45:06.714 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T12:45:06.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.811+0000 7fcd98cb1640 10 monclient: get_monmap_and_config 2024-08-19T12:45:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.811+0000 7fcd98cb1640 10 monclient: build_initial_monmap 2024-08-19T12:45:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.812+0000 7fcd98cb1640 10 monclient: monmap: 2024-08-19T12:45:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:06.813845+0000 2024-08-19T12:45:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:06.813845+0000 2024-08-19T12:45:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.814 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:06.815 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:06.815 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.812+0000 7fcd98cb1640 1 Processor -- start 2024-08-19T12:45:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.812+0000 7fcd98cb1640 1 -- start start 2024-08-19T12:45:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.812+0000 7fcd98cb1640 10 monclient: init 2024-08-19T12:45:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.813+0000 7fcd98cb1640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.813+0000 7fcd98cb1640 10 monclient: _add_conns ranks=[1,8,7,5,0,3,4,6,2] 2024-08-19T12:45:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.813+0000 7fcd98cb1640 10 monclient(hunting): picked mon.noname-f con 0x7fcd9406e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.813+0000 7fcd98cb1640 10 monclient(hunting): picked mon.noname-e con 0x7fcd940a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.813+0000 7fcd98cb1640 10 monclient(hunting): picked mon.noname-i con 0x7fcd940a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.813+0000 7fcd98cb1640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd94136790 con 0x7fcd9406e7b0 2024-08-19T12:45:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.813+0000 7fcd98cb1640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd9409ddf0 con 0x7fcd940a5d50 2024-08-19T12:45:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.813+0000 7fcd98cb1640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd9409e0d0 con 0x7fcd940a5920 2024-08-19T12:45:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.813+0000 7fcd98cb1640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:06.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.813+0000 7fcd98cb1640 10 monclient(hunting): authenticate will time out at 1100.901266s 2024-08-19T12:45:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.813+0000 7fcd91d9b640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fcd940a5920 0x7fcd9413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34282/0 (socket says 172.21.15.26:34282) 2024-08-19T12:45:06.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.813+0000 7fcd91d9b640 1 -- 172.21.15.26:0/437733984 learned_addr learned my addr 172.21.15.26:0/437733984 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:06.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd91d9b640 1 -- 172.21.15.26:0/437733984 >> v1:172.21.15.26:6793/0 conn(0x7fcd940a5920 legacy=0x7fcd9413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9259c640 1 -- 172.21.15.26:0/437733984 >> v1:172.21.15.104:6789/0 conn(0x7fcd9406e7b0 legacy=0x7fcd94136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd92d9d640 1 -- 172.21.15.26:0/437733984 >> v1:172.21.15.104:6792/0 conn(0x7fcd940a5d50 legacy=0x7fcd94141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3593294701 0 0) 0x7fcd9409e0d0 con 0x7fcd940a5920 2024-08-19T12:45:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 10 monclient(hunting): my global_id is 26144 2024-08-19T12:45:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd80003520 con 0x7fcd940a5920 2024-08-19T12:45:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3212147975 0 0) 0x7fcd94136790 con 0x7fcd9406e7b0 2024-08-19T12:45:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 10 monclient(hunting): my global_id is 27037 2024-08-19T12:45:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd9409e0d0 con 0x7fcd9406e7b0 2024-08-19T12:45:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 519683832 0 0) 0x7fcd8c0013d0 con 0x7fcd940a5d50 2024-08-19T12:45:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 10 monclient(hunting): my global_id is 135214 2024-08-19T12:45:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.815+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd94136790 con 0x7fcd940a5d50 2024-08-19T12:45:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 85334767 0 0) 0x7fcd80003520 con 0x7fcd940a5920 2024-08-19T12:45:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 >> v1:172.21.15.104:6792/0 conn(0x7fcd940a5d50 legacy=0x7fcd94141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 >> v1:172.21.15.104:6789/0 conn(0x7fcd9406e7b0 legacy=0x7fcd94136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 10 monclient: _finish_hunting 0 2024-08-19T12:45:06.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 1 monclient: found mon.noname-e 2024-08-19T12:45:06.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:06.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:45:06.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd9409e3b0 con 0x7fcd940a5920 2024-08-19T12:45:06.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 10 monclient: _finish_auth 0 2024-08-19T12:45:06.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 10 monclient: _check_auth_tickets 2024-08-19T12:45:06.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:06.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcd7c001ae0 con 0x7fcd940a5920 2024-08-19T12:45:06.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:06.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 10 monclient: dump: 2024-08-19T12:45:06.829 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.829 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.829 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.829 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.829 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.829 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.829 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.829 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.830 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.830 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.830 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.830 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.830 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.830 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.830 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.830 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd98cb1640 5 monclient: authenticate success, global_id 26144 2024-08-19T12:45:06.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd98cb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:06.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcd7c0048a0 con 0x7fcd940a5920 2024-08-19T12:45:06.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:06.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 1 -- 172.21.15.26:0/437733984 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcd7c004e10 con 0x7fcd940a5920 2024-08-19T12:45:06.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd98cb1640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:06.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd98cb1640 4 set_mon_vals no callback set 2024-08-19T12:45:06.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:06.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd9159a640 10 monclient: dump: 2024-08-19T12:45:06.831 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.831 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.832 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd98cb1640 10 monclient: shutdown 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd98cb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd98cb1640 1 -- 172.21.15.26:0/437733984 >> v1:172.21.15.26:6793/0 conn(0x7fcd940a5920 legacy=0x7fcd9413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd98cb1640 1 -- 172.21.15.26:0/437733984 shutdown_connections 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd98cb1640 1 -- 172.21.15.26:0/437733984 >> 172.21.15.26:0/437733984 conn(0x7fcd940a8740 msgr2=0x7fcd940a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd98cb1640 1 -- 172.21.15.26:0/437733984 shutdown_connections 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.816+0000 7fcd98cb1640 1 -- 172.21.15.26:0/437733984 wait complete. 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.817+0000 7fcd98cb1640 10 monclient: build_initial_monmap 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.817+0000 7fcd98cb1640 10 monclient: monmap: 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:06.833 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:06.834 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:06.834 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:06.834 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:06.834 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.834 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:06.834 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:06.834 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:06.834 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:06.834 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:06.834 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:06.834 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:06.835 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.817+0000 7fcd98cb1640 1 librados: starting msgr at 2024-08-19T12:45:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.817+0000 7fcd98cb1640 1 librados: starting objecter 2024-08-19T12:45:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.817+0000 7fcd98cb1640 1 Processor -- start 2024-08-19T12:45:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.817+0000 7fcd98cb1640 1 -- start start 2024-08-19T12:45:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.817+0000 7fcd98cb1640 1 librados: setting wanted keys 2024-08-19T12:45:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.817+0000 7fcd98cb1640 1 librados: calling monclient init 2024-08-19T12:45:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.817+0000 7fcd98cb1640 10 monclient: init 2024-08-19T12:45:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.818+0000 7fcd98cb1640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:06.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.818+0000 7fcd98cb1640 10 monclient: _add_conns ranks=[8,1,3,2,7,4,0,6,5] 2024-08-19T12:45:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.818+0000 7fcd98cb1640 10 monclient(hunting): picked mon.noname-e con 0x7fcd9406e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.818+0000 7fcd98cb1640 10 monclient(hunting): picked mon.noname-f con 0x7fcd940a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.818+0000 7fcd98cb1640 10 monclient(hunting): picked mon.noname-g con 0x7fcd940a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.818+0000 7fcd98cb1640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd940a18d0 con 0x7fcd940a5920 2024-08-19T12:45:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.818+0000 7fcd98cb1640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd94134a70 con 0x7fcd940a5d50 2024-08-19T12:45:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.818+0000 7fcd98cb1640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd94134e60 con 0x7fcd9406e7b0 2024-08-19T12:45:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.818+0000 7fcd98cb1640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.818+0000 7fcd98cb1640 10 monclient(hunting): authenticate will time out at 1100.906406s 2024-08-19T12:45:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.818+0000 7fcd9259c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fcd9406e7b0 0x7fcd941ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34290/0 (socket says 172.21.15.26:34290) 2024-08-19T12:45:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.818+0000 7fcd9259c640 1 -- 172.21.15.26:0/511120244 learned_addr learned my addr 172.21.15.26:0/511120244 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:06.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd9259c640 1 -- 172.21.15.26:0/511120244 >> v1:172.21.15.26:6793/0 conn(0x7fcd9406e7b0 legacy=0x7fcd941ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 client.?.objecter ms_handle_connect 0x7fcd9406e7b0 2024-08-19T12:45:06.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:06.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd92d9d640 1 -- 172.21.15.26:0/511120244 >> v1:172.21.15.104:6790/0 conn(0x7fcd940a5d50 legacy=0x7fcd941d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2040401366 0 0) 0x7fcd94134e60 con 0x7fcd9406e7b0 2024-08-19T12:45:06.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 monclient(hunting): my global_id is 26153 2024-08-19T12:45:06.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd91d9b640 1 -- 172.21.15.26:0/511120244 >> v1:172.21.15.104:6789/0 conn(0x7fcd940a5920 legacy=0x7fcd941d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:06.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd74004ee0 con 0x7fcd9406e7b0 2024-08-19T12:45:06.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 client.?.objecter ms_handle_connect 0x7fcd940a5d50 2024-08-19T12:45:06.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:06.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 client.?.objecter ms_handle_connect 0x7fcd940a5920 2024-08-19T12:45:06.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:06.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4114848222 0 0) 0x7fcd940a18d0 con 0x7fcd940a5920 2024-08-19T12:45:06.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 monclient(hunting): my global_id is 27046 2024-08-19T12:45:06.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd94134e60 con 0x7fcd940a5920 2024-08-19T12:45:06.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1677497686 0 0) 0x7fcd94136790 con 0x7fcd940a5d50 2024-08-19T12:45:06.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 10 monclient(hunting): my global_id is 154155 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd940a18d0 con 0x7fcd940a5d50 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1799830597 0 0) 0x7fcd74004ee0 con 0x7fcd9406e7b0 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.820+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd94136790 con 0x7fcd9406e7b0 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcd84003990 con 0x7fcd9406e7b0 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient(hunting): dump: 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.840 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.840 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.840 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.840 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.840 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.840 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.840 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.840 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.840 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.840 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.840 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.840 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 638757660 0 0) 0x7fcd94134e60 con 0x7fcd940a5920 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd74004ee0 con 0x7fcd940a5920 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcd7c004d90 con 0x7fcd940a5920 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient(hunting): dump: 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.841 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.842 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.842 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.842 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.842 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.842 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.842 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.842 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.842 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.842 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3058387195 0 0) 0x7fcd94136790 con 0x7fcd9406e7b0 2024-08-19T12:45:06.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 >> v1:172.21.15.104:6790/0 conn(0x7fcd940a5d50 legacy=0x7fcd941d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 >> v1:172.21.15.104:6789/0 conn(0x7fcd940a5920 legacy=0x7fcd941d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 1 monclient: found mon.e 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd940a2bf0 con 0x7fcd9406e7b0 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient: _finish_auth 0 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd98cb1640 5 monclient: authenticate success, global_id 26153 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd98cb1640 10 monclient: _renew_subs 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd98cb1640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd98cb1640 1 -- 172.21.15.26:0/511120244 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcd940a3de0 con 0x7fcd9406e7b0 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd98cb1640 10 client.26153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:06.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd98cb1640 10 monclient: _renew_subs 2024-08-19T12:45:06.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd98cb1640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:06.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd98cb1640 1 -- 172.21.15.26:0/511120244 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fcd940a0a20 con 0x7fcd9406e7b0 2024-08-19T12:45:06.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd98cb1640 1 librados: init done 2024-08-19T12:45:06.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcd84003f60 con 0x7fcd9406e7b0 2024-08-19T12:45:06.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:06.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcd84004490 con 0x7fcd9406e7b0 2024-08-19T12:45:06.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:06.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:06.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd90d99640 4 set_mon_vals no callback set 2024-08-19T12:45:06.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.821+0000 7fcd8affd640 10 monclient: dump: 2024-08-19T12:45:06.844 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:06.844 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:06.845 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.845 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:06.845 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:06.845 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:06.845 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:06.845 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:06.845 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:06.845 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:06.845 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:06.845 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:06.845 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:06.845 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:06.846 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:06.846 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:06.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.822+0000 7fcd98cb1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:06.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.822+0000 7fcd98cb1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:06.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.822+0000 7fcd98cb1640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:06.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.822+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fcd84004630 con 0x7fcd9406e7b0 2024-08-19T12:45:06.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.822+0000 7fcd98cb1640 1 -- 172.21.15.26:0/511120244 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd78005080 con 0x7fcd9406e7b0 2024-08-19T12:45:06.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.822+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(279..279 src has 1..279) ==== 5338+0+0 (unknown 3765987609 0 0) 0x7fcd8404e650 con 0x7fcd9406e7b0 2024-08-19T12:45:06.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.822+0000 7fcd8affd640 10 client.26153.objecter ms_dispatch 0x7fcd94000b90 osd_map(279..279 src has 1..279) 2024-08-19T12:45:06.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.822+0000 7fcd8affd640 3 client.26153.objecter handle_osd_map got epochs [279,279] > 0 2024-08-19T12:45:06.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.822+0000 7fcd8affd640 3 client.26153.objecter handle_osd_map decoding full epoch 279 2024-08-19T12:45:06.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.822+0000 7fcd8affd640 20 client.26153.objecter dump_active .. 0 homeless 2024-08-19T12:45:06.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.822+0000 7fcd91d9b640 10 monclient: get_auth_request con 0x7fcd74041500 auth_method 0 2024-08-19T12:45:06.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.825+0000 7fcd8affd640 10 client.26153.objecter ms_handle_connect 0x7fcd74041500 2024-08-19T12:45:06.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.827+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fcd84017a60 con 0x7fcd9406e7b0 2024-08-19T12:45:06.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.827+0000 7fcd8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:06.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:06.827+0000 7fcd8affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:07.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.084+0000 7fcd98cb1640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T12:45:07.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.084+0000 7fcd98cb1640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T12:45:07.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.084+0000 7fcd98cb1640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:07.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.084+0000 7fcd98cb1640 1 -- 172.21.15.26:0/511120244 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fcd78005c40 con 0x7fcd9406e7b0 2024-08-19T12:45:07.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.086+0000 7fcd8affd640 1 -- 172.21.15.26:0/511120244 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fcd84017360 con 0x7fcd9406e7b0 2024-08-19T12:45:07.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.086+0000 7fcd8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T12:45:07.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.086+0000 7fcd8affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:07.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.088+0000 7fcd98cb1640 10 librados: watch_flush enter 2024-08-19T12:45:07.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.088+0000 7fcd98cb1640 10 librados: watch_flush exit 2024-08-19T12:45:07.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.088+0000 7fcd98cb1640 20 client.26153.objecter shutdown clearing up homeless session... 2024-08-19T12:45:07.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.088+0000 7fcd98cb1640 10 client.26153.objecter successfully canceled tick 2024-08-19T12:45:07.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.088+0000 7fcd98cb1640 1 -- 172.21.15.26:0/511120244 >> v1:172.21.15.104:6800/899887506 conn(0x7fcd74041500 legacy=0x7fcd740439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:07.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.088+0000 7fcd98cb1640 10 monclient: shutdown 2024-08-19T12:45:07.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.088+0000 7fcd98cb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:07.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.088+0000 7fcd98cb1640 1 -- 172.21.15.26:0/511120244 >> v1:172.21.15.26:6793/0 conn(0x7fcd9406e7b0 legacy=0x7fcd941ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:07.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.088+0000 7fcd98cb1640 1 -- 172.21.15.26:0/511120244 shutdown_connections 2024-08-19T12:45:07.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.088+0000 7fcd98cb1640 1 -- 172.21.15.26:0/511120244 >> 172.21.15.26:0/511120244 conn(0x7fcd940a8740 msgr2=0x7fcd940aa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:07.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.088+0000 7fcd98cb1640 1 -- 172.21.15.26:0/511120244 shutdown_connections 2024-08-19T12:45:07.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.088+0000 7fcd98cb1640 1 -- 172.21.15.26:0/511120244 wait complete. 2024-08-19T12:45:07.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:07.089+0000 7fcd98cb1640 1 librados: shutdown 2024-08-19T12:45:07.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T12:45:07.193 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.193 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=1,health=12,kv=13,logm=53} crc {auth=3949891568,config=1228700967,health=1497070696,kv=1344821656,logm=2584336645}) 2024-08-19T12:45:07.193 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=1701167229,config=1097430493,health=1497070696,kv=3460335929,logm=3916501930}) 2024-08-19T12:45:07.193 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.193 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=1,health=12,kv=13,logm=53} crc {auth=3949891568,config=1228700967,health=1497070696,kv=1344821656,logm=2584336645}) 2024-08-19T12:45:07.193 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=12,kv=12,logm=54} crc {auth=3026278071,config=1097430493,health=1497070696,kv=352587344,logm=4238019036}) 2024-08-19T12:45:07.193 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.193 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=1,health=12,kv=13,logm=53} crc {auth=3949891568,config=1228700967,health=1497070696,kv=1344821656,logm=2584336645}) 2024-08-19T12:45:07.193 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=19,config=2,health=12,kv=13,logm=54} crc {auth=805637857,config=1097430493,health=1497070696,kv=1344821656,logm=2612756727}) 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=1,health=12,kv=13,logm=53} crc {auth=3949891568,config=1228700967,health=1497070696,kv=1344821656,logm=2584336645}) 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=22,config=2,health=12,kv=13,logm=51} crc {auth=540256240,config=1097430493,health=1497070696,kv=1344821656,logm=3183913417}) 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=1,health=12,kv=13,logm=53} crc {auth=3949891568,config=1228700967,health=1497070696,kv=1344821656,logm=2584336645}) 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=21,config=2,health=10,kv=13,logm=54} crc {auth=3324370996,config=1097430493,health=3524487152,kv=1344821656,logm=849204005}) 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=1,health=12,kv=13,logm=53} crc {auth=3949891568,config=1228700967,health=1497070696,kv=1344821656,logm=2584336645}) 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=22,config=2,health=11,kv=12,logm=53} crc {auth=2399734105,config=1097430493,health=2069771779,kv=1344821656,logm=918305965}) 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=1,health=12,kv=13,logm=53} crc {auth=3949891568,config=1228700967,health=1497070696,kv=1344821656,logm=2584336645}) 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=20,config=2,health=11,kv=13,logm=54} crc {auth=242592504,config=1097430493,health=2069771779,kv=1344821656,logm=659224385}) 2024-08-19T12:45:07.194 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.195 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=1,health=12,kv=13,logm=53} crc {auth=3949891568,config=1228700967,health=1497070696,kv=1344821656,logm=2584336645}) 2024-08-19T12:45:07.195 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=20,health=12,kv=13,logm=55} crc {auth=3906541458,health=1497070696,kv=701615758,logm=384082457}) 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422675750}) 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=384258251}) 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422675750}) 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564318773}) 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422675750}) 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3756649136}) 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422675750}) 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3432907938}) 2024-08-19T12:45:07.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422675750}) 2024-08-19T12:45:07.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2792332176}) 2024-08-19T12:45:07.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422675750}) 2024-08-19T12:45:07.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=713310022}) 2024-08-19T12:45:07.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422675750}) 2024-08-19T12:45:07.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3919844574}) 2024-08-19T12:45:07.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422675750}) 2024-08-19T12:45:07.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3560388907}) 2024-08-19T12:45:07.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865908434}) 2024-08-19T12:45:07.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=108703435}) 2024-08-19T12:45:07.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865908434}) 2024-08-19T12:45:07.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2459719342}) 2024-08-19T12:45:07.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865908434}) 2024-08-19T12:45:07.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3346951442}) 2024-08-19T12:45:07.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865908434}) 2024-08-19T12:45:07.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=265729036}) 2024-08-19T12:45:07.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865908434}) 2024-08-19T12:45:07.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1257713099}) 2024-08-19T12:45:07.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865908434}) 2024-08-19T12:45:07.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4233311265}) 2024-08-19T12:45:07.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865908434}) 2024-08-19T12:45:07.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2262919904}) 2024-08-19T12:45:07.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865908434}) 2024-08-19T12:45:07.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=942692010}) 2024-08-19T12:45:07.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524402883}) 2024-08-19T12:45:07.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1013349319}) 2024-08-19T12:45:07.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524402883}) 2024-08-19T12:45:07.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783504363}) 2024-08-19T12:45:07.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524402883}) 2024-08-19T12:45:07.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2053862228}) 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524402883}) 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2648601119}) 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524402883}) 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=194070883}) 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524402883}) 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2633048737}) 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524402883}) 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2104582621}) 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524402883}) 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=250857355}) 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536546825}) 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1641233545}) 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536546825}) 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2390707353}) 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536546825}) 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1784037577}) 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536546825}) 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=669326767}) 2024-08-19T12:45:07.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536546825}) 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3914806947}) 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536546825}) 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4082401658}) 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536546825}) 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1726545045}) 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536546825}) 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.217+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=210726619}) 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244840707}) 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020372983}) 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244840707}) 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=695768220}) 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244840707}) 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1504081055}) 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244840707}) 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3954403607}) 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244840707}) 2024-08-19T12:45:07.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=438907882}) 2024-08-19T12:45:07.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244840707}) 2024-08-19T12:45:07.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3266145848}) 2024-08-19T12:45:07.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244840707}) 2024-08-19T12:45:07.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1143178409}) 2024-08-19T12:45:07.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244840707}) 2024-08-19T12:45:07.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=409882945}) 2024-08-19T12:45:07.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123074743}) 2024-08-19T12:45:07.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817257903}) 2024-08-19T12:45:07.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123074743}) 2024-08-19T12:45:07.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332712761}) 2024-08-19T12:45:07.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123074743}) 2024-08-19T12:45:07.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2396489904}) 2024-08-19T12:45:07.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123074743}) 2024-08-19T12:45:07.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1743683330}) 2024-08-19T12:45:07.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123074743}) 2024-08-19T12:45:07.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3075605212}) 2024-08-19T12:45:07.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123074743}) 2024-08-19T12:45:07.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3884193234}) 2024-08-19T12:45:07.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123074743}) 2024-08-19T12:45:07.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3458350466}) 2024-08-19T12:45:07.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123074743}) 2024-08-19T12:45:07.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.327+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2806167003}) 2024-08-19T12:45:07.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354187970}) 2024-08-19T12:45:07.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=212878186}) 2024-08-19T12:45:07.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354187970}) 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3265768702}) 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354187970}) 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1142631426}) 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354187970}) 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4208077878}) 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354187970}) 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=589203636}) 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354187970}) 2024-08-19T12:45:07.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=851754458}) 2024-08-19T12:45:07.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354187970}) 2024-08-19T12:45:07.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=335178447}) 2024-08-19T12:45:07.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354187970}) 2024-08-19T12:45:07.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.331+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1459439415}) 2024-08-19T12:45:07.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130385374}) 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3762059197}) 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130385374}) 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1477440876}) 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130385374}) 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3734647037}) 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130385374}) 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4009342235}) 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130385374}) 2024-08-19T12:45:07.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3952933320}) 2024-08-19T12:45:07.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130385374}) 2024-08-19T12:45:07.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2935169484}) 2024-08-19T12:45:07.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130385374}) 2024-08-19T12:45:07.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4105306160}) 2024-08-19T12:45:07.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:07.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130385374}) 2024-08-19T12:45:07.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:07.356+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3060424174}) 2024-08-19T12:45:08.103 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:45:08.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:45:08.190 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.190 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595446171}) 2024-08-19T12:45:08.190 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3941400611}) 2024-08-19T12:45:08.190 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.190 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595446171}) 2024-08-19T12:45:08.190 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=484110357}) 2024-08-19T12:45:08.190 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.190 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595446171}) 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=129878102}) 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595446171}) 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2827048444}) 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595446171}) 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3572729012}) 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595446171}) 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1609388602}) 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595446171}) 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1155195531}) 2024-08-19T12:45:08.191 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.192 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595446171}) 2024-08-19T12:45:08.192 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.187+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2756047464}) 2024-08-19T12:45:08.196 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.196 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207079236}) 2024-08-19T12:45:08.196 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1051266571}) 2024-08-19T12:45:08.196 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.197 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207079236}) 2024-08-19T12:45:08.197 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=451675820}) 2024-08-19T12:45:08.197 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.197 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207079236}) 2024-08-19T12:45:08.197 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=321199933}) 2024-08-19T12:45:08.197 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.197 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207079236}) 2024-08-19T12:45:08.198 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3408587688}) 2024-08-19T12:45:08.198 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.198 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207079236}) 2024-08-19T12:45:08.198 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2571259558}) 2024-08-19T12:45:08.198 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.198 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207079236}) 2024-08-19T12:45:08.198 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2941566006}) 2024-08-19T12:45:08.198 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.199 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207079236}) 2024-08-19T12:45:08.199 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1161839899}) 2024-08-19T12:45:08.199 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.199 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207079236}) 2024-08-19T12:45:08.199 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.193+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4043795727}) 2024-08-19T12:45:08.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.202 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006789671}) 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=736271031}) 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006789671}) 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=85029265}) 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006789671}) 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3792592749}) 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006789671}) 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2814549708}) 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006789671}) 2024-08-19T12:45:08.203 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1439546120}) 2024-08-19T12:45:08.204 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.204 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006789671}) 2024-08-19T12:45:08.204 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3027378313}) 2024-08-19T12:45:08.204 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.204 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006789671}) 2024-08-19T12:45:08.204 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=693278030}) 2024-08-19T12:45:08.204 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.204 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006789671}) 2024-08-19T12:45:08.204 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.200+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2169102355}) 2024-08-19T12:45:08.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.219+0000 7fb23bac7640 10 monclient: get_monmap_and_config 2024-08-19T12:45:08.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.219+0000 7fb23bac7640 10 monclient: build_initial_monmap 2024-08-19T12:45:08.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.219+0000 7fb23bac7640 10 monclient: monmap: 2024-08-19T12:45:08.222 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:08.222 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.222 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:08.221807+0000 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:08.221807+0000 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.220+0000 7fb23bac7640 1 Processor -- start 2024-08-19T12:45:08.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.220+0000 7fb23bac7640 1 -- start start 2024-08-19T12:45:08.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.220+0000 7fb23bac7640 10 monclient: init 2024-08-19T12:45:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb23bac7640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb23bac7640 10 monclient: _add_conns ranks=[5,0,8,3,6,7,2,4,1] 2024-08-19T12:45:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb23bac7640 10 monclient(hunting): picked mon.noname-h con 0x7fb234073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb23bac7640 10 monclient(hunting): picked mon.noname-a con 0x7fb2340a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb23bac7640 10 monclient(hunting): picked mon.noname-e con 0x7fb23413f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb23bac7640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb23409dae0 con 0x7fb2340a57a0 2024-08-19T12:45:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb23bac7640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb234136910 con 0x7fb234073470 2024-08-19T12:45:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb23bac7640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb23409dd50 con 0x7fb23413f2e0 2024-08-19T12:45:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb23bac7640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb23bac7640 10 monclient(hunting): authenticate will time out at 1102.309388s 2024-08-19T12:45:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb23a064640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fb23413f2e0 0x7fb2341416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:34308/0 (socket says 172.21.15.26:34308) 2024-08-19T12:45:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb239062640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb2340a57a0 0x7fb2340a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:45296/0 (socket says 172.21.15.26:45296) 2024-08-19T12:45:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.221+0000 7fb23a064640 1 -- 172.21.15.26:0/3277468938 learned_addr learned my addr 172.21.15.26:0/3277468938 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb239062640 1 -- 172.21.15.26:0/3277468938 >> v1:172.21.15.26:6789/0 conn(0x7fb2340a57a0 legacy=0x7fb2340a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb23a064640 1 -- 172.21.15.26:0/3277468938 >> v1:172.21.15.26:6793/0 conn(0x7fb23413f2e0 legacy=0x7fb2341416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3019236006 0 0) 0x7fb23409dae0 con 0x7fb2340a57a0 2024-08-19T12:45:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb238861640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb238861640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb239863640 1 -- 172.21.15.26:0/3277468938 >> v1:172.21.15.104:6791/0 conn(0x7fb234073470 legacy=0x7fb234136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb238861640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb238861640 10 monclient(hunting): my global_id is 144504 2024-08-19T12:45:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb220003520 con 0x7fb2340a57a0 2024-08-19T12:45:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2672560633 0 0) 0x7fb23409dd50 con 0x7fb23413f2e0 2024-08-19T12:45:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb238861640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb238861640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb238861640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb238861640 10 monclient(hunting): my global_id is 26162 2024-08-19T12:45:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.223+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb23409dae0 con 0x7fb23413f2e0 2024-08-19T12:45:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3948676311 0 0) 0x7fb234136910 con 0x7fb234073470 2024-08-19T12:45:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient(hunting): my global_id is 26285 2024-08-19T12:45:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb23409dd50 con 0x7fb234073470 2024-08-19T12:45:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 454672278 0 0) 0x7fb23409dae0 con 0x7fb23413f2e0 2024-08-19T12:45:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 >> v1:172.21.15.104:6791/0 conn(0x7fb234073470 legacy=0x7fb234136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 >> v1:172.21.15.26:6789/0 conn(0x7fb2340a57a0 legacy=0x7fb2340a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient: _finish_hunting 0 2024-08-19T12:45:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 1 monclient: found mon.noname-e 2024-08-19T12:45:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:45:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23409e030 con 0x7fb23413f2e0 2024-08-19T12:45:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient: _finish_auth 0 2024-08-19T12:45:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient: _check_auth_tickets 2024-08-19T12:45:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb23bac7640 5 monclient: authenticate success, global_id 26162 2024-08-19T12:45:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb22c001f00 con 0x7fb23413f2e0 2024-08-19T12:45:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient: dump: 2024-08-19T12:45:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.243 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb23bac7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb22c0020a0 con 0x7fb23413f2e0 2024-08-19T12:45:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 1 -- 172.21.15.26:0/3277468938 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb22c004f60 con 0x7fb23413f2e0 2024-08-19T12:45:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb238861640 10 monclient: dump: 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.245 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb23bac7640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb23bac7640 4 set_mon_vals no callback set 2024-08-19T12:45:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb23bac7640 10 monclient: shutdown 2024-08-19T12:45:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.224+0000 7fb23bac7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.225+0000 7fb23bac7640 1 -- 172.21.15.26:0/3277468938 >> v1:172.21.15.26:6793/0 conn(0x7fb23413f2e0 legacy=0x7fb2341416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.225+0000 7fb23bac7640 1 -- 172.21.15.26:0/3277468938 shutdown_connections 2024-08-19T12:45:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.225+0000 7fb23bac7640 1 -- 172.21.15.26:0/3277468938 >> 172.21.15.26:0/3277468938 conn(0x7fb2340abff0 msgr2=0x7fb2340ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.225+0000 7fb23bac7640 1 -- 172.21.15.26:0/3277468938 shutdown_connections 2024-08-19T12:45:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.225+0000 7fb23bac7640 1 -- 172.21.15.26:0/3277468938 wait complete. 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.225+0000 7fb23bac7640 10 monclient: build_initial_monmap 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.225+0000 7fb23bac7640 10 monclient: monmap: 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:08.246 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.225+0000 7fb23bac7640 1 librados: starting msgr at 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.225+0000 7fb23bac7640 1 librados: starting objecter 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 1 Processor -- start 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 1 -- start start 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 1 librados: setting wanted keys 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 1 librados: calling monclient init 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 10 monclient: init 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:08.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 10 monclient: _add_conns ranks=[1,7,3,2,6,4,0,8,5] 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 10 monclient(hunting): picked mon.noname-f con 0x7fb234073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 10 monclient(hunting): picked mon.noname-i con 0x7fb2340a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 10 monclient(hunting): picked mon.noname-g con 0x7fb23413f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb234142b60 con 0x7fb234073470 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb23413f0b0 con 0x7fb23413f2e0 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb234136e30 con 0x7fb2340a57a0 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23bac7640 10 monclient(hunting): authenticate will time out at 1102.314606s 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb239863640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb234073470 0x7fb2341ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:52218/0 (socket says 172.21.15.26:52218) 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb239062640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7fb2340a57a0 0x7fb2341d3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:33838/0 (socket says 172.21.15.26:33838) 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb23a064640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fb23413f2e0 0x7fb2341d6a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:44348/0 (socket says 172.21.15.26:44348) 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.226+0000 7fb239863640 1 -- 172.21.15.26:0/1224161978 learned_addr learned my addr 172.21.15.26:0/1224161978 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:08.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb23a064640 1 -- 172.21.15.26:0/1224161978 >> v1:172.21.15.104:6790/0 conn(0x7fb23413f2e0 legacy=0x7fb2341d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 client.?.objecter ms_handle_connect 0x7fb23413f2e0 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 client.?.objecter ms_handle_connect 0x7fb2340a57a0 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 client.?.objecter ms_handle_connect 0x7fb234073470 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb239062640 1 -- 172.21.15.26:0/1224161978 >> v1:172.21.15.104:6792/0 conn(0x7fb2340a57a0 legacy=0x7fb2341d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb239863640 1 -- 172.21.15.26:0/1224161978 >> v1:172.21.15.104:6789/0 conn(0x7fb234073470 legacy=0x7fb2341ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1502620410 0 0) 0x7fb23413f0b0 con 0x7fb23413f2e0 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): my global_id is 154164 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb218004eb0 con 0x7fb23413f2e0 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 315538082 0 0) 0x7fb234136e30 con 0x7fb2340a57a0 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): my global_id is 135223 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb23413f0b0 con 0x7fb2340a57a0 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543368276 0 0) 0x7fb23409dd50 con 0x7fb234073470 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): my global_id is 27055 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb234136e30 con 0x7fb234073470 2024-08-19T12:45:08.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3815444841 0 0) 0x7fb23413f0b0 con 0x7fb2340a57a0 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb23409dd50 con 0x7fb2340a57a0 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb2240047e0 con 0x7fb2340a57a0 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): dump: 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 692431760 0 0) 0x7fb218004eb0 con 0x7fb23413f2e0 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb23413f0b0 con 0x7fb23413f2e0 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2507217436 0 0) 0x7fb234136e30 con 0x7fb234073470 2024-08-19T12:45:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb218004eb0 con 0x7fb234073470 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb22c004ea0 con 0x7fb23413f2e0 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): dump: 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb230003610 con 0x7fb234073470 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient(hunting): dump: 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2470515868 0 0) 0x7fb23409dd50 con 0x7fb2340a57a0 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 >> v1:172.21.15.104:6790/0 conn(0x7fb23413f2e0 legacy=0x7fb2341d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 >> v1:172.21.15.104:6789/0 conn(0x7fb234073470 legacy=0x7fb2341ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 monclient: found mon.i 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2340a4730 con 0x7fb2340a57a0 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:45:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb23bac7640 5 monclient: authenticate success, global_id 135223 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb23bac7640 10 monclient: _renew_subs 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb23bac7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb23bac7640 1 -- 172.21.15.26:0/1224161978 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb2340a3c50 con 0x7fb2340a57a0 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb23bac7640 10 client.135223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb23bac7640 10 monclient: _renew_subs 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb23bac7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb23bac7640 1 -- 172.21.15.26:0/1224161978 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fb2340a35e0 con 0x7fb2340a57a0 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb23bac7640 1 librados: init done 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb224004c30 con 0x7fb2340a57a0 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb224005120 con 0x7fb2340a57a0 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:45:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22bfff640 4 set_mon_vals no callback set 2024-08-19T12:45:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.229+0000 7fb22a7fc640 10 monclient: dump: 2024-08-19T12:45:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.230+0000 7fb23bac7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.230+0000 7fb23bac7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.230+0000 7fb23bac7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.230+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fb224011580 con 0x7fb2340a57a0 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.230+0000 7fb23bac7640 1 -- 172.21.15.26:0/1224161978 --> v1:172.21.15.104:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb21c005080 con 0x7fb2340a57a0 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.231+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(281..281 src has 1..281) ==== 5364+0+0 (unknown 2673407375 0 0) 0x7fb22404e130 con 0x7fb2340a57a0 2024-08-19T12:45:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.231+0000 7fb22a7fc640 10 client.135223.objecter ms_dispatch 0x7fb234000b90 osd_map(281..281 src has 1..281) 2024-08-19T12:45:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.231+0000 7fb22a7fc640 3 client.135223.objecter handle_osd_map got epochs [281,281] > 0 2024-08-19T12:45:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.231+0000 7fb22a7fc640 3 client.135223.objecter handle_osd_map decoding full epoch 281 2024-08-19T12:45:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.231+0000 7fb22a7fc640 20 client.135223.objecter dump_active .. 0 homeless 2024-08-19T12:45:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.231+0000 7fb239863640 10 monclient: get_auth_request con 0x7fb218041b70 auth_method 0 2024-08-19T12:45:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.233+0000 7fb22a7fc640 10 client.135223.objecter ms_handle_connect 0x7fb218041b70 2024-08-19T12:45:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.235+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.7 v1:172.21.15.104:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fb224017520 con 0x7fb2340a57a0 2024-08-19T12:45:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.235+0000 7fb22a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.235+0000 7fb22a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:08.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644629117}) 2024-08-19T12:45:08.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=870576662}) 2024-08-19T12:45:08.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644629117}) 2024-08-19T12:45:08.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=200845485}) 2024-08-19T12:45:08.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644629117}) 2024-08-19T12:45:08.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1890913456}) 2024-08-19T12:45:08.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644629117}) 2024-08-19T12:45:08.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3245000743}) 2024-08-19T12:45:08.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644629117}) 2024-08-19T12:45:08.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2507303266}) 2024-08-19T12:45:08.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644629117}) 2024-08-19T12:45:08.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=241629077}) 2024-08-19T12:45:08.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644629117}) 2024-08-19T12:45:08.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1670838765}) 2024-08-19T12:45:08.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644629117}) 2024-08-19T12:45:08.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1089943315}) 2024-08-19T12:45:08.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629543711}) 2024-08-19T12:45:08.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1109183362}) 2024-08-19T12:45:08.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629543711}) 2024-08-19T12:45:08.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3418619842}) 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629543711}) 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1810484947}) 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629543711}) 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2693727203}) 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629543711}) 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=619583910}) 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629543711}) 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=477900070}) 2024-08-19T12:45:08.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629543711}) 2024-08-19T12:45:08.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3947175515}) 2024-08-19T12:45:08.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629543711}) 2024-08-19T12:45:08.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.351+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3593577885}) 2024-08-19T12:45:08.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.496+0000 7fb23bac7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:45:08.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.496+0000 7fb23bac7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:45:08.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.496+0000 7fb23bac7640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:45:08.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.496+0000 7fb23bac7640 1 -- 172.21.15.26:0/1224161978 --> v1:172.21.15.104:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb21c005370 con 0x7fb2340a57a0 2024-08-19T12:45:08.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.497+0000 7fb22a7fc640 1 -- 172.21.15.26:0/1224161978 <== mon.7 v1:172.21.15.104:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 2415747349) 0x7fb224016e20 con 0x7fb2340a57a0 2024-08-19T12:45:08.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.497+0000 7fb22a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:45:08.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.497+0000 7fb22a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:08.499 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":42,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.499+0000 7fb23bac7640 10 librados: watch_flush enter 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.499+0000 7fb23bac7640 10 librados: watch_flush exit 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.499+0000 7fb23bac7640 20 client.135223.objecter shutdown clearing up homeless session... 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.499+0000 7fb23bac7640 10 client.135223.objecter successfully canceled tick 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.499+0000 7fb23bac7640 1 -- 172.21.15.26:0/1224161978 >> v1:172.21.15.104:6800/899887506 conn(0x7fb218041b70 legacy=0x7fb218044030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.499+0000 7fb23bac7640 10 monclient: shutdown 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.499+0000 7fb23bac7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.499+0000 7fb23bac7640 1 -- 172.21.15.26:0/1224161978 >> v1:172.21.15.104:6792/0 conn(0x7fb2340a57a0 legacy=0x7fb2341d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.499+0000 7fb23bac7640 1 -- 172.21.15.26:0/1224161978 shutdown_connections 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.499+0000 7fb23bac7640 1 -- 172.21.15.26:0/1224161978 >> 172.21.15.26:0/1224161978 conn(0x7fb2340abff0 msgr2=0x7fb234134910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.499+0000 7fb23bac7640 1 -- 172.21.15.26:0/1224161978 shutdown_connections 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.499+0000 7fb23bac7640 1 -- 172.21.15.26:0/1224161978 wait complete. 2024-08-19T12:45:08.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.500+0000 7fb23bac7640 1 librados: shutdown 2024-08-19T12:45:08.513 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:45:08.513 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T12:45:08.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:45:08.513 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:45:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.611+0000 7f1f94141640 10 monclient: get_monmap_and_config 2024-08-19T12:45:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.611+0000 7f1f94141640 10 monclient: build_initial_monmap 2024-08-19T12:45:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.611+0000 7f1f94141640 10 monclient: monmap: 2024-08-19T12:45:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:08.613361+0000 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:08.613361+0000 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:08.615 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 1 Processor -- start 2024-08-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 1 -- start start 2024-08-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 10 monclient: init 2024-08-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 10 monclient: _add_conns ranks=[3,1,6,7,8,5,2,4,0] 2024-08-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 10 monclient(hunting): picked mon.noname-g con 0x7f1f8c06e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 10 monclient(hunting): picked mon.noname-f con 0x7f1f8c0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 10 monclient(hunting): picked mon.noname-d con 0x7f1f8c0a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c136790 con 0x7f1f8c0a5920 2024-08-19T12:45:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c09ddf0 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c09e0d0 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f94141640 10 monclient(hunting): authenticate will time out at 1102.700821s 2024-08-19T12:45:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f926de640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f1f8c0a5d50 0x7f1f8c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49884/0 (socket says 172.21.15.26:49884) 2024-08-19T12:45:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.612+0000 7f1f926de640 1 -- 172.21.15.26:0/376677389 learned_addr learned my addr 172.21.15.26:0/376677389 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f926de640 1 -- 172.21.15.26:0/376677389 >> v1:172.21.15.26:6792/0 conn(0x7f1f8c0a5d50 legacy=0x7f1f8c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 1 -- 172.21.15.26:0/376677389 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3490267739 0 0) 0x7f1f8c09e0d0 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient(hunting): my global_id is 26619 2024-08-19T12:45:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 1 -- 172.21.15.26:0/376677389 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f74003520 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f916dc640 1 -- 172.21.15.26:0/376677389 >> v1:172.21.15.104:6789/0 conn(0x7f1f8c0a5920 legacy=0x7f1f8c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f91edd640 1 -- 172.21.15.26:0/376677389 >> v1:172.21.15.104:6790/0 conn(0x7f1f8c06e7b0 legacy=0x7f1f8c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 1 -- 172.21.15.26:0/376677389 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2097121746 0 0) 0x7f1f74003520 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 1 -- 172.21.15.26:0/376677389 >> v1:172.21.15.104:6790/0 conn(0x7f1f8c06e7b0 legacy=0x7f1f8c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 1 -- 172.21.15.26:0/376677389 >> v1:172.21.15.104:6789/0 conn(0x7f1f8c0a5920 legacy=0x7f1f8c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient: _finish_hunting 0 2024-08-19T12:45:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 1 monclient: found mon.noname-d 2024-08-19T12:45:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:45:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 1 -- 172.21.15.26:0/376677389 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c09e3b0 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient: _finish_auth 0 2024-08-19T12:45:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient: _check_auth_tickets 2024-08-19T12:45:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 1 -- 172.21.15.26:0/376677389 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1f7c001ae0 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:45:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient: dump: 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f94141640 5 monclient: authenticate success, global_id 26619 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f94141640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 1 -- 172.21.15.26:0/376677389 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1f7c0048a0 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 1 -- 172.21.15.26:0/376677389 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1f7c004e10 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f94141640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f94141640 4 set_mon_vals no callback set 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.615+0000 7f1f90edb640 10 monclient: dump: 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.629 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.616+0000 7f1f94141640 10 monclient: shutdown 2024-08-19T12:45:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.616+0000 7f1f94141640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.616+0000 7f1f94141640 1 -- 172.21.15.26:0/376677389 >> v1:172.21.15.26:6792/0 conn(0x7f1f8c0a5d50 legacy=0x7f1f8c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.616+0000 7f1f94141640 1 -- 172.21.15.26:0/376677389 shutdown_connections 2024-08-19T12:45:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.616+0000 7f1f94141640 1 -- 172.21.15.26:0/376677389 >> 172.21.15.26:0/376677389 conn(0x7f1f8c0a8740 msgr2=0x7f1f8c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.616+0000 7f1f94141640 1 -- 172.21.15.26:0/376677389 shutdown_connections 2024-08-19T12:45:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.616+0000 7f1f94141640 1 -- 172.21.15.26:0/376677389 wait complete. 2024-08-19T12:45:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.616+0000 7f1f94141640 10 monclient: build_initial_monmap 2024-08-19T12:45:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.616+0000 7f1f94141640 10 monclient: monmap: 2024-08-19T12:45:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.616+0000 7f1f94141640 1 librados: starting msgr at 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.616+0000 7f1f94141640 1 librados: starting objecter 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 1 Processor -- start 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 1 -- start start 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 1 librados: setting wanted keys 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 1 librados: calling monclient init 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 10 monclient: init 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 10 monclient: _add_conns ranks=[6,2,0,7,8,4,1,5,3] 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 10 monclient(hunting): picked mon.noname-d con 0x7f1f8c06e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 10 monclient(hunting): picked mon.noname-b con 0x7f1f8c0a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8c0a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c134980 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c1345c0 con 0x7f1f8c0a5920 2024-08-19T12:45:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f8c1382e0 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f94141640 10 monclient(hunting): authenticate will time out at 1102.705532s 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f91edd640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f1f8c06e7b0 0x7f1f8c1374c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:49890/0 (socket says 172.21.15.26:49890) 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.617+0000 7f1f91edd640 1 -- 172.21.15.26:0/693370172 learned_addr learned my addr 172.21.15.26:0/693370172 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f91edd640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.26:6792/0 conn(0x7f1f8c06e7b0 legacy=0x7f1f8c1374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 client.?.objecter ms_handle_connect 0x7f1f8c06e7b0 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 client.?.objecter ms_handle_connect 0x7f1f8c0a5d50 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f926de640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.26:6789/0 conn(0x7f1f8c0a5d50 legacy=0x7f1f8c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f916dc640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.26:6790/0 conn(0x7f1f8c0a5920 legacy=0x7f1f8c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 client.?.objecter ms_handle_connect 0x7f1f8c0a5920 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3602827066 0 0) 0x7f1f8c1382e0 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 monclient(hunting): my global_id is 26628 2024-08-19T12:45:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f78004eb0 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3000037095 0 0) 0x7f1f8c09e3b0 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 monclient(hunting): my global_id is 144513 2024-08-19T12:45:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f8c1382e0 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4212453554 0 0) 0x7f1f8c136790 con 0x7f1f8c0a5920 2024-08-19T12:45:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 10 monclient(hunting): my global_id is 26273 2024-08-19T12:45:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.619+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f8c09e3b0 con 0x7f1f8c0a5920 2024-08-19T12:45:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2746139084 0 0) 0x7f1f78004eb0 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f8c136790 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1f84003610 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:45:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient(hunting): dump: 2024-08-19T12:45:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.637 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.637 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.637 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.637 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.637 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.637 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.637 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.637 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.637 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3801208985 0 0) 0x7f1f8c09e3b0 con 0x7f1f8c0a5920 2024-08-19T12:45:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f78004eb0 con 0x7f1f8c0a5920 2024-08-19T12:45:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2763557783 0 0) 0x7f1f8c1382e0 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f8c09e3b0 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1f80002090 con 0x7f1f8c0a5920 2024-08-19T12:45:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient(hunting): dump: 2024-08-19T12:45:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.640 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1f7c001d30 con 0x7f1f8c0a5d50 2024-08-19T12:45:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient(hunting): dump: 2024-08-19T12:45:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.642 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.642 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.642 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.642 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1855797323 0 0) 0x7f1f8c136790 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.26:6790/0 conn(0x7f1f8c0a5920 legacy=0x7f1f8c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.26:6789/0 conn(0x7f1f8c0a5d50 legacy=0x7f1f8c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:45:08.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 monclient: found mon.d 2024-08-19T12:45:08.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:08.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c13ae60 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:45:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:45:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f94141640 5 monclient: authenticate success, global_id 26628 2024-08-19T12:45:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f94141640 10 monclient: _renew_subs 2024-08-19T12:45:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f94141640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f94141640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1f8c13b140 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f94141640 10 client.26628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f94141640 10 monclient: _renew_subs 2024-08-19T12:45:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f94141640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f94141640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f1f8c1428b0 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f94141640 1 librados: init done 2024-08-19T12:45:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1f84003b90 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1f84004100 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8bfff640 4 set_mon_vals no callback set 2024-08-19T12:45:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.620+0000 7f1f8a7fc640 10 monclient: dump: 2024-08-19T12:45:08.644 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.645 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.621+0000 7f1f94141640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.621+0000 7f1f94141640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1f6c0065c0 con 0x7f1f6c002860 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.621+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f1f840042a0 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.622+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(281..281 src has 1..281) ==== 5364+0+0 (unknown 2673407375 0 0) 0x7f1f8404e1b0 con 0x7f1f8c06e7b0 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.622+0000 7f1f8a7fc640 10 client.26628.objecter ms_dispatch 0x7f1f8c000b90 osd_map(281..281 src has 1..281) 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.622+0000 7f1f8a7fc640 3 client.26628.objecter handle_osd_map got epochs [281,281] > 0 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.622+0000 7f1f8a7fc640 3 client.26628.objecter handle_osd_map decoding full epoch 281 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.622+0000 7f1f8a7fc640 20 client.26628.objecter dump_active .. 0 homeless 2024-08-19T12:45:08.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.622+0000 7f1f926de640 10 monclient: get_auth_request con 0x7f1f78041b20 auth_method 0 2024-08-19T12:45:08.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.623+0000 7f1f916dc640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.26:6789/0 conn(0x7f1f6c002860 legacy=0x7f1f6c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.623+0000 7f1f8a7fc640 10 client.26628.objecter ms_handle_connect 0x7f1f6c002860 2024-08-19T12:45:08.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.623+0000 7f1f8a7fc640 10 client.26628.objecter resend_mon_ops 2024-08-19T12:45:08.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.623+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 217312975 0 0) 0x7f1f6c0065c0 con 0x7f1f6c002860 2024-08-19T12:45:08.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.623+0000 7f1f8a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.623+0000 7f1f8a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.623+0000 7f1f8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.623+0000 7f1f8a7fc640 10 monclient(hunting): my global_id is 144522 2024-08-19T12:45:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.623+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f8c136790 con 0x7f1f6c002860 2024-08-19T12:45:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 260436876 0 0) 0x7f1f8c136790 con 0x7f1f6c002860 2024-08-19T12:45:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f6c0065c0 con 0x7f1f6c002860 2024-08-19T12:45:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1f80004310 con 0x7f1f6c002860 2024-08-19T12:45:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 10 monclient: dump: 2024-08-19T12:45:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.651 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.652 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.652 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.652 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.652 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.652 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.652 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.652 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.652 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.652 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.652 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.652 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.652 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2496289796 0 0) 0x7f1f6c0065c0 con 0x7f1f6c002860 2024-08-19T12:45:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f6c00a160 con 0x7f1f6c002860 2024-08-19T12:45:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1f800045e0 con 0x7f1f6c002860 2024-08-19T12:45:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.26:6789/0 conn(0x7f1f6c002860 legacy=0x7f1f6c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.26:6789/0 conn(0x7f1f6c002860 legacy=0x7f1f6c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.624+0000 7f1f8a7fc640 10 client.26628.objecter ms_handle_connect 0x7f1f78041b20 2024-08-19T12:45:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.630+0000 7f1f94141640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.630+0000 7f1f94141640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1f6c009e50 con 0x7f1f6c009110 2024-08-19T12:45:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.632+0000 7f1f916dc640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.26:6789/0 conn(0x7f1f6c009110 legacy=0x7f1f6c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.632+0000 7f1f8a7fc640 10 client.26628.objecter ms_handle_connect 0x7f1f6c009110 2024-08-19T12:45:08.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.632+0000 7f1f8a7fc640 10 client.26628.objecter resend_mon_ops 2024-08-19T12:45:08.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1273760394 0 0) 0x7f1f6c009e50 con 0x7f1f6c009110 2024-08-19T12:45:08.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 10 monclient(hunting): my global_id is 144531 2024-08-19T12:45:08.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f6c0065c0 con 0x7f1f6c009110 2024-08-19T12:45:08.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 187620433 0 0) 0x7f1f6c0065c0 con 0x7f1f6c009110 2024-08-19T12:45:08.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1f6c009e50 con 0x7f1f6c009110 2024-08-19T12:45:08.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1f80006070 con 0x7f1f6c009110 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 10 monclient: dump: 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.656 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.656 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.656 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.656 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.656 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.656 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2834623651 0 0) 0x7f1f6c009e50 con 0x7f1f6c009110 2024-08-19T12:45:08.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1f6c00c3a0 con 0x7f1f6c009110 2024-08-19T12:45:08.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7713 (unknown 0 0 988633387) 0x7f1f800082f0 con 0x7f1f6c009110 2024-08-19T12:45:08.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:08.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:08.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.26:6789/0 conn(0x7f1f6c009110 legacy=0x7f1f6c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f8a7fc640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.26:6789/0 conn(0x7f1f6c009110 legacy=0x7f1f6c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:08.657 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:45:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 160551, 2024-08-19T12:45:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:45:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:08.665 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.666 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.667 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.669 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:08.672 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:08.674 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.676 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.677 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.678 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:08.678 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:08.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f94141640 10 librados: watch_flush enter 2024-08-19T12:45:08.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f94141640 10 librados: watch_flush exit 2024-08-19T12:45:08.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f94141640 20 client.26628.objecter shutdown clearing up homeless session... 2024-08-19T12:45:08.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f94141640 10 client.26628.objecter successfully canceled tick 2024-08-19T12:45:08.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f94141640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.104:6800/899887506 conn(0x7f1f78041b20 legacy=0x7f1f78043fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f94141640 10 monclient: shutdown 2024-08-19T12:45:08.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f94141640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:08.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f94141640 1 -- 172.21.15.26:0/693370172 >> v1:172.21.15.26:6792/0 conn(0x7f1f8c06e7b0 legacy=0x7f1f8c1374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f926de640 1 -- 172.21.15.26:0/693370172 reap_dead start 2024-08-19T12:45:08.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f926de640 1 -- 172.21.15.26:0/693370172 reap_dead start 2024-08-19T12:45:08.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f94141640 1 -- 172.21.15.26:0/693370172 shutdown_connections 2024-08-19T12:45:08.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.633+0000 7f1f94141640 1 -- 172.21.15.26:0/693370172 >> 172.21.15.26:0/693370172 conn(0x7f1f8c0a8740 msgr2=0x7f1f8c06dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:08.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.634+0000 7f1f94141640 1 -- 172.21.15.26:0/693370172 shutdown_connections 2024-08-19T12:45:08.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.634+0000 7f1f94141640 1 -- 172.21.15.26:0/693370172 wait complete. 2024-08-19T12:45:08.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.634+0000 7f1f94141640 1 librados: shutdown 2024-08-19T12:45:08.682 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:08.682 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:45:08.682 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:45:08.698 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.698 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830483323}) 2024-08-19T12:45:08.698 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=627071153}) 2024-08-19T12:45:08.698 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.698 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830483323}) 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1971094116}) 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830483323}) 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1468157275}) 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830483323}) 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1646979921}) 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830483323}) 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2651034212}) 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830483323}) 2024-08-19T12:45:08.699 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=257443603}) 2024-08-19T12:45:08.700 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.700 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830483323}) 2024-08-19T12:45:08.700 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1136611197}) 2024-08-19T12:45:08.700 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.700 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830483323}) 2024-08-19T12:45:08.700 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.654+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3282198427}) 2024-08-19T12:45:08.700 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.700 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3743848578}) 2024-08-19T12:45:08.700 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4096275386}) 2024-08-19T12:45:08.700 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.700 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3743848578}) 2024-08-19T12:45:08.700 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=351925941}) 2024-08-19T12:45:08.701 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.701 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3743848578}) 2024-08-19T12:45:08.701 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3318129946}) 2024-08-19T12:45:08.701 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.701 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3743848578}) 2024-08-19T12:45:08.701 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2866428006}) 2024-08-19T12:45:08.701 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.701 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3743848578}) 2024-08-19T12:45:08.701 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4141774744}) 2024-08-19T12:45:08.701 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.701 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3743848578}) 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2941119807}) 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3743848578}) 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2770192186}) 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3743848578}) 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.655+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=164226261}) 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917916636}) 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3360642218}) 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917916636}) 2024-08-19T12:45:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=494875312}) 2024-08-19T12:45:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917916636}) 2024-08-19T12:45:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3010373802}) 2024-08-19T12:45:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917916636}) 2024-08-19T12:45:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=419424458}) 2024-08-19T12:45:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917916636}) 2024-08-19T12:45:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=314672346}) 2024-08-19T12:45:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917916636}) 2024-08-19T12:45:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4033770161}) 2024-08-19T12:45:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917916636}) 2024-08-19T12:45:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=832833548}) 2024-08-19T12:45:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917916636}) 2024-08-19T12:45:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.656+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3156333802}) 2024-08-19T12:45:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437024466}) 2024-08-19T12:45:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2746699338}) 2024-08-19T12:45:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437024466}) 2024-08-19T12:45:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010855799}) 2024-08-19T12:45:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437024466}) 2024-08-19T12:45:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=58584516}) 2024-08-19T12:45:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437024466}) 2024-08-19T12:45:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1461382356}) 2024-08-19T12:45:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437024466}) 2024-08-19T12:45:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2220625779}) 2024-08-19T12:45:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437024466}) 2024-08-19T12:45:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=715388625}) 2024-08-19T12:45:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.657+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.658+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437024466}) 2024-08-19T12:45:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.658+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2192279619}) 2024-08-19T12:45:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.658+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.658+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437024466}) 2024-08-19T12:45:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.658+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1277613772}) 2024-08-19T12:45:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129039213}) 2024-08-19T12:45:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=325753404}) 2024-08-19T12:45:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129039213}) 2024-08-19T12:45:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2194477466}) 2024-08-19T12:45:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129039213}) 2024-08-19T12:45:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=410546901}) 2024-08-19T12:45:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129039213}) 2024-08-19T12:45:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1352581859}) 2024-08-19T12:45:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129039213}) 2024-08-19T12:45:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3482224628}) 2024-08-19T12:45:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129039213}) 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1625840308}) 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129039213}) 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2113479705}) 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129039213}) 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.659+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=18073822}) 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285320052}) 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2971366776}) 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285320052}) 2024-08-19T12:45:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3919195797}) 2024-08-19T12:45:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285320052}) 2024-08-19T12:45:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3526246251}) 2024-08-19T12:45:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285320052}) 2024-08-19T12:45:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1626520194}) 2024-08-19T12:45:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285320052}) 2024-08-19T12:45:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=681736464}) 2024-08-19T12:45:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285320052}) 2024-08-19T12:45:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=540101299}) 2024-08-19T12:45:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285320052}) 2024-08-19T12:45:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1396937915}) 2024-08-19T12:45:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285320052}) 2024-08-19T12:45:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.660+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2257101507}) 2024-08-19T12:45:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598037680}) 2024-08-19T12:45:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3403838754}) 2024-08-19T12:45:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598037680}) 2024-08-19T12:45:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1093219615}) 2024-08-19T12:45:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598037680}) 2024-08-19T12:45:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4097634424}) 2024-08-19T12:45:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598037680}) 2024-08-19T12:45:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=720767246}) 2024-08-19T12:45:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598037680}) 2024-08-19T12:45:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=992785459}) 2024-08-19T12:45:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598037680}) 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4026328808}) 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598037680}) 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3779762797}) 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598037680}) 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.661+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3737781444}) 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583451944}) 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2999195750}) 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583451944}) 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2683881934}) 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583451944}) 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2786988065}) 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583451944}) 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1317909472}) 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.662+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583451944}) 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.663+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=362866727}) 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.663+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.663+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583451944}) 2024-08-19T12:45:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.663+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2891484210}) 2024-08-19T12:45:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.663+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.663+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583451944}) 2024-08-19T12:45:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.663+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=431870049}) 2024-08-19T12:45:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.663+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.663+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583451944}) 2024-08-19T12:45:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.663+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3915651991}) 2024-08-19T12:45:08.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077075916}) 2024-08-19T12:45:08.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3951707471}) 2024-08-19T12:45:08.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077075916}) 2024-08-19T12:45:08.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=430772157}) 2024-08-19T12:45:08.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077075916}) 2024-08-19T12:45:08.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3243803661}) 2024-08-19T12:45:08.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077075916}) 2024-08-19T12:45:08.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1881974466}) 2024-08-19T12:45:08.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077075916}) 2024-08-19T12:45:08.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1547159598}) 2024-08-19T12:45:08.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077075916}) 2024-08-19T12:45:08.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2609268491}) 2024-08-19T12:45:08.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077075916}) 2024-08-19T12:45:08.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4138331841}) 2024-08-19T12:45:08.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077075916}) 2024-08-19T12:45:08.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2686530199}) 2024-08-19T12:45:08.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715259569}) 2024-08-19T12:45:08.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778350317}) 2024-08-19T12:45:08.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715259569}) 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3727603301}) 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715259569}) 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1504475750}) 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715259569}) 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1639006369}) 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715259569}) 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1109443994}) 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.725+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.725+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715259569}) 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.725+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2157230737}) 2024-08-19T12:45:08.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.725+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.725+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715259569}) 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.725+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1099266919}) 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.725+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.725+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715259569}) 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.725+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1695437544}) 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265510646}) 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1783609287}) 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265510646}) 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3888172652}) 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265510646}) 2024-08-19T12:45:08.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033323915}) 2024-08-19T12:45:08.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265510646}) 2024-08-19T12:45:08.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1825849320}) 2024-08-19T12:45:08.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265510646}) 2024-08-19T12:45:08.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2097788708}) 2024-08-19T12:45:08.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265510646}) 2024-08-19T12:45:08.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1422006042}) 2024-08-19T12:45:08.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265510646}) 2024-08-19T12:45:08.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2450649204}) 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265510646}) 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1806894811}) 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229271478}) 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313005767}) 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229271478}) 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=159477176}) 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229271478}) 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4255742218}) 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229271478}) 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1609022614}) 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229271478}) 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=469706559}) 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229271478}) 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3816701441}) 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229271478}) 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3934033286}) 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229271478}) 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.727+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3692275009}) 2024-08-19T12:45:08.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592222821}) 2024-08-19T12:45:08.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2329343105}) 2024-08-19T12:45:08.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592222821}) 2024-08-19T12:45:08.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219080556}) 2024-08-19T12:45:08.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592222821}) 2024-08-19T12:45:08.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4063844306}) 2024-08-19T12:45:08.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592222821}) 2024-08-19T12:45:08.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2525864964}) 2024-08-19T12:45:08.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592222821}) 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3990025221}) 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592222821}) 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3622847205}) 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592222821}) 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4226933015}) 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592222821}) 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2353295539}) 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833415147}) 2024-08-19T12:45:08.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1341556575}) 2024-08-19T12:45:08.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833415147}) 2024-08-19T12:45:08.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2417687324}) 2024-08-19T12:45:08.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833415147}) 2024-08-19T12:45:08.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2013695981}) 2024-08-19T12:45:08.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833415147}) 2024-08-19T12:45:08.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1898507099}) 2024-08-19T12:45:08.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833415147}) 2024-08-19T12:45:08.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3896162842}) 2024-08-19T12:45:08.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.729+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833415147}) 2024-08-19T12:45:08.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=306454226}) 2024-08-19T12:45:08.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833415147}) 2024-08-19T12:45:08.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1152675893}) 2024-08-19T12:45:08.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833415147}) 2024-08-19T12:45:08.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=183186229}) 2024-08-19T12:45:08.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893858409}) 2024-08-19T12:45:08.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050595897}) 2024-08-19T12:45:08.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893858409}) 2024-08-19T12:45:08.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2778989120}) 2024-08-19T12:45:08.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893858409}) 2024-08-19T12:45:08.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2811305531}) 2024-08-19T12:45:08.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893858409}) 2024-08-19T12:45:08.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2054396717}) 2024-08-19T12:45:08.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893858409}) 2024-08-19T12:45:08.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1837476137}) 2024-08-19T12:45:08.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893858409}) 2024-08-19T12:45:08.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=976682076}) 2024-08-19T12:45:08.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893858409}) 2024-08-19T12:45:08.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2021340599}) 2024-08-19T12:45:08.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1893858409}) 2024-08-19T12:45:08.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.731+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3746501867}) 2024-08-19T12:45:08.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950671713}) 2024-08-19T12:45:08.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1260140348}) 2024-08-19T12:45:08.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950671713}) 2024-08-19T12:45:08.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541600569}) 2024-08-19T12:45:08.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950671713}) 2024-08-19T12:45:08.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1909411879}) 2024-08-19T12:45:08.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950671713}) 2024-08-19T12:45:08.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1440750420}) 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950671713}) 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=762967709}) 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950671713}) 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3012594640}) 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950671713}) 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3360478735}) 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950671713}) 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2113683148}) 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631721102}) 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1721567804}) 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631721102}) 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688978667}) 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631721102}) 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=475715396}) 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631721102}) 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3747457517}) 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631721102}) 2024-08-19T12:45:08.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1416160275}) 2024-08-19T12:45:08.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631721102}) 2024-08-19T12:45:08.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1478114395}) 2024-08-19T12:45:08.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631721102}) 2024-08-19T12:45:08.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1489672810}) 2024-08-19T12:45:08.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631721102}) 2024-08-19T12:45:08.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.733+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=864416701}) 2024-08-19T12:45:08.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901628532}) 2024-08-19T12:45:08.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=604473646}) 2024-08-19T12:45:08.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901628532}) 2024-08-19T12:45:08.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1000035121}) 2024-08-19T12:45:08.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901628532}) 2024-08-19T12:45:08.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2249023387}) 2024-08-19T12:45:08.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901628532}) 2024-08-19T12:45:08.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3761134258}) 2024-08-19T12:45:08.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901628532}) 2024-08-19T12:45:08.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2937563864}) 2024-08-19T12:45:08.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901628532}) 2024-08-19T12:45:08.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4213532611}) 2024-08-19T12:45:08.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901628532}) 2024-08-19T12:45:08.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1071651949}) 2024-08-19T12:45:08.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901628532}) 2024-08-19T12:45:08.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.735+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3459250860}) 2024-08-19T12:45:08.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856648410}) 2024-08-19T12:45:08.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1676010907}) 2024-08-19T12:45:08.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856648410}) 2024-08-19T12:45:08.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1239569232}) 2024-08-19T12:45:08.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856648410}) 2024-08-19T12:45:08.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2755070235}) 2024-08-19T12:45:08.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856648410}) 2024-08-19T12:45:08.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1975441397}) 2024-08-19T12:45:08.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856648410}) 2024-08-19T12:45:08.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=453961349}) 2024-08-19T12:45:08.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856648410}) 2024-08-19T12:45:08.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2928088197}) 2024-08-19T12:45:08.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856648410}) 2024-08-19T12:45:08.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3561296048}) 2024-08-19T12:45:08.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856648410}) 2024-08-19T12:45:08.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2561385319}) 2024-08-19T12:45:08.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589732777}) 2024-08-19T12:45:08.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816069906}) 2024-08-19T12:45:08.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589732777}) 2024-08-19T12:45:08.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688990593}) 2024-08-19T12:45:08.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589732777}) 2024-08-19T12:45:08.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2475118932}) 2024-08-19T12:45:08.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.764 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589732777}) 2024-08-19T12:45:08.764 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2023204518}) 2024-08-19T12:45:08.764 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.764 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589732777}) 2024-08-19T12:45:08.764 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3582732027}) 2024-08-19T12:45:08.764 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.764 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589732777}) 2024-08-19T12:45:08.764 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1787529255}) 2024-08-19T12:45:08.764 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.764 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589732777}) 2024-08-19T12:45:08.764 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1854171057}) 2024-08-19T12:45:08.764 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589732777}) 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=402187566}) 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946332076}) 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=520049813}) 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946332076}) 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2496738503}) 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946332076}) 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1188632516}) 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.765 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946332076}) 2024-08-19T12:45:08.766 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1743052125}) 2024-08-19T12:45:08.766 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.766 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946332076}) 2024-08-19T12:45:08.766 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=670712050}) 2024-08-19T12:45:08.766 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.766 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946332076}) 2024-08-19T12:45:08.766 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3282345626}) 2024-08-19T12:45:08.766 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.766 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946332076}) 2024-08-19T12:45:08.766 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1339726256}) 2024-08-19T12:45:08.766 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.766 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946332076}) 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3616733219}) 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848856195}) 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2354062849}) 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848856195}) 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3764485631}) 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848856195}) 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2933963763}) 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848856195}) 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3081390608}) 2024-08-19T12:45:08.767 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.768 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848856195}) 2024-08-19T12:45:08.768 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3005337985}) 2024-08-19T12:45:08.768 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.768 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848856195}) 2024-08-19T12:45:08.768 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4286169498}) 2024-08-19T12:45:08.768 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.768 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848856195}) 2024-08-19T12:45:08.768 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=609837517}) 2024-08-19T12:45:08.768 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.768 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848856195}) 2024-08-19T12:45:08.768 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.748+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=237149804}) 2024-08-19T12:45:08.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287102478}) 2024-08-19T12:45:08.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2366507539}) 2024-08-19T12:45:08.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287102478}) 2024-08-19T12:45:08.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=690069536}) 2024-08-19T12:45:08.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287102478}) 2024-08-19T12:45:08.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1878641950}) 2024-08-19T12:45:08.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287102478}) 2024-08-19T12:45:08.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2917271574}) 2024-08-19T12:45:08.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287102478}) 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1579068552}) 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287102478}) 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4051328995}) 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287102478}) 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=263292292}) 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4287102478}) 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3087100509}) 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573339904}) 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2999244324}) 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573339904}) 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1479405965}) 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573339904}) 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=214911120}) 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573339904}) 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4285782102}) 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573339904}) 2024-08-19T12:45:08.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2711282274}) 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573339904}) 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1322388344}) 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573339904}) 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=105949173}) 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573339904}) 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3743196364}) 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728326899}) 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=543184875}) 2024-08-19T12:45:08.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728326899}) 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280555886}) 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728326899}) 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1824390166}) 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728326899}) 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2932256294}) 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728326899}) 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3482151208}) 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728326899}) 2024-08-19T12:45:08.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2581387314}) 2024-08-19T12:45:08.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728326899}) 2024-08-19T12:45:08.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2195224381}) 2024-08-19T12:45:08.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728326899}) 2024-08-19T12:45:08.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.763+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2881209463}) 2024-08-19T12:45:08.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196175843}) 2024-08-19T12:45:08.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1769828723}) 2024-08-19T12:45:08.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196175843}) 2024-08-19T12:45:08.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2378705648}) 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196175843}) 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1979144183}) 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196175843}) 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1224107517}) 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196175843}) 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2149640905}) 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196175843}) 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3416497645}) 2024-08-19T12:45:08.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196175843}) 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=618049921}) 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196175843}) 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.765+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2987734521}) 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2716387228}) 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1825551258}) 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2716387228}) 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3271424813}) 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.797 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2716387228}) 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=474227196}) 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2716387228}) 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2132115075}) 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2716387228}) 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3273520321}) 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2716387228}) 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3127109470}) 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2716387228}) 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2066336044}) 2024-08-19T12:45:08.798 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2716387228}) 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.766+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3890162754}) 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.767+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.767+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3803579366}) 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.767+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1378106237}) 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.767+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.767+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3803579366}) 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.767+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4070118887}) 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.767+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.767+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3803579366}) 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.767+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=230386522}) 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3803579366}) 2024-08-19T12:45:08.799 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=224267265}) 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3803579366}) 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=279620149}) 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3803579366}) 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2971143181}) 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3803579366}) 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1903930125}) 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3803579366}) 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.768+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3643343408}) 2024-08-19T12:45:08.800 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382075300}) 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3740416460}) 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382075300}) 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1007776495}) 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382075300}) 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1351688057}) 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382075300}) 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3202714908}) 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382075300}) 2024-08-19T12:45:08.801 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3440201968}) 2024-08-19T12:45:08.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382075300}) 2024-08-19T12:45:08.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=31197654}) 2024-08-19T12:45:08.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382075300}) 2024-08-19T12:45:08.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=887919361}) 2024-08-19T12:45:08.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382075300}) 2024-08-19T12:45:08.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.769+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3374588672}) 2024-08-19T12:45:08.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680288190}) 2024-08-19T12:45:08.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3675286830}) 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680288190}) 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3757788101}) 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680288190}) 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3431674484}) 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680288190}) 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3418954346}) 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680288190}) 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2936018387}) 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680288190}) 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1898219338}) 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680288190}) 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=755777717}) 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680288190}) 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.770+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3352618523}) 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138185839}) 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=998250922}) 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138185839}) 2024-08-19T12:45:08.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3332969840}) 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138185839}) 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=421855143}) 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138185839}) 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2921119926}) 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138185839}) 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1560176822}) 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138185839}) 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2504695430}) 2024-08-19T12:45:08.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138185839}) 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2377513204}) 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138185839}) 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=524149771}) 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3522396175}) 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3830135594}) 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3522396175}) 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2096251066}) 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3522396175}) 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=546086551}) 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3522396175}) 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=921438115}) 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3522396175}) 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=239137452}) 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3522396175}) 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3517074141}) 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3522396175}) 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=116893693}) 2024-08-19T12:45:08.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3522396175}) 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2298014470}) 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511803806}) 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544257360}) 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511803806}) 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4176235927}) 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511803806}) 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1580616012}) 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511803806}) 2024-08-19T12:45:08.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2831651294}) 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511803806}) 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3725091315}) 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511803806}) 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3245678206}) 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511803806}) 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=435577376}) 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511803806}) 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3208230955}) 2024-08-19T12:45:08.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=606871678,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2756268331}) 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=2137838063,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=4285519102}) 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=606871678,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2756268331}) 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=3731585149,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2826520744}) 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=606871678,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2756268331}) 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=78,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=3} crc {logm=636408789,mdsmap=182599002,mgr=3738928481,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2756268331,mon_config_key=1564873177}) 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=606871678,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2756268331}) 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=80,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=131397519,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=33987383}) 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.810 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=606871678,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2756268331}) 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=2397224978,mdsmap=3707151070,mgr=1806791599,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=4163806741}) 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=606871678,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2756268331}) 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=84,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=3462420018,mdsmap=182599002,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=1614155652}) 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=606871678,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2756268331}) 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=80,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=1541874325,mdsmap=3707151070,mgr=3553849969,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=33987383}) 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=606871678,mdsmap=3707151070,mgr=3314961562,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2756268331}) 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=80,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=1} crc {logm=3113292256,mdsmap=182599002,mgr=963350020,mgr_command_descs=3626727126,mgr_metadata=426268268,mgrstat=2756268331,mon_config_key=1967427191}) 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=45} crc {mon_config_key=2323740666,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=1700087648,osdmap=891391437}) 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=19,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=46} crc {mon_config_key=2735668574,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=3421697156,osdmap=2971600474}) 2024-08-19T12:45:08.811 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=45} crc {mon_config_key=2323740666,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=1700087648,osdmap=891391437}) 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=45} crc {mon_config_key=2171442413,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=2940424483,osdmap=3357105035}) 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=45} crc {mon_config_key=2323740666,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=1700087648,osdmap=891391437}) 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=44} crc {mon_config_key=2171442413,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=2835486546,osdmap=241774659}) 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=45} crc {mon_config_key=2323740666,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=1700087648,osdmap=891391437}) 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=45} crc {mon_config_key=2171442413,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=3635537877,osdmap=1470307326}) 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=45} crc {mon_config_key=2323740666,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=1700087648,osdmap=891391437}) 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {mon_config_key=18,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=47} crc {mon_config_key=2340938430,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=3862563611,osdmap=2208148283}) 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=45} crc {mon_config_key=2323740666,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=1700087648,osdmap=891391437}) 2024-08-19T12:45:08.812 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {mon_config_key=20,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=25,osdmap=47} crc {mon_config_key=2171442413,monmap=2932333919,osd_metadata=2768981642,osd_pg_creating=761603184,osd_snap=4275584257,osdmap=3222985185}) 2024-08-19T12:45:08.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=45} crc {mon_config_key=2323740666,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=1700087648,osdmap=891391437}) 2024-08-19T12:45:08.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=25,osdmap=46} crc {mon_config_key=2171442413,monmap=1261126655,osd_metadata=438114253,osd_pg_creating=761603184,osd_snap=1178575182,osdmap=2442840499}) 2024-08-19T12:45:08.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=45} crc {mon_config_key=2323740666,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=1700087648,osdmap=891391437}) 2024-08-19T12:45:08.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25,osdmap=45} crc {mon_config_key=2171442413,monmap=1261126655,osd_metadata=3824660376,osd_pg_creating=761603184,osd_snap=3939737339,osdmap=3074873358}) 2024-08-19T12:45:08.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=361514021}) 2024-08-19T12:45:08.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2052971102}) 2024-08-19T12:45:08.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=361514021}) 2024-08-19T12:45:08.813 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1645227548}) 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=361514021}) 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1788293703}) 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=361514021}) 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3467031242}) 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=361514021}) 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2548814217}) 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=361514021}) 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3092568019}) 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.814 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=361514021}) 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=36278655}) 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=361514021}) 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.779+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1994760419}) 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4024165386}) 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=826650330}) 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4024165386}) 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2433282213}) 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4024165386}) 2024-08-19T12:45:08.815 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3280854918}) 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4024165386}) 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3291776503}) 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4024165386}) 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=947408180}) 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4024165386}) 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3612015281}) 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4024165386}) 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=510483090}) 2024-08-19T12:45:08.816 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.817 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4024165386}) 2024-08-19T12:45:08.817 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.781+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3532926062}) 2024-08-19T12:45:08.820 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.820 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2174609311}) 2024-08-19T12:45:08.820 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3793893909}) 2024-08-19T12:45:08.820 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.820 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2174609311}) 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=893196620}) 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2174609311}) 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1533874596}) 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2174609311}) 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1060232408}) 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2174609311}) 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=730802912}) 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2174609311}) 2024-08-19T12:45:08.821 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=964572366}) 2024-08-19T12:45:08.822 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.822 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2174609311}) 2024-08-19T12:45:08.822 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2360197504}) 2024-08-19T12:45:08.822 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.822 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2174609311}) 2024-08-19T12:45:08.822 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.794+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2314328730}) 2024-08-19T12:45:08.822 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.822 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=74} crc {osdmap=1273308344}) 2024-08-19T12:45:08.822 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=67} crc {osdmap=642734530}) 2024-08-19T12:45:08.822 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.822 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=74} crc {osdmap=1273308344}) 2024-08-19T12:45:08.822 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=72} crc {osdmap=2728631335}) 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=74} crc {osdmap=1273308344}) 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=74} crc {osdmap=3502978665}) 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=74} crc {osdmap=1273308344}) 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=75} crc {osdmap=2835743977}) 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=74} crc {osdmap=1273308344}) 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=72} crc {osdmap=2481672207}) 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=74} crc {osdmap=1273308344}) 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=73} crc {osdmap=645070594}) 2024-08-19T12:45:08.823 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.824 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=74} crc {osdmap=1273308344}) 2024-08-19T12:45:08.824 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=77} crc {osdmap=3569675746}) 2024-08-19T12:45:08.824 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:45:08.824 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=74} crc {osdmap=1273308344}) 2024-08-19T12:45:08.824 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:45:08.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=75} crc {osdmap=2079542106}) 2024-08-19T12:45:08.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.828+0000 7f64be627640 10 monclient: get_monmap_and_config 2024-08-19T12:45:08.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.828+0000 7f64be627640 10 monclient: build_initial_monmap 2024-08-19T12:45:08.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.828+0000 7f64be627640 10 monclient: monmap: 2024-08-19T12:45:08.830 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:08.830 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.830 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:08.829941+0000 2024-08-19T12:45:08.830 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:08.829941+0000 2024-08-19T12:45:08.830 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:08.830 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.830 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:08.830 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:08.830 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:08.830 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:08.830 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:08.830 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:08.831 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:08.831 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:08.831 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:08.831 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.828+0000 7f64be627640 1 Processor -- start 2024-08-19T12:45:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.828+0000 7f64be627640 1 -- start start 2024-08-19T12:45:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.828+0000 7f64be627640 10 monclient: init 2024-08-19T12:45:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64be627640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64be627640 10 monclient: _add_conns ranks=[0,4,5,1,2,6,3,8,7] 2024-08-19T12:45:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64be627640 10 monclient(hunting): picked mon.noname-a con 0x7f64b8073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64be627640 10 monclient(hunting): picked mon.noname-c con 0x7f64b80a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64be627640 10 monclient(hunting): picked mon.noname-h con 0x7f64b813f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64be627640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b809dae0 con 0x7f64b8073470 2024-08-19T12:45:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64be627640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b8136910 con 0x7f64b80a57a0 2024-08-19T12:45:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64be627640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b809dd50 con 0x7f64b813f2e0 2024-08-19T12:45:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64be627640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64be627640 10 monclient(hunting): authenticate will time out at 1102.917363s 2024-08-19T12:45:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64b7fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f64b8073470 0x7f64b8136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38206/0 (socket says 172.21.15.26:38206) 2024-08-19T12:45:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64b77fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f64b80a57a0 0x7f64b80a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:44196/0 (socket says 172.21.15.26:44196) 2024-08-19T12:45:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.829+0000 7f64b7fff640 1 -- 172.21.15.26:0/4104886506 learned_addr learned my addr 172.21.15.26:0/4104886506 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:08.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b7fff640 1 -- 172.21.15.26:0/4104886506 >> v1:172.21.15.26:6789/0 conn(0x7f64b8073470 legacy=0x7f64b8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b77fe640 1 -- 172.21.15.26:0/4104886506 >> v1:172.21.15.26:6791/0 conn(0x7f64b80a57a0 legacy=0x7f64b80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b6ffd640 1 -- 172.21.15.26:0/4104886506 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4081212226 0 0) 0x7f64b809dae0 con 0x7f64b8073470 2024-08-19T12:45:08.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b6ffd640 10 monclient(hunting): my global_id is 144540 2024-08-19T12:45:08.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b6ffd640 1 -- 172.21.15.26:0/4104886506 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64a0003520 con 0x7f64b8073470 2024-08-19T12:45:08.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b6ffd640 1 -- 172.21.15.26:0/4104886506 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2554099383 0 0) 0x7f64b8136910 con 0x7f64b80a57a0 2024-08-19T12:45:08.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b6ffd640 10 monclient(hunting): my global_id is 26608 2024-08-19T12:45:08.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64b6ffd640 1 -- 172.21.15.26:0/4104886506 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64b809dae0 con 0x7f64b80a57a0 2024-08-19T12:45:08.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.831+0000 7f64bcbc4640 1 -- 172.21.15.26:0/4104886506 >> v1:172.21.15.104:6791/0 conn(0x7f64b813f2e0 legacy=0x7f64b81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 1 -- 172.21.15.26:0/4104886506 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 214400870 0 0) 0x7f64a0003520 con 0x7f64b8073470 2024-08-19T12:45:08.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 1 -- 172.21.15.26:0/4104886506 >> v1:172.21.15.104:6791/0 conn(0x7f64b813f2e0 legacy=0x7f64b81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 1 -- 172.21.15.26:0/4104886506 >> v1:172.21.15.26:6791/0 conn(0x7f64b80a57a0 legacy=0x7f64b80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:08.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 1 monclient: found mon.noname-a 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 1 -- 172.21.15.26:0/4104886506 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b809e030 con 0x7f64b8073470 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 10 monclient: _finish_auth 0 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 1 -- 172.21.15.26:0/4104886506 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64a8001ba0 con 0x7f64b8073470 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64be627640 5 monclient: authenticate success, global_id 144540 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64be627640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 10 monclient: dump: 2024-08-19T12:45:08.839 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.840 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64be627640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 1 -- 172.21.15.26:0/4104886506 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f64a80048a0 con 0x7f64b8073470 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 1 -- 172.21.15.26:0/4104886506 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64a8004e10 con 0x7f64b8073470 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64be627640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64be627640 4 set_mon_vals no callback set 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64b6ffd640 10 monclient: dump: 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.841 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.842 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64be627640 10 monclient: shutdown 2024-08-19T12:45:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64be627640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64be627640 1 -- 172.21.15.26:0/4104886506 >> v1:172.21.15.26:6789/0 conn(0x7f64b8073470 legacy=0x7f64b8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64be627640 1 -- 172.21.15.26:0/4104886506 shutdown_connections 2024-08-19T12:45:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64be627640 1 -- 172.21.15.26:0/4104886506 >> 172.21.15.26:0/4104886506 conn(0x7f64b80abff0 msgr2=0x7f64b80ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64be627640 1 -- 172.21.15.26:0/4104886506 shutdown_connections 2024-08-19T12:45:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.832+0000 7f64be627640 1 -- 172.21.15.26:0/4104886506 wait complete. 2024-08-19T12:45:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.833+0000 7f64be627640 10 monclient: build_initial_monmap 2024-08-19T12:45:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.833+0000 7f64be627640 10 monclient: monmap: 2024-08-19T12:45:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:08.843 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:08.844 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.833+0000 7f64be627640 1 librados: starting msgr at 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.833+0000 7f64be627640 1 librados: starting objecter 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.833+0000 7f64be627640 1 Processor -- start 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.833+0000 7f64be627640 1 -- start start 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.833+0000 7f64be627640 1 librados: setting wanted keys 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.833+0000 7f64be627640 1 librados: calling monclient init 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.833+0000 7f64be627640 10 monclient: init 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.834+0000 7f64be627640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.834+0000 7f64be627640 10 monclient: _add_conns ranks=[5,7,0,1,3,8,6,2,4] 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.834+0000 7f64be627640 10 monclient(hunting): picked mon.noname-h con 0x7f64b8073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.834+0000 7f64be627640 10 monclient(hunting): picked mon.noname-i con 0x7f64b80a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.834+0000 7f64be627640 10 monclient(hunting): picked mon.noname-a con 0x7f64b813f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:08.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.834+0000 7f64be627640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b812f4c0 con 0x7f64b813f2e0 2024-08-19T12:45:08.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.834+0000 7f64be627640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b8142bc0 con 0x7f64b8073470 2024-08-19T12:45:08.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.834+0000 7f64be627640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b812ea30 con 0x7f64b80a57a0 2024-08-19T12:45:08.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.834+0000 7f64be627640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:08.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.834+0000 7f64be627640 10 monclient(hunting): authenticate will time out at 1102.922451s 2024-08-19T12:45:08.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.834+0000 7f64bcbc4640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f64b813f2e0 0x7f64b81d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38212/0 (socket says 172.21.15.26:38212) 2024-08-19T12:45:08.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.834+0000 7f64bcbc4640 1 -- 172.21.15.26:0/1970859368 learned_addr learned my addr 172.21.15.26:0/1970859368 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.836+0000 7f64bcbc4640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.26:6789/0 conn(0x7f64b813f2e0 legacy=0x7f64b81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.836+0000 7f64b4ff9640 10 client.?.objecter ms_handle_connect 0x7f64b813f2e0 2024-08-19T12:45:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.836+0000 7f64b4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.836+0000 7f64b4ff9640 10 client.?.objecter ms_handle_connect 0x7f64b80a57a0 2024-08-19T12:45:08.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.836+0000 7f64b4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.836+0000 7f64b77fe640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.104:6792/0 conn(0x7f64b80a57a0 legacy=0x7f64b812e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b7fff640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.104:6791/0 conn(0x7f64b8073470 legacy=0x7f64b812dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 client.?.objecter ms_handle_connect 0x7f64b8073470 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2973462314 0 0) 0x7f64b812f4c0 con 0x7f64b813f2e0 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): my global_id is 144549 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64a4004eb0 con 0x7f64b813f2e0 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1420025130 0 0) 0x7f64b812ea30 con 0x7f64b80a57a0 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): my global_id is 135232 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64b812f4c0 con 0x7f64b80a57a0 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1167819723 0 0) 0x7f64b8142bc0 con 0x7f64b8073470 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): my global_id is 26303 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64b812ea30 con 0x7f64b8073470 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4014270061 0 0) 0x7f64a4004eb0 con 0x7f64b813f2e0 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64b8142bc0 con 0x7f64b813f2e0 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64b0003990 con 0x7f64b813f2e0 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:08.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:45:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): dump: 2024-08-19T12:45:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.856 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 775281355 0 0) 0x7f64b812f4c0 con 0x7f64b80a57a0 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64a4004eb0 con 0x7f64b80a57a0 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64ac0047e0 con 0x7f64b80a57a0 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): dump: 2024-08-19T12:45:08.857 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.858 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1939889237 0 0) 0x7f64b812ea30 con 0x7f64b8073470 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64b812f4c0 con 0x7f64b8073470 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64a8001b60 con 0x7f64b8073470 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient(hunting): dump: 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.859 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2729623544 0 0) 0x7f64b8142bc0 con 0x7f64b813f2e0 2024-08-19T12:45:08.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.104:6792/0 conn(0x7f64b80a57a0 legacy=0x7f64b812e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.104:6791/0 conn(0x7f64b8073470 legacy=0x7f64b812dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:45:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 monclient: found mon.a 2024-08-19T12:45:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b8131820 con 0x7f64b813f2e0 2024-08-19T12:45:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient: _finish_auth 0 2024-08-19T12:45:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:45:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64be627640 5 monclient: authenticate success, global_id 144549 2024-08-19T12:45:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64be627640 10 monclient: _renew_subs 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64be627640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64be627640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f64b8130020 con 0x7f64b813f2e0 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64be627640 10 client.144549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64be627640 10 monclient: _renew_subs 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64be627640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64be627640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f64b8132f30 con 0x7f64b813f2e0 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64be627640 1 librados: init done 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f64b0003e90 con 0x7f64b813f2e0 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64b00043c0 con 0x7f64b813f2e0 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b67fc640 4 set_mon_vals no callback set 2024-08-19T12:45:08.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.837+0000 7f64b4ff9640 10 monclient: dump: 2024-08-19T12:45:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.863 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.838+0000 7f64be627640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.838+0000 7f64be627640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f64980065c0 con 0x7f6498002860 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.838+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f64b0004560 con 0x7f64b813f2e0 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.838+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(281..281 src has 1..281) ==== 5364+0+0 (unknown 2673407375 0 0) 0x7f64b004e3d0 con 0x7f64b813f2e0 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.838+0000 7f64b4ff9640 10 client.144549.objecter ms_dispatch 0x7f64b8000b90 osd_map(281..281 src has 1..281) 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.838+0000 7f64b4ff9640 3 client.144549.objecter handle_osd_map got epochs [281,281] > 0 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.838+0000 7f64b4ff9640 3 client.144549.objecter handle_osd_map decoding full epoch 281 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.839+0000 7f64b4ff9640 20 client.144549.objecter dump_active .. 0 homeless 2024-08-19T12:45:08.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.839+0000 7f64b77fe640 10 monclient: get_auth_request con 0x7f64a40419c0 auth_method 0 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.840+0000 7f64b7fff640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.26:6790/0 conn(0x7f6498002860 legacy=0x7f6498004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.840+0000 7f64b4ff9640 10 client.144549.objecter ms_handle_connect 0x7f6498002860 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.840+0000 7f64b4ff9640 10 client.144549.objecter resend_mon_ops 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4166993015 0 0) 0x7f64980065c0 con 0x7f6498002860 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 10 monclient(hunting): my global_id is 26282 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64b8142bc0 con 0x7f6498002860 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1894170583 0 0) 0x7f64b8142bc0 con 0x7f6498002860 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64980065c0 con 0x7f6498002860 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64a80048a0 con 0x7f6498002860 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 10 monclient: dump: 2024-08-19T12:45:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.866 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 10 client.144549.objecter ms_handle_connect 0x7f64a40419c0 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3742731808 0 0) 0x7f64980065c0 con 0x7f6498002860 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f649800a160 con 0x7f6498002860 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f64a8001d00 con 0x7f6498002860 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.26:6790/0 conn(0x7f6498002860 legacy=0x7f6498004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.841+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.26:6790/0 conn(0x7f6498002860 legacy=0x7f6498004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:08.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.847+0000 7f64be627640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.848+0000 7f64be627640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6498009e50 con 0x7f6498009110 2024-08-19T12:45:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b7fff640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.26:6790/0 conn(0x7f6498009110 legacy=0x7f64980095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 10 client.144549.objecter ms_handle_connect 0x7f6498009110 2024-08-19T12:45:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 10 client.144549.objecter resend_mon_ops 2024-08-19T12:45:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1759911039 0 0) 0x7f6498009e50 con 0x7f6498009110 2024-08-19T12:45:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 10 monclient(hunting): my global_id is 26291 2024-08-19T12:45:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64980065c0 con 0x7f6498009110 2024-08-19T12:45:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1721317070 0 0) 0x7f64980065c0 con 0x7f6498009110 2024-08-19T12:45:08.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6498009e50 con 0x7f6498009110 2024-08-19T12:45:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64a8005a80 con 0x7f6498009110 2024-08-19T12:45:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 10 monclient: dump: 2024-08-19T12:45:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:08.869 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3677785415 0 0) 0x7f6498009e50 con 0x7f6498009110 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.850+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f649800c3a0 con 0x7f6498009110 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1376732841) 0x7f64a8005f00 con 0x7f6498009110 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64b4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64b4ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:08.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.26:6790/0 conn(0x7f6498009110 legacy=0x7f64980095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64b4ff9640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.26:6790/0 conn(0x7f6498009110 legacy=0x7f64980095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:08.871 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 421905, 2024-08-19T12:45:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:08.890 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:08.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 10 librados: watch_flush enter 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 10 librados: watch_flush exit 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 20 client.144549.objecter shutdown clearing up homeless session... 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 10 client.144549.objecter successfully canceled tick 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.104:6800/899887506 conn(0x7f64a40419c0 legacy=0x7f64a4043e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 10 monclient: shutdown 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 1 -- 172.21.15.26:0/1970859368 >> v1:172.21.15.26:6789/0 conn(0x7f64b813f2e0 legacy=0x7f64b81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64bcbc4640 1 -- 172.21.15.26:0/1970859368 reap_dead start 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64bcbc4640 1 -- 172.21.15.26:0/1970859368 reap_dead start 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 1 -- 172.21.15.26:0/1970859368 shutdown_connections 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 1 -- 172.21.15.26:0/1970859368 >> 172.21.15.26:0/1970859368 conn(0x7f64b80abff0 msgr2=0x7f64b80a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 1 -- 172.21.15.26:0/1970859368 shutdown_connections 2024-08-19T12:45:08.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 1 -- 172.21.15.26:0/1970859368 wait complete. 2024-08-19T12:45:08.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:08.851+0000 7f64be627640 1 librados: shutdown 2024-08-19T12:45:08.894 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:08.894 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:45:08.894 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:45:09.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.032+0000 7ff0483fd640 10 monclient: get_monmap_and_config 2024-08-19T12:45:09.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.032+0000 7ff0483fd640 10 monclient: build_initial_monmap 2024-08-19T12:45:09.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.032+0000 7ff0483fd640 10 monclient: monmap: 2024-08-19T12:45:09.034 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:09.034 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.034 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:09.034159+0000 2024-08-19T12:45:09.034 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:09.034159+0000 2024-08-19T12:45:09.034 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:09.035 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.035 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:09.035 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:09.035 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:09.035 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:09.035 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:09.035 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:09.035 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:09.035 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:09.035 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:09.035 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.032+0000 7ff0483fd640 1 Processor -- start 2024-08-19T12:45:09.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.032+0000 7ff0483fd640 1 -- start start 2024-08-19T12:45:09.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.032+0000 7ff0483fd640 10 monclient: init 2024-08-19T12:45:09.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.033+0000 7ff0483fd640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.033+0000 7ff0483fd640 10 monclient: _add_conns ranks=[0,1,5,3,8,4,7,6,2] 2024-08-19T12:45:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.033+0000 7ff0483fd640 10 monclient(hunting): picked mon.noname-a con 0x7ff040073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.033+0000 7ff0483fd640 10 monclient(hunting): picked mon.noname-f con 0x7ff0400a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.033+0000 7ff0483fd640 10 monclient(hunting): picked mon.noname-h con 0x7ff04013f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.033+0000 7ff0483fd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff04009dae0 con 0x7ff040073470 2024-08-19T12:45:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.033+0000 7ff0483fd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff040136910 con 0x7ff0400a57a0 2024-08-19T12:45:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.033+0000 7ff0483fd640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff04009dd50 con 0x7ff04013f2e0 2024-08-19T12:45:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.033+0000 7ff0483fd640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.033+0000 7ff0483fd640 10 monclient(hunting): authenticate will time out at 1103.121593s 2024-08-19T12:45:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.033+0000 7ff046199640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff040073470 0x7ff040136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38216/0 (socket says 172.21.15.26:38216) 2024-08-19T12:45:09.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.033+0000 7ff046199640 1 -- 172.21.15.26:0/3537862286 learned_addr learned my addr 172.21.15.26:0/3537862286 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.035+0000 7ff046199640 1 -- 172.21.15.26:0/3537862286 >> v1:172.21.15.26:6789/0 conn(0x7ff040073470 legacy=0x7ff040136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff04699a640 1 -- 172.21.15.26:0/3537862286 >> v1:172.21.15.104:6791/0 conn(0x7ff04013f2e0 legacy=0x7ff0401416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 1 -- 172.21.15.26:0/3537862286 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3686673575 0 0) 0x7ff04009dae0 con 0x7ff040073470 2024-08-19T12:45:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient(hunting): my global_id is 144558 2024-08-19T12:45:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045998640 1 -- 172.21.15.26:0/3537862286 >> v1:172.21.15.104:6789/0 conn(0x7ff0400a57a0 legacy=0x7ff0400a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 1 -- 172.21.15.26:0/3537862286 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff02c003520 con 0x7ff040073470 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 1 -- 172.21.15.26:0/3537862286 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 95133705 0 0) 0x7ff02c003520 con 0x7ff040073470 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 1 -- 172.21.15.26:0/3537862286 >> v1:172.21.15.104:6791/0 conn(0x7ff04013f2e0 legacy=0x7ff0401416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 1 -- 172.21.15.26:0/3537862286 >> v1:172.21.15.104:6789/0 conn(0x7ff0400a57a0 legacy=0x7ff0400a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient: _finish_hunting 0 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 1 monclient: found mon.noname-a 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 1 -- 172.21.15.26:0/3537862286 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04009e030 con 0x7ff040073470 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient: _finish_auth 0 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient: _check_auth_tickets 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 1 -- 172.21.15.26:0/3537862286 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff03c001ae0 con 0x7ff040073470 2024-08-19T12:45:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff0483fd640 5 monclient: authenticate success, global_id 144558 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff0483fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient: dump: 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.048 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff0483fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 1 -- 172.21.15.26:0/3537862286 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff03c0048a0 con 0x7ff040073470 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 1 -- 172.21.15.26:0/3537862286 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff03c004dd0 con 0x7ff040073470 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff045197640 10 monclient: dump: 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.050 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff0483fd640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.036+0000 7ff0483fd640 4 set_mon_vals no callback set 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.037+0000 7ff0483fd640 10 monclient: shutdown 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.037+0000 7ff0483fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.037+0000 7ff0483fd640 1 -- 172.21.15.26:0/3537862286 >> v1:172.21.15.26:6789/0 conn(0x7ff040073470 legacy=0x7ff040136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.037+0000 7ff0483fd640 1 -- 172.21.15.26:0/3537862286 shutdown_connections 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.037+0000 7ff0483fd640 1 -- 172.21.15.26:0/3537862286 >> 172.21.15.26:0/3537862286 conn(0x7ff0400abff0 msgr2=0x7ff0400ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.037+0000 7ff0483fd640 1 -- 172.21.15.26:0/3537862286 shutdown_connections 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.037+0000 7ff0483fd640 1 -- 172.21.15.26:0/3537862286 wait complete. 2024-08-19T12:45:09.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.037+0000 7ff0483fd640 10 monclient: build_initial_monmap 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.037+0000 7ff0483fd640 10 monclient: monmap: 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:09.052 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.037+0000 7ff0483fd640 1 librados: starting msgr at 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.037+0000 7ff0483fd640 1 librados: starting objecter 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 1 Processor -- start 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 1 -- start start 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 1 librados: setting wanted keys 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 1 librados: calling monclient init 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 10 monclient: init 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 10 monclient: _add_conns ranks=[1,7,3,5,4,0,6,2,8] 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 10 monclient(hunting): picked mon.noname-f con 0x7ff040073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:09.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 10 monclient(hunting): picked mon.noname-i con 0x7ff0400a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 10 monclient(hunting): picked mon.noname-g con 0x7ff04013f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff040137180 con 0x7ff040073470 2024-08-19T12:45:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff040137460 con 0x7ff04013f2e0 2024-08-19T12:45:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0400a4850 con 0x7ff0400a57a0 2024-08-19T12:45:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff0483fd640 10 monclient(hunting): authenticate will time out at 1103.126543s 2024-08-19T12:45:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff046199640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff040073470 0x7ff0401ce3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:53268/0 (socket says 172.21.15.26:53268) 2024-08-19T12:45:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.038+0000 7ff046199640 1 -- 172.21.15.26:0/2842067275 learned_addr learned my addr 172.21.15.26:0/2842067275 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff045998640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.104:6792/0 conn(0x7ff0400a57a0 legacy=0x7ff0401d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff046199640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.104:6789/0 conn(0x7ff040073470 legacy=0x7ff0401ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 client.?.objecter ms_handle_connect 0x7ff0400a57a0 2024-08-19T12:45:09.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 client.?.objecter ms_handle_connect 0x7ff040073470 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 client.?.objecter ms_handle_connect 0x7ff04013f2e0 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff04699a640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.104:6790/0 conn(0x7ff04013f2e0 legacy=0x7ff0401d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287431176 0 0) 0x7ff0400a4850 con 0x7ff0400a57a0 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): my global_id is 135241 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff024004eb0 con 0x7ff0400a57a0 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff046199640 1 --1- 172.21.15.26:0/2842067275 >> v1:172.21.15.104:6789/0 conn(0x7ff040073470 0x7ff0401ce3a0 :-1 s=OPENED pgs=3386 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7ff040137180 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:45:09.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3680235628 0 0) 0x7ff040137460 con 0x7ff04013f2e0 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): my global_id is 154182 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0400a4850 con 0x7ff04013f2e0 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 391142330 0 0) 0x7ff024004eb0 con 0x7ff0400a57a0 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff040137460 con 0x7ff0400a57a0 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1818648596 0 0) 0x7ff0400a4850 con 0x7ff04013f2e0 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff024004eb0 con 0x7ff04013f2e0 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff030001c80 con 0x7ff0400a57a0 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:45:09.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): dump: 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.057 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff038002030 con 0x7ff04013f2e0 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient(hunting): dump: 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.058 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3630222268 0 0) 0x7ff024004eb0 con 0x7ff04013f2e0 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.104:6792/0 conn(0x7ff0400a57a0 legacy=0x7ff0401d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.104:6789/0 conn(0x7ff040073470 legacy=0x7ff0401ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 monclient: found mon.g 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff040142b80 con 0x7ff04013f2e0 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient: _finish_auth 0 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff0483fd640 5 monclient: authenticate success, global_id 154182 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3516740501 0 0) 0x7ff040137180 con 0x7ff040073470 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff0483fd640 10 monclient: _renew_subs 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff0483fd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff0483fd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff04013e3c0 con 0x7ff04013f2e0 2024-08-19T12:45:09.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.041+0000 7ff036ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff0483fd640 10 client.154182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff0483fd640 10 monclient: _renew_subs 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff0483fd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff0483fd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff040136bc0 con 0x7ff04013f2e0 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff0483fd640 1 librados: init done 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.3 v1:172.21.15.104:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff038003de0 con 0x7ff04013f2e0 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff036ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.3 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff038004350 con 0x7ff04013f2e0 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff036ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff044996640 4 set_mon_vals no callback set 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff036ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff036ffd640 10 monclient: dump: 2024-08-19T12:45:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.062 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.063 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.063 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff0483fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:09.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.042+0000 7ff0483fd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff0280065c0 con 0x7ff028002860 2024-08-19T12:45:09.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.043+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.3 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff0380044f0 con 0x7ff04013f2e0 2024-08-19T12:45:09.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.043+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.3 v1:172.21.15.104:6790/0 8 ==== osd_map(281..281 src has 1..281) ==== 5364+0+0 (unknown 2673407375 0 0) 0x7ff03804e230 con 0x7ff04013f2e0 2024-08-19T12:45:09.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.043+0000 7ff036ffd640 10 client.154182.objecter ms_dispatch 0x7ff040000b90 osd_map(281..281 src has 1..281) 2024-08-19T12:45:09.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.043+0000 7ff036ffd640 3 client.154182.objecter handle_osd_map got epochs [281,281] > 0 2024-08-19T12:45:09.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.043+0000 7ff036ffd640 3 client.154182.objecter handle_osd_map decoding full epoch 281 2024-08-19T12:45:09.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.043+0000 7ff036ffd640 20 client.154182.objecter dump_active .. 0 homeless 2024-08-19T12:45:09.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.043+0000 7ff045998640 10 monclient: get_auth_request con 0x7ff0240414f0 auth_method 0 2024-08-19T12:45:09.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.044+0000 7ff046199640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.26:6791/0 conn(0x7ff028002860 legacy=0x7ff028004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.044+0000 7ff036ffd640 10 client.154182.objecter ms_handle_connect 0x7ff028002860 2024-08-19T12:45:09.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.044+0000 7ff036ffd640 10 client.154182.objecter resend_mon_ops 2024-08-19T12:45:09.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3396966824 0 0) 0x7ff0280065c0 con 0x7ff028002860 2024-08-19T12:45:09.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 10 monclient(hunting): my global_id is 26617 2024-08-19T12:45:09.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff040137180 con 0x7ff028002860 2024-08-19T12:45:09.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4265965662 0 0) 0x7ff040137180 con 0x7ff028002860 2024-08-19T12:45:09.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0280065c0 con 0x7ff028002860 2024-08-19T12:45:09.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff03c004be0 con 0x7ff028002860 2024-08-19T12:45:09.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 10 monclient: dump: 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.068 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2713214793 0 0) 0x7ff0280065c0 con 0x7ff028002860 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff02800a160 con 0x7ff028002860 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 10 client.154182.objecter ms_handle_connect 0x7ff0240414f0 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff03c004be0 con 0x7ff028002860 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.26:6791/0 conn(0x7ff028002860 legacy=0x7ff028004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.045+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.26:6791/0 conn(0x7ff028002860 legacy=0x7ff028004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.051+0000 7ff0483fd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:09.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.051+0000 7ff0483fd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff028009e50 con 0x7ff028009110 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff046199640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.26:6791/0 conn(0x7ff028009110 legacy=0x7ff0280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 10 client.154182.objecter ms_handle_connect 0x7ff028009110 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 10 client.154182.objecter resend_mon_ops 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 105263727 0 0) 0x7ff028009e50 con 0x7ff028009110 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 10 monclient(hunting): my global_id is 26626 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0280065c0 con 0x7ff028009110 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2452428773 0 0) 0x7ff0280065c0 con 0x7ff028009110 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff028009e50 con 0x7ff028009110 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff03c005450 con 0x7ff028009110 2024-08-19T12:45:09.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 10 monclient: dump: 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.071 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.072 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3737312835 0 0) 0x7ff028009e50 con 0x7ff028009110 2024-08-19T12:45:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff02800c3f0 con 0x7ff028009110 2024-08-19T12:45:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7878 (unknown 0 0 4007053001) 0x7ff03c0085c0 con 0x7ff028009110 2024-08-19T12:45:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.26:6791/0 conn(0x7ff028009110 legacy=0x7ff0280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff036ffd640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.26:6791/0 conn(0x7ff028009110 legacy=0x7ff0280095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:09.073 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 422085, 2024-08-19T12:45:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:09.073 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:09.074 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:09.075 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:09.076 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:09.077 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:09.078 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.083 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:09.084 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.085 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.088 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:09.089 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:09.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:09.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:09.094 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 10 librados: watch_flush enter 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 10 librados: watch_flush exit 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 20 client.154182.objecter shutdown clearing up homeless session... 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 10 client.154182.objecter successfully canceled tick 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.104:6800/899887506 conn(0x7ff0240414f0 legacy=0x7ff0240439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 10 monclient: shutdown 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 1 -- 172.21.15.26:0/2842067275 >> v1:172.21.15.104:6790/0 conn(0x7ff04013f2e0 legacy=0x7ff0401d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff04699a640 1 -- 172.21.15.26:0/2842067275 reap_dead start 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff04699a640 1 -- 172.21.15.26:0/2842067275 reap_dead start 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 1 -- 172.21.15.26:0/2842067275 shutdown_connections 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 1 -- 172.21.15.26:0/2842067275 >> 172.21.15.26:0/2842067275 conn(0x7ff0400abff0 msgr2=0x7ff040133c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:09.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 1 -- 172.21.15.26:0/2842067275 shutdown_connections 2024-08-19T12:45:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 1 -- 172.21.15.26:0/2842067275 wait complete. 2024-08-19T12:45:09.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.055+0000 7ff0483fd640 1 librados: shutdown 2024-08-19T12:45:09.097 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:09.097 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:45:09.097 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:45:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.195+0000 7fc1aacc5640 10 monclient: get_monmap_and_config 2024-08-19T12:45:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.195+0000 7fc1aacc5640 10 monclient: build_initial_monmap 2024-08-19T12:45:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.195+0000 7fc1aacc5640 10 monclient: monmap: 2024-08-19T12:45:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:09.197106+0000 2024-08-19T12:45:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:09.197106+0000 2024-08-19T12:45:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:09.198 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 1 Processor -- start 2024-08-19T12:45:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 1 -- start start 2024-08-19T12:45:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 10 monclient: init 2024-08-19T12:45:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 10 monclient: _add_conns ranks=[7,4,3,8,6,1,2,0,5] 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 10 monclient(hunting): picked mon.noname-i con 0x7fc1a4073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 10 monclient(hunting): picked mon.noname-c con 0x7fc1a40a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 10 monclient(hunting): picked mon.noname-g con 0x7fc1a413f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a409dae0 con 0x7fc1a413f2e0 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a4136910 con 0x7fc1a40a57a0 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a409dd50 con 0x7fc1a4073470 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1aacc5640 10 monclient(hunting): authenticate will time out at 1103.284578s 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1a3fff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fc1a40a57a0 0x7fc1a40a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:44212/0 (socket says 172.21.15.26:44212) 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.196+0000 7fc1a3fff640 1 -- 172.21.15.26:0/1903796090 learned_addr learned my addr 172.21.15.26:0/1903796090 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.198+0000 7fc1a3fff640 1 -- 172.21.15.26:0/1903796090 >> v1:172.21.15.26:6791/0 conn(0x7fc1a40a57a0 legacy=0x7fc1a40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.198+0000 7fc1a8a61640 1 -- 172.21.15.26:0/1903796090 >> v1:172.21.15.104:6792/0 conn(0x7fc1a4073470 legacy=0x7fc1a4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.198+0000 7fc1a9262640 1 -- 172.21.15.26:0/1903796090 >> v1:172.21.15.104:6790/0 conn(0x7fc1a413f2e0 legacy=0x7fc1a41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.683+0000 7fc1a37fe640 1 -- 172.21.15.26:0/1903796090 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3488587135 0 0) 0x7fc1a4136910 con 0x7fc1a40a57a0 2024-08-19T12:45:09.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.683+0000 7fc1a37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.683+0000 7fc1a37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.683+0000 7fc1a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.683+0000 7fc1a37fe640 10 monclient(hunting): my global_id is 26635 2024-08-19T12:45:09.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 1 -- 172.21.15.26:0/1903796090 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc194003520 con 0x7fc1a40a57a0 2024-08-19T12:45:09.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 1 -- 172.21.15.26:0/1903796090 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2996573517 0 0) 0x7fc194003520 con 0x7fc1a40a57a0 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 1 -- 172.21.15.26:0/1903796090 >> v1:172.21.15.104:6792/0 conn(0x7fc1a4073470 legacy=0x7fc1a4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 1 -- 172.21.15.26:0/1903796090 >> v1:172.21.15.104:6790/0 conn(0x7fc1a413f2e0 legacy=0x7fc1a41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 10 monclient: _finish_hunting 0 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 1 monclient: found mon.noname-c 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 1 -- 172.21.15.26:0/1903796090 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a409e030 con 0x7fc1a40a57a0 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 10 monclient: _finish_auth 0 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 10 monclient: _check_auth_tickets 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 1 -- 172.21.15.26:0/1903796090 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc190001ae0 con 0x7fc1a40a57a0 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1aacc5640 5 monclient: authenticate success, global_id 26635 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1aacc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:09.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 10 monclient: dump: 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.689 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1aacc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 1 -- 172.21.15.26:0/1903796090 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1900048a0 con 0x7fc1a40a57a0 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 1 -- 172.21.15.26:0/1903796090 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc190004dd0 con 0x7fc1a40a57a0 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1aacc5640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1aacc5640 4 set_mon_vals no callback set 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.684+0000 7fc1a37fe640 10 monclient: dump: 2024-08-19T12:45:09.690 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.691 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.692 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.685+0000 7fc1aacc5640 10 monclient: shutdown 2024-08-19T12:45:09.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.685+0000 7fc1aacc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:09.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.685+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1903796090 >> v1:172.21.15.26:6791/0 conn(0x7fc1a40a57a0 legacy=0x7fc1a40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.685+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1903796090 shutdown_connections 2024-08-19T12:45:09.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.685+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1903796090 >> 172.21.15.26:0/1903796090 conn(0x7fc1a40abff0 msgr2=0x7fc1a40ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:09.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.685+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1903796090 shutdown_connections 2024-08-19T12:45:09.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.685+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1903796090 wait complete. 2024-08-19T12:45:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.686+0000 7fc1aacc5640 10 monclient: build_initial_monmap 2024-08-19T12:45:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.686+0000 7fc1aacc5640 10 monclient: monmap: 2024-08-19T12:45:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.686+0000 7fc1aacc5640 1 librados: starting msgr at 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.686+0000 7fc1aacc5640 1 librados: starting objecter 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.686+0000 7fc1aacc5640 1 Processor -- start 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.687+0000 7fc1aacc5640 1 -- start start 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.687+0000 7fc1aacc5640 1 librados: setting wanted keys 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.687+0000 7fc1aacc5640 1 librados: calling monclient init 2024-08-19T12:45:09.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.687+0000 7fc1aacc5640 10 monclient: init 2024-08-19T12:45:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.688+0000 7fc1aacc5640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.688+0000 7fc1aacc5640 10 monclient: _add_conns ranks=[2,0,5,7,6,3,4,8,1] 2024-08-19T12:45:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.688+0000 7fc1aacc5640 10 monclient(hunting): picked mon.noname-b con 0x7fc1a4073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.688+0000 7fc1aacc5640 10 monclient(hunting): picked mon.noname-a con 0x7fc1a40a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.688+0000 7fc1aacc5640 10 monclient(hunting): picked mon.noname-h con 0x7fc1a413f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.688+0000 7fc1aacc5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a4137280 con 0x7fc1a40a57a0 2024-08-19T12:45:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.688+0000 7fc1aacc5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a4135420 con 0x7fc1a4073470 2024-08-19T12:45:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.688+0000 7fc1aacc5640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a413f0b0 con 0x7fc1a413f2e0 2024-08-19T12:45:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.688+0000 7fc1aacc5640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.688+0000 7fc1aacc5640 10 monclient(hunting): authenticate will time out at 1103.776339s 2024-08-19T12:45:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.688+0000 7fc1a8a61640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc1a4073470 0x7fc1a41ce3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40066/0 (socket says 172.21.15.26:40066) 2024-08-19T12:45:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.688+0000 7fc1a8a61640 1 -- 172.21.15.26:0/1002625397 learned_addr learned my addr 172.21.15.26:0/1002625397 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.690+0000 7fc1a8a61640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.26:6790/0 conn(0x7fc1a4073470 legacy=0x7fc1a41ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.690+0000 7fc1a17fa640 10 client.?.objecter ms_handle_connect 0x7fc1a4073470 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.690+0000 7fc1a17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.691+0000 7fc1a9262640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.104:6791/0 conn(0x7fc1a413f2e0 legacy=0x7fc1a41d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.691+0000 7fc1a17fa640 10 client.?.objecter ms_handle_connect 0x7fc1a413f2e0 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.691+0000 7fc1a17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.692+0000 7fc1a17fa640 10 client.?.objecter ms_handle_connect 0x7fc1a40a57a0 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.692+0000 7fc1a17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.692+0000 7fc1a3fff640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.26:6789/0 conn(0x7fc1a40a57a0 legacy=0x7fc1a41d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.692+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4255268305 0 0) 0x7fc1a4137280 con 0x7fc1a40a57a0 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.692+0000 7fc1a17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.692+0000 7fc1a17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.692+0000 7fc1a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.692+0000 7fc1a17fa640 10 monclient(hunting): my global_id is 144567 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.692+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc188004eb0 con 0x7fc1a40a57a0 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.692+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1549291149 0 0) 0x7fc188004eb0 con 0x7fc1a40a57a0 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1a4137280 con 0x7fc1a40a57a0 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc190004d90 con 0x7fc1a40a57a0 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 10 monclient(hunting): dump: 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.701 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3271306663 0 0) 0x7fc1a4137280 con 0x7fc1a40a57a0 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.104:6791/0 conn(0x7fc1a413f2e0 legacy=0x7fc1a41d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.26:6790/0 conn(0x7fc1a4073470 legacy=0x7fc1a41ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 10 monclient: _finish_hunting 0 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 1 monclient: found mon.a 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a4142b60 con 0x7fc1a40a57a0 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 10 monclient: _finish_auth 0 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 10 monclient: _check_auth_tickets 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1aacc5640 5 monclient: authenticate success, global_id 144567 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1aacc5640 10 monclient: _renew_subs 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1aacc5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc1a4136fa0 con 0x7fc1a40a57a0 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1aacc5640 10 client.144567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1aacc5640 10 monclient: _renew_subs 2024-08-19T12:45:09.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1aacc5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc1a413e9f0 con 0x7fc1a40a57a0 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1aacc5640 1 librados: init done 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc190001d30 con 0x7fc1a40a57a0 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc190005700 con 0x7fc1a40a57a0 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a2ffd640 4 set_mon_vals no callback set 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.693+0000 7fc1a17fa640 10 monclient: dump: 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.694+0000 7fc1aacc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:09.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.694+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc1840065c0 con 0x7fc184002860 2024-08-19T12:45:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.694+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fc190011b60 con 0x7fc1a40a57a0 2024-08-19T12:45:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.694+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(282..282 src has 1..282) ==== 5364+0+0 (unknown 3942427206 0 0) 0x7fc19004e1f0 con 0x7fc1a40a57a0 2024-08-19T12:45:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.694+0000 7fc1a17fa640 10 client.144567.objecter ms_dispatch 0x7fc1a4000b90 osd_map(282..282 src has 1..282) 2024-08-19T12:45:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.694+0000 7fc1a17fa640 3 client.144567.objecter handle_osd_map got epochs [282,282] > 0 2024-08-19T12:45:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.694+0000 7fc1a17fa640 3 client.144567.objecter handle_osd_map decoding full epoch 282 2024-08-19T12:45:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.695+0000 7fc1a17fa640 20 client.144567.objecter dump_active .. 0 homeless 2024-08-19T12:45:09.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.695+0000 7fc1a9262640 10 monclient: get_auth_request con 0x7fc188040f30 auth_method 0 2024-08-19T12:45:09.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.696+0000 7fc1a8a61640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.26:6792/0 conn(0x7fc184002860 legacy=0x7fc184004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.696+0000 7fc1a17fa640 10 client.144567.objecter ms_handle_connect 0x7fc184002860 2024-08-19T12:45:09.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.696+0000 7fc1a17fa640 10 client.144567.objecter resend_mon_ops 2024-08-19T12:45:09.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.697+0000 7fc1a17fa640 10 client.144567.objecter ms_handle_connect 0x7fc188040f30 2024-08-19T12:45:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3407989306 0 0) 0x7fc1840065c0 con 0x7fc184002860 2024-08-19T12:45:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 10 monclient(hunting): my global_id is 26637 2024-08-19T12:45:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1a4137280 con 0x7fc184002860 2024-08-19T12:45:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2608736878 0 0) 0x7fc1a4137280 con 0x7fc184002860 2024-08-19T12:45:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1840065c0 con 0x7fc184002860 2024-08-19T12:45:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc198001bc0 con 0x7fc184002860 2024-08-19T12:45:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 10 monclient: dump: 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.724 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.725 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1907938390 0 0) 0x7fc1840065c0 con 0x7fc184002860 2024-08-19T12:45:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.720+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc18400a160 con 0x7fc184002860 2024-08-19T12:45:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.721+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc1980057c0 con 0x7fc184002860 2024-08-19T12:45:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.721+0000 7fc1a17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.721+0000 7fc1a17fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.721+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.26:6792/0 conn(0x7fc184002860 legacy=0x7fc184004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.721+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.26:6792/0 conn(0x7fc184002860 legacy=0x7fc184004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:09.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.727+0000 7fc1aacc5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:09.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.727+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc184009e70 con 0x7fc184009130 2024-08-19T12:45:09.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a8a61640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.26:6792/0 conn(0x7fc184009130 legacy=0x7fc1840095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 10 client.144567.objecter ms_handle_connect 0x7fc184009130 2024-08-19T12:45:09.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 10 client.144567.objecter resend_mon_ops 2024-08-19T12:45:09.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 659099627 0 0) 0x7fc184009e70 con 0x7fc184009130 2024-08-19T12:45:09.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 10 monclient(hunting): my global_id is 26646 2024-08-19T12:45:09.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1840065c0 con 0x7fc184009130 2024-08-19T12:45:09.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 518960241 0 0) 0x7fc1840065c0 con 0x7fc184009130 2024-08-19T12:45:09.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc184009e70 con 0x7fc184009130 2024-08-19T12:45:09.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc198003d00 con 0x7fc184009130 2024-08-19T12:45:09.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:09.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 10 monclient: dump: 2024-08-19T12:45:09.734 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.734 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.734 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.734 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.734 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.734 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.734 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.734 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.734 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.735 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.735 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.735 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.735 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.735 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.735 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.735 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 937087022 0 0) 0x7fc184009e70 con 0x7fc184009130 2024-08-19T12:45:09.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.730+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc18400c410 con 0x7fc184009130 2024-08-19T12:45:09.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 4264055761) 0x7fc198005a30 con 0x7fc184009130 2024-08-19T12:45:09.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1a17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:09.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1a17fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:09.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.26:6792/0 conn(0x7fc184009130 legacy=0x7fc1840095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1a17fa640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.26:6792/0 conn(0x7fc184009130 legacy=0x7fc1840095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:09.736 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:09.736 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 422771, 2024-08-19T12:45:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:09.737 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:45:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:09.738 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.739 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:09.740 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:09.742 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:09.743 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:09.743 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:09.743 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:09.743 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:09.743 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:09.743 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:09.743 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:09.743 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.743 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:09.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.743 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:09.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.744 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:09.744 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:09.744 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.744 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:09.744 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.744 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.745 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:09.745 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:09.745 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.745 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.745 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.745 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:09.745 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:09.745 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.745 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.746 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.746 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:09.746 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.746 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.746 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.746 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:09.746 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:09.746 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.746 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.746 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.747 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:09.747 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:09.747 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.747 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.747 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.747 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:09.747 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.747 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.748 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.749 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.749 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:09.749 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.749 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.749 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.749 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:09.749 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:09.749 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.749 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.749 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.750 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:09.750 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:09.750 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.750 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.750 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.750 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:09.750 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.751 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.751 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:09.751 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:09.751 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.751 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.751 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.751 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:09.751 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:09.751 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.752 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.753 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:09.753 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:09.753 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.753 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:09.753 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.753 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.754 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.754 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:09.754 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:09.754 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.754 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.754 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.754 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:09.754 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:09.754 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.755 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:09.755 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.755 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:09.755 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:09.755 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.755 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.756 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.756 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:09.756 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:09.756 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.756 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.756 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.756 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:09.756 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.757 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.757 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:09.757 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:09.757 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.757 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.757 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.757 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.757 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:09.758 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:09.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.758 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:09.758 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:09.758 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:09.758 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.758 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.758 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:45:09.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.758 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:09.758 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:09.759 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:09.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.759 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.759 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:09.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.759 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:09.759 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:09.759 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:09.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.759 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.759 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:09.760 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:09.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 10 librados: watch_flush enter 2024-08-19T12:45:09.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 10 librados: watch_flush exit 2024-08-19T12:45:09.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 20 client.144567.objecter shutdown clearing up homeless session... 2024-08-19T12:45:09.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 10 client.144567.objecter successfully canceled tick 2024-08-19T12:45:09.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.104:6800/899887506 conn(0x7fc188040f30 legacy=0x7fc1880433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 10 monclient: shutdown 2024-08-19T12:45:09.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:09.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1002625397 >> v1:172.21.15.26:6789/0 conn(0x7fc1a40a57a0 legacy=0x7fc1a41d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1a9262640 1 -- 172.21.15.26:0/1002625397 reap_dead start 2024-08-19T12:45:09.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1a9262640 1 -- 172.21.15.26:0/1002625397 reap_dead start 2024-08-19T12:45:09.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1002625397 shutdown_connections 2024-08-19T12:45:09.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1002625397 >> 172.21.15.26:0/1002625397 conn(0x7fc1a40abff0 msgr2=0x7fc1a41349a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:09.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1002625397 shutdown_connections 2024-08-19T12:45:09.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 1 -- 172.21.15.26:0/1002625397 wait complete. 2024-08-19T12:45:09.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.731+0000 7fc1aacc5640 1 librados: shutdown 2024-08-19T12:45:09.763 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:09.763 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:45:09.763 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:45:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.861+0000 7f6b8b20e640 10 monclient: get_monmap_and_config 2024-08-19T12:45:09.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.861+0000 7f6b8b20e640 10 monclient: build_initial_monmap 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.861+0000 7f6b8b20e640 10 monclient: monmap: 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:09.863703+0000 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:09.863703+0000 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:09.864 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:09.865 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:09.865 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:09.865 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:09.865 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.862+0000 7f6b8b20e640 1 Processor -- start 2024-08-19T12:45:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.862+0000 7f6b8b20e640 1 -- start start 2024-08-19T12:45:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.862+0000 7f6b8b20e640 10 monclient: init 2024-08-19T12:45:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b8b20e640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b8b20e640 10 monclient: _add_conns ranks=[6,4,1,8,2,5,3,7,0] 2024-08-19T12:45:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b8b20e640 10 monclient(hunting): picked mon.noname-d con 0x7f6b8406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b8b20e640 10 monclient(hunting): picked mon.noname-c con 0x7f6b840a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:09.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b8b20e640 10 monclient(hunting): picked mon.noname-f con 0x7f6b840a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b8b20e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b84136790 con 0x7f6b840a5d50 2024-08-19T12:45:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b8b20e640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8409ddf0 con 0x7f6b840a5920 2024-08-19T12:45:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b8b20e640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8409e0d0 con 0x7f6b8406e7b0 2024-08-19T12:45:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b8b20e640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b8b20e640 10 monclient(hunting): authenticate will time out at 1103.951131s 2024-08-19T12:45:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b88faa640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f6b8406e7b0 0x7f6b84136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:55182/0 (socket says 172.21.15.26:55182) 2024-08-19T12:45:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b88faa640 1 -- 172.21.15.26:0/2317769708 learned_addr learned my addr 172.21.15.26:0/2317769708 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:09.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.863+0000 7f6b83fff640 1 --1- 172.21.15.26:0/2317769708 >> v1:172.21.15.26:6791/0 conn(0x7f6b840a5920 0x7f6b8413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:44228/0 (socket says 172.21.15.26:44228) 2024-08-19T12:45:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b83fff640 1 -- 172.21.15.26:0/2317769708 >> v1:172.21.15.26:6791/0 conn(0x7f6b840a5920 legacy=0x7f6b8413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b88faa640 1 -- 172.21.15.26:0/2317769708 >> v1:172.21.15.26:6792/0 conn(0x7f6b8406e7b0 legacy=0x7f6b84136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 1 -- 172.21.15.26:0/2317769708 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1285296477 0 0) 0x7f6b6c001450 con 0x7f6b840a5920 2024-08-19T12:45:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b897ab640 1 -- 172.21.15.26:0/2317769708 >> v1:172.21.15.104:6789/0 conn(0x7f6b840a5d50 legacy=0x7f6b84141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient(hunting): my global_id is 26644 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 1 -- 172.21.15.26:0/2317769708 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b70003520 con 0x7f6b840a5920 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 1 -- 172.21.15.26:0/2317769708 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1516132615 0 0) 0x7f6b8409e0d0 con 0x7f6b8406e7b0 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient(hunting): my global_id is 26655 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 1 -- 172.21.15.26:0/2317769708 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b6c001450 con 0x7f6b8406e7b0 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 1 -- 172.21.15.26:0/2317769708 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2226928156 0 0) 0x7f6b70003520 con 0x7f6b840a5920 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 1 -- 172.21.15.26:0/2317769708 >> v1:172.21.15.26:6792/0 conn(0x7f6b8406e7b0 legacy=0x7f6b84136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 1 -- 172.21.15.26:0/2317769708 >> v1:172.21.15.104:6789/0 conn(0x7f6b840a5d50 legacy=0x7f6b84141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient: _finish_hunting 0 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 1 monclient: found mon.noname-c 2024-08-19T12:45:09.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:45:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 1 -- 172.21.15.26:0/2317769708 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8409e3b0 con 0x7f6b840a5920 2024-08-19T12:45:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient: _finish_auth 0 2024-08-19T12:45:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient: _check_auth_tickets 2024-08-19T12:45:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 1 -- 172.21.15.26:0/2317769708 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6b6c001cb0 con 0x7f6b840a5920 2024-08-19T12:45:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b8b20e640 5 monclient: authenticate success, global_id 26644 2024-08-19T12:45:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b8b20e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:45:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient: dump: 2024-08-19T12:45:09.873 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b8b20e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 1 -- 172.21.15.26:0/2317769708 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6b6c004890 con 0x7f6b840a5920 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b8b20e640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 1 -- 172.21.15.26:0/2317769708 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6b6c004e00 con 0x7f6b840a5920 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b8b20e640 4 set_mon_vals no callback set 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.865+0000 7f6b837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.866+0000 7f6b837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.866+0000 7f6b837fe640 10 monclient: dump: 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.875 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.866+0000 7f6b8b20e640 10 monclient: shutdown 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.866+0000 7f6b8b20e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.866+0000 7f6b8b20e640 1 -- 172.21.15.26:0/2317769708 >> v1:172.21.15.26:6791/0 conn(0x7f6b840a5920 legacy=0x7f6b8413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.866+0000 7f6b8b20e640 1 -- 172.21.15.26:0/2317769708 shutdown_connections 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.866+0000 7f6b8b20e640 1 -- 172.21.15.26:0/2317769708 >> 172.21.15.26:0/2317769708 conn(0x7f6b840a8740 msgr2=0x7f6b840a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.866+0000 7f6b8b20e640 1 -- 172.21.15.26:0/2317769708 shutdown_connections 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.866+0000 7f6b8b20e640 1 -- 172.21.15.26:0/2317769708 wait complete. 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.867+0000 7f6b8b20e640 10 monclient: build_initial_monmap 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.867+0000 7f6b8b20e640 10 monclient: monmap: 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:09.877 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.867+0000 7f6b8b20e640 1 librados: starting msgr at 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.867+0000 7f6b8b20e640 1 librados: starting objecter 2024-08-19T12:45:09.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.867+0000 7f6b8b20e640 1 Processor -- start 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.867+0000 7f6b8b20e640 1 -- start start 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.867+0000 7f6b8b20e640 1 librados: setting wanted keys 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.867+0000 7f6b8b20e640 1 librados: calling monclient init 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.867+0000 7f6b8b20e640 10 monclient: init 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.868+0000 7f6b8b20e640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.868+0000 7f6b8b20e640 10 monclient: _add_conns ranks=[5,8,7,3,6,4,2,1,0] 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.868+0000 7f6b8b20e640 10 monclient(hunting): picked mon.noname-h con 0x7f6b8406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.868+0000 7f6b8b20e640 10 monclient(hunting): picked mon.noname-e con 0x7f6b840a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.868+0000 7f6b8b20e640 10 monclient(hunting): picked mon.noname-i con 0x7f6b840a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.868+0000 7f6b8b20e640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b841392f0 con 0x7f6b8406e7b0 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.868+0000 7f6b8b20e640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b84134600 con 0x7f6b840a5d50 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.868+0000 7f6b8b20e640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8413b650 con 0x7f6b840a5920 2024-08-19T12:45:09.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.868+0000 7f6b8b20e640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:09.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.868+0000 7f6b8b20e640 10 monclient(hunting): authenticate will time out at 1103.956200s 2024-08-19T12:45:09.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.868+0000 7f6b83fff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f6b840a5920 0x7f6b841d3690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50836/0 (socket says 172.21.15.26:50836) 2024-08-19T12:45:09.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.868+0000 7f6b83fff640 1 -- 172.21.15.26:0/974090318 learned_addr learned my addr 172.21.15.26:0/974090318 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b83fff640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.26:6793/0 conn(0x7f6b840a5920 legacy=0x7f6b841d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 client.?.objecter ms_handle_connect 0x7f6b840a5920 2024-08-19T12:45:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b897ab640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.104:6792/0 conn(0x7f6b840a5d50 legacy=0x7f6b841d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 client.?.objecter ms_handle_connect 0x7f6b840a5d50 2024-08-19T12:45:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:09.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b88faa640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.104:6791/0 conn(0x7f6b8406e7b0 legacy=0x7f6b84132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 249155448 0 0) 0x7f6b8413b650 con 0x7f6b840a5920 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): my global_id is 26171 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b64004eb0 con 0x7f6b840a5920 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 client.?.objecter ms_handle_connect 0x7f6b8406e7b0 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2802861160 0 0) 0x7f6b84134600 con 0x7f6b840a5d50 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): my global_id is 135259 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b8413b650 con 0x7f6b840a5d50 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4184469171 0 0) 0x7f6b841392f0 con 0x7f6b8406e7b0 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): my global_id is 26321 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b84134600 con 0x7f6b8406e7b0 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2112780727 0 0) 0x7f6b64004eb0 con 0x7f6b840a5920 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b841392f0 con 0x7f6b840a5920 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6b6c004d80 con 0x7f6b840a5920 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:09.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): dump: 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.887 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1601169590 0 0) 0x7f6b841392f0 con 0x7f6b840a5920 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.104:6792/0 conn(0x7f6b840a5d50 legacy=0x7f6b841d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.104:6791/0 conn(0x7f6b8406e7b0 legacy=0x7f6b84132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient: _finish_hunting 0 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 monclient: found mon.e 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8413adb0 con 0x7f6b840a5920 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient: _finish_auth 0 2024-08-19T12:45:09.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 10 monclient: _check_auth_tickets 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b8b20e640 5 monclient: authenticate success, global_id 26171 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b8b20e640 10 monclient: _renew_subs 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b8b20e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.870+0000 7f6b8b20e640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6b84139e50 con 0x7f6b840a5920 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b8b20e640 10 client.26171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b8b20e640 10 monclient: _renew_subs 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b8b20e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b8b20e640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f6b841387f0 con 0x7f6b840a5920 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b8b20e640 1 librados: init done 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6b6c004810 con 0x7f6b840a5920 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b817fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:09.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6b6c001c50 con 0x7f6b840a5920 2024-08-19T12:45:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b82ffd640 4 set_mon_vals no callback set 2024-08-19T12:45:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b817fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b817fa640 10 monclient: dump: 2024-08-19T12:45:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.891 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b8b20e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.871+0000 7f6b8b20e640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6b680065c0 con 0x7f6b68002860 2024-08-19T12:45:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.872+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6b6c0119f0 con 0x7f6b840a5920 2024-08-19T12:45:09.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.872+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(282..282 src has 1..282) ==== 5364+0+0 (unknown 3942427206 0 0) 0x7f6b6c04e2f0 con 0x7f6b840a5920 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.872+0000 7f6b817fa640 10 client.26171.objecter ms_dispatch 0x7f6b84000b90 osd_map(282..282 src has 1..282) 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.872+0000 7f6b817fa640 3 client.26171.objecter handle_osd_map got epochs [282,282] > 0 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.872+0000 7f6b817fa640 3 client.26171.objecter handle_osd_map decoding full epoch 282 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.872+0000 7f6b817fa640 20 client.26171.objecter dump_active .. 0 homeless 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.872+0000 7f6b897ab640 10 monclient: get_auth_request con 0x7f6b64040cd0 auth_method 0 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b88faa640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.26:6793/0 conn(0x7f6b68002860 legacy=0x7f6b68004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 10 client.26171.objecter ms_handle_connect 0x7f6b68002860 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 10 client.26171.objecter resend_mon_ops 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1403917372 0 0) 0x7f6b680065c0 con 0x7f6b68002860 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 10 monclient(hunting): my global_id is 26180 2024-08-19T12:45:09.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b841392f0 con 0x7f6b68002860 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 867971975 0 0) 0x7f6b841392f0 con 0x7f6b68002860 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b680065c0 con 0x7f6b68002860 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6b74005230 con 0x7f6b68002860 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 10 monclient: dump: 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.893 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 471057735 0 0) 0x7f6b680065c0 con 0x7f6b68002860 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6b6800a160 con 0x7f6b68002860 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 10 client.26171.objecter ms_handle_connect 0x7f6b64040cd0 2024-08-19T12:45:09.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6b74005360 con 0x7f6b68002860 2024-08-19T12:45:09.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:09.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:09.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.26:6793/0 conn(0x7f6b68002860 legacy=0x7f6b68004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.874+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.26:6793/0 conn(0x7f6b68002860 legacy=0x7f6b68004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:09.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.880+0000 7f6b8b20e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:09.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.880+0000 7f6b8b20e640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6b68009e50 con 0x7f6b68009110 2024-08-19T12:45:09.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b88faa640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.26:6793/0 conn(0x7f6b68009110 legacy=0x7f6b680095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:09.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 10 client.26171.objecter ms_handle_connect 0x7f6b68009110 2024-08-19T12:45:09.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 10 client.26171.objecter resend_mon_ops 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1973347868 0 0) 0x7f6b68009e50 con 0x7f6b68009110 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 10 monclient(hunting): my global_id is 26189 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b680065c0 con 0x7f6b68009110 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 14125471 0 0) 0x7f6b680065c0 con 0x7f6b68009110 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b68009e50 con 0x7f6b68009110 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6b74005c00 con 0x7f6b68009110 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 10 monclient: dump: 2024-08-19T12:45:09.897 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:09.898 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:09.899 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:09.899 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:09.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2274756961 0 0) 0x7f6b68009e50 con 0x7f6b68009110 2024-08-19T12:45:09.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6b6800c3f0 con 0x7f6b68009110 2024-08-19T12:45:09.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 2011130195) 0x7f6b74008010 con 0x7f6b68009110 2024-08-19T12:45:09.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:09.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:09.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.26:6793/0 conn(0x7f6b68009110 legacy=0x7f6b680095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b817fa640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.26:6793/0 conn(0x7f6b68009110 legacy=0x7f6b680095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:09.899 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 422901, 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:09.900 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:09.901 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:09.902 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:09.903 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:09.904 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:09.908 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.909 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.910 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.911 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.912 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:09.913 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.914 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.915 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.916 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.917 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.918 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:09.919 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:09.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 10 librados: watch_flush enter 2024-08-19T12:45:09.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 10 librados: watch_flush exit 2024-08-19T12:45:09.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 20 client.26171.objecter shutdown clearing up homeless session... 2024-08-19T12:45:09.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 10 client.26171.objecter successfully canceled tick 2024-08-19T12:45:09.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.104:6800/899887506 conn(0x7f6b64040cd0 legacy=0x7f6b64043190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 10 monclient: shutdown 2024-08-19T12:45:09.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:09.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 1 -- 172.21.15.26:0/974090318 >> v1:172.21.15.26:6793/0 conn(0x7f6b840a5920 legacy=0x7f6b841d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:09.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b897ab640 1 -- 172.21.15.26:0/974090318 reap_dead start 2024-08-19T12:45:09.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b897ab640 1 -- 172.21.15.26:0/974090318 reap_dead start 2024-08-19T12:45:09.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 1 -- 172.21.15.26:0/974090318 shutdown_connections 2024-08-19T12:45:09.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 1 -- 172.21.15.26:0/974090318 >> 172.21.15.26:0/974090318 conn(0x7f6b840a8740 msgr2=0x7f6b84135140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:09.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 1 -- 172.21.15.26:0/974090318 shutdown_connections 2024-08-19T12:45:09.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 1 -- 172.21.15.26:0/974090318 wait complete. 2024-08-19T12:45:09.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:09.884+0000 7f6b8b20e640 1 librados: shutdown 2024-08-19T12:45:09.923 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:09.923 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:45:09.923 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:45:10.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.021+0000 7f64cb74c640 10 monclient: get_monmap_and_config 2024-08-19T12:45:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.021+0000 7f64cb74c640 10 monclient: build_initial_monmap 2024-08-19T12:45:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.021+0000 7f64cb74c640 10 monclient: monmap: 2024-08-19T12:45:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:10.022928+0000 2024-08-19T12:45:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:10.022928+0000 2024-08-19T12:45:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:10.023 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:10.024 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:10.024 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:10.024 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:10.024 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:10.024 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:10.024 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.021+0000 7f64cb74c640 1 Processor -- start 2024-08-19T12:45:10.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.021+0000 7f64cb74c640 1 -- start start 2024-08-19T12:45:10.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.021+0000 7f64cb74c640 10 monclient: init 2024-08-19T12:45:10.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.022+0000 7f64cb74c640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.022+0000 7f64cb74c640 10 monclient: _add_conns ranks=[8,2,0,7,5,1,3,6,4] 2024-08-19T12:45:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.022+0000 7f64cb74c640 10 monclient(hunting): picked mon.noname-e con 0x7f64c4073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.022+0000 7f64cb74c640 10 monclient(hunting): picked mon.noname-b con 0x7f64c40a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.022+0000 7f64cb74c640 10 monclient(hunting): picked mon.noname-a con 0x7f64c413f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.022+0000 7f64cb74c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c409dae0 con 0x7f64c413f2e0 2024-08-19T12:45:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.022+0000 7f64cb74c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c4136910 con 0x7f64c40a57a0 2024-08-19T12:45:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.022+0000 7f64cb74c640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c409dd50 con 0x7f64c4073470 2024-08-19T12:45:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.022+0000 7f64cb74c640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.022+0000 7f64cb74c640 10 monclient(hunting): authenticate will time out at 1104.110441s 2024-08-19T12:45:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.022+0000 7f64c94e8640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f64c4073470 0x7f64c4136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50854/0 (socket says 172.21.15.26:50854) 2024-08-19T12:45:10.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.022+0000 7f64c94e8640 1 -- 172.21.15.26:0/2039547029 learned_addr learned my addr 172.21.15.26:0/2039547029 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:10.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.024+0000 7f64c94e8640 1 -- 172.21.15.26:0/2039547029 >> v1:172.21.15.26:6793/0 conn(0x7f64c4073470 legacy=0x7f64c4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.024+0000 7f64c8ce7640 1 -- 172.21.15.26:0/2039547029 >> v1:172.21.15.26:6790/0 conn(0x7f64c40a57a0 legacy=0x7f64c40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.024+0000 7f64c9ce9640 1 -- 172.21.15.26:0/2039547029 >> v1:172.21.15.26:6789/0 conn(0x7f64c413f2e0 legacy=0x7f64c41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.024+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2321002507 0 0) 0x7f64c409dd50 con 0x7f64c4073470 2024-08-19T12:45:10.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient(hunting): my global_id is 26198 2024-08-19T12:45:10.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64ac003520 con 0x7f64c4073470 2024-08-19T12:45:10.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4081584453 0 0) 0x7f64c4136910 con 0x7f64c40a57a0 2024-08-19T12:45:10.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient(hunting): my global_id is 26300 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64c409dd50 con 0x7f64c40a57a0 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660359611 0 0) 0x7f64c409dae0 con 0x7f64c413f2e0 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient(hunting): my global_id is 144576 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64c4136910 con 0x7f64c413f2e0 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1511717853 0 0) 0x7f64ac003520 con 0x7f64c4073470 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 >> v1:172.21.15.26:6790/0 conn(0x7f64c40a57a0 legacy=0x7f64c40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 >> v1:172.21.15.26:6789/0 conn(0x7f64c413f2e0 legacy=0x7f64c41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient: _finish_hunting 0 2024-08-19T12:45:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 monclient: found mon.noname-e 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c409e030 con 0x7f64c4073470 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient: _finish_auth 0 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient: _check_auth_tickets 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64b4001e60 con 0x7f64c4073470 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient: dump: 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64cb74c640 5 monclient: authenticate success, global_id 26198 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64cb74c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:10.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64bc001a50 con 0x7f64c413f2e0 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f64b4004c20 con 0x7f64c4073470 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 1 -- 172.21.15.26:0/2039547029 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64b4005190 con 0x7f64c4073470 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64c3fff640 10 monclient: dump: 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64cb74c640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64cb74c640 4 set_mon_vals no callback set 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64cb74c640 10 monclient: shutdown 2024-08-19T12:45:10.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64cb74c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64cb74c640 1 -- 172.21.15.26:0/2039547029 >> v1:172.21.15.26:6793/0 conn(0x7f64c4073470 legacy=0x7f64c4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.025+0000 7f64cb74c640 1 -- 172.21.15.26:0/2039547029 shutdown_connections 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.026+0000 7f64cb74c640 1 -- 172.21.15.26:0/2039547029 >> 172.21.15.26:0/2039547029 conn(0x7f64c40abff0 msgr2=0x7f64c40ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.026+0000 7f64cb74c640 1 -- 172.21.15.26:0/2039547029 shutdown_connections 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.026+0000 7f64cb74c640 1 -- 172.21.15.26:0/2039547029 wait complete. 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.026+0000 7f64cb74c640 10 monclient: build_initial_monmap 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.026+0000 7f64cb74c640 10 monclient: monmap: 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:10.040 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.026+0000 7f64cb74c640 1 librados: starting msgr at 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.026+0000 7f64cb74c640 1 librados: starting objecter 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.026+0000 7f64cb74c640 1 Processor -- start 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 1 -- start start 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 1 librados: setting wanted keys 2024-08-19T12:45:10.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 1 librados: calling monclient init 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 10 monclient: init 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 10 monclient: _add_conns ranks=[2,3,8,7,1,0,4,6,5] 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 10 monclient(hunting): picked mon.noname-b con 0x7f64c4073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 10 monclient(hunting): picked mon.noname-g con 0x7f64c40a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 10 monclient(hunting): picked mon.noname-e con 0x7f64c413f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c413f070 con 0x7f64c4073470 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c412f3d0 con 0x7f64c40a57a0 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c412e8d0 con 0x7f64c413f2e0 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64cb74c640 10 monclient(hunting): authenticate will time out at 1104.115455s 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64c94e8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f64c4073470 0x7f64c412dab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40088/0 (socket says 172.21.15.26:40088) 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.027+0000 7f64c94e8640 1 -- 172.21.15.26:0/3153058188 learned_addr learned my addr 172.21.15.26:0/3153058188 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:10.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c94e8640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.26:6790/0 conn(0x7f64c4073470 legacy=0x7f64c412dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c9ce9640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.26:6793/0 conn(0x7f64c413f2e0 legacy=0x7f64c41d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 client.?.objecter ms_handle_connect 0x7f64c4073470 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 client.?.objecter ms_handle_connect 0x7f64c413f2e0 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 client.?.objecter ms_handle_connect 0x7f64c40a57a0 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c8ce7640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.104:6790/0 conn(0x7f64c40a57a0 legacy=0x7f64c412e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 975508145 0 0) 0x7f64c413f070 con 0x7f64c4073470 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 monclient(hunting): my global_id is 26309 2024-08-19T12:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64b0004eb0 con 0x7f64c4073470 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298763568 0 0) 0x7f64c412e8d0 con 0x7f64c413f2e0 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 10 monclient(hunting): my global_id is 26207 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.029+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64c413f070 con 0x7f64c413f2e0 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4046617486 0 0) 0x7f64c413f070 con 0x7f64c413f2e0 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64c412e8d0 con 0x7f64c413f2e0 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64bc003950 con 0x7f64c413f2e0 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient(hunting): dump: 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.045 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3513927624 0 0) 0x7f64b0004eb0 con 0x7f64c4073470 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64c413f070 con 0x7f64c4073470 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64b4005110 con 0x7f64c4073470 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient(hunting): dump: 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.046 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1835046849 0 0) 0x7f64c412e8d0 con 0x7f64c413f2e0 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.104:6790/0 conn(0x7f64c40a57a0 legacy=0x7f64c412e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.26:6790/0 conn(0x7f64c4073470 legacy=0x7f64c412dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:45:10.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 monclient: found mon.e 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c4132370 con 0x7f64c413f2e0 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient: _finish_auth 0 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64cb74c640 5 monclient: authenticate success, global_id 26207 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64cb74c640 10 monclient: _renew_subs 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64cb74c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64cb74c640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f64c4130e90 con 0x7f64c413f2e0 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f64bc001ce0 con 0x7f64c413f2e0 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64bc003f30 con 0x7f64c413f2e0 2024-08-19T12:45:10.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c1ffb640 10 monclient: dump: 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.049 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64cb74c640 10 client.26207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64cb74c640 10 monclient: _renew_subs 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64cb74c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64cb74c640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f64c4130110 con 0x7f64c413f2e0 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64cb74c640 1 librados: init done 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.030+0000 7f64c37fe640 4 set_mon_vals no callback set 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.031+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f64bc0040d0 con 0x7f64c413f2e0 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.031+0000 7f64cb74c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.031+0000 7f64cb74c640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f64c4131270 con 0x7f64c4142df0 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.031+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(282..282 src has 1..282) ==== 5364+0+0 (unknown 3942427206 0 0) 0x7f64bc04df30 con 0x7f64c413f2e0 2024-08-19T12:45:10.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.031+0000 7f64c1ffb640 10 client.26207.objecter ms_dispatch 0x7f64c4000b90 osd_map(282..282 src has 1..282) 2024-08-19T12:45:10.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.031+0000 7f64c1ffb640 3 client.26207.objecter handle_osd_map got epochs [282,282] > 0 2024-08-19T12:45:10.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.031+0000 7f64c1ffb640 3 client.26207.objecter handle_osd_map decoding full epoch 282 2024-08-19T12:45:10.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.031+0000 7f64c1ffb640 20 client.26207.objecter dump_active .. 0 homeless 2024-08-19T12:45:10.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.031+0000 7f64c8ce7640 10 monclient: get_auth_request con 0x7f64b00414e0 auth_method 0 2024-08-19T12:45:10.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.033+0000 7f64c94e8640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.104:6789/0 conn(0x7f64c4142df0 legacy=0x7f64c4134790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.033+0000 7f64c1ffb640 10 client.26207.objecter ms_handle_connect 0x7f64c4142df0 2024-08-19T12:45:10.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.033+0000 7f64c1ffb640 10 client.26207.objecter resend_mon_ops 2024-08-19T12:45:10.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.033+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2442303332 0 0) 0x7f64c4131270 con 0x7f64c4142df0 2024-08-19T12:45:10.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.033+0000 7f64c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.033+0000 7f64c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.033+0000 7f64c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.033+0000 7f64c1ffb640 10 monclient(hunting): my global_id is 27091 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.033+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64c412e8d0 con 0x7f64c4142df0 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.033+0000 7f64c1ffb640 10 client.26207.objecter ms_handle_connect 0x7f64b00414e0 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1476789050 0 0) 0x7f64c412e8d0 con 0x7f64c4142df0 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64c4131270 con 0x7f64c4142df0 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64b40020b0 con 0x7f64c4142df0 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 10 monclient: dump: 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.054 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.055 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 636617008 0 0) 0x7f64c4131270 con 0x7f64c4142df0 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64c4131690 con 0x7f64c4142df0 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f64b4005810 con 0x7f64c4142df0 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.104:6789/0 conn(0x7f64c4142df0 legacy=0x7f64c4134790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.034+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.104:6789/0 conn(0x7f64c4142df0 legacy=0x7f64c4134790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.040+0000 7f64cb74c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.040+0000 7f64cb74c640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f64c4133270 con 0x7f64c4132e90 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c94e8640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.104:6789/0 conn(0x7f64c4132e90 legacy=0x7f64c4134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 10 client.26207.objecter ms_handle_connect 0x7f64c4132e90 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 10 client.26207.objecter resend_mon_ops 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690111003 0 0) 0x7f64c4133270 con 0x7f64c4132e90 2024-08-19T12:45:10.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 10 monclient(hunting): my global_id is 27100 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64c4131270 con 0x7f64c4132e90 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2562640995 0 0) 0x7f64c4131270 con 0x7f64c4132e90 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64c4133270 con 0x7f64c4132e90 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f64b4006030 con 0x7f64c4132e90 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 10 monclient: dump: 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.057 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2817731274 0 0) 0x7f64c4133270 con 0x7f64c4132e90 2024-08-19T12:45:10.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f64c406c840 con 0x7f64c4132e90 2024-08-19T12:45:10.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1326734182) 0x7f64b40063e0 con 0x7f64c4132e90 2024-08-19T12:45:10.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:10.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:10.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.104:6789/0 conn(0x7f64c4132e90 legacy=0x7f64c4134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c1ffb640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.104:6789/0 conn(0x7f64c4132e90 legacy=0x7f64c4134ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:10.059 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:10.059 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:10.059 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:10.059 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 423048, 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:10.060 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.061 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:45:10.061 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:10.061 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:10.061 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:10.061 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:10.061 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:10.061 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:10.061 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:10.061 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:10.061 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:10.061 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:10.061 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:10.062 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:10.062 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:10.062 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:10.062 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.062 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:10.062 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:10.062 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:10.062 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:10.062 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:10.062 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:10.062 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:10.062 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:10.063 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:10.064 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.065 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.066 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.067 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.068 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:10.069 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.070 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.072 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:10.073 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.074 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.075 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:10.076 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.079 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:10.079 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:10.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 10 librados: watch_flush enter 2024-08-19T12:45:10.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 10 librados: watch_flush exit 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 20 client.26207.objecter shutdown clearing up homeless session... 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 10 client.26207.objecter successfully canceled tick 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.104:6800/899887506 conn(0x7f64b00414e0 legacy=0x7f64b00439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 10 monclient: shutdown 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 1 -- 172.21.15.26:0/3153058188 >> v1:172.21.15.26:6793/0 conn(0x7f64c413f2e0 legacy=0x7f64c41d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c9ce9640 1 -- 172.21.15.26:0/3153058188 reap_dead start 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64c9ce9640 1 -- 172.21.15.26:0/3153058188 reap_dead start 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 1 -- 172.21.15.26:0/3153058188 shutdown_connections 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 1 -- 172.21.15.26:0/3153058188 >> 172.21.15.26:0/3153058188 conn(0x7f64c40abff0 msgr2=0x7f64c40a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 1 -- 172.21.15.26:0/3153058188 shutdown_connections 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 1 -- 172.21.15.26:0/3153058188 wait complete. 2024-08-19T12:45:10.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.044+0000 7f64cb74c640 1 librados: shutdown 2024-08-19T12:45:10.082 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:10.082 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:45:10.082 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:45:10.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.179+0000 7f933878a640 10 monclient: get_monmap_and_config 2024-08-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.179+0000 7f933878a640 10 monclient: build_initial_monmap 2024-08-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.180+0000 7f933878a640 10 monclient: monmap: 2024-08-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:10.181842+0000 2024-08-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:10.181842+0000 2024-08-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:10.182 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:10.183 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:10.183 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:10.183 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:10.183 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:10.183 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:10.183 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.180+0000 7f933878a640 1 Processor -- start 2024-08-19T12:45:10.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.180+0000 7f933878a640 1 -- start start 2024-08-19T12:45:10.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.180+0000 7f933878a640 10 monclient: init 2024-08-19T12:45:10.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.181+0000 7f933878a640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:10.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.181+0000 7f933878a640 10 monclient: _add_conns ranks=[2,8,4,3,1,0,5,7,6] 2024-08-19T12:45:10.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.181+0000 7f933878a640 10 monclient(hunting): picked mon.noname-b con 0x7f933006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:10.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.181+0000 7f933878a640 10 monclient(hunting): picked mon.noname-e con 0x7f93300a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:10.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.181+0000 7f933878a640 10 monclient(hunting): picked mon.noname-c con 0x7f93300a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:10.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.181+0000 7f933878a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9330136790 con 0x7f933006e7b0 2024-08-19T12:45:10.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.181+0000 7f933878a640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933009ddf0 con 0x7f93300a5d50 2024-08-19T12:45:10.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.181+0000 7f933878a640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933009e0d0 con 0x7f93300a5920 2024-08-19T12:45:10.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.181+0000 7f933878a640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:10.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.181+0000 7f933878a640 10 monclient(hunting): authenticate will time out at 1104.269298s 2024-08-19T12:45:10.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.181+0000 7f9336526640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f933006e7b0 0x7f9330136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40096/0 (socket says 172.21.15.26:40096) 2024-08-19T12:45:10.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.181+0000 7f9336526640 1 -- 172.21.15.26:0/3758901284 learned_addr learned my addr 172.21.15.26:0/3758901284 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:10.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335d25640 1 -- 172.21.15.26:0/3758901284 >> v1:172.21.15.26:6793/0 conn(0x7f93300a5920 legacy=0x7f933013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9336d27640 1 -- 172.21.15.26:0/3758901284 >> v1:172.21.15.26:6791/0 conn(0x7f93300a5d50 legacy=0x7f9330141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9336526640 1 -- 172.21.15.26:0/3758901284 >> v1:172.21.15.26:6790/0 conn(0x7f933006e7b0 legacy=0x7f9330136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1861920195 0 0) 0x7f933009e0d0 con 0x7f93300a5920 2024-08-19T12:45:10.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 10 monclient(hunting): my global_id is 26216 2024-08-19T12:45:10.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9318003520 con 0x7f93300a5920 2024-08-19T12:45:10.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3444256620 0 0) 0x7f9320001750 con 0x7f93300a5d50 2024-08-19T12:45:10.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 10 monclient(hunting): my global_id is 26653 2024-08-19T12:45:10.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f933009e0d0 con 0x7f93300a5d50 2024-08-19T12:45:10.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3443284851 0 0) 0x7f9330136790 con 0x7f933006e7b0 2024-08-19T12:45:10.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 10 monclient(hunting): my global_id is 26318 2024-08-19T12:45:10.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.183+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9320001750 con 0x7f933006e7b0 2024-08-19T12:45:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4125374544 0 0) 0x7f933009e0d0 con 0x7f93300a5d50 2024-08-19T12:45:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 >> v1:172.21.15.26:6793/0 conn(0x7f93300a5920 legacy=0x7f933013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 >> v1:172.21.15.26:6790/0 conn(0x7f933006e7b0 legacy=0x7f9330136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 10 monclient: _finish_hunting 0 2024-08-19T12:45:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 1 monclient: found mon.noname-c 2024-08-19T12:45:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:45:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933009e3b0 con 0x7f93300a5d50 2024-08-19T12:45:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 10 monclient: _finish_auth 0 2024-08-19T12:45:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 10 monclient: _check_auth_tickets 2024-08-19T12:45:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:10.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9320001f10 con 0x7f93300a5d50 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f933878a640 5 monclient: authenticate success, global_id 26653 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f933878a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 10 monclient: dump: 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.197 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f933878a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9320003990 con 0x7f93300a5d50 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f933878a640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 1 -- 172.21.15.26:0/3758901284 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9320003f00 con 0x7f93300a5d50 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f933878a640 4 set_mon_vals no callback set 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f9335524640 10 monclient: dump: 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f933878a640 10 monclient: shutdown 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f933878a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f933878a640 1 -- 172.21.15.26:0/3758901284 >> v1:172.21.15.26:6791/0 conn(0x7f93300a5d50 legacy=0x7f9330141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f933878a640 1 -- 172.21.15.26:0/3758901284 shutdown_connections 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f933878a640 1 -- 172.21.15.26:0/3758901284 >> 172.21.15.26:0/3758901284 conn(0x7f93300a8740 msgr2=0x7f93300a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f933878a640 1 -- 172.21.15.26:0/3758901284 shutdown_connections 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.184+0000 7f933878a640 1 -- 172.21.15.26:0/3758901284 wait complete. 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.185+0000 7f933878a640 10 monclient: build_initial_monmap 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.185+0000 7f933878a640 10 monclient: monmap: 2024-08-19T12:45:10.200 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:10.201 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.185+0000 7f933878a640 1 librados: starting msgr at 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.185+0000 7f933878a640 1 librados: starting objecter 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.185+0000 7f933878a640 1 Processor -- start 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.185+0000 7f933878a640 1 -- start start 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.185+0000 7f933878a640 1 librados: setting wanted keys 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.185+0000 7f933878a640 1 librados: calling monclient init 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.185+0000 7f933878a640 10 monclient: init 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f933878a640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f933878a640 10 monclient: _add_conns ranks=[5,0,8,2,4,6,1,3,7] 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f933878a640 10 monclient(hunting): picked mon.noname-h con 0x7f933006e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f933878a640 10 monclient(hunting): picked mon.noname-a con 0x7f93300a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f933878a640 10 monclient(hunting): picked mon.noname-e con 0x7f93300a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f933878a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933013b240 con 0x7f93300a5920 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f933878a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933013b520 con 0x7f933006e7b0 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f933878a640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9330139d20 con 0x7f93300a5d50 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f933878a640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f933878a640 10 monclient(hunting): authenticate will time out at 1104.274248s 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f9335d25640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f93300a5920 0x7f93301d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38250/0 (socket says 172.21.15.26:38250) 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f9335d25640 1 -- 172.21.15.26:0/3760891387 learned_addr learned my addr 172.21.15.26:0/3760891387 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.186+0000 7f9336d27640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f93300a5d50 0x7f93301d6bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50882/0 (socket says 172.21.15.26:50882) 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f9336d27640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.26:6793/0 conn(0x7f93300a5d50 legacy=0x7f93301d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 client.?.objecter ms_handle_connect 0x7f93300a5d50 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f9335d25640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.26:6789/0 conn(0x7f93300a5920 legacy=0x7f93301d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 client.?.objecter ms_handle_connect 0x7f93300a5920 2024-08-19T12:45:10.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 112521931 0 0) 0x7f9330139d20 con 0x7f93300a5d50 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f9336526640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.104:6791/0 conn(0x7f933006e7b0 legacy=0x7f93301ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): my global_id is 26225 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f931c004eb0 con 0x7f93300a5d50 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 client.?.objecter ms_handle_connect 0x7f933006e7b0 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3969640222 0 0) 0x7f933013b240 con 0x7f93300a5920 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): my global_id is 144585 2024-08-19T12:45:10.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9330139d20 con 0x7f93300a5920 2024-08-19T12:45:10.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2483462885 0 0) 0x7f931c004eb0 con 0x7f93300a5d50 2024-08-19T12:45:10.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f933013b240 con 0x7f93300a5d50 2024-08-19T12:45:10.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9320003e80 con 0x7f93300a5d50 2024-08-19T12:45:10.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:10.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:10.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): dump: 2024-08-19T12:45:10.206 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.206 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.206 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.206 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:10.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1375268153 0 0) 0x7f933013b520 con 0x7f933006e7b0 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 10 monclient(hunting): my global_id is 26330 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f931c004eb0 con 0x7f933006e7b0 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.188+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3843678927 0 0) 0x7f9330139d20 con 0x7f93300a5920 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f933013b520 con 0x7f93300a5920 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9324003950 con 0x7f93300a5920 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 10 monclient(hunting): dump: 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.208 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.209 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1006387075 0 0) 0x7f933013b240 con 0x7f93300a5d50 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.104:6791/0 conn(0x7f933006e7b0 legacy=0x7f93301ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.26:6789/0 conn(0x7f93300a5920 legacy=0x7f93301d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 1 monclient: found mon.e 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93301391c0 con 0x7f93300a5d50 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 10 monclient: _finish_auth 0 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f933878a640 5 monclient: authenticate success, global_id 26225 2024-08-19T12:45:10.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f933878a640 10 monclient: _renew_subs 2024-08-19T12:45:10.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f933878a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:10.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f933878a640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f933013a9a0 con 0x7f93300a5d50 2024-08-19T12:45:10.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f933878a640 10 client.26225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:10.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f933878a640 10 monclient: _renew_subs 2024-08-19T12:45:10.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f933878a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:45:10.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f933878a640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f933013bed0 con 0x7f93300a5d50 2024-08-19T12:45:10.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f933878a640 1 librados: init done 2024-08-19T12:45:10.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9320001750 con 0x7f93300a5d50 2024-08-19T12:45:10.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:10.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9320004610 con 0x7f93300a5d50 2024-08-19T12:45:10.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f9334d23640 4 set_mon_vals no callback set 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.189+0000 7f932effd640 10 monclient: dump: 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.190+0000 7f933878a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.190+0000 7f933878a640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f93100065c0 con 0x7f9310002860 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.190+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f9320011e00 con 0x7f93300a5d50 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.190+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(282..282 src has 1..282) ==== 5364+0+0 (unknown 3942427206 0 0) 0x7f932004e270 con 0x7f93300a5d50 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.190+0000 7f932effd640 10 client.26225.objecter ms_dispatch 0x7f9330000b90 osd_map(282..282 src has 1..282) 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.190+0000 7f932effd640 3 client.26225.objecter handle_osd_map got epochs [282,282] > 0 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.190+0000 7f932effd640 3 client.26225.objecter handle_osd_map decoding full epoch 282 2024-08-19T12:45:10.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.190+0000 7f932effd640 20 client.26225.objecter dump_active .. 0 homeless 2024-08-19T12:45:10.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.190+0000 7f9335d25640 10 monclient: get_auth_request con 0x7f931c041680 auth_method 0 2024-08-19T12:45:10.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.192+0000 7f9336526640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.104:6790/0 conn(0x7f9310002860 legacy=0x7f9310004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.192+0000 7f932effd640 10 client.26225.objecter ms_handle_connect 0x7f9310002860 2024-08-19T12:45:10.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.192+0000 7f932effd640 10 client.26225.objecter resend_mon_ops 2024-08-19T12:45:10.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.192+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3152733701 0 0) 0x7f93100065c0 con 0x7f9310002860 2024-08-19T12:45:10.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.192+0000 7f932effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.192+0000 7f932effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.192+0000 7f932effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.192+0000 7f932effd640 10 monclient(hunting): my global_id is 154191 2024-08-19T12:45:10.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.192+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f933013b240 con 0x7f9310002860 2024-08-19T12:45:10.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 10 client.26225.objecter ms_handle_connect 0x7f931c041680 2024-08-19T12:45:10.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 731952097 0 0) 0x7f933013b240 con 0x7f9310002860 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93100065c0 con 0x7f9310002860 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9328001320 con 0x7f9310002860 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 10 monclient: dump: 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.220 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3355821689 0 0) 0x7f93100065c0 con 0x7f9310002860 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f931000a160 con 0x7f9310002860 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9328004910 con 0x7f9310002860 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:10.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.104:6790/0 conn(0x7f9310002860 legacy=0x7f9310004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.193+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.104:6790/0 conn(0x7f9310002860 legacy=0x7f9310004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:10.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.200+0000 7f933878a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:10.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.200+0000 7f933878a640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9310009e50 con 0x7f9310009110 2024-08-19T12:45:10.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.202+0000 7f9336526640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.104:6790/0 conn(0x7f9310009110 legacy=0x7f93100095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.202+0000 7f932effd640 10 client.26225.objecter ms_handle_connect 0x7f9310009110 2024-08-19T12:45:10.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.202+0000 7f932effd640 10 client.26225.objecter resend_mon_ops 2024-08-19T12:45:10.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4115932125 0 0) 0x7f9310009e50 con 0x7f9310009110 2024-08-19T12:45:10.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 10 monclient(hunting): my global_id is 154200 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93100065c0 con 0x7f9310009110 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 889993870 0 0) 0x7f93100065c0 con 0x7f9310009110 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9310009e50 con 0x7f9310009110 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9328005190 con 0x7f9310009110 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 10 monclient: dump: 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.223 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 950293345 0 0) 0x7f9310009e50 con 0x7f9310009110 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f931000c3f0 con 0x7f9310009110 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 1152473148) 0x7f93280074f0 con 0x7f9310009110 2024-08-19T12:45:10.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:10.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:10.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.104:6790/0 conn(0x7f9310009110 legacy=0x7f93100095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.203+0000 7f932effd640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.104:6790/0 conn(0x7f9310009110 legacy=0x7f93100095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:10.225 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:10.225 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 21933, 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.226 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:10.227 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:10.228 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:10.229 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:10.230 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.231 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.232 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.233 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:10.235 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.236 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:10.238 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:10.238 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.238 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.238 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.238 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:10.238 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:10.238 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:10.240 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.241 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:10.244 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.245 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:10.245 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 10 librados: watch_flush enter 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 10 librados: watch_flush exit 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 20 client.26225.objecter shutdown clearing up homeless session... 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 10 client.26225.objecter successfully canceled tick 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.104:6800/899887506 conn(0x7f931c041680 legacy=0x7f931c043b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 10 monclient: shutdown 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 1 -- 172.21.15.26:0/3760891387 >> v1:172.21.15.26:6793/0 conn(0x7f93300a5d50 legacy=0x7f93301d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f9336d27640 1 -- 172.21.15.26:0/3760891387 reap_dead start 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f9336d27640 1 -- 172.21.15.26:0/3760891387 reap_dead start 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 1 -- 172.21.15.26:0/3760891387 shutdown_connections 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 1 -- 172.21.15.26:0/3760891387 >> 172.21.15.26:0/3760891387 conn(0x7f93300a8740 msgr2=0x7f93300aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:10.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 1 -- 172.21.15.26:0/3760891387 shutdown_connections 2024-08-19T12:45:10.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 1 -- 172.21.15.26:0/3760891387 wait complete. 2024-08-19T12:45:10.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.204+0000 7f933878a640 1 librados: shutdown 2024-08-19T12:45:10.248 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:10.248 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:45:10.248 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:45:10.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.347+0000 7fd06559f640 10 monclient: get_monmap_and_config 2024-08-19T12:45:10.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.347+0000 7fd06559f640 10 monclient: build_initial_monmap 2024-08-19T12:45:10.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.347+0000 7fd06559f640 10 monclient: monmap: 2024-08-19T12:45:10.350 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:10.350 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.350 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:10.349786+0000 2024-08-19T12:45:10.350 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:10.349786+0000 2024-08-19T12:45:10.350 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:10.350 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.350 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:10.351 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:10.351 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:10.351 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:10.351 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:10.351 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:10.351 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:10.351 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:10.351 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:10.351 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.348+0000 7fd06559f640 1 Processor -- start 2024-08-19T12:45:10.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.348+0000 7fd06559f640 1 -- start start 2024-08-19T12:45:10.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.348+0000 7fd06559f640 10 monclient: init 2024-08-19T12:45:10.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.349+0000 7fd06559f640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:10.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.349+0000 7fd06559f640 10 monclient: _add_conns ranks=[0,7,8,1,5,4,3,2,6] 2024-08-19T12:45:10.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.349+0000 7fd06559f640 10 monclient(hunting): picked mon.noname-a con 0x7fd06006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:10.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.349+0000 7fd06559f640 10 monclient(hunting): picked mon.noname-i con 0x7fd0600a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:10.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.349+0000 7fd06559f640 10 monclient(hunting): picked mon.noname-e con 0x7fd0600a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.349+0000 7fd06559f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd060136790 con 0x7fd06006e7b0 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.349+0000 7fd06559f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd06009ddf0 con 0x7fd0600a5920 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.349+0000 7fd05effd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd06006e7b0 0x7fd060136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38258/0 (socket says 172.21.15.26:38258) 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.349+0000 7fd05effd640 1 -- 172.21.15.26:0/1556043724 learned_addr learned my addr 172.21.15.26:0/1556043724 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.349+0000 7fd06559f640 1 -- 172.21.15.26:0/1556043724 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd06009e0d0 con 0x7fd0600a5d50 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.349+0000 7fd06559f640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.349+0000 7fd06559f640 10 monclient(hunting): authenticate will time out at 1104.437313s 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05f7fe640 1 -- 172.21.15.26:0/1556043724 >> v1:172.21.15.26:6793/0 conn(0x7fd0600a5d50 legacy=0x7fd060141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05effd640 1 -- 172.21.15.26:0/1556043724 >> v1:172.21.15.26:6789/0 conn(0x7fd06006e7b0 legacy=0x7fd060136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05e7fc640 1 -- 172.21.15.26:0/1556043724 >> v1:172.21.15.104:6792/0 conn(0x7fd0600a5920 legacy=0x7fd06013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4020629666 0 0) 0x7fd060136790 con 0x7fd06006e7b0 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient(hunting): my global_id is 144594 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd04c003520 con 0x7fd06006e7b0 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 650913146 0 0) 0x7fd06009e0d0 con 0x7fd0600a5d50 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient(hunting): my global_id is 26234 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd060136790 con 0x7fd0600a5d50 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1308756015 0 0) 0x7fd0480014a0 con 0x7fd0600a5920 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient(hunting): my global_id is 135268 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd06009e0d0 con 0x7fd0600a5920 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2796750652 0 0) 0x7fd060136790 con 0x7fd0600a5d50 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 >> v1:172.21.15.104:6792/0 conn(0x7fd0600a5920 legacy=0x7fd06013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 >> v1:172.21.15.26:6789/0 conn(0x7fd06006e7b0 legacy=0x7fd060136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 1 monclient: found mon.noname-e 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06009e3b0 con 0x7fd0600a5d50 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient: _finish_auth 0 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd06559f640 5 monclient: authenticate success, global_id 26234 2024-08-19T12:45:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd06559f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd050001a90 con 0x7fd0600a5d50 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.351+0000 7fd05dffb640 10 monclient: dump: 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.360 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd06559f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd050003610 con 0x7fd0600a5d50 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd05dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd05dffb640 1 -- 172.21.15.26:0/1556043724 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd050003b80 con 0x7fd0600a5d50 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd06559f640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd06559f640 4 set_mon_vals no callback set 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd05dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd05dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd05dffb640 10 monclient: dump: 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.363 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.363 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.363 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd06559f640 10 monclient: shutdown 2024-08-19T12:45:10.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd06559f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:10.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd06559f640 1 -- 172.21.15.26:0/1556043724 >> v1:172.21.15.26:6793/0 conn(0x7fd0600a5d50 legacy=0x7fd060141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd06559f640 1 -- 172.21.15.26:0/1556043724 shutdown_connections 2024-08-19T12:45:10.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd06559f640 1 -- 172.21.15.26:0/1556043724 >> 172.21.15.26:0/1556043724 conn(0x7fd0600a8740 msgr2=0x7fd0600a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd06559f640 1 -- 172.21.15.26:0/1556043724 shutdown_connections 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.352+0000 7fd06559f640 1 -- 172.21.15.26:0/1556043724 wait complete. 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.353+0000 7fd06559f640 10 monclient: build_initial_monmap 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.353+0000 7fd06559f640 10 monclient: monmap: 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.353+0000 7fd06559f640 1 librados: starting msgr at 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.353+0000 7fd06559f640 1 librados: starting objecter 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.353+0000 7fd06559f640 1 Processor -- start 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.353+0000 7fd06559f640 1 -- start start 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.353+0000 7fd06559f640 1 librados: setting wanted keys 2024-08-19T12:45:10.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.353+0000 7fd06559f640 1 librados: calling monclient init 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.353+0000 7fd06559f640 10 monclient: init 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.354+0000 7fd06559f640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.354+0000 7fd06559f640 10 monclient: _add_conns ranks=[7,6,3,1,8,4,2,5,0] 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.354+0000 7fd06559f640 10 monclient(hunting): picked mon.noname-i con 0x7fd06006e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.354+0000 7fd06559f640 10 monclient(hunting): picked mon.noname-d con 0x7fd0600a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.354+0000 7fd06559f640 10 monclient(hunting): picked mon.noname-g con 0x7fd0600a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.354+0000 7fd06559f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0601349e0 con 0x7fd0600a5d50 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.354+0000 7fd06559f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd060134600 con 0x7fd0600a5920 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.354+0000 7fd06559f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0601391d0 con 0x7fd06006e7b0 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.354+0000 7fd06559f640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.354+0000 7fd06559f640 10 monclient(hunting): authenticate will time out at 1104.442317s 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.354+0000 7fd05e7fc640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fd0600a5920 0x7fd0601d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:55184/0 (socket says 172.21.15.26:55184) 2024-08-19T12:45:10.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.354+0000 7fd05e7fc640 1 -- 172.21.15.26:0/4056483706 learned_addr learned my addr 172.21.15.26:0/4056483706 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:10.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.356+0000 7fd05e7fc640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.26:6792/0 conn(0x7fd0600a5920 legacy=0x7fd0601d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.356+0000 7fd0577fe640 10 client.?.objecter ms_handle_connect 0x7fd0600a5920 2024-08-19T12:45:10.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.356+0000 7fd0577fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:10.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.356+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3856775347 0 0) 0x7fd060134600 con 0x7fd0600a5920 2024-08-19T12:45:10.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.356+0000 7fd0577fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.356+0000 7fd0577fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.356+0000 7fd0577fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.356+0000 7fd0577fe640 10 monclient(hunting): my global_id is 26664 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.356+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd040004eb0 con 0x7fd0600a5920 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.356+0000 7fd05effd640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.104:6792/0 conn(0x7fd06006e7b0 legacy=0x7fd060132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.356+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.356+0000 7fd05f7fe640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.104:6790/0 conn(0x7fd0600a5d50 legacy=0x7fd0601d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 client.?.objecter ms_handle_connect 0x7fd06006e7b0 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 client.?.objecter ms_handle_connect 0x7fd0600a5d50 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3329558624 0 0) 0x7fd040004eb0 con 0x7fd0600a5920 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd060134600 con 0x7fd0600a5920 2024-08-19T12:45:10.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4254236648 0 0) 0x7fd06009e3b0 con 0x7fd0600a5d50 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 monclient(hunting): my global_id is 154209 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd040004eb0 con 0x7fd0600a5d50 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 57574442 0 0) 0x7fd0601391d0 con 0x7fd06006e7b0 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 monclient(hunting): my global_id is 135277 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd06009e3b0 con 0x7fd06006e7b0 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd048003950 con 0x7fd0600a5920 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:10.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:45:10.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 monclient(hunting): dump: 2024-08-19T12:45:10.373 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.373 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.373 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.373 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.373 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.373 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.373 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.358+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3208028117 0 0) 0x7fd060134600 con 0x7fd0600a5920 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.104:6792/0 conn(0x7fd06006e7b0 legacy=0x7fd060132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.104:6790/0 conn(0x7fd0600a5d50 legacy=0x7fd0601d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 10 monclient: _finish_hunting 0 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 1 monclient: found mon.d 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06013b530 con 0x7fd0600a5920 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 10 monclient: _finish_auth 0 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 10 monclient: _check_auth_tickets 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd06559f640 5 monclient: authenticate success, global_id 26664 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd06559f640 10 monclient: _renew_subs 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd06559f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd06559f640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd06013a960 con 0x7fd0600a5920 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd06559f640 10 client.26664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd06559f640 10 monclient: _renew_subs 2024-08-19T12:45:10.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd06559f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd06559f640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fd060133160 con 0x7fd0600a5920 2024-08-19T12:45:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd06559f640 1 librados: init done 2024-08-19T12:45:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd048001c60 con 0x7fd0600a5920 2024-08-19T12:45:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd048004290 con 0x7fd0600a5920 2024-08-19T12:45:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd05d7fa640 4 set_mon_vals no callback set 2024-08-19T12:45:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 10 monclient: dump: 2024-08-19T12:45:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.376 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.377 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.359+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd048011a80 con 0x7fd0600a5920 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.360+0000 7fd06559f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.360+0000 7fd06559f640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd0440065c0 con 0x7fd044002860 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.360+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(282..282 src has 1..282) ==== 5364+0+0 (unknown 3942427206 0 0) 0x7fd04804e300 con 0x7fd0600a5920 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.360+0000 7fd0577fe640 10 client.26664.objecter ms_dispatch 0x7fd060000b90 osd_map(282..282 src has 1..282) 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.360+0000 7fd0577fe640 3 client.26664.objecter handle_osd_map got epochs [282,282] > 0 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.360+0000 7fd0577fe640 3 client.26664.objecter handle_osd_map decoding full epoch 282 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.360+0000 7fd0577fe640 20 client.26664.objecter dump_active .. 0 homeless 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.360+0000 7fd05f7fe640 10 monclient: get_auth_request con 0x7fd040041280 auth_method 0 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd05effd640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.104:6791/0 conn(0x7fd044002860 legacy=0x7fd044004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 10 client.26664.objecter ms_handle_connect 0x7fd044002860 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 10 client.26664.objecter resend_mon_ops 2024-08-19T12:45:10.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1587123022 0 0) 0x7fd0440065c0 con 0x7fd044002860 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 10 monclient(hunting): my global_id is 26339 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd060134600 con 0x7fd044002860 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 10 client.26664.objecter ms_handle_connect 0x7fd040041280 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 494393330 0 0) 0x7fd060134600 con 0x7fd044002860 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0440065c0 con 0x7fd044002860 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd058001eb0 con 0x7fd044002860 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.362+0000 7fd0577fe640 10 monclient: dump: 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.379 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.380 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.363+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2651216987 0 0) 0x7fd0440065c0 con 0x7fd044002860 2024-08-19T12:45:10.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.363+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd04400a160 con 0x7fd044002860 2024-08-19T12:45:10.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.363+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd058004ce0 con 0x7fd044002860 2024-08-19T12:45:10.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.363+0000 7fd0577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:10.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.363+0000 7fd0577fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:10.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.363+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.104:6791/0 conn(0x7fd044002860 legacy=0x7fd044004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.363+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.104:6791/0 conn(0x7fd044002860 legacy=0x7fd044004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:10.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.369+0000 7fd06559f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:10.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.369+0000 7fd06559f640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd044009e70 con 0x7fd044009130 2024-08-19T12:45:10.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd05effd640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.104:6791/0 conn(0x7fd044009130 legacy=0x7fd0440095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 10 client.26664.objecter ms_handle_connect 0x7fd044009130 2024-08-19T12:45:10.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 10 client.26664.objecter resend_mon_ops 2024-08-19T12:45:10.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 199335950 0 0) 0x7fd044009e70 con 0x7fd044009130 2024-08-19T12:45:10.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 10 monclient(hunting): my global_id is 26348 2024-08-19T12:45:10.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0440065c0 con 0x7fd044009130 2024-08-19T12:45:10.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 844650218 0 0) 0x7fd0440065c0 con 0x7fd044009130 2024-08-19T12:45:10.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd044009e70 con 0x7fd044009130 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd0580054d0 con 0x7fd044009130 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 10 monclient: dump: 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.384 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3867451329 0 0) 0x7fd044009e70 con 0x7fd044009130 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd04400c410 con 0x7fd044009130 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1701560773) 0x7fd058006a20 con 0x7fd044009130 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.104:6791/0 conn(0x7fd044009130 legacy=0x7fd0440095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd0577fe640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.104:6791/0 conn(0x7fd044009130 legacy=0x7fd0440095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:10.386 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:10.386 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:10.386 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:10.386 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:10.386 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:10.386 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 423396, 2024-08-19T12:45:10.386 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:10.386 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:10.387 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:10.388 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:10.389 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:10.389 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:10.389 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:10.389 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:10.389 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:10.389 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:10.389 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:10.389 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:10.389 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:10.389 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:10.389 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:10.389 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.390 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:10.390 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:10.390 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:10.390 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:10.390 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:10.390 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:10.390 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:10.390 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:10.391 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:10.392 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:10.392 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:10.392 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:10.392 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:10.392 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:10.392 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:10.392 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.392 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:10.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.392 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:10.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.392 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.393 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.394 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.394 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:10.394 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:10.394 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.394 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.394 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.394 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:10.394 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.394 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.396 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.397 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.399 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.400 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:10.401 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.402 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.403 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.404 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.406 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:10.407 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:10.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 10 librados: watch_flush enter 2024-08-19T12:45:10.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 10 librados: watch_flush exit 2024-08-19T12:45:10.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 20 client.26664.objecter shutdown clearing up homeless session... 2024-08-19T12:45:10.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 10 client.26664.objecter successfully canceled tick 2024-08-19T12:45:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.104:6800/899887506 conn(0x7fd040041280 legacy=0x7fd040043740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 10 monclient: shutdown 2024-08-19T12:45:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 1 -- 172.21.15.26:0/4056483706 >> v1:172.21.15.26:6792/0 conn(0x7fd0600a5920 legacy=0x7fd0601d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd05f7fe640 1 -- 172.21.15.26:0/4056483706 reap_dead start 2024-08-19T12:45:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd05f7fe640 1 -- 172.21.15.26:0/4056483706 reap_dead start 2024-08-19T12:45:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 1 -- 172.21.15.26:0/4056483706 shutdown_connections 2024-08-19T12:45:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 1 -- 172.21.15.26:0/4056483706 >> 172.21.15.26:0/4056483706 conn(0x7fd0600a8740 msgr2=0x7fd060135140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 1 -- 172.21.15.26:0/4056483706 shutdown_connections 2024-08-19T12:45:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 1 -- 172.21.15.26:0/4056483706 wait complete. 2024-08-19T12:45:10.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.373+0000 7fd06559f640 1 librados: shutdown 2024-08-19T12:45:10.410 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:10.410 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:45:10.410 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:45:10.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.507+0000 7f65517aa640 10 monclient: get_monmap_and_config 2024-08-19T12:45:10.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.507+0000 7f65517aa640 10 monclient: build_initial_monmap 2024-08-19T12:45:10.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.507+0000 7f65517aa640 10 monclient: monmap: 2024-08-19T12:45:10.550 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:10.550 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.551 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:10.509658+0000 2024-08-19T12:45:10.551 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:10.509658+0000 2024-08-19T12:45:10.551 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:10.551 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.552 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:10.552 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:10.552 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:10.552 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:10.552 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:10.552 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:10.552 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:10.553 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:10.553 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:10.553 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.508+0000 7f65517aa640 1 Processor -- start 2024-08-19T12:45:10.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.508+0000 7f65517aa640 1 -- start start 2024-08-19T12:45:10.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.508+0000 7f65517aa640 10 monclient: init 2024-08-19T12:45:10.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.508+0000 7f65517aa640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:10.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.508+0000 7f65517aa640 10 monclient: _add_conns ranks=[0,1,3,2,6,4,7,5,8] 2024-08-19T12:45:10.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.509+0000 7f65517aa640 10 monclient(hunting): picked mon.noname-a con 0x7f654c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:10.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.509+0000 7f65517aa640 10 monclient(hunting): picked mon.noname-f con 0x7f654c0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:10.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.509+0000 7f65517aa640 10 monclient(hunting): picked mon.noname-g con 0x7f654c0a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:10.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.509+0000 7f65517aa640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f654c136790 con 0x7f654c06e7b0 2024-08-19T12:45:10.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.509+0000 7f65517aa640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f654c09ddf0 con 0x7f654c0a5920 2024-08-19T12:45:10.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.509+0000 7f65517aa640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f654c09e0d0 con 0x7f654c0a5d50 2024-08-19T12:45:10.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.509+0000 7f65517aa640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:10.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.509+0000 7f65517aa640 10 monclient(hunting): authenticate will time out at 1104.597107s 2024-08-19T12:45:10.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.509+0000 7f654affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f654c06e7b0 0x7f654c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38260/0 (socket says 172.21.15.26:38260) 2024-08-19T12:45:10.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.509+0000 7f654affd640 1 -- 172.21.15.26:0/173323826 learned_addr learned my addr 172.21.15.26:0/173323826 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:10.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f654affd640 1 -- 172.21.15.26:0/173323826 >> v1:172.21.15.26:6789/0 conn(0x7f654c06e7b0 legacy=0x7f654c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 1 -- 172.21.15.26:0/173323826 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2135794140 0 0) 0x7f654c136790 con 0x7f654c06e7b0 2024-08-19T12:45:10.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient(hunting): my global_id is 144603 2024-08-19T12:45:10.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 1 -- 172.21.15.26:0/173323826 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6538003520 con 0x7f654c06e7b0 2024-08-19T12:45:10.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f654b7fe640 1 -- 172.21.15.26:0/173323826 >> v1:172.21.15.104:6790/0 conn(0x7f654c0a5d50 legacy=0x7f654c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f654a7fc640 1 -- 172.21.15.26:0/173323826 >> v1:172.21.15.104:6789/0 conn(0x7f654c0a5920 legacy=0x7f654c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 1 -- 172.21.15.26:0/173323826 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1529155505 0 0) 0x7f6538003520 con 0x7f654c06e7b0 2024-08-19T12:45:10.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 1 -- 172.21.15.26:0/173323826 >> v1:172.21.15.104:6790/0 conn(0x7f654c0a5d50 legacy=0x7f654c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 1 -- 172.21.15.26:0/173323826 >> v1:172.21.15.104:6789/0 conn(0x7f654c0a5920 legacy=0x7f654c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:45:10.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 1 monclient: found mon.noname-a 2024-08-19T12:45:10.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:10.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:45:10.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 1 -- 172.21.15.26:0/173323826 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654c09e3b0 con 0x7f654c06e7b0 2024-08-19T12:45:10.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient: _finish_auth 0 2024-08-19T12:45:10.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:45:10.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:10.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 1 -- 172.21.15.26:0/173323826 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6544001e60 con 0x7f654c06e7b0 2024-08-19T12:45:10.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:45:10.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient: dump: 2024-08-19T12:45:10.559 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.559 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.559 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.560 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.560 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.560 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.560 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.560 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.560 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.560 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.561 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.561 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.561 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.561 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.561 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.562 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f65517aa640 5 monclient: authenticate success, global_id 144603 2024-08-19T12:45:10.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f65517aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:10.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 1 -- 172.21.15.26:0/173323826 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6544004c20 con 0x7f654c06e7b0 2024-08-19T12:45:10.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:10.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 1 -- 172.21.15.26:0/173323826 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6544005150 con 0x7f654c06e7b0 2024-08-19T12:45:10.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f65517aa640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:10.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f65517aa640 4 set_mon_vals no callback set 2024-08-19T12:45:10.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:10.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.511+0000 7f6549ffb640 10 monclient: dump: 2024-08-19T12:45:10.564 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.564 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.564 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.564 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.564 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.564 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.564 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.565 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.565 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.565 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.565 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.565 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.566 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.566 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.566 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.566 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.512+0000 7f65517aa640 10 monclient: shutdown 2024-08-19T12:45:10.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.512+0000 7f65517aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:10.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.512+0000 7f65517aa640 1 -- 172.21.15.26:0/173323826 >> v1:172.21.15.26:6789/0 conn(0x7f654c06e7b0 legacy=0x7f654c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.512+0000 7f65517aa640 1 -- 172.21.15.26:0/173323826 shutdown_connections 2024-08-19T12:45:10.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.512+0000 7f65517aa640 1 -- 172.21.15.26:0/173323826 >> 172.21.15.26:0/173323826 conn(0x7f654c0a8740 msgr2=0x7f654c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:10.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.512+0000 7f65517aa640 1 -- 172.21.15.26:0/173323826 shutdown_connections 2024-08-19T12:45:10.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.512+0000 7f65517aa640 1 -- 172.21.15.26:0/173323826 wait complete. 2024-08-19T12:45:10.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.512+0000 7f65517aa640 10 monclient: build_initial_monmap 2024-08-19T12:45:10.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.512+0000 7f65517aa640 10 monclient: monmap: 2024-08-19T12:45:10.568 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:10.568 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:10.568 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:10.568 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:10.568 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:10.569 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.569 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:10.569 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:10.569 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:10.569 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:10.569 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:10.569 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:10.570 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:10.570 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:10.570 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:10.570 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 1 librados: starting msgr at 2024-08-19T12:45:10.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 1 librados: starting objecter 2024-08-19T12:45:10.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 1 Processor -- start 2024-08-19T12:45:10.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 1 -- start start 2024-08-19T12:45:10.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 1 librados: setting wanted keys 2024-08-19T12:45:10.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 1 librados: calling monclient init 2024-08-19T12:45:10.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 10 monclient: init 2024-08-19T12:45:10.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:10.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 10 monclient: _add_conns ranks=[4,6,0,7,2,8,3,5,1] 2024-08-19T12:45:10.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 10 monclient(hunting): picked mon.noname-c con 0x7f654c06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:10.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 10 monclient(hunting): picked mon.noname-d con 0x7f654c0a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:10.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 10 monclient(hunting): picked mon.noname-a con 0x7f654c0a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:10.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f654c142000 con 0x7f654c0a5d50 2024-08-19T12:45:10.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f654c138f10 con 0x7f654c06e7b0 2024-08-19T12:45:10.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f654c1383e0 con 0x7f654c0a5920 2024-08-19T12:45:10.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:10.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.513+0000 7f65517aa640 10 monclient(hunting): authenticate will time out at 1104.601937s 2024-08-19T12:45:10.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.514+0000 7f654b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f654c0a5d50 0x7f654c1d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38276/0 (socket says 172.21.15.26:38276) 2024-08-19T12:45:10.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.514+0000 7f654affd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f654c06e7b0 0x7f654c1375c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:44242/0 (socket says 172.21.15.26:44242) 2024-08-19T12:45:10.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.514+0000 7f654a7fc640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f654c0a5920 0x7f654c137cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:55200/0 (socket says 172.21.15.26:55200) 2024-08-19T12:45:10.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.514+0000 7f654b7fe640 1 -- 172.21.15.26:0/3935806362 learned_addr learned my addr 172.21.15.26:0/3935806362 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:10.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f654b7fe640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.26:6789/0 conn(0x7f654c0a5d50 legacy=0x7f654c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f654a7fc640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.26:6792/0 conn(0x7f654c0a5920 legacy=0x7f654c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f654affd640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.26:6791/0 conn(0x7f654c06e7b0 legacy=0x7f654c1375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 client.?.objecter ms_handle_connect 0x7f654c0a5d50 2024-08-19T12:45:10.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:10.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 client.?.objecter ms_handle_connect 0x7f654c0a5920 2024-08-19T12:45:10.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:10.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 client.?.objecter ms_handle_connect 0x7f654c06e7b0 2024-08-19T12:45:10.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:10.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f654affd640 1 --1- 172.21.15.26:0/3935806362 >> v1:172.21.15.26:6791/0 conn(0x7f654c06e7b0 0x7f654c1375c0 :-1 s=OPENED pgs=3294 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f654c138f10 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:45:10.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1592931209 0 0) 0x7f654c1383e0 con 0x7f654c0a5920 2024-08-19T12:45:10.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): my global_id is 26673 2024-08-19T12:45:10.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f652c004eb0 con 0x7f654c0a5920 2024-08-19T12:45:10.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 602815251 0 0) 0x7f654c142000 con 0x7f654c0a5d50 2024-08-19T12:45:10.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): my global_id is 144612 2024-08-19T12:45:10.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f654c1383e0 con 0x7f654c0a5d50 2024-08-19T12:45:10.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3988246745 0 0) 0x7f652c004eb0 con 0x7f654c0a5920 2024-08-19T12:45:10.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f654c142000 con 0x7f654c0a5920 2024-08-19T12:45:10.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6534003610 con 0x7f654c0a5920 2024-08-19T12:45:10.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:10.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:45:10.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): dump: 2024-08-19T12:45:10.577 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.577 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.578 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.578 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.578 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.578 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.578 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.579 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.579 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.579 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.579 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.579 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.579 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.579 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.579 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.580 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:10.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3250413255 0 0) 0x7f654c1383e0 con 0x7f654c0a5d50 2024-08-19T12:45:10.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f652c004eb0 con 0x7f654c0a5d50 2024-08-19T12:45:10.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f653c001cd0 con 0x7f654c0a5d50 2024-08-19T12:45:10.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:10.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:10.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient(hunting): dump: 2024-08-19T12:45:10.580 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.581 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.581 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.581 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.581 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.582 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.582 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.582 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.582 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.582 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.582 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.583 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4140269102 0 0) 0x7f654c142000 con 0x7f654c0a5920 2024-08-19T12:45:10.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.26:6791/0 conn(0x7f654c06e7b0 legacy=0x7f654c1375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.26:6789/0 conn(0x7f654c0a5d50 legacy=0x7f654c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient: _finish_hunting 0 2024-08-19T12:45:10.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 monclient: found mon.d 2024-08-19T12:45:10.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:10.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:10.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654c13a9d0 con 0x7f654c0a5920 2024-08-19T12:45:10.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient: _finish_auth 0 2024-08-19T12:45:10.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient: _check_auth_tickets 2024-08-19T12:45:10.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:10.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65517aa640 5 monclient: authenticate success, global_id 26673 2024-08-19T12:45:10.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2564566802 0 0) 0x7f654c138f10 con 0x7f654c06e7b0 2024-08-19T12:45:10.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65517aa640 10 monclient: _renew_subs 2024-08-19T12:45:10.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65517aa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:10.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65517aa640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f654c139d10 con 0x7f654c0a5920 2024-08-19T12:45:10.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.516+0000 7f65437fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:45:10.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65517aa640 10 client.26673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:10.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65517aa640 10 monclient: _renew_subs 2024-08-19T12:45:10.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65517aa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:45:10.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65517aa640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f654c13b690 con 0x7f654c0a5920 2024-08-19T12:45:10.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65517aa640 1 librados: init done 2024-08-19T12:45:10.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6534003b10 con 0x7f654c0a5920 2024-08-19T12:45:10.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65437fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:10.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6534004080 con 0x7f654c0a5920 2024-08-19T12:45:10.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65497fa640 4 set_mon_vals no callback set 2024-08-19T12:45:10.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65437fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:45:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65437fe640 10 monclient: dump: 2024-08-19T12:45:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.589 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6534004220 con 0x7f654c0a5920 2024-08-19T12:45:10.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65517aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:10.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.517+0000 7f65517aa640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f65300065c0 con 0x7f6530002860 2024-08-19T12:45:10.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.518+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(282..282 src has 1..282) ==== 5364+0+0 (unknown 3942427206 0 0) 0x7f653404e1c0 con 0x7f654c0a5920 2024-08-19T12:45:10.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.518+0000 7f65437fe640 10 client.26673.objecter ms_dispatch 0x7f654c000b90 osd_map(282..282 src has 1..282) 2024-08-19T12:45:10.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.518+0000 7f65437fe640 3 client.26673.objecter handle_osd_map got epochs [282,282] > 0 2024-08-19T12:45:10.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.518+0000 7f65437fe640 3 client.26673.objecter handle_osd_map decoding full epoch 282 2024-08-19T12:45:10.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.518+0000 7f65437fe640 20 client.26673.objecter dump_active .. 0 homeless 2024-08-19T12:45:10.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.518+0000 7f654b7fe640 10 monclient: get_auth_request con 0x7f652c041580 auth_method 0 2024-08-19T12:45:10.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 10 client.26673.objecter ms_handle_connect 0x7f6530002860 2024-08-19T12:45:10.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 10 client.26673.objecter resend_mon_ops 2024-08-19T12:45:10.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f654affd640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.104:6792/0 conn(0x7f6530002860 legacy=0x7f6530004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2932706534 0 0) 0x7f65300065c0 con 0x7f6530002860 2024-08-19T12:45:10.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 10 monclient(hunting): my global_id is 135286 2024-08-19T12:45:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f654c138f10 con 0x7f6530002860 2024-08-19T12:45:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 10 client.26673.objecter ms_handle_connect 0x7f652c041580 2024-08-19T12:45:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3975974319 0 0) 0x7f654c138f10 con 0x7f6530002860 2024-08-19T12:45:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65300065c0 con 0x7f6530002860 2024-08-19T12:45:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6544004f30 con 0x7f6530002860 2024-08-19T12:45:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:45:10.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 10 monclient: dump: 2024-08-19T12:45:10.592 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.592 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.592 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.592 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.592 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.592 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.593 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.593 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.593 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.593 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.593 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.593 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.593 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.593 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.594 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.594 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4235660850 0 0) 0x7f65300065c0 con 0x7f6530002860 2024-08-19T12:45:10.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.520+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f653000a160 con 0x7f6530002860 2024-08-19T12:45:10.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.521+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6544001d10 con 0x7f6530002860 2024-08-19T12:45:10.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.521+0000 7f65437fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:10.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.521+0000 7f65437fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:10.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.521+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.104:6792/0 conn(0x7f6530002860 legacy=0x7f6530004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.521+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.104:6792/0 conn(0x7f6530002860 legacy=0x7f6530004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:10.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.527+0000 7f65517aa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:10.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.527+0000 7f65517aa640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6530009e70 con 0x7f6530009130 2024-08-19T12:45:10.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f654affd640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.104:6792/0 conn(0x7f6530009130 legacy=0x7f65300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 10 client.26673.objecter ms_handle_connect 0x7f6530009130 2024-08-19T12:45:10.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 10 client.26673.objecter resend_mon_ops 2024-08-19T12:45:10.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3418217110 0 0) 0x7f6530009e70 con 0x7f6530009130 2024-08-19T12:45:10.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:10.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:10.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:10.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 10 monclient(hunting): my global_id is 135295 2024-08-19T12:45:10.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65300065c0 con 0x7f6530009130 2024-08-19T12:45:10.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3846006742 0 0) 0x7f65300065c0 con 0x7f6530009130 2024-08-19T12:45:10.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6530009e70 con 0x7f6530009130 2024-08-19T12:45:10.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f65440057d0 con 0x7f6530009130 2024-08-19T12:45:10.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:10.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:45:10.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 10 monclient: dump: 2024-08-19T12:45:10.597 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:10.597 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.597 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.597 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:10.597 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:10.598 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.598 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:10.598 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:10.598 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:10.598 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:10.598 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:10.598 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:10.598 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:10.599 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:10.599 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:10.599 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 190819449 0 0) 0x7f6530009e70 con 0x7f6530009130 2024-08-19T12:45:10.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f653000c410 con 0x7f6530009130 2024-08-19T12:45:10.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1075665242) 0x7f65440057d0 con 0x7f6530009130 2024-08-19T12:45:10.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:10.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:10.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.104:6792/0 conn(0x7f6530009130 legacy=0x7f65300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65437fe640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.104:6792/0 conn(0x7f6530009130 legacy=0x7f65300095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:10.600 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:10.600 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:10.600 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:10.601 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:10.601 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 42, 2024-08-19T12:45:10.601 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 280889, 2024-08-19T12:45:10.601 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:10.601 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:10.601 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:10.601 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:10.601 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:10.602 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:10.602 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:10.602 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:45:10.602 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:10.602 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:10.602 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.603 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:45:10.603 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:10.603 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:10.603 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:10.603 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:10.603 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:10.603 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:10.603 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:10.604 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:10.604 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:10.604 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:10.604 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:10.604 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:10.604 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:10.604 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:10.604 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.605 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:10.605 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:10.605 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:10.605 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:10.605 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:10.605 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:10.605 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:10.605 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:10.606 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:10.606 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:10.606 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:10.606 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:10.606 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:10.606 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.606 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:10.607 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:10.607 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:10.607 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:10.607 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:10.607 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:10.607 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:10.607 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:10.607 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:10.608 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:10.608 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:10.608 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:10.608 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:10.608 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:10.608 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:10.608 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:10.608 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:10.608 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:10.609 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:10.609 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:10.609 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:10.609 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:10.609 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:10.609 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:10.609 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:10.609 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:10.610 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:10.610 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:10.610 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.610 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:10.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.610 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:10.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.610 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:10.611 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:10.611 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:10.611 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.611 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.612 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:10.612 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:10.612 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.612 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.612 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.613 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:10.613 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:10.613 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:10.614 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.614 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:10.615 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:10.615 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.615 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.615 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.615 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:10.615 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:10.616 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.616 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.616 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.616 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:10.616 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.617 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.617 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:10.617 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:10.617 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.617 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.617 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.618 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:10.618 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:10.618 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.618 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.619 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:10.619 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.619 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.619 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:10.620 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:10.620 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.621 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.621 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.622 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:10.622 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:10.622 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.622 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.622 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:10.623 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.624 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.624 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.624 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:10.624 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:10.624 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.624 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.625 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.625 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:10.625 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.626 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:10.626 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:10.626 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.626 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.626 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.627 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:10.627 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:10.627 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.627 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.627 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.627 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:10.628 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.628 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.628 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:10.628 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:10.628 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.629 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.629 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.629 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:10.629 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:10.629 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.629 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.630 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:10.630 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.630 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:10.631 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:10.631 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.631 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.631 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.631 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:10.632 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:10.632 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:10.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:10.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.632 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:10.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:10.632 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:10.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.633 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:10.633 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:10.633 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:10.633 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:10.633 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:10.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.634 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.634 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:10.634 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:10.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.634 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:10.635 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:10.635 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:10.635 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.635 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:10.635 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:10.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:10.635 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:10.635 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:10.636 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:10.636 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:10.636 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:10.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:10.636 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:10.636 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:10.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 10 librados: watch_flush enter 2024-08-19T12:45:10.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 10 librados: watch_flush exit 2024-08-19T12:45:10.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 20 client.26673.objecter shutdown clearing up homeless session... 2024-08-19T12:45:10.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 10 client.26673.objecter successfully canceled tick 2024-08-19T12:45:10.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.104:6800/899887506 conn(0x7f652c041580 legacy=0x7f652c043a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 10 monclient: shutdown 2024-08-19T12:45:10.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:10.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 1 -- 172.21.15.26:0/3935806362 >> v1:172.21.15.26:6792/0 conn(0x7f654c0a5920 legacy=0x7f654c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:10.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f654b7fe640 1 -- 172.21.15.26:0/3935806362 reap_dead start 2024-08-19T12:45:10.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f654b7fe640 1 -- 172.21.15.26:0/3935806362 reap_dead start 2024-08-19T12:45:10.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 1 -- 172.21.15.26:0/3935806362 shutdown_connections 2024-08-19T12:45:10.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 1 -- 172.21.15.26:0/3935806362 >> 172.21.15.26:0/3935806362 conn(0x7f654c0a8740 msgr2=0x7f654c06dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:10.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 1 -- 172.21.15.26:0/3935806362 shutdown_connections 2024-08-19T12:45:10.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 1 -- 172.21.15.26:0/3935806362 wait complete. 2024-08-19T12:45:10.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.531+0000 7f65517aa640 1 librados: shutdown 2024-08-19T12:45:10.642 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:45:10.642 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['d'] 2024-08-19T12:45:10.642 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-08-19T12:45:10.643 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-08-19T12:45:10.643 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-08-19T12:45:10.643 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-08-19T12:45:10.643 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:45:10.657 INFO:tasks.ceph.mon.d:Stopped 2024-08-19T12:45:10.657 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-08-19T12:45:10.657 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:45:10.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.794+0000 7fb14f755640 10 monclient: get_monmap_and_config 2024-08-19T12:45:10.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.794+0000 7fb14f755640 10 monclient: build_initial_monmap 2024-08-19T12:45:10.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.794+0000 7fb14f755640 10 monclient: monmap: 2024-08-19T12:45:10.796 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:10.795945+0000 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:10.795945+0000 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:10.797 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:10.798 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:10.798 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:10.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.794+0000 7fb14f755640 1 Processor -- start 2024-08-19T12:45:10.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.794+0000 7fb14f755640 1 -- start start 2024-08-19T12:45:10.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.794+0000 7fb14f755640 10 monclient: init 2024-08-19T12:45:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.795+0000 7fb14f755640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.795+0000 7fb14f755640 10 monclient: _add_conns ranks=[8,7,5,6,3,1,2,0,4] 2024-08-19T12:45:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.795+0000 7fb14f755640 10 monclient(hunting): picked mon.noname-e con 0x7fb148073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.795+0000 7fb14f755640 10 monclient(hunting): picked mon.noname-i con 0x7fb1480a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.795+0000 7fb14f755640 10 monclient(hunting): picked mon.noname-h con 0x7fb14813f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.795+0000 7fb14f755640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb14809dae0 con 0x7fb14813f2e0 2024-08-19T12:45:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.795+0000 7fb14f755640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb148136910 con 0x7fb1480a57a0 2024-08-19T12:45:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.795+0000 7fb14f755640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb14809dd50 con 0x7fb148073470 2024-08-19T12:45:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.795+0000 7fb14f755640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.795+0000 7fb14f755640 10 monclient(hunting): authenticate will time out at 1104.883389s 2024-08-19T12:45:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.795+0000 7fb14d4f1640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fb148073470 0x7fb148136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:50912/0 (socket says 172.21.15.26:50912) 2024-08-19T12:45:10.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.795+0000 7fb14d4f1640 1 -- 172.21.15.26:0/4143082915 learned_addr learned my addr 172.21.15.26:0/4143082915 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:10.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.797+0000 7fb14d4f1640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6793/0 conn(0x7fb148073470 legacy=0x7fb148136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.797+0000 7fb14d4f1640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6793/0 conn(0x7fb148073470 legacy=0x7fb148136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-08-19T12:45:10.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.797+0000 7fb14d4f1640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6793/0 conn(0x7fb148073470 legacy=0x7fb148136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:45:10.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.797+0000 7fb14d4f1640 1 --1- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6793/0 conn(0x7fb148073470 0x7fb148136200 :-1 s=OPENED pgs=3338 cs=1 l=1).handle_message read tag failed 2024-08-19T12:45:10.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.797+0000 7fb14d4f1640 1 --1- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6793/0 conn(0x7fb148073470 0x7fb148136200 :-1 s=OPENED pgs=3338 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:45:10.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.797+0000 7fb147fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6793/0 2024-08-19T12:45:10.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.797+0000 7fb14dcf2640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.104:6791/0 conn(0x7fb14813f2e0 legacy=0x7fb1481416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:10.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:10.798+0000 7fb14ccf0640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.104:6792/0 conn(0x7fb1480a57a0 legacy=0x7fb1480a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:13.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 10 monclient(hunting): tick 2024-08-19T12:45:13.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:45:13.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 1 monclient(hunting): continuing hunt 2024-08-19T12:45:13.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:45:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6793/0 conn(0x7fb148073470 legacy=0x7fb148136200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.104:6792/0 conn(0x7fb1480a57a0 legacy=0x7fb1480a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.104:6791/0 conn(0x7fb14813f2e0 legacy=0x7fb1481416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 10 monclient: _add_conns ranks=[6,0,4,3,2,1] 2024-08-19T12:45:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 10 monclient(hunting): picked mon.noname-d con 0x7fb134003810 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 10 monclient(hunting): picked mon.noname-a con 0x7fb134007480 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fb13400b1f0 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 1 -- 172.21.15.26:0/4143082915 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb14809e030 con 0x7fb134007480 2024-08-19T12:45:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 1 -- 172.21.15.26:0/4143082915 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb134010030 con 0x7fb13400b1f0 2024-08-19T12:45:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb14d4f1640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6792/0 conn(0x7fb134003810 legacy=0x7fb134005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 1 -- 172.21.15.26:0/4143082915 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1340112d0 con 0x7fb134003810 2024-08-19T12:45:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.796+0000 7fb146ffd640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:13.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb14dcf2640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6791/0 conn(0x7fb13400b1f0 legacy=0x7fb13400d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:13.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb14ccf0640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6789/0 conn(0x7fb134007480 legacy=0x7fb134009920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:13.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 1 -- 172.21.15.26:0/4143082915 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2128715491 0 0) 0x7fb14809e030 con 0x7fb134007480 2024-08-19T12:45:13.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 10 monclient(hunting): _negotiate 2024-08-19T12:45:13.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:13.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:13.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 10 monclient(hunting): my global_id is 144621 2024-08-19T12:45:13.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 1 -- 172.21.15.26:0/4143082915 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb130003520 con 0x7fb134007480 2024-08-19T12:45:13.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb14ccf0640 1 --1- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6789/0 conn(0x7fb134007480 0x7fb134009920 :-1 s=OPENED pgs=3352 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fb130003520 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:45:13.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 1 -- 172.21.15.26:0/4143082915 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 74528269 0 0) 0x7fb130003520 con 0x7fb134007480 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6792/0 conn(0x7fb134003810 legacy=0x7fb134005c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6791/0 conn(0x7fb13400b1f0 legacy=0x7fb13400d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 10 monclient: _finish_hunting 0 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 1 monclient: found mon.noname-a 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb14dcf2640 1 -- 172.21.15.26:0/4143082915 reap_dead start 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 1 -- 172.21.15.26:0/4143082915 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1340125d0 con 0x7fb134007480 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 10 monclient: _finish_auth 0 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 10 monclient: _check_auth_tickets 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb14f755640 5 monclient: authenticate success, global_id 144621 2024-08-19T12:45:13.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb14f755640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 1 -- 172.21.15.26:0/4143082915 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb13c001ab0 con 0x7fb134007480 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 10 monclient: dump: 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:13.804 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb147fff640 1 -- 172.21.15.26:0/4143082915 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb13c001d00 con 0x7fb134007480 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.799+0000 7fb14f755640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.800+0000 7fb147fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.800+0000 7fb14f755640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.800+0000 7fb14f755640 4 set_mon_vals no callback set 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.800+0000 7fb14f755640 10 monclient: shutdown 2024-08-19T12:45:13.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.800+0000 7fb14f755640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:13.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.800+0000 7fb14f755640 1 -- 172.21.15.26:0/4143082915 >> v1:172.21.15.26:6789/0 conn(0x7fb134007480 legacy=0x7fb134009920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:13.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.800+0000 7fb147fff640 1 -- 172.21.15.26:0/4143082915 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb13c0039d0 con 0x7fb134007480 2024-08-19T12:45:13.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.800+0000 7fb147fff640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:45:13.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.800+0000 7fb14f755640 1 -- 172.21.15.26:0/4143082915 shutdown_connections 2024-08-19T12:45:13.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.800+0000 7fb14f755640 1 -- 172.21.15.26:0/4143082915 >> 172.21.15.26:0/4143082915 conn(0x7fb1480abff0 msgr2=0x7fb1480ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:13.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.801+0000 7fb14f755640 1 -- 172.21.15.26:0/4143082915 shutdown_connections 2024-08-19T12:45:13.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.801+0000 7fb14f755640 1 -- 172.21.15.26:0/4143082915 wait complete. 2024-08-19T12:45:13.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.801+0000 7fb14f755640 10 monclient: build_initial_monmap 2024-08-19T12:45:13.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.801+0000 7fb14f755640 10 monclient: monmap: 2024-08-19T12:45:13.808 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:13.808 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:13.808 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:13.808 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:13.808 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:13.808 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:13.808 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:13.808 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:13.808 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:13.808 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.801+0000 7fb14f755640 1 librados: starting msgr at 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.801+0000 7fb14f755640 1 librados: starting objecter 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 1 Processor -- start 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 1 -- start start 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 1 librados: setting wanted keys 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 1 librados: calling monclient init 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 10 monclient: init 2024-08-19T12:45:13.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 10 monclient: _add_conns ranks=[2,5,3,6,0,4,8,1,7] 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 10 monclient(hunting): picked mon.noname-b con 0x7fb134007480 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 10 monclient(hunting): picked mon.noname-h con 0x7fb1480a4910 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 10 monclient(hunting): picked mon.noname-g con 0x7fb1481d48d0 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1480a3850 con 0x7fb134007480 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1480a1140 con 0x7fb1481d48d0 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb14813f060 con 0x7fb1480a4910 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14f755640 10 monclient(hunting): authenticate will time out at 1107.889848s 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14d4f1640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb134007480 0x7fb1480a0320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:40110/0 (socket says 172.21.15.26:40110) 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.802+0000 7fb14d4f1640 1 -- 172.21.15.26:0/3437907145 learned_addr learned my addr 172.21.15.26:0/3437907145 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.805+0000 7fb14d4f1640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6790/0 conn(0x7fb134007480 legacy=0x7fb1480a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:13.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.805+0000 7fb145ffb640 10 client.?.objecter ms_handle_connect 0x7fb134007480 2024-08-19T12:45:13.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.805+0000 7fb145ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:13.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.805+0000 7fb14dcf2640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6790/0 conn(0x7fb1481d48d0 legacy=0x7fb1481d6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:13.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.805+0000 7fb145ffb640 10 client.?.objecter ms_handle_connect 0x7fb1481d48d0 2024-08-19T12:45:13.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.805+0000 7fb145ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:13.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.805+0000 7fb145ffb640 10 client.?.objecter ms_handle_connect 0x7fb1480a4910 2024-08-19T12:45:13.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.805+0000 7fb145ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:13.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:13.805+0000 7fb14ccf0640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6791/0 conn(0x7fb1480a4910 legacy=0x7fb1480a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:16.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.802+0000 7fb144ff9640 10 monclient(hunting): tick 2024-08-19T12:45:16.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.802+0000 7fb144ff9640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:45:16.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.802+0000 7fb144ff9640 1 monclient(hunting): continuing hunt 2024-08-19T12:45:16.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.802+0000 7fb144ff9640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:45:16.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.802+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6791/0 conn(0x7fb1480a4910 legacy=0x7fb1480a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:16.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.802+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6790/0 conn(0x7fb1481d48d0 legacy=0x7fb1481d6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:16.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.802+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6790/0 conn(0x7fb134007480 legacy=0x7fb1480a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:16.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.803+0000 7fb144ff9640 10 monclient: _add_conns ranks=[6,4,8,7,1,0] 2024-08-19T12:45:16.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.803+0000 7fb144ff9640 10 monclient(hunting): picked mon.noname-d con 0x7fb128003960 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:16.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.803+0000 7fb144ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fb1280075f0 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:16.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.803+0000 7fb144ff9640 10 monclient(hunting): picked mon.noname-e con 0x7fb12800b320 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:16.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1480a1d90 con 0x7fb1280075f0 2024-08-19T12:45:16.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb128010160 con 0x7fb128003960 2024-08-19T12:45:16.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb128011400 con 0x7fb12800b320 2024-08-19T12:45:16.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.803+0000 7fb144ff9640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:16.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.803+0000 7fb14d4f1640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6792/0 conn(0x7fb128003960 legacy=0x7fb128005db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:16.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.805+0000 7fb145ffb640 10 client.?.objecter ms_handle_connect 0x7fb1280075f0 2024-08-19T12:45:16.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.805+0000 7fb14ccf0640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6791/0 conn(0x7fb1280075f0 legacy=0x7fb128009a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:16.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.805+0000 7fb145ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:16.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.805+0000 7fb145ffb640 10 client.?.objecter ms_handle_connect 0x7fb12800b320 2024-08-19T12:45:16.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.805+0000 7fb145ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:16.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:16.805+0000 7fb14dcf2640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6793/0 conn(0x7fb12800b320 legacy=0x7fb12800d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:17.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:17.004+0000 7fb14d4f1640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6792/0 conn(0x7fb128003960 legacy=0x7fb128005db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:17.405+0000 7fb14d4f1640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6792/0 conn(0x7fb128003960 legacy=0x7fb128005db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:18.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:18.206+0000 7fb14d4f1640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6792/0 conn(0x7fb128003960 legacy=0x7fb128005db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:19.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 10 monclient(hunting): tick 2024-08-19T12:45:19.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:45:19.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 1 monclient(hunting): continuing hunt 2024-08-19T12:45:19.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:45:19.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6793/0 conn(0x7fb12800b320 legacy=0x7fb12800d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:19.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6792/0 conn(0x7fb128003960 legacy=0x7fb128005db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:45:19.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6791/0 conn(0x7fb1280075f0 legacy=0x7fb128009a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:19.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 10 monclient: _add_conns ranks=[1,0,7] 2024-08-19T12:45:19.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb14dcf2640 1 -- 172.21.15.26:0/3437907145 reap_dead start 2024-08-19T12:45:19.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 10 monclient(hunting): picked mon.noname-f con 0x7fb128011930 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:19.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb14dcf2640 1 -- 172.21.15.26:0/3437907145 reap_dead start 2024-08-19T12:45:19.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fb128003960 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:19.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 10 monclient(hunting): picked mon.noname-i con 0x7fb1280075f0 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:19.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1280126a0 con 0x7fb128003960 2024-08-19T12:45:19.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb128010160 con 0x7fb128011930 2024-08-19T12:45:19.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb12800f180 con 0x7fb1280075f0 2024-08-19T12:45:19.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.803+0000 7fb144ff9640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:19.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.805+0000 7fb14ccf0640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6789/0 conn(0x7fb128003960 legacy=0x7fb128005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:19.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.805+0000 7fb145ffb640 10 client.?.objecter ms_handle_connect 0x7fb128003960 2024-08-19T12:45:19.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.805+0000 7fb145ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:19.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.806+0000 7fb145ffb640 10 client.?.objecter ms_handle_connect 0x7fb1280075f0 2024-08-19T12:45:19.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.806+0000 7fb145ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:19.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.806+0000 7fb14dcf2640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6792/0 conn(0x7fb1280075f0 legacy=0x7fb128009a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:19.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.806+0000 7fb145ffb640 10 client.?.objecter ms_handle_connect 0x7fb128011930 2024-08-19T12:45:19.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.806+0000 7fb145ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:19.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:19.806+0000 7fb14d4f1640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6789/0 conn(0x7fb128011930 legacy=0x7fb128011d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:22.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 10 monclient(hunting): tick 2024-08-19T12:45:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:45:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 1 monclient(hunting): continuing hunt 2024-08-19T12:45:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:45:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6792/0 conn(0x7fb1280075f0 legacy=0x7fb128009a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6789/0 conn(0x7fb128011930 legacy=0x7fb128011d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6789/0 conn(0x7fb128003960 legacy=0x7fb128005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 10 monclient: _add_conns ranks=[3,6,5,8,1,2,4,7,0] 2024-08-19T12:45:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 10 monclient(hunting): picked mon.noname-g con 0x7fb12800b320 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 10 monclient(hunting): picked mon.noname-d con 0x7fb12800fa60 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 10 monclient(hunting): picked mon.noname-h con 0x7fb128018110 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:22.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb12800d890 con 0x7fb12800b320 2024-08-19T12:45:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb12801ccf0 con 0x7fb128018110 2024-08-19T12:45:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb12801df70 con 0x7fb12800fa60 2024-08-19T12:45:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb144ff9640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.803+0000 7fb14ccf0640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6792/0 conn(0x7fb12800fa60 legacy=0x7fb1280169f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:22.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.806+0000 7fb14d4f1640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6790/0 conn(0x7fb12800b320 legacy=0x7fb12800f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:22.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.806+0000 7fb145ffb640 10 client.?.objecter ms_handle_connect 0x7fb12800b320 2024-08-19T12:45:22.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.806+0000 7fb145ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:22.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.806+0000 7fb145ffb640 10 client.?.objecter ms_handle_connect 0x7fb128018110 2024-08-19T12:45:22.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.806+0000 7fb145ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:22.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:22.806+0000 7fb14dcf2640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6791/0 conn(0x7fb128018110 legacy=0x7fb12801a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:23.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:23.005+0000 7fb14ccf0640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6792/0 conn(0x7fb12800fa60 legacy=0x7fb1280169f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:23.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:23.406+0000 7fb14ccf0640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6792/0 conn(0x7fb12800fa60 legacy=0x7fb1280169f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.207+0000 7fb14ccf0640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6792/0 conn(0x7fb12800fa60 legacy=0x7fb1280169f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.733+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3507327818 0 0) 0x7fb12800d890 con 0x7fb12800b320 2024-08-19T12:45:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.734+0000 7fb145ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.734+0000 7fb145ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.734+0000 7fb145ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.734+0000 7fb145ffb640 10 monclient(hunting): my global_id is 154227 2024-08-19T12:45:24.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.734+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb134007590 con 0x7fb12800b320 2024-08-19T12:45:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.734+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2402839514 0 0) 0x7fb134007590 con 0x7fb12800b320 2024-08-19T12:45:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.734+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb12800d890 con 0x7fb12800b320 2024-08-19T12:45:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.735+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb138004b60 con 0x7fb12800b320 2024-08-19T12:45:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.735+0000 7fb145ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.735+0000 7fb145ffb640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:45:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.735+0000 7fb145ffb640 10 monclient(hunting): dump: 2024-08-19T12:45:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:24.736 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:24.737 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.735+0000 7fb145ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.735+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1005294627 0 0) 0x7fb12801ccf0 con 0x7fb128018110 2024-08-19T12:45:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.735+0000 7fb145ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.735+0000 7fb145ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.735+0000 7fb145ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.735+0000 7fb145ffb640 10 monclient(hunting): my global_id is 26357 2024-08-19T12:45:24.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.735+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb134007590 con 0x7fb128018110 2024-08-19T12:45:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3677243206 0 0) 0x7fb134007590 con 0x7fb128018110 2024-08-19T12:45:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb12801ccf0 con 0x7fb128018110 2024-08-19T12:45:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb140001bd0 con 0x7fb128018110 2024-08-19T12:45:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:45:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 10 monclient(hunting): dump: 2024-08-19T12:45:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:24.746 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:24.747 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:24.747 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2879599747 0 0) 0x7fb12800d890 con 0x7fb12800b320 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.26:6792/0 conn(0x7fb12800fa60 legacy=0x7fb1280169f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6791/0 conn(0x7fb128018110 legacy=0x7fb12801a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 1 monclient: found mon.g 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12801f1f0 con 0x7fb12800b320 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb14dcf2640 1 -- 172.21.15.26:0/3437907145 reap_dead start 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 10 monclient: _finish_auth 0 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb145ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb14f755640 5 monclient: authenticate success, global_id 154227 2024-08-19T12:45:24.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb14f755640 10 monclient: _renew_subs 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb14f755640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.742+0000 7fb14f755640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb1480a40f0 con 0x7fb12800b320 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.3 v1:172.21.15.104:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb138004fd0 con 0x7fb12800b320 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb145ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.3 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb138005540 con 0x7fb12800b320 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb145ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb145ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb145ffb640 10 monclient: dump: 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:24.749 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb1477fe640 4 set_mon_vals no callback set 2024-08-19T12:45:24.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb14f755640 10 client.154227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb14f755640 10 monclient: _renew_subs 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb14f755640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb14f755640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1480a0320 con 0x7fb12800b320 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb14f755640 1 librados: init done 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.743+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.3 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fb1380056e0 con 0x7fb12800b320 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.744+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.3 v1:172.21.15.104:6790/0 8 ==== osd_map(283..283 src has 1..283) ==== 5364+0+0 (unknown 985969326 0 0) 0x7fb13804e4b0 con 0x7fb12800b320 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.744+0000 7fb145ffb640 10 client.154227.objecter ms_dispatch 0x7fb148000b90 osd_map(283..283 src has 1..283) 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.744+0000 7fb145ffb640 3 client.154227.objecter handle_osd_map got epochs [283,283] > 0 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.744+0000 7fb145ffb640 3 client.154227.objecter handle_osd_map decoding full epoch 283 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.744+0000 7fb14ccf0640 10 monclient: get_auth_request con 0x7fb1340424f0 auth_method 0 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.744+0000 7fb145ffb640 20 client.154227.objecter dump_active .. 0 homeless 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.744+0000 7fb14f755640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:24.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.744+0000 7fb14f755640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:24.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.744+0000 7fb14f755640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:45:24.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.744+0000 7fb14f755640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb148133970 con 0x7fb12800b320 2024-08-19T12:45:24.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.746+0000 7fb145ffb640 10 client.154227.objecter ms_handle_connect 0x7fb1340424f0 2024-08-19T12:45:24.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.749+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.3 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fb148133970 con 0x7fb12800b320 2024-08-19T12:45:24.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.749+0000 7fb145ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:24.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:24.749+0000 7fb145ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:25.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.009+0000 7fb14f755640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:45:25.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.009+0000 7fb14f755640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:45:25.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.009+0000 7fb14f755640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:45:25.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.009+0000 7fb14f755640 1 -- 172.21.15.26:0/3437907145 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb148134240 con 0x7fb12800b320 2024-08-19T12:45:25.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.010+0000 7fb145ffb640 1 -- 172.21.15.26:0/3437907145 <== mon.3 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2723 (unknown 254746657 0 325423493) 0x7fb148134240 con 0x7fb12800b320 2024-08-19T12:45:25.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.010+0000 7fb145ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:45:25.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.010+0000 7fb145ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:25.011 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":46,"quorum":[0,1,2,3,4,5,7,8],"quorum_names":["a","f","b","g","c","h","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:45:25.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 10 librados: watch_flush enter 2024-08-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 10 librados: watch_flush exit 2024-08-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 20 client.154227.objecter shutdown clearing up homeless session... 2024-08-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 10 client.154227.objecter successfully canceled tick 2024-08-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6800/899887506 conn(0x7fb1340424f0 legacy=0x7fb1340449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 10 monclient: shutdown 2024-08-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 1 -- 172.21.15.26:0/3437907145 >> v1:172.21.15.104:6790/0 conn(0x7fb12800b320 legacy=0x7fb12800f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 1 -- 172.21.15.26:0/3437907145 shutdown_connections 2024-08-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 1 -- 172.21.15.26:0/3437907145 >> 172.21.15.26:0/3437907145 conn(0x7fb1480abff0 msgr2=0x7fb148132b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 1 -- 172.21.15.26:0/3437907145 shutdown_connections 2024-08-19T12:45:25.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 1 -- 172.21.15.26:0/3437907145 wait complete. 2024-08-19T12:45:25.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.012+0000 7fb14f755640 1 librados: shutdown 2024-08-19T12:45:25.025 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-08-19T12:45:25.025 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-08-19T12:45:25.025 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:45:25.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.124+0000 7fe1bfc8b640 10 monclient: get_monmap_and_config 2024-08-19T12:45:25.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.124+0000 7fe1bfc8b640 10 monclient: build_initial_monmap 2024-08-19T12:45:25.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.124+0000 7fe1bfc8b640 10 monclient: monmap: 2024-08-19T12:45:25.125 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:25.125 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.125 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:25.125077+0000 2024-08-19T12:45:25.125 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:25.125077+0000 2024-08-19T12:45:25.125 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:25.125 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.125 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:25.126 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:25.126 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:25.126 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:25.126 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:25.126 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:25.126 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:25.126 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:25.126 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:25.126 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.124+0000 7fe1bfc8b640 1 Processor -- start 2024-08-19T12:45:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bfc8b640 1 -- start start 2024-08-19T12:45:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bfc8b640 10 monclient: init 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bfc8b640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bfc8b640 10 monclient: _add_conns ranks=[2,6,5,0,3,1,8,7,4] 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bfc8b640 10 monclient(hunting): picked mon.noname-b con 0x7fe1b806e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bfc8b640 10 monclient(hunting): picked mon.noname-d con 0x7fe1b80a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bfc8b640 10 monclient(hunting): picked mon.noname-h con 0x7fe1b80a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bfc8b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b8136790 con 0x7fe1b806e7b0 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bfc8b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b809ddf0 con 0x7fe1b80a5d50 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bfc8b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b809e0d0 con 0x7fe1b80a5920 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bfc8b640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bfc8b640 10 monclient(hunting): authenticate will time out at 1119.212518s 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bd226640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fe1b80a5920 legacy=0x7fe1b813e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bda27640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe1b806e7b0 0x7fe1b8136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38854/0 (socket says 172.21.15.26:38854) 2024-08-19T12:45:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.125+0000 7fe1bda27640 1 -- 172.21.15.26:0/2129301499 learned_addr learned my addr 172.21.15.26:0/2129301499 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.127+0000 7fe1bda27640 1 -- 172.21.15.26:0/2129301499 >> v1:172.21.15.26:6790/0 conn(0x7fe1b806e7b0 legacy=0x7fe1b8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.127+0000 7fe1bca25640 1 -- 172.21.15.26:0/2129301499 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 833808067 0 0) 0x7fe1b8136790 con 0x7fe1b806e7b0 2024-08-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.127+0000 7fe1bca25640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.127+0000 7fe1bca25640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.127+0000 7fe1bca25640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.127+0000 7fe1bca25640 10 monclient(hunting): my global_id is 26327 2024-08-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.127+0000 7fe1bca25640 1 -- 172.21.15.26:0/2129301499 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1a4003520 con 0x7fe1b806e7b0 2024-08-19T12:45:25.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1be228640 1 -- 172.21.15.26:0/2129301499 >> v1:172.21.15.104:6791/0 conn(0x7fe1b80a5d50 legacy=0x7fe1b8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 1 -- 172.21.15.26:0/2129301499 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2276122136 0 0) 0x7fe1a4003520 con 0x7fe1b806e7b0 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 1 -- 172.21.15.26:0/2129301499 >> v1:172.21.15.26:6792/0 conn(0x7fe1b80a5920 legacy=0x7fe1b813e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 1 -- 172.21.15.26:0/2129301499 >> v1:172.21.15.104:6791/0 conn(0x7fe1b80a5d50 legacy=0x7fe1b8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 10 monclient: _finish_hunting 0 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 1 monclient: found mon.noname-b 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 1 -- 172.21.15.26:0/2129301499 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b809e3b0 con 0x7fe1b806e7b0 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 10 monclient: _finish_auth 0 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 10 monclient: _check_auth_tickets 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 1 -- 172.21.15.26:0/2129301499 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1b4001ae0 con 0x7fe1b806e7b0 2024-08-19T12:45:25.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 10 monclient: dump: 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.138 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bfc8b640 5 monclient: authenticate success, global_id 26327 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bfc8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 1 -- 172.21.15.26:0/2129301499 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1b40048a0 con 0x7fe1b806e7b0 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 1 -- 172.21.15.26:0/2129301499 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1b4004e10 con 0x7fe1b806e7b0 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bfc8b640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bfc8b640 4 set_mon_vals no callback set 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bca25640 10 monclient: dump: 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bfc8b640 10 monclient: shutdown 2024-08-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bfc8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/2129301499 >> v1:172.21.15.26:6790/0 conn(0x7fe1b806e7b0 legacy=0x7fe1b8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/2129301499 shutdown_connections 2024-08-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/2129301499 >> 172.21.15.26:0/2129301499 conn(0x7fe1b80a8740 msgr2=0x7fe1b80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/2129301499 shutdown_connections 2024-08-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.128+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/2129301499 wait complete. 2024-08-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.129+0000 7fe1bfc8b640 10 monclient: build_initial_monmap 2024-08-19T12:45:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.129+0000 7fe1bfc8b640 10 monclient: monmap: 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.129+0000 7fe1bfc8b640 1 librados: starting msgr at 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.129+0000 7fe1bfc8b640 1 librados: starting objecter 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.129+0000 7fe1bfc8b640 1 Processor -- start 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.129+0000 7fe1bfc8b640 1 -- start start 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.129+0000 7fe1bfc8b640 1 librados: setting wanted keys 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.129+0000 7fe1bfc8b640 1 librados: calling monclient init 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.129+0000 7fe1bfc8b640 10 monclient: init 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1bfc8b640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1bfc8b640 10 monclient: _add_conns ranks=[2,4,6,3,1,7,8,5,0] 2024-08-19T12:45:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1bfc8b640 10 monclient(hunting): picked mon.noname-b con 0x7fe1b806e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1bfc8b640 10 monclient(hunting): picked mon.noname-c con 0x7fe1b80a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1bfc8b640 10 monclient(hunting): picked mon.noname-d con 0x7fe1b80a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1bfc8b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b812df10 con 0x7fe1b806e7b0 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1bfc8b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b8124330 con 0x7fe1b80a5920 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1bfc8b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b8126800 con 0x7fe1b80a5d50 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1be228640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fe1b80a5d50 legacy=0x7fe1b81d6290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1bfc8b640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1bfc8b640 10 monclient(hunting): authenticate will time out at 1119.217322s 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1bda27640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe1b806e7b0 0x7fe1b8123510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38864/0 (socket says 172.21.15.26:38864) 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.130+0000 7fe1bda27640 1 -- 172.21.15.26:0/1435872619 learned_addr learned my addr 172.21.15.26:0/1435872619 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1bd226640 1 -- 172.21.15.26:0/1435872619 >> v1:172.21.15.26:6791/0 conn(0x7fe1b80a5920 legacy=0x7fe1b8123c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1bda27640 1 -- 172.21.15.26:0/1435872619 >> v1:172.21.15.26:6790/0 conn(0x7fe1b806e7b0 legacy=0x7fe1b8123510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 client.?.objecter ms_handle_connect 0x7fe1b80a5920 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 client.?.objecter ms_handle_connect 0x7fe1b806e7b0 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1508982399 0 0) 0x7fe1b809e3b0 con 0x7fe1b806e7b0 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): my global_id is 26336 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe19c004eb0 con 0x7fe1b806e7b0 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 163997120 0 0) 0x7fe1b8124330 con 0x7fe1b80a5920 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): my global_id is 26671 2024-08-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1b809e3b0 con 0x7fe1b80a5920 2024-08-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1787780853 0 0) 0x7fe19c004eb0 con 0x7fe1b806e7b0 2024-08-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1b8124330 con 0x7fe1b806e7b0 2024-08-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1b4004d90 con 0x7fe1b806e7b0 2024-08-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): dump: 2024-08-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1451082048 0 0) 0x7fe1b809e3b0 con 0x7fe1b80a5920 2024-08-19T12:45:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe19c004eb0 con 0x7fe1b80a5920 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a8004550 con 0x7fe1b80a5920 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.132+0000 7fe1ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 10 monclient(hunting): dump: 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1413235741 0 0) 0x7fe1b8124330 con 0x7fe1b806e7b0 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 >> v1:172.21.15.26:6792/0 conn(0x7fe1b80a5d50 legacy=0x7fe1b81d6290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 >> v1:172.21.15.26:6791/0 conn(0x7fe1b80a5920 legacy=0x7fe1b8123c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 1 monclient: found mon.b 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b8133220 con 0x7fe1b806e7b0 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 10 monclient: _finish_auth 0 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1bfc8b640 5 monclient: authenticate success, global_id 26336 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1bfc8b640 10 monclient: _renew_subs 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1bfc8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b8125b10 con 0x7fe1b806e7b0 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1bfc8b640 10 client.26336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1bfc8b640 10 monclient: _renew_subs 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1bfc8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1b81d79b0 con 0x7fe1b806e7b0 2024-08-19T12:45:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1bfc8b640 1 librados: init done 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1b4001d30 con 0x7fe1b806e7b0 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1b40048a0 con 0x7fe1b806e7b0 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1affff640 4 set_mon_vals no callback set 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.133+0000 7fe1ae7fc640 10 monclient: dump: 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.134+0000 7fe1bfc8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.134+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1a00065c0 con 0x7fe1a0002860 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.134+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe1b4011a00 con 0x7fe1b806e7b0 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.134+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(283..283 src has 1..283) ==== 5364+0+0 (unknown 985969326 0 0) 0x7fe1b404e290 con 0x7fe1b806e7b0 2024-08-19T12:45:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.134+0000 7fe1ae7fc640 10 client.26336.objecter ms_dispatch 0x7fe1b8000b90 osd_map(283..283 src has 1..283) 2024-08-19T12:45:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.134+0000 7fe1ae7fc640 3 client.26336.objecter handle_osd_map got epochs [283,283] > 0 2024-08-19T12:45:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.134+0000 7fe1ae7fc640 3 client.26336.objecter handle_osd_map decoding full epoch 283 2024-08-19T12:45:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.134+0000 7fe1ae7fc640 20 client.26336.objecter dump_active .. 0 homeless 2024-08-19T12:45:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.134+0000 7fe1be228640 10 monclient: get_auth_request con 0x7fe19c0416d0 auth_method 0 2024-08-19T12:45:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1bd226640 1 -- 172.21.15.26:0/1435872619 >> v1:172.21.15.26:6789/0 conn(0x7fe1a0002860 legacy=0x7fe1a0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 10 client.26336.objecter ms_handle_connect 0x7fe1a0002860 2024-08-19T12:45:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 10 client.26336.objecter resend_mon_ops 2024-08-19T12:45:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3746696176 0 0) 0x7fe1a00065c0 con 0x7fe1a0002860 2024-08-19T12:45:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 10 monclient(hunting): my global_id is 144630 2024-08-19T12:45:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1b8124330 con 0x7fe1a0002860 2024-08-19T12:45:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1492575912 0 0) 0x7fe1b8124330 con 0x7fe1a0002860 2024-08-19T12:45:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1a00065c0 con 0x7fe1a0002860 2024-08-19T12:45:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a8004cf0 con 0x7fe1a0002860 2024-08-19T12:45:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.136+0000 7fe1ae7fc640 10 monclient: dump: 2024-08-19T12:45:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.137+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2167522996 0 0) 0x7fe1a00065c0 con 0x7fe1a0002860 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.137+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe1a000a160 con 0x7fe1a0002860 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.137+0000 7fe1ae7fc640 10 client.26336.objecter ms_handle_connect 0x7fe19c0416d0 2024-08-19T12:45:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.137+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe1a8006290 con 0x7fe1a0002860 2024-08-19T12:45:25.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.137+0000 7fe1ae7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:25.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.137+0000 7fe1ae7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:25.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.137+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 >> v1:172.21.15.26:6789/0 conn(0x7fe1a0002860 legacy=0x7fe1a0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.137+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 >> v1:172.21.15.26:6789/0 conn(0x7fe1a0002860 legacy=0x7fe1a0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:25.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.143+0000 7fe1bfc8b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:25.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.143+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1a0009e70 con 0x7fe1a0009130 2024-08-19T12:45:25.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1bd226640 1 -- 172.21.15.26:0/1435872619 >> v1:172.21.15.26:6789/0 conn(0x7fe1a0009130 legacy=0x7fe1a00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 10 client.26336.objecter ms_handle_connect 0x7fe1a0009130 2024-08-19T12:45:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 10 client.26336.objecter resend_mon_ops 2024-08-19T12:45:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464499240 0 0) 0x7fe1a0009e70 con 0x7fe1a0009130 2024-08-19T12:45:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 10 monclient(hunting): my global_id is 144639 2024-08-19T12:45:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1a00065c0 con 0x7fe1a0009130 2024-08-19T12:45:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3520571027 0 0) 0x7fe1a00065c0 con 0x7fe1a0009130 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1a0009e70 con 0x7fe1a0009130 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a8006a70 con 0x7fe1a0009130 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 10 monclient: dump: 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2652273098 0 0) 0x7fe1a0009e70 con 0x7fe1a0009130 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe1a000c3f0 con 0x7fe1a0009130 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7702 (unknown 0 0 3222219812) 0x7fe1a8006d10 con 0x7fe1a0009130 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 >> v1:172.21.15.26:6789/0 conn(0x7fe1a0009130 legacy=0x7fe1a00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.146+0000 7fe1ae7fc640 1 -- 172.21.15.26:0/1435872619 >> v1:172.21.15.26:6789/0 conn(0x7fe1a0009130 legacy=0x7fe1a00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:25.165 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:25.165 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:25.165 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:25.165 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:45:25.165 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 46, 2024-08-19T12:45:25.165 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 177064, 2024-08-19T12:45:25.165 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:25.165 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:25.166 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:25.166 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:25.166 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:25.166 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:25.166 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:25.166 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:25.166 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:25.166 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.166 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:45:25.166 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:25.166 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:25.167 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:25.167 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.167 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.167 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.167 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.167 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.167 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.167 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.167 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.167 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.167 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.167 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:25.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 10 librados: watch_flush enter 2024-08-19T12:45:25.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 10 librados: watch_flush exit 2024-08-19T12:45:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 20 client.26336.objecter shutdown clearing up homeless session... 2024-08-19T12:45:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 10 client.26336.objecter successfully canceled tick 2024-08-19T12:45:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/1435872619 >> v1:172.21.15.104:6800/899887506 conn(0x7fe19c0416d0 legacy=0x7fe19c043b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 10 monclient: shutdown 2024-08-19T12:45:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/1435872619 >> v1:172.21.15.26:6790/0 conn(0x7fe1b806e7b0 legacy=0x7fe1b8123510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1be228640 1 -- 172.21.15.26:0/1435872619 reap_dead start 2024-08-19T12:45:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1be228640 1 -- 172.21.15.26:0/1435872619 reap_dead start 2024-08-19T12:45:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/1435872619 shutdown_connections 2024-08-19T12:45:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/1435872619 >> 172.21.15.26:0/1435872619 conn(0x7fe1b80a8740 msgr2=0x7fe1b806d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/1435872619 shutdown_connections 2024-08-19T12:45:25.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 1 -- 172.21.15.26:0/1435872619 wait complete. 2024-08-19T12:45:25.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.147+0000 7fe1bfc8b640 1 librados: shutdown 2024-08-19T12:45:25.190 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:45:25.190 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-08-19T12:45:25.190 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:45:25.290 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.290+0000 7f2f2557f640 10 monclient: get_monmap_and_config 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.290+0000 7f2f2557f640 10 monclient: build_initial_monmap 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.290+0000 7f2f2557f640 10 monclient: monmap: 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:25.290920+0000 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:25.290920+0000 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:25.291 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:25.292 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:25.292 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:25.292 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:25.292 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:25.292 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.290+0000 7f2f2557f640 1 Processor -- start 2024-08-19T12:45:25.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.290+0000 7f2f2557f640 1 -- start start 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.290+0000 7f2f2557f640 10 monclient: init 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f2557f640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f2557f640 10 monclient: _add_conns ranks=[7,3,5,4,1,6,0,8,2] 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f2557f640 10 monclient(hunting): picked mon.noname-i con 0x7f2f20073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f2557f640 10 monclient(hunting): picked mon.noname-g con 0x7f2f200a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f2557f640 10 monclient(hunting): picked mon.noname-h con 0x7f2f2013f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f2557f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f2009dae0 con 0x7f2f200a57a0 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f2557f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f20136910 con 0x7f2f2013f2e0 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f2557f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f2009dd50 con 0x7f2f20073470 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f2557f640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f2557f640 10 monclient(hunting): authenticate will time out at 1119.378221s 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f1effd640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f2f20073470 0x7f2f20136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:38578/0 (socket says 172.21.15.26:38578) 2024-08-19T12:45:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f1effd640 1 -- 172.21.15.26:0/3321408199 learned_addr learned my addr 172.21.15.26:0/3321408199 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:25.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.291+0000 7f2f1f7fe640 1 --1- 172.21.15.26:0/3321408199 >> v1:172.21.15.104:6791/0 conn(0x7f2f2013f2e0 0x7f2f201416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:34336/0 (socket says 172.21.15.26:34336) 2024-08-19T12:45:25.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.293+0000 7f2f1e7fc640 1 -- 172.21.15.26:0/3321408199 >> v1:172.21.15.104:6790/0 conn(0x7f2f200a57a0 legacy=0x7f2f200a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.293+0000 7f2f1f7fe640 1 -- 172.21.15.26:0/3321408199 >> v1:172.21.15.104:6791/0 conn(0x7f2f2013f2e0 legacy=0x7f2f201416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.293+0000 7f2f1effd640 1 -- 172.21.15.26:0/3321408199 >> v1:172.21.15.104:6792/0 conn(0x7f2f20073470 legacy=0x7f2f20136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.293+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3331325859 0 0) 0x7f2f2009dae0 con 0x7f2f200a57a0 2024-08-19T12:45:25.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient(hunting): my global_id is 154236 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f0c003520 con 0x7f2f200a57a0 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3633552768 0 0) 0x7f2f2009dd50 con 0x7f2f20073470 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient(hunting): my global_id is 135304 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f2009dae0 con 0x7f2f20073470 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1046930858 0 0) 0x7f2f20136910 con 0x7f2f2013f2e0 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient(hunting): my global_id is 26366 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f2009dd50 con 0x7f2f2013f2e0 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3407401659 0 0) 0x7f2f0c003520 con 0x7f2f200a57a0 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 >> v1:172.21.15.104:6792/0 conn(0x7f2f20073470 legacy=0x7f2f20136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 >> v1:172.21.15.104:6791/0 conn(0x7f2f2013f2e0 legacy=0x7f2f201416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 monclient: found mon.noname-g 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2009e030 con 0x7f2f200a57a0 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient: _finish_auth 0 2024-08-19T12:45:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f2557f640 5 monclient: authenticate success, global_id 154236 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f2557f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2f08001b30 con 0x7f2f200a57a0 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient: dump: 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f2557f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f08003610 con 0x7f2f200a57a0 2024-08-19T12:45:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f2557f640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 1 -- 172.21.15.26:0/3321408199 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2f08003b80 con 0x7f2f200a57a0 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f2557f640 4 set_mon_vals no callback set 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.294+0000 7f2f1dffb640 10 monclient: dump: 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.295+0000 7f2f2557f640 10 monclient: shutdown 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.295+0000 7f2f2557f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.295+0000 7f2f2557f640 1 -- 172.21.15.26:0/3321408199 >> v1:172.21.15.104:6790/0 conn(0x7f2f200a57a0 legacy=0x7f2f200a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.295+0000 7f2f2557f640 1 -- 172.21.15.26:0/3321408199 shutdown_connections 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.295+0000 7f2f2557f640 1 -- 172.21.15.26:0/3321408199 >> 172.21.15.26:0/3321408199 conn(0x7f2f200abff0 msgr2=0x7f2f200ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.295+0000 7f2f2557f640 1 -- 172.21.15.26:0/3321408199 shutdown_connections 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.295+0000 7f2f2557f640 1 -- 172.21.15.26:0/3321408199 wait complete. 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.295+0000 7f2f2557f640 10 monclient: build_initial_monmap 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.295+0000 7f2f2557f640 10 monclient: monmap: 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:25.305 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.295+0000 7f2f2557f640 1 librados: starting msgr at 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.295+0000 7f2f2557f640 1 librados: starting objecter 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 1 Processor -- start 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 1 -- start start 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 1 librados: setting wanted keys 2024-08-19T12:45:25.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 1 librados: calling monclient init 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 10 monclient: init 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 10 monclient: _add_conns ranks=[6,7,2,3,8,1,0,5,4] 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 10 monclient(hunting): picked mon.noname-d con 0x7f2f20073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 10 monclient(hunting): picked mon.noname-i con 0x7f2f200a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 10 monclient(hunting): picked mon.noname-b con 0x7f2f2013f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f20131be0 con 0x7f2f2013f2e0 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f200aa160 con 0x7f2f20073470 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f1effd640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f2f20073470 legacy=0x7f2f2012deb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f2012f5b0 con 0x7f2f200a57a0 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f2557f640 10 monclient(hunting): authenticate will time out at 1119.383740s 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f1f7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2f2013f2e0 0x7f2f201d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38866/0 (socket says 172.21.15.26:38866) 2024-08-19T12:45:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.296+0000 7f2f1f7fe640 1 -- 172.21.15.26:0/1873612295 learned_addr learned my addr 172.21.15.26:0/1873612295 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.298+0000 7f2f1f7fe640 1 -- 172.21.15.26:0/1873612295 >> v1:172.21.15.26:6790/0 conn(0x7f2f2013f2e0 legacy=0x7f2f201d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.298+0000 7f2f177fe640 10 client.?.objecter ms_handle_connect 0x7f2f2013f2e0 2024-08-19T12:45:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.298+0000 7f2f177fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3568132507 0 0) 0x7f2f20131be0 con 0x7f2f2013f2e0 2024-08-19T12:45:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient(hunting): my global_id is 26345 2024-08-19T12:45:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f00004eb0 con 0x7f2f2013f2e0 2024-08-19T12:45:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 client.?.objecter ms_handle_connect 0x7f2f200a57a0 2024-08-19T12:45:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f1e7fc640 1 -- 172.21.15.26:0/1873612295 >> v1:172.21.15.104:6792/0 conn(0x7f2f200a57a0 legacy=0x7f2f2012e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3285717057 0 0) 0x7f2f2012f5b0 con 0x7f2f200a57a0 2024-08-19T12:45:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient(hunting): my global_id is 135313 2024-08-19T12:45:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f20131be0 con 0x7f2f200a57a0 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1017548162 0 0) 0x7f2f00004eb0 con 0x7f2f2013f2e0 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f2012f5b0 con 0x7f2f2013f2e0 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2f18001cd0 con 0x7f2f2013f2e0 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient(hunting): dump: 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1317098303 0 0) 0x7f2f2012f5b0 con 0x7f2f2013f2e0 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 >> v1:172.21.15.104:6792/0 conn(0x7f2f200a57a0 legacy=0x7f2f2012e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 >> v1:172.21.15.26:6792/0 conn(0x7f2f20073470 legacy=0x7f2f2012deb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:45:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient: _finish_hunting 0 2024-08-19T12:45:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 monclient: found mon.b 2024-08-19T12:45:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:45:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2012ecd0 con 0x7f2f2013f2e0 2024-08-19T12:45:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient: _finish_auth 0 2024-08-19T12:45:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient: _check_auth_tickets 2024-08-19T12:45:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f2557f640 5 monclient: authenticate success, global_id 26345 2024-08-19T12:45:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f2557f640 10 monclient: _renew_subs 2024-08-19T12:45:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f2557f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:45:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f2557f640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2f20130ff0 con 0x7f2f2013f2e0 2024-08-19T12:45:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f2557f640 10 client.26345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f2557f640 10 monclient: _renew_subs 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f2557f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f2557f640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2f20132e60 con 0x7f2f2013f2e0 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f2557f640 1 librados: init done 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f18004a90 con 0x7f2f2013f2e0 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2f18004f80 con 0x7f2f2013f2e0 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f1cff9640 4 set_mon_vals no callback set 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.299+0000 7f2f177fe640 10 monclient: dump: 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.318 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.319 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.300+0000 7f2f2557f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.300+0000 7f2f2557f640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2f040065c0 con 0x7f2f04002860 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.300+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2f18005120 con 0x7f2f2013f2e0 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.300+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(283..283 src has 1..283) ==== 5364+0+0 (unknown 985969326 0 0) 0x7f2f1804e070 con 0x7f2f2013f2e0 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.300+0000 7f2f177fe640 10 client.26345.objecter ms_dispatch 0x7f2f20000b90 osd_map(283..283 src has 1..283) 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.300+0000 7f2f177fe640 3 client.26345.objecter handle_osd_map got epochs [283,283] > 0 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.300+0000 7f2f177fe640 3 client.26345.objecter handle_osd_map decoding full epoch 283 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.300+0000 7f2f177fe640 20 client.26345.objecter dump_active .. 0 homeless 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.300+0000 7f2f1e7fc640 10 monclient: get_auth_request con 0x7f2f00040f70 auth_method 0 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.302+0000 7f2f1effd640 1 -- 172.21.15.26:0/1873612295 >> v1:172.21.15.26:6790/0 conn(0x7f2f04002860 legacy=0x7f2f04004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.302+0000 7f2f177fe640 10 client.26345.objecter ms_handle_connect 0x7f2f04002860 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.302+0000 7f2f177fe640 10 client.26345.objecter resend_mon_ops 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.302+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3280294621 0 0) 0x7f2f040065c0 con 0x7f2f04002860 2024-08-19T12:45:25.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.302+0000 7f2f177fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.302+0000 7f2f177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.302+0000 7f2f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.302+0000 7f2f177fe640 10 monclient(hunting): my global_id is 26354 2024-08-19T12:45:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.302+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f2012f5b0 con 0x7f2f04002860 2024-08-19T12:45:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2928815221 0 0) 0x7f2f2012f5b0 con 0x7f2f04002860 2024-08-19T12:45:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f040065c0 con 0x7f2f04002860 2024-08-19T12:45:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2f10001f20 con 0x7f2f04002860 2024-08-19T12:45:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 10 monclient: dump: 2024-08-19T12:45:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 10 client.26345.objecter ms_handle_connect 0x7f2f00040f70 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2263420637 0 0) 0x7f2f040065c0 con 0x7f2f04002860 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f0400a160 con 0x7f2f04002860 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2f10007480 con 0x7f2f04002860 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 >> v1:172.21.15.26:6790/0 conn(0x7f2f04002860 legacy=0x7f2f04004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.303+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 >> v1:172.21.15.26:6790/0 conn(0x7f2f04002860 legacy=0x7f2f04004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.309+0000 7f2f2557f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.309+0000 7f2f2557f640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2f04009e50 con 0x7f2f04009110 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f1effd640 1 -- 172.21.15.26:0/1873612295 >> v1:172.21.15.26:6790/0 conn(0x7f2f04009110 legacy=0x7f2f040095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 10 client.26345.objecter ms_handle_connect 0x7f2f04009110 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 10 client.26345.objecter resend_mon_ops 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1333047596 0 0) 0x7f2f04009e50 con 0x7f2f04009110 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 10 monclient(hunting): my global_id is 26363 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f040065c0 con 0x7f2f04009110 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2186976013 0 0) 0x7f2f040065c0 con 0x7f2f04009110 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f04009e50 con 0x7f2f04009110 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2f10005570 con 0x7f2f04009110 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 10 monclient: dump: 2024-08-19T12:45:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.326 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1595179153 0 0) 0x7f2f04009e50 con 0x7f2f04009110 2024-08-19T12:45:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2f0400c3f0 con 0x7f2f04009110 2024-08-19T12:45:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 186418277) 0x7f2f10007630 con 0x7f2f04009110 2024-08-19T12:45:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 >> v1:172.21.15.26:6790/0 conn(0x7f2f04009110 legacy=0x7f2f040095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.312+0000 7f2f177fe640 1 -- 172.21.15.26:0/1873612295 >> v1:172.21.15.26:6790/0 conn(0x7f2f04009110 legacy=0x7f2f040095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:25.326 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 46, 2024-08-19T12:45:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 438367, 2024-08-19T12:45:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:25.327 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.328 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.329 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:25.330 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:25.331 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.332 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:25.333 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:25.334 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:25.337 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:45:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:25.346 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 10 librados: watch_flush enter 2024-08-19T12:45:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 10 librados: watch_flush exit 2024-08-19T12:45:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 20 client.26345.objecter shutdown clearing up homeless session... 2024-08-19T12:45:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 10 client.26345.objecter successfully canceled tick 2024-08-19T12:45:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 1 -- 172.21.15.26:0/1873612295 >> v1:172.21.15.104:6800/899887506 conn(0x7f2f00040f70 legacy=0x7f2f00043430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 10 monclient: shutdown 2024-08-19T12:45:25.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:25.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 1 -- 172.21.15.26:0/1873612295 >> v1:172.21.15.26:6790/0 conn(0x7f2f2013f2e0 legacy=0x7f2f201d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f1f7fe640 1 -- 172.21.15.26:0/1873612295 reap_dead start 2024-08-19T12:45:25.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 1 -- 172.21.15.26:0/1873612295 shutdown_connections 2024-08-19T12:45:25.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 1 -- 172.21.15.26:0/1873612295 >> 172.21.15.26:0/1873612295 conn(0x7f2f200abff0 msgr2=0x7f2f200a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:25.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 1 -- 172.21.15.26:0/1873612295 shutdown_connections 2024-08-19T12:45:25.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 1 -- 172.21.15.26:0/1873612295 wait complete. 2024-08-19T12:45:25.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.313+0000 7f2f2557f640 1 librados: shutdown 2024-08-19T12:45:25.349 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:45:25.349 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-08-19T12:45:25.350 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:45:25.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.448+0000 7fd5b05f4640 10 monclient: get_monmap_and_config 2024-08-19T12:45:25.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.448+0000 7fd5b05f4640 10 monclient: build_initial_monmap 2024-08-19T12:45:25.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.448+0000 7fd5b05f4640 10 monclient: monmap: 2024-08-19T12:45:25.449 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:25.449 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.449 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:25.449358+0000 2024-08-19T12:45:25.449 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:25.449358+0000 2024-08-19T12:45:25.450 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:25.450 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.450 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:25.450 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:25.450 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:25.450 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:25.450 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:25.450 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:25.450 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:25.450 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:25.450 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:25.450 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 1 Processor -- start 2024-08-19T12:45:25.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 1 -- start start 2024-08-19T12:45:25.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 10 monclient: init 2024-08-19T12:45:25.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:25.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 10 monclient: _add_conns ranks=[8,1,2,5,7,0,3,6,4] 2024-08-19T12:45:25.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 10 monclient(hunting): picked mon.noname-e con 0x7fd5a806e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:25.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 10 monclient(hunting): picked mon.noname-f con 0x7fd5a80a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:25.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 10 monclient(hunting): picked mon.noname-b con 0x7fd5a80a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:25.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a8136790 con 0x7fd5a80a5920 2024-08-19T12:45:25.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a809ddf0 con 0x7fd5a80a5d50 2024-08-19T12:45:25.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a809e0d0 con 0x7fd5a806e7b0 2024-08-19T12:45:25.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:25.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5b05f4640 10 monclient(hunting): authenticate will time out at 1119.536789s 2024-08-19T12:45:25.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5aeb91640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd5a80a5d50 0x7fd5a8141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38896/0 (socket says 172.21.15.26:38896) 2024-08-19T12:45:25.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5ae390640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fd5a806e7b0 0x7fd5a8136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:41572/0 (socket says 172.21.15.26:41572) 2024-08-19T12:45:25.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.449+0000 7fd5aeb91640 1 -- 172.21.15.26:0/2795571738 learned_addr learned my addr 172.21.15.26:0/2795571738 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:25.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5aeb91640 1 -- 172.21.15.26:0/2795571738 >> v1:172.21.15.26:6790/0 conn(0x7fd5a80a5d50 legacy=0x7fd5a8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ae390640 1 -- 172.21.15.26:0/2795571738 >> v1:172.21.15.26:6793/0 conn(0x7fd5a806e7b0 legacy=0x7fd5a8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 1 -- 172.21.15.26:0/2795571738 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1978756631 0 0) 0x7fd5a809e0d0 con 0x7fd5a806e7b0 2024-08-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient(hunting): my global_id is 26243 2024-08-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 1 -- 172.21.15.26:0/2795571738 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd594003520 con 0x7fd5a806e7b0 2024-08-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 1 -- 172.21.15.26:0/2795571738 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1741941177 0 0) 0x7fd5a00017d0 con 0x7fd5a80a5d50 2024-08-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient(hunting): my global_id is 26372 2024-08-19T12:45:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 1 -- 172.21.15.26:0/2795571738 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5a809e0d0 con 0x7fd5a80a5d50 2024-08-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5adb8f640 1 -- 172.21.15.26:0/2795571738 >> v1:172.21.15.104:6789/0 conn(0x7fd5a80a5920 legacy=0x7fd5a813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 1 -- 172.21.15.26:0/2795571738 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2607750976 0 0) 0x7fd594003520 con 0x7fd5a806e7b0 2024-08-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 1 -- 172.21.15.26:0/2795571738 >> v1:172.21.15.26:6790/0 conn(0x7fd5a80a5d50 legacy=0x7fd5a8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 1 -- 172.21.15.26:0/2795571738 >> v1:172.21.15.104:6789/0 conn(0x7fd5a80a5920 legacy=0x7fd5a813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient: _finish_hunting 0 2024-08-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 1 monclient: found mon.noname-e 2024-08-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 1 -- 172.21.15.26:0/2795571738 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a809e3b0 con 0x7fd5a806e7b0 2024-08-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient: _finish_auth 0 2024-08-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient: _check_auth_tickets 2024-08-19T12:45:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 1 -- 172.21.15.26:0/2795571738 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd5a4001a10 con 0x7fd5a806e7b0 2024-08-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient: dump: 2024-08-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.463 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5b05f4640 5 monclient: authenticate success, global_id 26243 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5b05f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 1 -- 172.21.15.26:0/2795571738 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd5a4001ce0 con 0x7fd5a806e7b0 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 1 -- 172.21.15.26:0/2795571738 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd5a4004b20 con 0x7fd5a806e7b0 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5b05f4640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5b05f4640 4 set_mon_vals no callback set 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.452+0000 7fd5ad38e640 10 monclient: dump: 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.466 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.466 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.466 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.466 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.466 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.466 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.466 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.466 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.466 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.453+0000 7fd5b05f4640 10 monclient: shutdown 2024-08-19T12:45:25.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.453+0000 7fd5b05f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:25.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.453+0000 7fd5b05f4640 1 -- 172.21.15.26:0/2795571738 >> v1:172.21.15.26:6793/0 conn(0x7fd5a806e7b0 legacy=0x7fd5a8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.453+0000 7fd5b05f4640 1 -- 172.21.15.26:0/2795571738 shutdown_connections 2024-08-19T12:45:25.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.453+0000 7fd5b05f4640 1 -- 172.21.15.26:0/2795571738 >> 172.21.15.26:0/2795571738 conn(0x7fd5a80a8740 msgr2=0x7fd5a80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:25.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.453+0000 7fd5b05f4640 1 -- 172.21.15.26:0/2795571738 shutdown_connections 2024-08-19T12:45:25.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.453+0000 7fd5b05f4640 1 -- 172.21.15.26:0/2795571738 wait complete. 2024-08-19T12:45:25.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.453+0000 7fd5b05f4640 10 monclient: build_initial_monmap 2024-08-19T12:45:25.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.453+0000 7fd5b05f4640 10 monclient: monmap: 2024-08-19T12:45:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 1 librados: starting msgr at 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 1 librados: starting objecter 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 1 Processor -- start 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 1 -- start start 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 1 librados: setting wanted keys 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 1 librados: calling monclient init 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 10 monclient: init 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 10 monclient: _add_conns ranks=[1,4,2,7,0,5,6,3,8] 2024-08-19T12:45:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 10 monclient(hunting): picked mon.noname-f con 0x7fd5a806e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 10 monclient(hunting): picked mon.noname-c con 0x7fd5a80a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 10 monclient(hunting): picked mon.noname-b con 0x7fd5a80a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a8128b70 con 0x7fd5a806e7b0 2024-08-19T12:45:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a81232d0 con 0x7fd5a80a5d50 2024-08-19T12:45:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a81235b0 con 0x7fd5a80a5920 2024-08-19T12:45:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5b05f4640 10 monclient(hunting): authenticate will time out at 1119.541664s 2024-08-19T12:45:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5aeb91640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd5a80a5d50 0x7fd5a81d6bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38910/0 (socket says 172.21.15.26:38910) 2024-08-19T12:45:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.454+0000 7fd5aeb91640 1 -- 172.21.15.26:0/1168237658 learned_addr learned my addr 172.21.15.26:0/1168237658 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.456+0000 7fd5aeb91640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.26:6790/0 conn(0x7fd5a80a5d50 legacy=0x7fd5a81d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.456+0000 7fd59effd640 10 client.?.objecter ms_handle_connect 0x7fd5a80a5d50 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.456+0000 7fd59effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.456+0000 7fd59effd640 10 client.?.objecter ms_handle_connect 0x7fd5a80a5920 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.456+0000 7fd59effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.456+0000 7fd5adb8f640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.26:6791/0 conn(0x7fd5a80a5920 legacy=0x7fd5a81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd5aeb91640 1 --1- 172.21.15.26:0/1168237658 >> v1:172.21.15.26:6790/0 conn(0x7fd5a80a5d50 0x7fd5a81d6bb0 :-1 s=OPENED pgs=3341 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fd5a81232d0 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4210825786 0 0) 0x7fd5a81235b0 con 0x7fd5a80a5920 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd5ae390640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.104:6789/0 conn(0x7fd5a806e7b0 legacy=0x7fd5a81ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient(hunting): my global_id is 26680 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd58c004eb0 con 0x7fd5a80a5920 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 client.?.objecter ms_handle_connect 0x7fd5a806e7b0 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 538292489 0 0) 0x7fd58c004eb0 con 0x7fd5a80a5920 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5a81235b0 con 0x7fd5a80a5920 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd598003610 con 0x7fd5a80a5920 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient(hunting): dump: 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1765526538 0 0) 0x7fd5a8128b70 con 0x7fd5a806e7b0 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient(hunting): my global_id is 27127 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd58c004eb0 con 0x7fd5a806e7b0 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 368050474 0 0) 0x7fd5a81235b0 con 0x7fd5a80a5920 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.26:6790/0 conn(0x7fd5a80a5d50 legacy=0x7fd5a81d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.104:6789/0 conn(0x7fd5a806e7b0 legacy=0x7fd5a81ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 monclient: found mon.c 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a8123890 con 0x7fd5a80a5920 2024-08-19T12:45:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient: _finish_auth 0 2024-08-19T12:45:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd5b05f4640 5 monclient: authenticate success, global_id 26680 2024-08-19T12:45:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635310299 0 0) 0x7fd5a81232d0 con 0x7fd5a80a5d50 2024-08-19T12:45:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd5b05f4640 10 monclient: _renew_subs 2024-08-19T12:45:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd5b05f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd5b05f4640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a8122290 con 0x7fd5a80a5920 2024-08-19T12:45:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:45:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd5b05f4640 10 client.26680.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd5b05f4640 10 monclient: _renew_subs 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd5b05f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd5b05f4640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fd5a812eb70 con 0x7fd5a80a5920 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd5b05f4640 1 librados: init done 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd598003ba0 con 0x7fd5a80a5920 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd5980040d0 con 0x7fd5a80a5920 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd5acb8d640 4 set_mon_vals no callback set 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.457+0000 7fd59effd640 10 monclient: dump: 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.458+0000 7fd5b05f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:25.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.458+0000 7fd5b05f4640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd5900065c0 con 0x7fd590002860 2024-08-19T12:45:25.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.458+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd598004270 con 0x7fd5a80a5920 2024-08-19T12:45:25.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.458+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(283..283 src has 1..283) ==== 5364+0+0 (unknown 985969326 0 0) 0x7fd59804e240 con 0x7fd5a80a5920 2024-08-19T12:45:25.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.458+0000 7fd59effd640 10 client.26680.objecter ms_dispatch 0x7fd5a8000b90 osd_map(283..283 src has 1..283) 2024-08-19T12:45:25.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.458+0000 7fd59effd640 3 client.26680.objecter handle_osd_map got epochs [283,283] > 0 2024-08-19T12:45:25.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.458+0000 7fd59effd640 3 client.26680.objecter handle_osd_map decoding full epoch 283 2024-08-19T12:45:25.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.458+0000 7fd59effd640 20 client.26680.objecter dump_active .. 0 homeless 2024-08-19T12:45:25.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.459+0000 7fd5aeb91640 10 monclient: get_auth_request con 0x7fd58c040f10 auth_method 0 2024-08-19T12:45:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd5ae390640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.26:6791/0 conn(0x7fd590002860 legacy=0x7fd590004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd59effd640 10 client.26680.objecter ms_handle_connect 0x7fd590002860 2024-08-19T12:45:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd59effd640 10 client.26680.objecter resend_mon_ops 2024-08-19T12:45:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336354644 0 0) 0x7fd5900065c0 con 0x7fd590002860 2024-08-19T12:45:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd59effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd59effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd59effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd59effd640 10 monclient(hunting): my global_id is 26689 2024-08-19T12:45:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5a81232d0 con 0x7fd590002860 2024-08-19T12:45:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2643736491 0 0) 0x7fd5a81232d0 con 0x7fd590002860 2024-08-19T12:45:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5900065c0 con 0x7fd590002860 2024-08-19T12:45:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd5a4004b60 con 0x7fd590002860 2024-08-19T12:45:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.460+0000 7fd59effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.461+0000 7fd59effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.461+0000 7fd59effd640 10 monclient: dump: 2024-08-19T12:45:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.461+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2602543361 0 0) 0x7fd5900065c0 con 0x7fd590002860 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.461+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd59000a160 con 0x7fd590002860 2024-08-19T12:45:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.461+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd5a4001580 con 0x7fd590002860 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.461+0000 7fd59effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.461+0000 7fd59effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.461+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.26:6791/0 conn(0x7fd590002860 legacy=0x7fd590004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.461+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.26:6791/0 conn(0x7fd590002860 legacy=0x7fd590004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.461+0000 7fd59effd640 10 client.26680.objecter ms_handle_connect 0x7fd58c040f10 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.467+0000 7fd5b05f4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.467+0000 7fd5b05f4640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd590009e70 con 0x7fd590009130 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd5ae390640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.26:6791/0 conn(0x7fd590009130 legacy=0x7fd5900095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 10 client.26680.objecter ms_handle_connect 0x7fd590009130 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 10 client.26680.objecter resend_mon_ops 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 825091965 0 0) 0x7fd590009e70 con 0x7fd590009130 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 10 monclient(hunting): my global_id is 26698 2024-08-19T12:45:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5900065c0 con 0x7fd590009130 2024-08-19T12:45:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2539444798 0 0) 0x7fd5900065c0 con 0x7fd590009130 2024-08-19T12:45:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd590009e70 con 0x7fd590009130 2024-08-19T12:45:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd5a4004e60 con 0x7fd590009130 2024-08-19T12:45:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 10 monclient: dump: 2024-08-19T12:45:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1962123963 0 0) 0x7fd590009e70 con 0x7fd590009130 2024-08-19T12:45:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd59000c410 con 0x7fd590009130 2024-08-19T12:45:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7867 (unknown 0 0 182238659) 0x7fd5a40053b0 con 0x7fd590009130 2024-08-19T12:45:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.26:6791/0 conn(0x7fd590009130 legacy=0x7fd5900095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd59effd640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.26:6791/0 conn(0x7fd590009130 legacy=0x7fd5900095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:25.488 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:25.488 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:25.488 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:25.488 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:25.488 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 46, 2024-08-19T12:45:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 438500, 2024-08-19T12:45:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.489 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.490 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.491 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:25.492 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:25.493 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:25.493 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:25.493 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:25.493 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:25.493 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:25.493 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:25.493 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:25.493 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:25.493 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.493 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.493 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.493 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.494 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.494 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.494 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.494 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.494 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.494 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.494 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.494 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.494 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:25.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.494 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.495 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:25.496 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:25.496 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.496 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.496 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.496 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:25.496 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:25.496 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.496 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.496 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.497 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:25.497 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.497 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.497 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.497 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:25.497 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:25.497 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.497 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.497 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.498 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:25.498 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:25.498 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.498 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.498 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.498 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:25.498 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.498 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.498 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:25.499 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:25.499 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.499 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.499 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.499 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:25.500 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:25.500 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:45:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:25.512 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.470+0000 7fd5b05f4640 10 librados: watch_flush enter 2024-08-19T12:45:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5b05f4640 10 librados: watch_flush exit 2024-08-19T12:45:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5b05f4640 20 client.26680.objecter shutdown clearing up homeless session... 2024-08-19T12:45:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5b05f4640 10 client.26680.objecter successfully canceled tick 2024-08-19T12:45:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5b05f4640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.104:6800/899887506 conn(0x7fd58c040f10 legacy=0x7fd58c0433d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5b05f4640 10 monclient: shutdown 2024-08-19T12:45:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5b05f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5b05f4640 1 -- 172.21.15.26:0/1168237658 >> v1:172.21.15.26:6791/0 conn(0x7fd5a80a5920 legacy=0x7fd5a81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5aeb91640 1 -- 172.21.15.26:0/1168237658 reap_dead start 2024-08-19T12:45:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5aeb91640 1 -- 172.21.15.26:0/1168237658 reap_dead start 2024-08-19T12:45:25.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5b05f4640 1 -- 172.21.15.26:0/1168237658 shutdown_connections 2024-08-19T12:45:25.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5b05f4640 1 -- 172.21.15.26:0/1168237658 >> 172.21.15.26:0/1168237658 conn(0x7fd5a80a8740 msgr2=0x7fd5a81351c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:25.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5b05f4640 1 -- 172.21.15.26:0/1168237658 shutdown_connections 2024-08-19T12:45:25.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5b05f4640 1 -- 172.21.15.26:0/1168237658 wait complete. 2024-08-19T12:45:25.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.471+0000 7fd5b05f4640 1 librados: shutdown 2024-08-19T12:45:25.515 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:45:25.515 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-08-19T12:45:25.515 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:45:25.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.614+0000 7f4c726b1640 10 monclient: get_monmap_and_config 2024-08-19T12:45:25.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.614+0000 7f4c726b1640 10 monclient: build_initial_monmap 2024-08-19T12:45:25.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.614+0000 7f4c726b1640 10 monclient: monmap: 2024-08-19T12:45:25.615 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:25.615 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.615 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:25.615013+0000 2024-08-19T12:45:25.615 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:25.615013+0000 2024-08-19T12:45:25.615 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:25.615 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.615 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:25.615 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:25.616 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:25.616 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:25.616 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:25.616 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:25.616 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:25.616 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:25.616 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:25.616 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.614+0000 7f4c726b1640 1 Processor -- start 2024-08-19T12:45:25.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.614+0000 7f4c726b1640 1 -- start start 2024-08-19T12:45:25.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.614+0000 7f4c726b1640 10 monclient: init 2024-08-19T12:45:25.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c726b1640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:25.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c726b1640 10 monclient: _add_conns ranks=[6,0,3,1,4,7,8,5,2] 2024-08-19T12:45:25.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c726b1640 10 monclient(hunting): picked mon.noname-d con 0x7f4c6c06e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:25.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c726b1640 10 monclient(hunting): picked mon.noname-a con 0x7f4c6c0a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:25.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c726b1640 10 monclient(hunting): picked mon.noname-g con 0x7f4c6c0a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:25.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c726b1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c6c136790 con 0x7f4c6c0a5920 2024-08-19T12:45:25.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c726b1640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c6c09ddf0 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c726b1640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c6c09e0d0 con 0x7f4c6c06e7b0 2024-08-19T12:45:25.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c726b1640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:25.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c726b1640 10 monclient(hunting): authenticate will time out at 1119.702478s 2024-08-19T12:45:25.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c6bfff640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f4c6c06e7b0 legacy=0x7f4c6c136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:25.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c6b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4c6c0a5920 0x7f4c6c13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38150/0 (socket says 172.21.15.26:38150) 2024-08-19T12:45:25.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.615+0000 7f4c6b7fe640 1 -- 172.21.15.26:0/1804819232 learned_addr learned my addr 172.21.15.26:0/1804819232 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:25.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.617+0000 7f4c6b7fe640 1 -- 172.21.15.26:0/1804819232 >> v1:172.21.15.26:6789/0 conn(0x7f4c6c0a5920 legacy=0x7f4c6c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.617+0000 7f4c70c4e640 1 -- 172.21.15.26:0/1804819232 >> v1:172.21.15.104:6790/0 conn(0x7f4c6c0a5d50 legacy=0x7f4c6c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.617+0000 7f4c6affd640 1 -- 172.21.15.26:0/1804819232 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 648595601 0 0) 0x7f4c6c136790 con 0x7f4c6c0a5920 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.617+0000 7f4c6affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.617+0000 7f4c6affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.617+0000 7f4c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.617+0000 7f4c6affd640 10 monclient(hunting): my global_id is 144648 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.617+0000 7f4c6affd640 1 -- 172.21.15.26:0/1804819232 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c54003520 con 0x7f4c6c0a5920 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 1 -- 172.21.15.26:0/1804819232 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1736257260 0 0) 0x7f4c64001820 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient(hunting): my global_id is 154245 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 1 -- 172.21.15.26:0/1804819232 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c6c136790 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 1 -- 172.21.15.26:0/1804819232 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2513450381 0 0) 0x7f4c54003520 con 0x7f4c6c0a5920 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 1 -- 172.21.15.26:0/1804819232 >> v1:172.21.15.26:6792/0 conn(0x7f4c6c06e7b0 legacy=0x7f4c6c136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 1 -- 172.21.15.26:0/1804819232 >> v1:172.21.15.104:6790/0 conn(0x7f4c6c0a5d50 legacy=0x7f4c6c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 1 monclient: found mon.noname-a 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 1 -- 172.21.15.26:0/1804819232 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6c09e3b0 con 0x7f4c6c0a5920 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient: _finish_auth 0 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c726b1640 5 monclient: authenticate success, global_id 144648 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 1 -- 172.21.15.26:0/1804819232 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c60001ae0 con 0x7f4c6c0a5920 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c726b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:45:25.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient: dump: 2024-08-19T12:45:25.625 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.625 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.625 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.625 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.625 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.625 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.625 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.625 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c726b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 1 -- 172.21.15.26:0/1804819232 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4c600048a0 con 0x7f4c6c0a5920 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 1 -- 172.21.15.26:0/1804819232 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c60004e10 con 0x7f4c6c0a5920 2024-08-19T12:45:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c726b1640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:25.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c726b1640 4 set_mon_vals no callback set 2024-08-19T12:45:25.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:25.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c6affd640 10 monclient: dump: 2024-08-19T12:45:25.627 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.627 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.627 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.627 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.627 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.628 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.628 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.628 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.628 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.628 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c726b1640 10 monclient: shutdown 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c726b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c726b1640 1 -- 172.21.15.26:0/1804819232 >> v1:172.21.15.26:6789/0 conn(0x7f4c6c0a5920 legacy=0x7f4c6c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c726b1640 1 -- 172.21.15.26:0/1804819232 shutdown_connections 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c726b1640 1 -- 172.21.15.26:0/1804819232 >> 172.21.15.26:0/1804819232 conn(0x7f4c6c0a8740 msgr2=0x7f4c6c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c726b1640 1 -- 172.21.15.26:0/1804819232 shutdown_connections 2024-08-19T12:45:25.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.618+0000 7f4c726b1640 1 -- 172.21.15.26:0/1804819232 wait complete. 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.619+0000 7f4c726b1640 10 monclient: build_initial_monmap 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.619+0000 7f4c726b1640 10 monclient: monmap: 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:25.630 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.619+0000 7f4c726b1640 1 librados: starting msgr at 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.619+0000 7f4c726b1640 1 librados: starting objecter 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.619+0000 7f4c726b1640 1 Processor -- start 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 1 -- start start 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 1 librados: setting wanted keys 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 1 librados: calling monclient init 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 10 monclient: init 2024-08-19T12:45:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 10 monclient: _add_conns ranks=[1,5,0,4,3,7,8,6,2] 2024-08-19T12:45:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 10 monclient(hunting): picked mon.noname-f con 0x7f4c6c06e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 10 monclient(hunting): picked mon.noname-h con 0x7f4c6c0a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 10 monclient(hunting): picked mon.noname-a con 0x7f4c6c0a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c6c124350 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c6c1228d0 con 0x7f4c6c06e7b0 2024-08-19T12:45:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c6c0ab8e0 con 0x7f4c6c0a5920 2024-08-19T12:45:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c726b1640 10 monclient(hunting): authenticate will time out at 1119.707511s 2024-08-19T12:45:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c70c4e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4c6c0a5d50 0x7f4c6c1d6f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38154/0 (socket says 172.21.15.26:38154) 2024-08-19T12:45:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.620+0000 7f4c70c4e640 1 -- 172.21.15.26:0/69442014 learned_addr learned my addr 172.21.15.26:0/69442014 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.622+0000 7f4c70c4e640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.26:6789/0 conn(0x7f4c6c0a5d50 legacy=0x7f4c6c1d6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.622+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:45:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.622+0000 7f4c6b7fe640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.104:6791/0 conn(0x7f4c6c0a5920 legacy=0x7f4c6c1d37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.622+0000 7f4c6bfff640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.104:6789/0 conn(0x7f4c6c06e7b0 legacy=0x7f4c6c1288b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 client.?.objecter ms_handle_connect 0x7f4c6c0a5d50 2024-08-19T12:45:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 client.?.objecter ms_handle_connect 0x7f4c6c0a5920 2024-08-19T12:45:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 client.?.objecter ms_handle_connect 0x7f4c6c06e7b0 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 972726302 0 0) 0x7f4c6c124350 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 monclient(hunting): my global_id is 144657 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c58004eb0 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104974721 0 0) 0x7f4c6c1228d0 con 0x7f4c6c06e7b0 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 monclient(hunting): my global_id is 27136 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c6c124350 con 0x7f4c6c06e7b0 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1365180914 0 0) 0x7f4c6c0ab8e0 con 0x7f4c6c0a5920 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 10 monclient(hunting): my global_id is 26375 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.624+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c6c1228d0 con 0x7f4c6c0a5920 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1485156721 0 0) 0x7f4c58004eb0 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c6c0ab8e0 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c64003990 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:45:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 10 monclient(hunting): dump: 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.640 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 422034403 0 0) 0x7f4c6c0ab8e0 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.104:6791/0 conn(0x7f4c6c0a5920 legacy=0x7f4c6c1d37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.104:6789/0 conn(0x7f4c6c06e7b0 legacy=0x7f4c6c1288b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 1 monclient: found mon.a 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6c134a10 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 10 monclient: _finish_auth 0 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c726b1640 5 monclient: authenticate success, global_id 144657 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c726b1640 10 monclient: _renew_subs 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c726b1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c726b1640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4c6c12f360 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c726b1640 10 client.144657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c726b1640 10 monclient: _renew_subs 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c726b1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c726b1640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4c6c122ce0 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c726b1640 1 librados: init done 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4c64003e90 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:25.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c64004400 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c6a7fc640 4 set_mon_vals no callback set 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.625+0000 7f4c68ff9640 10 monclient: dump: 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.626+0000 7f4c726b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.626+0000 7f4c726b1640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4c4c0065c0 con 0x7f4c4c002860 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.626+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f4c640045a0 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.626+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(283..283 src has 1..283) ==== 5364+0+0 (unknown 985969326 0 0) 0x7f4c6404e480 con 0x7f4c6c0a5d50 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.626+0000 7f4c68ff9640 10 client.144657.objecter ms_dispatch 0x7f4c6c000b90 osd_map(283..283 src has 1..283) 2024-08-19T12:45:25.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.626+0000 7f4c68ff9640 3 client.144657.objecter handle_osd_map got epochs [283,283] > 0 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.626+0000 7f4c68ff9640 3 client.144657.objecter handle_osd_map decoding full epoch 283 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.626+0000 7f4c68ff9640 20 client.144657.objecter dump_active .. 0 homeless 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.626+0000 7f4c6b7fe640 10 monclient: get_auth_request con 0x7f4c58041220 auth_method 0 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c6bfff640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.26:6793/0 conn(0x7f4c4c002860 legacy=0x7f4c4c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 10 client.144657.objecter ms_handle_connect 0x7f4c4c002860 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 10 client.144657.objecter resend_mon_ops 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 213463685 0 0) 0x7f4c4c0065c0 con 0x7f4c4c002860 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 10 monclient(hunting): my global_id is 26252 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c6c0ab8e0 con 0x7f4c4c002860 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3408036790 0 0) 0x7f4c6c0ab8e0 con 0x7f4c4c002860 2024-08-19T12:45:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c4c0065c0 con 0x7f4c4c002860 2024-08-19T12:45:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c5c004470 con 0x7f4c4c002860 2024-08-19T12:45:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.628+0000 7f4c68ff9640 10 monclient: dump: 2024-08-19T12:45:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.629+0000 7f4c68ff9640 10 client.144657.objecter ms_handle_connect 0x7f4c58041220 2024-08-19T12:45:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.629+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3889454236 0 0) 0x7f4c4c0065c0 con 0x7f4c4c002860 2024-08-19T12:45:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.629+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c4c00a160 con 0x7f4c4c002860 2024-08-19T12:45:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.629+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4c5c005940 con 0x7f4c4c002860 2024-08-19T12:45:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.629+0000 7f4c68ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.629+0000 7f4c68ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.629+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.26:6793/0 conn(0x7f4c4c002860 legacy=0x7f4c4c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.629+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.26:6793/0 conn(0x7f4c4c002860 legacy=0x7f4c4c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.635+0000 7f4c726b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.635+0000 7f4c726b1640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4c4c009e50 con 0x7f4c4c009110 2024-08-19T12:45:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.637+0000 7f4c6bfff640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.26:6793/0 conn(0x7f4c4c009110 legacy=0x7f4c4c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.637+0000 7f4c68ff9640 10 client.144657.objecter ms_handle_connect 0x7f4c4c009110 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.637+0000 7f4c68ff9640 10 client.144657.objecter resend_mon_ops 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.637+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4166551771 0 0) 0x7f4c4c009e50 con 0x7f4c4c009110 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 10 monclient(hunting): my global_id is 26261 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c4c0065c0 con 0x7f4c4c009110 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4268572386 0 0) 0x7f4c4c0065c0 con 0x7f4c4c009110 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c4c009e50 con 0x7f4c4c009110 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c5c004c00 con 0x7f4c4c009110 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 10 monclient: dump: 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2182738368 0 0) 0x7f4c4c009e50 con 0x7f4c4c009110 2024-08-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4c4c00c3f0 con 0x7f4c4c009110 2024-08-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7700 (unknown 0 0 1883807018) 0x7f4c5c004f80 con 0x7f4c4c009110 2024-08-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.26:6793/0 conn(0x7f4c4c009110 legacy=0x7f4c4c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c68ff9640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.26:6793/0 conn(0x7f4c4c009110 legacy=0x7f4c4c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:25.652 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 46, 2024-08-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 438655, 2024-08-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:25.653 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.654 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.655 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.655 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.655 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.655 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.655 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.655 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.655 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:25.655 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:25.655 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.655 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.656 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.656 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.656 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.656 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.656 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.656 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.656 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.656 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.656 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.656 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:25.657 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:25.657 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:25.657 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:25.657 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:25.657 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:25.657 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:25.657 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:25.657 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:25.657 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:25.657 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:25.657 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:25.657 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:25.658 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:25.658 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:25.658 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:25.658 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:25.658 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.658 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.658 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.658 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.658 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.658 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.658 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.658 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.659 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.659 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.659 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.659 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.659 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:25.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.659 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:25.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.659 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:25.659 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:25.659 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:25.661 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:25.662 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:25.663 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:25.663 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.663 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.663 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:25.674 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:25.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c726b1640 10 librados: watch_flush enter 2024-08-19T12:45:25.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c726b1640 10 librados: watch_flush exit 2024-08-19T12:45:25.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c726b1640 20 client.144657.objecter shutdown clearing up homeless session... 2024-08-19T12:45:25.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c726b1640 10 client.144657.objecter successfully canceled tick 2024-08-19T12:45:25.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c726b1640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.104:6800/899887506 conn(0x7f4c58041220 legacy=0x7f4c580436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c726b1640 10 monclient: shutdown 2024-08-19T12:45:25.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c726b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:25.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c726b1640 1 -- 172.21.15.26:0/69442014 >> v1:172.21.15.26:6789/0 conn(0x7f4c6c0a5d50 legacy=0x7f4c6c1d6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c70c4e640 1 -- 172.21.15.26:0/69442014 reap_dead start 2024-08-19T12:45:25.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.638+0000 7f4c70c4e640 1 -- 172.21.15.26:0/69442014 reap_dead start 2024-08-19T12:45:25.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.639+0000 7f4c726b1640 1 -- 172.21.15.26:0/69442014 shutdown_connections 2024-08-19T12:45:25.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.639+0000 7f4c726b1640 1 -- 172.21.15.26:0/69442014 >> 172.21.15.26:0/69442014 conn(0x7f4c6c0a8740 msgr2=0x7f4c6c135100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:25.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.639+0000 7f4c726b1640 1 -- 172.21.15.26:0/69442014 shutdown_connections 2024-08-19T12:45:25.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.639+0000 7f4c726b1640 1 -- 172.21.15.26:0/69442014 wait complete. 2024-08-19T12:45:25.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.639+0000 7f4c726b1640 1 librados: shutdown 2024-08-19T12:45:25.677 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:45:25.677 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-08-19T12:45:25.678 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:45:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.781+0000 7f40a2615640 10 monclient: get_monmap_and_config 2024-08-19T12:45:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.781+0000 7f40a2615640 10 monclient: build_initial_monmap 2024-08-19T12:45:25.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.781+0000 7f40a2615640 10 monclient: monmap: 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:25.782595+0000 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:25.782595+0000 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:25.783 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:25.784 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.782+0000 7f40a2615640 1 Processor -- start 2024-08-19T12:45:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.782+0000 7f40a2615640 1 -- start start 2024-08-19T12:45:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.782+0000 7f40a2615640 10 monclient: init 2024-08-19T12:45:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.782+0000 7f40a2615640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.782+0000 7f40a2615640 10 monclient: _add_conns ranks=[1,7,6,2,0,4,3,8,5] 2024-08-19T12:45:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.783+0000 7f40a2615640 10 monclient(hunting): picked mon.noname-f con 0x7f409c06e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.783+0000 7f40a2615640 10 monclient(hunting): picked mon.noname-i con 0x7f409c0a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.783+0000 7f40a2615640 10 monclient(hunting): picked mon.noname-d con 0x7f409c0a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.783+0000 7f40a2615640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409c136790 con 0x7f409c06e7b0 2024-08-19T12:45:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.783+0000 7f40a2615640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409c09ddf0 con 0x7f409c0a5d50 2024-08-19T12:45:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.783+0000 7f40a2615640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409c09e0d0 con 0x7f409c0a5920 2024-08-19T12:45:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.783+0000 7f40a2615640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.783+0000 7f40a0bb2640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f409c0a5d50 legacy=0x7f409c141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.783+0000 7f40a2615640 10 monclient(hunting): authenticate will time out at 1119.870125s 2024-08-19T12:45:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.783+0000 7f409b7fe640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f409c0a5920 0x7f409c13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:38584/0 (socket says 172.21.15.26:38584) 2024-08-19T12:45:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.783+0000 7f409b7fe640 1 -- 172.21.15.26:0/2369964980 learned_addr learned my addr 172.21.15.26:0/2369964980 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:25.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.783+0000 7f409bfff640 1 --1- 172.21.15.26:0/2369964980 >> v1:172.21.15.104:6789/0 conn(0x7f409c06e7b0 0x7f409c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:60120/0 (socket says 172.21.15.26:60120) 2024-08-19T12:45:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409b7fe640 1 -- 172.21.15.26:0/2369964980 >> v1:172.21.15.104:6792/0 conn(0x7f409c0a5920 legacy=0x7f409c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409bfff640 1 -- 172.21.15.26:0/2369964980 >> v1:172.21.15.104:6789/0 conn(0x7f409c06e7b0 legacy=0x7f409c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 1 -- 172.21.15.26:0/2369964980 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2371902739 0 0) 0x7f409c09e0d0 con 0x7f409c0a5920 2024-08-19T12:45:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient(hunting): my global_id is 135322 2024-08-19T12:45:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 1 -- 172.21.15.26:0/2369964980 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4084003520 con 0x7f409c0a5920 2024-08-19T12:45:25.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 1 -- 172.21.15.26:0/2369964980 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1184976785 0 0) 0x7f409c136790 con 0x7f409c06e7b0 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient(hunting): my global_id is 27145 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 1 -- 172.21.15.26:0/2369964980 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f409c09e0d0 con 0x7f409c06e7b0 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 1 -- 172.21.15.26:0/2369964980 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3132953693 0 0) 0x7f4084003520 con 0x7f409c0a5920 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 1 -- 172.21.15.26:0/2369964980 >> v1:172.21.15.26:6792/0 conn(0x7f409c0a5d50 legacy=0x7f409c141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 1 -- 172.21.15.26:0/2369964980 >> v1:172.21.15.104:6789/0 conn(0x7f409c06e7b0 legacy=0x7f409c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 1 monclient: found mon.noname-i 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:45:25.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 1 -- 172.21.15.26:0/2369964980 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409c09e3b0 con 0x7f409c0a5920 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient: _finish_auth 0 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 1 -- 172.21.15.26:0/2369964980 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4090001ae0 con 0x7f409c0a5920 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f40a2615640 5 monclient: authenticate success, global_id 135322 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f40a2615640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f409affd640 10 monclient: dump: 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.792 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.785+0000 7f40a2615640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:25.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f409affd640 1 -- 172.21.15.26:0/2369964980 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40900048a0 con 0x7f409c0a5920 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f409affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f409affd640 1 -- 172.21.15.26:0/2369964980 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4090004e10 con 0x7f409c0a5920 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f40a2615640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f40a2615640 4 set_mon_vals no callback set 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f409affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f409affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f409affd640 10 monclient: dump: 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f40a2615640 10 monclient: shutdown 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f40a2615640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f40a2615640 1 -- 172.21.15.26:0/2369964980 >> v1:172.21.15.104:6792/0 conn(0x7f409c0a5920 legacy=0x7f409c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f40a2615640 1 -- 172.21.15.26:0/2369964980 shutdown_connections 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f40a2615640 1 -- 172.21.15.26:0/2369964980 >> 172.21.15.26:0/2369964980 conn(0x7f409c0a8740 msgr2=0x7f409c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f40a2615640 1 -- 172.21.15.26:0/2369964980 shutdown_connections 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.786+0000 7f40a2615640 1 -- 172.21.15.26:0/2369964980 wait complete. 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 10 monclient: build_initial_monmap 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 10 monclient: monmap: 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 1 librados: starting msgr at 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 1 librados: starting objecter 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 1 Processor -- start 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 1 -- start start 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 1 librados: setting wanted keys 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 1 librados: calling monclient init 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 10 monclient: init 2024-08-19T12:45:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 10 monclient: _add_conns ranks=[0,4,6,1,8,2,5,7,3] 2024-08-19T12:45:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 10 monclient(hunting): picked mon.noname-a con 0x7f409c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 10 monclient(hunting): picked mon.noname-c con 0x7f409c0a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.787+0000 7f40a2615640 10 monclient(hunting): picked mon.noname-d con 0x7f409c0a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.788+0000 7f40a2615640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409c09f6d0 con 0x7f409c06e7b0 2024-08-19T12:45:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.788+0000 7f40a2615640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409c1344e0 con 0x7f409c0a5920 2024-08-19T12:45:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.788+0000 7f40a2615640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409c134a10 con 0x7f409c0a5d50 2024-08-19T12:45:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.788+0000 7f40a2615640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.788+0000 7f40a2615640 10 monclient(hunting): authenticate will time out at 1119.875047s 2024-08-19T12:45:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.788+0000 7f40a0bb2640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f409c0a5d50 legacy=0x7f409c1d66b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.788+0000 7f409b7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f409c0a5920 0x7f409c09efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:45914/0 (socket says 172.21.15.26:45914) 2024-08-19T12:45:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.788+0000 7f409b7fe640 1 -- 172.21.15.26:0/3649531549 learned_addr learned my addr 172.21.15.26:0/3649531549 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f409bfff640 1 -- 172.21.15.26:0/3649531549 >> v1:172.21.15.26:6789/0 conn(0x7f409c06e7b0 legacy=0x7f409c09e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f409b7fe640 1 -- 172.21.15.26:0/3649531549 >> v1:172.21.15.26:6791/0 conn(0x7f409c0a5920 legacy=0x7f409c09efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 client.?.objecter ms_handle_connect 0x7f409c06e7b0 2024-08-19T12:45:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 client.?.objecter ms_handle_connect 0x7f409c0a5920 2024-08-19T12:45:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 749195139 0 0) 0x7f409c1344e0 con 0x7f409c0a5920 2024-08-19T12:45:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): my global_id is 26707 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4088004eb0 con 0x7f409c0a5920 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1505333331 0 0) 0x7f409c09f6d0 con 0x7f409c06e7b0 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): my global_id is 144666 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f409c1344e0 con 0x7f409c06e7b0 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3696082886 0 0) 0x7f4088004eb0 con 0x7f409c0a5920 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f409c09f6d0 con 0x7f409c0a5920 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4090004d90 con 0x7f409c0a5920 2024-08-19T12:45:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): dump: 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1570862748 0 0) 0x7f409c1344e0 con 0x7f409c06e7b0 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4088004eb0 con 0x7f409c06e7b0 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f408c001d30 con 0x7f409c06e7b0 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient(hunting): dump: 2024-08-19T12:45:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.807 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2130971571 0 0) 0x7f409c09f6d0 con 0x7f409c0a5920 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 >> v1:172.21.15.26:6792/0 conn(0x7f409c0a5d50 legacy=0x7f409c1d66b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 >> v1:172.21.15.26:6789/0 conn(0x7f409c06e7b0 legacy=0x7f409c09e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 monclient: found mon.c 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409c09fbb0 con 0x7f409c0a5920 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient: _finish_auth 0 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f4098ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:25.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f40a2615640 5 monclient: authenticate success, global_id 26707 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f40a2615640 10 monclient: _renew_subs 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f40a2615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f40a2615640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f409c0a0df0 con 0x7f409c0a5920 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f40a2615640 10 client.26707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f40a2615640 10 monclient: _renew_subs 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f40a2615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.790+0000 7f40a2615640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f409c132690 con 0x7f409c0a5920 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.791+0000 7f40a2615640 1 librados: init done 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.791+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4090001d30 con 0x7f409c0a5920 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.791+0000 7f4098ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.791+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f40900048a0 con 0x7f409c0a5920 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.791+0000 7f4098ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.791+0000 7f409a7fc640 4 set_mon_vals no callback set 2024-08-19T12:45:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.791+0000 7f4098ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.791+0000 7f4098ff9640 10 monclient: dump: 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.791+0000 7f40a2615640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.791+0000 7f40a2615640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f407c0065c0 con 0x7f407c002860 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.791+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f4090011a00 con 0x7f409c0a5920 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.792+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(284..284 src has 1..284) ==== 5699+0+0 (unknown 2697114685 0 0) 0x7f409004e350 con 0x7f409c0a5920 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.792+0000 7f4098ff9640 10 client.26707.objecter ms_dispatch 0x7f409c000b90 osd_map(284..284 src has 1..284) 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.792+0000 7f4098ff9640 3 client.26707.objecter handle_osd_map got epochs [284,284] > 0 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.792+0000 7f4098ff9640 3 client.26707.objecter handle_osd_map decoding full epoch 284 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.792+0000 7f4098ff9640 20 client.26707.objecter dump_active .. 0 homeless 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.792+0000 7f40a0bb2640 10 monclient: get_auth_request con 0x7f40880417a0 auth_method 0 2024-08-19T12:45:25.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f409bfff640 1 -- 172.21.15.26:0/3649531549 >> v1:172.21.15.104:6789/0 conn(0x7f407c002860 legacy=0x7f407c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 10 client.26707.objecter ms_handle_connect 0x7f407c002860 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 10 client.26707.objecter resend_mon_ops 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2746150397 0 0) 0x7f407c0065c0 con 0x7f407c002860 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 10 monclient(hunting): my global_id is 27154 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f409c09f6d0 con 0x7f407c002860 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 10 client.26707.objecter ms_handle_connect 0x7f40880417a0 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 133454662 0 0) 0x7f409c09f6d0 con 0x7f407c002860 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f407c0065c0 con 0x7f407c002860 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f408c004d50 con 0x7f407c002860 2024-08-19T12:45:25.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 10 monclient: dump: 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 629689004 0 0) 0x7f407c0065c0 con 0x7f407c002860 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.794+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f407c00a160 con 0x7f407c002860 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.795+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f408c0070b0 con 0x7f407c002860 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.795+0000 7f4098ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.795+0000 7f4098ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.795+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 >> v1:172.21.15.104:6789/0 conn(0x7f407c002860 legacy=0x7f407c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.795+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 >> v1:172.21.15.104:6789/0 conn(0x7f407c002860 legacy=0x7f407c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.801+0000 7f40a2615640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.801+0000 7f40a2615640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f407c00c4f0 con 0x7f407c009110 2024-08-19T12:45:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.803+0000 7f409bfff640 1 -- 172.21.15.26:0/3649531549 >> v1:172.21.15.104:6789/0 conn(0x7f407c009110 legacy=0x7f407c009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.803+0000 7f4098ff9640 10 client.26707.objecter ms_handle_connect 0x7f407c009110 2024-08-19T12:45:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.803+0000 7f4098ff9640 10 client.26707.objecter resend_mon_ops 2024-08-19T12:45:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.803+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3102146617 0 0) 0x7f407c00c4f0 con 0x7f407c009110 2024-08-19T12:45:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.803+0000 7f4098ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.803+0000 7f4098ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.803+0000 7f4098ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.803+0000 7f4098ff9640 10 monclient(hunting): my global_id is 27163 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.803+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f407c0065c0 con 0x7f407c009110 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.803+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1866742440 0 0) 0x7f407c0065c0 con 0x7f407c009110 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f407c00c4f0 con 0x7f407c009110 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f408c0057f0 con 0x7f407c009110 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f4098ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f4098ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f4098ff9640 10 monclient: dump: 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2959512026 0 0) 0x7f407c00c4f0 con 0x7f407c009110 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f407c009ce0 con 0x7f407c009110 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 3721452944) 0x7f408c00a010 con 0x7f407c009110 2024-08-19T12:45:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f4098ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f4098ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 >> v1:172.21.15.104:6789/0 conn(0x7f407c009110 legacy=0x7f407c009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f4098ff9640 1 -- 172.21.15.26:0/3649531549 >> v1:172.21.15.104:6789/0 conn(0x7f407c009110 legacy=0x7f407c009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:25.819 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:25.819 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:25.819 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:25.819 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:25.819 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 46, 2024-08-19T12:45:25.819 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 438808, 2024-08-19T12:45:25.819 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:25.820 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:25.823 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.824 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.826 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:25.826 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:25.826 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:25.826 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.826 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.826 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:25.829 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.830 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:45:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:25.839 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:25.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f40a2615640 10 librados: watch_flush enter 2024-08-19T12:45:25.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f40a2615640 10 librados: watch_flush exit 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f40a2615640 20 client.26707.objecter shutdown clearing up homeless session... 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f40a2615640 10 client.26707.objecter successfully canceled tick 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f40a2615640 1 -- 172.21.15.26:0/3649531549 >> v1:172.21.15.104:6800/899887506 conn(0x7f40880417a0 legacy=0x7f4088043c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f40a2615640 10 monclient: shutdown 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f40a2615640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f40a2615640 1 -- 172.21.15.26:0/3649531549 >> v1:172.21.15.26:6791/0 conn(0x7f409c0a5920 legacy=0x7f409c09efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f40a0bb2640 1 -- 172.21.15.26:0/3649531549 reap_dead start 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f40a0bb2640 1 -- 172.21.15.26:0/3649531549 reap_dead start 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f40a2615640 1 -- 172.21.15.26:0/3649531549 shutdown_connections 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.804+0000 7f40a2615640 1 -- 172.21.15.26:0/3649531549 >> 172.21.15.26:0/3649531549 conn(0x7f409c0a8740 msgr2=0x7f409c06d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.805+0000 7f40a2615640 1 -- 172.21.15.26:0/3649531549 shutdown_connections 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.805+0000 7f40a2615640 1 -- 172.21.15.26:0/3649531549 wait complete. 2024-08-19T12:45:25.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.805+0000 7f40a2615640 1 librados: shutdown 2024-08-19T12:45:25.842 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:45:25.842 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-08-19T12:45:25.842 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:45:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.942+0000 7f0629260640 10 monclient: get_monmap_and_config 2024-08-19T12:45:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.942+0000 7f0629260640 10 monclient: build_initial_monmap 2024-08-19T12:45:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.942+0000 7f0629260640 10 monclient: monmap: 2024-08-19T12:45:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:25.943171+0000 2024-08-19T12:45:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:25.943171+0000 2024-08-19T12:45:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.943 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:25.944 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:25.944 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.942+0000 7f0629260640 1 Processor -- start 2024-08-19T12:45:25.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.942+0000 7f0629260640 1 -- start start 2024-08-19T12:45:25.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.942+0000 7f0629260640 10 monclient: init 2024-08-19T12:45:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f0629260640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f0629260640 10 monclient: _add_conns ranks=[8,5,6,1,2,0,4,3,7] 2024-08-19T12:45:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f0629260640 10 monclient(hunting): picked mon.noname-e con 0x7f062406e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f0629260640 10 monclient(hunting): picked mon.noname-h con 0x7f06240a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:45:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f0629260640 10 monclient(hunting): picked mon.noname-d con 0x7f06240a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f0629260640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0624136790 con 0x7f06240a5920 2024-08-19T12:45:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f0629260640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f062409ddf0 con 0x7f06240a5d50 2024-08-19T12:45:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f0629260640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f062409e0d0 con 0x7f062406e7b0 2024-08-19T12:45:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f0629260640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f0629260640 10 monclient(hunting): authenticate will time out at 1120.030478s 2024-08-19T12:45:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f062359e640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f06240a5d50 legacy=0x7f0624141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:25.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f0622d9d640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f062406e7b0 0x7f0624136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:41602/0 (socket says 172.21.15.26:41602) 2024-08-19T12:45:25.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.943+0000 7f0622d9d640 1 -- 172.21.15.26:0/1983949862 learned_addr learned my addr 172.21.15.26:0/1983949862 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.945+0000 7f0622d9d640 1 -- 172.21.15.26:0/1983949862 >> v1:172.21.15.26:6793/0 conn(0x7f062406e7b0 legacy=0x7f0624136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.945+0000 7f0621d9b640 1 -- 172.21.15.26:0/1983949862 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2294912811 0 0) 0x7f062409e0d0 con 0x7f062406e7b0 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.945+0000 7f0621d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.945+0000 7f0621d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.945+0000 7f0621d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.945+0000 7f0621d9b640 10 monclient(hunting): my global_id is 26270 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.945+0000 7f0621d9b640 1 -- 172.21.15.26:0/1983949862 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f060c003520 con 0x7f062406e7b0 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f062259c640 1 -- 172.21.15.26:0/1983949862 >> v1:172.21.15.104:6791/0 conn(0x7f06240a5920 legacy=0x7f062413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 1 -- 172.21.15.26:0/1983949862 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 261071045 0 0) 0x7f060c003520 con 0x7f062406e7b0 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 1 -- 172.21.15.26:0/1983949862 >> v1:172.21.15.26:6792/0 conn(0x7f06240a5d50 legacy=0x7f0624141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 1 -- 172.21.15.26:0/1983949862 >> v1:172.21.15.104:6791/0 conn(0x7f06240a5920 legacy=0x7f062413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 1 monclient: found mon.noname-e 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 1 -- 172.21.15.26:0/1983949862 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062409e3b0 con 0x7f062406e7b0 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 10 monclient: _finish_auth 0 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0629260640 5 monclient: authenticate success, global_id 26270 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0629260640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 1 -- 172.21.15.26:0/1983949862 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0614001ae0 con 0x7f062406e7b0 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 10 monclient: dump: 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.952 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.953 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0629260640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:25.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 1 -- 172.21.15.26:0/1983949862 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f06140048a0 con 0x7f062406e7b0 2024-08-19T12:45:25.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:25.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 1 -- 172.21.15.26:0/1983949862 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0614004e10 con 0x7f062406e7b0 2024-08-19T12:45:25.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0629260640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:25.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0629260640 4 set_mon_vals no callback set 2024-08-19T12:45:25.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:25.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0621d9b640 10 monclient: dump: 2024-08-19T12:45:25.954 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.955 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0629260640 10 monclient: shutdown 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0629260640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0629260640 1 -- 172.21.15.26:0/1983949862 >> v1:172.21.15.26:6793/0 conn(0x7f062406e7b0 legacy=0x7f0624136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0629260640 1 -- 172.21.15.26:0/1983949862 shutdown_connections 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0629260640 1 -- 172.21.15.26:0/1983949862 >> 172.21.15.26:0/1983949862 conn(0x7f06240a8740 msgr2=0x7f06240a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0629260640 1 -- 172.21.15.26:0/1983949862 shutdown_connections 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.946+0000 7f0629260640 1 -- 172.21.15.26:0/1983949862 wait complete. 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.947+0000 7f0629260640 10 monclient: build_initial_monmap 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.947+0000 7f0629260640 10 monclient: monmap: 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:25.956 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.947+0000 7f0629260640 1 librados: starting msgr at 2024-08-19T12:45:25.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.947+0000 7f0629260640 1 librados: starting objecter 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.947+0000 7f0629260640 1 Processor -- start 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 1 -- start start 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 1 librados: setting wanted keys 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 1 librados: calling monclient init 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 10 monclient: init 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 10 monclient: _add_conns ranks=[3,1,7,4,5,6,8,0,2] 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 10 monclient(hunting): picked mon.noname-g con 0x7f062406e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 10 monclient(hunting): picked mon.noname-f con 0x7f06240a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 10 monclient(hunting): picked mon.noname-i con 0x7f06240a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f062409e9e0 con 0x7f06240a5920 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f062409f2c0 con 0x7f062406e7b0 2024-08-19T12:45:25.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06240a0d40 con 0x7f06240a5d50 2024-08-19T12:45:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f0629260640 10 monclient(hunting): authenticate will time out at 1120.035474s 2024-08-19T12:45:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f062359e640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f06240a5d50 0x7f0624132ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:38598/0 (socket says 172.21.15.26:38598) 2024-08-19T12:45:25.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.948+0000 7f062359e640 1 -- 172.21.15.26:0/2971351076 learned_addr learned my addr 172.21.15.26:0/2971351076 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f062259c640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6789/0 conn(0x7f06240a5920 legacy=0x7f06241325d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 client.?.objecter ms_handle_connect 0x7f06240a5920 2024-08-19T12:45:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f062359e640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6792/0 conn(0x7f06240a5d50 legacy=0x7f0624132ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 client.?.objecter ms_handle_connect 0x7f06240a5d50 2024-08-19T12:45:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 client.?.objecter ms_handle_connect 0x7f062406e7b0 2024-08-19T12:45:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f0622d9d640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6790/0 conn(0x7f062406e7b0 legacy=0x7f0624131ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3895542097 0 0) 0x7f06240a0d40 con 0x7f06240a5d50 2024-08-19T12:45:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): my global_id is 135331 2024-08-19T12:45:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0610004eb0 con 0x7f06240a5d50 2024-08-19T12:45:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3236646744 0 0) 0x7f062409e9e0 con 0x7f06240a5920 2024-08-19T12:45:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): my global_id is 27172 2024-08-19T12:45:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06240a0d40 con 0x7f06240a5920 2024-08-19T12:45:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3639985469 0 0) 0x7f062409f2c0 con 0x7f062406e7b0 2024-08-19T12:45:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): my global_id is 154254 2024-08-19T12:45:25.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f062409e9e0 con 0x7f062406e7b0 2024-08-19T12:45:25.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1879181442 0 0) 0x7f062409e9e0 con 0x7f062406e7b0 2024-08-19T12:45:25.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f062409f2c0 con 0x7f062406e7b0 2024-08-19T12:45:25.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4070895660 0 0) 0x7f06240a0d40 con 0x7f06240a5920 2024-08-19T12:45:25.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f062409e9e0 con 0x7f06240a5920 2024-08-19T12:45:25.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3889647605 0 0) 0x7f0610004eb0 con 0x7f06240a5d50 2024-08-19T12:45:25.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06240a0d40 con 0x7f06240a5d50 2024-08-19T12:45:25.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0614004d90 con 0x7f062406e7b0 2024-08-19T12:45:25.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:25.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): dump: 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.967 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0618003610 con 0x7f06240a5920 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): dump: 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.968 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f061c004830 con 0x7f06240a5d50 2024-08-19T12:45:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:45:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient(hunting): dump: 2024-08-19T12:45:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.970 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1121188732 0 0) 0x7f062409f2c0 con 0x7f062406e7b0 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6792/0 conn(0x7f06240a5d50 legacy=0x7f0624132ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6789/0 conn(0x7f06240a5920 legacy=0x7f06241325d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 monclient: found mon.g 2024-08-19T12:45:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:45:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06240a04a0 con 0x7f062406e7b0 2024-08-19T12:45:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient: _finish_auth 0 2024-08-19T12:45:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:45:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f0629260640 5 monclient: authenticate success, global_id 154254 2024-08-19T12:45:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f0629260640 10 monclient: _renew_subs 2024-08-19T12:45:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f0629260640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:45:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f0629260640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f06241d72a0 con 0x7f062406e7b0 2024-08-19T12:45:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f0629260640 10 client.154254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f0629260640 10 monclient: _renew_subs 2024-08-19T12:45:25.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f0629260640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f0629260640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f06241d7650 con 0x7f062406e7b0 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f0629260640 1 librados: init done 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.951+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0614001320 con 0x7f062406e7b0 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.952+0000 7f060b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.952+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0614001580 con 0x7f062406e7b0 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.952+0000 7f060b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.952+0000 7f060b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.952+0000 7f062159a640 4 set_mon_vals no callback set 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.952+0000 7f060b7fe640 10 monclient: dump: 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.974 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.952+0000 7f0629260640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.952+0000 7f0629260640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f06040065c0 con 0x7f0604002860 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.952+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f0614011a00 con 0x7f062406e7b0 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.953+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 8 ==== osd_map(284..284 src has 1..284) ==== 5699+0+0 (unknown 2697114685 0 0) 0x7f061404e1c0 con 0x7f062406e7b0 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.953+0000 7f060b7fe640 10 client.154254.objecter ms_dispatch 0x7f0624000b90 osd_map(284..284 src has 1..284) 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.953+0000 7f060b7fe640 3 client.154254.objecter handle_osd_map got epochs [284,284] > 0 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.953+0000 7f060b7fe640 3 client.154254.objecter handle_osd_map decoding full epoch 284 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.953+0000 7f060b7fe640 20 client.154254.objecter dump_active .. 0 homeless 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.953+0000 7f062359e640 10 monclient: get_auth_request con 0x7f0610041b70 auth_method 0 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f062259c640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6790/0 conn(0x7f0604002860 legacy=0x7f0604004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 10 client.154254.objecter ms_handle_connect 0x7f0604002860 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 10 client.154254.objecter resend_mon_ops 2024-08-19T12:45:25.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3629241897 0 0) 0x7f06040065c0 con 0x7f0604002860 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 10 monclient(hunting): my global_id is 154263 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f062409f2c0 con 0x7f0604002860 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1626730891 0 0) 0x7f062409f2c0 con 0x7f0604002860 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06040065c0 con 0x7f0604002860 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0618003e10 con 0x7f0604002860 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 10 monclient: dump: 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.976 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.977 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 10 client.154254.objecter ms_handle_connect 0x7f0610041b70 2024-08-19T12:45:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4148529410 0 0) 0x7f06040065c0 con 0x7f0604002860 2024-08-19T12:45:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.955+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f060400a160 con 0x7f0604002860 2024-08-19T12:45:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.956+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0618003ea0 con 0x7f0604002860 2024-08-19T12:45:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.956+0000 7f060b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.956+0000 7f060b7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.956+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6790/0 conn(0x7f0604002860 legacy=0x7f0604004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.956+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6790/0 conn(0x7f0604002860 legacy=0x7f0604004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.962+0000 7f0629260640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:25.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.962+0000 7f0629260640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0604009e70 con 0x7f0604009130 2024-08-19T12:45:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 10 client.154254.objecter ms_handle_connect 0x7f0604009130 2024-08-19T12:45:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f062259c640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6790/0 conn(0x7f0604009130 legacy=0x7f06040095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 10 client.154254.objecter resend_mon_ops 2024-08-19T12:45:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3836059043 0 0) 0x7f0604009e70 con 0x7f0604009130 2024-08-19T12:45:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:45:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 10 monclient(hunting): my global_id is 154272 2024-08-19T12:45:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06040065c0 con 0x7f0604009130 2024-08-19T12:45:25.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 719269173 0 0) 0x7f06040065c0 con 0x7f0604009130 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0604009e70 con 0x7f0604009130 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0618004390 con 0x7f0604009130 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 10 monclient: dump: 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:25.981 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1780917704 0 0) 0x7f0604009e70 con 0x7f0604009130 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.965+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f060400c3f0 con 0x7f0604009130 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7532 (unknown 0 0 1340486595) 0x7f0618007ba0 con 0x7f0604009130 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f060b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:25.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f060b7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6790/0 conn(0x7f0604009130 legacy=0x7f06040095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f060b7fe640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6790/0 conn(0x7f0604009130 legacy=0x7f06040095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:25.983 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:25.983 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:25.983 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:25.983 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:25.983 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 46, 2024-08-19T12:45:25.983 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 37694, 2024-08-19T12:45:25.983 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:25.983 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:25.984 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:25.984 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:25.984 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:25.984 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:25.984 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:25.984 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:25.984 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:25.984 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.986 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:45:25.986 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:25.986 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:25.986 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.987 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:25.989 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:25.990 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.991 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:25.993 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.996 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:25.996 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.996 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.996 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:25.996 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:25.996 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.996 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.996 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:25.997 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:25.998 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:25.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.003 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:26.004 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:26.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 10 librados: watch_flush enter 2024-08-19T12:45:26.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 10 librados: watch_flush exit 2024-08-19T12:45:26.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 20 client.154254.objecter shutdown clearing up homeless session... 2024-08-19T12:45:26.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 10 client.154254.objecter successfully canceled tick 2024-08-19T12:45:26.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6800/899887506 conn(0x7f0610041b70 legacy=0x7f0610044030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 10 monclient: shutdown 2024-08-19T12:45:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 1 -- 172.21.15.26:0/2971351076 >> v1:172.21.15.104:6790/0 conn(0x7f062406e7b0 legacy=0x7f0624131ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f062359e640 1 -- 172.21.15.26:0/2971351076 reap_dead start 2024-08-19T12:45:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f062359e640 1 -- 172.21.15.26:0/2971351076 reap_dead start 2024-08-19T12:45:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 1 -- 172.21.15.26:0/2971351076 shutdown_connections 2024-08-19T12:45:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 1 -- 172.21.15.26:0/2971351076 >> 172.21.15.26:0/2971351076 conn(0x7f06240a8740 msgr2=0x7f0624133970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 1 -- 172.21.15.26:0/2971351076 shutdown_connections 2024-08-19T12:45:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 1 -- 172.21.15.26:0/2971351076 wait complete. 2024-08-19T12:45:26.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:25.966+0000 7f0629260640 1 librados: shutdown 2024-08-19T12:45:26.007 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:45:26.008 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-08-19T12:45:26.008 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:45:26.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.108+0000 7fa9199de640 10 monclient: get_monmap_and_config 2024-08-19T12:45:26.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.108+0000 7fa9199de640 10 monclient: build_initial_monmap 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.108+0000 7fa9199de640 10 monclient: monmap: 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:26.109740+0000 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:26.109740+0000 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:26.110 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:26.111 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:26.111 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:26.111 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.109+0000 7fa9199de640 1 Processor -- start 2024-08-19T12:45:26.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.109+0000 7fa9199de640 1 -- start start 2024-08-19T12:45:26.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.109+0000 7fa9199de640 10 monclient: init 2024-08-19T12:45:26.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.109+0000 7fa9199de640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:26.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.109+0000 7fa9199de640 10 monclient: _add_conns ranks=[6,0,4,7,8,5,1,2,3] 2024-08-19T12:45:26.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.110+0000 7fa9199de640 10 monclient(hunting): picked mon.noname-d con 0x7fa91406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:45:26.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.110+0000 7fa9199de640 10 monclient(hunting): picked mon.noname-a con 0x7fa9140a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.110+0000 7fa9199de640 10 monclient(hunting): picked mon.noname-c con 0x7fa9140a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.110+0000 7fa9199de640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa914136790 con 0x7fa9140a5920 2024-08-19T12:45:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.110+0000 7fa9199de640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa91409ddf0 con 0x7fa9140a5d50 2024-08-19T12:45:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.110+0000 7fa9199de640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa91409e0d0 con 0x7fa91406e7b0 2024-08-19T12:45:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.110+0000 7fa9199de640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.110+0000 7fa9199de640 10 monclient(hunting): authenticate will time out at 1120.197082s 2024-08-19T12:45:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.110+0000 7fa912ffd640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fa91406e7b0 legacy=0x7fa914136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:45:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.110+0000 7fa9127fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa9140a5920 0x7fa91413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38166/0 (socket says 172.21.15.26:38166) 2024-08-19T12:45:26.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.110+0000 7fa9127fc640 1 -- 172.21.15.26:0/1148340387 learned_addr learned my addr 172.21.15.26:0/1148340387 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa9127fc640 1 -- 172.21.15.26:0/1148340387 >> v1:172.21.15.26:6789/0 conn(0x7fa9140a5920 legacy=0x7fa91413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa9137fe640 1 -- 172.21.15.26:0/1148340387 >> v1:172.21.15.26:6791/0 conn(0x7fa9140a5d50 legacy=0x7fa914141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 1 -- 172.21.15.26:0/1148340387 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3500288451 0 0) 0x7fa914136790 con 0x7fa9140a5920 2024-08-19T12:45:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient(hunting): my global_id is 144675 2024-08-19T12:45:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 1 -- 172.21.15.26:0/1148340387 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa900003520 con 0x7fa9140a5920 2024-08-19T12:45:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 1 -- 172.21.15.26:0/1148340387 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464699812 0 0) 0x7fa90c001820 con 0x7fa9140a5d50 2024-08-19T12:45:26.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient(hunting): my global_id is 26716 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 1 -- 172.21.15.26:0/1148340387 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa914136790 con 0x7fa9140a5d50 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 1 -- 172.21.15.26:0/1148340387 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3981917613 0 0) 0x7fa914136790 con 0x7fa9140a5d50 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 1 -- 172.21.15.26:0/1148340387 >> v1:172.21.15.26:6792/0 conn(0x7fa91406e7b0 legacy=0x7fa914136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 1 -- 172.21.15.26:0/1148340387 >> v1:172.21.15.26:6789/0 conn(0x7fa9140a5920 legacy=0x7fa91413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 1 monclient: found mon.noname-c 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 1 -- 172.21.15.26:0/1148340387 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91409e3b0 con 0x7fa9140a5d50 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient: _finish_auth 0 2024-08-19T12:45:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 1 -- 172.21.15.26:0/1148340387 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa90c002060 con 0x7fa9140a5d50 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa9199de640 5 monclient: authenticate success, global_id 26716 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa9199de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa911ffb640 10 monclient: dump: 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.119 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.112+0000 7fa9199de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa911ffb640 1 -- 172.21.15.26:0/1148340387 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa90c003a40 con 0x7fa9140a5d50 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa911ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa911ffb640 1 -- 172.21.15.26:0/1148340387 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa90c003fb0 con 0x7fa9140a5d50 2024-08-19T12:45:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa9199de640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:26.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa9199de640 4 set_mon_vals no callback set 2024-08-19T12:45:26.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa911ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:26.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa911ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:26.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa911ffb640 10 monclient: dump: 2024-08-19T12:45:26.121 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.121 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.121 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.121 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.121 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa9199de640 10 monclient: shutdown 2024-08-19T12:45:26.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa9199de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa9199de640 1 -- 172.21.15.26:0/1148340387 >> v1:172.21.15.26:6791/0 conn(0x7fa9140a5d50 legacy=0x7fa914141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa9199de640 1 -- 172.21.15.26:0/1148340387 shutdown_connections 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa9199de640 1 -- 172.21.15.26:0/1148340387 >> 172.21.15.26:0/1148340387 conn(0x7fa9140a8740 msgr2=0x7fa9140a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa9199de640 1 -- 172.21.15.26:0/1148340387 shutdown_connections 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.113+0000 7fa9199de640 1 -- 172.21.15.26:0/1148340387 wait complete. 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.114+0000 7fa9199de640 10 monclient: build_initial_monmap 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.114+0000 7fa9199de640 10 monclient: monmap: 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.123 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.114+0000 7fa9199de640 1 librados: starting msgr at 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.114+0000 7fa9199de640 1 librados: starting objecter 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.114+0000 7fa9199de640 1 Processor -- start 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.114+0000 7fa9199de640 1 -- start start 2024-08-19T12:45:26.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.114+0000 7fa9199de640 1 librados: setting wanted keys 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.114+0000 7fa9199de640 1 librados: calling monclient init 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.114+0000 7fa9199de640 10 monclient: init 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.115+0000 7fa9199de640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.115+0000 7fa9199de640 10 monclient: _add_conns ranks=[8,1,4,5,7,6,0,2,3] 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.115+0000 7fa9199de640 10 monclient(hunting): picked mon.noname-e con 0x7fa91406e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.115+0000 7fa9199de640 10 monclient(hunting): picked mon.noname-f con 0x7fa9140a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.115+0000 7fa9199de640 10 monclient(hunting): picked mon.noname-c con 0x7fa9140a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.115+0000 7fa9199de640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa914142ec0 con 0x7fa9140a5920 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.115+0000 7fa9199de640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa914138fc0 con 0x7fa9140a5d50 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.115+0000 7fa9199de640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa91413aa80 con 0x7fa91406e7b0 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.115+0000 7fa9199de640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.115+0000 7fa9199de640 10 monclient(hunting): authenticate will time out at 1120.202209s 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.115+0000 7fa912ffd640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fa91406e7b0 0x7fa9141ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:41610/0 (socket says 172.21.15.26:41610) 2024-08-19T12:45:26.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.115+0000 7fa912ffd640 1 -- 172.21.15.26:0/3955424759 learned_addr learned my addr 172.21.15.26:0/3955424759 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:26.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa9137fe640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.26:6791/0 conn(0x7fa9140a5d50 legacy=0x7fa9141d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 client.?.objecter ms_handle_connect 0x7fa9140a5d50 2024-08-19T12:45:26.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:26.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 client.?.objecter ms_handle_connect 0x7fa91406e7b0 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa912ffd640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.26:6793/0 conn(0x7fa91406e7b0 legacy=0x7fa9141ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3572158339 0 0) 0x7fa914138fc0 con 0x7fa9140a5d50 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): my global_id is 26725 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8f4004eb0 con 0x7fa9140a5d50 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 516744545 0 0) 0x7fa91413aa80 con 0x7fa91406e7b0 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): my global_id is 26279 2024-08-19T12:45:26.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa914138fc0 con 0x7fa91406e7b0 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 client.?.objecter ms_handle_connect 0x7fa9140a5920 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa9127fc640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.104:6789/0 conn(0x7fa9140a5920 legacy=0x7fa9141d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2676384070 0 0) 0x7fa8f4004eb0 con 0x7fa9140a5d50 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa91413aa80 con 0x7fa9140a5d50 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa90c003f30 con 0x7fa9140a5d50 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): dump: 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.132 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:26.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4154757627 0 0) 0x7fa914138fc0 con 0x7fa91406e7b0 2024-08-19T12:45:26.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8f4004eb0 con 0x7fa91406e7b0 2024-08-19T12:45:26.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa904004620 con 0x7fa91406e7b0 2024-08-19T12:45:26.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:26.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:45:26.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): dump: 2024-08-19T12:45:26.134 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.134 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.134 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.134 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.134 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.134 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.134 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1737642780 0 0) 0x7fa914142ec0 con 0x7fa9140a5920 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 10 monclient(hunting): my global_id is 27181 2024-08-19T12:45:26.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.117+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa914138fc0 con 0x7fa9140a5920 2024-08-19T12:45:26.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3254390433 0 0) 0x7fa91413aa80 con 0x7fa9140a5d50 2024-08-19T12:45:26.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.26:6793/0 conn(0x7fa91406e7b0 legacy=0x7fa9141ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.104:6789/0 conn(0x7fa9140a5920 legacy=0x7fa9141d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:45:26.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 1 monclient: found mon.c 2024-08-19T12:45:26.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:26.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:26.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91413ad60 con 0x7fa9140a5d50 2024-08-19T12:45:26.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 10 monclient: _finish_auth 0 2024-08-19T12:45:26.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:45:26.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:26.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa9199de640 5 monclient: authenticate success, global_id 26725 2024-08-19T12:45:26.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa9199de640 10 monclient: _renew_subs 2024-08-19T12:45:26.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa9199de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:26.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa9199de640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa91413a650 con 0x7fa9140a5d50 2024-08-19T12:45:26.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa9199de640 10 client.26725.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:26.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa9199de640 10 monclient: _renew_subs 2024-08-19T12:45:26.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa9199de640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:45:26.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa9199de640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9141386b0 con 0x7fa9140a5d50 2024-08-19T12:45:26.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa9199de640 1 librados: init done 2024-08-19T12:45:26.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa90c0039c0 con 0x7fa9140a5d50 2024-08-19T12:45:26.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:26.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa90c0046c0 con 0x7fa9140a5d50 2024-08-19T12:45:26.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:26.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa9117fa640 4 set_mon_vals no callback set 2024-08-19T12:45:26.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:26.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 10 monclient: dump: 2024-08-19T12:45:26.138 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.138 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.138 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.138 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.138 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.138 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa9199de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa9199de640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa8f80065c0 con 0x7fa8f8002860 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.118+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa90c011e00 con 0x7fa9140a5d50 2024-08-19T12:45:26.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.119+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(284..284 src has 1..284) ==== 5699+0+0 (unknown 2697114685 0 0) 0x7fa90c04e430 con 0x7fa9140a5d50 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.119+0000 7fa90bfff640 10 client.26725.objecter ms_dispatch 0x7fa914000b90 osd_map(284..284 src has 1..284) 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.119+0000 7fa90bfff640 3 client.26725.objecter handle_osd_map got epochs [284,284] > 0 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.119+0000 7fa90bfff640 3 client.26725.objecter handle_osd_map decoding full epoch 284 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.119+0000 7fa90bfff640 20 client.26725.objecter dump_active .. 0 homeless 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.119+0000 7fa9127fc640 10 monclient: get_auth_request con 0x7fa8f4041670 auth_method 0 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa912ffd640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.104:6791/0 conn(0x7fa8f8002860 legacy=0x7fa8f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 10 client.26725.objecter ms_handle_connect 0x7fa8f8002860 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 10 client.26725.objecter resend_mon_ops 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 16142111 0 0) 0x7fa8f80065c0 con 0x7fa8f8002860 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 10 monclient(hunting): my global_id is 26384 2024-08-19T12:45:26.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa91413aa80 con 0x7fa8f8002860 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 10 client.26725.objecter ms_handle_connect 0x7fa8f4041670 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 173352410 0 0) 0x7fa91413aa80 con 0x7fa8f8002860 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8f80065c0 con 0x7fa8f8002860 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa904005160 con 0x7fa8f8002860 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 10 monclient: dump: 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.141 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1991949207 0 0) 0x7fa8f80065c0 con 0x7fa8f8002860 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.121+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa8f800a160 con 0x7fa8f8002860 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.122+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa9040074a0 con 0x7fa8f8002860 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.122+0000 7fa90bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:26.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.122+0000 7fa90bfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:26.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.122+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.104:6791/0 conn(0x7fa8f8002860 legacy=0x7fa8f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.122+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.104:6791/0 conn(0x7fa8f8002860 legacy=0x7fa8f8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:26.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.128+0000 7fa9199de640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:26.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.128+0000 7fa9199de640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa8f8009e70 con 0x7fa8f8009130 2024-08-19T12:45:26.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa912ffd640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.104:6791/0 conn(0x7fa8f8009130 legacy=0x7fa8f80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 10 client.26725.objecter ms_handle_connect 0x7fa8f8009130 2024-08-19T12:45:26.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 10 client.26725.objecter resend_mon_ops 2024-08-19T12:45:26.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1760240153 0 0) 0x7fa8f8009e70 con 0x7fa8f8009130 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 10 monclient(hunting): my global_id is 26393 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8f80065c0 con 0x7fa8f8009130 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4048760510 0 0) 0x7fa8f80065c0 con 0x7fa8f8009130 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8f8009e70 con 0x7fa8f8009130 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa904006bb0 con 0x7fa8f8009130 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 10 monclient: dump: 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.145 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 776248129 0 0) 0x7fa8f8009e70 con 0x7fa8f8009130 2024-08-19T12:45:26.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.131+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa8f800c410 con 0x7fa8f8009130 2024-08-19T12:45:26.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7700 (unknown 0 0 2351210582) 0x7fa904006f80 con 0x7fa8f8009130 2024-08-19T12:45:26.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa90bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:26.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa90bfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:26.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.104:6791/0 conn(0x7fa8f8009130 legacy=0x7fa8f80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa90bfff640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.104:6791/0 conn(0x7fa8f8009130 legacy=0x7fa8f80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:26.147 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:26.147 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:26.147 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:26.147 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 46, 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 439154, 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:26.148 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:26.149 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:26.150 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:26.151 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:26.152 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.153 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:26.155 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:26.156 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.157 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:26.159 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.160 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:26.163 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.164 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:26.165 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.166 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:26.167 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:26.167 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-08-19T12:45:26.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.167 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:26.167 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 10 librados: watch_flush enter 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 10 librados: watch_flush exit 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 20 client.26725.objecter shutdown clearing up homeless session... 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 10 client.26725.objecter successfully canceled tick 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.104:6800/899887506 conn(0x7fa8f4041670 legacy=0x7fa8f4043b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 10 monclient: shutdown 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 1 -- 172.21.15.26:0/3955424759 >> v1:172.21.15.26:6791/0 conn(0x7fa9140a5d50 legacy=0x7fa9141d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9137fe640 1 -- 172.21.15.26:0/3955424759 reap_dead start 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9137fe640 1 -- 172.21.15.26:0/3955424759 reap_dead start 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 1 -- 172.21.15.26:0/3955424759 shutdown_connections 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 1 -- 172.21.15.26:0/3955424759 >> 172.21.15.26:0/3955424759 conn(0x7fa9140a8740 msgr2=0x7fa9140ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:26.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 1 -- 172.21.15.26:0/3955424759 shutdown_connections 2024-08-19T12:45:26.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 1 -- 172.21.15.26:0/3955424759 wait complete. 2024-08-19T12:45:26.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.132+0000 7fa9199de640 1 librados: shutdown 2024-08-19T12:45:26.170 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:45:26.170 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-08-19T12:45:26.170 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:45:26.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.270+0000 7f1e98f2f640 10 monclient: get_monmap_and_config 2024-08-19T12:45:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.270+0000 7f1e98f2f640 10 monclient: build_initial_monmap 2024-08-19T12:45:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.270+0000 7f1e98f2f640 10 monclient: monmap: 2024-08-19T12:45:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:45:26.270878+0000 2024-08-19T12:45:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:45:26.270878+0000 2024-08-19T12:45:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:26.271 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:26.272 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:26.272 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:26.272 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:26.272 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:26.272 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:26.272 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.270+0000 7f1e98f2f640 1 Processor -- start 2024-08-19T12:45:26.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.270+0000 7f1e98f2f640 1 -- start start 2024-08-19T12:45:26.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.270+0000 7f1e98f2f640 10 monclient: init 2024-08-19T12:45:26.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e98f2f640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:26.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e98f2f640 10 monclient: _add_conns ranks=[4,7,2,1,5,8,6,3,0] 2024-08-19T12:45:26.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e98f2f640 10 monclient(hunting): picked mon.noname-c con 0x7f1e94073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:45:26.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e98f2f640 10 monclient(hunting): picked mon.noname-i con 0x7f1e940a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:26.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e98f2f640 10 monclient(hunting): picked mon.noname-b con 0x7f1e9413f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:45:26.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e98f2f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e9409dae0 con 0x7f1e9413f2e0 2024-08-19T12:45:26.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e98f2f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e94136910 con 0x7f1e94073470 2024-08-19T12:45:26.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e98f2f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e9409dd50 con 0x7f1e940a57a0 2024-08-19T12:45:26.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e98f2f640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:26.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e98f2f640 10 monclient(hunting): authenticate will time out at 1120.358365s 2024-08-19T12:45:26.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e92d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f1e9413f2e0 0x7f1e941416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38914/0 (socket says 172.21.15.26:38914) 2024-08-19T12:45:26.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e9259c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f1e94073470 0x7f1e94136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:45944/0 (socket says 172.21.15.26:45944) 2024-08-19T12:45:26.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.271+0000 7f1e92d9d640 1 -- 172.21.15.26:0/3359539574 learned_addr learned my addr 172.21.15.26:0/3359539574 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:26.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9259c640 1 -- 172.21.15.26:0/3359539574 >> v1:172.21.15.26:6791/0 conn(0x7f1e94073470 legacy=0x7f1e94136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e92d9d640 1 -- 172.21.15.26:0/3359539574 >> v1:172.21.15.26:6790/0 conn(0x7f1e9413f2e0 legacy=0x7f1e941416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e91d9b640 1 -- 172.21.15.26:0/3359539574 >> v1:172.21.15.104:6792/0 conn(0x7f1e940a57a0 legacy=0x7f1e940a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3635124440 0 0) 0x7f1e94136910 con 0x7f1e94073470 2024-08-19T12:45:26.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9159a640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9159a640 10 monclient(hunting): my global_id is 26734 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e80003520 con 0x7f1e94073470 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2835479271 0 0) 0x7f1e9409dae0 con 0x7f1e9413f2e0 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9159a640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9159a640 10 monclient(hunting): my global_id is 26390 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.273+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e94136910 con 0x7f1e9413f2e0 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1000001402 0 0) 0x7f1e9409dd50 con 0x7f1e940a57a0 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient(hunting): my global_id is 135340 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e9409dae0 con 0x7f1e940a57a0 2024-08-19T12:45:26.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1049052274 0 0) 0x7f1e80003520 con 0x7f1e94073470 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 >> v1:172.21.15.104:6792/0 conn(0x7f1e940a57a0 legacy=0x7f1e940a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 >> v1:172.21.15.26:6790/0 conn(0x7f1e9413f2e0 legacy=0x7f1e941416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient: _finish_hunting 0 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 1 monclient: found mon.noname-c 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9409e030 con 0x7f1e94073470 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient: _finish_auth 0 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient: _check_auth_tickets 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1e8c001e60 con 0x7f1e94073470 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:45:26.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient: dump: 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.281 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e98f2f640 5 monclient: authenticate success, global_id 26734 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e98f2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1e8c004b60 con 0x7f1e94073470 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 1 -- 172.21.15.26:0/3359539574 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1e8c0050d0 con 0x7f1e94073470 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e98f2f640 10 monclient: get_monmap_and_config success 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e98f2f640 4 set_mon_vals no callback set 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e9159a640 10 monclient: dump: 2024-08-19T12:45:26.282 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.283 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e98f2f640 10 monclient: shutdown 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e98f2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e98f2f640 1 -- 172.21.15.26:0/3359539574 >> v1:172.21.15.26:6791/0 conn(0x7f1e94073470 legacy=0x7f1e94136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e98f2f640 1 -- 172.21.15.26:0/3359539574 shutdown_connections 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e98f2f640 1 -- 172.21.15.26:0/3359539574 >> 172.21.15.26:0/3359539574 conn(0x7f1e940abff0 msgr2=0x7f1e940ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e98f2f640 1 -- 172.21.15.26:0/3359539574 shutdown_connections 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.274+0000 7f1e98f2f640 1 -- 172.21.15.26:0/3359539574 wait complete. 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.275+0000 7f1e98f2f640 10 monclient: build_initial_monmap 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.275+0000 7f1e98f2f640 10 monclient: monmap: 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:45:26.284 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.275+0000 7f1e98f2f640 1 librados: starting msgr at 2024-08-19T12:45:26.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.275+0000 7f1e98f2f640 1 librados: starting objecter 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.275+0000 7f1e98f2f640 1 Processor -- start 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.275+0000 7f1e98f2f640 1 -- start start 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.275+0000 7f1e98f2f640 1 librados: setting wanted keys 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.275+0000 7f1e98f2f640 1 librados: calling monclient init 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.275+0000 7f1e98f2f640 10 monclient: init 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.276+0000 7f1e98f2f640 10 monclient: _reopen_session rank -1 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.276+0000 7f1e98f2f640 10 monclient: _add_conns ranks=[7,0,1,3,2,8,4,5,6] 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.276+0000 7f1e98f2f640 10 monclient(hunting): picked mon.noname-i con 0x7f1e94073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.276+0000 7f1e98f2f640 10 monclient(hunting): picked mon.noname-a con 0x7f1e940a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.276+0000 7f1e98f2f640 10 monclient(hunting): picked mon.noname-f con 0x7f1e9413f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.276+0000 7f1e98f2f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e9413dd00 con 0x7f1e940a57a0 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.276+0000 7f1e98f2f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e94134f90 con 0x7f1e9413f2e0 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.276+0000 7f1e98f2f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e9413edb0 con 0x7f1e94073470 2024-08-19T12:45:26.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.276+0000 7f1e98f2f640 10 monclient(hunting): _renew_subs 2024-08-19T12:45:26.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.276+0000 7f1e98f2f640 10 monclient(hunting): authenticate will time out at 1120.363428s 2024-08-19T12:45:26.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.276+0000 7f1e91d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f1e940a57a0 0x7f1e940a25a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:38168/0 (socket says 172.21.15.26:38168) 2024-08-19T12:45:26.287 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.276+0000 7f1e91d9b640 1 -- 172.21.15.26:0/4201328935 learned_addr learned my addr 172.21.15.26:0/4201328935 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:45:26.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e91d9b640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.26:6789/0 conn(0x7f1e940a57a0 legacy=0x7f1e940a25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e8affd640 10 client.?.objecter ms_handle_connect 0x7f1e940a57a0 2024-08-19T12:45:26.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e8affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:26.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e8affd640 10 client.?.objecter ms_handle_connect 0x7f1e94073470 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e8affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e9259c640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.104:6792/0 conn(0x7f1e94073470 legacy=0x7f1e940a1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e92d9d640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.104:6789/0 conn(0x7f1e9413f2e0 legacy=0x7f1e940a2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 220186005 0 0) 0x7f1e9413dd00 con 0x7f1e940a57a0 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e8affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e8affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e8affd640 10 monclient(hunting): my global_id is 144684 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e74004eb0 con 0x7f1e940a57a0 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e8affd640 10 client.?.objecter ms_handle_connect 0x7f1e9413f2e0 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.278+0000 7f1e8affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4083602065 0 0) 0x7f1e74004eb0 con 0x7f1e940a57a0 2024-08-19T12:45:26.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e9413dd00 con 0x7f1e940a57a0 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1e7c003950 con 0x7f1e940a57a0 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient(hunting): dump: 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.293 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129408661 0 0) 0x7f1e94134f90 con 0x7f1e9413f2e0 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient(hunting): my global_id is 27190 2024-08-19T12:45:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e74004eb0 con 0x7f1e9413f2e0 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2534167689 0 0) 0x7f1e9413edb0 con 0x7f1e94073470 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient(hunting): my global_id is 135349 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e94134f90 con 0x7f1e94073470 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1032810370 0 0) 0x7f1e9413dd00 con 0x7f1e940a57a0 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.104:6792/0 conn(0x7f1e94073470 legacy=0x7f1e940a1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.104:6789/0 conn(0x7f1e9413f2e0 legacy=0x7f1e940a2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient: _finish_hunting 0 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 monclient: found mon.a 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:26.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e94137070 con 0x7f1e940a57a0 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient: _finish_auth 0 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient: _check_auth_tickets 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e98f2f640 5 monclient: authenticate success, global_id 144684 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e98f2f640 10 monclient: _renew_subs 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e98f2f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e98f2f640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1e940a4740 con 0x7f1e940a57a0 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e98f2f640 10 client.144684.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e98f2f640 10 monclient: _renew_subs 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e98f2f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e98f2f640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1e941d7340 con 0x7f1e940a57a0 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e98f2f640 1 librados: init done 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1e7c001d20 con 0x7f1e940a57a0 2024-08-19T12:45:26.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1e7c0041a0 con 0x7f1e940a57a0 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e90d99640 4 set_mon_vals no callback set 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.279+0000 7f1e8affd640 10 monclient: dump: 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.297 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.280+0000 7f1e98f2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.280+0000 7f1e98f2f640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1e780065c0 con 0x7f1e78002860 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.280+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f1e7c011a80 con 0x7f1e940a57a0 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.280+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(284..284 src has 1..284) ==== 5699+0+0 (unknown 2697114685 0 0) 0x7f1e7c04e590 con 0x7f1e940a57a0 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.280+0000 7f1e8affd640 10 client.144684.objecter ms_dispatch 0x7f1e94000b90 osd_map(284..284 src has 1..284) 2024-08-19T12:45:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.280+0000 7f1e8affd640 3 client.144684.objecter handle_osd_map got epochs [284,284] > 0 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.280+0000 7f1e8affd640 3 client.144684.objecter handle_osd_map decoding full epoch 284 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.280+0000 7f1e8affd640 20 client.144684.objecter dump_active .. 0 homeless 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.280+0000 7f1e92d9d640 10 monclient: get_auth_request con 0x7f1e74041120 auth_method 0 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e9259c640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.104:6792/0 conn(0x7f1e78002860 legacy=0x7f1e78004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 10 client.144684.objecter ms_handle_connect 0x7f1e78002860 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 10 client.144684.objecter resend_mon_ops 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2610440571 0 0) 0x7f1e780065c0 con 0x7f1e78002860 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 10 monclient(hunting): my global_id is 135358 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e9413dd00 con 0x7f1e78002860 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3889475452 0 0) 0x7f1e9413dd00 con 0x7f1e78002860 2024-08-19T12:45:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e780065c0 con 0x7f1e78002860 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1e8c004ea0 con 0x7f1e78002860 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.282+0000 7f1e8affd640 10 monclient: dump: 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.300 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.283+0000 7f1e8affd640 10 client.144684.objecter ms_handle_connect 0x7f1e74041120 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.283+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1597267490 0 0) 0x7f1e780065c0 con 0x7f1e78002860 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.283+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e7800a160 con 0x7f1e78002860 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.283+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1e8c001da0 con 0x7f1e78002860 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.283+0000 7f1e8affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.283+0000 7f1e8affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.283+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.104:6792/0 conn(0x7f1e78002860 legacy=0x7f1e78004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.283+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.104:6792/0 conn(0x7f1e78002860 legacy=0x7f1e78004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:26.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.289+0000 7f1e98f2f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:45:26.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.289+0000 7f1e98f2f640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1e78009e70 con 0x7f1e78009130 2024-08-19T12:45:26.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e9259c640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.104:6792/0 conn(0x7f1e78009130 legacy=0x7f1e780095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:45:26.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 10 client.144684.objecter ms_handle_connect 0x7f1e78009130 2024-08-19T12:45:26.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 10 client.144684.objecter resend_mon_ops 2024-08-19T12:45:26.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2497543544 0 0) 0x7f1e78009e70 con 0x7f1e78009130 2024-08-19T12:45:26.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 10 monclient(hunting): _negotiate 2024-08-19T12:45:26.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 10 monclient(hunting): my global_id is 135367 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e780065c0 con 0x7f1e78009130 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 161314953 0 0) 0x7f1e780065c0 con 0x7f1e78009130 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1e78009e70 con 0x7f1e78009130 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f1e8c005190 con 0x7f1e78009130 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 10 monclient: dump: 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.304 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4292953746 0 0) 0x7f1e78009e70 con 0x7f1e78009130 2024-08-19T12:45:26.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.292+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1e7800c410 con 0x7f1e78009130 2024-08-19T12:45:26.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 1990625819) 0x7f1e8c0057b0 con 0x7f1e78009130 2024-08-19T12:45:26.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e8affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:45:26.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e8affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:45:26.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.104:6792/0 conn(0x7f1e78009130 legacy=0x7f1e780095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e8affd640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.104:6792/0 conn(0x7f1e78009130 legacy=0x7f1e780095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 46, 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 296649, 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:45:26.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 10 librados: watch_flush enter 2024-08-19T12:45:26.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 10 librados: watch_flush exit 2024-08-19T12:45:26.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 20 client.144684.objecter shutdown clearing up homeless session... 2024-08-19T12:45:26.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 10 client.144684.objecter successfully canceled tick 2024-08-19T12:45:26.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.104:6800/899887506 conn(0x7f1e74041120 legacy=0x7f1e740435e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 10 monclient: shutdown 2024-08-19T12:45:26.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:45:26.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 1 -- 172.21.15.26:0/4201328935 >> v1:172.21.15.26:6789/0 conn(0x7f1e940a57a0 legacy=0x7f1e940a25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:45:26.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e92d9d640 1 -- 172.21.15.26:0/4201328935 reap_dead start 2024-08-19T12:45:26.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e92d9d640 1 -- 172.21.15.26:0/4201328935 reap_dead start 2024-08-19T12:45:26.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 1 -- 172.21.15.26:0/4201328935 shutdown_connections 2024-08-19T12:45:26.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 1 -- 172.21.15.26:0/4201328935 >> 172.21.15.26:0/4201328935 conn(0x7f1e940abff0 msgr2=0x7f1e94134810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:45:26.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 1 -- 172.21.15.26:0/4201328935 shutdown_connections 2024-08-19T12:45:26.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 1 -- 172.21.15.26:0/4201328935 wait complete. 2024-08-19T12:45:26.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:45:26.293+0000 7f1e98f2f640 1 librados: shutdown 2024-08-19T12:45:26.328 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:45:26.328 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T12:45:31.218 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ o.RoundTripAppendPP (3005 ms) 2024-08-19T12:45:31.218 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-08-19T12:45:31.218 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3026 ms) 2024-08-19T12:45:31.218 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-08-19T12:45:31.218 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2734 ms) 2024-08-19T12:45:31.218 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-08-19T12:45:31.219 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3025 ms) 2024-08-19T12:45:31.219 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-08-19T12:45:31.219 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: setting pool EIO 2024-08-19T12:45:31.219 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: max_success 98, min_failed 99 2024-08-19T12:45:31.219 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4026 ms) 2024-08-19T12:45:31.219 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-08-19T12:45:31.219 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (2928 ms) 2024-08-19T12:45:31.219 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (139108 ms total) 2024-08-19T12:45:31.219 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: 2024-08-19T12:45:31.219 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-08-19T12:45:31.219 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-08-19T12:45:31.220 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3064 ms) 2024-08-19T12:45:31.220 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-08-19T12:45:31.220 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9353 ms) 2024-08-19T12:45:31.220 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-08-19T12:45:31.220 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (16433 ms) 2024-08-19T12:45:31.220 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-08-19T12:45:31.220 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3008 ms) 2024-08-19T12:45:31.220 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (31859 ms total) 2024-08-19T12:45:31.220 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: 2024-08-19T12:45:31.220 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-08-19T12:45:31.220 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-08-19T12:45:31.220 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (23518 ms) 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (23518 ms total) 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (33595 ms) 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7973 ms) 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (9006 ms) 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7000 ms) 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-08-19T12:45:31.221 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3005 ms) 2024-08-19T12:45:31.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-08-19T12:45:31.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7016 ms) 2024-08-19T12:45:31.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-08-19T12:45:31.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (8004 ms) 2024-08-19T12:45:31.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-08-19T12:45:31.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6982 ms) 2024-08-19T12:45:31.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-08-19T12:45:31.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7753 ms) 2024-08-19T12:45:31.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-08-19T12:45:31.222 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7257 ms) 2024-08-19T12:45:31.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-08-19T12:45:31.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7953 ms) 2024-08-19T12:45:31.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-08-19T12:45:31.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6057 ms) 2024-08-19T12:45:31.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-08-19T12:45:31.223 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (49612 ms) 2024-08-19T12:46:22.464 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-08-19T12:46:22.464 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3042 ms) 2024-08-19T12:46:22.464 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-08-19T12:46:22.464 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (9552 ms) 2024-08-19T12:46:22.465 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-08-19T12:46:22.465 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6401 ms) 2024-08-19T12:46:22.465 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-08-19T12:46:22.465 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (8798 ms) 2024-08-19T12:46:22.465 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-08-19T12:46:22.465 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7467 ms) 2024-08-19T12:46:22.465 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-08-19T12:46:22.465 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (8669 ms) 2024-08-19T12:46:22.465 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-08-19T12:46:22.465 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7311 ms) 2024-08-19T12:46:22.465 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (212454 ms total) 2024-08-19T12:46:22.466 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: 2024-08-19T12:46:22.466 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-08-19T12:46:22.466 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (406939 ms total) 2024-08-19T12:46:22.466 INFO:tasks.workunit.client.0.smithi104.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-08-19T12:46:56.329 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-08-19T12:46:56.330 INFO:tasks.ceph.mon.d:Restarting daemon 2024-08-19T12:46:56.330 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-08-19T12:46:56.334 INFO:tasks.ceph.mon.d:Started 2024-08-19T12:46:56.334 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:46:56.334 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:46:56.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.463+0000 7f0936911640 10 monclient: get_monmap_and_config 2024-08-19T12:46:56.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.463+0000 7f0936911640 10 monclient: build_initial_monmap 2024-08-19T12:46:56.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.463+0000 7f0936911640 10 monclient: monmap: 2024-08-19T12:46:56.464 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:56.464 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:56.464 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:46:56.464269+0000 2024-08-19T12:46:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:46:56.464269+0000 2024-08-19T12:46:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:56.466 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:56.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f0936911640 1 Processor -- start 2024-08-19T12:46:56.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f0936911640 1 -- start start 2024-08-19T12:46:56.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f0936911640 10 monclient: init 2024-08-19T12:46:56.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f0936911640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:56.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f0936911640 10 monclient: _add_conns ranks=[4,8,2,7,0,6,1,5,3] 2024-08-19T12:46:56.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f0936911640 10 monclient(hunting): picked mon.noname-c con 0x7f0930073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:46:56.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f0936911640 10 monclient(hunting): picked mon.noname-e con 0x7f09300a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:46:56.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f0936911640 10 monclient(hunting): picked mon.noname-b con 0x7f093013f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:46:56.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f0936911640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f093009dae0 con 0x7f093013f2e0 2024-08-19T12:46:56.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f0936911640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0930136910 con 0x7f0930073470 2024-08-19T12:46:56.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f0936911640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f093009dd50 con 0x7f09300a57a0 2024-08-19T12:46:56.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f092ffff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f0930073470 0x7f0930136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41762/0 (socket says 172.21.15.26:41762) 2024-08-19T12:46:56.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.464+0000 7f092ffff640 1 -- 172.21.15.26:0/1872015708 learned_addr learned my addr 172.21.15.26:0/1872015708 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:56.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.465+0000 7f0936911640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:56.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.465+0000 7f0936911640 10 monclient(hunting): authenticate will time out at 1210.552047s 2024-08-19T12:46:56.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.467+0000 7f092ffff640 1 -- 172.21.15.26:0/1872015708 >> v1:172.21.15.26:6791/0 conn(0x7f0930073470 legacy=0x7f0930136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:56.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.467+0000 7f0934eae640 1 -- 172.21.15.26:0/1872015708 >> v1:172.21.15.26:6790/0 conn(0x7f093013f2e0 legacy=0x7f09301416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:56.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:56.467+0000 7f092f7fe640 1 -- 172.21.15.26:0/1872015708 >> v1:172.21.15.26:6793/0 conn(0x7f09300a57a0 legacy=0x7f09300a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:57.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3391155010 0 0) 0x7f093009dd50 con 0x7f09300a57a0 2024-08-19T12:46:57.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:57.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:57.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:57.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 10 monclient(hunting): my global_id is 26315 2024-08-19T12:46:57.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0918003520 con 0x7f09300a57a0 2024-08-19T12:46:57.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4101707691 0 0) 0x7f0930136910 con 0x7f0930073470 2024-08-19T12:46:57.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:57.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:57.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:57.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 10 monclient(hunting): my global_id is 26779 2024-08-19T12:46:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f093009dd50 con 0x7f0930073470 2024-08-19T12:46:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3196408617 0 0) 0x7f093009dae0 con 0x7f093013f2e0 2024-08-19T12:46:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 10 monclient(hunting): my global_id is 26435 2024-08-19T12:46:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0930136910 con 0x7f093013f2e0 2024-08-19T12:46:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2903821769 0 0) 0x7f093009dd50 con 0x7f0930073470 2024-08-19T12:46:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 >> v1:172.21.15.26:6793/0 conn(0x7f09300a57a0 legacy=0x7f09300a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092f7fe640 1 --1- 172.21.15.26:0/1872015708 >> v1:172.21.15.26:6793/0 conn(0x7f09300a57a0 0x7f09300a5b80 :-1 s=OPENED pgs=3391 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f0924001b30 mon_map magic: 0 2024-08-19T12:46:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.894+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 >> v1:172.21.15.26:6790/0 conn(0x7f093013f2e0 legacy=0x7f09301416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:57.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 10 monclient: _finish_hunting 0 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 1 monclient: found mon.noname-c 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f093009e030 con 0x7f0930073470 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 10 monclient: _finish_auth 0 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 10 monclient: _check_auth_tickets 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0920001ac0 con 0x7f0930073470 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 10 monclient: dump: 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:57.899 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:57.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f0936911640 5 monclient: authenticate success, global_id 26779 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0924001b30 con 0x7f09300a57a0 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f0936911640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f09200048a0 con 0x7f0930073470 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 1 -- 172.21.15.26:0/1872015708 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0920004d90 con 0x7f0930073470 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f092effd640 10 monclient: dump: 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:57.901 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f0936911640 10 monclient: get_monmap_and_config success 2024-08-19T12:46:57.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.895+0000 7f0936911640 4 set_mon_vals no callback set 2024-08-19T12:46:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.896+0000 7f0936911640 10 monclient: shutdown 2024-08-19T12:46:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.896+0000 7f0936911640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.896+0000 7f0936911640 1 -- 172.21.15.26:0/1872015708 >> v1:172.21.15.26:6791/0 conn(0x7f0930073470 legacy=0x7f0930136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.896+0000 7f0936911640 1 -- 172.21.15.26:0/1872015708 shutdown_connections 2024-08-19T12:46:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.896+0000 7f0936911640 1 -- 172.21.15.26:0/1872015708 >> 172.21.15.26:0/1872015708 conn(0x7f09300abff0 msgr2=0x7f09300ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.896+0000 7f0936911640 1 -- 172.21.15.26:0/1872015708 shutdown_connections 2024-08-19T12:46:57.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.896+0000 7f0936911640 1 -- 172.21.15.26:0/1872015708 wait complete. 2024-08-19T12:46:57.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.897+0000 7f0936911640 10 monclient: build_initial_monmap 2024-08-19T12:46:57.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.897+0000 7f0936911640 10 monclient: monmap: 2024-08-19T12:46:57.906 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:57.906 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:46:57.906 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:46:57.906 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:46:57.906 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:57.907 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:57.907 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:57.907 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:57.907 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:57.907 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:57.907 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:57.908 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:57.908 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:57.908 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:57.908 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:57.908 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:57.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.897+0000 7f0936911640 1 librados: starting msgr at 2024-08-19T12:46:57.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.897+0000 7f0936911640 1 librados: starting objecter 2024-08-19T12:46:57.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.897+0000 7f0936911640 1 Processor -- start 2024-08-19T12:46:57.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.897+0000 7f0936911640 1 -- start start 2024-08-19T12:46:57.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.897+0000 7f0936911640 1 librados: setting wanted keys 2024-08-19T12:46:57.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.897+0000 7f0936911640 1 librados: calling monclient init 2024-08-19T12:46:57.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.897+0000 7f0936911640 10 monclient: init 2024-08-19T12:46:57.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.898+0000 7f0936911640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:57.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.898+0000 7f0936911640 10 monclient: _add_conns ranks=[8,2,5,7,1,6,3,0,4] 2024-08-19T12:46:57.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.898+0000 7f0936911640 10 monclient(hunting): picked mon.noname-e con 0x7f0930073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:46:57.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.898+0000 7f0936911640 10 monclient(hunting): picked mon.noname-b con 0x7f09300a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:46:57.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.898+0000 7f0936911640 10 monclient(hunting): picked mon.noname-h con 0x7f093013f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:46:57.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.898+0000 7f0936911640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09301318e0 con 0x7f09300a57a0 2024-08-19T12:46:57.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.898+0000 7f0936911640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0930131bc0 con 0x7f093013f2e0 2024-08-19T12:46:57.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.898+0000 7f0936911640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09301cef20 con 0x7f0930073470 2024-08-19T12:46:57.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.898+0000 7f0936911640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:57.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.898+0000 7f0936911640 10 monclient(hunting): authenticate will time out at 1211.985501s 2024-08-19T12:46:57.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.898+0000 7f092ffff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f0930073470 0x7f09301ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:41682/0 (socket says 172.21.15.26:41682) 2024-08-19T12:46:57.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.898+0000 7f092ffff640 1 -- 172.21.15.26:0/1998635717 learned_addr learned my addr 172.21.15.26:0/1998635717 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.900+0000 7f092f7fe640 1 -- 172.21.15.26:0/1998635717 >> v1:172.21.15.26:6790/0 conn(0x7f09300a57a0 legacy=0x7f09301ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.900+0000 7f092ffff640 1 -- 172.21.15.26:0/1998635717 >> v1:172.21.15.26:6793/0 conn(0x7f0930073470 legacy=0x7f09301ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.900+0000 7f092cff9640 10 client.?.objecter ms_handle_connect 0x7f09300a57a0 2024-08-19T12:46:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.900+0000 7f092cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.900+0000 7f092cff9640 10 client.?.objecter ms_handle_connect 0x7f0930073470 2024-08-19T12:46:57.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.900+0000 7f092cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:57.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.900+0000 7f0934eae640 1 -- 172.21.15.26:0/1998635717 >> v1:172.21.15.104:6791/0 conn(0x7f093013f2e0 legacy=0x7f09301d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:57.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.900+0000 7f092cff9640 10 client.?.objecter ms_handle_connect 0x7f093013f2e0 2024-08-19T12:46:57.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:57.900+0000 7f092cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:58.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.198+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3795358826 0 0) 0x7f0930131bc0 con 0x7f093013f2e0 2024-08-19T12:46:58.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.198+0000 7f092cff9640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.198+0000 7f092cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.198+0000 7f092cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.198+0000 7f092cff9640 10 monclient(hunting): my global_id is 26438 2024-08-19T12:46:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.198+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f091c004eb0 con 0x7f093013f2e0 2024-08-19T12:46:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1192774791 0 0) 0x7f09301318e0 con 0x7f09300a57a0 2024-08-19T12:46:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient(hunting): my global_id is 26444 2024-08-19T12:46:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0930131bc0 con 0x7f09300a57a0 2024-08-19T12:46:58.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2197640286 0 0) 0x7f091c004eb0 con 0x7f093013f2e0 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09301318e0 con 0x7f093013f2e0 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0928002070 con 0x7f093013f2e0 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient(hunting): dump: 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.204 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1616251219 0 0) 0x7f0930131bc0 con 0x7f09300a57a0 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f091c004eb0 con 0x7f09300a57a0 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0924003610 con 0x7f09300a57a0 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:58.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient(hunting): dump: 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.206 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 91557099 0 0) 0x7f09301318e0 con 0x7f093013f2e0 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 >> v1:172.21.15.26:6793/0 conn(0x7f0930073470 legacy=0x7f09301ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 >> v1:172.21.15.26:6790/0 conn(0x7f09300a57a0 legacy=0x7f09301ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient: _finish_hunting 0 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 monclient: found mon.h 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f093012f860 con 0x7f093013f2e0 2024-08-19T12:46:58.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient: _finish_auth 0 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient: _check_auth_tickets 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f0936911640 5 monclient: authenticate success, global_id 26438 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f0936911640 10 monclient: _renew_subs 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f0936911640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f0936911640 1 -- 172.21.15.26:0/1998635717 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0930133320 con 0x7f093013f2e0 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f0936911640 10 client.26438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f0936911640 10 monclient: _renew_subs 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f0936911640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f0936911640 1 -- 172.21.15.26:0/1998635717 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f0930131fe0 con 0x7f093013f2e0 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f0936911640 1 librados: init done 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0928003c40 con 0x7f093013f2e0 2024-08-19T12:46:58.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:58.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0928004170 con 0x7f093013f2e0 2024-08-19T12:46:58.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:58.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:46:58.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092e7fc640 4 set_mon_vals no callback set 2024-08-19T12:46:58.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.199+0000 7f092cff9640 10 monclient: dump: 2024-08-19T12:46:58.209 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.209 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.209 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.209 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.209 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.209 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.209 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.200+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f0928004310 con 0x7f093013f2e0 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.200+0000 7f0936911640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.200+0000 7f0936911640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:58.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.200+0000 7f0936911640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:46:58.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.200+0000 7f0936911640 1 -- 172.21.15.26:0/1998635717 --> v1:172.21.15.104:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0910005080 con 0x7f093013f2e0 2024-08-19T12:46:58.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.200+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(349..349 src has 1..349) ==== 5346+0+0 (unknown 3654949895 0 0) 0x7f092804e380 con 0x7f093013f2e0 2024-08-19T12:46:58.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.200+0000 7f092cff9640 10 client.26438.objecter ms_dispatch 0x7f0930000b90 osd_map(349..349 src has 1..349) 2024-08-19T12:46:58.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.200+0000 7f092cff9640 3 client.26438.objecter handle_osd_map got epochs [349,349] > 0 2024-08-19T12:46:58.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.200+0000 7f092cff9640 3 client.26438.objecter handle_osd_map decoding full epoch 349 2024-08-19T12:46:58.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.201+0000 7f092cff9640 20 client.26438.objecter dump_active .. 0 homeless 2024-08-19T12:46:58.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.201+0000 7f092ffff640 10 monclient: get_auth_request con 0x7f091c0414d0 auth_method 0 2024-08-19T12:46:58.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.203+0000 7f092cff9640 10 client.26438.objecter ms_handle_connect 0x7f091c0414d0 2024-08-19T12:46:58.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.206+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.5 v1:172.21.15.104:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f0928017130 con 0x7f093013f2e0 2024-08-19T12:46:58.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.206+0000 7f092cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:58.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.206+0000 7f092cff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:46:58.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.464+0000 7f0936911640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:46:58.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.464+0000 7f0936911640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:46:58.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.464+0000 7f0936911640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:46:58.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.464+0000 7f0936911640 1 -- 172.21.15.26:0/1998635717 --> v1:172.21.15.104:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0910005370 con 0x7f093013f2e0 2024-08-19T12:46:58.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.464+0000 7f092cff9640 1 -- 172.21.15.26:0/1998635717 <== mon.5 v1:172.21.15.104:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 4238620516) 0x7f0928020070 con 0x7f093013f2e0 2024-08-19T12:46:58.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.464+0000 7f092cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:46:58.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.464+0000 7f092cff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:46:58.466 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":52,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:46:58.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.466+0000 7f0936911640 10 librados: watch_flush enter 2024-08-19T12:46:58.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.466+0000 7f0936911640 10 librados: watch_flush exit 2024-08-19T12:46:58.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.466+0000 7f0936911640 20 client.26438.objecter shutdown clearing up homeless session... 2024-08-19T12:46:58.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.467+0000 7f0936911640 10 client.26438.objecter successfully canceled tick 2024-08-19T12:46:58.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.467+0000 7f0936911640 1 -- 172.21.15.26:0/1998635717 >> v1:172.21.15.104:6800/899887506 conn(0x7f091c0414d0 legacy=0x7f091c043990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.467+0000 7f0936911640 10 monclient: shutdown 2024-08-19T12:46:58.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.467+0000 7f0936911640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:58.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.467+0000 7f0936911640 1 -- 172.21.15.26:0/1998635717 >> v1:172.21.15.104:6791/0 conn(0x7f093013f2e0 legacy=0x7f09301d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.467+0000 7f0936911640 1 -- 172.21.15.26:0/1998635717 shutdown_connections 2024-08-19T12:46:58.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.467+0000 7f0936911640 1 -- 172.21.15.26:0/1998635717 >> 172.21.15.26:0/1998635717 conn(0x7f09300abff0 msgr2=0x7f09300a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:58.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.467+0000 7f0936911640 1 -- 172.21.15.26:0/1998635717 shutdown_connections 2024-08-19T12:46:58.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.467+0000 7f0936911640 1 -- 172.21.15.26:0/1998635717 wait complete. 2024-08-19T12:46:58.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.467+0000 7f0936911640 1 librados: shutdown 2024-08-19T12:46:58.478 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:46:58.478 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:46:58.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:46:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.579+0000 7f45a736b640 10 monclient: get_monmap_and_config 2024-08-19T12:46:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.579+0000 7f45a736b640 10 monclient: build_initial_monmap 2024-08-19T12:46:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.579+0000 7f45a736b640 10 monclient: monmap: 2024-08-19T12:46:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.580 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:46:58.580180+0000 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:46:58.580180+0000 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.579+0000 7f45a736b640 1 Processor -- start 2024-08-19T12:46:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a736b640 1 -- start start 2024-08-19T12:46:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a736b640 10 monclient: init 2024-08-19T12:46:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a736b640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a736b640 10 monclient: _add_conns ranks=[8,5,0,4,2,6,3,1,7] 2024-08-19T12:46:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a736b640 10 monclient(hunting): picked mon.noname-e con 0x7f45a006e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:46:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a736b640 10 monclient(hunting): picked mon.noname-h con 0x7f45a00a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:46:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a736b640 10 monclient(hunting): picked mon.noname-a con 0x7f45a00a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:46:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a736b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a0136790 con 0x7f45a00a5d50 2024-08-19T12:46:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a736b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a009ddf0 con 0x7f45a00a5920 2024-08-19T12:46:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a736b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a009e0d0 con 0x7f45a006e7b0 2024-08-19T12:46:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a736b640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:58.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a736b640 10 monclient(hunting): authenticate will time out at 1212.667614s 2024-08-19T12:46:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a5107640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f45a006e7b0 0x7f45a0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:41696/0 (socket says 172.21.15.26:41696) 2024-08-19T12:46:58.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.580+0000 7f45a5107640 1 -- 172.21.15.26:0/1512674867 learned_addr learned my addr 172.21.15.26:0/1512674867 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:58.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.582+0000 7f45a5908640 1 -- 172.21.15.26:0/1512674867 >> v1:172.21.15.26:6789/0 conn(0x7f45a00a5d50 legacy=0x7f45a0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.582+0000 7f45a5107640 1 -- 172.21.15.26:0/1512674867 >> v1:172.21.15.26:6793/0 conn(0x7f45a006e7b0 legacy=0x7f45a0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f45a4906640 1 -- 172.21.15.26:0/1512674867 >> v1:172.21.15.104:6791/0 conn(0x7f45a00a5920 legacy=0x7f45a013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 1 -- 172.21.15.26:0/1512674867 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4106441856 0 0) 0x7f45a0136790 con 0x7f45a00a5d50 2024-08-19T12:46:58.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient(hunting): my global_id is 144747 2024-08-19T12:46:58.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 1 -- 172.21.15.26:0/1512674867 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4588003520 con 0x7f45a00a5d50 2024-08-19T12:46:58.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 1 -- 172.21.15.26:0/1512674867 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 97226567 0 0) 0x7f45a009e0d0 con 0x7f45a006e7b0 2024-08-19T12:46:58.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient(hunting): my global_id is 26324 2024-08-19T12:46:58.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 1 -- 172.21.15.26:0/1512674867 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45a0136790 con 0x7f45a006e7b0 2024-08-19T12:46:58.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 1 -- 172.21.15.26:0/1512674867 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1304845060 0 0) 0x7f4588003520 con 0x7f45a00a5d50 2024-08-19T12:46:58.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 1 -- 172.21.15.26:0/1512674867 >> v1:172.21.15.26:6793/0 conn(0x7f45a006e7b0 legacy=0x7f45a0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 1 -- 172.21.15.26:0/1512674867 >> v1:172.21.15.104:6791/0 conn(0x7f45a00a5920 legacy=0x7f45a013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient: _finish_hunting 0 2024-08-19T12:46:58.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 1 monclient: found mon.noname-a 2024-08-19T12:46:58.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:58.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:46:58.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 1 -- 172.21.15.26:0/1512674867 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a009e3b0 con 0x7f45a00a5d50 2024-08-19T12:46:58.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient: _finish_auth 0 2024-08-19T12:46:58.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient: _check_auth_tickets 2024-08-19T12:46:58.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:58.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 1 -- 172.21.15.26:0/1512674867 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4598001b30 con 0x7f45a00a5d50 2024-08-19T12:46:58.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f45a736b640 5 monclient: authenticate success, global_id 144747 2024-08-19T12:46:58.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f45a736b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:46:58.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:58.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:46:58.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient: dump: 2024-08-19T12:46:58.598 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.598 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.599 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.600 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f45a736b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:46:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 1 -- 172.21.15.26:0/1512674867 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4598003610 con 0x7f45a00a5d50 2024-08-19T12:46:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f45a736b640 10 monclient: get_monmap_and_config success 2024-08-19T12:46:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 1 -- 172.21.15.26:0/1512674867 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4598003b80 con 0x7f45a00a5d50 2024-08-19T12:46:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f45a736b640 4 set_mon_vals no callback set 2024-08-19T12:46:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:46:58.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f459ffff640 10 monclient: dump: 2024-08-19T12:46:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.601 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.602 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.602 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.602 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.602 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.602 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f45a736b640 10 monclient: shutdown 2024-08-19T12:46:58.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f45a736b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:58.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.583+0000 7f45a736b640 1 -- 172.21.15.26:0/1512674867 >> v1:172.21.15.26:6789/0 conn(0x7f45a00a5d50 legacy=0x7f45a0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.584+0000 7f45a736b640 1 -- 172.21.15.26:0/1512674867 shutdown_connections 2024-08-19T12:46:58.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.584+0000 7f45a736b640 1 -- 172.21.15.26:0/1512674867 >> 172.21.15.26:0/1512674867 conn(0x7f45a00a8740 msgr2=0x7f45a00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:58.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.584+0000 7f45a736b640 1 -- 172.21.15.26:0/1512674867 shutdown_connections 2024-08-19T12:46:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.584+0000 7f45a736b640 1 -- 172.21.15.26:0/1512674867 wait complete. 2024-08-19T12:46:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.584+0000 7f45a736b640 10 monclient: build_initial_monmap 2024-08-19T12:46:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.584+0000 7f45a736b640 10 monclient: monmap: 2024-08-19T12:46:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:46:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:46:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:46:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:58.604 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:58.604 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:58.604 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:58.604 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:58.604 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:58.604 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:58.604 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.584+0000 7f45a736b640 1 librados: starting msgr at 2024-08-19T12:46:58.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.584+0000 7f45a736b640 1 librados: starting objecter 2024-08-19T12:46:58.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.584+0000 7f45a736b640 1 Processor -- start 2024-08-19T12:46:58.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 1 -- start start 2024-08-19T12:46:58.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 1 librados: setting wanted keys 2024-08-19T12:46:58.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 1 librados: calling monclient init 2024-08-19T12:46:58.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 10 monclient: init 2024-08-19T12:46:58.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:58.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 10 monclient: _add_conns ranks=[7,5,4,8,1,6,0,3,2] 2024-08-19T12:46:58.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 10 monclient(hunting): picked mon.noname-i con 0x7f45a006e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:46:58.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 10 monclient(hunting): picked mon.noname-h con 0x7f45a00a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:46:58.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 10 monclient(hunting): picked mon.noname-c con 0x7f45a00a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:46:58.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a01222b0 con 0x7f45a00a5d50 2024-08-19T12:46:58.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a0123750 con 0x7f45a00a5920 2024-08-19T12:46:58.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a0142ec0 con 0x7f45a006e7b0 2024-08-19T12:46:58.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:58.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a736b640 10 monclient(hunting): authenticate will time out at 1212.672465s 2024-08-19T12:46:58.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a5908640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f45a00a5d50 0x7f45a0131b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41774/0 (socket says 172.21.15.26:41774) 2024-08-19T12:46:58.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.585+0000 7f45a5908640 1 -- 172.21.15.26:0/1979516515 learned_addr learned my addr 172.21.15.26:0/1979516515 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:58.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f45a5908640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.26:6791/0 conn(0x7f45a00a5d50 legacy=0x7f45a0131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 client.?.objecter ms_handle_connect 0x7f45a00a5d50 2024-08-19T12:46:58.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:58.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 client.?.objecter ms_handle_connect 0x7f45a00a5920 2024-08-19T12:46:58.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:58.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f45a4906640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.104:6791/0 conn(0x7f45a00a5920 legacy=0x7f45a0131480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 client.?.objecter ms_handle_connect 0x7f45a006e7b0 2024-08-19T12:46:58.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f45a5107640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.104:6792/0 conn(0x7f45a006e7b0 legacy=0x7f45a0130d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:58.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3507103584 0 0) 0x7f45a01222b0 con 0x7f45a00a5d50 2024-08-19T12:46:58.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 monclient(hunting): my global_id is 26788 2024-08-19T12:46:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f458c004eb0 con 0x7f45a00a5d50 2024-08-19T12:46:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3860044966 0 0) 0x7f45a0123750 con 0x7f45a00a5920 2024-08-19T12:46:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 10 monclient(hunting): my global_id is 26456 2024-08-19T12:46:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.587+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45a01222b0 con 0x7f45a00a5920 2024-08-19T12:46:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3406977465 0 0) 0x7f45a0142ec0 con 0x7f45a006e7b0 2024-08-19T12:46:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient(hunting): my global_id is 135439 2024-08-19T12:46:58.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45a0123750 con 0x7f45a006e7b0 2024-08-19T12:46:58.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1297480900 0 0) 0x7f458c004eb0 con 0x7f45a00a5d50 2024-08-19T12:46:58.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45a0142ec0 con 0x7f45a00a5d50 2024-08-19T12:46:58.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4598003b00 con 0x7f45a00a5d50 2024-08-19T12:46:58.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:58.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:46:58.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient(hunting): dump: 2024-08-19T12:46:58.611 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.611 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.611 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.611 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.611 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.611 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.611 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.611 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.611 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.611 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.612 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.612 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.612 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.612 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.612 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.612 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:46:58.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 318514420 0 0) 0x7f45a01222b0 con 0x7f45a00a5920 2024-08-19T12:46:58.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f458c004eb0 con 0x7f45a00a5920 2024-08-19T12:46:58.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4594003950 con 0x7f45a00a5920 2024-08-19T12:46:58.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:58.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:46:58.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient(hunting): dump: 2024-08-19T12:46:58.614 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.614 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.614 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.614 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.614 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.615 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.615 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.615 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.615 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.615 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.615 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.615 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.615 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.615 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.615 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.616 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3208343578 0 0) 0x7f45a0142ec0 con 0x7f45a00a5d50 2024-08-19T12:46:58.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.104:6792/0 conn(0x7f45a006e7b0 legacy=0x7f45a0130d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.104:6791/0 conn(0x7f45a00a5920 legacy=0x7f45a0131480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:46:58.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 monclient: found mon.c 2024-08-19T12:46:58.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:58.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:46:58.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a0132670 con 0x7f45a00a5d50 2024-08-19T12:46:58.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient: _finish_auth 0 2024-08-19T12:46:58.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:46:58.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:58.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f45a736b640 5 monclient: authenticate success, global_id 26788 2024-08-19T12:46:58.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f45a736b640 10 monclient: _renew_subs 2024-08-19T12:46:58.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f45a736b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:46:58.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f45a736b640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f45a0123470 con 0x7f45a00a5d50 2024-08-19T12:46:58.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4598001b50 con 0x7f45a00a5d50 2024-08-19T12:46:58.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:58.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4598004340 con 0x7f45a00a5d50 2024-08-19T12:46:58.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:58.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:46:58.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459dffb640 10 monclient: dump: 2024-08-19T12:46:58.618 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.618 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.618 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.618 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.619 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.619 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.619 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.619 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.619 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.619 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.619 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.619 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.619 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.619 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.619 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.619 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f45a736b640 10 client.26788.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f45a736b640 10 monclient: _renew_subs 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f45a736b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f45a736b640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f45a01d7480 con 0x7f45a00a5d50 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f459f7fe640 4 set_mon_vals no callback set 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.588+0000 7f45a736b640 1 librados: init done 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.589+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f4598011a80 con 0x7f45a00a5d50 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.589+0000 7f45a736b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.589+0000 7f45a736b640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f45a00008d0 con 0x7f45a012dc30 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.589+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(349..349 src has 1..349) ==== 5346+0+0 (unknown 3654949895 0 0) 0x7f459804dee0 con 0x7f45a00a5d50 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.589+0000 7f459dffb640 10 client.26788.objecter ms_dispatch 0x7f45a0000b90 osd_map(349..349 src has 1..349) 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.589+0000 7f459dffb640 3 client.26788.objecter handle_osd_map got epochs [349,349] > 0 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.589+0000 7f459dffb640 3 client.26788.objecter handle_osd_map decoding full epoch 349 2024-08-19T12:46:58.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.589+0000 7f459dffb640 20 client.26788.objecter dump_active .. 0 homeless 2024-08-19T12:46:58.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.589+0000 7f45a4906640 10 monclient: get_auth_request con 0x7f458c041480 auth_method 0 2024-08-19T12:46:58.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.591+0000 7f459dffb640 10 client.26788.objecter ms_handle_connect 0x7f45a012dc30 2024-08-19T12:46:58.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.591+0000 7f459dffb640 10 client.26788.objecter resend_mon_ops 2024-08-19T12:46:58.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.591+0000 7f45a5107640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.26:6789/0 conn(0x7f45a012dc30 legacy=0x7f45a012e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.591+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 612410599 0 0) 0x7f45a00008d0 con 0x7f45a012dc30 2024-08-19T12:46:58.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.591+0000 7f459dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.591+0000 7f459dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.591+0000 7f459dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.591+0000 7f459dffb640 10 monclient(hunting): my global_id is 144756 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.591+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45a0142ec0 con 0x7f45a012dc30 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3070354006 0 0) 0x7f45a0142ec0 con 0x7f45a012dc30 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45a00008d0 con 0x7f45a012dc30 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4590005230 con 0x7f45a012dc30 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 10 monclient: dump: 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.622 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.623 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.623 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.623 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.623 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.623 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.623 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.623 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.623 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.623 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.623 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 10 client.26788.objecter ms_handle_connect 0x7f458c041480 2024-08-19T12:46:58.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2363687259 0 0) 0x7f45a00008d0 con 0x7f45a012dc30 2024-08-19T12:46:58.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45a01347b0 con 0x7f45a012dc30 2024-08-19T12:46:58.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f45900075d0 con 0x7f45a012dc30 2024-08-19T12:46:58.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:58.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:46:58.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.26:6789/0 conn(0x7f45a012dc30 legacy=0x7f45a012e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.592+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.26:6789/0 conn(0x7f45a012dc30 legacy=0x7f45a012e010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:58.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.598+0000 7f45a736b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:46:58.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.598+0000 7f45a736b640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f45a012eb00 con 0x7f45a012e720 2024-08-19T12:46:58.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 10 client.26788.objecter ms_handle_connect 0x7f45a012e720 2024-08-19T12:46:58.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 10 client.26788.objecter resend_mon_ops 2024-08-19T12:46:58.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f45a5107640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.26:6789/0 conn(0x7f45a012e720 legacy=0x7f45a01db670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2405807083 0 0) 0x7f45a012eb00 con 0x7f45a012e720 2024-08-19T12:46:58.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 10 monclient(hunting): my global_id is 144765 2024-08-19T12:46:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45a00008d0 con 0x7f45a012e720 2024-08-19T12:46:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4153784641 0 0) 0x7f45a00008d0 con 0x7f45a012e720 2024-08-19T12:46:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45a012eb00 con 0x7f45a012e720 2024-08-19T12:46:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4590005cd0 con 0x7f45a012e720 2024-08-19T12:46:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:46:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 10 monclient: dump: 2024-08-19T12:46:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.627 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.628 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.628 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.628 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.628 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.628 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.628 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.628 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.628 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.629 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1592335424 0 0) 0x7f45a012eb00 con 0x7f45a012e720 2024-08-19T12:46:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f45a0143140 con 0x7f45a012e720 2024-08-19T12:46:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7880 (unknown 0 0 559406623) 0x7f45900070d0 con 0x7f45a012e720 2024-08-19T12:46:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:46:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:46:58.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.26:6789/0 conn(0x7f45a012e720 legacy=0x7f45a01db670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.601+0000 7f459dffb640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.26:6789/0 conn(0x7f45a012e720 legacy=0x7f45a01db670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:58.630 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:46:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:46:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:46:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:46:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:46:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 270519, 2024-08-19T12:46:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:46:58.630 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:58.631 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:46:58.632 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:46:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:58.633 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:46:58.634 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:58.635 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:46:58.636 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:58.637 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.638 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.639 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:46:58.640 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:58.641 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:46:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:46:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:46:58.642 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:46:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:46:58.643 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:46:58.644 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:46:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:46:58.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:46:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:46:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:58.646 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:46:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:46:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:58.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:46:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.648 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:46:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:46:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:58.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:46:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:46:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:58.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:46:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:46:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:46:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.652 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:46:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:46:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:46:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:58.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:46:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:46:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:58.654 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:46:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:46:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.655 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:46:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:46:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.656 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:46:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:46:58.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:46:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:58.658 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:46:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:46:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:58.659 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:58.660 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:46:58.660 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:46:58.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 10 librados: watch_flush enter 2024-08-19T12:46:58.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 10 librados: watch_flush exit 2024-08-19T12:46:58.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 20 client.26788.objecter shutdown clearing up homeless session... 2024-08-19T12:46:58.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 10 client.26788.objecter successfully canceled tick 2024-08-19T12:46:58.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.104:6800/899887506 conn(0x7f458c041480 legacy=0x7f458c043940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 10 monclient: shutdown 2024-08-19T12:46:58.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:58.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 1 -- 172.21.15.26:0/1979516515 >> v1:172.21.15.26:6791/0 conn(0x7f45a00a5d50 legacy=0x7f45a0131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a5908640 1 -- 172.21.15.26:0/1979516515 reap_dead start 2024-08-19T12:46:58.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a5908640 1 -- 172.21.15.26:0/1979516515 reap_dead start 2024-08-19T12:46:58.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 1 -- 172.21.15.26:0/1979516515 shutdown_connections 2024-08-19T12:46:58.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 1 -- 172.21.15.26:0/1979516515 >> 172.21.15.26:0/1979516515 conn(0x7f45a00a8740 msgr2=0x7f45a006d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:58.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 1 -- 172.21.15.26:0/1979516515 shutdown_connections 2024-08-19T12:46:58.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 1 -- 172.21.15.26:0/1979516515 wait complete. 2024-08-19T12:46:58.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.602+0000 7f45a736b640 1 librados: shutdown 2024-08-19T12:46:58.665 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:46:58.665 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:46:58.665 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:46:58.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.765+0000 7f9f77858640 10 monclient: get_monmap_and_config 2024-08-19T12:46:58.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.765+0000 7f9f77858640 10 monclient: build_initial_monmap 2024-08-19T12:46:58.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.765+0000 7f9f77858640 10 monclient: monmap: 2024-08-19T12:46:58.767 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:58.767 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.767 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:46:58.765886+0000 2024-08-19T12:46:58.767 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:46:58.765886+0000 2024-08-19T12:46:58.767 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:58.767 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.767 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:58.768 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:58.768 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:58.768 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:58.768 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:58.768 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:58.768 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:58.768 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:58.769 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:58.769 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.765+0000 7f9f77858640 1 Processor -- start 2024-08-19T12:46:58.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.765+0000 7f9f77858640 1 -- start start 2024-08-19T12:46:58.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.765+0000 7f9f77858640 10 monclient: init 2024-08-19T12:46:58.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.766+0000 7f9f77858640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:58.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.766+0000 7f9f77858640 10 monclient: _add_conns ranks=[5,3,7,4,8,6,1,2,0] 2024-08-19T12:46:58.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.766+0000 7f9f77858640 10 monclient(hunting): picked mon.noname-h con 0x7f9f70073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:46:58.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.766+0000 7f9f77858640 10 monclient(hunting): picked mon.noname-g con 0x7f9f700a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:46:58.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.766+0000 7f9f77858640 10 monclient(hunting): picked mon.noname-i con 0x7f9f7013f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:46:58.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.766+0000 7f9f77858640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f7009dae0 con 0x7f9f700a57a0 2024-08-19T12:46:58.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.766+0000 7f9f77858640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f70136910 con 0x7f9f70073470 2024-08-19T12:46:58.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.766+0000 7f9f77858640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f7009dd50 con 0x7f9f7013f2e0 2024-08-19T12:46:58.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.766+0000 7f9f77858640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:58.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.766+0000 7f9f77858640 10 monclient(hunting): authenticate will time out at 1212.853449s 2024-08-19T12:46:58.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.766+0000 7f9f755f4640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f9f70073470 0x7f9f70136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:47254/0 (socket says 172.21.15.26:47254) 2024-08-19T12:46:58.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.766+0000 7f9f755f4640 1 -- 172.21.15.26:0/1242712790 learned_addr learned my addr 172.21.15.26:0/1242712790 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:58.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.768+0000 7f9f75df5640 1 -- 172.21.15.26:0/1242712790 >> v1:172.21.15.104:6792/0 conn(0x7f9f7013f2e0 legacy=0x7f9f701416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.768+0000 7f9f755f4640 1 -- 172.21.15.26:0/1242712790 >> v1:172.21.15.104:6791/0 conn(0x7f9f70073470 legacy=0x7f9f70136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.768+0000 7f9f74df3640 1 -- 172.21.15.26:0/1242712790 >> v1:172.21.15.104:6790/0 conn(0x7f9f700a57a0 legacy=0x7f9f700a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2110647768 0 0) 0x7f9f70136910 con 0x7f9f70073470 2024-08-19T12:46:58.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient(hunting): my global_id is 26465 2024-08-19T12:46:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f58003520 con 0x7f9f70073470 2024-08-19T12:46:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3910350316 0 0) 0x7f9f7009dd50 con 0x7f9f7013f2e0 2024-08-19T12:46:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient(hunting): my global_id is 135448 2024-08-19T12:46:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f70136910 con 0x7f9f7013f2e0 2024-08-19T12:46:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 635435384 0 0) 0x7f9f7009dae0 con 0x7f9f700a57a0 2024-08-19T12:46:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient(hunting): my global_id is 154299 2024-08-19T12:46:58.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f7009dd50 con 0x7f9f700a57a0 2024-08-19T12:46:58.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2911293420 0 0) 0x7f9f58003520 con 0x7f9f70073470 2024-08-19T12:46:58.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 >> v1:172.21.15.104:6792/0 conn(0x7f9f7013f2e0 legacy=0x7f9f701416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 >> v1:172.21.15.104:6790/0 conn(0x7f9f700a57a0 legacy=0x7f9f700a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient: _finish_hunting 0 2024-08-19T12:46:58.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 monclient: found mon.noname-h 2024-08-19T12:46:58.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:58.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:46:58.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7009e030 con 0x7f9f70073470 2024-08-19T12:46:58.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient: _finish_auth 0 2024-08-19T12:46:58.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient: _check_auth_tickets 2024-08-19T12:46:58.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:58.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9f68001ba0 con 0x7f9f70073470 2024-08-19T12:46:58.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f77858640 5 monclient: authenticate success, global_id 26465 2024-08-19T12:46:58.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f77858640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:46:58.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:58.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:46:58.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient: dump: 2024-08-19T12:46:58.786 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.786 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.787 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.787 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.787 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.787 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.787 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.787 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.787 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.787 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f77858640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9f680048a0 con 0x7f9f70073470 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f77858640 10 monclient: get_monmap_and_config success 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f77858640 4 set_mon_vals no callback set 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 1 -- 172.21.15.26:0/1242712790 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9f68004e10 con 0x7f9f70073470 2024-08-19T12:46:58.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.769+0000 7f9f6ffff640 10 monclient: dump: 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.789 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.770+0000 7f9f77858640 10 monclient: shutdown 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.770+0000 7f9f77858640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.770+0000 7f9f77858640 1 -- 172.21.15.26:0/1242712790 >> v1:172.21.15.104:6791/0 conn(0x7f9f70073470 legacy=0x7f9f70136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.770+0000 7f9f77858640 1 -- 172.21.15.26:0/1242712790 shutdown_connections 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.770+0000 7f9f77858640 1 -- 172.21.15.26:0/1242712790 >> 172.21.15.26:0/1242712790 conn(0x7f9f700abff0 msgr2=0x7f9f700ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.770+0000 7f9f77858640 1 -- 172.21.15.26:0/1242712790 shutdown_connections 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.770+0000 7f9f77858640 1 -- 172.21.15.26:0/1242712790 wait complete. 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.770+0000 7f9f77858640 10 monclient: build_initial_monmap 2024-08-19T12:46:58.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.770+0000 7f9f77858640 10 monclient: monmap: 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:58.791 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.770+0000 7f9f77858640 1 librados: starting msgr at 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.770+0000 7f9f77858640 1 librados: starting objecter 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 1 Processor -- start 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 1 -- start start 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 1 librados: setting wanted keys 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 1 librados: calling monclient init 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 10 monclient: init 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 10 monclient: _add_conns ranks=[6,4,5,3,7,8,2,1,0] 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 10 monclient(hunting): picked mon.noname-d con 0x7f9f70073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 10 monclient(hunting): picked mon.noname-c con 0x7f9f700a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:46:58.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 10 monclient(hunting): picked mon.noname-h con 0x7f9f7013f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f7013f0b0 con 0x7f9f700a57a0 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f70136ea0 con 0x7f9f7013f2e0 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f70137180 con 0x7f9f70073470 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f77858640 10 monclient(hunting): authenticate will time out at 1212.858708s 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f74df3640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f9f700a57a0 0x7f9f701d3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:34584/0 (socket says 172.21.15.26:34584) 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f755f4640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f9f70073470 0x7f9f701ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:43026/0 (socket says 172.21.15.26:43026) 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.771+0000 7f9f74df3640 1 -- 172.21.15.26:0/1497648277 learned_addr learned my addr 172.21.15.26:0/1497648277 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.773+0000 7f9f74df3640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.26:6791/0 conn(0x7f9f700a57a0 legacy=0x7f9f701d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.773+0000 7f9f6dffb640 10 client.?.objecter ms_handle_connect 0x7f9f700a57a0 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.773+0000 7f9f6dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.773+0000 7f9f755f4640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.26:6792/0 conn(0x7f9f70073470 legacy=0x7f9f701ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.773+0000 7f9f6dffb640 10 client.?.objecter ms_handle_connect 0x7f9f70073470 2024-08-19T12:46:58.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.773+0000 7f9f6dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:58.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f75df5640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.104:6791/0 conn(0x7f9f7013f2e0 legacy=0x7f9f701d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 client.?.objecter ms_handle_connect 0x7f9f7013f2e0 2024-08-19T12:46:58.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:58.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1200743227 0 0) 0x7f9f7009dd50 con 0x7f9f700a57a0 2024-08-19T12:46:58.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): my global_id is 26797 2024-08-19T12:46:58.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f5c004eb0 con 0x7f9f700a57a0 2024-08-19T12:46:58.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1040079863 0 0) 0x7f9f70136ea0 con 0x7f9f7013f2e0 2024-08-19T12:46:58.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:46:58.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:58.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:58.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): my global_id is 26474 2024-08-19T12:46:58.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f7009dd50 con 0x7f9f7013f2e0 2024-08-19T12:46:58.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1856823517 0 0) 0x7f9f5c004eb0 con 0x7f9f700a57a0 2024-08-19T12:46:58.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f70136ea0 con 0x7f9f700a57a0 2024-08-19T12:46:58.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9f64003990 con 0x7f9f700a57a0 2024-08-19T12:46:58.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:58.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:46:58.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): dump: 2024-08-19T12:46:58.795 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.796 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3251061851 0 0) 0x7f9f7009dd50 con 0x7f9f7013f2e0 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f5c004eb0 con 0x7f9f7013f2e0 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9f60001cb0 con 0x7f9f7013f2e0 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient(hunting): dump: 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.797 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.798 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.798 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.798 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.798 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.798 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.798 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.798 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.798 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.798 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.798 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1562888805 0 0) 0x7f9f70136ea0 con 0x7f9f700a57a0 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.26:6792/0 conn(0x7f9f70073470 legacy=0x7f9f701ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.104:6791/0 conn(0x7f9f7013f2e0 legacy=0x7f9f701d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 monclient: found mon.c 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f700a53c0 con 0x7f9f700a57a0 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient: _finish_auth 0 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f77858640 5 monclient: authenticate success, global_id 26797 2024-08-19T12:46:58.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f77858640 10 monclient: _renew_subs 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f77858640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f77858640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9f70135420 con 0x7f9f700a57a0 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9f64003ee0 con 0x7f9f700a57a0 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9f64004410 con 0x7f9f700a57a0 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6dffb640 10 monclient: dump: 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:58.800 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f6f7fe640 4 set_mon_vals no callback set 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f77858640 10 client.26797.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f77858640 10 monclient: _renew_subs 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f77858640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:46:58.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f77858640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f9f700a3140 con 0x7f9f700a57a0 2024-08-19T12:46:58.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.774+0000 7f9f77858640 1 librados: init done 2024-08-19T12:46:58.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.775+0000 7f9f77858640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:58.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.775+0000 7f9f77858640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9f70137570 con 0x7f9f70142df0 2024-08-19T12:46:58.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.776+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f9f640046b0 con 0x7f9f700a57a0 2024-08-19T12:46:58.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.776+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(349..349 src has 1..349) ==== 5346+0+0 (unknown 3654949895 0 0) 0x7f9f6404e5d0 con 0x7f9f700a57a0 2024-08-19T12:46:58.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.776+0000 7f9f6dffb640 10 client.26797.objecter ms_dispatch 0x7f9f70000b90 osd_map(349..349 src has 1..349) 2024-08-19T12:46:58.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.776+0000 7f9f6dffb640 3 client.26797.objecter handle_osd_map got epochs [349,349] > 0 2024-08-19T12:46:58.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.776+0000 7f9f6dffb640 3 client.26797.objecter handle_osd_map decoding full epoch 349 2024-08-19T12:46:58.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.776+0000 7f9f6dffb640 20 client.26797.objecter dump_active .. 0 homeless 2024-08-19T12:46:58.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.776+0000 7f9f75df5640 10 monclient: get_auth_request con 0x7f9f5c041520 auth_method 0 2024-08-19T12:46:58.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.778+0000 7f9f755f4640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.26:6790/0 conn(0x7f9f70142df0 legacy=0x7f9f700aa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:58.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.778+0000 7f9f6dffb640 10 client.26797.objecter ms_handle_connect 0x7f9f70142df0 2024-08-19T12:46:58.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.778+0000 7f9f6dffb640 10 client.26797.objecter resend_mon_ops 2024-08-19T12:46:58.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:58.778+0000 7f9f6dffb640 10 client.26797.objecter ms_handle_connect 0x7f9f5c041520 2024-08-19T12:46:59.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.231+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1586186318 0 0) 0x7f9f70137570 con 0x7f9f70142df0 2024-08-19T12:46:59.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.231+0000 7f9f6dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.231+0000 7f9f6dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.231+0000 7f9f6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.231+0000 7f9f6dffb640 10 monclient(hunting): my global_id is 26453 2024-08-19T12:46:59.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.231+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f70136ea0 con 0x7f9f70142df0 2024-08-19T12:46:59.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.231+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 485273935 0 0) 0x7f9f70136ea0 con 0x7f9f70142df0 2024-08-19T12:46:59.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.231+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f70137570 con 0x7f9f70142df0 2024-08-19T12:46:59.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.231+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9f68001ba0 con 0x7f9f70142df0 2024-08-19T12:46:59.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.231+0000 7f9f6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.231+0000 7f9f6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.231+0000 7f9f6dffb640 10 monclient: dump: 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.235 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.235 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.235 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.235 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.235 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.247+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2299495140 0 0) 0x7f9f70137570 con 0x7f9f70142df0 2024-08-19T12:46:59.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.247+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f7013e2e0 con 0x7f9f70142df0 2024-08-19T12:46:59.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.247+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9f680047e0 con 0x7f9f70142df0 2024-08-19T12:46:59.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.247+0000 7f9f6dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:59.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.247+0000 7f9f6dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:46:59.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.247+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.26:6790/0 conn(0x7f9f70142df0 legacy=0x7f9f700aa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.247+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.26:6790/0 conn(0x7f9f70142df0 legacy=0x7f9f700aa080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:59.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.254+0000 7f9f77858640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:46:59.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.254+0000 7f9f77858640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9f700a3be0 con 0x7f9f701cea80 2024-08-19T12:46:59.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.256+0000 7f9f6dffb640 10 client.26797.objecter ms_handle_connect 0x7f9f701cea80 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.256+0000 7f9f6dffb640 10 client.26797.objecter resend_mon_ops 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.256+0000 7f9f755f4640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.26:6790/0 conn(0x7f9f701cea80 legacy=0x7f9f700aa790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3732823584 0 0) 0x7f9f700a3be0 con 0x7f9f701cea80 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 10 monclient(hunting): my global_id is 26462 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f70137570 con 0x7f9f701cea80 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2048980850 0 0) 0x7f9f70137570 con 0x7f9f701cea80 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9f700a3be0 con 0x7f9f701cea80 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9f68005410 con 0x7f9f701cea80 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:46:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 10 monclient: dump: 2024-08-19T12:46:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.260 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.261 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3252348050 0 0) 0x7f9f700a3be0 con 0x7f9f701cea80 2024-08-19T12:46:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9f700008d0 con 0x7f9f701cea80 2024-08-19T12:46:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 4079763952) 0x7f9f68007d80 con 0x7f9f701cea80 2024-08-19T12:46:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:46:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:46:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.26:6790/0 conn(0x7f9f701cea80 legacy=0x7f9f700aa790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.257+0000 7f9f6dffb640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.26:6790/0 conn(0x7f9f701cea80 legacy=0x7f9f700aa790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:59.262 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:46:59.262 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:46:59.262 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:46:59.262 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:46:59.262 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:46:59.262 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 532312, 2024-08-19T12:46:59.262 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:46:59.262 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:46:59.262 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:46:59.262 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.263 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.264 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.264 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.264 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.264 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.264 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.264 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.265 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.266 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.266 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.266 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.266 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.266 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:46:59.266 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:46:59.266 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:46:59.266 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:46:59.266 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:46:59.267 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:46:59.267 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:46:59.267 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:46:59.267 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:46:59.267 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:46:59.267 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:46:59.267 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:46:59.267 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:46:59.267 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:46:59.267 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:46:59.267 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:46:59.267 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:46:59.268 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.268 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.268 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.268 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.268 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.268 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.268 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.268 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.268 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.269 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.269 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.269 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.269 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:46:59.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.269 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:46:59.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.269 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:46:59.269 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:46:59.269 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.269 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.271 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:46:59.271 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:46:59.271 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.271 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.271 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.271 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:46:59.271 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.271 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.271 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:46:59.272 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:46:59.272 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.272 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.272 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.272 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:46:59.272 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:46:59.272 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.272 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:46:59.273 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.273 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:46:59.273 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:46:59.273 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.273 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.273 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.274 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:46:59.274 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:46:59.274 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.274 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:46:59.274 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.274 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.275 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:46:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:46:59.277 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.277 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.277 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.277 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:46:59.277 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.277 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.277 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.277 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:46:59.277 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:46:59.278 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.278 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.278 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.278 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:46:59.278 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:46:59.278 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.278 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.280 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:46:59.280 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:46:59.280 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.280 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.281 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:46:59.281 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.281 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:46:59.281 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:46:59.281 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.281 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.282 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.282 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:46:59.282 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:46:59.282 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.282 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.282 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.283 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:46:59.283 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.283 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.283 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:46:59.283 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:46:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:46:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:46:59.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:46:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:46:59.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:46:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.286 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:46:59.286 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:46:59.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 10 librados: watch_flush enter 2024-08-19T12:46:59.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 10 librados: watch_flush exit 2024-08-19T12:46:59.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 20 client.26797.objecter shutdown clearing up homeless session... 2024-08-19T12:46:59.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 10 client.26797.objecter successfully canceled tick 2024-08-19T12:46:59.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.104:6800/899887506 conn(0x7f9f5c041520 legacy=0x7f9f5c0439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 10 monclient: shutdown 2024-08-19T12:46:59.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:59.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 1 -- 172.21.15.26:0/1497648277 >> v1:172.21.15.26:6791/0 conn(0x7f9f700a57a0 legacy=0x7f9f701d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f75df5640 1 -- 172.21.15.26:0/1497648277 reap_dead start 2024-08-19T12:46:59.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f75df5640 1 -- 172.21.15.26:0/1497648277 reap_dead start 2024-08-19T12:46:59.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 1 -- 172.21.15.26:0/1497648277 shutdown_connections 2024-08-19T12:46:59.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 1 -- 172.21.15.26:0/1497648277 >> 172.21.15.26:0/1497648277 conn(0x7f9f700abff0 msgr2=0x7f9f70134a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:59.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 1 -- 172.21.15.26:0/1497648277 shutdown_connections 2024-08-19T12:46:59.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 1 -- 172.21.15.26:0/1497648277 wait complete. 2024-08-19T12:46:59.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.258+0000 7f9f77858640 1 librados: shutdown 2024-08-19T12:46:59.289 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:46:59.290 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:46:59.290 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:46:59.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.389+0000 7fc3d833a640 10 monclient: get_monmap_and_config 2024-08-19T12:46:59.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.389+0000 7fc3d833a640 10 monclient: build_initial_monmap 2024-08-19T12:46:59.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.389+0000 7fc3d833a640 10 monclient: monmap: 2024-08-19T12:46:59.390 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:59.390 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.390 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:46:59.390027+0000 2024-08-19T12:46:59.390 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:46:59.390027+0000 2024-08-19T12:46:59.390 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:59.390 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.390 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:59.390 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:59.390 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:59.391 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:59.391 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:59.391 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:59.391 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:59.391 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:59.391 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:59.391 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.389+0000 7fc3d833a640 1 Processor -- start 2024-08-19T12:46:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.389+0000 7fc3d833a640 1 -- start start 2024-08-19T12:46:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.389+0000 7fc3d833a640 10 monclient: init 2024-08-19T12:46:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d833a640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d833a640 10 monclient: _add_conns ranks=[2,8,0,3,5,1,4,7,6] 2024-08-19T12:46:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d833a640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:46:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d833a640 10 monclient(hunting): picked mon.noname-e con 0x7fc3d00a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:46:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d833a640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d00a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:46:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d833a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d0136790 con 0x7fc3d00a5d50 2024-08-19T12:46:59.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d833a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d009ddf0 con 0x7fc3d006e7b0 2024-08-19T12:46:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d833a640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d009e0d0 con 0x7fc3d00a5920 2024-08-19T12:46:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d833a640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d833a640 10 monclient(hunting): authenticate will time out at 1213.477448s 2024-08-19T12:46:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d60d6640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc3d006e7b0 0x7fc3d0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38942/0 (socket says 172.21.15.26:38942) 2024-08-19T12:46:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d60d6640 1 -- 172.21.15.26:0/2680207381 learned_addr learned my addr 172.21.15.26:0/2680207381 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.390+0000 7fc3d58d5640 1 --1- 172.21.15.26:0/2680207381 >> v1:172.21.15.26:6793/0 conn(0x7fc3d00a5920 0x7fc3d013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:56106/0 (socket says 172.21.15.26:56106) 2024-08-19T12:46:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.392+0000 7fc3d68d7640 1 -- 172.21.15.26:0/2680207381 >> v1:172.21.15.26:6789/0 conn(0x7fc3d00a5d50 legacy=0x7fc3d0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.392+0000 7fc3d58d5640 1 -- 172.21.15.26:0/2680207381 >> v1:172.21.15.26:6793/0 conn(0x7fc3d00a5920 legacy=0x7fc3d013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.392+0000 7fc3d60d6640 1 -- 172.21.15.26:0/2680207381 >> v1:172.21.15.26:6790/0 conn(0x7fc3d006e7b0 legacy=0x7fc3d0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.392+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4018386787 0 0) 0x7fc3d0136790 con 0x7fc3d00a5d50 2024-08-19T12:46:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.392+0000 7fc3d50d4640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.392+0000 7fc3d50d4640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.392+0000 7fc3d50d4640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.392+0000 7fc3d50d4640 10 monclient(hunting): my global_id is 144774 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.392+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3b8003520 con 0x7fc3d00a5d50 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.392+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1222131359 0 0) 0x7fc3d009e0d0 con 0x7fc3d00a5920 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient(hunting): my global_id is 26333 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d0136790 con 0x7fc3d00a5920 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4146016357 0 0) 0x7fc3c8001450 con 0x7fc3d006e7b0 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient(hunting): my global_id is 26471 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d009e0d0 con 0x7fc3d006e7b0 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d58d5640 1 -- 172.21.15.26:0/2680207381 >> v1:172.21.15.26:6793/0 conn(0x7fc3d00a5920 legacy=0x7fc3d013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d58d5640 1 -- 172.21.15.26:0/2680207381 >> v1:172.21.15.26:6793/0 conn(0x7fc3d00a5920 legacy=0x7fc3d013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d58d5640 1 --1- 172.21.15.26:0/2680207381 >> v1:172.21.15.26:6793/0 conn(0x7fc3d00a5920 0x7fc3d013e2f0 :-1 s=OPENED pgs=3395 cs=1 l=1).handle_message read tag failed 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d58d5640 1 --1- 172.21.15.26:0/2680207381 >> v1:172.21.15.26:6793/0 conn(0x7fc3d00a5920 0x7fc3d013e2f0 :-1 s=OPENED pgs=3395 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6793/0 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 667253857 0 0) 0x7fc3b8003520 con 0x7fc3d00a5d50 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 >> v1:172.21.15.26:6793/0 conn(0x7fc3d00a5920 legacy=0x7fc3d013e2f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 >> v1:172.21.15.26:6790/0 conn(0x7fc3d006e7b0 legacy=0x7fc3d0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient: _finish_hunting 0 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 1 monclient: found mon.noname-a 2024-08-19T12:46:59.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:46:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d009e3b0 con 0x7fc3d00a5d50 2024-08-19T12:46:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient: _finish_auth 0 2024-08-19T12:46:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient: _check_auth_tickets 2024-08-19T12:46:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc3c0001eb0 con 0x7fc3d00a5d50 2024-08-19T12:46:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d833a640 5 monclient: authenticate success, global_id 144774 2024-08-19T12:46:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d833a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:46:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:46:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient: dump: 2024-08-19T12:46:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.410 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d833a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:46:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3c0003990 con 0x7fc3d00a5d50 2024-08-19T12:46:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 1 -- 172.21.15.26:0/2680207381 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc3c0003f40 con 0x7fc3d00a5d50 2024-08-19T12:46:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d833a640 10 monclient: get_monmap_and_config success 2024-08-19T12:46:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d833a640 4 set_mon_vals no callback set 2024-08-19T12:46:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:46:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d50d4640 10 monclient: dump: 2024-08-19T12:46:59.410 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.411 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.412 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d833a640 10 monclient: shutdown 2024-08-19T12:46:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d833a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.393+0000 7fc3d833a640 1 -- 172.21.15.26:0/2680207381 >> v1:172.21.15.26:6789/0 conn(0x7fc3d00a5d50 legacy=0x7fc3d0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.394+0000 7fc3d833a640 1 -- 172.21.15.26:0/2680207381 shutdown_connections 2024-08-19T12:46:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.394+0000 7fc3d833a640 1 -- 172.21.15.26:0/2680207381 >> 172.21.15.26:0/2680207381 conn(0x7fc3d00a8740 msgr2=0x7fc3d00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:59.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.394+0000 7fc3d833a640 1 -- 172.21.15.26:0/2680207381 shutdown_connections 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.394+0000 7fc3d833a640 1 -- 172.21.15.26:0/2680207381 wait complete. 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.394+0000 7fc3d833a640 10 monclient: build_initial_monmap 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.394+0000 7fc3d833a640 10 monclient: monmap: 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:59.413 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:59.414 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:59.414 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:59.414 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:59.414 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:59.414 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.394+0000 7fc3d833a640 1 librados: starting msgr at 2024-08-19T12:46:59.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.394+0000 7fc3d833a640 1 librados: starting objecter 2024-08-19T12:46:59.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.394+0000 7fc3d833a640 1 Processor -- start 2024-08-19T12:46:59.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 1 -- start start 2024-08-19T12:46:59.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 1 librados: setting wanted keys 2024-08-19T12:46:59.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 1 librados: calling monclient init 2024-08-19T12:46:59.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 10 monclient: init 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 10 monclient: _add_conns ranks=[3,8,2,7,0,6,5,4,1] 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 10 monclient(hunting): picked mon.noname-g con 0x7fc3d006e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 10 monclient(hunting): picked mon.noname-e con 0x7fc3d00a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d00a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d00a0ce0 con 0x7fc3d00a5d50 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d00a1840 con 0x7fc3d006e7b0 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d00a27a0 con 0x7fc3d00a5920 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d833a640 10 monclient(hunting): authenticate will time out at 1213.482555s 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d68d7640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc3d00a5d50 0x7fc3d01d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38958/0 (socket says 172.21.15.26:38958) 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d68d7640 1 -- 172.21.15.26:0/196573507 learned_addr learned my addr 172.21.15.26:0/196573507 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:59.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.395+0000 7fc3d58d5640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fc3d00a5920 0x7fc3d009fd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:56110/0 (socket says 172.21.15.26:56110) 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3d58d5640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.26:6793/0 conn(0x7fc3d00a5920 legacy=0x7fc3d009fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 10 client.?.objecter ms_handle_connect 0x7fc3d00a5920 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3d68d7640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.26:6790/0 conn(0x7fc3d00a5d50 legacy=0x7fc3d01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 10 client.?.objecter ms_handle_connect 0x7fc3d00a5d50 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2593763879 0 0) 0x7fc3d00a27a0 con 0x7fc3d00a5920 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 10 monclient(hunting): my global_id is 26342 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3bc004eb0 con 0x7fc3d00a5920 2024-08-19T12:46:59.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2723572961 0 0) 0x7fc3d00a0ce0 con 0x7fc3d00a5d50 2024-08-19T12:46:59.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 10 monclient(hunting): my global_id is 26480 2024-08-19T12:46:59.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.397+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d00a27a0 con 0x7fc3d00a5d50 2024-08-19T12:46:59.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3d60d6640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.104:6790/0 conn(0x7fc3d006e7b0 legacy=0x7fc3d009f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 client.?.objecter ms_handle_connect 0x7fc3d006e7b0 2024-08-19T12:46:59.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:59.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2657667296 0 0) 0x7fc3bc004eb0 con 0x7fc3d00a5920 2024-08-19T12:46:59.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3d00a0ce0 con 0x7fc3d00a5920 2024-08-19T12:46:59.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc3c40047e0 con 0x7fc3d00a5920 2024-08-19T12:46:59.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient(hunting): dump: 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.418 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.419 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.419 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.419 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.419 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.419 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:46:59.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 135081166 0 0) 0x7fc3d00a27a0 con 0x7fc3d00a5d50 2024-08-19T12:46:59.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3bc004eb0 con 0x7fc3d00a5d50 2024-08-19T12:46:59.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc3c0001e30 con 0x7fc3d00a5d50 2024-08-19T12:46:59.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:59.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:46:59.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient(hunting): dump: 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.420 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.421 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.421 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.421 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1013032633 0 0) 0x7fc3d00a0ce0 con 0x7fc3d00a5920 2024-08-19T12:46:59.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.104:6790/0 conn(0x7fc3d006e7b0 legacy=0x7fc3d009f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.26:6790/0 conn(0x7fc3d00a5d50 legacy=0x7fc3d01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient: _finish_hunting 0 2024-08-19T12:46:59.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 monclient: found mon.e 2024-08-19T12:46:59.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:59.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:46:59.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d00a2a80 con 0x7fc3d00a5920 2024-08-19T12:46:59.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient: _finish_auth 0 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient: _check_auth_tickets 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3d833a640 5 monclient: authenticate success, global_id 26342 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3d833a640 10 monclient: _renew_subs 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3d833a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3d833a640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d00a2d60 con 0x7fc3d00a5920 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3d833a640 10 client.26342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3d833a640 10 monclient: _renew_subs 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3d833a640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3d833a640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3d00a2430 con 0x7fc3d00a5920 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3d833a640 1 librados: init done 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3c4004d60 con 0x7fc3d00a5920 2024-08-19T12:46:59.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc3c4005250 con 0x7fc3d00a5920 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3d48d3640 4 set_mon_vals no callback set 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.398+0000 7fc3ceffd640 10 monclient: dump: 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.423 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.399+0000 7fc3d833a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.399+0000 7fc3d833a640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc3b00065c0 con 0x7fc3b0002860 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.399+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fc3c40116b0 con 0x7fc3d00a5920 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.399+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(350..350 src has 1..350) ==== 5346+0+0 (unknown 4082435385 0 0) 0x7fc3c404e2b0 con 0x7fc3d00a5920 2024-08-19T12:46:59.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.399+0000 7fc3ceffd640 10 client.26342.objecter ms_dispatch 0x7fc3d0000b90 osd_map(350..350 src has 1..350) 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.399+0000 7fc3ceffd640 3 client.26342.objecter handle_osd_map got epochs [350,350] > 0 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.399+0000 7fc3ceffd640 3 client.26342.objecter handle_osd_map decoding full epoch 350 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.399+0000 7fc3ceffd640 20 client.26342.objecter dump_active .. 0 homeless 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.400+0000 7fc3d68d7640 10 monclient: get_auth_request con 0x7fc3bc0413f0 auth_method 0 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3d60d6640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.26:6791/0 conn(0x7fc3b0002860 legacy=0x7fc3b0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 10 client.26342.objecter ms_handle_connect 0x7fc3b0002860 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 10 client.26342.objecter resend_mon_ops 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2625009168 0 0) 0x7fc3b00065c0 con 0x7fc3b0002860 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 10 monclient(hunting): my global_id is 26806 2024-08-19T12:46:59.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d00a0ce0 con 0x7fc3b0002860 2024-08-19T12:46:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 678090653 0 0) 0x7fc3d00a0ce0 con 0x7fc3b0002860 2024-08-19T12:46:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3b00065c0 con 0x7fc3b0002860 2024-08-19T12:46:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc3c8001a10 con 0x7fc3b0002860 2024-08-19T12:46:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:46:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.401+0000 7fc3ceffd640 10 monclient: dump: 2024-08-19T12:46:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.426 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.402+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3541480136 0 0) 0x7fc3b00065c0 con 0x7fc3b0002860 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.402+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3b000a160 con 0x7fc3b0002860 2024-08-19T12:46:59.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.402+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc3c8004940 con 0x7fc3b0002860 2024-08-19T12:46:59.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.402+0000 7fc3ceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:59.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.402+0000 7fc3ceffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:46:59.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.402+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.26:6791/0 conn(0x7fc3b0002860 legacy=0x7fc3b0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.402+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.26:6791/0 conn(0x7fc3b0002860 legacy=0x7fc3b0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:59.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.402+0000 7fc3ceffd640 10 client.26342.objecter ms_handle_connect 0x7fc3bc0413f0 2024-08-19T12:46:59.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.408+0000 7fc3d833a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:46:59.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.408+0000 7fc3d833a640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc3b0009e50 con 0x7fc3b0009110 2024-08-19T12:46:59.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.410+0000 7fc3d60d6640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.26:6791/0 conn(0x7fc3b0009110 legacy=0x7fc3b00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.410+0000 7fc3ceffd640 10 client.26342.objecter ms_handle_connect 0x7fc3b0009110 2024-08-19T12:46:59.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.410+0000 7fc3ceffd640 10 client.26342.objecter resend_mon_ops 2024-08-19T12:46:59.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.410+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1888727462 0 0) 0x7fc3b0009e50 con 0x7fc3b0009110 2024-08-19T12:46:59.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.410+0000 7fc3ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.410+0000 7fc3ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.410+0000 7fc3ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.410+0000 7fc3ceffd640 10 monclient(hunting): my global_id is 26815 2024-08-19T12:46:59.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.410+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3b00065c0 con 0x7fc3b0009110 2024-08-19T12:46:59.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3314054182 0 0) 0x7fc3b00065c0 con 0x7fc3b0009110 2024-08-19T12:46:59.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3b0009e50 con 0x7fc3b0009110 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc3c8005140 con 0x7fc3b0009110 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 10 monclient: dump: 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.431 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1054082016 0 0) 0x7fc3b0009e50 con 0x7fc3b0009110 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc3b000c3f0 con 0x7fc3b0009110 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1490531634) 0x7fc3c8005270 con 0x7fc3b0009110 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.26:6791/0 conn(0x7fc3b0009110 legacy=0x7fc3b00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3ceffd640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.26:6791/0 conn(0x7fc3b0009110 legacy=0x7fc3b00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:59.433 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:46:59.433 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:46:59.433 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:46:59.433 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:46:59.433 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:46:59.433 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 532441, 2024-08-19T12:46:59.433 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:46:59.433 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:46:59.433 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:46:59.433 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:46:59.434 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:46:59.434 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:46:59.434 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:46:59.434 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:46:59.434 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:46:59.434 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:46:59.434 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.434 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:46:59.434 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:46:59.434 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:46:59.434 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:46:59.434 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:46:59.435 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.436 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:46:59.437 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:46:59.438 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:46:59.438 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:46:59.438 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:46:59.438 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.438 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.438 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.438 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.438 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.438 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.438 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.438 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.438 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:46:59.440 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.441 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:46:59.444 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.445 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.446 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.448 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:46:59.449 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.450 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:59.451 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:46:59.452 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:46:59.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d833a640 10 librados: watch_flush enter 2024-08-19T12:46:59.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d833a640 10 librados: watch_flush exit 2024-08-19T12:46:59.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d833a640 20 client.26342.objecter shutdown clearing up homeless session... 2024-08-19T12:46:59.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d833a640 10 client.26342.objecter successfully canceled tick 2024-08-19T12:46:59.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d833a640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.104:6800/899887506 conn(0x7fc3bc0413f0 legacy=0x7fc3bc0438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d833a640 10 monclient: shutdown 2024-08-19T12:46:59.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d833a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:59.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d833a640 1 -- 172.21.15.26:0/196573507 >> v1:172.21.15.26:6793/0 conn(0x7fc3d00a5920 legacy=0x7fc3d009fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d68d7640 1 -- 172.21.15.26:0/196573507 reap_dead start 2024-08-19T12:46:59.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d68d7640 1 -- 172.21.15.26:0/196573507 reap_dead start 2024-08-19T12:46:59.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d833a640 1 -- 172.21.15.26:0/196573507 shutdown_connections 2024-08-19T12:46:59.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d833a640 1 -- 172.21.15.26:0/196573507 >> 172.21.15.26:0/196573507 conn(0x7fc3d00a8740 msgr2=0x7fc3d00aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:59.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d833a640 1 -- 172.21.15.26:0/196573507 shutdown_connections 2024-08-19T12:46:59.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.411+0000 7fc3d833a640 1 -- 172.21.15.26:0/196573507 wait complete. 2024-08-19T12:46:59.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.412+0000 7fc3d833a640 1 librados: shutdown 2024-08-19T12:46:59.456 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:46:59.456 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:46:59.456 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:46:59.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.555+0000 7fe20833a640 10 monclient: get_monmap_and_config 2024-08-19T12:46:59.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.555+0000 7fe20833a640 10 monclient: build_initial_monmap 2024-08-19T12:46:59.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.555+0000 7fe20833a640 10 monclient: monmap: 2024-08-19T12:46:59.556 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:59.556 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.556 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:46:59.556519+0000 2024-08-19T12:46:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:46:59.556519+0000 2024-08-19T12:46:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:59.557 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:59.558 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:59.558 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 1 Processor -- start 2024-08-19T12:46:59.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 1 -- start start 2024-08-19T12:46:59.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 10 monclient: init 2024-08-19T12:46:59.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:59.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 10 monclient: _add_conns ranks=[5,2,7,1,8,0,3,6,4] 2024-08-19T12:46:59.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 10 monclient(hunting): picked mon.noname-h con 0x7fe20006e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:46:59.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 10 monclient(hunting): picked mon.noname-b con 0x7fe2000a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:46:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 10 monclient(hunting): picked mon.noname-i con 0x7fe2000a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:46:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe200136790 con 0x7fe2000a5920 2024-08-19T12:46:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe20009ddf0 con 0x7fe20006e7b0 2024-08-19T12:46:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe20009e0d0 con 0x7fe2000a5d50 2024-08-19T12:46:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.556+0000 7fe20833a640 10 monclient(hunting): authenticate will time out at 1213.643975s 2024-08-19T12:46:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.557+0000 7fe2058d5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe2000a5920 0x7fe20013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38972/0 (socket says 172.21.15.26:38972) 2024-08-19T12:46:59.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.557+0000 7fe2058d5640 1 -- 172.21.15.26:0/4026929886 learned_addr learned my addr 172.21.15.26:0/4026929886 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:59.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2058d5640 1 -- 172.21.15.26:0/4026929886 >> v1:172.21.15.26:6790/0 conn(0x7fe2000a5920 legacy=0x7fe20013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 1 -- 172.21.15.26:0/4026929886 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1181945032 0 0) 0x7fe200136790 con 0x7fe2000a5920 2024-08-19T12:46:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient(hunting): my global_id is 26489 2024-08-19T12:46:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 1 -- 172.21.15.26:0/4026929886 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1ec003520 con 0x7fe2000a5920 2024-08-19T12:46:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2060d6640 1 -- 172.21.15.26:0/4026929886 >> v1:172.21.15.104:6791/0 conn(0x7fe20006e7b0 legacy=0x7fe200136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2068d7640 1 -- 172.21.15.26:0/4026929886 >> v1:172.21.15.104:6792/0 conn(0x7fe2000a5d50 legacy=0x7fe200141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 1 -- 172.21.15.26:0/4026929886 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1162731344 0 0) 0x7fe1ec003520 con 0x7fe2000a5920 2024-08-19T12:46:59.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 1 -- 172.21.15.26:0/4026929886 >> v1:172.21.15.104:6792/0 conn(0x7fe2000a5d50 legacy=0x7fe200141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 1 -- 172.21.15.26:0/4026929886 >> v1:172.21.15.104:6791/0 conn(0x7fe20006e7b0 legacy=0x7fe200136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient: _finish_hunting 0 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 1 monclient: found mon.noname-b 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 1 -- 172.21.15.26:0/4026929886 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20009e3b0 con 0x7fe2000a5920 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient: _finish_auth 0 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient: _check_auth_tickets 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 1 -- 172.21.15.26:0/4026929886 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1f0001ae0 con 0x7fe2000a5920 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:46:59.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient: dump: 2024-08-19T12:46:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.573 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.574 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe20833a640 5 monclient: authenticate success, global_id 26489 2024-08-19T12:46:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe20833a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:46:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 1 -- 172.21.15.26:0/4026929886 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1f00048a0 con 0x7fe2000a5920 2024-08-19T12:46:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 1 -- 172.21.15.26:0/4026929886 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1f0004dd0 con 0x7fe2000a5920 2024-08-19T12:46:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe20833a640 10 monclient: get_monmap_and_config success 2024-08-19T12:46:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe20833a640 4 set_mon_vals no callback set 2024-08-19T12:46:59.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.559+0000 7fe2050d4640 10 monclient: dump: 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.560+0000 7fe20833a640 10 monclient: shutdown 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.560+0000 7fe20833a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.560+0000 7fe20833a640 1 -- 172.21.15.26:0/4026929886 >> v1:172.21.15.26:6790/0 conn(0x7fe2000a5920 legacy=0x7fe20013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.560+0000 7fe20833a640 1 -- 172.21.15.26:0/4026929886 shutdown_connections 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.560+0000 7fe20833a640 1 -- 172.21.15.26:0/4026929886 >> 172.21.15.26:0/4026929886 conn(0x7fe2000a8740 msgr2=0x7fe2000a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.560+0000 7fe20833a640 1 -- 172.21.15.26:0/4026929886 shutdown_connections 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.560+0000 7fe20833a640 1 -- 172.21.15.26:0/4026929886 wait complete. 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.560+0000 7fe20833a640 10 monclient: build_initial_monmap 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 10 monclient: monmap: 2024-08-19T12:46:59.576 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:46:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:46:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:46:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:59.577 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:59.578 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:59.578 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:59.578 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 1 librados: starting msgr at 2024-08-19T12:46:59.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 1 librados: starting objecter 2024-08-19T12:46:59.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 1 Processor -- start 2024-08-19T12:46:59.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 1 -- start start 2024-08-19T12:46:59.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 1 librados: setting wanted keys 2024-08-19T12:46:59.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 1 librados: calling monclient init 2024-08-19T12:46:59.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 10 monclient: init 2024-08-19T12:46:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 10 monclient: _add_conns ranks=[4,0,6,5,2,3,1,7,8] 2024-08-19T12:46:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 10 monclient(hunting): picked mon.noname-c con 0x7fe20006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:46:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 10 monclient(hunting): picked mon.noname-a con 0x7fe2000a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:46:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 10 monclient(hunting): picked mon.noname-d con 0x7fe2000a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:46:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe20013b1f0 con 0x7fe2000a5920 2024-08-19T12:46:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe200134650 con 0x7fe20006e7b0 2024-08-19T12:46:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe200139150 con 0x7fe2000a5d50 2024-08-19T12:46:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe20833a640 10 monclient(hunting): authenticate will time out at 1213.648895s 2024-08-19T12:46:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe2060d6640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fe20006e7b0 0x7fe2001ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:34616/0 (socket says 172.21.15.26:34616) 2024-08-19T12:46:59.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.561+0000 7fe2060d6640 1 -- 172.21.15.26:0/3508469866 learned_addr learned my addr 172.21.15.26:0/3508469866 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe2060d6640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.26:6791/0 conn(0x7fe20006e7b0 legacy=0x7fe2001ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 client.?.objecter ms_handle_connect 0x7fe20006e7b0 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 client.?.objecter ms_handle_connect 0x7fe2000a5d50 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe2068d7640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.26:6792/0 conn(0x7fe2000a5d50 legacy=0x7fe2001d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 client.?.objecter ms_handle_connect 0x7fe2000a5920 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe2058d5640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.26:6789/0 conn(0x7fe2000a5920 legacy=0x7fe2001d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2216772661 0 0) 0x7fe200134650 con 0x7fe20006e7b0 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): my global_id is 26824 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1e4004eb0 con 0x7fe20006e7b0 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2549920054 0 0) 0x7fe200139150 con 0x7fe2000a5d50 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): my global_id is 164103 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe200134650 con 0x7fe2000a5d50 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247301805 0 0) 0x7fe20013b1f0 con 0x7fe2000a5920 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): my global_id is 144783 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe200139150 con 0x7fe2000a5920 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3411216540 0 0) 0x7fe200134650 con 0x7fe2000a5d50 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe20013b1f0 con 0x7fe2000a5d50 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2744111149 0 0) 0x7fe1e4004eb0 con 0x7fe20006e7b0 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe200134650 con 0x7fe20006e7b0 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2273358920 0 0) 0x7fe200139150 con 0x7fe2000a5920 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1e4004eb0 con 0x7fe2000a5920 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1f8002030 con 0x7fe2000a5d50 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): dump: 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.582 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.583 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1fc003610 con 0x7fe20006e7b0 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): dump: 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.584 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1f0004d90 con 0x7fe2000a5920 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient(hunting): dump: 2024-08-19T12:46:59.585 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.586 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3279635888 0 0) 0x7fe20013b1f0 con 0x7fe2000a5d50 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.26:6791/0 conn(0x7fe20006e7b0 legacy=0x7fe2001ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.26:6789/0 conn(0x7fe2000a5920 legacy=0x7fe2001d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 monclient: found mon.d 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20013ac10 con 0x7fe2000a5d50 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient: _finish_auth 0 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:46:59.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe1f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe20833a640 5 monclient: authenticate success, global_id 164103 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe20833a640 10 monclient: _renew_subs 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe20833a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe20833a640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe20013b4b0 con 0x7fe2000a5d50 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe20833a640 10 client.164103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe20833a640 10 monclient: _renew_subs 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe20833a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe20833a640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe2001387b0 con 0x7fe2000a5d50 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.564+0000 7fe20833a640 1 librados: init done 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.565+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1f8003de0 con 0x7fe2000a5d50 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.565+0000 7fe1f6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.565+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1f8004350 con 0x7fe2000a5d50 2024-08-19T12:46:59.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.565+0000 7fe1f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.565+0000 7fe2048d3640 4 set_mon_vals no callback set 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.565+0000 7fe1f6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.565+0000 7fe1f6ffd640 10 monclient: dump: 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.589 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.565+0000 7fe20833a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.565+0000 7fe20833a640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1e80065c0 con 0x7fe1e8002860 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.565+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe1f80044f0 con 0x7fe2000a5d50 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.566+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(350..350 src has 1..350) ==== 5346+0+0 (unknown 4082435385 0 0) 0x7fe1f804e2e0 con 0x7fe2000a5d50 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.566+0000 7fe1f6ffd640 10 client.164103.objecter ms_dispatch 0x7fe200000b90 osd_map(350..350 src has 1..350) 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.566+0000 7fe1f6ffd640 3 client.164103.objecter handle_osd_map got epochs [350,350] > 0 2024-08-19T12:46:59.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.566+0000 7fe1f6ffd640 3 client.164103.objecter handle_osd_map decoding full epoch 350 2024-08-19T12:46:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.566+0000 7fe1f6ffd640 20 client.164103.objecter dump_active .. 0 homeless 2024-08-19T12:46:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.566+0000 7fe2058d5640 10 monclient: get_auth_request con 0x7fe1e4041800 auth_method 0 2024-08-19T12:46:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.567+0000 7fe2060d6640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.26:6792/0 conn(0x7fe1e8002860 legacy=0x7fe1e8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.567+0000 7fe1f6ffd640 10 client.164103.objecter ms_handle_connect 0x7fe1e8002860 2024-08-19T12:46:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.567+0000 7fe1f6ffd640 10 client.164103.objecter resend_mon_ops 2024-08-19T12:46:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 430531895 0 0) 0x7fe1e80065c0 con 0x7fe1e8002860 2024-08-19T12:46:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 10 monclient(hunting): my global_id is 164112 2024-08-19T12:46:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe20013b1f0 con 0x7fe1e8002860 2024-08-19T12:46:59.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4095813131 0 0) 0x7fe20013b1f0 con 0x7fe1e8002860 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1e80065c0 con 0x7fe1e8002860 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1fc004140 con 0x7fe1e8002860 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 10 monclient: dump: 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.592 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 585013354 0 0) 0x7fe1e80065c0 con 0x7fe1e8002860 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe1e800a160 con 0x7fe1e8002860 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 10 client.164103.objecter ms_handle_connect 0x7fe1e4041800 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe1fc004430 con 0x7fe1e8002860 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:59.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:46:59.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.26:6792/0 conn(0x7fe1e8002860 legacy=0x7fe1e8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.568+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.26:6792/0 conn(0x7fe1e8002860 legacy=0x7fe1e8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:59.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.574+0000 7fe20833a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:46:59.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.574+0000 7fe20833a640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1e8009e50 con 0x7fe1e8009110 2024-08-19T12:46:59.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe2060d6640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.26:6792/0 conn(0x7fe1e8009110 legacy=0x7fe1e80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 10 client.164103.objecter ms_handle_connect 0x7fe1e8009110 2024-08-19T12:46:59.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 10 client.164103.objecter resend_mon_ops 2024-08-19T12:46:59.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3261718662 0 0) 0x7fe1e8009e50 con 0x7fe1e8009110 2024-08-19T12:46:59.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 10 monclient(hunting): my global_id is 164121 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1e80065c0 con 0x7fe1e8009110 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3992562690 0 0) 0x7fe1e80065c0 con 0x7fe1e8009110 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1e8009e50 con 0x7fe1e8009110 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1fc005cf0 con 0x7fe1e8009110 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 10 monclient: dump: 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.597 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.598 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.598 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.598 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.598 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.598 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.598 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.598 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.598 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.598 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.598 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3150052748 0 0) 0x7fe1e8009e50 con 0x7fe1e8009110 2024-08-19T12:46:59.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe1e800c3f0 con 0x7fe1e8009110 2024-08-19T12:46:59.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 2889026118) 0x7fe1fc0072a0 con 0x7fe1e8009110 2024-08-19T12:46:59.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:46:59.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:46:59.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.26:6792/0 conn(0x7fe1e8009110 legacy=0x7fe1e80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.577+0000 7fe1f6ffd640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.26:6792/0 conn(0x7fe1e8009110 legacy=0x7fe1e80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:59.599 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:46:59.599 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:46:59.599 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:46:59.599 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 3131, 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:46:59.600 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.601 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:46:59.601 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:46:59.601 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:46:59.601 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:46:59.601 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.601 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.601 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.601 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.601 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.601 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.601 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.601 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.602 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.602 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.602 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.602 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.602 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:46:59.602 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:46:59.602 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.602 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.602 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:46:59.603 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:46:59.604 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:46:59.605 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:46:59.605 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.605 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.605 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.605 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.605 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.605 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.605 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.606 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.606 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.606 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.606 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.606 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.606 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:46:59.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.606 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:46:59.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.606 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:46:59.606 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:46:59.606 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:46:59.607 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.607 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:46:59.607 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:46:59.607 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:46:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:46:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:46:59.608 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:46:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:46:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.609 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:46:59.610 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:46:59.610 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.610 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:46:59.610 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.610 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.610 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:46:59.610 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:46:59.611 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.612 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.612 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:46:59.612 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:46:59.612 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.612 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.612 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.613 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:46:59.613 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:46:59.613 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:46:59.614 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.614 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:46:59.614 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:46:59.614 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.614 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.614 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.615 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:46:59.616 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:46:59.616 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.616 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.616 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.616 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:46:59.616 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:46:59.616 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.617 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.617 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:46:59.617 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.617 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.617 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.617 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:46:59.618 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:46:59.618 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.618 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.618 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.618 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:46:59.618 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:46:59.619 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.619 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.619 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:46:59.619 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:46:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:46:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:46:59.621 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:46:59.622 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:46:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:46:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.623 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:46:59.623 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:46:59.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 10 librados: watch_flush enter 2024-08-19T12:46:59.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 10 librados: watch_flush exit 2024-08-19T12:46:59.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 20 client.164103.objecter shutdown clearing up homeless session... 2024-08-19T12:46:59.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 10 client.164103.objecter successfully canceled tick 2024-08-19T12:46:59.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.104:6800/899887506 conn(0x7fe1e4041800 legacy=0x7fe1e4043cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 10 monclient: shutdown 2024-08-19T12:46:59.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:59.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 1 -- 172.21.15.26:0/3508469866 >> v1:172.21.15.26:6792/0 conn(0x7fe2000a5d50 legacy=0x7fe2001d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe2068d7640 1 -- 172.21.15.26:0/3508469866 reap_dead start 2024-08-19T12:46:59.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe2068d7640 1 -- 172.21.15.26:0/3508469866 reap_dead start 2024-08-19T12:46:59.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 1 -- 172.21.15.26:0/3508469866 shutdown_connections 2024-08-19T12:46:59.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 1 -- 172.21.15.26:0/3508469866 >> 172.21.15.26:0/3508469866 conn(0x7fe2000a8740 msgr2=0x7fe2000aa000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:59.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 1 -- 172.21.15.26:0/3508469866 shutdown_connections 2024-08-19T12:46:59.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 1 -- 172.21.15.26:0/3508469866 wait complete. 2024-08-19T12:46:59.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.578+0000 7fe20833a640 1 librados: shutdown 2024-08-19T12:46:59.627 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:46:59.627 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:46:59.627 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:46:59.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.726+0000 7f93b2820640 10 monclient: get_monmap_and_config 2024-08-19T12:46:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.726+0000 7f93b2820640 10 monclient: build_initial_monmap 2024-08-19T12:46:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.726+0000 7f93b2820640 10 monclient: monmap: 2024-08-19T12:46:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:46:59.726874+0000 2024-08-19T12:46:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:46:59.726874+0000 2024-08-19T12:46:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:59.727 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:59.728 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:59.728 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.726+0000 7f93b2820640 1 Processor -- start 2024-08-19T12:46:59.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.726+0000 7f93b2820640 1 -- start start 2024-08-19T12:46:59.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.726+0000 7f93b2820640 10 monclient: init 2024-08-19T12:46:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.727+0000 7f93b2820640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.727+0000 7f93b2820640 10 monclient: _add_conns ranks=[6,1,5,2,0,3,7,8,4] 2024-08-19T12:46:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.727+0000 7f93b2820640 10 monclient(hunting): picked mon.noname-d con 0x7f93ac06e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:46:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.727+0000 7f93b2820640 10 monclient(hunting): picked mon.noname-f con 0x7f93ac0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:46:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.727+0000 7f93b2820640 10 monclient(hunting): picked mon.noname-h con 0x7f93ac0a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:46:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.727+0000 7f93b2820640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93ac136790 con 0x7f93ac0a5920 2024-08-19T12:46:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.727+0000 7f93b2820640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93ac09ddf0 con 0x7f93ac0a5d50 2024-08-19T12:46:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.727+0000 7f93b2820640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93ac09e0d0 con 0x7f93ac06e7b0 2024-08-19T12:46:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.727+0000 7f93b2820640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:59.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.727+0000 7f93b2820640 10 monclient(hunting): authenticate will time out at 1213.814297s 2024-08-19T12:46:59.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.727+0000 7f93abfff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f93ac06e7b0 0x7f93ac136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:43070/0 (socket says 172.21.15.26:43070) 2024-08-19T12:46:59.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.727+0000 7f93abfff640 1 -- 172.21.15.26:0/1796998078 learned_addr learned my addr 172.21.15.26:0/1796998078 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:59.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93abfff640 1 -- 172.21.15.26:0/1796998078 >> v1:172.21.15.26:6792/0 conn(0x7f93ac06e7b0 legacy=0x7f93ac136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93b0dbd640 1 -- 172.21.15.26:0/1796998078 >> v1:172.21.15.104:6791/0 conn(0x7f93ac0a5d50 legacy=0x7f93ac141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93ab7fe640 1 -- 172.21.15.26:0/1796998078 >> v1:172.21.15.104:6789/0 conn(0x7f93ac0a5920 legacy=0x7f93ac13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1659921760 0 0) 0x7f93ac09e0d0 con 0x7f93ac06e7b0 2024-08-19T12:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 10 monclient(hunting): my global_id is 164130 2024-08-19T12:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9394003520 con 0x7f93ac06e7b0 2024-08-19T12:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2715606152 0 0) 0x7f93a4001820 con 0x7f93ac0a5d50 2024-08-19T12:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 10 monclient(hunting): my global_id is 26492 2024-08-19T12:46:59.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93ac09e0d0 con 0x7f93ac0a5d50 2024-08-19T12:46:59.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2084844751 0 0) 0x7f93ac136790 con 0x7f93ac0a5920 2024-08-19T12:46:59.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 10 monclient(hunting): my global_id is 27262 2024-08-19T12:46:59.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.729+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93a4001820 con 0x7f93ac0a5920 2024-08-19T12:46:59.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1057515536 0 0) 0x7f9394003520 con 0x7f93ac06e7b0 2024-08-19T12:46:59.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 >> v1:172.21.15.104:6791/0 conn(0x7f93ac0a5d50 legacy=0x7f93ac141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 >> v1:172.21.15.104:6789/0 conn(0x7f93ac0a5920 legacy=0x7f93ac13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 10 monclient: _finish_hunting 0 2024-08-19T12:46:59.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 1 monclient: found mon.noname-d 2024-08-19T12:46:59.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:59.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:46:59.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ac09e3b0 con 0x7f93ac06e7b0 2024-08-19T12:46:59.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 10 monclient: _finish_auth 0 2024-08-19T12:46:59.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 10 monclient: _check_auth_tickets 2024-08-19T12:46:59.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:59.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93b2820640 5 monclient: authenticate success, global_id 164130 2024-08-19T12:46:59.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f939c001ae0 con 0x7f93ac06e7b0 2024-08-19T12:46:59.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93b2820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 10 monclient: dump: 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.745 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.746 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.746 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.746 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.746 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.746 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93b2820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:46:59.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f939c001d30 con 0x7f93ac06e7b0 2024-08-19T12:46:59.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:59.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93b2820640 10 monclient: get_monmap_and_config success 2024-08-19T12:46:59.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 1 -- 172.21.15.26:0/1796998078 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f939c004be0 con 0x7f93ac06e7b0 2024-08-19T12:46:59.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93b2820640 4 set_mon_vals no callback set 2024-08-19T12:46:59.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:46:59.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93aaffd640 10 monclient: dump: 2024-08-19T12:46:59.747 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.747 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.747 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.747 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.747 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.748 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.748 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.748 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.748 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.748 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.748 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93b2820640 10 monclient: shutdown 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93b2820640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93b2820640 1 -- 172.21.15.26:0/1796998078 >> v1:172.21.15.26:6792/0 conn(0x7f93ac06e7b0 legacy=0x7f93ac136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93b2820640 1 -- 172.21.15.26:0/1796998078 shutdown_connections 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93b2820640 1 -- 172.21.15.26:0/1796998078 >> 172.21.15.26:0/1796998078 conn(0x7f93ac0a8740 msgr2=0x7f93ac0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93b2820640 1 -- 172.21.15.26:0/1796998078 shutdown_connections 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.730+0000 7f93b2820640 1 -- 172.21.15.26:0/1796998078 wait complete. 2024-08-19T12:46:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.731+0000 7f93b2820640 10 monclient: build_initial_monmap 2024-08-19T12:46:59.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.731+0000 7f93b2820640 10 monclient: monmap: 2024-08-19T12:46:59.750 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:59.750 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:46:59.750 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:46:59.750 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:46:59.750 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:59.750 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.750 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:59.750 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:59.750 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:59.750 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:59.750 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:59.751 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.731+0000 7f93b2820640 1 librados: starting msgr at 2024-08-19T12:46:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.731+0000 7f93b2820640 1 librados: starting objecter 2024-08-19T12:46:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.731+0000 7f93b2820640 1 Processor -- start 2024-08-19T12:46:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.731+0000 7f93b2820640 1 -- start start 2024-08-19T12:46:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.731+0000 7f93b2820640 1 librados: setting wanted keys 2024-08-19T12:46:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.731+0000 7f93b2820640 1 librados: calling monclient init 2024-08-19T12:46:59.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.731+0000 7f93b2820640 10 monclient: init 2024-08-19T12:46:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.732+0000 7f93b2820640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.732+0000 7f93b2820640 10 monclient: _add_conns ranks=[5,2,0,4,6,7,8,3,1] 2024-08-19T12:46:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.732+0000 7f93b2820640 10 monclient(hunting): picked mon.noname-h con 0x7f93ac06e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:46:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.732+0000 7f93b2820640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac0a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:46:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.732+0000 7f93b2820640 10 monclient(hunting): picked mon.noname-a con 0x7f93ac0a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:46:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.732+0000 7f93b2820640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93ac13aa00 con 0x7f93ac0a5d50 2024-08-19T12:46:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.732+0000 7f93b2820640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93ac1346c0 con 0x7f93ac0a5920 2024-08-19T12:46:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.732+0000 7f93b2820640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93ac139200 con 0x7f93ac06e7b0 2024-08-19T12:46:59.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.732+0000 7f93b2820640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:59.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.732+0000 7f93b2820640 10 monclient(hunting): authenticate will time out at 1213.819257s 2024-08-19T12:46:59.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.732+0000 7f93ab7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f93ac0a5920 0x7f93ac1d34f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38984/0 (socket says 172.21.15.26:38984) 2024-08-19T12:46:59.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.732+0000 7f93ab7fe640 1 -- 172.21.15.26:0/905826216 learned_addr learned my addr 172.21.15.26:0/905826216 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:59.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93ab7fe640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.26:6790/0 conn(0x7f93ac0a5920 legacy=0x7f93ac1d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93b0dbd640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.26:6789/0 conn(0x7f93ac0a5d50 legacy=0x7f93ac1d6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 client.?.objecter ms_handle_connect 0x7f93ac0a5920 2024-08-19T12:46:59.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:59.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 client.?.objecter ms_handle_connect 0x7f93ac0a5d50 2024-08-19T12:46:59.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:59.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 client.?.objecter ms_handle_connect 0x7f93ac06e7b0 2024-08-19T12:46:59.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93abfff640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.104:6791/0 conn(0x7f93ac06e7b0 legacy=0x7f93ac1ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2381928285 0 0) 0x7f93ac1346c0 con 0x7f93ac0a5920 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): my global_id is 26498 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9398004eb0 con 0x7f93ac0a5920 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2018104071 0 0) 0x7f93ac13aa00 con 0x7f93ac0a5d50 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): my global_id is 144792 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93ac1346c0 con 0x7f93ac0a5d50 2024-08-19T12:46:59.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1449442104 0 0) 0x7f93ac139200 con 0x7f93ac06e7b0 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): my global_id is 26501 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93ac13aa00 con 0x7f93ac06e7b0 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1910513893 0 0) 0x7f9398004eb0 con 0x7f93ac0a5920 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93ac139200 con 0x7f93ac0a5920 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f93a0003610 con 0x7f93ac0a5920 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): dump: 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.755 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.756 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.756 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.757 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.757 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.757 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.757 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.757 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.757 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.757 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.757 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 448414993 0 0) 0x7f93ac1346c0 con 0x7f93ac0a5d50 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9398004eb0 con 0x7f93ac0a5d50 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f93a4003cd0 con 0x7f93ac0a5d50 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.734+0000 7f93a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient(hunting): dump: 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.758 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1359975824 0 0) 0x7f93ac13aa00 con 0x7f93ac06e7b0 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93ac1346c0 con 0x7f93ac06e7b0 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f939c004b20 con 0x7f93ac06e7b0 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient(hunting): dump: 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.761 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.761 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.761 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.761 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.761 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.761 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.761 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3653760848 0 0) 0x7f93ac139200 con 0x7f93ac0a5920 2024-08-19T12:46:59.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.104:6791/0 conn(0x7f93ac06e7b0 legacy=0x7f93ac1ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.26:6789/0 conn(0x7f93ac0a5d50 legacy=0x7f93ac1d6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:46:59.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 1 monclient: found mon.b 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ac13b560 con 0x7f93ac0a5920 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient: _finish_auth 0 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93b2820640 5 monclient: authenticate success, global_id 26498 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93b2820640 10 monclient: _renew_subs 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93b2820640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93b2820640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f93ac13acc0 con 0x7f93ac0a5920 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93b2820640 10 client.26498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93b2820640 10 monclient: _renew_subs 2024-08-19T12:46:59.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93b2820640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93b2820640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f93ac13bf90 con 0x7f93ac0a5920 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93b2820640 1 librados: init done 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93a0003b60 con 0x7f93ac0a5920 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f93a00040d0 con 0x7f93ac0a5920 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93aa7fc640 4 set_mon_vals no callback set 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.735+0000 7f93a8ff9640 10 monclient: dump: 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.763 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.736+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f93a0004270 con 0x7f93ac0a5920 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.736+0000 7f93b2820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:59.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.736+0000 7f93b2820640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f938c0065c0 con 0x7f938c002860 2024-08-19T12:46:59.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.736+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(350..350 src has 1..350) ==== 5346+0+0 (unknown 4082435385 0 0) 0x7f93a004e230 con 0x7f93ac0a5920 2024-08-19T12:46:59.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.736+0000 7f93a8ff9640 10 client.26498.objecter ms_dispatch 0x7f93ac000b90 osd_map(350..350 src has 1..350) 2024-08-19T12:46:59.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.736+0000 7f93a8ff9640 3 client.26498.objecter handle_osd_map got epochs [350,350] > 0 2024-08-19T12:46:59.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.736+0000 7f93a8ff9640 3 client.26498.objecter handle_osd_map decoding full epoch 350 2024-08-19T12:46:59.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.736+0000 7f93b0dbd640 10 monclient: get_auth_request con 0x7f93980419d0 auth_method 0 2024-08-19T12:46:59.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.736+0000 7f93a8ff9640 20 client.26498.objecter dump_active .. 0 homeless 2024-08-19T12:46:59.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93abfff640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.26:6793/0 conn(0x7f938c002860 legacy=0x7f938c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 10 client.26498.objecter ms_handle_connect 0x7f938c002860 2024-08-19T12:46:59.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 10 client.26498.objecter resend_mon_ops 2024-08-19T12:46:59.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 918650813 0 0) 0x7f938c0065c0 con 0x7f938c002860 2024-08-19T12:46:59.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 10 monclient(hunting): my global_id is 26351 2024-08-19T12:46:59.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93ac139200 con 0x7f938c002860 2024-08-19T12:46:59.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2549320437 0 0) 0x7f93ac139200 con 0x7f938c002860 2024-08-19T12:46:59.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f938c0065c0 con 0x7f938c002860 2024-08-19T12:46:59.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 10 client.26498.objecter ms_handle_connect 0x7f93980419d0 2024-08-19T12:46:59.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f939c001d30 con 0x7f938c002860 2024-08-19T12:46:59.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.738+0000 7f93a8ff9640 10 monclient: dump: 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.767 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.768 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.768 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.768 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.739+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3000024687 0 0) 0x7f938c0065c0 con 0x7f938c002860 2024-08-19T12:46:59.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.739+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f938c00a160 con 0x7f938c002860 2024-08-19T12:46:59.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.739+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f939c0052f0 con 0x7f938c002860 2024-08-19T12:46:59.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.739+0000 7f93a8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:59.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.739+0000 7f93a8ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:46:59.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.739+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.26:6793/0 conn(0x7f938c002860 legacy=0x7f938c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.739+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.26:6793/0 conn(0x7f938c002860 legacy=0x7f938c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:59.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.745+0000 7f93b2820640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:46:59.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.745+0000 7f93b2820640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f938c009e70 con 0x7f938c009130 2024-08-19T12:46:59.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.747+0000 7f93abfff640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.26:6793/0 conn(0x7f938c009130 legacy=0x7f938c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.747+0000 7f93a8ff9640 10 client.26498.objecter ms_handle_connect 0x7f938c009130 2024-08-19T12:46:59.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.747+0000 7f93a8ff9640 10 client.26498.objecter resend_mon_ops 2024-08-19T12:46:59.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2830413783 0 0) 0x7f938c009e70 con 0x7f938c009130 2024-08-19T12:46:59.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 10 monclient(hunting): my global_id is 26360 2024-08-19T12:46:59.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f938c0065c0 con 0x7f938c009130 2024-08-19T12:46:59.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1858667793 0 0) 0x7f938c0065c0 con 0x7f938c009130 2024-08-19T12:46:59.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f938c009e70 con 0x7f938c009130 2024-08-19T12:46:59.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f939c005a50 con 0x7f938c009130 2024-08-19T12:46:59.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:46:59.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 10 monclient: dump: 2024-08-19T12:46:59.772 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.773 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.774 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.774 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2375155511 0 0) 0x7f938c009e70 con 0x7f938c009130 2024-08-19T12:46:59.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f938c00c410 con 0x7f938c009130 2024-08-19T12:46:59.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3205020622) 0x7f939c007e80 con 0x7f938c009130 2024-08-19T12:46:59.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:46:59.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:46:59.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.26:6793/0 conn(0x7f938c009130 legacy=0x7f938c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93a8ff9640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.26:6793/0 conn(0x7f938c009130 legacy=0x7f938c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:59.775 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:46:59.775 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:46:59.775 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:46:59.775 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:46:59.775 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:46:59.775 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 532765, 2024-08-19T12:46:59.775 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:46:59.775 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:46:59.775 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:46:59.775 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.777 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:46:59.778 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:46:59.779 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:46:59.780 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.780 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.780 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.780 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.780 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.780 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.780 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.780 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.780 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.780 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.780 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.780 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.781 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.782 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.784 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:46:59.784 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:46:59.784 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.784 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.784 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.784 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:46:59.784 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.784 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.785 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.786 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.786 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:46:59.786 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.786 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.786 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.786 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:46:59.786 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:46:59.786 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.787 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.790 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:46:59.790 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:46:59.790 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.790 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:46:59.790 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.792 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.793 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.794 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.794 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.794 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.794 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:46:59.794 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:46:59.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.794 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:59.794 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:59.794 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:46:59.794 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.794 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.795 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:46:59.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.795 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:59.795 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:59.795 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:46:59.795 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.795 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.795 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:46:59.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.795 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:59.795 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:59.795 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:46:59.796 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.796 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.796 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:46:59.796 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:46:59.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93b2820640 10 librados: watch_flush enter 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93b2820640 10 librados: watch_flush exit 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93b2820640 20 client.26498.objecter shutdown clearing up homeless session... 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.748+0000 7f93b2820640 10 client.26498.objecter successfully canceled tick 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.749+0000 7f93b2820640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.104:6800/899887506 conn(0x7f93980419d0 legacy=0x7f9398043e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.749+0000 7f93b2820640 10 monclient: shutdown 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.749+0000 7f93b2820640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.749+0000 7f93b2820640 1 -- 172.21.15.26:0/905826216 >> v1:172.21.15.26:6790/0 conn(0x7f93ac0a5920 legacy=0x7f93ac1d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.749+0000 7f93b0dbd640 1 -- 172.21.15.26:0/905826216 reap_dead start 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.749+0000 7f93b0dbd640 1 -- 172.21.15.26:0/905826216 reap_dead start 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.749+0000 7f93b2820640 1 -- 172.21.15.26:0/905826216 shutdown_connections 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.749+0000 7f93b2820640 1 -- 172.21.15.26:0/905826216 >> 172.21.15.26:0/905826216 conn(0x7f93ac0a8740 msgr2=0x7f93ac135200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.749+0000 7f93b2820640 1 -- 172.21.15.26:0/905826216 shutdown_connections 2024-08-19T12:46:59.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.749+0000 7f93b2820640 1 -- 172.21.15.26:0/905826216 wait complete. 2024-08-19T12:46:59.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.749+0000 7f93b2820640 1 librados: shutdown 2024-08-19T12:46:59.799 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:46:59.799 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:46:59.799 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:46:59.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.898+0000 7f400e14c640 10 monclient: get_monmap_and_config 2024-08-19T12:46:59.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.898+0000 7f400e14c640 10 monclient: build_initial_monmap 2024-08-19T12:46:59.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.898+0000 7f400e14c640 10 monclient: monmap: 2024-08-19T12:46:59.899 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:59.899 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.899 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:46:59.899381+0000 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:46:59.899381+0000 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:59.900 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 1 Processor -- start 2024-08-19T12:46:59.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 1 -- start start 2024-08-19T12:46:59.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 10 monclient: init 2024-08-19T12:46:59.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:59.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 10 monclient: _add_conns ranks=[4,1,7,3,5,6,2,8,0] 2024-08-19T12:46:59.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 10 monclient(hunting): picked mon.noname-c con 0x7f400806e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:46:59.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 10 monclient(hunting): picked mon.noname-f con 0x7f40080a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:46:59.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 10 monclient(hunting): picked mon.noname-i con 0x7f40080a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:46:59.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4008136790 con 0x7f40080a5920 2024-08-19T12:46:59.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f400809ddf0 con 0x7f400806e7b0 2024-08-19T12:46:59.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f400809e0d0 con 0x7f40080a5d50 2024-08-19T12:46:59.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:59.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.899+0000 7f400e14c640 10 monclient(hunting): authenticate will time out at 1213.986999s 2024-08-19T12:46:59.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.900+0000 7f40077fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f400806e7b0 0x7f4008136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:34628/0 (socket says 172.21.15.26:34628) 2024-08-19T12:46:59.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.900+0000 7f40077fe640 1 -- 172.21.15.26:0/3081223343 learned_addr learned my addr 172.21.15.26:0/3081223343 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:59.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f40077fe640 1 -- 172.21.15.26:0/3081223343 >> v1:172.21.15.26:6791/0 conn(0x7f400806e7b0 legacy=0x7f4008136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 1 -- 172.21.15.26:0/3081223343 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 298636905 0 0) 0x7f3ff8001450 con 0x7f400806e7b0 2024-08-19T12:46:59.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient(hunting): my global_id is 26833 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 1 -- 172.21.15.26:0/3081223343 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ff4003520 con 0x7f400806e7b0 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f3ffffff640 1 -- 172.21.15.26:0/3081223343 >> v1:172.21.15.104:6789/0 conn(0x7f40080a5920 legacy=0x7f400813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4007fff640 1 -- 172.21.15.26:0/3081223343 >> v1:172.21.15.104:6792/0 conn(0x7f40080a5d50 legacy=0x7f4008141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 1 -- 172.21.15.26:0/3081223343 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 791776339 0 0) 0x7f3ff4003520 con 0x7f400806e7b0 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 1 -- 172.21.15.26:0/3081223343 >> v1:172.21.15.104:6792/0 conn(0x7f40080a5d50 legacy=0x7f4008141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 1 -- 172.21.15.26:0/3081223343 >> v1:172.21.15.104:6789/0 conn(0x7f40080a5920 legacy=0x7f400813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 1 monclient: found mon.noname-c 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:59.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 1 -- 172.21.15.26:0/3081223343 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400809e3b0 con 0x7f400806e7b0 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient: _finish_auth 0 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 1 -- 172.21.15.26:0/3081223343 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3ff8001cb0 con 0x7f400806e7b0 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f400e14c640 5 monclient: authenticate success, global_id 26833 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f400e14c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient: dump: 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.912 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.913 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.913 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.913 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.913 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.913 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.913 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.913 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.913 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.913 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.913 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f400e14c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 1 -- 172.21.15.26:0/3081223343 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ff8004990 con 0x7f400806e7b0 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 1 -- 172.21.15.26:0/3081223343 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3ff8004f00 con 0x7f400806e7b0 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f400e14c640 10 monclient: get_monmap_and_config success 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f400e14c640 4 set_mon_vals no callback set 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.902+0000 7f4006ffd640 10 monclient: dump: 2024-08-19T12:46:59.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.915 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.915 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.915 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.915 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.915 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.915 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.915 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.915 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.915 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.915 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.915 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.915 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.903+0000 7f400e14c640 10 monclient: shutdown 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.903+0000 7f400e14c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.903+0000 7f400e14c640 1 -- 172.21.15.26:0/3081223343 >> v1:172.21.15.26:6791/0 conn(0x7f400806e7b0 legacy=0x7f4008136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.903+0000 7f400e14c640 1 -- 172.21.15.26:0/3081223343 shutdown_connections 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.903+0000 7f400e14c640 1 -- 172.21.15.26:0/3081223343 >> 172.21.15.26:0/3081223343 conn(0x7f40080a8740 msgr2=0x7f40080a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.903+0000 7f400e14c640 1 -- 172.21.15.26:0/3081223343 shutdown_connections 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.903+0000 7f400e14c640 1 -- 172.21.15.26:0/3081223343 wait complete. 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.903+0000 7f400e14c640 10 monclient: build_initial_monmap 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.903+0000 7f400e14c640 10 monclient: monmap: 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:46:59.916 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:46:59.917 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 1 librados: starting msgr at 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 1 librados: starting objecter 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 1 Processor -- start 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 1 -- start start 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 1 librados: setting wanted keys 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 1 librados: calling monclient init 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 10 monclient: init 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 10 monclient: _reopen_session rank -1 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 10 monclient: _add_conns ranks=[1,0,4,5,2,8,7,3,6] 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 10 monclient(hunting): picked mon.noname-f con 0x7f400806e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 10 monclient(hunting): picked mon.noname-a con 0x7f40080a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 10 monclient(hunting): picked mon.noname-c con 0x7f40080a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:46:59.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40080a50a0 con 0x7f40080a5920 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40080a1990 con 0x7f400806e7b0 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4008134a30 con 0x7f40080a5d50 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 10 monclient(hunting): _renew_subs 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f400e14c640 10 monclient(hunting): authenticate will time out at 1213.991861s 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f4007fff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f40080a5d50 0x7f40081d6bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:34640/0 (socket says 172.21.15.26:34640) 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f4007fff640 1 -- 172.21.15.26:0/614865707 learned_addr learned my addr 172.21.15.26:0/614865707 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.904+0000 7f3ffffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f40080a5920 0x7f40081d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47924/0 (socket says 172.21.15.26:47924) 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f3ffffff640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.26:6789/0 conn(0x7f40080a5920 legacy=0x7f40081d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 client.?.objecter ms_handle_connect 0x7f40080a5920 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 client.?.objecter ms_handle_connect 0x7f40080a5d50 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:59.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4007fff640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.26:6791/0 conn(0x7f40080a5d50 legacy=0x7f40081d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2418264560 0 0) 0x7f40080a50a0 con 0x7f40080a5920 2024-08-19T12:46:59.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): my global_id is 144801 2024-08-19T12:46:59.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fe8004eb0 con 0x7f40080a5920 2024-08-19T12:46:59.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f40077fe640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.104:6789/0 conn(0x7f400806e7b0 legacy=0x7f40081ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 client.?.objecter ms_handle_connect 0x7f400806e7b0 2024-08-19T12:46:59.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:46:59.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3367760162 0 0) 0x7f4008134a30 con 0x7f40080a5d50 2024-08-19T12:46:59.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): my global_id is 26842 2024-08-19T12:46:59.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40080a50a0 con 0x7f40080a5d50 2024-08-19T12:46:59.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3592468655 0 0) 0x7f40080a1990 con 0x7f400806e7b0 2024-08-19T12:46:59.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): my global_id is 27280 2024-08-19T12:46:59.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4008134a30 con 0x7f400806e7b0 2024-08-19T12:46:59.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 990845972 0 0) 0x7f3fe8004eb0 con 0x7f40080a5920 2024-08-19T12:46:59.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40080a1990 con 0x7f40080a5920 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3ff0003610 con 0x7f40080a5920 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): dump: 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.923 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2562190903 0 0) 0x7f40080a50a0 con 0x7f40080a5d50 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3fe8004eb0 con 0x7f40080a5d50 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4000002050 con 0x7f40080a5d50 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:46:59.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient(hunting): dump: 2024-08-19T12:46:59.925 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.925 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.925 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.925 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.925 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.925 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.925 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.925 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.925 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.925 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.925 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.926 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.926 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.926 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.926 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.926 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3390262324 0 0) 0x7f40080a1990 con 0x7f40080a5920 2024-08-19T12:46:59.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.26:6791/0 conn(0x7f40080a5d50 legacy=0x7f40081d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.104:6789/0 conn(0x7f400806e7b0 legacy=0x7f40081ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:46:59.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 monclient: found mon.a 2024-08-19T12:46:59.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40080a0eb0 con 0x7f40080a5920 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient: _finish_auth 0 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f400e14c640 5 monclient: authenticate success, global_id 144801 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f400e14c640 10 monclient: _renew_subs 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f400e14c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f400e14c640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f40080a28f0 con 0x7f40080a5920 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f400e14c640 10 client.144801.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f400e14c640 10 monclient: _renew_subs 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f400e14c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:46:59.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f400e14c640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f40080a07f0 con 0x7f40080a5920 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f400e14c640 1 librados: init done 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ff0003a80 con 0x7f40080a5920 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3ff0003fb0 con 0x7f40080a5920 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f40067fc640 4 set_mon_vals no callback set 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.907+0000 7f4004ff9640 10 monclient: dump: 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.908+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3ff0004150 con 0x7f40080a5920 2024-08-19T12:46:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.908+0000 7f400e14c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:59.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.908+0000 7f400e14c640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3fec0065c0 con 0x7f3fec002860 2024-08-19T12:46:59.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.908+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(350..350 src has 1..350) ==== 5346+0+0 (unknown 4082435385 0 0) 0x7f3ff004e1f0 con 0x7f40080a5920 2024-08-19T12:46:59.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.908+0000 7f4004ff9640 10 client.144801.objecter ms_dispatch 0x7f4008000b90 osd_map(350..350 src has 1..350) 2024-08-19T12:46:59.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.908+0000 7f4004ff9640 3 client.144801.objecter handle_osd_map got epochs [350,350] > 0 2024-08-19T12:46:59.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.908+0000 7f4004ff9640 3 client.144801.objecter handle_osd_map decoding full epoch 350 2024-08-19T12:46:59.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.909+0000 7f4004ff9640 20 client.144801.objecter dump_active .. 0 homeless 2024-08-19T12:46:59.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.909+0000 7f4007fff640 10 monclient: get_auth_request con 0x7f3fe80414d0 auth_method 0 2024-08-19T12:46:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f40077fe640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.104:6789/0 conn(0x7f3fec002860 legacy=0x7f3fec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 10 client.144801.objecter ms_handle_connect 0x7f3fec002860 2024-08-19T12:46:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 10 client.144801.objecter resend_mon_ops 2024-08-19T12:46:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1731516116 0 0) 0x7f3fec0065c0 con 0x7f3fec002860 2024-08-19T12:46:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 10 monclient(hunting): my global_id is 27289 2024-08-19T12:46:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40080a1990 con 0x7f3fec002860 2024-08-19T12:46:59.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 10 client.144801.objecter ms_handle_connect 0x7f3fe80414d0 2024-08-19T12:46:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3538323877 0 0) 0x7f40080a1990 con 0x7f3fec002860 2024-08-19T12:46:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3fec0065c0 con 0x7f3fec002860 2024-08-19T12:46:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3ff8004ec0 con 0x7f3fec002860 2024-08-19T12:46:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:46:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 10 monclient: dump: 2024-08-19T12:46:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.934 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.935 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.937 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1264645016 0 0) 0x7f3fec0065c0 con 0x7f3fec002860 2024-08-19T12:46:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.911+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3fec00a160 con 0x7f3fec002860 2024-08-19T12:46:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.912+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3ff8005240 con 0x7f3fec002860 2024-08-19T12:46:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.912+0000 7f4004ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:46:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.912+0000 7f4004ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:46:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.912+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.104:6789/0 conn(0x7f3fec002860 legacy=0x7f3fec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.912+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.104:6789/0 conn(0x7f3fec002860 legacy=0x7f3fec004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.918+0000 7f400e14c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:46:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.918+0000 7f400e14c640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3fec009e50 con 0x7f3fec009110 2024-08-19T12:46:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f40077fe640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.104:6789/0 conn(0x7f3fec009110 legacy=0x7f3fec0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:46:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 10 client.144801.objecter ms_handle_connect 0x7f3fec009110 2024-08-19T12:46:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 10 client.144801.objecter resend_mon_ops 2024-08-19T12:46:59.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1303957881 0 0) 0x7f3fec009e50 con 0x7f3fec009110 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 10 monclient(hunting): my global_id is 27298 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fec0065c0 con 0x7f3fec009110 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3252771781 0 0) 0x7f3fec0065c0 con 0x7f3fec009110 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3fec009e50 con 0x7f3fec009110 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3ff8005b80 con 0x7f3fec009110 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.920+0000 7f4004ff9640 10 monclient: dump: 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:46:59.939 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:46:59.940 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:46:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1176939411 0 0) 0x7f3fec009e50 con 0x7f3fec009110 2024-08-19T12:46:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3fec00c3f0 con 0x7f3fec009110 2024-08-19T12:46:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 402039114) 0x7f3ff80086f0 con 0x7f3fec009110 2024-08-19T12:46:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f4004ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:46:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f4004ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:46:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.104:6789/0 conn(0x7f3fec009110 legacy=0x7f3fec0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f4004ff9640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.104:6789/0 conn(0x7f3fec009110 legacy=0x7f3fec0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:46:59.941 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:46:59.941 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:46:59.941 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 532925, 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:46:59.942 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:46:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:46:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:46:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:46:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.943 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:46:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:46:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:46:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:46:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:46:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:46:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:46:59.946 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:46:59.949 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:46:59.949 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.949 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.949 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.949 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:46:59.949 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:46:59.949 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.949 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.949 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.950 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:46:59.951 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.952 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.952 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.952 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:46:59.952 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:46:59.952 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:46:59.952 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:46:59.953 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.954 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:46:59.957 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.958 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:59.960 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:46:59.961 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:46:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:46:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:46:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:46:59.962 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:46:59.962 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:46:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 10 librados: watch_flush enter 2024-08-19T12:46:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 10 librados: watch_flush exit 2024-08-19T12:46:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 20 client.144801.objecter shutdown clearing up homeless session... 2024-08-19T12:46:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 10 client.144801.objecter successfully canceled tick 2024-08-19T12:46:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.104:6800/899887506 conn(0x7f3fe80414d0 legacy=0x7f3fe8043990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 10 monclient: shutdown 2024-08-19T12:46:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:46:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 1 -- 172.21.15.26:0/614865707 >> v1:172.21.15.26:6789/0 conn(0x7f40080a5920 legacy=0x7f40081d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:46:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f4007fff640 1 -- 172.21.15.26:0/614865707 reap_dead start 2024-08-19T12:46:59.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f4007fff640 1 -- 172.21.15.26:0/614865707 reap_dead start 2024-08-19T12:46:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 1 -- 172.21.15.26:0/614865707 shutdown_connections 2024-08-19T12:46:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 1 -- 172.21.15.26:0/614865707 >> 172.21.15.26:0/614865707 conn(0x7f40080a8740 msgr2=0x7f400806dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:46:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 1 -- 172.21.15.26:0/614865707 shutdown_connections 2024-08-19T12:46:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 1 -- 172.21.15.26:0/614865707 wait complete. 2024-08-19T12:46:59.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:46:59.921+0000 7f400e14c640 1 librados: shutdown 2024-08-19T12:46:59.965 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:46:59.965 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:46:59.965 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:47:00.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.065+0000 7fd25493a640 10 monclient: get_monmap_and_config 2024-08-19T12:47:00.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.065+0000 7fd25493a640 10 monclient: build_initial_monmap 2024-08-19T12:47:00.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.065+0000 7fd25493a640 10 monclient: monmap: 2024-08-19T12:47:00.066 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:00.066 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.066 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:00.065922+0000 2024-08-19T12:47:00.066 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:00.065922+0000 2024-08-19T12:47:00.066 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:00.066 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:00.067 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.065+0000 7fd25493a640 1 Processor -- start 2024-08-19T12:47:00.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.065+0000 7fd25493a640 1 -- start start 2024-08-19T12:47:00.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.065+0000 7fd25493a640 10 monclient: init 2024-08-19T12:47:00.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.066+0000 7fd25493a640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.066+0000 7fd25493a640 10 monclient: _add_conns ranks=[5,4,1,0,2,7,6,8,3] 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.066+0000 7fd25493a640 10 monclient(hunting): picked mon.noname-h con 0x7fd250073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.066+0000 7fd25493a640 10 monclient(hunting): picked mon.noname-c con 0x7fd2500a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.066+0000 7fd25493a640 10 monclient(hunting): picked mon.noname-f con 0x7fd25013f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.066+0000 7fd25493a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25009dae0 con 0x7fd25013f2e0 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.066+0000 7fd25493a640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd250136910 con 0x7fd2500a57a0 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.066+0000 7fd25493a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25009dd50 con 0x7fd250073470 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.066+0000 7fd25493a640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.066+0000 7fd25493a640 10 monclient(hunting): authenticate will time out at 1214.153361s 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.066+0000 7fd24dd9b640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fd2500a57a0 0x7fd2500a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:34642/0 (socket says 172.21.15.26:34642) 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.066+0000 7fd24dd9b640 1 -- 172.21.15.26:0/598829327 learned_addr learned my addr 172.21.15.26:0/598829327 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.068+0000 7fd24dd9b640 1 -- 172.21.15.26:0/598829327 >> v1:172.21.15.26:6791/0 conn(0x7fd2500a57a0 legacy=0x7fd2500a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.068+0000 7fd24e59c640 1 -- 172.21.15.26:0/598829327 >> v1:172.21.15.104:6791/0 conn(0x7fd250073470 legacy=0x7fd250136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.068+0000 7fd24ed9d640 1 -- 172.21.15.26:0/598829327 >> v1:172.21.15.104:6789/0 conn(0x7fd25013f2e0 legacy=0x7fd2501416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.068+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1764153097 0 0) 0x7fd250136910 con 0x7fd2500a57a0 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.068+0000 7fd24d59a640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.068+0000 7fd24d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.068+0000 7fd24d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.068+0000 7fd24d59a640 10 monclient(hunting): my global_id is 26851 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.068+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd23c003520 con 0x7fd2500a57a0 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3700590175 0 0) 0x7fd25009dae0 con 0x7fd25013f2e0 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient(hunting): my global_id is 27307 2024-08-19T12:47:00.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd250136910 con 0x7fd25013f2e0 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346867675 0 0) 0x7fd25009dd50 con 0x7fd250073470 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient(hunting): my global_id is 26510 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd25009dae0 con 0x7fd250073470 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2801949288 0 0) 0x7fd23c003520 con 0x7fd2500a57a0 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 >> v1:172.21.15.104:6791/0 conn(0x7fd250073470 legacy=0x7fd250136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 >> v1:172.21.15.104:6789/0 conn(0x7fd25013f2e0 legacy=0x7fd2501416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient: _finish_hunting 0 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 1 monclient: found mon.noname-c 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:00.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:47:00.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25009e030 con 0x7fd2500a57a0 2024-08-19T12:47:00.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient: _finish_auth 0 2024-08-19T12:47:00.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient: _check_auth_tickets 2024-08-19T12:47:00.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:00.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd238001ae0 con 0x7fd2500a57a0 2024-08-19T12:47:00.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd25493a640 5 monclient: authenticate success, global_id 26851 2024-08-19T12:47:00.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:47:00.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient: dump: 2024-08-19T12:47:00.077 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.078 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.079 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.079 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.079 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.079 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd25493a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:00.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd2380048a0 con 0x7fd2500a57a0 2024-08-19T12:47:00.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:00.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 1 -- 172.21.15.26:0/598829327 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd238004e10 con 0x7fd2500a57a0 2024-08-19T12:47:00.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:00.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd24d59a640 10 monclient: dump: 2024-08-19T12:47:00.080 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.080 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.080 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.080 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.080 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.080 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.080 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.081 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.081 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.081 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.081 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.081 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.081 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.081 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.081 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.081 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd25493a640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:00.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd25493a640 4 set_mon_vals no callback set 2024-08-19T12:47:00.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd25493a640 10 monclient: shutdown 2024-08-19T12:47:00.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd25493a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:00.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd25493a640 1 -- 172.21.15.26:0/598829327 >> v1:172.21.15.26:6791/0 conn(0x7fd2500a57a0 legacy=0x7fd2500a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd25493a640 1 -- 172.21.15.26:0/598829327 shutdown_connections 2024-08-19T12:47:00.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd25493a640 1 -- 172.21.15.26:0/598829327 >> 172.21.15.26:0/598829327 conn(0x7fd2500abff0 msgr2=0x7fd2500ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:00.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd25493a640 1 -- 172.21.15.26:0/598829327 shutdown_connections 2024-08-19T12:47:00.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.069+0000 7fd25493a640 1 -- 172.21.15.26:0/598829327 wait complete. 2024-08-19T12:47:00.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.070+0000 7fd25493a640 10 monclient: build_initial_monmap 2024-08-19T12:47:00.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.070+0000 7fd25493a640 10 monclient: monmap: 2024-08-19T12:47:00.083 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:00.083 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:00.083 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:00.083 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:00.083 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:00.083 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.083 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:00.083 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:00.083 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:00.084 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.070+0000 7fd25493a640 1 librados: starting msgr at 2024-08-19T12:47:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.070+0000 7fd25493a640 1 librados: starting objecter 2024-08-19T12:47:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.070+0000 7fd25493a640 1 Processor -- start 2024-08-19T12:47:00.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.070+0000 7fd25493a640 1 -- start start 2024-08-19T12:47:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.070+0000 7fd25493a640 1 librados: setting wanted keys 2024-08-19T12:47:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.070+0000 7fd25493a640 1 librados: calling monclient init 2024-08-19T12:47:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.070+0000 7fd25493a640 10 monclient: init 2024-08-19T12:47:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.071+0000 7fd25493a640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.071+0000 7fd25493a640 10 monclient: _add_conns ranks=[1,5,3,4,0,7,2,8,6] 2024-08-19T12:47:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.071+0000 7fd25493a640 10 monclient(hunting): picked mon.noname-f con 0x7fd250073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.071+0000 7fd25493a640 10 monclient(hunting): picked mon.noname-h con 0x7fd2500a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.071+0000 7fd25493a640 10 monclient(hunting): picked mon.noname-g con 0x7fd25013f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.071+0000 7fd25493a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2501371b0 con 0x7fd250073470 2024-08-19T12:47:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.071+0000 7fd25493a640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2500a43c0 con 0x7fd25013f2e0 2024-08-19T12:47:00.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.071+0000 7fd25493a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2500a4660 con 0x7fd2500a57a0 2024-08-19T12:47:00.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.071+0000 7fd25493a640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:00.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.071+0000 7fd25493a640 10 monclient(hunting): authenticate will time out at 1214.158462s 2024-08-19T12:47:00.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.071+0000 7fd24e59c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd250073470 0x7fd2501ce3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:35820/0 (socket says 172.21.15.26:35820) 2024-08-19T12:47:00.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.071+0000 7fd24e59c640 1 -- 172.21.15.26:0/1743895434 learned_addr learned my addr 172.21.15.26:0/1743895434 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd24dd9b640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6791/0 conn(0x7fd2500a57a0 legacy=0x7fd2501d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 10 client.?.objecter ms_handle_connect 0x7fd2500a57a0 2024-08-19T12:47:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:00.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 10 client.?.objecter ms_handle_connect 0x7fd250073470 2024-08-19T12:47:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd24e59c640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6789/0 conn(0x7fd250073470 legacy=0x7fd2501ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 10 client.?.objecter ms_handle_connect 0x7fd25013f2e0 2024-08-19T12:47:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd24ed9d640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6790/0 conn(0x7fd25013f2e0 legacy=0x7fd2501d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4220262388 0 0) 0x7fd2500a4660 con 0x7fd2500a57a0 2024-08-19T12:47:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 10 monclient(hunting): my global_id is 26519 2024-08-19T12:47:00.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd230004eb0 con 0x7fd2500a57a0 2024-08-19T12:47:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.073+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3683439832 0 0) 0x7fd2501371b0 con 0x7fd250073470 2024-08-19T12:47:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): my global_id is 27316 2024-08-19T12:47:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2500a4660 con 0x7fd250073470 2024-08-19T12:47:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3227860497 0 0) 0x7fd2500a43c0 con 0x7fd25013f2e0 2024-08-19T12:47:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): my global_id is 154317 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2501371b0 con 0x7fd25013f2e0 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 302897799 0 0) 0x7fd230004eb0 con 0x7fd2500a57a0 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2500a43c0 con 0x7fd2500a57a0 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd238004d90 con 0x7fd2500a57a0 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): dump: 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.095 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1018041134 0 0) 0x7fd2501371b0 con 0x7fd25013f2e0 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd230004eb0 con 0x7fd25013f2e0 2024-08-19T12:47:00.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2698317819 0 0) 0x7fd2500a4660 con 0x7fd250073470 2024-08-19T12:47:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2501371b0 con 0x7fd250073470 2024-08-19T12:47:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd248001c60 con 0x7fd25013f2e0 2024-08-19T12:47:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:47:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): dump: 2024-08-19T12:47:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.097 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd240003990 con 0x7fd250073470 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:00.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient(hunting): dump: 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.099 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.100 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2765942223 0 0) 0x7fd2500a43c0 con 0x7fd2500a57a0 2024-08-19T12:47:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6790/0 conn(0x7fd25013f2e0 legacy=0x7fd2501d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6789/0 conn(0x7fd250073470 legacy=0x7fd2501ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:47:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 monclient: found mon.h 2024-08-19T12:47:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:00.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:47:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2500a3020 con 0x7fd2500a57a0 2024-08-19T12:47:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient: _finish_auth 0 2024-08-19T12:47:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:47:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd25493a640 5 monclient: authenticate success, global_id 26519 2024-08-19T12:47:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd25493a640 10 monclient: _renew_subs 2024-08-19T12:47:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd25493a640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:47:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd25493a640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd2500a32b0 con 0x7fd2500a57a0 2024-08-19T12:47:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd25493a640 10 client.26519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd25493a640 10 monclient: _renew_subs 2024-08-19T12:47:00.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd25493a640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd25493a640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fd2501cec90 con 0x7fd2500a57a0 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd25493a640 1 librados: init done 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd238001d30 con 0x7fd2500a57a0 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd238001b00 con 0x7fd2500a57a0 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd24cd99640 4 set_mon_vals no callback set 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.074+0000 7fd246ffd640 10 monclient: dump: 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.102 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.103 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.104 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.075+0000 7fd25493a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.075+0000 7fd25493a640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd2340065c0 con 0x7fd234002860 2024-08-19T12:47:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.075+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd238011a00 con 0x7fd2500a57a0 2024-08-19T12:47:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.075+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(350..350 src has 1..350) ==== 5346+0+0 (unknown 4082435385 0 0) 0x7fd23804e230 con 0x7fd2500a57a0 2024-08-19T12:47:00.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.075+0000 7fd246ffd640 10 client.26519.objecter ms_dispatch 0x7fd250000b90 osd_map(350..350 src has 1..350) 2024-08-19T12:47:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.075+0000 7fd246ffd640 3 client.26519.objecter handle_osd_map got epochs [350,350] > 0 2024-08-19T12:47:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.075+0000 7fd246ffd640 3 client.26519.objecter handle_osd_map decoding full epoch 350 2024-08-19T12:47:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.075+0000 7fd246ffd640 20 client.26519.objecter dump_active .. 0 homeless 2024-08-19T12:47:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.075+0000 7fd24ed9d640 10 monclient: get_auth_request con 0x7fd230041a10 auth_method 0 2024-08-19T12:47:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.077+0000 7fd24e59c640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6790/0 conn(0x7fd234002860 legacy=0x7fd234004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.077+0000 7fd246ffd640 10 client.26519.objecter ms_handle_connect 0x7fd234002860 2024-08-19T12:47:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.077+0000 7fd246ffd640 10 client.26519.objecter resend_mon_ops 2024-08-19T12:47:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.077+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3503573945 0 0) 0x7fd2340065c0 con 0x7fd234002860 2024-08-19T12:47:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.077+0000 7fd246ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.077+0000 7fd246ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.077+0000 7fd246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.077+0000 7fd246ffd640 10 monclient(hunting): my global_id is 154326 2024-08-19T12:47:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.077+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2500a43c0 con 0x7fd234002860 2024-08-19T12:47:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3600549545 0 0) 0x7fd2500a43c0 con 0x7fd234002860 2024-08-19T12:47:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2340065c0 con 0x7fd234002860 2024-08-19T12:47:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd240004190 con 0x7fd234002860 2024-08-19T12:47:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:47:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 10 monclient: dump: 2024-08-19T12:47:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.107 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.108 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2451345925 0 0) 0x7fd2340065c0 con 0x7fd234002860 2024-08-19T12:47:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd23400a160 con 0x7fd234002860 2024-08-19T12:47:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 10 client.26519.objecter ms_handle_connect 0x7fd230041a10 2024-08-19T12:47:00.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd240005b40 con 0x7fd234002860 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6790/0 conn(0x7fd234002860 legacy=0x7fd234004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.078+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6790/0 conn(0x7fd234002860 legacy=0x7fd234004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.084+0000 7fd25493a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.084+0000 7fd25493a640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd234009e70 con 0x7fd234009130 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd24e59c640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6790/0 conn(0x7fd234009130 legacy=0x7fd2340095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 10 client.26519.objecter ms_handle_connect 0x7fd234009130 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 10 client.26519.objecter resend_mon_ops 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 445315758 0 0) 0x7fd234009e70 con 0x7fd234009130 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 10 monclient(hunting): my global_id is 154335 2024-08-19T12:47:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2340065c0 con 0x7fd234009130 2024-08-19T12:47:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2784594853 0 0) 0x7fd2340065c0 con 0x7fd234009130 2024-08-19T12:47:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd234009e70 con 0x7fd234009130 2024-08-19T12:47:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd240004740 con 0x7fd234009130 2024-08-19T12:47:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:47:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.087+0000 7fd246ffd640 10 monclient: dump: 2024-08-19T12:47:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.110 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.111 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.112 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 203016315 0 0) 0x7fd234009e70 con 0x7fd234009130 2024-08-19T12:47:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd23400c3f0 con 0x7fd234009130 2024-08-19T12:47:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 641562852) 0x7fd2400080f0 con 0x7fd234009130 2024-08-19T12:47:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd246ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd246ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6790/0 conn(0x7fd234009130 legacy=0x7fd2340095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd246ffd640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6790/0 conn(0x7fd234009130 legacy=0x7fd2340095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:00.113 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:47:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 131817, 2024-08-19T12:47:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:47:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:47:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:47:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:00.131 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.132 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:00.134 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:00.135 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.135 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.135 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:00.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.135 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:00.135 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:00.135 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:00.135 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.135 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.135 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:00.135 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:00.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 10 librados: watch_flush enter 2024-08-19T12:47:00.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 10 librados: watch_flush exit 2024-08-19T12:47:00.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 20 client.26519.objecter shutdown clearing up homeless session... 2024-08-19T12:47:00.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 10 client.26519.objecter successfully canceled tick 2024-08-19T12:47:00.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6800/899887506 conn(0x7fd230041a10 legacy=0x7fd230043ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 10 monclient: shutdown 2024-08-19T12:47:00.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:00.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 1 -- 172.21.15.26:0/1743895434 >> v1:172.21.15.104:6791/0 conn(0x7fd2500a57a0 legacy=0x7fd2501d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd24ed9d640 1 -- 172.21.15.26:0/1743895434 reap_dead start 2024-08-19T12:47:00.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd24ed9d640 1 -- 172.21.15.26:0/1743895434 reap_dead start 2024-08-19T12:47:00.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 1 -- 172.21.15.26:0/1743895434 shutdown_connections 2024-08-19T12:47:00.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 1 -- 172.21.15.26:0/1743895434 >> 172.21.15.26:0/1743895434 conn(0x7fd2500abff0 msgr2=0x7fd2501338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:00.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 1 -- 172.21.15.26:0/1743895434 shutdown_connections 2024-08-19T12:47:00.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 1 -- 172.21.15.26:0/1743895434 wait complete. 2024-08-19T12:47:00.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.088+0000 7fd25493a640 1 librados: shutdown 2024-08-19T12:47:00.139 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:47:00.139 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:47:00.139 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:47:00.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.237+0000 7ff53d343640 10 monclient: get_monmap_and_config 2024-08-19T12:47:00.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.237+0000 7ff53d343640 10 monclient: build_initial_monmap 2024-08-19T12:47:00.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.238+0000 7ff53d343640 10 monclient: monmap: 2024-08-19T12:47:00.239 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:00.240 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.240 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:00.238835+0000 2024-08-19T12:47:00.240 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:00.238835+0000 2024-08-19T12:47:00.240 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:00.240 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.241 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:00.241 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:00.241 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:00.241 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:00.242 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:00.242 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:00.242 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:00.242 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:00.242 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:00.242 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.238+0000 7ff53d343640 1 Processor -- start 2024-08-19T12:47:00.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.238+0000 7ff53d343640 1 -- start start 2024-08-19T12:47:00.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.238+0000 7ff53d343640 10 monclient: init 2024-08-19T12:47:00.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff53d343640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:00.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff53d343640 10 monclient: _add_conns ranks=[3,6,2,7,1,4,5,8,0] 2024-08-19T12:47:00.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff53d343640 10 monclient(hunting): picked mon.noname-g con 0x7ff538073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:00.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff53d343640 10 monclient(hunting): picked mon.noname-d con 0x7ff5380a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:00.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff53d343640 10 monclient(hunting): picked mon.noname-b con 0x7ff53813f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:00.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff53d343640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff53809dae0 con 0x7ff53813f2e0 2024-08-19T12:47:00.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff53d343640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff538136910 con 0x7ff538073470 2024-08-19T12:47:00.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff53d343640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff53809dd50 con 0x7ff5380a57a0 2024-08-19T12:47:00.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff53d343640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:00.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff53d343640 10 monclient(hunting): authenticate will time out at 1214.326212s 2024-08-19T12:47:00.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff5377fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff53813f2e0 0x7ff5381416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38986/0 (socket says 172.21.15.26:38986) 2024-08-19T12:47:00.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff5377fe640 1 -- 172.21.15.26:0/1473101339 learned_addr learned my addr 172.21.15.26:0/1473101339 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:00.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.239+0000 7ff5367fc640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7ff5380a57a0 0x7ff5380a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:43076/0 (socket says 172.21.15.26:43076) 2024-08-19T12:47:00.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.241+0000 7ff5377fe640 1 -- 172.21.15.26:0/1473101339 >> v1:172.21.15.26:6790/0 conn(0x7ff53813f2e0 legacy=0x7ff5381416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.241+0000 7ff5367fc640 1 -- 172.21.15.26:0/1473101339 >> v1:172.21.15.26:6792/0 conn(0x7ff5380a57a0 legacy=0x7ff5380a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.241+0000 7ff536ffd640 1 -- 172.21.15.26:0/1473101339 >> v1:172.21.15.104:6790/0 conn(0x7ff538073470 legacy=0x7ff538136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.246+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 564294488 0 0) 0x7ff53809dd50 con 0x7ff5380a57a0 2024-08-19T12:47:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.246+0000 7ff535ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.246+0000 7ff535ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.246+0000 7ff535ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.246+0000 7ff535ffb640 10 monclient(hunting): my global_id is 164139 2024-08-19T12:47:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.246+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff524003520 con 0x7ff5380a57a0 2024-08-19T12:47:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.247+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1255793148 0 0) 0x7ff53809dae0 con 0x7ff53813f2e0 2024-08-19T12:47:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.247+0000 7ff535ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.247+0000 7ff535ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.247+0000 7ff535ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.247+0000 7ff535ffb640 10 monclient(hunting): my global_id is 26507 2024-08-19T12:47:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.247+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff53809dd50 con 0x7ff53813f2e0 2024-08-19T12:47:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.248+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2739761376 0 0) 0x7ff538136910 con 0x7ff538073470 2024-08-19T12:47:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.248+0000 7ff535ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.248+0000 7ff535ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.248+0000 7ff535ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.248+0000 7ff535ffb640 10 monclient(hunting): my global_id is 154344 2024-08-19T12:47:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.248+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff53809dae0 con 0x7ff538073470 2024-08-19T12:47:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2375229858 0 0) 0x7ff53809dae0 con 0x7ff538073470 2024-08-19T12:47:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 >> v1:172.21.15.26:6792/0 conn(0x7ff5380a57a0 legacy=0x7ff5380a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 >> v1:172.21.15.26:6790/0 conn(0x7ff53813f2e0 legacy=0x7ff5381416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:47:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 1 monclient: found mon.noname-g 2024-08-19T12:47:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:47:00.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53809e030 con 0x7ff538073470 2024-08-19T12:47:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 10 monclient: _finish_auth 0 2024-08-19T12:47:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:47:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff530001ad0 con 0x7ff538073470 2024-08-19T12:47:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff53d343640 5 monclient: authenticate success, global_id 154344 2024-08-19T12:47:00.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff53d343640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:00.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:47:00.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 10 monclient: dump: 2024-08-19T12:47:00.268 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.268 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.268 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.269 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.269 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.269 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.269 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.269 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.269 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.270 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.271 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff53d343640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff536ffd640 1 --1- 172.21.15.26:0/1473101339 >> v1:172.21.15.104:6790/0 conn(0x7ff538073470 0x7ff538136200 :-1 s=OPENED pgs=3292 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7ff530003950 mon_map magic: 0 2024-08-19T12:47:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff530001d20 con 0x7ff538073470 2024-08-19T12:47:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:00.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff53d343640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:00.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff53d343640 4 set_mon_vals no callback set 2024-08-19T12:47:00.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff53d343640 10 monclient: shutdown 2024-08-19T12:47:00.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff53d343640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:00.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff53d343640 1 -- 172.21.15.26:0/1473101339 >> v1:172.21.15.104:6790/0 conn(0x7ff538073470 legacy=0x7ff538136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 1 -- 172.21.15.26:0/1473101339 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff530003950 con 0x7ff538073470 2024-08-19T12:47:00.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.249+0000 7ff535ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:47:00.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.250+0000 7ff53d343640 1 -- 172.21.15.26:0/1473101339 shutdown_connections 2024-08-19T12:47:00.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.250+0000 7ff53d343640 1 -- 172.21.15.26:0/1473101339 >> 172.21.15.26:0/1473101339 conn(0x7ff5380abff0 msgr2=0x7ff5380ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:00.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.250+0000 7ff53d343640 1 -- 172.21.15.26:0/1473101339 shutdown_connections 2024-08-19T12:47:00.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.250+0000 7ff53d343640 1 -- 172.21.15.26:0/1473101339 wait complete. 2024-08-19T12:47:00.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.250+0000 7ff53d343640 10 monclient: build_initial_monmap 2024-08-19T12:47:00.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.250+0000 7ff53d343640 10 monclient: monmap: 2024-08-19T12:47:00.275 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:00.275 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:00.275 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:00.275 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:00.275 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:00.275 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.276 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:00.276 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:00.277 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:00.277 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:00.277 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:00.277 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:00.277 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:00.277 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:00.278 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:00.278 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.250+0000 7ff53d343640 1 librados: starting msgr at 2024-08-19T12:47:00.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.250+0000 7ff53d343640 1 librados: starting objecter 2024-08-19T12:47:00.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 1 Processor -- start 2024-08-19T12:47:00.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 1 -- start start 2024-08-19T12:47:00.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 1 librados: setting wanted keys 2024-08-19T12:47:00.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 1 librados: calling monclient init 2024-08-19T12:47:00.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 10 monclient: init 2024-08-19T12:47:00.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:00.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 10 monclient: _add_conns ranks=[8,4,6,7,3,2,1,5,0] 2024-08-19T12:47:00.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 10 monclient(hunting): picked mon.noname-e con 0x7ff538073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:00.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 10 monclient(hunting): picked mon.noname-c con 0x7ff5380a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:00.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 10 monclient(hunting): picked mon.noname-d con 0x7ff53813f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:00.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff538131920 con 0x7ff5380a57a0 2024-08-19T12:47:00.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff53812f5c0 con 0x7ff53813f2e0 2024-08-19T12:47:00.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff538131080 con 0x7ff538073470 2024-08-19T12:47:00.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:00.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff53d343640 10 monclient(hunting): authenticate will time out at 1214.338642s 2024-08-19T12:47:00.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff536ffd640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7ff538073470 0x7ff53812dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:56142/0 (socket says 172.21.15.26:56142) 2024-08-19T12:47:00.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff5377fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7ff53813f2e0 0x7ff53812eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:43080/0 (socket says 172.21.15.26:43080) 2024-08-19T12:47:00.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff536ffd640 1 -- 172.21.15.26:0/2845365679 learned_addr learned my addr 172.21.15.26:0/2845365679 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:00.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.251+0000 7ff5367fc640 1 --1- 172.21.15.26:0/2845365679 >> v1:172.21.15.26:6791/0 conn(0x7ff5380a57a0 0x7ff53812e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:34656/0 (socket says 172.21.15.26:34656) 2024-08-19T12:47:00.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.253+0000 7ff5367fc640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.26:6791/0 conn(0x7ff5380a57a0 legacy=0x7ff53812e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.253+0000 7ff52f7fe640 10 client.?.objecter ms_handle_connect 0x7ff5380a57a0 2024-08-19T12:47:00.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.253+0000 7ff52f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:00.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.253+0000 7ff5377fe640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.26:6792/0 conn(0x7ff53813f2e0 legacy=0x7ff53812eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.253+0000 7ff52f7fe640 10 client.?.objecter ms_handle_connect 0x7ff538073470 2024-08-19T12:47:00.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.253+0000 7ff52f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:00.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.253+0000 7ff52f7fe640 10 client.?.objecter ms_handle_connect 0x7ff53813f2e0 2024-08-19T12:47:00.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.253+0000 7ff52f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:00.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.253+0000 7ff536ffd640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.26:6793/0 conn(0x7ff538073470 legacy=0x7ff53812dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.262+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2440761749 0 0) 0x7ff538131920 con 0x7ff5380a57a0 2024-08-19T12:47:00.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.262+0000 7ff52f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.262+0000 7ff52f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.262+0000 7ff52f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.262+0000 7ff52f7fe640 10 monclient(hunting): my global_id is 26860 2024-08-19T12:47:00.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.262+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff518004eb0 con 0x7ff5380a57a0 2024-08-19T12:47:00.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.262+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 357778031 0 0) 0x7ff53812f5c0 con 0x7ff53813f2e0 2024-08-19T12:47:00.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.262+0000 7ff52f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.262+0000 7ff52f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.262+0000 7ff52f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.262+0000 7ff52f7fe640 10 monclient(hunting): my global_id is 164148 2024-08-19T12:47:00.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.262+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff538131920 con 0x7ff53813f2e0 2024-08-19T12:47:00.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 288006948 0 0) 0x7ff538131920 con 0x7ff53813f2e0 2024-08-19T12:47:00.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff53812f5c0 con 0x7ff53813f2e0 2024-08-19T12:47:00.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1499876568 0 0) 0x7ff538131080 con 0x7ff538073470 2024-08-19T12:47:00.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient(hunting): my global_id is 26369 2024-08-19T12:47:00.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff538131920 con 0x7ff538073470 2024-08-19T12:47:00.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2135760371 0 0) 0x7ff518004eb0 con 0x7ff5380a57a0 2024-08-19T12:47:00.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff538131080 con 0x7ff5380a57a0 2024-08-19T12:47:00.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff528002020 con 0x7ff53813f2e0 2024-08-19T12:47:00.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:00.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:47:00.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient(hunting): dump: 2024-08-19T12:47:00.303 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.303 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.303 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.303 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.304 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.304 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.304 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.304 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.304 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.304 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.305 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.305 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.305 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.305 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.305 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.306 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:00.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff5200047e0 con 0x7ff5380a57a0 2024-08-19T12:47:00.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:00.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:00.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient(hunting): dump: 2024-08-19T12:47:00.308 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.308 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.308 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.309 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.309 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.309 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.309 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.309 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.309 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.310 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.310 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.310 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.310 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.310 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.310 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.311 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3848056361 0 0) 0x7ff53812f5c0 con 0x7ff53813f2e0 2024-08-19T12:47:00.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.26:6793/0 conn(0x7ff538073470 legacy=0x7ff53812dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.26:6791/0 conn(0x7ff5380a57a0 legacy=0x7ff53812e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:47:00.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 monclient: found mon.d 2024-08-19T12:47:00.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:00.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:00.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff538130120 con 0x7ff53813f2e0 2024-08-19T12:47:00.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:47:00.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:47:00.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff52f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:00.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff53d343640 5 monclient: authenticate success, global_id 164148 2024-08-19T12:47:00.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff53d343640 10 monclient: _renew_subs 2024-08-19T12:47:00.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff53d343640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:00.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff53d343640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff5381303b0 con 0x7ff53813f2e0 2024-08-19T12:47:00.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff53d343640 10 client.164148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:00.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff53d343640 10 monclient: _renew_subs 2024-08-19T12:47:00.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.263+0000 7ff53d343640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:00.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.264+0000 7ff53d343640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7ff53806ec30 con 0x7ff53813f2e0 2024-08-19T12:47:00.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.264+0000 7ff53d343640 1 librados: init done 2024-08-19T12:47:00.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.264+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff528004e10 con 0x7ff53813f2e0 2024-08-19T12:47:00.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.264+0000 7ff52f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:00.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.264+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff528005380 con 0x7ff53813f2e0 2024-08-19T12:47:00.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.264+0000 7ff52f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.264+0000 7ff52f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:00.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.264+0000 7ff52f7fe640 10 monclient: dump: 2024-08-19T12:47:00.315 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.315 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.315 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.315 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.316 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.316 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.316 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.316 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.316 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.316 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.316 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.316 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.316 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.316 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.317 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.317 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.264+0000 7ff5357fa640 4 set_mon_vals no callback set 2024-08-19T12:47:00.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.264+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff5280117e0 con 0x7ff53813f2e0 2024-08-19T12:47:00.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.265+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(351..351 src has 1..351) ==== 4961+0+0 (unknown 598841625 0 0) 0x7ff52804dfe0 con 0x7ff53813f2e0 2024-08-19T12:47:00.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.265+0000 7ff52f7fe640 10 client.164148.objecter ms_dispatch 0x7ff538000b90 osd_map(351..351 src has 1..351) 2024-08-19T12:47:00.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.265+0000 7ff52f7fe640 3 client.164148.objecter handle_osd_map got epochs [351,351] > 0 2024-08-19T12:47:00.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.265+0000 7ff52f7fe640 3 client.164148.objecter handle_osd_map decoding full epoch 351 2024-08-19T12:47:00.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.265+0000 7ff52f7fe640 20 client.164148.objecter dump_active .. 0 homeless 2024-08-19T12:47:00.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.265+0000 7ff53d343640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:00.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.265+0000 7ff53d343640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff51c0065c0 con 0x7ff51c002860 2024-08-19T12:47:00.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.266+0000 7ff536ffd640 10 monclient: get_auth_request con 0x7ff518049f50 auth_method 0 2024-08-19T12:47:00.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.269+0000 7ff5367fc640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff51c002860 legacy=0x7ff51c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.269+0000 7ff52f7fe640 10 client.164148.objecter ms_handle_connect 0x7ff51c002860 2024-08-19T12:47:00.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.269+0000 7ff52f7fe640 10 client.164148.objecter resend_mon_ops 2024-08-19T12:47:00.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.269+0000 7ff5367fc640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff51c002860 legacy=0x7ff51c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T12:47:00.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.269+0000 7ff5367fc640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff51c002860 legacy=0x7ff51c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:47:00.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.269+0000 7ff5367fc640 1 --1- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff51c002860 0x7ff51c004d00 :-1 s=OPENED pgs=3335 cs=1 l=1).handle_message read tag failed 2024-08-19T12:47:00.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.269+0000 7ff5367fc640 1 --1- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff51c002860 0x7ff51c004d00 :-1 s=OPENED pgs=3335 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:47:00.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.269+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff51c002860 legacy=0x7ff51c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:00.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.269+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff51c002860 legacy=0x7ff51c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:00.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.269+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff53812f5c0 con 0x7ff518052a20 2024-08-19T12:47:00.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.269+0000 7ff52f7fe640 10 client.164148.objecter ms_handle_connect 0x7ff518049f50 2024-08-19T12:47:00.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.272+0000 7ff52f7fe640 10 client.164148.objecter ms_handle_connect 0x7ff518052a20 2024-08-19T12:47:00.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.272+0000 7ff5367fc640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff518052a20 legacy=0x7ff518054e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.272+0000 7ff52f7fe640 10 client.164148.objecter resend_mon_ops 2024-08-19T12:47:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.272+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 125494114 0 0) 0x7ff53812f5c0 con 0x7ff518052a20 2024-08-19T12:47:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.272+0000 7ff52f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.272+0000 7ff52f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.272+0000 7ff52f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.272+0000 7ff52f7fe640 10 monclient(hunting): my global_id is 26528 2024-08-19T12:47:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.272+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff538131080 con 0x7ff518052a20 2024-08-19T12:47:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3967339619 0 0) 0x7ff538131080 con 0x7ff518052a20 2024-08-19T12:47:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff53812f5c0 con 0x7ff518052a20 2024-08-19T12:47:00.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff520004ef0 con 0x7ff518052a20 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 10 monclient: dump: 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.321 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3453501196 0 0) 0x7ff53812f5c0 con 0x7ff518052a20 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff518057720 con 0x7ff518052a20 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff520007230 con 0x7ff518052a20 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff518052a20 legacy=0x7ff518054e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.273+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff518052a20 legacy=0x7ff518054e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.279+0000 7ff53d343640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:00.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.280+0000 7ff53d343640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff51c009d50 con 0x7ff51c009140 2024-08-19T12:47:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.282+0000 7ff5367fc640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff51c009140 legacy=0x7ff51c009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.282+0000 7ff52f7fe640 10 client.164148.objecter ms_handle_connect 0x7ff51c009140 2024-08-19T12:47:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.282+0000 7ff52f7fe640 10 client.164148.objecter resend_mon_ops 2024-08-19T12:47:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.282+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3147907012 0 0) 0x7ff51c009d50 con 0x7ff51c009140 2024-08-19T12:47:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.282+0000 7ff52f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.282+0000 7ff52f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.282+0000 7ff52f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.282+0000 7ff52f7fe640 10 monclient(hunting): my global_id is 26537 2024-08-19T12:47:00.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.282+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff53812f5c0 con 0x7ff51c009140 2024-08-19T12:47:00.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 175301407 0 0) 0x7ff53812f5c0 con 0x7ff51c009140 2024-08-19T12:47:00.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff51c009d50 con 0x7ff51c009140 2024-08-19T12:47:00.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff520005930 con 0x7ff51c009140 2024-08-19T12:47:00.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:47:00.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 10 monclient: dump: 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.325 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.326 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.326 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1695157309 0 0) 0x7ff51c009d50 con 0x7ff51c009140 2024-08-19T12:47:00.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff51c00c350 con 0x7ff51c009140 2024-08-19T12:47:00.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 66444838) 0x7ff52000a240 con 0x7ff51c009140 2024-08-19T12:47:00.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:00.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:00.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff51c009140 legacy=0x7ff51c009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff52f7fe640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6791/0 conn(0x7ff51c009140 legacy=0x7ff51c009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:00.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff5377fe640 1 -- 172.21.15.26:0/2845365679 reap_dead start 2024-08-19T12:47:00.327 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:00.327 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:00.327 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:00.327 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:00.327 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:47:00.327 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 533306, 2024-08-19T12:47:00.327 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:00.328 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:00.329 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.330 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:00.330 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:00.330 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:00.330 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:00.330 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:00.330 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:00.330 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:00.330 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:00.330 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:00.330 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:00.331 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:00.332 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:00.332 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:00.332 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:00.332 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:00.332 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:00.332 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:00.332 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:00.332 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:00.332 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:00.332 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:00.332 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:00.333 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:00.333 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:00.333 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:00.333 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:00.333 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:00.333 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:00.333 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:00.333 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:00.333 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:00.334 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.334 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:00.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.334 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:00.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.334 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:00.334 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:00.334 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.334 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.334 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:00.335 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.335 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.335 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.335 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:00.335 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:00.335 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.335 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.335 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.336 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:00.336 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:00.336 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.336 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.336 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:00.336 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.336 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.336 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:00.337 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:00.337 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.337 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.337 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.337 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:00.337 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:00.337 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.337 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.338 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.338 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:00.338 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.338 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.338 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.338 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:00.338 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:00.338 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.338 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.339 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:00.340 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.340 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.340 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:00.340 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:00.340 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.340 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.340 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.341 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:00.341 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:00.341 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:00.341 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.342 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.342 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:00.342 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:00.342 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.342 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.342 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.342 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:00.343 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:00.343 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.343 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:00.344 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.344 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.344 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:00.344 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:00.344 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.344 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.345 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.345 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:00.345 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:00.345 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.345 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.345 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.345 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:00.345 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.346 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.346 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.346 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:00.346 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:00.346 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.346 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.347 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:00.348 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:00.348 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.348 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.348 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.348 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:00.348 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:00.349 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:00.349 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.349 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.350 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:00.350 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:00.350 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.350 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.350 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.350 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.350 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:00.351 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:00.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.351 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:00.351 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:00.351 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:00.351 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.351 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.351 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:47:00.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.352 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:00.352 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:00.352 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:00.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.352 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.352 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:00.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.352 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:00.352 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:00.352 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:47:00.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.353 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:00.353 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff53d343640 10 librados: watch_flush enter 2024-08-19T12:47:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff53d343640 10 librados: watch_flush exit 2024-08-19T12:47:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff53d343640 20 client.164148.objecter shutdown clearing up homeless session... 2024-08-19T12:47:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff53d343640 10 client.164148.objecter successfully canceled tick 2024-08-19T12:47:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff53d343640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.104:6800/899887506 conn(0x7ff518049f50 legacy=0x7ff51804c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff53d343640 10 monclient: shutdown 2024-08-19T12:47:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff53d343640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.283+0000 7ff53d343640 1 -- 172.21.15.26:0/2845365679 >> v1:172.21.15.26:6792/0 conn(0x7ff53813f2e0 legacy=0x7ff53812eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.284+0000 7ff53d343640 1 -- 172.21.15.26:0/2845365679 shutdown_connections 2024-08-19T12:47:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.284+0000 7ff53d343640 1 -- 172.21.15.26:0/2845365679 >> 172.21.15.26:0/2845365679 conn(0x7ff5380abff0 msgr2=0x7ff538133df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.284+0000 7ff53d343640 1 -- 172.21.15.26:0/2845365679 shutdown_connections 2024-08-19T12:47:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.284+0000 7ff53d343640 1 -- 172.21.15.26:0/2845365679 wait complete. 2024-08-19T12:47:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.284+0000 7ff53d343640 1 librados: shutdown 2024-08-19T12:47:00.357 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:47:00.357 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:47:00.357 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:47:00.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.455+0000 7fcebb9de640 10 monclient: get_monmap_and_config 2024-08-19T12:47:00.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.455+0000 7fcebb9de640 10 monclient: build_initial_monmap 2024-08-19T12:47:00.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.455+0000 7fcebb9de640 10 monclient: monmap: 2024-08-19T12:47:00.456 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:00.456582+0000 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:00.456582+0000 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:00.457 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:00.458 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 1 Processor -- start 2024-08-19T12:47:00.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 1 -- start start 2024-08-19T12:47:00.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 10 monclient: init 2024-08-19T12:47:00.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:00.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 10 monclient: _add_conns ranks=[5,0,3,7,2,4,8,1,6] 2024-08-19T12:47:00.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 10 monclient(hunting): picked mon.noname-h con 0x7fceb406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:00.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 10 monclient(hunting): picked mon.noname-a con 0x7fceb40a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:00.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 10 monclient(hunting): picked mon.noname-g con 0x7fceb40a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:00.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceb4136790 con 0x7fceb40a5920 2024-08-19T12:47:00.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceb409ddf0 con 0x7fceb40a5d50 2024-08-19T12:47:00.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceb409e0d0 con 0x7fceb406e7b0 2024-08-19T12:47:00.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:00.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.456+0000 7fcebb9de640 10 monclient(hunting): authenticate will time out at 1214.543996s 2024-08-19T12:47:00.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.457+0000 7fceb8f79640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fceb40a5920 0x7fceb413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47948/0 (socket says 172.21.15.26:47948) 2024-08-19T12:47:00.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.457+0000 7fceb8f79640 1 -- 172.21.15.26:0/1353212622 learned_addr learned my addr 172.21.15.26:0/1353212622 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:00.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceb8f79640 1 -- 172.21.15.26:0/1353212622 >> v1:172.21.15.26:6789/0 conn(0x7fceb40a5920 legacy=0x7fceb413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 1 -- 172.21.15.26:0/1353212622 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2109608206 0 0) 0x7fceb4136790 con 0x7fceb40a5920 2024-08-19T12:47:00.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient(hunting): my global_id is 144810 2024-08-19T12:47:00.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 1 -- 172.21.15.26:0/1353212622 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcea0003520 con 0x7fceb40a5920 2024-08-19T12:47:00.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceb977a640 1 -- 172.21.15.26:0/1353212622 >> v1:172.21.15.104:6791/0 conn(0x7fceb406e7b0 legacy=0x7fceb4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceb9f7b640 1 -- 172.21.15.26:0/1353212622 >> v1:172.21.15.104:6790/0 conn(0x7fceb40a5d50 legacy=0x7fceb4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 1 -- 172.21.15.26:0/1353212622 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2832349519 0 0) 0x7fceb409e0d0 con 0x7fceb406e7b0 2024-08-19T12:47:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient(hunting): my global_id is 26546 2024-08-19T12:47:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 1 -- 172.21.15.26:0/1353212622 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceb4136790 con 0x7fceb406e7b0 2024-08-19T12:47:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 1 -- 172.21.15.26:0/1353212622 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3125720900 0 0) 0x7fcea0003520 con 0x7fceb40a5920 2024-08-19T12:47:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 1 -- 172.21.15.26:0/1353212622 >> v1:172.21.15.104:6791/0 conn(0x7fceb406e7b0 legacy=0x7fceb4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 1 -- 172.21.15.26:0/1353212622 >> v1:172.21.15.104:6790/0 conn(0x7fceb40a5d50 legacy=0x7fceb4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient: _finish_hunting 0 2024-08-19T12:47:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 1 monclient: found mon.noname-a 2024-08-19T12:47:00.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:47:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 1 -- 172.21.15.26:0/1353212622 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb409e3b0 con 0x7fceb40a5920 2024-08-19T12:47:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient: _finish_auth 0 2024-08-19T12:47:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient: _check_auth_tickets 2024-08-19T12:47:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fcebb9de640 5 monclient: authenticate success, global_id 144810 2024-08-19T12:47:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 1 -- 172.21.15.26:0/1353212622 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcea4001ae0 con 0x7fceb40a5920 2024-08-19T12:47:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fcebb9de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:47:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient: dump: 2024-08-19T12:47:00.466 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.467 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.468 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.468 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fcebb9de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:00.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 1 -- 172.21.15.26:0/1353212622 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcea4001d30 con 0x7fceb40a5920 2024-08-19T12:47:00.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:00.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 1 -- 172.21.15.26:0/1353212622 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcea4004be0 con 0x7fceb40a5920 2024-08-19T12:47:00.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fcebb9de640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:00.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fcebb9de640 4 set_mon_vals no callback set 2024-08-19T12:47:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.459+0000 7fceabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.460+0000 7fceabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.460+0000 7fceabfff640 10 monclient: dump: 2024-08-19T12:47:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.469 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.470 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.460+0000 7fcebb9de640 10 monclient: shutdown 2024-08-19T12:47:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.460+0000 7fcebb9de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.460+0000 7fcebb9de640 1 -- 172.21.15.26:0/1353212622 >> v1:172.21.15.26:6789/0 conn(0x7fceb40a5920 legacy=0x7fceb413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.460+0000 7fcebb9de640 1 -- 172.21.15.26:0/1353212622 shutdown_connections 2024-08-19T12:47:00.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.460+0000 7fcebb9de640 1 -- 172.21.15.26:0/1353212622 >> 172.21.15.26:0/1353212622 conn(0x7fceb40a8740 msgr2=0x7fceb40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.460+0000 7fcebb9de640 1 -- 172.21.15.26:0/1353212622 shutdown_connections 2024-08-19T12:47:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.460+0000 7fcebb9de640 1 -- 172.21.15.26:0/1353212622 wait complete. 2024-08-19T12:47:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 10 monclient: build_initial_monmap 2024-08-19T12:47:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 10 monclient: monmap: 2024-08-19T12:47:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:00.471 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:00.472 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 1 librados: starting msgr at 2024-08-19T12:47:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 1 librados: starting objecter 2024-08-19T12:47:00.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 1 Processor -- start 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 1 -- start start 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 1 librados: setting wanted keys 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 1 librados: calling monclient init 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 10 monclient: init 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 10 monclient: _add_conns ranks=[6,3,1,5,7,4,8,0,2] 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 10 monclient(hunting): picked mon.noname-d con 0x7fceb406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 10 monclient(hunting): picked mon.noname-g con 0x7fceb40a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 10 monclient(hunting): picked mon.noname-f con 0x7fceb40a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceb40a0600 con 0x7fceb40a5d50 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceb41283b0 con 0x7fceb40a5920 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceb40a20b0 con 0x7fceb406e7b0 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:00.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.461+0000 7fcebb9de640 10 monclient(hunting): authenticate will time out at 1214.548935s 2024-08-19T12:47:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.462+0000 7fceb977a640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fceb406e7b0 0x7fceb409f7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:43094/0 (socket says 172.21.15.26:43094) 2024-08-19T12:47:00.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.462+0000 7fceb977a640 1 -- 172.21.15.26:0/3674481485 learned_addr learned my addr 172.21.15.26:0/3674481485 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fceb977a640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.26:6792/0 conn(0x7fceb406e7b0 legacy=0x7fceb409f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 client.?.objecter ms_handle_connect 0x7fceb406e7b0 2024-08-19T12:47:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 client.?.objecter ms_handle_connect 0x7fceb40a5920 2024-08-19T12:47:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fceb8f79640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.104:6790/0 conn(0x7fceb40a5920 legacy=0x7fceb409fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:00.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 client.?.objecter ms_handle_connect 0x7fceb40a5d50 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fceb9f7b640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.104:6789/0 conn(0x7fceb40a5d50 legacy=0x7fceb41d6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 673336763 0 0) 0x7fceb40a20b0 con 0x7fceb406e7b0 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): my global_id is 164157 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce98004eb0 con 0x7fceb406e7b0 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1806384055 0 0) 0x7fceb41283b0 con 0x7fceb40a5920 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): my global_id is 154362 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceb40a20b0 con 0x7fceb40a5920 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3925997244 0 0) 0x7fceb40a0600 con 0x7fceb40a5d50 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): my global_id is 27325 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceb41283b0 con 0x7fceb40a5d50 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2979269287 0 0) 0x7fce98004eb0 con 0x7fceb406e7b0 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fceb40a0600 con 0x7fceb406e7b0 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fceb0003610 con 0x7fceb406e7b0 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:47:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): dump: 2024-08-19T12:47:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.483 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1289716717 0 0) 0x7fceb40a20b0 con 0x7fceb40a5920 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce98004eb0 con 0x7fceb40a5920 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcea4004b20 con 0x7fceb40a5920 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): dump: 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.484 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.485 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 741535424 0 0) 0x7fceb41283b0 con 0x7fceb40a5d50 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fceb40a20b0 con 0x7fceb40a5d50 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fceac0017b0 con 0x7fceb40a5d50 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient(hunting): dump: 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.486 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4095326157 0 0) 0x7fceb40a0600 con 0x7fceb406e7b0 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.104:6790/0 conn(0x7fceb40a5920 legacy=0x7fceb409fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.104:6789/0 conn(0x7fceb40a5d50 legacy=0x7fceb41d6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:47:00.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 monclient: found mon.d 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb40a1dd0 con 0x7fceb406e7b0 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient: _finish_auth 0 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcea9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcebb9de640 5 monclient: authenticate success, global_id 164157 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcebb9de640 10 monclient: _renew_subs 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcebb9de640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.464+0000 7fcebb9de640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fceb40a0e70 con 0x7fceb406e7b0 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fceb0003bc0 con 0x7fceb406e7b0 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcea9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:00.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fceb00040b0 con 0x7fceb406e7b0 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcea9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcea9ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcea9ffb640 10 monclient: dump: 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.489 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fceab7fe640 4 set_mon_vals no callback set 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcebb9de640 10 client.164157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcebb9de640 10 monclient: _renew_subs 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcebb9de640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcebb9de640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fceb4134740 con 0x7fceb406e7b0 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcebb9de640 1 librados: init done 2024-08-19T12:47:00.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fceb0004250 con 0x7fceb406e7b0 2024-08-19T12:47:00.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcebb9de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:00.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcebb9de640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fceb4077470 con 0x7fceb402e340 2024-08-19T12:47:00.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(351..351 src has 1..351) ==== 4961+0+0 (unknown 598841625 0 0) 0x7fceb004e060 con 0x7fceb406e7b0 2024-08-19T12:47:00.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcea9ffb640 10 client.164157.objecter ms_dispatch 0x7fceb4000b90 osd_map(351..351 src has 1..351) 2024-08-19T12:47:00.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcea9ffb640 3 client.164157.objecter handle_osd_map got epochs [351,351] > 0 2024-08-19T12:47:00.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.465+0000 7fcea9ffb640 3 client.164157.objecter handle_osd_map decoding full epoch 351 2024-08-19T12:47:00.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.466+0000 7fcea9ffb640 20 client.164157.objecter dump_active .. 0 homeless 2024-08-19T12:47:00.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.466+0000 7fceb9f7b640 10 monclient: get_auth_request con 0x7fce98041800 auth_method 0 2024-08-19T12:47:00.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fceb8f79640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.104:6792/0 conn(0x7fceb402e340 legacy=0x7fceb4063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 10 client.164157.objecter ms_handle_connect 0x7fceb402e340 2024-08-19T12:47:00.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 10 client.164157.objecter resend_mon_ops 2024-08-19T12:47:00.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4149485841 0 0) 0x7fceb409e3b0 con 0x7fceb402e340 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 10 monclient(hunting): my global_id is 135475 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceb40a0600 con 0x7fceb402e340 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 10 client.164157.objecter ms_handle_connect 0x7fce98041800 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2226672208 0 0) 0x7fceb40a0600 con 0x7fceb402e340 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fceb409e3b0 con 0x7fceb402e340 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcea4001580 con 0x7fceb402e340 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 10 monclient: dump: 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.492 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.493 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 402657256 0 0) 0x7fceb409e3b0 con 0x7fceb402e340 2024-08-19T12:47:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.468+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fceb402e720 con 0x7fceb402e340 2024-08-19T12:47:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.469+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcea40055a0 con 0x7fceb402e340 2024-08-19T12:47:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.469+0000 7fcea9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.469+0000 7fcea9ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.469+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.104:6792/0 conn(0x7fceb402e340 legacy=0x7fceb4063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.469+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.104:6792/0 conn(0x7fceb402e340 legacy=0x7fceb4063a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.475+0000 7fcebb9de640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.475+0000 7fcebb9de640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fceb4134490 con 0x7fceb4125700 2024-08-19T12:47:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.477+0000 7fceb8f79640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.104:6792/0 conn(0x7fceb4125700 legacy=0x7fceb41dbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.477+0000 7fcea9ffb640 10 client.164157.objecter ms_handle_connect 0x7fceb4125700 2024-08-19T12:47:00.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.477+0000 7fcea9ffb640 10 client.164157.objecter resend_mon_ops 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.477+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3712734283 0 0) 0x7fcea4005750 con 0x7fceb4125700 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.477+0000 7fcea9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.477+0000 7fcea9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.477+0000 7fcea9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.477+0000 7fcea9ffb640 10 monclient(hunting): my global_id is 135484 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.477+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceb409e3b0 con 0x7fceb4125700 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1874748744 0 0) 0x7fceb409e3b0 con 0x7fceb4125700 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcea4005750 con 0x7fceb4125700 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fcea4005ed0 con 0x7fceb4125700 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 10 monclient: dump: 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.495 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.496 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 651022606 0 0) 0x7fcea4005750 con 0x7fceb4125700 2024-08-19T12:47:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fceb40a25d0 con 0x7fceb4125700 2024-08-19T12:47:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 2051112587) 0x7fcea4008170 con 0x7fceb4125700 2024-08-19T12:47:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.104:6792/0 conn(0x7fceb4125700 legacy=0x7fceb41dbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.478+0000 7fcea9ffb640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.104:6792/0 conn(0x7fceb4125700 legacy=0x7fceb41dbed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:00.497 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 390835, 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:00.498 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:00.499 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:47:00.499 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:47:00.499 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.499 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:47:00.499 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:00.499 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:00.499 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:00.499 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:00.499 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:00.499 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:00.499 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:00.499 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:00.500 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:00.501 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:00.502 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:00.502 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:00.502 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:00.502 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:00.502 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:00.502 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:00.502 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:00.502 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:00.502 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:00.502 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:00.502 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:00.502 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.503 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.504 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.505 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.506 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:00.507 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.508 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:00.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.510 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.510 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:00.510 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:00.510 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.510 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.513 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.513 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:00.513 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.513 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.513 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.513 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:00.513 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:00.513 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.513 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.513 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.514 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:00.515 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.516 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.517 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:00.518 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:00.518 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:00.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 10 librados: watch_flush enter 2024-08-19T12:47:00.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 10 librados: watch_flush exit 2024-08-19T12:47:00.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 20 client.164157.objecter shutdown clearing up homeless session... 2024-08-19T12:47:00.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 10 client.164157.objecter successfully canceled tick 2024-08-19T12:47:00.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.104:6800/899887506 conn(0x7fce98041800 legacy=0x7fce98043cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 10 monclient: shutdown 2024-08-19T12:47:00.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:00.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 1 -- 172.21.15.26:0/3674481485 >> v1:172.21.15.26:6792/0 conn(0x7fceb406e7b0 legacy=0x7fceb409f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fceb9f7b640 1 -- 172.21.15.26:0/3674481485 reap_dead start 2024-08-19T12:47:00.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fceb9f7b640 1 -- 172.21.15.26:0/3674481485 reap_dead start 2024-08-19T12:47:00.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 1 -- 172.21.15.26:0/3674481485 shutdown_connections 2024-08-19T12:47:00.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 1 -- 172.21.15.26:0/3674481485 >> 172.21.15.26:0/3674481485 conn(0x7fceb40a8740 msgr2=0x7fceb41327b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:00.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 1 -- 172.21.15.26:0/3674481485 shutdown_connections 2024-08-19T12:47:00.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 1 -- 172.21.15.26:0/3674481485 wait complete. 2024-08-19T12:47:00.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.479+0000 7fcebb9de640 1 librados: shutdown 2024-08-19T12:47:00.521 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:47:00.522 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T12:47:00.522 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T12:47:00.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.619+0000 7f01f04d3640 10 monclient: get_monmap_and_config 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.619+0000 7f01f04d3640 10 monclient: build_initial_monmap 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.619+0000 7f01f04d3640 10 monclient: monmap: 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:00.620524+0000 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:00.620524+0000 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:00.621 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:00.622 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:00.622 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:00.622 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:00.622 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:00.622 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 1 Processor -- start 2024-08-19T12:47:00.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 1 -- start start 2024-08-19T12:47:00.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 10 monclient: init 2024-08-19T12:47:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 10 monclient: _add_conns ranks=[0,1,7,4,2,5,8,6,3] 2024-08-19T12:47:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 10 monclient(hunting): picked mon.noname-a con 0x7f01e806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 10 monclient(hunting): picked mon.noname-f con 0x7f01e80a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 10 monclient(hunting): picked mon.noname-i con 0x7f01e80a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:47:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e8136790 con 0x7f01e806e7b0 2024-08-19T12:47:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e809ddf0 con 0x7f01e80a5920 2024-08-19T12:47:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e809e0d0 con 0x7f01e80a5d50 2024-08-19T12:47:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.620+0000 7f01f04d3640 10 monclient(hunting): authenticate will time out at 1214.707970s 2024-08-19T12:47:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.621+0000 7f01ee26f640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f01e806e7b0 0x7f01e8136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47958/0 (socket says 172.21.15.26:47958) 2024-08-19T12:47:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.621+0000 7f01ee26f640 1 -- 172.21.15.26:0/4054778738 learned_addr learned my addr 172.21.15.26:0/4054778738 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:00.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ee26f640 1 -- 172.21.15.26:0/4054778738 >> v1:172.21.15.26:6789/0 conn(0x7f01e806e7b0 legacy=0x7f01e8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 1 -- 172.21.15.26:0/4054778738 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 694150517 0 0) 0x7f01e8136790 con 0x7f01e806e7b0 2024-08-19T12:47:00.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient(hunting): my global_id is 144819 2024-08-19T12:47:00.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 1 -- 172.21.15.26:0/4054778738 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01d4003520 con 0x7f01e806e7b0 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01eda6e640 1 -- 172.21.15.26:0/4054778738 >> v1:172.21.15.104:6789/0 conn(0x7f01e80a5920 legacy=0x7f01e813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01eea70640 1 -- 172.21.15.26:0/4054778738 >> v1:172.21.15.104:6792/0 conn(0x7f01e80a5d50 legacy=0x7f01e8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 1 -- 172.21.15.26:0/4054778738 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1014169157 0 0) 0x7f01d4003520 con 0x7f01e806e7b0 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 1 -- 172.21.15.26:0/4054778738 >> v1:172.21.15.104:6792/0 conn(0x7f01e80a5d50 legacy=0x7f01e8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 1 -- 172.21.15.26:0/4054778738 >> v1:172.21.15.104:6789/0 conn(0x7f01e80a5920 legacy=0x7f01e813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient: _finish_hunting 0 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 1 monclient: found mon.noname-a 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 1 -- 172.21.15.26:0/4054778738 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e809e3b0 con 0x7f01e806e7b0 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient: _finish_auth 0 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient: _check_auth_tickets 2024-08-19T12:47:00.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 1 -- 172.21.15.26:0/4054778738 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f01e4001ba0 con 0x7f01e806e7b0 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01f04d3640 5 monclient: authenticate success, global_id 144819 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01f04d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient: dump: 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.637 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01f04d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 1 -- 172.21.15.26:0/4054778738 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01e40048a0 con 0x7f01e806e7b0 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 1 -- 172.21.15.26:0/4054778738 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f01e4004e10 con 0x7f01e806e7b0 2024-08-19T12:47:00.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01f04d3640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01f04d3640 4 set_mon_vals no callback set 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.623+0000 7f01ed26d640 10 monclient: dump: 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.624+0000 7f01f04d3640 10 monclient: shutdown 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.624+0000 7f01f04d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.624+0000 7f01f04d3640 1 -- 172.21.15.26:0/4054778738 >> v1:172.21.15.26:6789/0 conn(0x7f01e806e7b0 legacy=0x7f01e8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.624+0000 7f01f04d3640 1 -- 172.21.15.26:0/4054778738 shutdown_connections 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.624+0000 7f01f04d3640 1 -- 172.21.15.26:0/4054778738 >> 172.21.15.26:0/4054778738 conn(0x7f01e80a8740 msgr2=0x7f01e80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.624+0000 7f01f04d3640 1 -- 172.21.15.26:0/4054778738 shutdown_connections 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.624+0000 7f01f04d3640 1 -- 172.21.15.26:0/4054778738 wait complete. 2024-08-19T12:47:00.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.624+0000 7f01f04d3640 10 monclient: build_initial_monmap 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.624+0000 7f01f04d3640 10 monclient: monmap: 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:00.641 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 1 librados: starting msgr at 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 1 librados: starting objecter 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 1 Processor -- start 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 1 -- start start 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 1 librados: setting wanted keys 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 1 librados: calling monclient init 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 10 monclient: init 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 10 monclient: _add_conns ranks=[1,5,6,7,2,0,8,3,4] 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 10 monclient(hunting): picked mon.noname-f con 0x7f01e806e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:00.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 10 monclient(hunting): picked mon.noname-h con 0x7f01e80a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 10 monclient(hunting): picked mon.noname-d con 0x7f01e80a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e809f8c0 con 0x7f01e806e7b0 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e8134660 con 0x7f01e80a5920 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e80a0b10 con 0x7f01e80a5d50 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01f04d3640 10 monclient(hunting): authenticate will time out at 1214.712718s 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01eea70640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f01e80a5d50 0x7f01e8132df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:43100/0 (socket says 172.21.15.26:43100) 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.625+0000 7f01eea70640 1 -- 172.21.15.26:0/3935244975 learned_addr learned my addr 172.21.15.26:0/3935244975 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.627+0000 7f01eea70640 1 -- 172.21.15.26:0/3935244975 >> v1:172.21.15.26:6792/0 conn(0x7f01e80a5d50 legacy=0x7f01e8132df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.627+0000 7f01deffd640 10 client.?.objecter ms_handle_connect 0x7f01e80a5d50 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 client.?.objecter ms_handle_connect 0x7f01e806e7b0 2024-08-19T12:47:00.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01ee26f640 1 -- 172.21.15.26:0/3935244975 >> v1:172.21.15.104:6789/0 conn(0x7f01e806e7b0 legacy=0x7f01e8131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 client.?.objecter ms_handle_connect 0x7f01e80a5920 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01eda6e640 1 -- 172.21.15.26:0/3935244975 >> v1:172.21.15.104:6791/0 conn(0x7f01e80a5920 legacy=0x7f01e81326e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3543787451 0 0) 0x7f01e80a0b10 con 0x7f01e80a5d50 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): my global_id is 164166 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01cc004eb0 con 0x7f01e80a5d50 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 268530143 0 0) 0x7f01d8001540 con 0x7f01e80a5920 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): my global_id is 26555 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e80a0b10 con 0x7f01e80a5920 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2398992009 0 0) 0x7f01e809f8c0 con 0x7f01e806e7b0 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): my global_id is 27343 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01d8001540 con 0x7f01e806e7b0 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 502392407 0 0) 0x7f01cc004eb0 con 0x7f01e80a5d50 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01e809f8c0 con 0x7f01e80a5d50 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f01e0002000 con 0x7f01e80a5d50 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): dump: 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.646 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 280379095 0 0) 0x7f01e809f8c0 con 0x7f01e80a5d50 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 >> v1:172.21.15.104:6791/0 conn(0x7f01e80a5920 legacy=0x7f01e81326e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 >> v1:172.21.15.104:6789/0 conn(0x7f01e806e7b0 legacy=0x7f01e8131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient: _finish_hunting 0 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 monclient: found mon.d 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e80a0df0 con 0x7f01e80a5d50 2024-08-19T12:47:00.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient: _finish_auth 0 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient: _check_auth_tickets 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f01e4001b60 con 0x7f01e806e7b0 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01f04d3640 5 monclient: authenticate success, global_id 164166 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01f04d3640 10 monclient: _renew_subs 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01f04d3640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01f04d3640 1 -- 172.21.15.26:0/3935244975 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f01e809f5f0 con 0x7f01e80a5d50 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01f04d3640 10 client.164166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01f04d3640 10 monclient: _renew_subs 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01f04d3640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01f04d3640 1 -- 172.21.15.26:0/3935244975 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f01e81d72a0 con 0x7f01e80a5d50 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01f04d3640 1 librados: init done 2024-08-19T12:47:00.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01e0003c40 con 0x7f01e80a5d50 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f01e0004170 con 0x7f01e80a5d50 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01eca6c640 4 set_mon_vals no callback set 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.628+0000 7f01deffd640 10 monclient: dump: 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:00.649 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.629+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f01e00043c0 con 0x7f01e80a5d50 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.629+0000 7f01f04d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.629+0000 7f01f04d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.629+0000 7f01f04d3640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.629+0000 7f01f04d3640 1 -- 172.21.15.26:0/3935244975 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d0005080 con 0x7f01e80a5d50 2024-08-19T12:47:00.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.629+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(351..351 src has 1..351) ==== 4961+0+0 (unknown 598841625 0 0) 0x7f01e004e160 con 0x7f01e80a5d50 2024-08-19T12:47:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.629+0000 7f01deffd640 10 client.164166.objecter ms_dispatch 0x7f01e8000b90 osd_map(351..351 src has 1..351) 2024-08-19T12:47:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.629+0000 7f01deffd640 3 client.164166.objecter handle_osd_map got epochs [351,351] > 0 2024-08-19T12:47:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.629+0000 7f01deffd640 3 client.164166.objecter handle_osd_map decoding full epoch 351 2024-08-19T12:47:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.629+0000 7f01deffd640 20 client.164166.objecter dump_active .. 0 homeless 2024-08-19T12:47:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.629+0000 7f01ee26f640 10 monclient: get_auth_request con 0x7f01cc040d90 auth_method 0 2024-08-19T12:47:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.632+0000 7f01deffd640 10 client.164166.objecter ms_handle_connect 0x7f01cc040d90 2024-08-19T12:47:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.634+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f01e0017090 con 0x7f01e80a5d50 2024-08-19T12:47:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.634+0000 7f01deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:00.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.634+0000 7f01deffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:00.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.889+0000 7f01f04d3640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T12:47:00.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.889+0000 7f01f04d3640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T12:47:00.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.889+0000 7f01f04d3640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:00.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.889+0000 7f01f04d3640 1 -- 172.21.15.26:0/3935244975 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f01d0005c40 con 0x7f01e80a5d50 2024-08-19T12:47:00.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.890+0000 7f01deffd640 1 -- 172.21.15.26:0/3935244975 <== mon.6 v1:172.21.15.26:6792/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f01e001ffd0 con 0x7f01e80a5d50 2024-08-19T12:47:00.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.890+0000 7f01deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T12:47:00.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.890+0000 7f01deffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:00.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=10,kv=13,logm=52} crc {auth=233369010,config=1097430493,health=3978515193,kv=1344821656,logm=3084378660}) 2024-08-19T12:47:00.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=12,kv=13,logm=51} crc {auth=1773997644,config=1097430493,health=2941319762,kv=1344821656,logm=410794029}) 2024-08-19T12:47:00.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=10,kv=13,logm=52} crc {auth=233369010,config=1097430493,health=3978515193,kv=1344821656,logm=3084378660}) 2024-08-19T12:47:00.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=13,logm=54} crc {auth=975611233,config=1097430493,health=590541244,kv=1344821656,logm=430767663}) 2024-08-19T12:47:00.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=10,kv=13,logm=52} crc {auth=233369010,config=1097430493,health=3978515193,kv=1344821656,logm=3084378660}) 2024-08-19T12:47:00.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=22,config=2,health=9,kv=12,logm=55} crc {auth=1139357816,config=1097430493,health=2238746497,kv=1344821656,logm=3890618698}) 2024-08-19T12:47:00.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.893 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=10,kv=13,logm=52} crc {auth=233369010,config=1097430493,health=3978515193,kv=1344821656,logm=3084378660}) 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=24,config=2,health=12,kv=13,logm=49} crc {auth=1280133455,config=1097430493,health=2941319762,kv=1344821656,logm=1150135378}) 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=10,kv=13,logm=52} crc {auth=233369010,config=1097430493,health=3978515193,kv=1344821656,logm=3084378660}) 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=23,config=2,health=12,kv=12,logm=51} crc {auth=4255190632,config=1097430493,health=2941319762,kv=3904374251,logm=473492267}) 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=10,kv=13,logm=52} crc {auth=233369010,config=1097430493,health=3978515193,kv=1344821656,logm=3084378660}) 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=23,config=2,health=12,kv=12,logm=51} crc {auth=3530883163,config=1097430493,health=2941319762,kv=437017915,logm=1045627562}) 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=10,kv=13,logm=52} crc {auth=233369010,config=1097430493,health=3978515193,kv=1344821656,logm=3084378660}) 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=23,config=2,health=12,kv=13,logm=50} crc {auth=1767943288,config=1097430493,health=2941319762,kv=1344821656,logm=2304607672}) 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=10,kv=13,logm=52} crc {auth=233369010,config=1097430493,health=3978515193,kv=1344821656,logm=3084378660}) 2024-08-19T12:47:00.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.891+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=24,config=2,health=11,kv=13,logm=50} crc {auth=455520962,config=1097430493,health=3660819857,kv=1283792330,logm=1228028048}) 2024-08-19T12:47:00.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 10 librados: watch_flush enter 2024-08-19T12:47:00.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 10 librados: watch_flush exit 2024-08-19T12:47:00.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 20 client.164166.objecter shutdown clearing up homeless session... 2024-08-19T12:47:00.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 10 client.164166.objecter successfully canceled tick 2024-08-19T12:47:00.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 1 -- 172.21.15.26:0/3935244975 >> v1:172.21.15.104:6800/899887506 conn(0x7f01cc040d90 legacy=0x7f01cc043250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 10 monclient: shutdown 2024-08-19T12:47:00.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:00.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 1 -- 172.21.15.26:0/3935244975 >> v1:172.21.15.26:6792/0 conn(0x7f01e80a5d50 legacy=0x7f01e8132df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:00.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 1 -- 172.21.15.26:0/3935244975 shutdown_connections 2024-08-19T12:47:00.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 1 -- 172.21.15.26:0/3935244975 >> 172.21.15.26:0/3935244975 conn(0x7f01e80a8740 msgr2=0x7f01e806d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:00.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 1 -- 172.21.15.26:0/3935244975 shutdown_connections 2024-08-19T12:47:00.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 1 -- 172.21.15.26:0/3935244975 wait complete. 2024-08-19T12:47:00.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:00.893+0000 7f01f04d3640 1 librados: shutdown 2024-08-19T12:47:00.905 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T12:47:00.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838895750}) 2024-08-19T12:47:00.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=999280320}) 2024-08-19T12:47:00.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838895750}) 2024-08-19T12:47:00.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=261687107}) 2024-08-19T12:47:00.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838895750}) 2024-08-19T12:47:00.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2676682095}) 2024-08-19T12:47:00.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838895750}) 2024-08-19T12:47:00.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2106702099}) 2024-08-19T12:47:00.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838895750}) 2024-08-19T12:47:00.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3287862707}) 2024-08-19T12:47:00.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838895750}) 2024-08-19T12:47:00.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3857584878}) 2024-08-19T12:47:00.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838895750}) 2024-08-19T12:47:00.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4135373340}) 2024-08-19T12:47:00.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838895750}) 2024-08-19T12:47:00.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=631600208}) 2024-08-19T12:47:00.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109562755}) 2024-08-19T12:47:00.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3072122049}) 2024-08-19T12:47:00.981 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109562755}) 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3005401354}) 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109562755}) 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=667291597}) 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109562755}) 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=858669777}) 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109562755}) 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3009116745}) 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109562755}) 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1123244638}) 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109562755}) 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4142835891}) 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109562755}) 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=376797577}) 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410432692}) 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1914147741}) 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410432692}) 2024-08-19T12:47:00.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1527658040}) 2024-08-19T12:47:00.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410432692}) 2024-08-19T12:47:00.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=435130796}) 2024-08-19T12:47:00.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410432692}) 2024-08-19T12:47:00.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1264664903}) 2024-08-19T12:47:00.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410432692}) 2024-08-19T12:47:00.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3937213787}) 2024-08-19T12:47:00.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410432692}) 2024-08-19T12:47:00.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=699273435}) 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410432692}) 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2588867717}) 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410432692}) 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.950+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=982230439}) 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=937362759}) 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=851530308}) 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=937362759}) 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3397839958}) 2024-08-19T12:47:00.985 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=937362759}) 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3342064749}) 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=937362759}) 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2626962906}) 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=937362759}) 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2495790589}) 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=937362759}) 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4129490539}) 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.986 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=937362759}) 2024-08-19T12:47:00.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2271853939}) 2024-08-19T12:47:00.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=937362759}) 2024-08-19T12:47:00.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.952+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=86985979}) 2024-08-19T12:47:00.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3997910038}) 2024-08-19T12:47:00.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3523679952}) 2024-08-19T12:47:00.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3997910038}) 2024-08-19T12:47:00.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4243433457}) 2024-08-19T12:47:00.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.987 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3997910038}) 2024-08-19T12:47:00.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3770140331}) 2024-08-19T12:47:00.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3997910038}) 2024-08-19T12:47:00.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4160800577}) 2024-08-19T12:47:00.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3997910038}) 2024-08-19T12:47:00.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=846241164}) 2024-08-19T12:47:00.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3997910038}) 2024-08-19T12:47:00.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2992176950}) 2024-08-19T12:47:00.988 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3997910038}) 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2453106003}) 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3997910038}) 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.953+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1268087205}) 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136765533}) 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1311317788}) 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136765533}) 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4110871087}) 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136765533}) 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3612068591}) 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136765533}) 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=392869553}) 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136765533}) 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=456931384}) 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136765533}) 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3304977193}) 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136765533}) 2024-08-19T12:47:00.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2161535876}) 2024-08-19T12:47:00.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136765533}) 2024-08-19T12:47:00.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.954+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2830027709}) 2024-08-19T12:47:00.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.955+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.955+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548125890}) 2024-08-19T12:47:00.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.955+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924309246}) 2024-08-19T12:47:00.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.955+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.955+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548125890}) 2024-08-19T12:47:00.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2450380122}) 2024-08-19T12:47:00.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548125890}) 2024-08-19T12:47:00.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1765570664}) 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548125890}) 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1670203984}) 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548125890}) 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4144639696}) 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548125890}) 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3638348501}) 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548125890}) 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=931029248}) 2024-08-19T12:47:00.992 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548125890}) 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.956+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2644476551}) 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227429930}) 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3988637704}) 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227429930}) 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2040569233}) 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227429930}) 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2742905402}) 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227429930}) 2024-08-19T12:47:00.993 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3769605142}) 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227429930}) 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4070469688}) 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227429930}) 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=29216610}) 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227429930}) 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1703869335}) 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227429930}) 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4115547936}) 2024-08-19T12:47:00.994 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072230638}) 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=14955339}) 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072230638}) 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3707924882}) 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072230638}) 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1670716074}) 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072230638}) 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3472457030}) 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.995 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072230638}) 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2661212055}) 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072230638}) 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3995021963}) 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072230638}) 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1984912195}) 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072230638}) 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3031753798}) 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3819564791}) 2024-08-19T12:47:00.996 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3670581398}) 2024-08-19T12:47:00.997 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.997 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3819564791}) 2024-08-19T12:47:00.997 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973224452}) 2024-08-19T12:47:00.997 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.997 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3819564791}) 2024-08-19T12:47:00.997 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=101964827}) 2024-08-19T12:47:00.997 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.997 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3819564791}) 2024-08-19T12:47:00.997 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3397814987}) 2024-08-19T12:47:00.997 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.997 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3819564791}) 2024-08-19T12:47:00.997 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2912119223}) 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3819564791}) 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2231554433}) 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3819564791}) 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2592028380}) 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3819564791}) 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.960+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1904572981}) 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939752895}) 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2567769532}) 2024-08-19T12:47:00.998 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939752895}) 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3501627573}) 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939752895}) 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1057262655}) 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939752895}) 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2655153108}) 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939752895}) 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4259895592}) 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:00.999 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939752895}) 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=881872892}) 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939752895}) 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3287807810}) 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939752895}) 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1301518879}) 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640861399}) 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1990594140}) 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640861399}) 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2601958501}) 2024-08-19T12:47:01.000 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.001 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640861399}) 2024-08-19T12:47:01.001 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1171956352}) 2024-08-19T12:47:01.001 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.001 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640861399}) 2024-08-19T12:47:01.001 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=678003030}) 2024-08-19T12:47:01.001 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.001 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640861399}) 2024-08-19T12:47:01.001 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3341172722}) 2024-08-19T12:47:01.001 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.001 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640861399}) 2024-08-19T12:47:01.001 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1374972930}) 2024-08-19T12:47:01.001 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.002 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640861399}) 2024-08-19T12:47:01.002 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1039210877}) 2024-08-19T12:47:01.002 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.002 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640861399}) 2024-08-19T12:47:01.002 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.962+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2185373727}) 2024-08-19T12:47:01.002 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.002 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876324453}) 2024-08-19T12:47:01.002 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755930927}) 2024-08-19T12:47:01.002 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.002 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876324453}) 2024-08-19T12:47:01.002 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3856384656}) 2024-08-19T12:47:01.002 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876324453}) 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1610022584}) 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.963+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876324453}) 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2992769166}) 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876324453}) 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2735412102}) 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876324453}) 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3364605830}) 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876324453}) 2024-08-19T12:47:01.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=977753035}) 2024-08-19T12:47:01.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876324453}) 2024-08-19T12:47:01.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.964+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=716024474}) 2024-08-19T12:47:01.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855357841}) 2024-08-19T12:47:01.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2244534793}) 2024-08-19T12:47:01.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855357841}) 2024-08-19T12:47:01.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=936065018}) 2024-08-19T12:47:01.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855357841}) 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=843936357}) 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855357841}) 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1300203631}) 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855357841}) 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3013269130}) 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855357841}) 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=434323880}) 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855357841}) 2024-08-19T12:47:01.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2389441097}) 2024-08-19T12:47:01.006 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.006 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855357841}) 2024-08-19T12:47:01.006 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.965+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3665706181}) 2024-08-19T12:47:01.006 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.006 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185543591}) 2024-08-19T12:47:01.006 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=504206561}) 2024-08-19T12:47:01.006 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.006 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185543591}) 2024-08-19T12:47:01.006 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3215515267}) 2024-08-19T12:47:01.006 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185543591}) 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2079204795}) 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185543591}) 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4087756600}) 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185543591}) 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1261669614}) 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185543591}) 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1195174162}) 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.007 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185543591}) 2024-08-19T12:47:01.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2221920960}) 2024-08-19T12:47:01.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185543591}) 2024-08-19T12:47:01.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.966+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3663588444}) 2024-08-19T12:47:01.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558202139}) 2024-08-19T12:47:01.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2387669823}) 2024-08-19T12:47:01.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558202139}) 2024-08-19T12:47:01.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3379985848}) 2024-08-19T12:47:01.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558202139}) 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3667243386}) 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558202139}) 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2531816491}) 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558202139}) 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1638951348}) 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558202139}) 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=252625220}) 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558202139}) 2024-08-19T12:47:01.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2585755722}) 2024-08-19T12:47:01.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558202139}) 2024-08-19T12:47:01.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.968+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3944020580}) 2024-08-19T12:47:01.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110512594}) 2024-08-19T12:47:01.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1646276404}) 2024-08-19T12:47:01.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110512594}) 2024-08-19T12:47:01.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179650798}) 2024-08-19T12:47:01.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110512594}) 2024-08-19T12:47:01.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4228969337}) 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110512594}) 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2975470499}) 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110512594}) 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4131252030}) 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110512594}) 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2138200298}) 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110512594}) 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2895023615}) 2024-08-19T12:47:01.011 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110512594}) 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.969+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1373722740}) 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843104821}) 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3472939143}) 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843104821}) 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001528696}) 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843104821}) 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1460004058}) 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.012 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843104821}) 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2209349462}) 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.970+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.971+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843104821}) 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.971+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1826939218}) 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.971+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.971+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843104821}) 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.971+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=931494687}) 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.971+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.971+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843104821}) 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.971+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1894218615}) 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.971+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.971+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843104821}) 2024-08-19T12:47:01.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:00.971+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3484002500}) 2024-08-19T12:47:01.208 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.209 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242600027}) 2024-08-19T12:47:01.209 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2170024292}) 2024-08-19T12:47:01.209 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.209 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242600027}) 2024-08-19T12:47:01.209 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3486239467}) 2024-08-19T12:47:01.210 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.210 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242600027}) 2024-08-19T12:47:01.210 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2616072993}) 2024-08-19T12:47:01.210 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.210 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242600027}) 2024-08-19T12:47:01.210 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2563294041}) 2024-08-19T12:47:01.210 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.210 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242600027}) 2024-08-19T12:47:01.210 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3197677419}) 2024-08-19T12:47:01.211 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.211 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242600027}) 2024-08-19T12:47:01.211 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=742519335}) 2024-08-19T12:47:01.211 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.189+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.211 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.190+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242600027}) 2024-08-19T12:47:01.211 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.190+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3841441586}) 2024-08-19T12:47:01.211 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.190+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.211 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.190+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242600027}) 2024-08-19T12:47:01.211 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.190+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1821520034}) 2024-08-19T12:47:01.211 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867702152}) 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4187033865}) 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867702152}) 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1055354554}) 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867702152}) 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=939217647}) 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867702152}) 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3021767960}) 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867702152}) 2024-08-19T12:47:01.212 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=394549806}) 2024-08-19T12:47:01.213 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.213 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867702152}) 2024-08-19T12:47:01.213 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1727345004}) 2024-08-19T12:47:01.213 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.213 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867702152}) 2024-08-19T12:47:01.213 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3861363541}) 2024-08-19T12:47:01.213 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.213 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867702152}) 2024-08-19T12:47:01.213 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.191+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2753171028}) 2024-08-19T12:47:01.213 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.213 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791894851}) 2024-08-19T12:47:01.214 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3403579531}) 2024-08-19T12:47:01.214 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.214 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791894851}) 2024-08-19T12:47:01.214 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1860138829}) 2024-08-19T12:47:01.214 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.214 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791894851}) 2024-08-19T12:47:01.214 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2003874768}) 2024-08-19T12:47:01.214 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.214 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791894851}) 2024-08-19T12:47:01.214 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2731214797}) 2024-08-19T12:47:01.214 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.214 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791894851}) 2024-08-19T12:47:01.215 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=755962593}) 2024-08-19T12:47:01.215 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.215 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791894851}) 2024-08-19T12:47:01.215 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3238003807}) 2024-08-19T12:47:01.215 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.215 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791894851}) 2024-08-19T12:47:01.215 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1245008559}) 2024-08-19T12:47:01.215 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.215 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791894851}) 2024-08-19T12:47:01.215 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.192+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3528815563}) 2024-08-19T12:47:01.215 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519751613}) 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094998833}) 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519751613}) 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=973326462}) 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519751613}) 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1985454663}) 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519751613}) 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=289402860}) 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519751613}) 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1684656030}) 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519751613}) 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2070389483}) 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519751613}) 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3719398634}) 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519751613}) 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.194+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2906488436}) 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803299575}) 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3777893155}) 2024-08-19T12:47:01.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803299575}) 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2381242177}) 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803299575}) 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3428704092}) 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803299575}) 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1635139707}) 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803299575}) 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=875444275}) 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803299575}) 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=365070559}) 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803299575}) 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1211799740}) 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803299575}) 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.195+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1610169493}) 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704342795}) 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1881202861}) 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704342795}) 2024-08-19T12:47:01.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382479919}) 2024-08-19T12:47:01.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704342795}) 2024-08-19T12:47:01.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1577232680}) 2024-08-19T12:47:01.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704342795}) 2024-08-19T12:47:01.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2212701923}) 2024-08-19T12:47:01.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704342795}) 2024-08-19T12:47:01.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2405650373}) 2024-08-19T12:47:01.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704342795}) 2024-08-19T12:47:01.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3068952449}) 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704342795}) 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3552719820}) 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704342795}) 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.197+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2895482668}) 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944343230}) 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=654384595}) 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944343230}) 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2390093141}) 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944343230}) 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=347220971}) 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944343230}) 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3288971851}) 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944343230}) 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2233241347}) 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944343230}) 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2182808417}) 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944343230}) 2024-08-19T12:47:01.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2984802437}) 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944343230}) 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.198+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3324851729}) 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413045412}) 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=71437145}) 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413045412}) 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2758286041}) 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413045412}) 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1969042780}) 2024-08-19T12:47:01.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413045412}) 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1779873293}) 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413045412}) 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=45140770}) 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413045412}) 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1565617570}) 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413045412}) 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3297354667}) 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.224 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413045412}) 2024-08-19T12:47:01.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.199+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1420121590}) 2024-08-19T12:47:01.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946660563}) 2024-08-19T12:47:01.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1495527929}) 2024-08-19T12:47:01.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946660563}) 2024-08-19T12:47:01.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2099935366}) 2024-08-19T12:47:01.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946660563}) 2024-08-19T12:47:01.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2403858554}) 2024-08-19T12:47:01.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946660563}) 2024-08-19T12:47:01.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3122563090}) 2024-08-19T12:47:01.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946660563}) 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1455040942}) 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946660563}) 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3171751550}) 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946660563}) 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3265769692}) 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946660563}) 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.211+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=744597813}) 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.212+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.212+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425538779}) 2024-08-19T12:47:01.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.212+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3628174948}) 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.212+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.212+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425538779}) 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.212+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3209666897}) 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.212+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.212+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425538779}) 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.212+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=962897964}) 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425538779}) 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2974168532}) 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425538779}) 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=998614586}) 2024-08-19T12:47:01.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425538779}) 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4000540846}) 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425538779}) 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=808736455}) 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425538779}) 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=316054676}) 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220500907}) 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4204676392}) 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220500907}) 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=110158097}) 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220500907}) 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1918212963}) 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220500907}) 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2886274343}) 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220500907}) 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1698319059}) 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220500907}) 2024-08-19T12:47:01.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=60079899}) 2024-08-19T12:47:01.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220500907}) 2024-08-19T12:47:01.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4206341707}) 2024-08-19T12:47:01.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220500907}) 2024-08-19T12:47:01.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.214+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2753785005}) 2024-08-19T12:47:01.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233812321}) 2024-08-19T12:47:01.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1970334878}) 2024-08-19T12:47:01.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233812321}) 2024-08-19T12:47:01.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2469767423}) 2024-08-19T12:47:01.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233812321}) 2024-08-19T12:47:01.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3529486320}) 2024-08-19T12:47:01.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233812321}) 2024-08-19T12:47:01.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=959154746}) 2024-08-19T12:47:01.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233812321}) 2024-08-19T12:47:01.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=776659554}) 2024-08-19T12:47:01.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233812321}) 2024-08-19T12:47:01.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2219055456}) 2024-08-19T12:47:01.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233812321}) 2024-08-19T12:47:01.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=371488554}) 2024-08-19T12:47:01.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233812321}) 2024-08-19T12:47:01.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.215+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2995431570}) 2024-08-19T12:47:01.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821583153}) 2024-08-19T12:47:01.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816177116}) 2024-08-19T12:47:01.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821583153}) 2024-08-19T12:47:01.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4190163444}) 2024-08-19T12:47:01.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821583153}) 2024-08-19T12:47:01.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3536377136}) 2024-08-19T12:47:01.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821583153}) 2024-08-19T12:47:01.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=618295132}) 2024-08-19T12:47:01.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821583153}) 2024-08-19T12:47:01.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2886389663}) 2024-08-19T12:47:01.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821583153}) 2024-08-19T12:47:01.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3081626923}) 2024-08-19T12:47:01.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821583153}) 2024-08-19T12:47:01.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=605479235}) 2024-08-19T12:47:01.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821583153}) 2024-08-19T12:47:01.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.218+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1625104768}) 2024-08-19T12:47:01.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582147986}) 2024-08-19T12:47:01.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3525880386}) 2024-08-19T12:47:01.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582147986}) 2024-08-19T12:47:01.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2297790922}) 2024-08-19T12:47:01.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582147986}) 2024-08-19T12:47:01.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1558246531}) 2024-08-19T12:47:01.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582147986}) 2024-08-19T12:47:01.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4184606776}) 2024-08-19T12:47:01.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582147986}) 2024-08-19T12:47:01.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=30430208}) 2024-08-19T12:47:01.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582147986}) 2024-08-19T12:47:01.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1191035391}) 2024-08-19T12:47:01.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582147986}) 2024-08-19T12:47:01.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3831617633}) 2024-08-19T12:47:01.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582147986}) 2024-08-19T12:47:01.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1056882414}) 2024-08-19T12:47:01.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.261+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431791530}) 2024-08-19T12:47:01.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2995404944}) 2024-08-19T12:47:01.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431791530}) 2024-08-19T12:47:01.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3742334221}) 2024-08-19T12:47:01.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431791530}) 2024-08-19T12:47:01.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3459029645}) 2024-08-19T12:47:01.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431791530}) 2024-08-19T12:47:01.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3994563466}) 2024-08-19T12:47:01.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431791530}) 2024-08-19T12:47:01.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2296846493}) 2024-08-19T12:47:01.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431791530}) 2024-08-19T12:47:01.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1248077035}) 2024-08-19T12:47:01.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431791530}) 2024-08-19T12:47:01.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=922950672}) 2024-08-19T12:47:01.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431791530}) 2024-08-19T12:47:01.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.262+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=174923569}) 2024-08-19T12:47:01.272 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.269+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.272 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.269+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404866726}) 2024-08-19T12:47:01.272 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.269+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3639348902}) 2024-08-19T12:47:01.272 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.269+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.272 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.269+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404866726}) 2024-08-19T12:47:01.272 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.269+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3053230499}) 2024-08-19T12:47:01.272 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.269+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.272 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.269+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404866726}) 2024-08-19T12:47:01.272 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4257841841}) 2024-08-19T12:47:01.272 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.272 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404866726}) 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3525457277}) 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404866726}) 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2366010832}) 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404866726}) 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=602281711}) 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404866726}) 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4026708306}) 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3404866726}) 2024-08-19T12:47:01.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.270+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1587965513}) 2024-08-19T12:47:01.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2746925375}) 2024-08-19T12:47:01.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=448911479}) 2024-08-19T12:47:01.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2746925375}) 2024-08-19T12:47:01.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=421242270}) 2024-08-19T12:47:01.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2746925375}) 2024-08-19T12:47:01.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=306659852}) 2024-08-19T12:47:01.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2746925375}) 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=540245350}) 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2746925375}) 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2237933475}) 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2746925375}) 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2930432924}) 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2746925375}) 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3882604911}) 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2746925375}) 2024-08-19T12:47:01.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3169105369}) 2024-08-19T12:47:01.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886309074}) 2024-08-19T12:47:01.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2031023417}) 2024-08-19T12:47:01.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.802 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886309074}) 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1329845953}) 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886309074}) 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=796064737}) 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886309074}) 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3442964736}) 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886309074}) 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2504463758}) 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886309074}) 2024-08-19T12:47:01.803 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3679669031}) 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886309074}) 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4233614017}) 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886309074}) 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.795+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=742237038}) 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3316753098}) 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060971405}) 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3316753098}) 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2212790984}) 2024-08-19T12:47:01.804 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3316753098}) 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=390891935}) 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3316753098}) 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1902754608}) 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3316753098}) 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=715793901}) 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3316753098}) 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2085888512}) 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3316753098}) 2024-08-19T12:47:01.805 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.796+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4129674318}) 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3316753098}) 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.797+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=189595319}) 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1488328833}) 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=687120247}) 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1488328833}) 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2133497426}) 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1488328833}) 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2636545986}) 2024-08-19T12:47:01.806 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1488328833}) 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3173944301}) 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1488328833}) 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3520163884}) 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1488328833}) 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4177186675}) 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1488328833}) 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1458736833}) 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.807 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1488328833}) 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.798+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3786185107}) 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227884237}) 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2821367090}) 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227884237}) 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4170423160}) 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227884237}) 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3115341980}) 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227884237}) 2024-08-19T12:47:01.808 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1679918772}) 2024-08-19T12:47:01.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227884237}) 2024-08-19T12:47:01.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2735205834}) 2024-08-19T12:47:01.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227884237}) 2024-08-19T12:47:01.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1363152276}) 2024-08-19T12:47:01.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227884237}) 2024-08-19T12:47:01.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=121917193}) 2024-08-19T12:47:01.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:01.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227884237}) 2024-08-19T12:47:01.809 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:01.799+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2046183714}) 2024-08-19T12:47:01.906 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:47:01.906 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:47:02.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.012+0000 7f921b8f6640 10 monclient: get_monmap_and_config 2024-08-19T12:47:02.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.012+0000 7f921b8f6640 10 monclient: build_initial_monmap 2024-08-19T12:47:02.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.012+0000 7f921b8f6640 10 monclient: monmap: 2024-08-19T12:47:02.013 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:02.013 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.013 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:02.012962+0000 2024-08-19T12:47:02.013 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:02.012962+0000 2024-08-19T12:47:02.013 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:02.013 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.013 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:02.013 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:02.013 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:02.014 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:02.014 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:02.014 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:02.014 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:02.014 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:02.014 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:02.014 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.012+0000 7f921b8f6640 1 Processor -- start 2024-08-19T12:47:02.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.012+0000 7f921b8f6640 1 -- start start 2024-08-19T12:47:02.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.012+0000 7f921b8f6640 10 monclient: init 2024-08-19T12:47:02.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f921b8f6640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:02.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f921b8f6640 10 monclient: _add_conns ranks=[8,1,2,4,7,3,6,0,5] 2024-08-19T12:47:02.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f921b8f6640 10 monclient(hunting): picked mon.noname-e con 0x7f9214073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:02.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f921b8f6640 10 monclient(hunting): picked mon.noname-f con 0x7f92140a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:02.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f921b8f6640 10 monclient(hunting): picked mon.noname-b con 0x7f921413f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:02.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f921b8f6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921409dae0 con 0x7f92140a57a0 2024-08-19T12:47:02.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f921b8f6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9214136910 con 0x7f921413f2e0 2024-08-19T12:47:02.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f921b8f6640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921409dd50 con 0x7f9214073470 2024-08-19T12:47:02.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f921b8f6640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:02.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f921b8f6640 10 monclient(hunting): authenticate will time out at 1216.100476s 2024-08-19T12:47:02.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f9219692640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f9214073470 0x7f9214136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:56158/0 (socket says 172.21.15.26:56158) 2024-08-19T12:47:02.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f9219692640 1 -- 172.21.15.26:0/4054199322 learned_addr learned my addr 172.21.15.26:0/4054199322 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:02.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.013+0000 7f9219e93640 1 --1- 172.21.15.26:0/4054199322 >> v1:172.21.15.26:6790/0 conn(0x7f921413f2e0 0x7f92141416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39002/0 (socket says 172.21.15.26:39002) 2024-08-19T12:47:02.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9219692640 1 -- 172.21.15.26:0/4054199322 >> v1:172.21.15.26:6793/0 conn(0x7f9214073470 legacy=0x7f9214136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9219e93640 1 -- 172.21.15.26:0/4054199322 >> v1:172.21.15.26:6790/0 conn(0x7f921413f2e0 legacy=0x7f92141416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9213fff640 1 -- 172.21.15.26:0/4054199322 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 230168744 0 0) 0x7f9214136910 con 0x7f921413f2e0 2024-08-19T12:47:02.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9213fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9213fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9213fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9213fff640 10 monclient(hunting): my global_id is 26516 2024-08-19T12:47:02.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9213fff640 1 -- 172.21.15.26:0/4054199322 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91fc003520 con 0x7f921413f2e0 2024-08-19T12:47:02.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9213fff640 1 -- 172.21.15.26:0/4054199322 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3684716538 0 0) 0x7f921409dd50 con 0x7f9214073470 2024-08-19T12:47:02.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9213fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9213fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9213fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.015+0000 7f9213fff640 10 monclient(hunting): my global_id is 26378 2024-08-19T12:47:02.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 1 -- 172.21.15.26:0/4054199322 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9214136910 con 0x7f9214073470 2024-08-19T12:47:02.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9218e91640 1 -- 172.21.15.26:0/4054199322 >> v1:172.21.15.104:6789/0 conn(0x7f92140a57a0 legacy=0x7f92140a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 1 -- 172.21.15.26:0/4054199322 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3842398802 0 0) 0x7f9214136910 con 0x7f9214073470 2024-08-19T12:47:02.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 1 -- 172.21.15.26:0/4054199322 >> v1:172.21.15.26:6790/0 conn(0x7f921413f2e0 legacy=0x7f92141416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 1 -- 172.21.15.26:0/4054199322 >> v1:172.21.15.104:6789/0 conn(0x7f92140a57a0 legacy=0x7f92140a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 10 monclient: _finish_hunting 0 2024-08-19T12:47:02.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 1 monclient: found mon.noname-e 2024-08-19T12:47:02.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:02.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:47:02.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 1 -- 172.21.15.26:0/4054199322 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921409e030 con 0x7f9214073470 2024-08-19T12:47:02.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 10 monclient: _finish_auth 0 2024-08-19T12:47:02.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 10 monclient: _check_auth_tickets 2024-08-19T12:47:02.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:02.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 1 -- 172.21.15.26:0/4054199322 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9204001a40 con 0x7f9214073470 2024-08-19T12:47:02.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:47:02.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 10 monclient: dump: 2024-08-19T12:47:02.024 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.024 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.024 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.024 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.024 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.024 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f921b8f6640 5 monclient: authenticate success, global_id 26378 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f921b8f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:02.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 1 -- 172.21.15.26:0/4054199322 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9204001d10 con 0x7f9214073470 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 1 -- 172.21.15.26:0/4054199322 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9204004be0 con 0x7f9214073470 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f921b8f6640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f921b8f6640 4 set_mon_vals no callback set 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f9213fff640 10 monclient: dump: 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.026 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f921b8f6640 10 monclient: shutdown 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f921b8f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f921b8f6640 1 -- 172.21.15.26:0/4054199322 >> v1:172.21.15.26:6793/0 conn(0x7f9214073470 legacy=0x7f9214136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f921b8f6640 1 -- 172.21.15.26:0/4054199322 shutdown_connections 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.016+0000 7f921b8f6640 1 -- 172.21.15.26:0/4054199322 >> 172.21.15.26:0/4054199322 conn(0x7f92140abff0 msgr2=0x7f92140ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.017+0000 7f921b8f6640 1 -- 172.21.15.26:0/4054199322 shutdown_connections 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.017+0000 7f921b8f6640 1 -- 172.21.15.26:0/4054199322 wait complete. 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.017+0000 7f921b8f6640 10 monclient: build_initial_monmap 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.017+0000 7f921b8f6640 10 monclient: monmap: 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:02.028 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:02.029 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:02.029 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:02.029 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:02.029 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:02.029 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:02.029 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:02.029 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.017+0000 7f921b8f6640 1 librados: starting msgr at 2024-08-19T12:47:02.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.017+0000 7f921b8f6640 1 librados: starting objecter 2024-08-19T12:47:02.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.017+0000 7f921b8f6640 1 Processor -- start 2024-08-19T12:47:02.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.017+0000 7f921b8f6640 1 -- start start 2024-08-19T12:47:02.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.017+0000 7f921b8f6640 1 librados: setting wanted keys 2024-08-19T12:47:02.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.017+0000 7f921b8f6640 1 librados: calling monclient init 2024-08-19T12:47:02.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.017+0000 7f921b8f6640 10 monclient: init 2024-08-19T12:47:02.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.018+0000 7f921b8f6640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:02.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.018+0000 7f921b8f6640 10 monclient: _add_conns ranks=[7,0,3,5,2,1,4,8,6] 2024-08-19T12:47:02.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.018+0000 7f921b8f6640 10 monclient(hunting): picked mon.noname-i con 0x7f9214073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:47:02.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.018+0000 7f921b8f6640 10 monclient(hunting): picked mon.noname-a con 0x7f92140a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:02.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.018+0000 7f921b8f6640 10 monclient(hunting): picked mon.noname-g con 0x7f921413f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:02.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.018+0000 7f921b8f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9214137280 con 0x7f92140a57a0 2024-08-19T12:47:02.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.018+0000 7f921b8f6640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92141cef20 con 0x7f921413f2e0 2024-08-19T12:47:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.018+0000 7f921b8f6640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921413f0b0 con 0x7f9214073470 2024-08-19T12:47:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.018+0000 7f921b8f6640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.018+0000 7f921b8f6640 10 monclient(hunting): authenticate will time out at 1216.105422s 2024-08-19T12:47:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.018+0000 7f9218e91640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f92140a57a0 0x7f92141ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47978/0 (socket says 172.21.15.26:47978) 2024-08-19T12:47:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.018+0000 7f9218e91640 1 -- 172.21.15.26:0/1879974607 learned_addr learned my addr 172.21.15.26:0/1879974607 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.020+0000 7f9218e91640 1 -- 172.21.15.26:0/1879974607 >> v1:172.21.15.26:6789/0 conn(0x7f92140a57a0 legacy=0x7f92141ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.020+0000 7f9211ffb640 10 client.?.objecter ms_handle_connect 0x7f92140a57a0 2024-08-19T12:47:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.020+0000 7f9211ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.020+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3679990438 0 0) 0x7f9214137280 con 0x7f92140a57a0 2024-08-19T12:47:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.020+0000 7f9211ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.020+0000 7f9211ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.020+0000 7f9211ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.020+0000 7f9211ffb640 10 monclient(hunting): my global_id is 144828 2024-08-19T12:47:02.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.020+0000 7f9219692640 1 -- 172.21.15.26:0/1879974607 >> v1:172.21.15.104:6792/0 conn(0x7f9214073470 legacy=0x7f92141ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.020+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9200004eb0 con 0x7f92140a57a0 2024-08-19T12:47:02.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.020+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:47:02.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.020+0000 7f9219e93640 1 -- 172.21.15.26:0/1879974607 >> v1:172.21.15.104:6790/0 conn(0x7f921413f2e0 legacy=0x7f92141d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 client.?.objecter ms_handle_connect 0x7f9214073470 2024-08-19T12:47:02.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 client.?.objecter ms_handle_connect 0x7f921413f2e0 2024-08-19T12:47:02.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1874580019 0 0) 0x7f921413f0b0 con 0x7f9214073470 2024-08-19T12:47:02.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 monclient(hunting): my global_id is 135502 2024-08-19T12:47:02.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9214137280 con 0x7f9214073470 2024-08-19T12:47:02.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 80589313 0 0) 0x7f9200004eb0 con 0x7f92140a57a0 2024-08-19T12:47:02.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f921413f0b0 con 0x7f92140a57a0 2024-08-19T12:47:02.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1544254832 0 0) 0x7f92141cef20 con 0x7f921413f2e0 2024-08-19T12:47:02.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 monclient(hunting): my global_id is 154371 2024-08-19T12:47:02.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9200004eb0 con 0x7f921413f2e0 2024-08-19T12:47:02.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9208003610 con 0x7f92140a57a0 2024-08-19T12:47:02.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:02.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:47:02.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 monclient(hunting): dump: 2024-08-19T12:47:02.038 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.038 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.038 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.038 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.038 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.038 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.022+0000 7f9211ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4114622561 0 0) 0x7f921413f0b0 con 0x7f92140a57a0 2024-08-19T12:47:02.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 >> v1:172.21.15.104:6792/0 conn(0x7f9214073470 legacy=0x7f92141ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 >> v1:172.21.15.104:6790/0 conn(0x7f921413f2e0 legacy=0x7f92141d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:47:02.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 1 monclient: found mon.a 2024-08-19T12:47:02.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:02.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:02.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9214142b60 con 0x7f92140a57a0 2024-08-19T12:47:02.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 10 monclient: _finish_auth 0 2024-08-19T12:47:02.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:47:02.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f921b8f6640 5 monclient: authenticate success, global_id 144828 2024-08-19T12:47:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f921b8f6640 10 monclient: _renew_subs 2024-08-19T12:47:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f921b8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f921b8f6640 1 -- 172.21.15.26:0/1879974607 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9214136fa0 con 0x7f92140a57a0 2024-08-19T12:47:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9208003bb0 con 0x7f92140a57a0 2024-08-19T12:47:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f92080040e0 con 0x7f92140a57a0 2024-08-19T12:47:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f9211ffb640 10 monclient: dump: 2024-08-19T12:47:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.041 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.042 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.043 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f92137fe640 4 set_mon_vals no callback set 2024-08-19T12:47:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f921b8f6640 10 client.144828.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f921b8f6640 10 monclient: _renew_subs 2024-08-19T12:47:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f921b8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f921b8f6640 1 -- 172.21.15.26:0/1879974607 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f92140a30d0 con 0x7f92140a57a0 2024-08-19T12:47:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.023+0000 7f921b8f6640 1 librados: init done 2024-08-19T12:47:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.024+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f9208004280 con 0x7f92140a57a0 2024-08-19T12:47:02.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.024+0000 7f921b8f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.024+0000 7f921b8f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.024+0000 7f921b8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.024+0000 7f921b8f6640 1 -- 172.21.15.26:0/1879974607 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92140008d0 con 0x7f92140a57a0 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.024+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(352..352 src has 1..352) ==== 5333+0+0 (unknown 318244571 0 0) 0x7f920804e1d0 con 0x7f92140a57a0 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.024+0000 7f9211ffb640 10 client.144828.objecter ms_dispatch 0x7f9214000b90 osd_map(352..352 src has 1..352) 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.024+0000 7f9211ffb640 3 client.144828.objecter handle_osd_map got epochs [352,352] > 0 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.024+0000 7f9211ffb640 3 client.144828.objecter handle_osd_map decoding full epoch 352 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.024+0000 7f9211ffb640 20 client.144828.objecter dump_active .. 0 homeless 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.024+0000 7f9219692640 10 monclient: get_auth_request con 0x7f9200041290 auth_method 0 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.027+0000 7f9211ffb640 10 client.144828.objecter ms_handle_connect 0x7f9200041290 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.029+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f9208017660 con 0x7f92140a57a0 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.029+0000 7f9211ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:02.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.029+0000 7f9211ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:02.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1014735678}) 2024-08-19T12:47:02.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3505850522}) 2024-08-19T12:47:02.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1014735678}) 2024-08-19T12:47:02.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696147538}) 2024-08-19T12:47:02.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1014735678}) 2024-08-19T12:47:02.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3457628655}) 2024-08-19T12:47:02.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1014735678}) 2024-08-19T12:47:02.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1457617414}) 2024-08-19T12:47:02.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1014735678}) 2024-08-19T12:47:02.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3159448203}) 2024-08-19T12:47:02.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1014735678}) 2024-08-19T12:47:02.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2957375517}) 2024-08-19T12:47:02.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1014735678}) 2024-08-19T12:47:02.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2455033114}) 2024-08-19T12:47:02.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1014735678}) 2024-08-19T12:47:02.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.106+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=212507617}) 2024-08-19T12:47:02.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860248660}) 2024-08-19T12:47:02.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3709408626}) 2024-08-19T12:47:02.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860248660}) 2024-08-19T12:47:02.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2573903698}) 2024-08-19T12:47:02.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860248660}) 2024-08-19T12:47:02.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2639562164}) 2024-08-19T12:47:02.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860248660}) 2024-08-19T12:47:02.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3295165060}) 2024-08-19T12:47:02.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860248660}) 2024-08-19T12:47:02.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1827356109}) 2024-08-19T12:47:02.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860248660}) 2024-08-19T12:47:02.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4204645613}) 2024-08-19T12:47:02.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860248660}) 2024-08-19T12:47:02.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=918321815}) 2024-08-19T12:47:02.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860248660}) 2024-08-19T12:47:02.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.108+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1022310911}) 2024-08-19T12:47:02.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205773727}) 2024-08-19T12:47:02.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584964882}) 2024-08-19T12:47:02.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205773727}) 2024-08-19T12:47:02.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3468405194}) 2024-08-19T12:47:02.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205773727}) 2024-08-19T12:47:02.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1958771379}) 2024-08-19T12:47:02.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205773727}) 2024-08-19T12:47:02.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2203074056}) 2024-08-19T12:47:02.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205773727}) 2024-08-19T12:47:02.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=69818573}) 2024-08-19T12:47:02.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.120 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205773727}) 2024-08-19T12:47:02.120 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4078931698}) 2024-08-19T12:47:02.120 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.120 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205773727}) 2024-08-19T12:47:02.120 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1390478843}) 2024-08-19T12:47:02.120 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.120 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205773727}) 2024-08-19T12:47:02.120 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.109+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=981255839}) 2024-08-19T12:47:02.230 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.230 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908418075}) 2024-08-19T12:47:02.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4277846414}) 2024-08-19T12:47:02.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908418075}) 2024-08-19T12:47:02.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3099449722}) 2024-08-19T12:47:02.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908418075}) 2024-08-19T12:47:02.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=233057562}) 2024-08-19T12:47:02.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908418075}) 2024-08-19T12:47:02.232 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1664941980}) 2024-08-19T12:47:02.232 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.232 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908418075}) 2024-08-19T12:47:02.232 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3569497549}) 2024-08-19T12:47:02.232 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.232 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908418075}) 2024-08-19T12:47:02.232 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3116347622}) 2024-08-19T12:47:02.232 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.232 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908418075}) 2024-08-19T12:47:02.232 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1548740296}) 2024-08-19T12:47:02.232 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.233 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908418075}) 2024-08-19T12:47:02.233 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.228+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3066393483}) 2024-08-19T12:47:02.235 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.235 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3171569307}) 2024-08-19T12:47:02.235 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2504520506}) 2024-08-19T12:47:02.235 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.235 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3171569307}) 2024-08-19T12:47:02.235 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3862626285}) 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3171569307}) 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2099973898}) 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3171569307}) 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=560063987}) 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3171569307}) 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=638413888}) 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3171569307}) 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=269233465}) 2024-08-19T12:47:02.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3171569307}) 2024-08-19T12:47:02.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3645309909}) 2024-08-19T12:47:02.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3171569307}) 2024-08-19T12:47:02.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3883812256}) 2024-08-19T12:47:02.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3245525319}) 2024-08-19T12:47:02.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2232696737}) 2024-08-19T12:47:02.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3245525319}) 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3806416669}) 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3245525319}) 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2403912435}) 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3245525319}) 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1375235352}) 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3245525319}) 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2404747728}) 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3245525319}) 2024-08-19T12:47:02.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1652419723}) 2024-08-19T12:47:02.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3245525319}) 2024-08-19T12:47:02.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2495958631}) 2024-08-19T12:47:02.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3245525319}) 2024-08-19T12:47:02.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.276+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2570549618}) 2024-08-19T12:47:02.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122640915}) 2024-08-19T12:47:02.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=510295723}) 2024-08-19T12:47:02.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122640915}) 2024-08-19T12:47:02.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1053907990}) 2024-08-19T12:47:02.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122640915}) 2024-08-19T12:47:02.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3754511085}) 2024-08-19T12:47:02.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122640915}) 2024-08-19T12:47:02.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1011962796}) 2024-08-19T12:47:02.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122640915}) 2024-08-19T12:47:02.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2139862897}) 2024-08-19T12:47:02.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122640915}) 2024-08-19T12:47:02.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4271100424}) 2024-08-19T12:47:02.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122640915}) 2024-08-19T12:47:02.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3172692283}) 2024-08-19T12:47:02.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122640915}) 2024-08-19T12:47:02.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2223591106}) 2024-08-19T12:47:02.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.303+0000 7f921b8f6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:47:02.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.303+0000 7f921b8f6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:47:02.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.303+0000 7f921b8f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:02.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.303+0000 7f921b8f6640 1 -- 172.21.15.26:0/1879974607 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f92140a3360 con 0x7f92140a57a0 2024-08-19T12:47:02.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.312+0000 7f9211ffb640 1 -- 172.21.15.26:0/1879974607 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 3984144131) 0x7f9208016f60 con 0x7f92140a57a0 2024-08-19T12:47:02.313 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":52,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:47:02.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.312+0000 7f9211ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:47:02.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.312+0000 7f9211ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:02.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.316+0000 7f921b8f6640 10 librados: watch_flush enter 2024-08-19T12:47:02.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.317+0000 7f921b8f6640 10 librados: watch_flush exit 2024-08-19T12:47:02.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.317+0000 7f921b8f6640 20 client.144828.objecter shutdown clearing up homeless session... 2024-08-19T12:47:02.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.317+0000 7f921b8f6640 10 client.144828.objecter successfully canceled tick 2024-08-19T12:47:02.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.317+0000 7f921b8f6640 1 -- 172.21.15.26:0/1879974607 >> v1:172.21.15.104:6800/899887506 conn(0x7f9200041290 legacy=0x7f9200043750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.317+0000 7f921b8f6640 10 monclient: shutdown 2024-08-19T12:47:02.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.317+0000 7f921b8f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:02.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.317+0000 7f921b8f6640 1 -- 172.21.15.26:0/1879974607 >> v1:172.21.15.26:6789/0 conn(0x7f92140a57a0 legacy=0x7f92141ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.317+0000 7f921b8f6640 1 -- 172.21.15.26:0/1879974607 shutdown_connections 2024-08-19T12:47:02.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.318+0000 7f921b8f6640 1 -- 172.21.15.26:0/1879974607 >> 172.21.15.26:0/1879974607 conn(0x7f92140abff0 msgr2=0x7f92141338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:02.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867400572}) 2024-08-19T12:47:02.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=299991555}) 2024-08-19T12:47:02.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867400572}) 2024-08-19T12:47:02.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1035082798}) 2024-08-19T12:47:02.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867400572}) 2024-08-19T12:47:02.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2853988512}) 2024-08-19T12:47:02.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867400572}) 2024-08-19T12:47:02.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3030253558}) 2024-08-19T12:47:02.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867400572}) 2024-08-19T12:47:02.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1522011762}) 2024-08-19T12:47:02.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867400572}) 2024-08-19T12:47:02.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2127560426}) 2024-08-19T12:47:02.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867400572}) 2024-08-19T12:47:02.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1124981487}) 2024-08-19T12:47:02.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:02.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867400572}) 2024-08-19T12:47:02.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:02.316+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3000566004}) 2024-08-19T12:47:02.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.320+0000 7f921b8f6640 1 -- 172.21.15.26:0/1879974607 shutdown_connections 2024-08-19T12:47:02.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.320+0000 7f921b8f6640 1 -- 172.21.15.26:0/1879974607 wait complete. 2024-08-19T12:47:02.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.321+0000 7f921b8f6640 1 librados: shutdown 2024-08-19T12:47:02.334 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:47:02.334 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T12:47:02.334 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:47:02.335 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:47:02.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.434+0000 7f06c84e5640 10 monclient: get_monmap_and_config 2024-08-19T12:47:02.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.434+0000 7f06c84e5640 10 monclient: build_initial_monmap 2024-08-19T12:47:02.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.434+0000 7f06c84e5640 10 monclient: monmap: 2024-08-19T12:47:02.435 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:02.436 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.436 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:02.435314+0000 2024-08-19T12:47:02.436 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:02.435314+0000 2024-08-19T12:47:02.436 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:02.436 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.436 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:02.436 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:02.436 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:02.436 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:02.436 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:02.436 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:02.436 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:02.437 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:02.437 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:02.437 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 1 Processor -- start 2024-08-19T12:47:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 1 -- start start 2024-08-19T12:47:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 10 monclient: init 2024-08-19T12:47:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 10 monclient: _add_conns ranks=[4,0,8,2,1,5,3,7,6] 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 10 monclient(hunting): picked mon.noname-c con 0x7f06c006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 10 monclient(hunting): picked mon.noname-a con 0x7f06c00a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 10 monclient(hunting): picked mon.noname-e con 0x7f06c00a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06c0136790 con 0x7f06c00a5920 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06c009ddf0 con 0x7f06c006e7b0 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06c009e0d0 con 0x7f06c00a5d50 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c84e5640 10 monclient(hunting): authenticate will time out at 1216.522863s 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c6281640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f06c006e7b0 0x7f06c0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:34668/0 (socket says 172.21.15.26:34668) 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c6281640 1 -- 172.21.15.26:0/1078422661 learned_addr learned my addr 172.21.15.26:0/1078422661 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.435+0000 7f06c5a80640 1 --1- 172.21.15.26:0/1078422661 >> v1:172.21.15.26:6789/0 conn(0x7f06c00a5920 0x7f06c013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47982/0 (socket says 172.21.15.26:47982) 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c6a82640 1 -- 172.21.15.26:0/1078422661 >> v1:172.21.15.26:6793/0 conn(0x7f06c00a5d50 legacy=0x7f06c0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c6281640 1 -- 172.21.15.26:0/1078422661 >> v1:172.21.15.26:6791/0 conn(0x7f06c006e7b0 legacy=0x7f06c0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c5a80640 1 -- 172.21.15.26:0/1078422661 >> v1:172.21.15.26:6789/0 conn(0x7f06c00a5920 legacy=0x7f06c013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 325844748 0 0) 0x7f06c009e0d0 con 0x7f06c00a5d50 2024-08-19T12:47:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient(hunting): my global_id is 26387 2024-08-19T12:47:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06a8003520 con 0x7f06c00a5d50 2024-08-19T12:47:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1842803951 0 0) 0x7f06b00017d0 con 0x7f06c006e7b0 2024-08-19T12:47:02.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient(hunting): my global_id is 26869 2024-08-19T12:47:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06c009e0d0 con 0x7f06c006e7b0 2024-08-19T12:47:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1199864398 0 0) 0x7f06c0136790 con 0x7f06c00a5920 2024-08-19T12:47:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient(hunting): my global_id is 144837 2024-08-19T12:47:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06b00017d0 con 0x7f06c00a5920 2024-08-19T12:47:02.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1956555573 0 0) 0x7f06a8003520 con 0x7f06c00a5d50 2024-08-19T12:47:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 >> v1:172.21.15.26:6791/0 conn(0x7f06c006e7b0 legacy=0x7f06c0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 >> v1:172.21.15.26:6789/0 conn(0x7f06c00a5920 legacy=0x7f06c013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient: _finish_hunting 0 2024-08-19T12:47:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 monclient: found mon.noname-e 2024-08-19T12:47:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:47:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c009e3b0 con 0x7f06c00a5d50 2024-08-19T12:47:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient: _finish_auth 0 2024-08-19T12:47:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient: _check_auth_tickets 2024-08-19T12:47:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06b8001b30 con 0x7f06c00a5d50 2024-08-19T12:47:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c84e5640 5 monclient: authenticate success, global_id 26387 2024-08-19T12:47:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c84e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:47:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient: dump: 2024-08-19T12:47:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.454 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c84e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f06b8003610 con 0x7f06c00a5d50 2024-08-19T12:47:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 1 -- 172.21.15.26:0/1078422661 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06b8003b80 con 0x7f06c00a5d50 2024-08-19T12:47:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c84e5640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c84e5640 4 set_mon_vals no callback set 2024-08-19T12:47:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:47:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.438+0000 7f06c527f640 10 monclient: dump: 2024-08-19T12:47:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.456 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.439+0000 7f06c84e5640 10 monclient: shutdown 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.439+0000 7f06c84e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.439+0000 7f06c84e5640 1 -- 172.21.15.26:0/1078422661 >> v1:172.21.15.26:6793/0 conn(0x7f06c00a5d50 legacy=0x7f06c0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.439+0000 7f06c84e5640 1 -- 172.21.15.26:0/1078422661 shutdown_connections 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.439+0000 7f06c84e5640 1 -- 172.21.15.26:0/1078422661 >> 172.21.15.26:0/1078422661 conn(0x7f06c00a8740 msgr2=0x7f06c00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.439+0000 7f06c84e5640 1 -- 172.21.15.26:0/1078422661 shutdown_connections 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.439+0000 7f06c84e5640 1 -- 172.21.15.26:0/1078422661 wait complete. 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.439+0000 7f06c84e5640 10 monclient: build_initial_monmap 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.439+0000 7f06c84e5640 10 monclient: monmap: 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:02.457 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:02.458 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 1 librados: starting msgr at 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 1 librados: starting objecter 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 1 Processor -- start 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 1 -- start start 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 1 librados: setting wanted keys 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 1 librados: calling monclient init 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 10 monclient: init 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 10 monclient: _add_conns ranks=[6,5,8,7,4,1,2,0,3] 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 10 monclient(hunting): picked mon.noname-d con 0x7f06c006e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 10 monclient(hunting): picked mon.noname-h con 0x7f06c00a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 10 monclient(hunting): picked mon.noname-e con 0x7f06c00a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:02.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06c013b260 con 0x7f06c00a5920 2024-08-19T12:47:02.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06c013b540 con 0x7f06c006e7b0 2024-08-19T12:47:02.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06c0134760 con 0x7f06c00a5d50 2024-08-19T12:47:02.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:02.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c84e5640 10 monclient(hunting): authenticate will time out at 1216.527813s 2024-08-19T12:47:02.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c6281640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f06c006e7b0 0x7f06c01ce450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:43106/0 (socket says 172.21.15.26:43106) 2024-08-19T12:47:02.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.440+0000 7f06c6281640 1 -- 172.21.15.26:0/4065723990 learned_addr learned my addr 172.21.15.26:0/4065723990 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:02.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.442+0000 7f06c6281640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.26:6792/0 conn(0x7f06c006e7b0 legacy=0x7f06c01ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.442+0000 7f06beffd640 10 client.?.objecter ms_handle_connect 0x7f06c006e7b0 2024-08-19T12:47:02.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 client.?.objecter ms_handle_connect 0x7f06c00a5d50 2024-08-19T12:47:02.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06c6a82640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.26:6793/0 conn(0x7f06c00a5d50 legacy=0x7f06c01d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1898438778 0 0) 0x7f06c013b540 con 0x7f06c006e7b0 2024-08-19T12:47:02.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): my global_id is 164175 2024-08-19T12:47:02.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06ac004eb0 con 0x7f06c006e7b0 2024-08-19T12:47:02.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 client.?.objecter ms_handle_connect 0x7f06c00a5920 2024-08-19T12:47:02.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06c5a80640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.104:6791/0 conn(0x7f06c00a5920 legacy=0x7f06c01d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2227858373 0 0) 0x7f06c0134760 con 0x7f06c00a5d50 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): my global_id is 26396 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06c013b540 con 0x7f06c00a5d50 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2427152527 0 0) 0x7f06c013b260 con 0x7f06c00a5920 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): my global_id is 26564 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06c0134760 con 0x7f06c00a5920 2024-08-19T12:47:02.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 385820034 0 0) 0x7f06ac004eb0 con 0x7f06c006e7b0 2024-08-19T12:47:02.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06c013b260 con 0x7f06c006e7b0 2024-08-19T12:47:02.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06b00017e0 con 0x7f06c006e7b0 2024-08-19T12:47:02.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:02.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:47:02.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): dump: 2024-08-19T12:47:02.463 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.463 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.463 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.463 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.463 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.463 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.463 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.464 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:02.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 66116923 0 0) 0x7f06c013b540 con 0x7f06c00a5d50 2024-08-19T12:47:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06ac004eb0 con 0x7f06c00a5d50 2024-08-19T12:47:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06b8003b00 con 0x7f06c00a5d50 2024-08-19T12:47:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:47:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient(hunting): dump: 2024-08-19T12:47:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.465 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.466 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.466 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.466 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.466 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.466 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.466 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.466 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.466 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.466 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1623730221 0 0) 0x7f06c013b260 con 0x7f06c006e7b0 2024-08-19T12:47:02.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.26:6793/0 conn(0x7f06c00a5d50 legacy=0x7f06c01d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.104:6791/0 conn(0x7f06c00a5920 legacy=0x7f06c01d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient: _finish_hunting 0 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 monclient: found mon.d 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c01391e0 con 0x7f06c006e7b0 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient: _finish_auth 0 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient: _check_auth_tickets 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06c84e5640 5 monclient: authenticate success, global_id 164175 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06c84e5640 10 monclient: _renew_subs 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06c84e5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06c84e5640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f06c013aca0 con 0x7f06c006e7b0 2024-08-19T12:47:02.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06c84e5640 10 client.164175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:02.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06c84e5640 10 monclient: _renew_subs 2024-08-19T12:47:02.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06c84e5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:02.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06c84e5640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f06c0139d40 con 0x7f06c006e7b0 2024-08-19T12:47:02.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06c84e5640 1 librados: init done 2024-08-19T12:47:02.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f06b00052f0 con 0x7f06c006e7b0 2024-08-19T12:47:02.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:02.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06b00057e0 con 0x7f06c006e7b0 2024-08-19T12:47:02.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06c4a7e640 4 set_mon_vals no callback set 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.443+0000 7f06beffd640 10 monclient: dump: 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.469 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.470 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.470 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.470 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.470 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.470 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.444+0000 7f06c84e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:02.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.444+0000 7f06c84e5640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f06a00065c0 con 0x7f06a0002860 2024-08-19T12:47:02.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.444+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f06b0011c40 con 0x7f06c006e7b0 2024-08-19T12:47:02.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.444+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(353..353 src has 1..353) ==== 5346+0+0 (unknown 3503228669 0 0) 0x7f06b004e720 con 0x7f06c006e7b0 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.444+0000 7f06beffd640 10 client.164175.objecter ms_dispatch 0x7f06c0000b90 osd_map(353..353 src has 1..353) 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.444+0000 7f06beffd640 3 client.164175.objecter handle_osd_map got epochs [353,353] > 0 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.444+0000 7f06beffd640 3 client.164175.objecter handle_osd_map decoding full epoch 353 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.445+0000 7f06beffd640 20 client.164175.objecter dump_active .. 0 homeless 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.445+0000 7f06c6a82640 10 monclient: get_auth_request con 0x7f06ac041660 auth_method 0 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.446+0000 7f06beffd640 10 client.164175.objecter ms_handle_connect 0x7f06a0002860 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.446+0000 7f06beffd640 10 client.164175.objecter resend_mon_ops 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.446+0000 7f06c5a80640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.26:6789/0 conn(0x7f06a0002860 legacy=0x7f06a0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.446+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2238150893 0 0) 0x7f06a00065c0 con 0x7f06a0002860 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.446+0000 7f06beffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.446+0000 7f06beffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.446+0000 7f06beffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.446+0000 7f06beffd640 10 monclient(hunting): my global_id is 144846 2024-08-19T12:47:02.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.446+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06c013b260 con 0x7f06a0002860 2024-08-19T12:47:02.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2175158061 0 0) 0x7f06c013b260 con 0x7f06a0002860 2024-08-19T12:47:02.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06a00065c0 con 0x7f06a0002860 2024-08-19T12:47:02.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06b4001320 con 0x7f06a0002860 2024-08-19T12:47:02.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:02.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 10 monclient: dump: 2024-08-19T12:47:02.472 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.472 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.472 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.472 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.473 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.473 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.473 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.473 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.473 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.473 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.473 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.473 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.473 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.473 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.473 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.473 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 91142630 0 0) 0x7f06a00065c0 con 0x7f06a0002860 2024-08-19T12:47:02.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06a000a160 con 0x7f06a0002860 2024-08-19T12:47:02.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 10 client.164175.objecter ms_handle_connect 0x7f06ac041660 2024-08-19T12:47:02.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f06b4004f00 con 0x7f06a0002860 2024-08-19T12:47:02.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:02.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:02.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.26:6789/0 conn(0x7f06a0002860 legacy=0x7f06a0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.447+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.26:6789/0 conn(0x7f06a0002860 legacy=0x7f06a0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:02.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.453+0000 7f06c84e5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:02.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.453+0000 7f06c84e5640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f06a0009e50 con 0x7f06a0009110 2024-08-19T12:47:02.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06c5a80640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.26:6789/0 conn(0x7f06a0009110 legacy=0x7f06a00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 10 client.164175.objecter ms_handle_connect 0x7f06a0009110 2024-08-19T12:47:02.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 10 client.164175.objecter resend_mon_ops 2024-08-19T12:47:02.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 511229954 0 0) 0x7f06a0009e50 con 0x7f06a0009110 2024-08-19T12:47:02.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 10 monclient(hunting): my global_id is 144855 2024-08-19T12:47:02.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06a00065c0 con 0x7f06a0009110 2024-08-19T12:47:02.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3511303562 0 0) 0x7f06a00065c0 con 0x7f06a0009110 2024-08-19T12:47:02.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06a0009e50 con 0x7f06a0009110 2024-08-19T12:47:02.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06b4005740 con 0x7f06a0009110 2024-08-19T12:47:02.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:02.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 10 monclient: dump: 2024-08-19T12:47:02.477 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.477 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.477 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.478 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1661842815 0 0) 0x7f06a0009e50 con 0x7f06a0009110 2024-08-19T12:47:02.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f06a000c3f0 con 0x7f06a0009110 2024-08-19T12:47:02.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7880 (unknown 0 0 3561302395) 0x7f06b4005a20 con 0x7f06a0009110 2024-08-19T12:47:02.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:02.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:02.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.26:6789/0 conn(0x7f06a0009110 legacy=0x7f06a00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.456+0000 7f06beffd640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.26:6789/0 conn(0x7f06a0009110 legacy=0x7f06a00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:02.479 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:02.479 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:02.480 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:02.480 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:47:02.480 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:47:02.480 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 274373, 2024-08-19T12:47:02.480 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:02.480 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:02.480 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:02.480 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:02.480 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:02.480 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:02.480 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:02.481 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:02.481 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:47:02.481 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:47:02.481 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.481 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:47:02.481 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:02.481 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:02.481 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:02.481 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:02.482 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:02.482 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:02.482 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:02.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:02.482 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:02.482 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:02.482 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:02.482 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:02.482 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:02.482 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:02.482 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.482 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:02.483 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:02.483 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:02.483 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:02.483 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:02.483 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:02.483 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:02.483 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:02.483 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:02.483 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:02.483 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:02.483 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:02.483 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:02.484 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.484 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:02.484 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:02.484 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:02.484 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:02.484 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:02.484 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:02.484 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:02.484 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:02.485 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:02.486 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:02.486 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:02.486 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:02.486 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:02.486 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:02.486 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:02.486 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.486 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:02.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.486 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:02.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.486 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:02.487 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.488 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.488 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.488 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:02.488 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:02.488 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.488 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:02.489 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:02.489 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:02.489 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.489 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.489 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:02.490 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.492 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.492 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:02.492 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:02.492 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.492 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.492 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.492 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:02.493 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:02.493 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.493 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.493 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.493 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:02.493 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.493 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.493 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.493 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:02.494 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:02.494 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.494 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.494 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.494 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:02.494 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:02.494 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.494 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.495 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.495 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:02.495 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.495 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.495 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.495 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:02.495 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:02.495 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.495 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.495 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:02.496 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:02.497 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:02.498 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.499 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.500 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.500 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.500 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:02.500 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:02.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.500 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:02.500 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:02.500 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:02.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.501 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.501 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:47:02.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.501 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:02.501 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:02.501 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:02.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.501 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.501 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:02.501 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.501 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.502 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:02.503 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:02.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 10 librados: watch_flush enter 2024-08-19T12:47:02.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 10 librados: watch_flush exit 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 20 client.164175.objecter shutdown clearing up homeless session... 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 10 client.164175.objecter successfully canceled tick 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.104:6800/899887506 conn(0x7f06ac041660 legacy=0x7f06ac043b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 10 monclient: shutdown 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 1 -- 172.21.15.26:0/4065723990 >> v1:172.21.15.26:6792/0 conn(0x7f06c006e7b0 legacy=0x7f06c01ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c6a82640 1 -- 172.21.15.26:0/4065723990 reap_dead start 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c6a82640 1 -- 172.21.15.26:0/4065723990 reap_dead start 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 1 -- 172.21.15.26:0/4065723990 shutdown_connections 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 1 -- 172.21.15.26:0/4065723990 >> 172.21.15.26:0/4065723990 conn(0x7f06c00a8740 msgr2=0x7f06c00aa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 1 -- 172.21.15.26:0/4065723990 shutdown_connections 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 1 -- 172.21.15.26:0/4065723990 wait complete. 2024-08-19T12:47:02.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.457+0000 7f06c84e5640 1 librados: shutdown 2024-08-19T12:47:02.506 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:47:02.506 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:47:02.506 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:47:02.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.606+0000 7f8983f9a640 10 monclient: get_monmap_and_config 2024-08-19T12:47:02.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.606+0000 7f8983f9a640 10 monclient: build_initial_monmap 2024-08-19T12:47:02.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.606+0000 7f8983f9a640 10 monclient: monmap: 2024-08-19T12:47:02.607 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:02.607 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.607 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:02.606942+0000 2024-08-19T12:47:02.607 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:02.606942+0000 2024-08-19T12:47:02.607 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:02.607 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:02.608 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:02.608 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.606+0000 7f8983f9a640 1 Processor -- start 2024-08-19T12:47:02.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.606+0000 7f8983f9a640 1 -- start start 2024-08-19T12:47:02.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.606+0000 7f8983f9a640 10 monclient: init 2024-08-19T12:47:02.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8983f9a640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:02.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8983f9a640 10 monclient: _add_conns ranks=[6,5,0,2,8,4,3,7,1] 2024-08-19T12:47:02.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8983f9a640 10 monclient(hunting): picked mon.noname-d con 0x7f897c073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:02.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8983f9a640 10 monclient(hunting): picked mon.noname-h con 0x7f897c0a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:02.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8983f9a640 10 monclient(hunting): picked mon.noname-a con 0x7f897c13f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:02.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8983f9a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f897c09dae0 con 0x7f897c13f2e0 2024-08-19T12:47:02.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8983f9a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f897c136910 con 0x7f897c0a57a0 2024-08-19T12:47:02.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8983f9a640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f897c09dd50 con 0x7f897c073470 2024-08-19T12:47:02.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8983f9a640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8983f9a640 10 monclient(hunting): authenticate will time out at 1216.694525s 2024-08-19T12:47:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8981d36640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f897c073470 0x7f897c136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:43108/0 (socket says 172.21.15.26:43108) 2024-08-19T12:47:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8981d36640 1 -- 172.21.15.26:0/1355418305 learned_addr learned my addr 172.21.15.26:0/1355418305 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.607+0000 7f8982537640 1 --1- 172.21.15.26:0/1355418305 >> v1:172.21.15.26:6789/0 conn(0x7f897c13f2e0 0x7f897c1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48006/0 (socket says 172.21.15.26:48006) 2024-08-19T12:47:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8981d36640 1 -- 172.21.15.26:0/1355418305 >> v1:172.21.15.26:6792/0 conn(0x7f897c073470 legacy=0x7f897c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8982537640 1 -- 172.21.15.26:0/1355418305 >> v1:172.21.15.26:6789/0 conn(0x7f897c13f2e0 legacy=0x7f897c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8980d34640 1 -- 172.21.15.26:0/1355418305 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1662155136 0 0) 0x7f897c09dd50 con 0x7f897c073470 2024-08-19T12:47:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8980d34640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8980d34640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8980d34640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8980d34640 10 monclient(hunting): my global_id is 164184 2024-08-19T12:47:02.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8980d34640 1 -- 172.21.15.26:0/1355418305 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8964003520 con 0x7f897c073470 2024-08-19T12:47:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8980d34640 1 -- 172.21.15.26:0/1355418305 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 757232087 0 0) 0x7f897c09dae0 con 0x7f897c13f2e0 2024-08-19T12:47:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8980d34640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8980d34640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8980d34640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8980d34640 10 monclient(hunting): my global_id is 144864 2024-08-19T12:47:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8981535640 1 -- 172.21.15.26:0/1355418305 >> v1:172.21.15.104:6791/0 conn(0x7f897c0a57a0 legacy=0x7f897c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.609+0000 7f8980d34640 1 -- 172.21.15.26:0/1355418305 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f897c09dd50 con 0x7f897c13f2e0 2024-08-19T12:47:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 1 -- 172.21.15.26:0/1355418305 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1067715518 0 0) 0x7f8964003520 con 0x7f897c073470 2024-08-19T12:47:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 1 -- 172.21.15.26:0/1355418305 >> v1:172.21.15.104:6791/0 conn(0x7f897c0a57a0 legacy=0x7f897c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 1 -- 172.21.15.26:0/1355418305 >> v1:172.21.15.26:6789/0 conn(0x7f897c13f2e0 legacy=0x7f897c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 10 monclient: _finish_hunting 0 2024-08-19T12:47:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 1 monclient: found mon.noname-d 2024-08-19T12:47:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:47:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 1 -- 172.21.15.26:0/1355418305 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897c09e030 con 0x7f897c073470 2024-08-19T12:47:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 10 monclient: _finish_auth 0 2024-08-19T12:47:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 10 monclient: _check_auth_tickets 2024-08-19T12:47:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 1 -- 172.21.15.26:0/1355418305 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8974001ae0 con 0x7f897c073470 2024-08-19T12:47:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8983f9a640 5 monclient: authenticate success, global_id 164184 2024-08-19T12:47:02.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8983f9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:47:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 10 monclient: dump: 2024-08-19T12:47:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.618 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.619 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8983f9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 1 -- 172.21.15.26:0/1355418305 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89740048a0 con 0x7f897c073470 2024-08-19T12:47:02.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 1 -- 172.21.15.26:0/1355418305 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8974004e10 con 0x7f897c073470 2024-08-19T12:47:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8980d34640 10 monclient: dump: 2024-08-19T12:47:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.620 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.621 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8983f9a640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:02.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8983f9a640 4 set_mon_vals no callback set 2024-08-19T12:47:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8983f9a640 10 monclient: shutdown 2024-08-19T12:47:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8983f9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8983f9a640 1 -- 172.21.15.26:0/1355418305 >> v1:172.21.15.26:6792/0 conn(0x7f897c073470 legacy=0x7f897c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8983f9a640 1 -- 172.21.15.26:0/1355418305 shutdown_connections 2024-08-19T12:47:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.610+0000 7f8983f9a640 1 -- 172.21.15.26:0/1355418305 >> 172.21.15.26:0/1355418305 conn(0x7f897c0abff0 msgr2=0x7f897c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.611+0000 7f8983f9a640 1 -- 172.21.15.26:0/1355418305 shutdown_connections 2024-08-19T12:47:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.611+0000 7f8983f9a640 1 -- 172.21.15.26:0/1355418305 wait complete. 2024-08-19T12:47:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.611+0000 7f8983f9a640 10 monclient: build_initial_monmap 2024-08-19T12:47:02.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.611+0000 7f8983f9a640 10 monclient: monmap: 2024-08-19T12:47:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:02.623 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:02.624 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.611+0000 7f8983f9a640 1 librados: starting msgr at 2024-08-19T12:47:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.611+0000 7f8983f9a640 1 librados: starting objecter 2024-08-19T12:47:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.611+0000 7f8983f9a640 1 Processor -- start 2024-08-19T12:47:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.611+0000 7f8983f9a640 1 -- start start 2024-08-19T12:47:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.611+0000 7f8983f9a640 1 librados: setting wanted keys 2024-08-19T12:47:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.611+0000 7f8983f9a640 1 librados: calling monclient init 2024-08-19T12:47:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.611+0000 7f8983f9a640 10 monclient: init 2024-08-19T12:47:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.612+0000 7f8983f9a640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.612+0000 7f8983f9a640 10 monclient: _add_conns ranks=[1,2,7,0,5,8,6,3,4] 2024-08-19T12:47:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.612+0000 7f8983f9a640 10 monclient(hunting): picked mon.noname-f con 0x7f897c073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.612+0000 7f8983f9a640 10 monclient(hunting): picked mon.noname-b con 0x7f897c0a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.612+0000 7f8983f9a640 10 monclient(hunting): picked mon.noname-i con 0x7f897c13f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:47:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.612+0000 7f8983f9a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f897c0a3920 con 0x7f897c073470 2024-08-19T12:47:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.612+0000 7f8983f9a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f897c1cef20 con 0x7f897c0a57a0 2024-08-19T12:47:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.612+0000 7f8983f9a640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f897c142b60 con 0x7f897c13f2e0 2024-08-19T12:47:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.612+0000 7f8983f9a640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.612+0000 7f8983f9a640 10 monclient(hunting): authenticate will time out at 1216.699459s 2024-08-19T12:47:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.612+0000 7f8981535640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f897c0a57a0 0x7f897c1ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39012/0 (socket says 172.21.15.26:39012) 2024-08-19T12:47:02.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.612+0000 7f8981535640 1 -- 172.21.15.26:0/767154799 learned_addr learned my addr 172.21.15.26:0/767154799 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f8981535640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.26:6790/0 conn(0x7f897c0a57a0 legacy=0x7f897c1ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f897a7fc640 10 client.?.objecter ms_handle_connect 0x7f897c0a57a0 2024-08-19T12:47:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f897a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f897a7fc640 10 client.?.objecter ms_handle_connect 0x7f897c073470 2024-08-19T12:47:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f897a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f8981d36640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.104:6789/0 conn(0x7f897c073470 legacy=0x7f897c1ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1087544906 0 0) 0x7f897c1cef20 con 0x7f897c0a57a0 2024-08-19T12:47:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f897a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f897a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f897a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f897a7fc640 10 monclient(hunting): my global_id is 26525 2024-08-19T12:47:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8968004ee0 con 0x7f897c0a57a0 2024-08-19T12:47:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f897a7fc640 10 client.?.objecter ms_handle_connect 0x7f897c13f2e0 2024-08-19T12:47:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f8982537640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.104:6792/0 conn(0x7f897c13f2e0 legacy=0x7f897c1d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.614+0000 7f897a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3951817138 0 0) 0x7f897c0a3920 con 0x7f897c073470 2024-08-19T12:47:02.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient(hunting): my global_id is 27361 2024-08-19T12:47:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f897c1cef20 con 0x7f897c073470 2024-08-19T12:47:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 93794760 0 0) 0x7f8968004ee0 con 0x7f897c0a57a0 2024-08-19T12:47:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f897c0a3920 con 0x7f897c0a57a0 2024-08-19T12:47:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8970003610 con 0x7f897c0a57a0 2024-08-19T12:47:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:47:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient(hunting): dump: 2024-08-19T12:47:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.634 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.635 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.636 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1627469251 0 0) 0x7f897c142b60 con 0x7f897c13f2e0 2024-08-19T12:47:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient(hunting): my global_id is 135511 2024-08-19T12:47:02.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8968004ee0 con 0x7f897c13f2e0 2024-08-19T12:47:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3238693429 0 0) 0x7f897c0a3920 con 0x7f897c0a57a0 2024-08-19T12:47:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.104:6792/0 conn(0x7f897c13f2e0 legacy=0x7f897c1d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.104:6789/0 conn(0x7f897c073470 legacy=0x7f897c1ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:47:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 monclient: found mon.b 2024-08-19T12:47:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:47:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897c0a53e0 con 0x7f897c0a57a0 2024-08-19T12:47:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:47:02.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:47:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8974004d90 con 0x7f897c073470 2024-08-19T12:47:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f8983f9a640 5 monclient: authenticate success, global_id 26525 2024-08-19T12:47:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f8983f9a640 10 monclient: _renew_subs 2024-08-19T12:47:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f8983f9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:47:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f8983f9a640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f897c136b20 con 0x7f897c0a57a0 2024-08-19T12:47:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:47:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f8983f9a640 10 client.26525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f8983f9a640 10 monclient: _renew_subs 2024-08-19T12:47:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f8983f9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:47:02.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f8983f9a640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f897c0a2ed0 con 0x7f897c0a57a0 2024-08-19T12:47:02.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f8983f9a640 1 librados: init done 2024-08-19T12:47:02.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8970003ac0 con 0x7f897c0a57a0 2024-08-19T12:47:02.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:02.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8970003fb0 con 0x7f897c0a57a0 2024-08-19T12:47:02.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897bfff640 4 set_mon_vals no callback set 2024-08-19T12:47:02.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:02.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.615+0000 7f897a7fc640 10 monclient: dump: 2024-08-19T12:47:02.639 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.640 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.640 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.640 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.640 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.640 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.640 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.640 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.640 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.640 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.640 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.640 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.641 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.641 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.641 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.641 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.616+0000 7f8983f9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:02.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.616+0000 7f8983f9a640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f895c0065c0 con 0x7f895c002860 2024-08-19T12:47:02.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.616+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f8970004150 con 0x7f897c0a57a0 2024-08-19T12:47:02.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.616+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(353..353 src has 1..353) ==== 5346+0+0 (unknown 3503228669 0 0) 0x7f897004e200 con 0x7f897c0a57a0 2024-08-19T12:47:02.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.616+0000 7f897a7fc640 10 client.26525.objecter ms_dispatch 0x7f897c000b90 osd_map(353..353 src has 1..353) 2024-08-19T12:47:02.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.616+0000 7f897a7fc640 3 client.26525.objecter handle_osd_map got epochs [353,353] > 0 2024-08-19T12:47:02.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.616+0000 7f897a7fc640 3 client.26525.objecter handle_osd_map decoding full epoch 353 2024-08-19T12:47:02.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.616+0000 7f8982537640 10 monclient: get_auth_request con 0x7f8968040ef0 auth_method 0 2024-08-19T12:47:02.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.616+0000 7f897a7fc640 20 client.26525.objecter dump_active .. 0 homeless 2024-08-19T12:47:02.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f8981d36640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.26:6790/0 conn(0x7f895c002860 legacy=0x7f895c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 10 client.26525.objecter ms_handle_connect 0x7f895c002860 2024-08-19T12:47:02.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 10 client.26525.objecter resend_mon_ops 2024-08-19T12:47:02.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1291230247 0 0) 0x7f895c0065c0 con 0x7f895c002860 2024-08-19T12:47:02.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 10 monclient(hunting): my global_id is 26534 2024-08-19T12:47:02.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f897c0a3920 con 0x7f895c002860 2024-08-19T12:47:02.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2200968646 0 0) 0x7f897c0a3920 con 0x7f895c002860 2024-08-19T12:47:02.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f895c0065c0 con 0x7f895c002860 2024-08-19T12:47:02.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8974001320 con 0x7f895c002860 2024-08-19T12:47:02.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:02.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.618+0000 7f897a7fc640 10 monclient: dump: 2024-08-19T12:47:02.643 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.643 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.643 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.644 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.644 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.644 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.644 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.644 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.644 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.644 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.644 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.644 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.644 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.645 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.645 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.645 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.619+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2083431511 0 0) 0x7f895c0065c0 con 0x7f895c002860 2024-08-19T12:47:02.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.619+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f895c00a160 con 0x7f895c002860 2024-08-19T12:47:02.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.619+0000 7f897a7fc640 10 client.26525.objecter ms_handle_connect 0x7f8968040ef0 2024-08-19T12:47:02.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.619+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8974005490 con 0x7f895c002860 2024-08-19T12:47:02.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.619+0000 7f897a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:02.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.619+0000 7f897a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:02.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.619+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.26:6790/0 conn(0x7f895c002860 legacy=0x7f895c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.619+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.26:6790/0 conn(0x7f895c002860 legacy=0x7f895c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:02.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.625+0000 7f8983f9a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:02.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.625+0000 7f8983f9a640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f895c009e50 con 0x7f895c009110 2024-08-19T12:47:02.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.627+0000 7f8981d36640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.26:6790/0 conn(0x7f895c009110 legacy=0x7f895c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.627+0000 7f897a7fc640 10 client.26525.objecter ms_handle_connect 0x7f895c009110 2024-08-19T12:47:02.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.627+0000 7f897a7fc640 10 client.26525.objecter resend_mon_ops 2024-08-19T12:47:02.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1786461892 0 0) 0x7f895c009e50 con 0x7f895c009110 2024-08-19T12:47:02.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 10 monclient(hunting): my global_id is 26543 2024-08-19T12:47:02.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f895c0065c0 con 0x7f895c009110 2024-08-19T12:47:02.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 743275579 0 0) 0x7f895c0065c0 con 0x7f895c009110 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f895c009e50 con 0x7f895c009110 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8974005c80 con 0x7f895c009110 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 10 monclient: dump: 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.647 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.648 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.648 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.648 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.648 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.648 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.648 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.648 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.648 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1109604310 0 0) 0x7f895c009e50 con 0x7f895c009110 2024-08-19T12:47:02.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f895c00c3a0 con 0x7f895c009110 2024-08-19T12:47:02.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 3456401263) 0x7f89740071d0 con 0x7f895c009110 2024-08-19T12:47:02.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:02.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:02.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.26:6790/0 conn(0x7f895c009110 legacy=0x7f895c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f897a7fc640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.26:6790/0 conn(0x7f895c009110 legacy=0x7f895c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:02.649 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:02.649 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:47:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 535683, 2024-08-19T12:47:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:02.650 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:02.651 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:02.652 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.654 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:02.654 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:02.654 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:02.654 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:02.654 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:02.654 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:02.654 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:02.654 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:02.654 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:02.655 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:02.656 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:02.657 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.657 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:02.657 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:02.658 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:02.658 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.658 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.658 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.658 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:02.658 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:02.658 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.658 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.659 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:02.659 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.659 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:02.659 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:02.659 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.659 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:02.663 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:02.664 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.664 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.664 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.664 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:02.664 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:02.664 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.664 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.664 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:02.665 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:02.666 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.666 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:02.666 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.666 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:02.666 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:02.667 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.667 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.667 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.667 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:02.667 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:02.667 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:02.668 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.668 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:02.668 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:02.668 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.668 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.668 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:02.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:02.670 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.670 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.670 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.670 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.670 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:02.670 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:02.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.670 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:02.671 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:02.671 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:02.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.671 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.671 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:02.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.671 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:02.671 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:02.671 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:47:02.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.672 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:02.672 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:02.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.628+0000 7f8983f9a640 10 librados: watch_flush enter 2024-08-19T12:47:02.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8983f9a640 10 librados: watch_flush exit 2024-08-19T12:47:02.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8983f9a640 20 client.26525.objecter shutdown clearing up homeless session... 2024-08-19T12:47:02.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8983f9a640 10 client.26525.objecter successfully canceled tick 2024-08-19T12:47:02.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8983f9a640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.104:6800/899887506 conn(0x7f8968040ef0 legacy=0x7f89680433b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8983f9a640 10 monclient: shutdown 2024-08-19T12:47:02.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8983f9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:02.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8983f9a640 1 -- 172.21.15.26:0/767154799 >> v1:172.21.15.26:6790/0 conn(0x7f897c0a57a0 legacy=0x7f897c1ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8982537640 1 -- 172.21.15.26:0/767154799 reap_dead start 2024-08-19T12:47:02.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8982537640 1 -- 172.21.15.26:0/767154799 reap_dead start 2024-08-19T12:47:02.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8983f9a640 1 -- 172.21.15.26:0/767154799 shutdown_connections 2024-08-19T12:47:02.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8983f9a640 1 -- 172.21.15.26:0/767154799 >> 172.21.15.26:0/767154799 conn(0x7f897c0abff0 msgr2=0x7f897c1338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:02.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8983f9a640 1 -- 172.21.15.26:0/767154799 shutdown_connections 2024-08-19T12:47:02.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8983f9a640 1 -- 172.21.15.26:0/767154799 wait complete. 2024-08-19T12:47:02.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.629+0000 7f8983f9a640 1 librados: shutdown 2024-08-19T12:47:02.675 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:47:02.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:47:02.676 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:47:02.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.774+0000 7fc63155a640 10 monclient: get_monmap_and_config 2024-08-19T12:47:02.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.774+0000 7fc63155a640 10 monclient: build_initial_monmap 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.774+0000 7fc63155a640 10 monclient: monmap: 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:02.775648+0000 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:02.775648+0000 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:02.776 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:02.777 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:02.777 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:02.777 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 1 Processor -- start 2024-08-19T12:47:02.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 1 -- start start 2024-08-19T12:47:02.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 10 monclient: init 2024-08-19T12:47:02.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:02.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 10 monclient: _add_conns ranks=[4,3,0,5,2,8,7,6,1] 2024-08-19T12:47:02.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 10 monclient(hunting): picked mon.noname-c con 0x7fc62c06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:02.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 10 monclient(hunting): picked mon.noname-g con 0x7fc62c0a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:02.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 10 monclient(hunting): picked mon.noname-a con 0x7fc62c0a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:02.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc62c136790 con 0x7fc62c0a5d50 2024-08-19T12:47:02.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc62c09ddf0 con 0x7fc62c0a5920 2024-08-19T12:47:02.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc62c09e0d0 con 0x7fc62c06e7b0 2024-08-19T12:47:02.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:02.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.775+0000 7fc63155a640 10 monclient(hunting): authenticate will time out at 1216.862938s 2024-08-19T12:47:02.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.776+0000 7fc62b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc62c0a5d50 0x7fc62c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48008/0 (socket says 172.21.15.26:48008) 2024-08-19T12:47:02.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.776+0000 7fc62affd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fc62c06e7b0 0x7fc62c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:34682/0 (socket says 172.21.15.26:34682) 2024-08-19T12:47:02.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.776+0000 7fc62b7fe640 1 -- 172.21.15.26:0/3229735786 learned_addr learned my addr 172.21.15.26:0/3229735786 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:02.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc62affd640 1 -- 172.21.15.26:0/3229735786 >> v1:172.21.15.26:6791/0 conn(0x7fc62c06e7b0 legacy=0x7fc62c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc62b7fe640 1 -- 172.21.15.26:0/3229735786 >> v1:172.21.15.26:6789/0 conn(0x7fc62c0a5d50 legacy=0x7fc62c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 648649961 0 0) 0x7fc62c09e0d0 con 0x7fc62c06e7b0 2024-08-19T12:47:02.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient(hunting): my global_id is 26878 2024-08-19T12:47:02.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc618003520 con 0x7fc62c06e7b0 2024-08-19T12:47:02.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3533865655 0 0) 0x7fc62c136790 con 0x7fc62c0a5d50 2024-08-19T12:47:02.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient(hunting): my global_id is 144873 2024-08-19T12:47:02.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc62c09e0d0 con 0x7fc62c0a5d50 2024-08-19T12:47:02.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc62a7fc640 1 -- 172.21.15.26:0/3229735786 >> v1:172.21.15.104:6790/0 conn(0x7fc62c0a5920 legacy=0x7fc62c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1291522797 0 0) 0x7fc614001460 con 0x7fc62c0a5920 2024-08-19T12:47:02.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient(hunting): my global_id is 154380 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc62c136790 con 0x7fc62c0a5920 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3013520657 0 0) 0x7fc618003520 con 0x7fc62c06e7b0 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 >> v1:172.21.15.104:6790/0 conn(0x7fc62c0a5920 legacy=0x7fc62c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 >> v1:172.21.15.26:6789/0 conn(0x7fc62c0a5d50 legacy=0x7fc62c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 monclient: found mon.noname-c 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62c09e3b0 con 0x7fc62c06e7b0 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient: _finish_auth 0 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:47:02.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc624001e60 con 0x7fc62c06e7b0 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc63155a640 5 monclient: authenticate success, global_id 26878 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc63155a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient: dump: 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.786 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc63155a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc624004b60 con 0x7fc62c06e7b0 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 1 -- 172.21.15.26:0/3229735786 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc6240050d0 con 0x7fc62c06e7b0 2024-08-19T12:47:02.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc63155a640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc63155a640 4 set_mon_vals no callback set 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.778+0000 7fc629ffb640 10 monclient: dump: 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.788 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.779+0000 7fc63155a640 10 monclient: shutdown 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.779+0000 7fc63155a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.779+0000 7fc63155a640 1 -- 172.21.15.26:0/3229735786 >> v1:172.21.15.26:6791/0 conn(0x7fc62c06e7b0 legacy=0x7fc62c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.779+0000 7fc63155a640 1 -- 172.21.15.26:0/3229735786 shutdown_connections 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.779+0000 7fc63155a640 1 -- 172.21.15.26:0/3229735786 >> 172.21.15.26:0/3229735786 conn(0x7fc62c0a8740 msgr2=0x7fc62c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.779+0000 7fc63155a640 1 -- 172.21.15.26:0/3229735786 shutdown_connections 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.779+0000 7fc63155a640 1 -- 172.21.15.26:0/3229735786 wait complete. 2024-08-19T12:47:02.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.779+0000 7fc63155a640 10 monclient: build_initial_monmap 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.779+0000 7fc63155a640 10 monclient: monmap: 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:02.790 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.780+0000 7fc63155a640 1 librados: starting msgr at 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.780+0000 7fc63155a640 1 librados: starting objecter 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.780+0000 7fc63155a640 1 Processor -- start 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.780+0000 7fc63155a640 1 -- start start 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.780+0000 7fc63155a640 1 librados: setting wanted keys 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.780+0000 7fc63155a640 1 librados: calling monclient init 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.780+0000 7fc63155a640 10 monclient: init 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.780+0000 7fc63155a640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.780+0000 7fc63155a640 10 monclient: _add_conns ranks=[0,2,4,8,7,6,1,5,3] 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.780+0000 7fc63155a640 10 monclient(hunting): picked mon.noname-a con 0x7fc62c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:02.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.780+0000 7fc63155a640 10 monclient(hunting): picked mon.noname-b con 0x7fc62c0a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.781+0000 7fc63155a640 10 monclient(hunting): picked mon.noname-c con 0x7fc62c0a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.781+0000 7fc63155a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc62c138ed0 con 0x7fc62c06e7b0 2024-08-19T12:47:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.781+0000 7fc63155a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc62c138360 con 0x7fc62c0a5920 2024-08-19T12:47:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.781+0000 7fc62affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc62c06e7b0 0x7fc62c137540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48022/0 (socket says 172.21.15.26:48022) 2024-08-19T12:47:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.781+0000 7fc62affd640 1 -- 172.21.15.26:0/1179119508 learned_addr learned my addr 172.21.15.26:0/1179119508 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.781+0000 7fc63155a640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc62c13a990 con 0x7fc62c0a5d50 2024-08-19T12:47:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.781+0000 7fc63155a640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:02.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.781+0000 7fc63155a640 10 monclient(hunting): authenticate will time out at 1216.868097s 2024-08-19T12:47:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.783+0000 7fc62affd640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.26:6789/0 conn(0x7fc62c06e7b0 legacy=0x7fc62c137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.783+0000 7fc62a7fc640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.26:6790/0 conn(0x7fc62c0a5920 legacy=0x7fc62c137c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.783+0000 7fc6237fe640 10 client.?.objecter ms_handle_connect 0x7fc62c06e7b0 2024-08-19T12:47:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.783+0000 7fc6237fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.783+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:47:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.783+0000 7fc62b7fe640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.26:6791/0 conn(0x7fc62c0a5d50 legacy=0x7fc62c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 client.?.objecter ms_handle_connect 0x7fc62c0a5920 2024-08-19T12:47:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 client.?.objecter ms_handle_connect 0x7fc62c0a5d50 2024-08-19T12:47:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2446203495 0 0) 0x7fc62c138360 con 0x7fc62c0a5920 2024-08-19T12:47:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): my global_id is 26552 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc60c004eb0 con 0x7fc62c0a5920 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3395812570 0 0) 0x7fc62c13a990 con 0x7fc62c0a5d50 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): my global_id is 26887 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc62c138360 con 0x7fc62c0a5d50 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1416714813 0 0) 0x7fc62c138ed0 con 0x7fc62c06e7b0 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): my global_id is 144882 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc62c13a990 con 0x7fc62c06e7b0 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2753379213 0 0) 0x7fc62c138360 con 0x7fc62c0a5d50 2024-08-19T12:47:02.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc62c138ed0 con 0x7fc62c0a5d50 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc61c001d10 con 0x7fc62c0a5d50 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): dump: 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.800 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2549860266 0 0) 0x7fc62c13a990 con 0x7fc62c06e7b0 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc62c138360 con 0x7fc62c06e7b0 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 814326698 0 0) 0x7fc60c004eb0 con 0x7fc62c0a5920 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc62c13a990 con 0x7fc62c0a5920 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc624005050 con 0x7fc62c06e7b0 2024-08-19T12:47:02.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): dump: 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.802 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc614003950 con 0x7fc62c0a5920 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient(hunting): dump: 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.803 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2431867921 0 0) 0x7fc62c138ed0 con 0x7fc62c0a5d50 2024-08-19T12:47:02.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.26:6790/0 conn(0x7fc62c0a5920 legacy=0x7fc62c137c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.26:6789/0 conn(0x7fc62c06e7b0 legacy=0x7fc62c137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient: _finish_hunting 0 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 monclient: found mon.c 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62c142910 con 0x7fc62c0a5d50 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient: _finish_auth 0 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 10 monclient: _check_auth_tickets 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc6237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc63155a640 5 monclient: authenticate success, global_id 26887 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc63155a640 10 monclient: _renew_subs 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc63155a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.785+0000 7fc63155a640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc62c139a30 con 0x7fc62c0a5d50 2024-08-19T12:47:02.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc63155a640 10 client.26887.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc63155a640 10 monclient: _renew_subs 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc63155a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc63155a640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fc62c1d79b0 con 0x7fc62c0a5d50 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc63155a640 1 librados: init done 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc61c004a90 con 0x7fc62c0a5d50 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc6237fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc61c004fc0 con 0x7fc62c0a5d50 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc6237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc6237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc6297fa640 4 set_mon_vals no callback set 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc6237fe640 10 monclient: dump: 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.806 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc63155a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc63155a640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc6100065c0 con 0x7fc610002860 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.786+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fc61c005160 con 0x7fc62c0a5d50 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.787+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(353..353 src has 1..353) ==== 5346+0+0 (unknown 3503228669 0 0) 0x7fc61c04dea0 con 0x7fc62c0a5d50 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.787+0000 7fc6237fe640 10 client.26887.objecter ms_dispatch 0x7fc62c000b90 osd_map(353..353 src has 1..353) 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.787+0000 7fc6237fe640 3 client.26887.objecter handle_osd_map got epochs [353,353] > 0 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.787+0000 7fc6237fe640 3 client.26887.objecter handle_osd_map decoding full epoch 353 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.787+0000 7fc6237fe640 20 client.26887.objecter dump_active .. 0 homeless 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.787+0000 7fc62a7fc640 10 monclient: get_auth_request con 0x7fc60c0418e0 auth_method 0 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc62affd640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.26:6791/0 conn(0x7fc610002860 legacy=0x7fc610004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 10 client.26887.objecter ms_handle_connect 0x7fc610002860 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 10 client.26887.objecter resend_mon_ops 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 752153085 0 0) 0x7fc6100065c0 con 0x7fc610002860 2024-08-19T12:47:02.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 10 monclient(hunting): my global_id is 26896 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc62c138ed0 con 0x7fc610002860 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2265333310 0 0) 0x7fc62c138ed0 con 0x7fc610002860 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6100065c0 con 0x7fc610002860 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc624004b60 con 0x7fc610002860 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 10 monclient: dump: 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.809 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 828331583 0 0) 0x7fc6100065c0 con 0x7fc610002860 2024-08-19T12:47:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc61000a160 con 0x7fc610002860 2024-08-19T12:47:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.789+0000 7fc6237fe640 10 client.26887.objecter ms_handle_connect 0x7fc60c0418e0 2024-08-19T12:47:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.790+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc624005a00 con 0x7fc610002860 2024-08-19T12:47:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.790+0000 7fc6237fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.790+0000 7fc6237fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.790+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.26:6791/0 conn(0x7fc610002860 legacy=0x7fc610004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.790+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.26:6791/0 conn(0x7fc610002860 legacy=0x7fc610004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.796+0000 7fc63155a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:02.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.796+0000 7fc63155a640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc610009e70 con 0x7fc610009130 2024-08-19T12:47:02.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc62affd640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.26:6791/0 conn(0x7fc610009130 legacy=0x7fc6100095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 10 client.26887.objecter ms_handle_connect 0x7fc610009130 2024-08-19T12:47:02.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 10 client.26887.objecter resend_mon_ops 2024-08-19T12:47:02.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 841146065 0 0) 0x7fc610009e70 con 0x7fc610009130 2024-08-19T12:47:02.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 10 monclient(hunting): my global_id is 26905 2024-08-19T12:47:02.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6100065c0 con 0x7fc610009130 2024-08-19T12:47:02.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3992236759 0 0) 0x7fc6100065c0 con 0x7fc610009130 2024-08-19T12:47:02.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc610009e70 con 0x7fc610009130 2024-08-19T12:47:02.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc624006120 con 0x7fc610009130 2024-08-19T12:47:02.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 10 monclient: dump: 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.814 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.815 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.815 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.815 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 409410340 0 0) 0x7fc610009e70 con 0x7fc610009130 2024-08-19T12:47:02.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.798+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc61000c3f0 con 0x7fc610009130 2024-08-19T12:47:02.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1177389885) 0x7fc624008480 con 0x7fc610009130 2024-08-19T12:47:02.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc6237fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:02.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc6237fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:02.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.26:6791/0 conn(0x7fc610009130 legacy=0x7fc6100095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc6237fe640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.26:6791/0 conn(0x7fc610009130 legacy=0x7fc6100095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:02.815 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 535829, 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:02.816 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:02.817 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:02.818 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:02.819 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:02.820 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:02.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:02.822 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.823 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:02.825 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:02.826 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:02.827 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.829 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:47:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:02.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 10 librados: watch_flush enter 2024-08-19T12:47:02.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 10 librados: watch_flush exit 2024-08-19T12:47:02.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 20 client.26887.objecter shutdown clearing up homeless session... 2024-08-19T12:47:02.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 10 client.26887.objecter successfully canceled tick 2024-08-19T12:47:02.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.104:6800/899887506 conn(0x7fc60c0418e0 legacy=0x7fc60c043da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 10 monclient: shutdown 2024-08-19T12:47:02.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:02.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 1 -- 172.21.15.26:0/1179119508 >> v1:172.21.15.26:6791/0 conn(0x7fc62c0a5d50 legacy=0x7fc62c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc62b7fe640 1 -- 172.21.15.26:0/1179119508 reap_dead start 2024-08-19T12:47:02.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc62b7fe640 1 -- 172.21.15.26:0/1179119508 reap_dead start 2024-08-19T12:47:02.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 1 -- 172.21.15.26:0/1179119508 shutdown_connections 2024-08-19T12:47:02.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 1 -- 172.21.15.26:0/1179119508 >> 172.21.15.26:0/1179119508 conn(0x7fc62c0a8740 msgr2=0x7fc62c1350a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:02.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 1 -- 172.21.15.26:0/1179119508 shutdown_connections 2024-08-19T12:47:02.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 1 -- 172.21.15.26:0/1179119508 wait complete. 2024-08-19T12:47:02.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.799+0000 7fc63155a640 1 librados: shutdown 2024-08-19T12:47:02.838 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:47:02.838 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:47:02.839 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:47:02.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.937+0000 7f17f9941640 10 monclient: get_monmap_and_config 2024-08-19T12:47:02.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.937+0000 7f17f9941640 10 monclient: build_initial_monmap 2024-08-19T12:47:02.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.937+0000 7f17f9941640 10 monclient: monmap: 2024-08-19T12:47:02.939 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:02.939 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.939 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:02.938779+0000 2024-08-19T12:47:02.939 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:02.938779+0000 2024-08-19T12:47:02.939 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:02.939 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.939 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:02.939 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:02.939 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:02.939 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:02.940 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:02.940 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:02.940 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:02.940 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:02.940 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:02.940 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.938+0000 7f17f9941640 1 Processor -- start 2024-08-19T12:47:02.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.938+0000 7f17f9941640 1 -- start start 2024-08-19T12:47:02.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.938+0000 7f17f9941640 10 monclient: init 2024-08-19T12:47:02.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17f9941640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17f9941640 10 monclient: _add_conns ranks=[6,2,3,4,1,5,0,7,8] 2024-08-19T12:47:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17f9941640 10 monclient(hunting): picked mon.noname-d con 0x7f17f406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17f9941640 10 monclient(hunting): picked mon.noname-b con 0x7f17f40a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17f9941640 10 monclient(hunting): picked mon.noname-g con 0x7f17f40a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17f9941640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f4136790 con 0x7f17f40a5920 2024-08-19T12:47:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17f9941640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f409ddf0 con 0x7f17f40a5d50 2024-08-19T12:47:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17f9941640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f409e0d0 con 0x7f17f406e7b0 2024-08-19T12:47:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17f9941640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17f9941640 10 monclient(hunting): authenticate will time out at 1217.026242s 2024-08-19T12:47:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17f2ffd640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f17f406e7b0 0x7f17f4136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:43116/0 (socket says 172.21.15.26:43116) 2024-08-19T12:47:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17ebfff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f17f40a5920 0x7f17f413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39052/0 (socket says 172.21.15.26:39052) 2024-08-19T12:47:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.939+0000 7f17f2ffd640 1 -- 172.21.15.26:0/3214813602 learned_addr learned my addr 172.21.15.26:0/3214813602 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f2ffd640 1 -- 172.21.15.26:0/3214813602 >> v1:172.21.15.26:6792/0 conn(0x7f17f406e7b0 legacy=0x7f17f4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17ebfff640 1 -- 172.21.15.26:0/3214813602 >> v1:172.21.15.26:6790/0 conn(0x7f17f40a5920 legacy=0x7f17f413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2612306424 0 0) 0x7f17f409e0d0 con 0x7f17f406e7b0 2024-08-19T12:47:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient(hunting): my global_id is 164193 2024-08-19T12:47:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17e0003520 con 0x7f17f406e7b0 2024-08-19T12:47:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2134756121 0 0) 0x7f17f4136790 con 0x7f17f40a5920 2024-08-19T12:47:02.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient(hunting): my global_id is 26561 2024-08-19T12:47:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17f409e0d0 con 0x7f17f40a5920 2024-08-19T12:47:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f37fe640 1 -- 172.21.15.26:0/3214813602 >> v1:172.21.15.104:6790/0 conn(0x7f17f40a5d50 legacy=0x7f17f4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1889943817 0 0) 0x7f17ec0017e0 con 0x7f17f40a5d50 2024-08-19T12:47:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.953 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient(hunting): my global_id is 154389 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17f4136790 con 0x7f17f40a5d50 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3203485420 0 0) 0x7f17f409e0d0 con 0x7f17f40a5920 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 >> v1:172.21.15.26:6792/0 conn(0x7f17f406e7b0 legacy=0x7f17f4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 >> v1:172.21.15.104:6790/0 conn(0x7f17f40a5d50 legacy=0x7f17f4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient: _finish_hunting 0 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 1 monclient: found mon.noname-b 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f409e3b0 con 0x7f17f40a5920 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient: _finish_auth 0 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 10 monclient: _check_auth_tickets 2024-08-19T12:47:02.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f17dc0019f0 con 0x7f17f40a5920 2024-08-19T12:47:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f9941640 5 monclient: authenticate success, global_id 26561 2024-08-19T12:47:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.941+0000 7f17f9941640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:47:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f27fc640 10 monclient: dump: 2024-08-19T12:47:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.955 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.956 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f9941640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17dc001cc0 con 0x7f17f40a5920 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f9941640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f27fc640 1 -- 172.21.15.26:0/3214813602 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f17dc004b20 con 0x7f17f40a5920 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f9941640 4 set_mon_vals no callback set 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f27fc640 10 monclient: dump: 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.957 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.958 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.958 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.958 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.958 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.958 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.958 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.958 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.958 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.958 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.958 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.958 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f9941640 10 monclient: shutdown 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f9941640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f9941640 1 -- 172.21.15.26:0/3214813602 >> v1:172.21.15.26:6790/0 conn(0x7f17f40a5920 legacy=0x7f17f413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f9941640 1 -- 172.21.15.26:0/3214813602 shutdown_connections 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f9941640 1 -- 172.21.15.26:0/3214813602 >> 172.21.15.26:0/3214813602 conn(0x7f17f40a8740 msgr2=0x7f17f40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f9941640 1 -- 172.21.15.26:0/3214813602 shutdown_connections 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.942+0000 7f17f9941640 1 -- 172.21.15.26:0/3214813602 wait complete. 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.943+0000 7f17f9941640 10 monclient: build_initial_monmap 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.943+0000 7f17f9941640 10 monclient: monmap: 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:02.959 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.943+0000 7f17f9941640 1 librados: starting msgr at 2024-08-19T12:47:02.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.943+0000 7f17f9941640 1 librados: starting objecter 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.943+0000 7f17f9941640 1 Processor -- start 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.943+0000 7f17f9941640 1 -- start start 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.943+0000 7f17f9941640 1 librados: setting wanted keys 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.943+0000 7f17f9941640 1 librados: calling monclient init 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.943+0000 7f17f9941640 10 monclient: init 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.943+0000 7f17f9941640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.943+0000 7f17f9941640 10 monclient: _add_conns ranks=[5,8,2,0,6,4,1,3,7] 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.944+0000 7f17f9941640 10 monclient(hunting): picked mon.noname-h con 0x7f17f406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.944+0000 7f17f9941640 10 monclient(hunting): picked mon.noname-e con 0x7f17f40a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.944+0000 7f17f9941640 10 monclient(hunting): picked mon.noname-b con 0x7f17f40a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.944+0000 7f17f9941640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f40702b0 con 0x7f17f40a5d50 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.944+0000 7f17f9941640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f4142c40 con 0x7f17f406e7b0 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.944+0000 7f17f9941640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f4071210 con 0x7f17f40a5920 2024-08-19T12:47:02.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.944+0000 7f17f9941640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.944+0000 7f17f9941640 10 monclient(hunting): authenticate will time out at 1217.031095s 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.944+0000 7f17ebfff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f17f40a5920 0x7f17f41d72b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:56184/0 (socket says 172.21.15.26:56184) 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.944+0000 7f17ebfff640 1 -- 172.21.15.26:0/530316923 learned_addr learned my addr 172.21.15.26:0/530316923 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.944+0000 7f17f37fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f17f40a5d50 0x7f17f41da9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39054/0 (socket says 172.21.15.26:39054) 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17ebfff640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.26:6793/0 conn(0x7f17f40a5920 legacy=0x7f17f41d72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f37fe640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.26:6790/0 conn(0x7f17f40a5d50 legacy=0x7f17f41da9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 client.?.objecter ms_handle_connect 0x7f17f40a5920 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 client.?.objecter ms_handle_connect 0x7f17f40a5d50 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 client.?.objecter ms_handle_connect 0x7f17f406e7b0 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f2ffd640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.104:6791/0 conn(0x7f17f406e7b0 legacy=0x7f17f40a4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3899261871 0 0) 0x7f17f40702b0 con 0x7f17f40a5d50 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): my global_id is 26570 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17d4004eb0 con 0x7f17f40a5d50 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847779162 0 0) 0x7f17f4071210 con 0x7f17f40a5920 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): my global_id is 26405 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17f40702b0 con 0x7f17f40a5920 2024-08-19T12:47:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 978764821 0 0) 0x7f17f4142c40 con 0x7f17f406e7b0 2024-08-19T12:47:02.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): my global_id is 26573 2024-08-19T12:47:02.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17f4071210 con 0x7f17f406e7b0 2024-08-19T12:47:02.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1692638679 0 0) 0x7f17d4004eb0 con 0x7f17f40a5d50 2024-08-19T12:47:02.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17f4142c40 con 0x7f17f40a5d50 2024-08-19T12:47:02.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f17ec003990 con 0x7f17f40a5d50 2024-08-19T12:47:02.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:02.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:47:02.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): dump: 2024-08-19T12:47:02.964 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.966 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.966 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:02.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 388104563 0 0) 0x7f17f40702b0 con 0x7f17f40a5920 2024-08-19T12:47:02.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d4004eb0 con 0x7f17f40a5920 2024-08-19T12:47:02.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f17dc004b20 con 0x7f17f40a5920 2024-08-19T12:47:02.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:02.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:47:02.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): dump: 2024-08-19T12:47:02.966 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.966 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.966 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.967 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 661425068 0 0) 0x7f17f4071210 con 0x7f17f406e7b0 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17f40702b0 con 0x7f17f406e7b0 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f17e40047e0 con 0x7f17f406e7b0 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 10 monclient(hunting): dump: 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.968 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.969 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.969 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.969 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.969 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.969 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.969 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.969 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.946+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 252277107 0 0) 0x7f17f4142c40 con 0x7f17f40a5d50 2024-08-19T12:47:02.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.26:6793/0 conn(0x7f17f40a5920 legacy=0x7f17f41d72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.104:6791/0 conn(0x7f17f406e7b0 legacy=0x7f17f40a4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:47:02.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 1 monclient: found mon.b 2024-08-19T12:47:02.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:02.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:47:02.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f40714f0 con 0x7f17f40a5d50 2024-08-19T12:47:02.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 10 monclient: _finish_auth 0 2024-08-19T12:47:02.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:47:02.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:02.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f9941640 5 monclient: authenticate success, global_id 26570 2024-08-19T12:47:02.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f9941640 10 monclient: _renew_subs 2024-08-19T12:47:02.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f9941640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:47:02.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f9941640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f17f406fe30 con 0x7f17f40a5d50 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f9941640 10 client.26570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f9941640 10 monclient: _renew_subs 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f9941640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f9941640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f17f40a55c0 con 0x7f17f40a5d50 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f9941640 1 librados: init done 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17ec003e90 con 0x7f17f40a5d50 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f17ec004400 con 0x7f17f40a5d50 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f1ffb640 4 set_mon_vals no callback set 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 10 monclient: dump: 2024-08-19T12:47:02.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.972 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.947+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f17ec0045a0 con 0x7f17f40a5d50 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.948+0000 7f17f9941640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.948+0000 7f17f9941640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f17d80065c0 con 0x7f17d8002860 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.948+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(353..353 src has 1..353) ==== 5346+0+0 (unknown 3503228669 0 0) 0x7f17ec04e2d0 con 0x7f17f40a5d50 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.948+0000 7f17f0ff9640 10 client.26570.objecter ms_dispatch 0x7f17f4000b90 osd_map(353..353 src has 1..353) 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.948+0000 7f17f0ff9640 3 client.26570.objecter handle_osd_map got epochs [353,353] > 0 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.948+0000 7f17f0ff9640 3 client.26570.objecter handle_osd_map decoding full epoch 353 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.948+0000 7f17f0ff9640 20 client.26570.objecter dump_active .. 0 homeless 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.948+0000 7f17ebfff640 10 monclient: get_auth_request con 0x7f17d4041910 auth_method 0 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 10 client.26570.objecter ms_handle_connect 0x7f17d8002860 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 10 client.26570.objecter resend_mon_ops 2024-08-19T12:47:02.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f2ffd640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.26:6792/0 conn(0x7f17d8002860 legacy=0x7f17d8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1255991117 0 0) 0x7f17d80065c0 con 0x7f17d8002860 2024-08-19T12:47:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 10 monclient(hunting): my global_id is 164202 2024-08-19T12:47:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17f4142c40 con 0x7f17d8002860 2024-08-19T12:47:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 694258447 0 0) 0x7f17f4142c40 con 0x7f17d8002860 2024-08-19T12:47:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d80065c0 con 0x7f17d8002860 2024-08-19T12:47:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f17e4004eb0 con 0x7f17d8002860 2024-08-19T12:47:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:02.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 10 monclient: dump: 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.975 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.976 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 258617858 0 0) 0x7f17d80065c0 con 0x7f17d8002860 2024-08-19T12:47:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17d800a160 con 0x7f17d8002860 2024-08-19T12:47:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.950+0000 7f17f0ff9640 10 client.26570.objecter ms_handle_connect 0x7f17d4041910 2024-08-19T12:47:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.951+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f17e4006370 con 0x7f17d8002860 2024-08-19T12:47:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.951+0000 7f17f0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.951+0000 7f17f0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.951+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.26:6792/0 conn(0x7f17d8002860 legacy=0x7f17d8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.951+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.26:6792/0 conn(0x7f17d8002860 legacy=0x7f17d8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.957+0000 7f17f9941640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.957+0000 7f17f9941640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f17d8009e50 con 0x7f17d8009110 2024-08-19T12:47:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 10 client.26570.objecter ms_handle_connect 0x7f17d8009110 2024-08-19T12:47:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f2ffd640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.26:6792/0 conn(0x7f17d8009110 legacy=0x7f17d80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 10 client.26570.objecter resend_mon_ops 2024-08-19T12:47:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 299458217 0 0) 0x7f17d8009e50 con 0x7f17d8009110 2024-08-19T12:47:02.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 10 monclient(hunting): my global_id is 164211 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17d80065c0 con 0x7f17d8009110 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1095966352 0 0) 0x7f17d80065c0 con 0x7f17d8009110 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d8009e50 con 0x7f17d8009110 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f17e4005460 con 0x7f17d8009110 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 10 monclient: dump: 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:02.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:02.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3762769614 0 0) 0x7f17d8009e50 con 0x7f17d8009110 2024-08-19T12:47:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.959+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f17d800c3f0 con 0x7f17d8009110 2024-08-19T12:47:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7542 (unknown 0 0 696770372) 0x7f17e40057f0 con 0x7f17d8009110 2024-08-19T12:47:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.26:6792/0 conn(0x7f17d8009110 legacy=0x7f17d80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:02.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f0ff9640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.26:6792/0 conn(0x7f17d8009110 legacy=0x7f17d80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:02.981 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:02.981 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:02.981 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:02.981 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 6514, 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:47:02.982 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:02.983 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:02.984 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:02.985 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:02.986 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:02.989 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.990 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.993 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.994 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.995 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:02.996 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.997 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:02.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.000 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:03.000 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 10 librados: watch_flush enter 2024-08-19T12:47:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 10 librados: watch_flush exit 2024-08-19T12:47:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 20 client.26570.objecter shutdown clearing up homeless session... 2024-08-19T12:47:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 10 client.26570.objecter successfully canceled tick 2024-08-19T12:47:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.104:6800/899887506 conn(0x7f17d4041910 legacy=0x7f17d4043dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 10 monclient: shutdown 2024-08-19T12:47:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:03.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 1 -- 172.21.15.26:0/530316923 >> v1:172.21.15.26:6790/0 conn(0x7f17f40a5d50 legacy=0x7f17f41da9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f37fe640 1 -- 172.21.15.26:0/530316923 reap_dead start 2024-08-19T12:47:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f37fe640 1 -- 172.21.15.26:0/530316923 reap_dead start 2024-08-19T12:47:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 1 -- 172.21.15.26:0/530316923 shutdown_connections 2024-08-19T12:47:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 1 -- 172.21.15.26:0/530316923 >> 172.21.15.26:0/530316923 conn(0x7f17f40a8740 msgr2=0x7f17f406dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 1 -- 172.21.15.26:0/530316923 shutdown_connections 2024-08-19T12:47:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 1 -- 172.21.15.26:0/530316923 wait complete. 2024-08-19T12:47:03.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:02.960+0000 7f17f9941640 1 librados: shutdown 2024-08-19T12:47:03.003 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:47:03.003 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:47:03.004 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:47:03.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.103+0000 7ff1bab7b640 10 monclient: get_monmap_and_config 2024-08-19T12:47:03.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.103+0000 7ff1bab7b640 10 monclient: build_initial_monmap 2024-08-19T12:47:03.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.103+0000 7ff1bab7b640 10 monclient: monmap: 2024-08-19T12:47:03.105 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:03.105 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.105 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:03.104531+0000 2024-08-19T12:47:03.105 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:03.104531+0000 2024-08-19T12:47:03.105 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:03.105 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.105 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:03.105 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:03.105 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:03.105 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:03.105 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:03.106 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:03.106 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:03.106 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:03.106 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:03.106 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 1 Processor -- start 2024-08-19T12:47:03.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 1 -- start start 2024-08-19T12:47:03.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 10 monclient: init 2024-08-19T12:47:03.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:03.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 10 monclient: _add_conns ranks=[6,2,5,7,8,0,1,4,3] 2024-08-19T12:47:03.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 10 monclient(hunting): picked mon.noname-d con 0x7ff1b406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:03.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 10 monclient(hunting): picked mon.noname-b con 0x7ff1b40a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:03.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 10 monclient(hunting): picked mon.noname-h con 0x7ff1b40a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:03.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1b4136790 con 0x7ff1b40a5920 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1b409ddf0 con 0x7ff1b40a5d50 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1b409e0d0 con 0x7ff1b406e7b0 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.104+0000 7ff1bab7b640 10 monclient(hunting): authenticate will time out at 1217.191949s 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.105+0000 7ff1b3fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff1b40a5920 0x7ff1b413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39066/0 (socket says 172.21.15.26:39066) 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.105+0000 7ff1b3fff640 1 -- 172.21.15.26:0/229790750 learned_addr learned my addr 172.21.15.26:0/229790750 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b3fff640 1 -- 172.21.15.26:0/229790750 >> v1:172.21.15.26:6790/0 conn(0x7ff1b40a5920 legacy=0x7ff1b413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b8917640 1 -- 172.21.15.26:0/229790750 >> v1:172.21.15.26:6792/0 conn(0x7ff1b406e7b0 legacy=0x7ff1b4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 1 -- 172.21.15.26:0/229790750 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4043695452 0 0) 0x7ff1b4136790 con 0x7ff1b40a5920 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient(hunting): my global_id is 26579 2024-08-19T12:47:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 1 -- 172.21.15.26:0/229790750 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1a4003520 con 0x7ff1b40a5920 2024-08-19T12:47:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 1 -- 172.21.15.26:0/229790750 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3766049764 0 0) 0x7ff1b409e0d0 con 0x7ff1b406e7b0 2024-08-19T12:47:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient(hunting): my global_id is 164220 2024-08-19T12:47:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 1 -- 172.21.15.26:0/229790750 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1b4136790 con 0x7ff1b406e7b0 2024-08-19T12:47:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b9118640 1 -- 172.21.15.26:0/229790750 >> v1:172.21.15.104:6791/0 conn(0x7ff1b40a5d50 legacy=0x7ff1b4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 1 -- 172.21.15.26:0/229790750 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 584157693 0 0) 0x7ff1a4003520 con 0x7ff1b40a5920 2024-08-19T12:47:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 1 -- 172.21.15.26:0/229790750 >> v1:172.21.15.26:6792/0 conn(0x7ff1b406e7b0 legacy=0x7ff1b4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 1 -- 172.21.15.26:0/229790750 >> v1:172.21.15.104:6791/0 conn(0x7ff1b40a5d50 legacy=0x7ff1b4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.113 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient: _finish_hunting 0 2024-08-19T12:47:03.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 1 monclient: found mon.noname-b 2024-08-19T12:47:03.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:03.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:47:03.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 1 -- 172.21.15.26:0/229790750 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b409e3b0 con 0x7ff1b40a5920 2024-08-19T12:47:03.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient: _finish_auth 0 2024-08-19T12:47:03.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient: _check_auth_tickets 2024-08-19T12:47:03.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:03.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 1 -- 172.21.15.26:0/229790750 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1a0001ae0 con 0x7ff1b40a5920 2024-08-19T12:47:03.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient: dump: 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.115 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1bab7b640 5 monclient: authenticate success, global_id 26579 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1bab7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 1 -- 172.21.15.26:0/229790750 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1a00048a0 con 0x7ff1b40a5920 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 1 -- 172.21.15.26:0/229790750 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1a0004e10 con 0x7ff1b40a5920 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1bab7b640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1bab7b640 4 set_mon_vals no callback set 2024-08-19T12:47:03.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:03.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.107+0000 7ff1b37fe640 10 monclient: dump: 2024-08-19T12:47:03.117 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.117 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.117 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.117 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.117 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.117 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.117 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.117 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.118 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.118 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.118 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.118 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.118 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.118 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.118 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.118 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.108+0000 7ff1bab7b640 10 monclient: shutdown 2024-08-19T12:47:03.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.108+0000 7ff1bab7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:03.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.108+0000 7ff1bab7b640 1 -- 172.21.15.26:0/229790750 >> v1:172.21.15.26:6790/0 conn(0x7ff1b40a5920 legacy=0x7ff1b413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.108+0000 7ff1bab7b640 1 -- 172.21.15.26:0/229790750 shutdown_connections 2024-08-19T12:47:03.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.108+0000 7ff1bab7b640 1 -- 172.21.15.26:0/229790750 >> 172.21.15.26:0/229790750 conn(0x7ff1b40a8740 msgr2=0x7ff1b40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:03.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.108+0000 7ff1bab7b640 1 -- 172.21.15.26:0/229790750 shutdown_connections 2024-08-19T12:47:03.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.108+0000 7ff1bab7b640 1 -- 172.21.15.26:0/229790750 wait complete. 2024-08-19T12:47:03.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.108+0000 7ff1bab7b640 10 monclient: build_initial_monmap 2024-08-19T12:47:03.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.108+0000 7ff1bab7b640 10 monclient: monmap: 2024-08-19T12:47:03.119 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:03.119 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:03.119 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:03.119 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:03.119 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:03.120 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.120 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:03.120 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:03.120 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:03.120 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:03.120 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:03.120 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:03.120 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:03.120 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:03.120 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:03.121 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 1 librados: starting msgr at 2024-08-19T12:47:03.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 1 librados: starting objecter 2024-08-19T12:47:03.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 1 Processor -- start 2024-08-19T12:47:03.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 1 -- start start 2024-08-19T12:47:03.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 1 librados: setting wanted keys 2024-08-19T12:47:03.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 1 librados: calling monclient init 2024-08-19T12:47:03.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 10 monclient: init 2024-08-19T12:47:03.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:03.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 10 monclient: _add_conns ranks=[3,0,1,6,8,2,7,4,5] 2024-08-19T12:47:03.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 10 monclient(hunting): picked mon.noname-g con 0x7ff1b406e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:03.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 10 monclient(hunting): picked mon.noname-a con 0x7ff1b40a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:03.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 10 monclient(hunting): picked mon.noname-f con 0x7ff1b40a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:03.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1b4126fe0 con 0x7ff1b40a5920 2024-08-19T12:47:03.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1b4134880 con 0x7ff1b40a5d50 2024-08-19T12:47:03.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1b4142ec0 con 0x7ff1b406e7b0 2024-08-19T12:47:03.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:03.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1bab7b640 10 monclient(hunting): authenticate will time out at 1217.196816s 2024-08-19T12:47:03.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1b3fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff1b40a5920 0x7ff1b4123df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48032/0 (socket says 172.21.15.26:48032) 2024-08-19T12:47:03.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.109+0000 7ff1b3fff640 1 -- 172.21.15.26:0/338349795 learned_addr learned my addr 172.21.15.26:0/338349795 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:03.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b3fff640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.26:6789/0 conn(0x7ff1b40a5920 legacy=0x7ff1b4123df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 client.?.objecter ms_handle_connect 0x7ff1b40a5920 2024-08-19T12:47:03.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 client.?.objecter ms_handle_connect 0x7ff1b406e7b0 2024-08-19T12:47:03.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 client.?.objecter ms_handle_connect 0x7ff1b40a5d50 2024-08-19T12:47:03.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b8917640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.104:6790/0 conn(0x7ff1b406e7b0 legacy=0x7ff1b41236e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b9118640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.104:6789/0 conn(0x7ff1b40a5d50 legacy=0x7ff1b41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 394728749 0 0) 0x7ff1b4126fe0 con 0x7ff1b40a5920 2024-08-19T12:47:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): my global_id is 144891 2024-08-19T12:47:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff198004eb0 con 0x7ff1b40a5920 2024-08-19T12:47:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 896319020 0 0) 0x7ff1b4142ec0 con 0x7ff1b406e7b0 2024-08-19T12:47:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): my global_id is 154398 2024-08-19T12:47:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1b4126fe0 con 0x7ff1b406e7b0 2024-08-19T12:47:03.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2251585610 0 0) 0x7ff1b4134880 con 0x7ff1b40a5d50 2024-08-19T12:47:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): my global_id is 27370 2024-08-19T12:47:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1b4142ec0 con 0x7ff1b40a5d50 2024-08-19T12:47:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3895521122 0 0) 0x7ff198004eb0 con 0x7ff1b40a5920 2024-08-19T12:47:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1b4134880 con 0x7ff1b40a5920 2024-08-19T12:47:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1a0004d90 con 0x7ff1b40a5920 2024-08-19T12:47:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:03.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): dump: 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.132 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.133 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4152182542 0 0) 0x7ff1b4126fe0 con 0x7ff1b406e7b0 2024-08-19T12:47:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff198004eb0 con 0x7ff1b406e7b0 2024-08-19T12:47:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1a8003610 con 0x7ff1b406e7b0 2024-08-19T12:47:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:47:03.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 10 monclient(hunting): dump: 2024-08-19T12:47:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.134 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1801092939 0 0) 0x7ff1b4134880 con 0x7ff1b40a5920 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.112+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.104:6790/0 conn(0x7ff1b406e7b0 legacy=0x7ff1b41236e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.104:6789/0 conn(0x7ff1b40a5d50 legacy=0x7ff1b41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 10 monclient: _finish_hunting 0 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 1 monclient: found mon.a 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:03.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b4124c80 con 0x7ff1b40a5920 2024-08-19T12:47:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 10 monclient: _finish_auth 0 2024-08-19T12:47:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 10 monclient: _check_auth_tickets 2024-08-19T12:47:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1bab7b640 5 monclient: authenticate success, global_id 144891 2024-08-19T12:47:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1bab7b640 10 monclient: _renew_subs 2024-08-19T12:47:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1bab7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1bab7b640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b4126d00 con 0x7ff1b40a5920 2024-08-19T12:47:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1bab7b640 10 client.144891.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1bab7b640 10 monclient: _renew_subs 2024-08-19T12:47:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1bab7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:03.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1bab7b640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1b412de70 con 0x7ff1b40a5920 2024-08-19T12:47:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1bab7b640 1 librados: init done 2024-08-19T12:47:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1a0001d30 con 0x7ff1b40a5920 2024-08-19T12:47:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1a00048a0 con 0x7ff1b40a5920 2024-08-19T12:47:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b2ffd640 4 set_mon_vals no callback set 2024-08-19T12:47:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 10 monclient: dump: 2024-08-19T12:47:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.137 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.138 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.139 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1bab7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1bab7b640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff1940065c0 con 0x7ff194002860 2024-08-19T12:47:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.113+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff1a0011a00 con 0x7ff1b40a5920 2024-08-19T12:47:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.114+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(353..353 src has 1..353) ==== 5346+0+0 (unknown 3503228669 0 0) 0x7ff1a004e150 con 0x7ff1b40a5920 2024-08-19T12:47:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.114+0000 7ff1b17fa640 10 client.144891.objecter ms_dispatch 0x7ff1b4000b90 osd_map(353..353 src has 1..353) 2024-08-19T12:47:03.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.114+0000 7ff1b17fa640 3 client.144891.objecter handle_osd_map got epochs [353,353] > 0 2024-08-19T12:47:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.114+0000 7ff1b17fa640 3 client.144891.objecter handle_osd_map decoding full epoch 353 2024-08-19T12:47:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.114+0000 7ff1b17fa640 20 client.144891.objecter dump_active .. 0 homeless 2024-08-19T12:47:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.114+0000 7ff1b9118640 10 monclient: get_auth_request con 0x7ff198041620 auth_method 0 2024-08-19T12:47:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b8917640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.26:6793/0 conn(0x7ff194002860 legacy=0x7ff194004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 10 client.144891.objecter ms_handle_connect 0x7ff194002860 2024-08-19T12:47:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 10 client.144891.objecter resend_mon_ops 2024-08-19T12:47:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3187238029 0 0) 0x7ff1940065c0 con 0x7ff194002860 2024-08-19T12:47:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 10 monclient(hunting): my global_id is 26414 2024-08-19T12:47:03.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1b4134880 con 0x7ff194002860 2024-08-19T12:47:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3326638512 0 0) 0x7ff1b4134880 con 0x7ff194002860 2024-08-19T12:47:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1940065c0 con 0x7ff194002860 2024-08-19T12:47:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1a8003da0 con 0x7ff194002860 2024-08-19T12:47:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:47:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 10 monclient: dump: 2024-08-19T12:47:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.141 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.142 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.142 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.142 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.142 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.142 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.142 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.142 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.142 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.142 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.142 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.143 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2225629519 0 0) 0x7ff1940065c0 con 0x7ff194002860 2024-08-19T12:47:03.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff19400a160 con 0x7ff194002860 2024-08-19T12:47:03.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 10 client.144891.objecter ms_handle_connect 0x7ff198041620 2024-08-19T12:47:03.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff1a80040c0 con 0x7ff194002860 2024-08-19T12:47:03.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:03.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:03.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.26:6793/0 conn(0x7ff194002860 legacy=0x7ff194004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.116+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.26:6793/0 conn(0x7ff194002860 legacy=0x7ff194004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:03.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.122+0000 7ff1bab7b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:03.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.123+0000 7ff1bab7b640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff194009e70 con 0x7ff194009130 2024-08-19T12:47:03.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b8917640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.26:6793/0 conn(0x7ff194009130 legacy=0x7ff1940095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 10 client.144891.objecter ms_handle_connect 0x7ff194009130 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 10 client.144891.objecter resend_mon_ops 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1994536513 0 0) 0x7ff194009e70 con 0x7ff194009130 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 10 monclient(hunting): my global_id is 26423 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1940065c0 con 0x7ff194009130 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3747675435 0 0) 0x7ff1940065c0 con 0x7ff194009130 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff194009e70 con 0x7ff194009130 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1a8005b40 con 0x7ff194009130 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:47:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 10 monclient: dump: 2024-08-19T12:47:03.145 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.145 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.145 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.145 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.145 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.145 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.145 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.145 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.145 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.145 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.145 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.146 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 588095522 0 0) 0x7ff194009e70 con 0x7ff194009130 2024-08-19T12:47:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff19400c410 con 0x7ff194009130 2024-08-19T12:47:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1580373837) 0x7ff1a8005e50 con 0x7ff194009130 2024-08-19T12:47:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.26:6793/0 conn(0x7ff194009130 legacy=0x7ff1940095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.125+0000 7ff1b17fa640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.26:6793/0 conn(0x7ff194009130 legacy=0x7ff1940095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:03.147 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:03.147 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:03.147 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:03.147 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:03.147 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:47:03.147 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 536142, 2024-08-19T12:47:03.147 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:03.148 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:03.149 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:03.149 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.149 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.149 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.149 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.149 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.149 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.149 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.149 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.150 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:03.151 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:03.152 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:03.152 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:03.152 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:03.152 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:03.152 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:03.152 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:03.152 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:03.152 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:03.152 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.153 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.154 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.156 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:03.156 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:03.156 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.156 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.156 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.156 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:03.156 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.156 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.157 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:03.158 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.158 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.158 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.158 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:03.158 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:03.158 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.158 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.158 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.158 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:03.159 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:03.159 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.159 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.159 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.159 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:03.159 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.159 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.159 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.159 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:03.159 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.162 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:03.165 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.166 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.168 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.169 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:03.169 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:03.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 10 librados: watch_flush enter 2024-08-19T12:47:03.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 10 librados: watch_flush exit 2024-08-19T12:47:03.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 20 client.144891.objecter shutdown clearing up homeless session... 2024-08-19T12:47:03.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 10 client.144891.objecter successfully canceled tick 2024-08-19T12:47:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.104:6800/899887506 conn(0x7ff198041620 legacy=0x7ff198043ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 10 monclient: shutdown 2024-08-19T12:47:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 1 -- 172.21.15.26:0/338349795 >> v1:172.21.15.26:6789/0 conn(0x7ff1b40a5920 legacy=0x7ff1b4123df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1b9118640 1 -- 172.21.15.26:0/338349795 reap_dead start 2024-08-19T12:47:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1b9118640 1 -- 172.21.15.26:0/338349795 reap_dead start 2024-08-19T12:47:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 1 -- 172.21.15.26:0/338349795 shutdown_connections 2024-08-19T12:47:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 1 -- 172.21.15.26:0/338349795 >> 172.21.15.26:0/338349795 conn(0x7ff1b40a8740 msgr2=0x7ff1b406d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 1 -- 172.21.15.26:0/338349795 shutdown_connections 2024-08-19T12:47:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 1 -- 172.21.15.26:0/338349795 wait complete. 2024-08-19T12:47:03.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.126+0000 7ff1bab7b640 1 librados: shutdown 2024-08-19T12:47:03.173 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:47:03.173 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:47:03.173 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:47:03.215 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.212+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.212+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003931542}) 2024-08-19T12:47:03.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.212+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3722881034}) 2024-08-19T12:47:03.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003931542}) 2024-08-19T12:47:03.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426987107}) 2024-08-19T12:47:03.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003931542}) 2024-08-19T12:47:03.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2240636615}) 2024-08-19T12:47:03.216 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003931542}) 2024-08-19T12:47:03.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1130002070}) 2024-08-19T12:47:03.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003931542}) 2024-08-19T12:47:03.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=375365940}) 2024-08-19T12:47:03.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003931542}) 2024-08-19T12:47:03.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3802623730}) 2024-08-19T12:47:03.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003931542}) 2024-08-19T12:47:03.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=722834747}) 2024-08-19T12:47:03.217 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2003931542}) 2024-08-19T12:47:03.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.213+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=327191280}) 2024-08-19T12:47:03.228 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.229 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=25004716}) 2024-08-19T12:47:03.229 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773797678}) 2024-08-19T12:47:03.229 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.229 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=25004716}) 2024-08-19T12:47:03.229 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=988931335}) 2024-08-19T12:47:03.229 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.229 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=25004716}) 2024-08-19T12:47:03.229 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4124616007}) 2024-08-19T12:47:03.229 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.229 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=25004716}) 2024-08-19T12:47:03.230 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2322738787}) 2024-08-19T12:47:03.230 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.230 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=25004716}) 2024-08-19T12:47:03.230 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3603263209}) 2024-08-19T12:47:03.230 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.230 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=25004716}) 2024-08-19T12:47:03.230 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1442497474}) 2024-08-19T12:47:03.230 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.230 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=25004716}) 2024-08-19T12:47:03.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3260702754}) 2024-08-19T12:47:03.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=25004716}) 2024-08-19T12:47:03.231 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1002007857}) 2024-08-19T12:47:03.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652184179}) 2024-08-19T12:47:03.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983497994}) 2024-08-19T12:47:03.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652184179}) 2024-08-19T12:47:03.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1236557297}) 2024-08-19T12:47:03.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652184179}) 2024-08-19T12:47:03.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3456177336}) 2024-08-19T12:47:03.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652184179}) 2024-08-19T12:47:03.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4294122429}) 2024-08-19T12:47:03.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.237 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652184179}) 2024-08-19T12:47:03.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1800104649}) 2024-08-19T12:47:03.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652184179}) 2024-08-19T12:47:03.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3174238427}) 2024-08-19T12:47:03.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652184179}) 2024-08-19T12:47:03.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3341910435}) 2024-08-19T12:47:03.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652184179}) 2024-08-19T12:47:03.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=310064221}) 2024-08-19T12:47:03.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809636981}) 2024-08-19T12:47:03.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2289595974}) 2024-08-19T12:47:03.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809636981}) 2024-08-19T12:47:03.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=858741127}) 2024-08-19T12:47:03.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809636981}) 2024-08-19T12:47:03.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2743659389}) 2024-08-19T12:47:03.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809636981}) 2024-08-19T12:47:03.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2949440540}) 2024-08-19T12:47:03.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809636981}) 2024-08-19T12:47:03.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2407422212}) 2024-08-19T12:47:03.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809636981}) 2024-08-19T12:47:03.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2170491798}) 2024-08-19T12:47:03.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809636981}) 2024-08-19T12:47:03.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2580736255}) 2024-08-19T12:47:03.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809636981}) 2024-08-19T12:47:03.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.279+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1914183227}) 2024-08-19T12:47:03.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2322445686}) 2024-08-19T12:47:03.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=90394535}) 2024-08-19T12:47:03.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2322445686}) 2024-08-19T12:47:03.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2431863285}) 2024-08-19T12:47:03.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2322445686}) 2024-08-19T12:47:03.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1625349075}) 2024-08-19T12:47:03.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2322445686}) 2024-08-19T12:47:03.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=823040751}) 2024-08-19T12:47:03.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2322445686}) 2024-08-19T12:47:03.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=154907847}) 2024-08-19T12:47:03.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2322445686}) 2024-08-19T12:47:03.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4294137221}) 2024-08-19T12:47:03.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2322445686}) 2024-08-19T12:47:03.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2985495683}) 2024-08-19T12:47:03.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2322445686}) 2024-08-19T12:47:03.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1058006279}) 2024-08-19T12:47:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.285+0000 7f48c4328640 10 monclient: get_monmap_and_config 2024-08-19T12:47:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.285+0000 7f48c4328640 10 monclient: build_initial_monmap 2024-08-19T12:47:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.285+0000 7f48c4328640 10 monclient: monmap: 2024-08-19T12:47:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:03.286307+0000 2024-08-19T12:47:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:03.286307+0000 2024-08-19T12:47:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:03.289 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:03.290 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:03.290 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.290+0000 7f48c4328640 1 Processor -- start 2024-08-19T12:47:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.290+0000 7f48c4328640 1 -- start start 2024-08-19T12:47:03.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.290+0000 7f48c4328640 10 monclient: init 2024-08-19T12:47:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.291+0000 7f48c4328640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:03.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.291+0000 7f48c4328640 10 monclient: _add_conns ranks=[4,8,7,1,2,5,6,3,0] 2024-08-19T12:47:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.291+0000 7f48c4328640 10 monclient(hunting): picked mon.noname-c con 0x7f48bc0a19f0 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.291+0000 7f48c4328640 10 monclient(hunting): picked mon.noname-e con 0x7f48bc1392c0 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.291+0000 7f48c4328640 10 monclient(hunting): picked mon.noname-i con 0x7f48bc1361a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:47:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.291+0000 7f48c4328640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48bc0a7300 con 0x7f48bc0a19f0 2024-08-19T12:47:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.291+0000 7f48c4328640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48bc0a0970 con 0x7f48bc1361a0 2024-08-19T12:47:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.291+0000 7f48c4328640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48bc09fbd0 con 0x7f48bc1392c0 2024-08-19T12:47:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.291+0000 7f48c4328640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.291+0000 7f48c4328640 10 monclient(hunting): authenticate will time out at 1217.378293s 2024-08-19T12:47:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.291+0000 7f48c28c5640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f48bc1361a0 0x7f48bc136630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:35916/0 (socket says 172.21.15.26:35916) 2024-08-19T12:47:03.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.291+0000 7f48c28c5640 1 -- 172.21.15.26:0/4209217833 learned_addr learned my addr 172.21.15.26:0/4209217833 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.293+0000 7f48c28c5640 1 -- 172.21.15.26:0/4209217833 >> v1:172.21.15.104:6792/0 conn(0x7f48bc1361a0 legacy=0x7f48bc136630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c20c4640 1 -- 172.21.15.26:0/4209217833 >> v1:172.21.15.26:6791/0 conn(0x7f48bc0a19f0 legacy=0x7f48bc0a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3061869287 0 0) 0x7f48bc0a0970 con 0x7f48bc1361a0 2024-08-19T12:47:03.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c18c3640 1 -- 172.21.15.26:0/4209217833 >> v1:172.21.15.26:6793/0 conn(0x7f48bc1392c0 legacy=0x7f48bc13b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 10 monclient(hunting): my global_id is 135520 2024-08-19T12:47:03.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48a8003520 con 0x7f48bc1361a0 2024-08-19T12:47:03.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3029705921 0 0) 0x7f48bc09fbd0 con 0x7f48bc1392c0 2024-08-19T12:47:03.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 10 monclient(hunting): my global_id is 26432 2024-08-19T12:47:03.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48bc0a0970 con 0x7f48bc1392c0 2024-08-19T12:47:03.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3907101382 0 0) 0x7f48bc0a7300 con 0x7f48bc0a19f0 2024-08-19T12:47:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 10 monclient(hunting): my global_id is 26914 2024-08-19T12:47:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.294+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48bc09fbd0 con 0x7f48bc0a19f0 2024-08-19T12:47:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 963223013 0 0) 0x7f48bc09fbd0 con 0x7f48bc0a19f0 2024-08-19T12:47:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c28c5640 1 --1- 172.21.15.26:0/4209217833 >> v1:172.21.15.104:6792/0 conn(0x7f48bc1361a0 0x7f48bc136630 :-1 s=OPENED pgs=438 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f48a8003520 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:47:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 >> v1:172.21.15.26:6793/0 conn(0x7f48bc1392c0 legacy=0x7f48bc13b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 >> v1:172.21.15.104:6792/0 conn(0x7f48bc1361a0 legacy=0x7f48bc136630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.298 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 10 monclient: _finish_hunting 0 2024-08-19T12:47:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 1 monclient: found mon.noname-c 2024-08-19T12:47:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:47:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48bc09feb0 con 0x7f48bc0a19f0 2024-08-19T12:47:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 10 monclient: _finish_auth 0 2024-08-19T12:47:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 10 monclient: _check_auth_tickets 2024-08-19T12:47:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c4328640 5 monclient: authenticate success, global_id 26914 2024-08-19T12:47:03.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c4328640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48b8001a50 con 0x7f48bc0a19f0 2024-08-19T12:47:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:47:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 10 monclient: dump: 2024-08-19T12:47:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.301 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.301 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c4328640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48b4001e60 con 0x7f48bc1361a0 2024-08-19T12:47:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:47:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48b8001d20 con 0x7f48bc0a19f0 2024-08-19T12:47:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 1 -- 172.21.15.26:0/4209217833 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48b8003950 con 0x7f48bc0a19f0 2024-08-19T12:47:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c4328640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c4328640 4 set_mon_vals no callback set 2024-08-19T12:47:03.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.295+0000 7f48c10c2640 10 monclient: dump: 2024-08-19T12:47:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.303 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.304 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.296+0000 7f48c4328640 10 monclient: shutdown 2024-08-19T12:47:03.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.296+0000 7f48c4328640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:03.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.296+0000 7f48c4328640 1 -- 172.21.15.26:0/4209217833 >> v1:172.21.15.26:6791/0 conn(0x7f48bc0a19f0 legacy=0x7f48bc0a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.296+0000 7f48c4328640 1 -- 172.21.15.26:0/4209217833 shutdown_connections 2024-08-19T12:47:03.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.296+0000 7f48c4328640 1 -- 172.21.15.26:0/4209217833 >> 172.21.15.26:0/4209217833 conn(0x7f48bc0a44d0 msgr2=0x7f48bc0a48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:03.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.296+0000 7f48c4328640 1 -- 172.21.15.26:0/4209217833 shutdown_connections 2024-08-19T12:47:03.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.297+0000 7f48c4328640 1 -- 172.21.15.26:0/4209217833 wait complete. 2024-08-19T12:47:03.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.297+0000 7f48c4328640 10 monclient: build_initial_monmap 2024-08-19T12:47:03.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.297+0000 7f48c4328640 10 monclient: monmap: 2024-08-19T12:47:03.308 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:03.308 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:03.308 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:03.308 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:03.308 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:03.308 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.309 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:03.309 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:03.309 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:03.309 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:03.309 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:03.309 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:03.309 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:03.309 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:03.309 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:03.309 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.297+0000 7f48c4328640 1 librados: starting msgr at 2024-08-19T12:47:03.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.297+0000 7f48c4328640 1 librados: starting objecter 2024-08-19T12:47:03.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 1 Processor -- start 2024-08-19T12:47:03.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 1 -- start start 2024-08-19T12:47:03.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 1 librados: setting wanted keys 2024-08-19T12:47:03.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 1 librados: calling monclient init 2024-08-19T12:47:03.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 10 monclient: init 2024-08-19T12:47:03.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:03.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 10 monclient: _add_conns ranks=[3,2,6,7,4,5,0,1,8] 2024-08-19T12:47:03.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 10 monclient(hunting): picked mon.noname-g con 0x7f48bc0a19f0 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:03.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 10 monclient(hunting): picked mon.noname-b con 0x7f48bc1361a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:03.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 10 monclient(hunting): picked mon.noname-d con 0x7f48bc1392c0 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:03.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48bc1208a0 con 0x7f48bc1361a0 2024-08-19T12:47:03.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48bc122360 con 0x7f48bc0a19f0 2024-08-19T12:47:03.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48bc122640 con 0x7f48bc1392c0 2024-08-19T12:47:03.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:03.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c4328640 10 monclient(hunting): authenticate will time out at 1217.385610s 2024-08-19T12:47:03.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c18c3640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f48bc1361a0 0x7f48bc135240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39082/0 (socket says 172.21.15.26:39082) 2024-08-19T12:47:03.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.298+0000 7f48c18c3640 1 -- 172.21.15.26:0/3968983558 learned_addr learned my addr 172.21.15.26:0/3968983558 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:03.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027048298}) 2024-08-19T12:47:03.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070786919}) 2024-08-19T12:47:03.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027048298}) 2024-08-19T12:47:03.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2991098715}) 2024-08-19T12:47:03.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027048298}) 2024-08-19T12:47:03.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3017367506}) 2024-08-19T12:47:03.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027048298}) 2024-08-19T12:47:03.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4019023299}) 2024-08-19T12:47:03.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027048298}) 2024-08-19T12:47:03.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=577862916}) 2024-08-19T12:47:03.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027048298}) 2024-08-19T12:47:03.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=413926977}) 2024-08-19T12:47:03.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027048298}) 2024-08-19T12:47:03.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1038642527}) 2024-08-19T12:47:03.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027048298}) 2024-08-19T12:47:03.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.300+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=810042417}) 2024-08-19T12:47:03.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.300+0000 7f48c18c3640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.26:6790/0 conn(0x7f48bc1361a0 legacy=0x7f48bc135240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.300+0000 7f48c28c5640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.26:6792/0 conn(0x7f48bc1392c0 legacy=0x7f48bc1d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.300+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:47:03.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.301+0000 7f48c20c4640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.104:6790/0 conn(0x7f48bc0a19f0 legacy=0x7f48bc1c9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.303+0000 7f48b2ffd640 10 client.?.objecter ms_handle_connect 0x7f48bc1361a0 2024-08-19T12:47:03.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.303+0000 7f48b2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.303+0000 7f48b2ffd640 10 client.?.objecter ms_handle_connect 0x7f48bc1392c0 2024-08-19T12:47:03.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.303+0000 7f48b2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.303+0000 7f48b2ffd640 10 client.?.objecter ms_handle_connect 0x7f48bc0a19f0 2024-08-19T12:47:03.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.303+0000 7f48b2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685668796}) 2024-08-19T12:47:03.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3980898650}) 2024-08-19T12:47:03.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685668796}) 2024-08-19T12:47:03.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=239616895}) 2024-08-19T12:47:03.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685668796}) 2024-08-19T12:47:03.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3610697409}) 2024-08-19T12:47:03.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685668796}) 2024-08-19T12:47:03.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3656277528}) 2024-08-19T12:47:03.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685668796}) 2024-08-19T12:47:03.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2370939659}) 2024-08-19T12:47:03.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.313+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685668796}) 2024-08-19T12:47:03.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.313+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3311892202}) 2024-08-19T12:47:03.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.313+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.313+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685668796}) 2024-08-19T12:47:03.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.313+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2346363180}) 2024-08-19T12:47:03.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.313+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.313+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685668796}) 2024-08-19T12:47:03.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.313+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=751351398}) 2024-08-19T12:47:03.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3210578934 0 0) 0x7f48bc122640 con 0x7f48bc1392c0 2024-08-19T12:47:03.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 10 monclient(hunting): my global_id is 164229 2024-08-19T12:47:03.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48a0004eb0 con 0x7f48bc1392c0 2024-08-19T12:47:03.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2522667816 0 0) 0x7f48a0004eb0 con 0x7f48bc1392c0 2024-08-19T12:47:03.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48bc122640 con 0x7f48bc1392c0 2024-08-19T12:47:03.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48b4004ea0 con 0x7f48bc1392c0 2024-08-19T12:47:03.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:03.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:47:03.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 10 monclient(hunting): dump: 2024-08-19T12:47:03.332 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.332 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.332 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.332 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.332 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.333 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.333 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.333 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.333 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.333 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.333 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.333 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.333 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.333 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.333 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.334 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:03.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1168717543 0 0) 0x7f48bc122640 con 0x7f48bc1392c0 2024-08-19T12:47:03.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.104:6790/0 conn(0x7f48bc0a19f0 legacy=0x7f48bc1c9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.26:6790/0 conn(0x7f48bc1361a0 legacy=0x7f48bc135240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:47:03.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 1 monclient: found mon.d 2024-08-19T12:47:03.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:03.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:03.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48bc135950 con 0x7f48bc1392c0 2024-08-19T12:47:03.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 10 monclient: _finish_auth 0 2024-08-19T12:47:03.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:47:03.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:03.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48c4328640 5 monclient: authenticate success, global_id 164229 2024-08-19T12:47:03.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48c4328640 10 monclient: _renew_subs 2024-08-19T12:47:03.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48c4328640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:03.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48c4328640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f48bc135be0 con 0x7f48bc1392c0 2024-08-19T12:47:03.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48c4328640 10 client.164229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:03.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48c4328640 10 monclient: _renew_subs 2024-08-19T12:47:03.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48c4328640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:03.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48c4328640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f48bc120050 con 0x7f48bc1392c0 2024-08-19T12:47:03.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.314+0000 7f48c4328640 1 librados: init done 2024-08-19T12:47:03.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.315+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48b40052f0 con 0x7f48bc1392c0 2024-08-19T12:47:03.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.315+0000 7f48b2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:03.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.315+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48b40057e0 con 0x7f48bc1392c0 2024-08-19T12:47:03.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.315+0000 7f48b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.315+0000 7f48c08c1640 4 set_mon_vals no callback set 2024-08-19T12:47:03.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.315+0000 7f48b2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:03.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.315+0000 7f48b2ffd640 10 monclient: dump: 2024-08-19T12:47:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.337 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.338 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.315+0000 7f48c4328640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.315+0000 7f48c4328640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f48a40065c0 con 0x7f48a4002860 2024-08-19T12:47:03.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.316+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f48b4011c40 con 0x7f48bc1392c0 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.316+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(354..354 src has 1..354) ==== 4961+0+0 (unknown 3168247542 0 0) 0x7f48b404e2a0 con 0x7f48bc1392c0 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.316+0000 7f48b2ffd640 10 client.164229.objecter ms_dispatch 0x7f48bc000b90 osd_map(354..354 src has 1..354) 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.316+0000 7f48b2ffd640 3 client.164229.objecter handle_osd_map got epochs [354,354] > 0 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.316+0000 7f48b2ffd640 3 client.164229.objecter handle_osd_map decoding full epoch 354 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.316+0000 7f48b2ffd640 20 client.164229.objecter dump_active .. 0 homeless 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.316+0000 7f48c18c3640 10 monclient: get_auth_request con 0x7f48a0040f00 auth_method 0 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48c20c4640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.104:6789/0 conn(0x7f48a4002860 legacy=0x7f48a4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 10 client.164229.objecter ms_handle_connect 0x7f48a4002860 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 10 client.164229.objecter resend_mon_ops 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3490526347 0 0) 0x7f48a40065c0 con 0x7f48a4002860 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 10 monclient(hunting): my global_id is 27379 2024-08-19T12:47:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48bc122640 con 0x7f48a4002860 2024-08-19T12:47:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1119985046 0 0) 0x7f48bc122640 con 0x7f48a4002860 2024-08-19T12:47:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48a40065c0 con 0x7f48a4002860 2024-08-19T12:47:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48b8003950 con 0x7f48a4002860 2024-08-19T12:47:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:47:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 10 monclient: dump: 2024-08-19T12:47:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.340 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.341 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.342 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.318+0000 7f48b2ffd640 10 client.164229.objecter ms_handle_connect 0x7f48a0040f00 2024-08-19T12:47:03.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.319+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1037905446 0 0) 0x7f48a40065c0 con 0x7f48a4002860 2024-08-19T12:47:03.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.319+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f48a400a160 con 0x7f48a4002860 2024-08-19T12:47:03.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.319+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f48b8001550 con 0x7f48a4002860 2024-08-19T12:47:03.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.319+0000 7f48b2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:03.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.319+0000 7f48b2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:03.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.319+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.104:6789/0 conn(0x7f48a4002860 legacy=0x7f48a4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.319+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.104:6789/0 conn(0x7f48a4002860 legacy=0x7f48a4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:03.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.325+0000 7f48c4328640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:03.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.325+0000 7f48c4328640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f48a4009e50 con 0x7f48a4009110 2024-08-19T12:47:03.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48c20c4640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.104:6789/0 conn(0x7f48a4009110 legacy=0x7f48a40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 10 client.164229.objecter ms_handle_connect 0x7f48a4009110 2024-08-19T12:47:03.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 10 client.164229.objecter resend_mon_ops 2024-08-19T12:47:03.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2898517744 0 0) 0x7f48a4009e50 con 0x7f48a4009110 2024-08-19T12:47:03.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 10 monclient(hunting): my global_id is 27388 2024-08-19T12:47:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48a40065c0 con 0x7f48a4009110 2024-08-19T12:47:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 464003741 0 0) 0x7f48a40065c0 con 0x7f48a4009110 2024-08-19T12:47:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48a4009e50 con 0x7f48a4009110 2024-08-19T12:47:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48b8003f50 con 0x7f48a4009110 2024-08-19T12:47:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:47:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 10 monclient: dump: 2024-08-19T12:47:03.344 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.345 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.346 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.346 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.346 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3864328955 0 0) 0x7f48a4009e50 con 0x7f48a4009110 2024-08-19T12:47:03.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f48a400c3f0 con 0x7f48a4009110 2024-08-19T12:47:03.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1670374605) 0x7f48b80041f0 con 0x7f48a4009110 2024-08-19T12:47:03.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:03.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:03.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.104:6789/0 conn(0x7f48a4009110 legacy=0x7f48a40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.328+0000 7f48b2ffd640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.104:6789/0 conn(0x7f48a4009110 legacy=0x7f48a40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:03.347 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:03.347 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:03.347 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:03.347 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:03.347 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:47:03.347 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 536332, 2024-08-19T12:47:03.347 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:03.348 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:03.348 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:03.348 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:03.348 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:03.348 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:03.348 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:03.348 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:03.348 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:47:03.348 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:47:03.348 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.348 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:47:03.348 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:03.349 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:03.349 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:03.349 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.349 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.349 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.349 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.349 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.349 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.349 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.350 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.350 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.350 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.350 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.350 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.350 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:03.350 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:03.350 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.350 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.350 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.350 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.351 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.351 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.351 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.351 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.351 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.351 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.351 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.351 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.351 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:03.351 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:03.351 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:03.352 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:03.352 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:03.352 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:03.352 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:03.352 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:03.352 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:03.352 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:03.352 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:03.352 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:03.352 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:03.352 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:03.352 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.353 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.354 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.354 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:03.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.354 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:03.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.354 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:03.354 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:03.354 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:03.355 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.355 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.355 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:03.355 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:03.355 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.355 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.355 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.355 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:03.356 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:03.356 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.356 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:03.356 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.356 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:03.356 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:03.357 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.358 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:03.358 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:03.358 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.358 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.358 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.358 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:03.358 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.359 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:03.362 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.362 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.362 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:03.362 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:03.362 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.362 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.362 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.362 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.363 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.364 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:47:03.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.368 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.368 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:03.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.368 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.368 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.368 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:47:03.368 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.368 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.368 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:03.368 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:03.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 10 librados: watch_flush enter 2024-08-19T12:47:03.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 10 librados: watch_flush exit 2024-08-19T12:47:03.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 20 client.164229.objecter shutdown clearing up homeless session... 2024-08-19T12:47:03.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 10 client.164229.objecter successfully canceled tick 2024-08-19T12:47:03.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.104:6800/899887506 conn(0x7f48a0040f00 legacy=0x7f48a00433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 10 monclient: shutdown 2024-08-19T12:47:03.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:03.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 1 -- 172.21.15.26:0/3968983558 >> v1:172.21.15.26:6792/0 conn(0x7f48bc1392c0 legacy=0x7f48bc1d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c28c5640 1 -- 172.21.15.26:0/3968983558 reap_dead start 2024-08-19T12:47:03.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c28c5640 1 -- 172.21.15.26:0/3968983558 reap_dead start 2024-08-19T12:47:03.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 1 -- 172.21.15.26:0/3968983558 shutdown_connections 2024-08-19T12:47:03.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 1 -- 172.21.15.26:0/3968983558 >> 172.21.15.26:0/3968983558 conn(0x7f48bc0a44d0 msgr2=0x7f48bc0a38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:03.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 1 -- 172.21.15.26:0/3968983558 shutdown_connections 2024-08-19T12:47:03.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 1 -- 172.21.15.26:0/3968983558 wait complete. 2024-08-19T12:47:03.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.329+0000 7f48c4328640 1 librados: shutdown 2024-08-19T12:47:03.372 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:47:03.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:47:03.372 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:47:03.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.469 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139536565}) 2024-08-19T12:47:03.469 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4070204477}) 2024-08-19T12:47:03.469 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.469 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139536565}) 2024-08-19T12:47:03.469 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4008492994}) 2024-08-19T12:47:03.469 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.469 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139536565}) 2024-08-19T12:47:03.469 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3908118870}) 2024-08-19T12:47:03.469 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.469 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139536565}) 2024-08-19T12:47:03.469 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1306951985}) 2024-08-19T12:47:03.469 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.470 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139536565}) 2024-08-19T12:47:03.470 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1345279376}) 2024-08-19T12:47:03.470 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.470 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139536565}) 2024-08-19T12:47:03.470 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1001025257}) 2024-08-19T12:47:03.470 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.470 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139536565}) 2024-08-19T12:47:03.470 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2861523509}) 2024-08-19T12:47:03.470 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.470 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139536565}) 2024-08-19T12:47:03.470 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.467+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=531899057}) 2024-08-19T12:47:03.472 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.472 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853325071}) 2024-08-19T12:47:03.472 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2933406223}) 2024-08-19T12:47:03.472 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.472 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853325071}) 2024-08-19T12:47:03.472 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3637184465}) 2024-08-19T12:47:03.472 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.472 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853325071}) 2024-08-19T12:47:03.472 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3718370095}) 2024-08-19T12:47:03.472 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.472 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853325071}) 2024-08-19T12:47:03.473 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1819002047}) 2024-08-19T12:47:03.473 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.473 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853325071}) 2024-08-19T12:47:03.473 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1508102623}) 2024-08-19T12:47:03.473 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.473 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853325071}) 2024-08-19T12:47:03.473 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4007574144}) 2024-08-19T12:47:03.473 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.474 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853325071}) 2024-08-19T12:47:03.474 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3222123121}) 2024-08-19T12:47:03.474 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.474 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853325071}) 2024-08-19T12:47:03.474 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.469+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=110062216}) 2024-08-19T12:47:03.484 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.478+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.484 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.478+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900990278}) 2024-08-19T12:47:03.484 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.478+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270130586}) 2024-08-19T12:47:03.484 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.478+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.484 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900990278}) 2024-08-19T12:47:03.484 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=671361604}) 2024-08-19T12:47:03.484 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.484 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900990278}) 2024-08-19T12:47:03.484 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=291052711}) 2024-08-19T12:47:03.484 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900990278}) 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1004358887}) 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900990278}) 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1579259028}) 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900990278}) 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1255248717}) 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900990278}) 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3534621524}) 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.485 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900990278}) 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.479+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3734246864}) 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660539961}) 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388249321}) 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660539961}) 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1750043532}) 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660539961}) 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2387786555}) 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660539961}) 2024-08-19T12:47:03.486 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4073155097}) 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660539961}) 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=431997654}) 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660539961}) 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1449975605}) 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660539961}) 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1428467661}) 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660539961}) 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.480+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=489213365}) 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.487 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545275071}) 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=280009815}) 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545275071}) 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=107190754}) 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545275071}) 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2506678672}) 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545275071}) 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2785267345}) 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545275071}) 2024-08-19T12:47:03.488 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1249517837}) 2024-08-19T12:47:03.489 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.489 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545275071}) 2024-08-19T12:47:03.489 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3447576460}) 2024-08-19T12:47:03.489 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.489 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545275071}) 2024-08-19T12:47:03.489 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4128768136}) 2024-08-19T12:47:03.489 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:03.489 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545275071}) 2024-08-19T12:47:03.489 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:03.481+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=803422867}) 2024-08-19T12:47:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.513+0000 7fb411aa6640 10 monclient: get_monmap_and_config 2024-08-19T12:47:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.513+0000 7fb411aa6640 10 monclient: build_initial_monmap 2024-08-19T12:47:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.513+0000 7fb411aa6640 10 monclient: monmap: 2024-08-19T12:47:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:03.514152+0000 2024-08-19T12:47:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:03.514152+0000 2024-08-19T12:47:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.514 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:03.515 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 1 Processor -- start 2024-08-19T12:47:03.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 1 -- start start 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 10 monclient: init 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 10 monclient: _add_conns ranks=[0,3,2,5,6,8,1,4,7] 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 10 monclient(hunting): picked mon.noname-a con 0x7fb40c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 10 monclient(hunting): picked mon.noname-g con 0x7fb40c0a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 10 monclient(hunting): picked mon.noname-b con 0x7fb40c0a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb40c136790 con 0x7fb40c06e7b0 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb40c09ddf0 con 0x7fb40c0a5d50 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb40c09e0d0 con 0x7fb40c0a5920 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb411aa6640 10 monclient(hunting): authenticate will time out at 1217.601588s 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb40b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb40c06e7b0 0x7fb40c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48064/0 (socket says 172.21.15.26:48064) 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb40b7fe640 1 -- 172.21.15.26:0/3328759972 learned_addr learned my addr 172.21.15.26:0/3328759972 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.514+0000 7fb40bfff640 1 --1- 172.21.15.26:0/3328759972 >> v1:172.21.15.26:6790/0 conn(0x7fb40c0a5d50 0x7fb40c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39098/0 (socket says 172.21.15.26:39098) 2024-08-19T12:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.516+0000 7fb40b7fe640 1 -- 172.21.15.26:0/3328759972 >> v1:172.21.15.26:6789/0 conn(0x7fb40c06e7b0 legacy=0x7fb40c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.516+0000 7fb40bfff640 1 -- 172.21.15.26:0/3328759972 >> v1:172.21.15.26:6790/0 conn(0x7fb40c0a5d50 legacy=0x7fb40c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40affd640 1 -- 172.21.15.26:0/3328759972 >> v1:172.21.15.104:6790/0 conn(0x7fb40c0a5920 legacy=0x7fb40c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 1 -- 172.21.15.26:0/3328759972 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 597467729 0 0) 0x7fb40c136790 con 0x7fb40c06e7b0 2024-08-19T12:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient(hunting): my global_id is 144900 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 1 -- 172.21.15.26:0/3328759972 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3f4003520 con 0x7fb40c06e7b0 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 1 -- 172.21.15.26:0/3328759972 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 230683305 0 0) 0x7fb4040017d0 con 0x7fb40c0a5d50 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient(hunting): my global_id is 26588 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 1 -- 172.21.15.26:0/3328759972 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb40c136790 con 0x7fb40c0a5d50 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 1 -- 172.21.15.26:0/3328759972 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1487516541 0 0) 0x7fb3f4003520 con 0x7fb40c06e7b0 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 1 -- 172.21.15.26:0/3328759972 >> v1:172.21.15.104:6790/0 conn(0x7fb40c0a5920 legacy=0x7fb40c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 1 -- 172.21.15.26:0/3328759972 >> v1:172.21.15.26:6790/0 conn(0x7fb40c0a5d50 legacy=0x7fb40c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 1 monclient: found mon.noname-a 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 1 -- 172.21.15.26:0/3328759972 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40c09e3b0 con 0x7fb40c06e7b0 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 1 -- 172.21.15.26:0/3328759972 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb3fc001ae0 con 0x7fb40c06e7b0 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb411aa6640 5 monclient: authenticate success, global_id 144900 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb411aa6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:47:03.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient: dump: 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.524 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb411aa6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 1 -- 172.21.15.26:0/3328759972 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb3fc0048a0 con 0x7fb40c06e7b0 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 1 -- 172.21.15.26:0/3328759972 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb3fc004e10 con 0x7fb40c06e7b0 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb40a7fc640 10 monclient: dump: 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.525 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb411aa6640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:03.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb411aa6640 4 set_mon_vals no callback set 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb411aa6640 10 monclient: shutdown 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb411aa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.517+0000 7fb411aa6640 1 -- 172.21.15.26:0/3328759972 >> v1:172.21.15.26:6789/0 conn(0x7fb40c06e7b0 legacy=0x7fb40c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.518+0000 7fb411aa6640 1 -- 172.21.15.26:0/3328759972 shutdown_connections 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.518+0000 7fb411aa6640 1 -- 172.21.15.26:0/3328759972 >> 172.21.15.26:0/3328759972 conn(0x7fb40c0a8740 msgr2=0x7fb40c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.518+0000 7fb411aa6640 1 -- 172.21.15.26:0/3328759972 shutdown_connections 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.518+0000 7fb411aa6640 1 -- 172.21.15.26:0/3328759972 wait complete. 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.518+0000 7fb411aa6640 10 monclient: build_initial_monmap 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.518+0000 7fb411aa6640 10 monclient: monmap: 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:03.527 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.518+0000 7fb411aa6640 1 librados: starting msgr at 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.518+0000 7fb411aa6640 1 librados: starting objecter 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.518+0000 7fb411aa6640 1 Processor -- start 2024-08-19T12:47:03.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 1 -- start start 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 1 librados: setting wanted keys 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 1 librados: calling monclient init 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 10 monclient: init 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 10 monclient: _add_conns ranks=[4,1,8,0,3,2,6,7,5] 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 10 monclient(hunting): picked mon.noname-c con 0x7fb40c06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 10 monclient(hunting): picked mon.noname-f con 0x7fb40c0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 10 monclient(hunting): picked mon.noname-e con 0x7fb40c0a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb40c13b150 con 0x7fb40c0a5920 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb40c138120 con 0x7fb40c06e7b0 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb40c13ae70 con 0x7fb40c0a5d50 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb411aa6640 10 monclient(hunting): authenticate will time out at 1217.606662s 2024-08-19T12:47:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb40b7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fb40c06e7b0 0x7fb40c137300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:34714/0 (socket says 172.21.15.26:34714) 2024-08-19T12:47:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.519+0000 7fb40b7fe640 1 -- 172.21.15.26:0/2427877358 learned_addr learned my addr 172.21.15.26:0/2427877358 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:03.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.521+0000 7fb40b7fe640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.26:6791/0 conn(0x7fb40c06e7b0 legacy=0x7fb40c137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.521+0000 7fb3f3fff640 10 client.?.objecter ms_handle_connect 0x7fb40c06e7b0 2024-08-19T12:47:03.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.521+0000 7fb40bfff640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.26:6793/0 conn(0x7fb40c0a5d50 legacy=0x7fb40c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.521+0000 7fb3f3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.521+0000 7fb3f3fff640 10 client.?.objecter ms_handle_connect 0x7fb40c0a5d50 2024-08-19T12:47:03.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.521+0000 7fb3f3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb40b7fe640 1 --1- 172.21.15.26:0/2427877358 >> v1:172.21.15.26:6791/0 conn(0x7fb40c06e7b0 0x7fb40c137300 :-1 s=OPENED pgs=3382 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fb40c138120 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2623351059 0 0) 0x7fb40c13ae70 con 0x7fb40c0a5d50 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient(hunting): my global_id is 26441 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3f8004eb0 con 0x7fb40c0a5d50 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 client.?.objecter ms_handle_connect 0x7fb40c0a5920 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb40affd640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.104:6789/0 conn(0x7fb40c0a5920 legacy=0x7fb40c137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3751846859 0 0) 0x7fb40c13b150 con 0x7fb40c0a5920 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient(hunting): my global_id is 27397 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb40c13ae70 con 0x7fb40c0a5920 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3526526663 0 0) 0x7fb3f8004eb0 con 0x7fb40c0a5d50 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb40c13b150 con 0x7fb40c0a5d50 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb404001850 con 0x7fb40c0a5d50 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient(hunting): dump: 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.537 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:03.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3617564470 0 0) 0x7fb40c13ae70 con 0x7fb40c0a5920 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3f8004eb0 con 0x7fb40c0a5920 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4283176657 0 0) 0x7fb40c13b150 con 0x7fb40c0a5d50 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.26:6791/0 conn(0x7fb40c06e7b0 legacy=0x7fb40c137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.104:6789/0 conn(0x7fb40c0a5920 legacy=0x7fb40c137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient: _finish_hunting 0 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 monclient: found mon.e 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40c139bc0 con 0x7fb40c0a5d50 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient: _finish_auth 0 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient: _check_auth_tickets 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:03.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb411aa6640 5 monclient: authenticate success, global_id 26441 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567792245 0 0) 0x7fb40c138120 con 0x7fb40c06e7b0 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb411aa6640 10 monclient: _renew_subs 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb411aa6640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb411aa6640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb40c1428c0 con 0x7fb40c0a5d50 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb411aa6640 10 client.26441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb411aa6640 10 monclient: _renew_subs 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb411aa6640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb411aa6640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fb40c1d7b90 con 0x7fb40c0a5d50 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb411aa6640 1 librados: init done 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb404004e10 con 0x7fb40c0a5d50 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb404005340 con 0x7fb40c0a5d50 2024-08-19T12:47:03.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb409ffb640 4 set_mon_vals no callback set 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.522+0000 7fb3f3fff640 10 monclient: dump: 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.541 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.523+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fb4040054e0 con 0x7fb40c0a5d50 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.523+0000 7fb411aa6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.523+0000 7fb411aa6640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb3ec0065c0 con 0x7fb3ec002860 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.523+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(354..354 src has 1..354) ==== 4961+0+0 (unknown 3168247542 0 0) 0x7fb40404e130 con 0x7fb40c0a5d50 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.523+0000 7fb3f3fff640 10 client.26441.objecter ms_dispatch 0x7fb40c000b90 osd_map(354..354 src has 1..354) 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.523+0000 7fb3f3fff640 3 client.26441.objecter handle_osd_map got epochs [354,354] > 0 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.523+0000 7fb3f3fff640 3 client.26441.objecter handle_osd_map decoding full epoch 354 2024-08-19T12:47:03.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.523+0000 7fb3f3fff640 20 client.26441.objecter dump_active .. 0 homeless 2024-08-19T12:47:03.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.523+0000 7fb40affd640 10 monclient: get_auth_request con 0x7fb3f8041240 auth_method 0 2024-08-19T12:47:03.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.525+0000 7fb40b7fe640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.104:6790/0 conn(0x7fb3ec002860 legacy=0x7fb3ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.525+0000 7fb3f3fff640 10 client.26441.objecter ms_handle_connect 0x7fb3ec002860 2024-08-19T12:47:03.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.525+0000 7fb3f3fff640 10 client.26441.objecter resend_mon_ops 2024-08-19T12:47:03.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3859366733 0 0) 0x7fb3ec0065c0 con 0x7fb3ec002860 2024-08-19T12:47:03.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 10 monclient(hunting): my global_id is 154407 2024-08-19T12:47:03.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb40c138120 con 0x7fb3ec002860 2024-08-19T12:47:03.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 10 client.26441.objecter ms_handle_connect 0x7fb3f8041240 2024-08-19T12:47:03.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2367415662 0 0) 0x7fb40c138120 con 0x7fb3ec002860 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3ec0065c0 con 0x7fb3ec002860 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb3fc004b60 con 0x7fb3ec002860 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 10 monclient: dump: 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.544 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 94725511 0 0) 0x7fb3ec0065c0 con 0x7fb3ec002860 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3ec00a160 con 0x7fb3ec002860 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb3fc001320 con 0x7fb3ec002860 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:03.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.104:6790/0 conn(0x7fb3ec002860 legacy=0x7fb3ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.526+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.104:6790/0 conn(0x7fb3ec002860 legacy=0x7fb3ec004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:03.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.532+0000 7fb411aa6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:03.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.532+0000 7fb411aa6640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb3ec009e70 con 0x7fb3ec009130 2024-08-19T12:47:03.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb40b7fe640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.104:6790/0 conn(0x7fb3ec009130 legacy=0x7fb3ec0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 10 client.26441.objecter ms_handle_connect 0x7fb3ec009130 2024-08-19T12:47:03.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 10 client.26441.objecter resend_mon_ops 2024-08-19T12:47:03.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 646519702 0 0) 0x7fb3ec009e70 con 0x7fb3ec009130 2024-08-19T12:47:03.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 10 monclient(hunting): my global_id is 154416 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3ec0065c0 con 0x7fb3ec009130 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 428653193 0 0) 0x7fb3ec0065c0 con 0x7fb3ec009130 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3ec009e70 con 0x7fb3ec009130 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb3fc005490 con 0x7fb3ec009130 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 10 monclient: dump: 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.548 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4158344963 0 0) 0x7fb3ec009e70 con 0x7fb3ec009130 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.535+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb3ec00c410 con 0x7fb3ec009130 2024-08-19T12:47:03.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1339459600) 0x7fb3fc007d80 con 0x7fb3ec009130 2024-08-19T12:47:03.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb3f3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:03.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb3f3fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:03.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.104:6790/0 conn(0x7fb3ec009130 legacy=0x7fb3ec0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb3f3fff640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.104:6790/0 conn(0x7fb3ec009130 legacy=0x7fb3ec0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:03.550 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:03.550 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:03.550 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:03.550 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:03.550 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 135264, 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:47:03.551 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:03.552 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:03.552 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:03.552 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.552 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.552 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.552 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.552 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.552 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.552 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.552 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.552 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.552 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.553 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.554 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.554 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.554 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.554 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:03.554 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:03.554 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:03.554 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:03.554 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:03.554 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:03.554 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:03.554 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.555 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.556 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:03.557 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.558 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:03.561 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:03.562 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.563 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:03.565 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:03.566 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.569 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:03.570 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.570 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.570 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:03.570 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:03.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 10 librados: watch_flush enter 2024-08-19T12:47:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 10 librados: watch_flush exit 2024-08-19T12:47:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 20 client.26441.objecter shutdown clearing up homeless session... 2024-08-19T12:47:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 10 client.26441.objecter successfully canceled tick 2024-08-19T12:47:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.104:6800/899887506 conn(0x7fb3f8041240 legacy=0x7fb3f8043700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 10 monclient: shutdown 2024-08-19T12:47:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 1 -- 172.21.15.26:0/2427877358 >> v1:172.21.15.26:6793/0 conn(0x7fb40c0a5d50 legacy=0x7fb40c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb40bfff640 1 -- 172.21.15.26:0/2427877358 reap_dead start 2024-08-19T12:47:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb40bfff640 1 -- 172.21.15.26:0/2427877358 reap_dead start 2024-08-19T12:47:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 1 -- 172.21.15.26:0/2427877358 shutdown_connections 2024-08-19T12:47:03.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 1 -- 172.21.15.26:0/2427877358 >> 172.21.15.26:0/2427877358 conn(0x7fb40c0a8740 msgr2=0x7fb40c0aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 1 -- 172.21.15.26:0/2427877358 shutdown_connections 2024-08-19T12:47:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 1 -- 172.21.15.26:0/2427877358 wait complete. 2024-08-19T12:47:03.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.536+0000 7fb411aa6640 1 librados: shutdown 2024-08-19T12:47:03.573 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:47:03.573 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:47:03.573 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:47:03.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.711+0000 7f9383586640 10 monclient: get_monmap_and_config 2024-08-19T12:47:03.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.711+0000 7f9383586640 10 monclient: build_initial_monmap 2024-08-19T12:47:03.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.711+0000 7f9383586640 10 monclient: monmap: 2024-08-19T12:47:03.712 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:03.712 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.713 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:03.712439+0000 2024-08-19T12:47:03.713 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:03.712439+0000 2024-08-19T12:47:03.713 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:03.713 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.713 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:03.713 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:03.713 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:03.713 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:03.713 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:03.713 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:03.714 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:03.714 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:03.714 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:03.714 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 1 Processor -- start 2024-08-19T12:47:03.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 1 -- start start 2024-08-19T12:47:03.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 10 monclient: init 2024-08-19T12:47:03.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:03.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 10 monclient: _add_conns ranks=[6,1,8,4,3,0,2,7,5] 2024-08-19T12:47:03.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 10 monclient(hunting): picked mon.noname-d con 0x7f937c06e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:03.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 10 monclient(hunting): picked mon.noname-f con 0x7f937c0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:03.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 10 monclient(hunting): picked mon.noname-e con 0x7f937c0a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:03.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f937c136790 con 0x7f937c0a5920 2024-08-19T12:47:03.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f937c09ddf0 con 0x7f937c06e7b0 2024-08-19T12:47:03.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f937c09e0d0 con 0x7f937c0a5d50 2024-08-19T12:47:03.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:03.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9383586640 10 monclient(hunting): authenticate will time out at 1217.799851s 2024-08-19T12:47:03.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9381322640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f937c06e7b0 0x7f937c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:43170/0 (socket says 172.21.15.26:43170) 2024-08-19T12:47:03.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.712+0000 7f9381322640 1 -- 172.21.15.26:0/2365789628 learned_addr learned my addr 172.21.15.26:0/2365789628 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:03.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9381b23640 1 -- 172.21.15.26:0/2365789628 >> v1:172.21.15.26:6793/0 conn(0x7f937c0a5d50 legacy=0x7f937c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9381322640 1 -- 172.21.15.26:0/2365789628 >> v1:172.21.15.26:6792/0 conn(0x7f937c06e7b0 legacy=0x7f937c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9380b21640 1 -- 172.21.15.26:0/2365789628 >> v1:172.21.15.104:6789/0 conn(0x7f937c0a5920 legacy=0x7f937c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1570983666 0 0) 0x7f937c09e0d0 con 0x7f937c0a5d50 2024-08-19T12:47:03.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient(hunting): my global_id is 26450 2024-08-19T12:47:03.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9368003520 con 0x7f937c0a5d50 2024-08-19T12:47:03.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 235868009 0 0) 0x7f9378001450 con 0x7f937c06e7b0 2024-08-19T12:47:03.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient(hunting): my global_id is 164238 2024-08-19T12:47:03.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f937c09e0d0 con 0x7f937c06e7b0 2024-08-19T12:47:03.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1757489703 0 0) 0x7f937c136790 con 0x7f937c0a5920 2024-08-19T12:47:03.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient(hunting): my global_id is 27406 2024-08-19T12:47:03.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9378001450 con 0x7f937c0a5920 2024-08-19T12:47:03.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9381b23640 1 --1- 172.21.15.26:0/2365789628 >> v1:172.21.15.26:6793/0 conn(0x7f937c0a5d50 0x7f937c141a20 :-1 s=OPENED pgs=3409 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f9374001eb0 mon_map magic: 0 2024-08-19T12:47:03.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 50671573 0 0) 0x7f937c09e0d0 con 0x7f937c06e7b0 2024-08-19T12:47:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 >> v1:172.21.15.26:6793/0 conn(0x7f937c0a5d50 legacy=0x7f937c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 >> v1:172.21.15.104:6789/0 conn(0x7f937c0a5920 legacy=0x7f937c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient: _finish_hunting 0 2024-08-19T12:47:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 monclient: found mon.noname-d 2024-08-19T12:47:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:47:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f937c09e3b0 con 0x7f937c06e7b0 2024-08-19T12:47:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient: _finish_auth 0 2024-08-19T12:47:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient: _check_auth_tickets 2024-08-19T12:47:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9378001c90 con 0x7f937c06e7b0 2024-08-19T12:47:03.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9383586640 5 monclient: authenticate success, global_id 164238 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient: dump: 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.725 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.726 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.726 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.726 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.726 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.726 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9374001eb0 con 0x7f937c0a5d50 2024-08-19T12:47:03.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9383586640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:03.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:47:03.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9378004920 con 0x7f937c06e7b0 2024-08-19T12:47:03.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:03.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 1 -- 172.21.15.26:0/2365789628 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9378004e90 con 0x7f937c06e7b0 2024-08-19T12:47:03.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9383586640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:03.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9383586640 4 set_mon_vals no callback set 2024-08-19T12:47:03.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.715+0000 7f9373fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.716+0000 7f9373fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:03.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.716+0000 7f9373fff640 10 monclient: dump: 2024-08-19T12:47:03.727 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.727 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.727 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.727 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.727 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.727 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.727 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.716+0000 7f9383586640 10 monclient: shutdown 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.716+0000 7f9383586640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.716+0000 7f9383586640 1 -- 172.21.15.26:0/2365789628 >> v1:172.21.15.26:6792/0 conn(0x7f937c06e7b0 legacy=0x7f937c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.716+0000 7f9383586640 1 -- 172.21.15.26:0/2365789628 shutdown_connections 2024-08-19T12:47:03.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.716+0000 7f9383586640 1 -- 172.21.15.26:0/2365789628 >> 172.21.15.26:0/2365789628 conn(0x7f937c0a8740 msgr2=0x7f937c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:03.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.716+0000 7f9383586640 1 -- 172.21.15.26:0/2365789628 shutdown_connections 2024-08-19T12:47:03.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.716+0000 7f9383586640 1 -- 172.21.15.26:0/2365789628 wait complete. 2024-08-19T12:47:03.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.717+0000 7f9383586640 10 monclient: build_initial_monmap 2024-08-19T12:47:03.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.717+0000 7f9383586640 10 monclient: monmap: 2024-08-19T12:47:03.729 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:03.729 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:03.729 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:03.729 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:03.729 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.717+0000 7f9383586640 1 librados: starting msgr at 2024-08-19T12:47:03.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.717+0000 7f9383586640 1 librados: starting objecter 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.717+0000 7f9383586640 1 Processor -- start 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.717+0000 7f9383586640 1 -- start start 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.717+0000 7f9383586640 1 librados: setting wanted keys 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.717+0000 7f9383586640 1 librados: calling monclient init 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.717+0000 7f9383586640 10 monclient: init 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.717+0000 7f9383586640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.717+0000 7f9383586640 10 monclient: _add_conns ranks=[0,4,5,1,6,2,3,7,8] 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.717+0000 7f9383586640 10 monclient(hunting): picked mon.noname-a con 0x7f937c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.718+0000 7f9383586640 10 monclient(hunting): picked mon.noname-c con 0x7f937c0a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.718+0000 7f9383586640 10 monclient(hunting): picked mon.noname-h con 0x7f937c0a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.718+0000 7f9383586640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f937c141d00 con 0x7f937c06e7b0 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.718+0000 7f9383586640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f937c1346f0 con 0x7f937c0a5920 2024-08-19T12:47:03.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.718+0000 7f9383586640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f937c142ea0 con 0x7f937c0a5d50 2024-08-19T12:47:03.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.718+0000 7f9383586640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:03.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.718+0000 7f9383586640 10 monclient(hunting): authenticate will time out at 1217.805088s 2024-08-19T12:47:03.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.718+0000 7f9381322640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f937c06e7b0 0x7f937c1374c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48076/0 (socket says 172.21.15.26:48076) 2024-08-19T12:47:03.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.718+0000 7f9381322640 1 -- 172.21.15.26:0/1715730471 learned_addr learned my addr 172.21.15.26:0/1715730471 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:03.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.720+0000 7f9380b21640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.26:6791/0 conn(0x7f937c0a5920 legacy=0x7f937c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.720+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:47:03.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.720+0000 7f9381322640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.26:6789/0 conn(0x7f937c06e7b0 legacy=0x7f937c1374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.720+0000 7f9381b23640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.104:6791/0 conn(0x7f937c0a5d50 legacy=0x7f937c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 client.?.objecter ms_handle_connect 0x7f937c0a5920 2024-08-19T12:47:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 client.?.objecter ms_handle_connect 0x7f937c06e7b0 2024-08-19T12:47:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 client.?.objecter ms_handle_connect 0x7f937c0a5d50 2024-08-19T12:47:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2401958530 0 0) 0x7f936c001350 con 0x7f937c0a5920 2024-08-19T12:47:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): my global_id is 26932 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9360004eb0 con 0x7f937c0a5920 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 799008369 0 0) 0x7f93740019b0 con 0x7f937c0a5d50 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): my global_id is 26591 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f936c001350 con 0x7f937c0a5d50 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 70865122 0 0) 0x7f937c141d00 con 0x7f937c06e7b0 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): my global_id is 144909 2024-08-19T12:47:03.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93740019b0 con 0x7f937c06e7b0 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3478655209 0 0) 0x7f93740019b0 con 0x7f937c06e7b0 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f937c141d00 con 0x7f937c06e7b0 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9378001d10 con 0x7f937c06e7b0 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): dump: 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.741 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 238130322 0 0) 0x7f9360004eb0 con 0x7f937c0a5920 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93740019b0 con 0x7f937c0a5920 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f936c003610 con 0x7f937c0a5920 2024-08-19T12:47:03.742 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient(hunting): dump: 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.743 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.744 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.746 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.746 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1222218526 0 0) 0x7f937c141d00 con 0x7f937c06e7b0 2024-08-19T12:47:03.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.104:6791/0 conn(0x7f937c0a5d50 legacy=0x7f937c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.26:6791/0 conn(0x7f937c0a5920 legacy=0x7f937c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:47:03.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 monclient: found mon.a 2024-08-19T12:47:03.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f937c1382e0 con 0x7f937c06e7b0 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient: _finish_auth 0 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9371ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9383586640 5 monclient: authenticate success, global_id 144909 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9383586640 10 monclient: _renew_subs 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9383586640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.722+0000 7f9383586640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f937c13ad90 con 0x7f937c06e7b0 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9378004ed0 con 0x7f937c06e7b0 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9371ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9378005510 con 0x7f937c06e7b0 2024-08-19T12:47:03.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9371ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9371ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9371ffb640 10 monclient: dump: 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.748 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9383586640 10 client.144909.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9383586640 10 monclient: _renew_subs 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f93737fe640 4 set_mon_vals no callback set 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9383586640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9383586640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f937c1427e0 con 0x7f937c06e7b0 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9383586640 1 librados: init done 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9383586640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9383586640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f937c0008d0 con 0x7f937c139a50 2024-08-19T12:47:03.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.723+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f9378011970 con 0x7f937c06e7b0 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.724+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(354..354 src has 1..354) ==== 4961+0+0 (unknown 3168247542 0 0) 0x7f937804dee0 con 0x7f937c06e7b0 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.724+0000 7f9371ffb640 10 client.144909.objecter ms_dispatch 0x7f937c000b90 osd_map(354..354 src has 1..354) 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.724+0000 7f9371ffb640 3 client.144909.objecter handle_osd_map got epochs [354,354] > 0 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.724+0000 7f9371ffb640 3 client.144909.objecter handle_osd_map decoding full epoch 354 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.724+0000 7f9371ffb640 20 client.144909.objecter dump_active .. 0 homeless 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.724+0000 7f9381b23640 10 monclient: get_auth_request con 0x7f93600417a0 auth_method 0 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9380b21640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.104:6791/0 conn(0x7f937c139a50 legacy=0x7f937c0a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 10 client.144909.objecter ms_handle_connect 0x7f937c139a50 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 10 client.144909.objecter resend_mon_ops 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 10 client.144909.objecter ms_handle_connect 0x7f93600417a0 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2722522430 0 0) 0x7f937c0008d0 con 0x7f937c139a50 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 10 monclient(hunting): my global_id is 26600 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f937c141d00 con 0x7f937c139a50 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3710405588 0 0) 0x7f937c141d00 con 0x7f937c139a50 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f937c0008d0 con 0x7f937c139a50 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f936c003e30 con 0x7f937c139a50 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.726+0000 7f9371ffb640 10 monclient: dump: 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.751 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.752 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.753 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.727+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3207002512 0 0) 0x7f937c0008d0 con 0x7f937c139a50 2024-08-19T12:47:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.727+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f937c143140 con 0x7f937c139a50 2024-08-19T12:47:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.727+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f936c004160 con 0x7f937c139a50 2024-08-19T12:47:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.727+0000 7f9371ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.727+0000 7f9371ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.727+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.104:6791/0 conn(0x7f937c139a50 legacy=0x7f937c0a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.727+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.104:6791/0 conn(0x7f937c139a50 legacy=0x7f937c0a08c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.733+0000 7f9383586640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.733+0000 7f9383586640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f937c02e5b0 con 0x7f937c02e1d0 2024-08-19T12:47:03.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.735+0000 7f9371ffb640 10 client.144909.objecter ms_handle_connect 0x7f937c02e1d0 2024-08-19T12:47:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.735+0000 7f9371ffb640 10 client.144909.objecter resend_mon_ops 2024-08-19T12:47:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.735+0000 7f9380b21640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.104:6791/0 conn(0x7f937c02e1d0 legacy=0x7f937c0a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2453524349 0 0) 0x7f937c02e5b0 con 0x7f937c02e1d0 2024-08-19T12:47:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 10 monclient(hunting): my global_id is 26609 2024-08-19T12:47:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f937c0008d0 con 0x7f937c02e1d0 2024-08-19T12:47:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3517866769 0 0) 0x7f937c0008d0 con 0x7f937c02e1d0 2024-08-19T12:47:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f937c02e5b0 con 0x7f937c02e1d0 2024-08-19T12:47:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f936c004460 con 0x7f937c02e1d0 2024-08-19T12:47:03.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 10 monclient: dump: 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.755 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.756 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1473495966 0 0) 0x7f937c02e5b0 con 0x7f937c02e1d0 2024-08-19T12:47:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.736+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f937c0a16e0 con 0x7f937c02e1d0 2024-08-19T12:47:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1923966162) 0x7f936c005e60 con 0x7f937c02e1d0 2024-08-19T12:47:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9371ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9371ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.104:6791/0 conn(0x7f937c02e1d0 legacy=0x7f937c0a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9371ffb640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.104:6791/0 conn(0x7f937c02e1d0 legacy=0x7f937c0a0fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:03.757 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:47:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 536759, 2024-08-19T12:47:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:03.757 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:03.758 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.759 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.760 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:03.761 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:03.762 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:03.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.764 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:03.765 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:03.767 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.768 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:03.770 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:03.771 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:03.772 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:03.774 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.775 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.776 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.777 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.778 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:03.779 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.780 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.781 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:47:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.782 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:47:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.783 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:03.783 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 10 librados: watch_flush enter 2024-08-19T12:47:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 10 librados: watch_flush exit 2024-08-19T12:47:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 20 client.144909.objecter shutdown clearing up homeless session... 2024-08-19T12:47:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 10 client.144909.objecter successfully canceled tick 2024-08-19T12:47:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.104:6800/899887506 conn(0x7f93600417a0 legacy=0x7f9360043c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 10 monclient: shutdown 2024-08-19T12:47:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 1 -- 172.21.15.26:0/1715730471 >> v1:172.21.15.26:6789/0 conn(0x7f937c06e7b0 legacy=0x7f937c1374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9381b23640 1 -- 172.21.15.26:0/1715730471 reap_dead start 2024-08-19T12:47:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9381b23640 1 -- 172.21.15.26:0/1715730471 reap_dead start 2024-08-19T12:47:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 1 -- 172.21.15.26:0/1715730471 shutdown_connections 2024-08-19T12:47:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 1 -- 172.21.15.26:0/1715730471 >> 172.21.15.26:0/1715730471 conn(0x7f937c0a8740 msgr2=0x7f937c0aaf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 1 -- 172.21.15.26:0/1715730471 shutdown_connections 2024-08-19T12:47:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 1 -- 172.21.15.26:0/1715730471 wait complete. 2024-08-19T12:47:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.737+0000 7f9383586640 1 librados: shutdown 2024-08-19T12:47:03.787 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:47:03.787 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:47:03.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:47:03.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.887+0000 7fd94d698640 10 monclient: get_monmap_and_config 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.887+0000 7fd94d698640 10 monclient: build_initial_monmap 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.887+0000 7fd94d698640 10 monclient: monmap: 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:03.888141+0000 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:03.888141+0000 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:03.888 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:03.889 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:03.889 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:03.889 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:03.889 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:03.889 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.887+0000 7fd94d698640 1 Processor -- start 2024-08-19T12:47:03.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.887+0000 7fd94d698640 1 -- start start 2024-08-19T12:47:03.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.887+0000 7fd94d698640 10 monclient: init 2024-08-19T12:47:03.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.888+0000 7fd94d698640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:03.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.888+0000 7fd94d698640 10 monclient: _add_conns ranks=[4,7,5,8,6,3,1,2,0] 2024-08-19T12:47:03.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.888+0000 7fd94d698640 10 monclient(hunting): picked mon.noname-c con 0x7fd94806e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:03.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.888+0000 7fd94d698640 10 monclient(hunting): picked mon.noname-i con 0x7fd9480a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:47:03.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.888+0000 7fd94d698640 10 monclient(hunting): picked mon.noname-h con 0x7fd9480a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:03.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.888+0000 7fd94d698640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd948136790 con 0x7fd94806e7b0 2024-08-19T12:47:03.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.888+0000 7fd94d698640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd94809ddf0 con 0x7fd9480a5d50 2024-08-19T12:47:03.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.888+0000 7fd94d698640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd94809e0d0 con 0x7fd9480a5920 2024-08-19T12:47:03.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.888+0000 7fd94d698640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:03.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.888+0000 7fd94d698640 10 monclient(hunting): authenticate will time out at 1217.975495s 2024-08-19T12:47:03.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.888+0000 7fd946ffd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fd94806e7b0 0x7fd948136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:34738/0 (socket says 172.21.15.26:34738) 2024-08-19T12:47:03.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.888+0000 7fd946ffd640 1 -- 172.21.15.26:0/307847901 learned_addr learned my addr 172.21.15.26:0/307847901 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.890+0000 7fd946ffd640 1 -- 172.21.15.26:0/307847901 >> v1:172.21.15.26:6791/0 conn(0x7fd94806e7b0 legacy=0x7fd948136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 1 -- 172.21.15.26:0/307847901 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4268988371 0 0) 0x7fd948136790 con 0x7fd94806e7b0 2024-08-19T12:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient(hunting): my global_id is 26941 2024-08-19T12:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 1 -- 172.21.15.26:0/307847901 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd934003520 con 0x7fd94806e7b0 2024-08-19T12:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd9477fe640 1 -- 172.21.15.26:0/307847901 >> v1:172.21.15.104:6791/0 conn(0x7fd9480a5d50 legacy=0x7fd948141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd9467fc640 1 -- 172.21.15.26:0/307847901 >> v1:172.21.15.104:6792/0 conn(0x7fd9480a5920 legacy=0x7fd94813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 1 -- 172.21.15.26:0/307847901 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 229148484 0 0) 0x7fd934003520 con 0x7fd94806e7b0 2024-08-19T12:47:03.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 1 -- 172.21.15.26:0/307847901 >> v1:172.21.15.104:6792/0 conn(0x7fd9480a5920 legacy=0x7fd94813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 1 -- 172.21.15.26:0/307847901 >> v1:172.21.15.104:6791/0 conn(0x7fd9480a5d50 legacy=0x7fd948141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 1 monclient: found mon.noname-c 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 1 -- 172.21.15.26:0/307847901 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94809e3b0 con 0x7fd94806e7b0 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient: _finish_auth 0 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 1 -- 172.21.15.26:0/307847901 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd938001e60 con 0x7fd94806e7b0 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient: dump: 2024-08-19T12:47:03.896 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.897 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd94d698640 5 monclient: authenticate success, global_id 26941 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd94d698640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 1 -- 172.21.15.26:0/307847901 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd938004c20 con 0x7fd94806e7b0 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd94d698640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd94d698640 4 set_mon_vals no callback set 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 1 -- 172.21.15.26:0/307847901 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd938005190 con 0x7fd94806e7b0 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd945ffb640 10 monclient: dump: 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.898 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.899 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd94d698640 10 monclient: shutdown 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd94d698640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd94d698640 1 -- 172.21.15.26:0/307847901 >> v1:172.21.15.26:6791/0 conn(0x7fd94806e7b0 legacy=0x7fd948136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd94d698640 1 -- 172.21.15.26:0/307847901 shutdown_connections 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.891+0000 7fd94d698640 1 -- 172.21.15.26:0/307847901 >> 172.21.15.26:0/307847901 conn(0x7fd9480a8740 msgr2=0x7fd9480a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.892+0000 7fd94d698640 1 -- 172.21.15.26:0/307847901 shutdown_connections 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.892+0000 7fd94d698640 1 -- 172.21.15.26:0/307847901 wait complete. 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.892+0000 7fd94d698640 10 monclient: build_initial_monmap 2024-08-19T12:47:03.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.892+0000 7fd94d698640 10 monclient: monmap: 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.892+0000 7fd94d698640 1 librados: starting msgr at 2024-08-19T12:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.892+0000 7fd94d698640 1 librados: starting objecter 2024-08-19T12:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 1 Processor -- start 2024-08-19T12:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 1 -- start start 2024-08-19T12:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 1 librados: setting wanted keys 2024-08-19T12:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 1 librados: calling monclient init 2024-08-19T12:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 10 monclient: init 2024-08-19T12:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 10 monclient: _add_conns ranks=[0,3,8,7,4,6,5,2,1] 2024-08-19T12:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 10 monclient(hunting): picked mon.noname-a con 0x7fd94806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:03.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 10 monclient(hunting): picked mon.noname-g con 0x7fd9480a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 10 monclient(hunting): picked mon.noname-e con 0x7fd9480a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9481349e0 con 0x7fd94806e7b0 2024-08-19T12:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd948134600 con 0x7fd9480a5920 2024-08-19T12:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9481391d0 con 0x7fd9480a5d50 2024-08-19T12:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd94d698640 10 monclient(hunting): authenticate will time out at 1217.980628s 2024-08-19T12:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd946ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd94806e7b0 0x7fd948132a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:48090/0 (socket says 172.21.15.26:48090) 2024-08-19T12:47:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.893+0000 7fd946ffd640 1 -- 172.21.15.26:0/4103753315 learned_addr learned my addr 172.21.15.26:0/4103753315 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.895+0000 7fd9477fe640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.26:6793/0 conn(0x7fd9480a5d50 legacy=0x7fd9481d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.895+0000 7fd93f7fe640 10 client.?.objecter ms_handle_connect 0x7fd9480a5d50 2024-08-19T12:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.895+0000 7fd93f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.895+0000 7fd93f7fe640 10 client.?.objecter ms_handle_connect 0x7fd94806e7b0 2024-08-19T12:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.895+0000 7fd93f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.895+0000 7fd946ffd640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.26:6789/0 conn(0x7fd94806e7b0 legacy=0x7fd948132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1248054428 0 0) 0x7fd9481391d0 con 0x7fd9480a5d50 2024-08-19T12:47:03.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): my global_id is 26459 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd928004eb0 con 0x7fd9480a5d50 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1617111151 0 0) 0x7fd94809e3b0 con 0x7fd94806e7b0 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): my global_id is 144918 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9481391d0 con 0x7fd94806e7b0 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 client.?.objecter ms_handle_connect 0x7fd9480a5920 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd9467fc640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.104:6790/0 conn(0x7fd9480a5920 legacy=0x7fd9481d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:03.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd946ffd640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.26:6789/0 conn(0x7fd94806e7b0 legacy=0x7fd948132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd946ffd640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.26:6789/0 conn(0x7fd94806e7b0 legacy=0x7fd948132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd946ffd640 1 --1- 172.21.15.26:0/4103753315 >> v1:172.21.15.26:6789/0 conn(0x7fd94806e7b0 0x7fd948132a50 :-1 s=OPENED pgs=3492 cs=1 l=1).handle_message read tag failed 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd946ffd640 1 --1- 172.21.15.26:0/4103753315 >> v1:172.21.15.26:6789/0 conn(0x7fd94806e7b0 0x7fd948132a50 :-1 s=OPENED pgs=3492 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6789/0 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1159749472 0 0) 0x7fd928004eb0 con 0x7fd9480a5d50 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd94809e3b0 con 0x7fd9480a5d50 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd940001cf0 con 0x7fd9480a5d50 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): dump: 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.910 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:03.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4112416180 0 0) 0x7fd948134600 con 0x7fd9480a5920 2024-08-19T12:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient(hunting): my global_id is 154425 2024-08-19T12:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd928004eb0 con 0x7fd9480a5920 2024-08-19T12:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 77348867 0 0) 0x7fd94809e3b0 con 0x7fd9480a5d50 2024-08-19T12:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.104:6790/0 conn(0x7fd9480a5920 legacy=0x7fd9481d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.26:6789/0 conn(0x7fd94806e7b0 legacy=0x7fd948132a50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:47:03.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 monclient: found mon.e 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94813b530 con 0x7fd9480a5d50 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd94d698640 5 monclient: authenticate success, global_id 26459 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd94d698640 10 monclient: _renew_subs 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd94d698640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd94d698640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd94813a960 con 0x7fd9480a5d50 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd94d698640 10 client.26459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd94d698640 10 monclient: _renew_subs 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd94d698640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:47:03.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd94d698640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fd948133160 con 0x7fd9480a5d50 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd94d698640 1 librados: init done 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9400039e0 con 0x7fd9480a5d50 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd940003f50 con 0x7fd9480a5d50 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd9457fa640 4 set_mon_vals no callback set 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.896+0000 7fd93f7fe640 10 monclient: dump: 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.914 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.897+0000 7fd94d698640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.897+0000 7fd94d698640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd92c0065c0 con 0x7fd92c002860 2024-08-19T12:47:03.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.897+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd9400040f0 con 0x7fd9480a5d50 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.897+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(354..354 src has 1..354) ==== 4961+0+0 (unknown 3168247542 0 0) 0x7fd94004dee0 con 0x7fd9480a5d50 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.897+0000 7fd93f7fe640 10 client.26459.objecter ms_dispatch 0x7fd948000b90 osd_map(354..354 src has 1..354) 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.897+0000 7fd93f7fe640 3 client.26459.objecter handle_osd_map got epochs [354,354] > 0 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.897+0000 7fd93f7fe640 3 client.26459.objecter handle_osd_map decoding full epoch 354 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.897+0000 7fd93f7fe640 20 client.26459.objecter dump_active .. 0 homeless 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.897+0000 7fd9467fc640 10 monclient: get_auth_request con 0x7fd928041110 auth_method 0 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.899+0000 7fd93f7fe640 10 client.26459.objecter ms_handle_connect 0x7fd92c002860 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.899+0000 7fd946ffd640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.104:6792/0 conn(0x7fd92c002860 legacy=0x7fd92c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.899+0000 7fd93f7fe640 10 client.26459.objecter resend_mon_ops 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440112114 0 0) 0x7fd92c0065c0 con 0x7fd92c002860 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 10 monclient(hunting): my global_id is 135538 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd94809e3b0 con 0x7fd92c002860 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 10 client.26459.objecter ms_handle_connect 0x7fd928041110 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 633132892 0 0) 0x7fd94809e3b0 con 0x7fd92c002860 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd92c0065c0 con 0x7fd92c002860 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd938004f60 con 0x7fd92c002860 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 10 monclient: dump: 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.917 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 889342267 0 0) 0x7fd92c0065c0 con 0x7fd92c002860 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd92c00a160 con 0x7fd92c002860 2024-08-19T12:47:03.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd938005180 con 0x7fd92c002860 2024-08-19T12:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.104:6792/0 conn(0x7fd92c002860 legacy=0x7fd92c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.900+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.104:6792/0 conn(0x7fd92c002860 legacy=0x7fd92c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.906+0000 7fd94d698640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:03.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.906+0000 7fd94d698640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd92c009e50 con 0x7fd92c009110 2024-08-19T12:47:03.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.909+0000 7fd946ffd640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.104:6792/0 conn(0x7fd92c009110 legacy=0x7fd92c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:03.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.909+0000 7fd93f7fe640 10 client.26459.objecter ms_handle_connect 0x7fd92c009110 2024-08-19T12:47:03.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.909+0000 7fd93f7fe640 10 client.26459.objecter resend_mon_ops 2024-08-19T12:47:03.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.909+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2475097919 0 0) 0x7fd92c009e50 con 0x7fd92c009110 2024-08-19T12:47:03.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.909+0000 7fd93f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:03.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.909+0000 7fd93f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:03.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.909+0000 7fd93f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.909+0000 7fd93f7fe640 10 monclient(hunting): my global_id is 135547 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.909+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd92c0065c0 con 0x7fd92c009110 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.910+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3911735434 0 0) 0x7fd92c0065c0 con 0x7fd92c009110 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.910+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd92c009e50 con 0x7fd92c009110 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.910+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd938001dc0 con 0x7fd92c009110 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.910+0000 7fd93f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.910+0000 7fd93f7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.910+0000 7fd93f7fe640 10 monclient: dump: 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:03.921 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.910+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1319833304 0 0) 0x7fd92c009e50 con 0x7fd92c009110 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.910+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd92c00c3f0 con 0x7fd92c009110 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 221414252) 0x7fd938005880 con 0x7fd92c009110 2024-08-19T12:47:03.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd93f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:03.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd93f7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:03.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.104:6792/0 conn(0x7fd92c009110 legacy=0x7fd92c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd93f7fe640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.104:6792/0 conn(0x7fd92c009110 legacy=0x7fd92c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 52, 2024-08-19T12:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 394267, 2024-08-19T12:47:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:03.943 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:03.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 10 librados: watch_flush enter 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 10 librados: watch_flush exit 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 20 client.26459.objecter shutdown clearing up homeless session... 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 10 client.26459.objecter successfully canceled tick 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.104:6800/899887506 conn(0x7fd928041110 legacy=0x7fd9280435d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 10 monclient: shutdown 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 1 -- 172.21.15.26:0/4103753315 >> v1:172.21.15.26:6793/0 conn(0x7fd9480a5d50 legacy=0x7fd9481d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd9477fe640 1 -- 172.21.15.26:0/4103753315 reap_dead start 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd9477fe640 1 -- 172.21.15.26:0/4103753315 reap_dead start 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 1 -- 172.21.15.26:0/4103753315 shutdown_connections 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 1 -- 172.21.15.26:0/4103753315 >> 172.21.15.26:0/4103753315 conn(0x7fd9480a8740 msgr2=0x7fd948135140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 1 -- 172.21.15.26:0/4103753315 shutdown_connections 2024-08-19T12:47:03.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 1 -- 172.21.15.26:0/4103753315 wait complete. 2024-08-19T12:47:03.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:03.911+0000 7fd94d698640 1 librados: shutdown 2024-08-19T12:47:03.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:47:03.946 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['e'] 2024-08-19T12:47:03.946 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-08-19T12:47:03.946 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-08-19T12:47:03.946 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-08-19T12:47:03.947 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-08-19T12:47:03.947 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:47:03.960 INFO:tasks.ceph.mon.e:Stopped 2024-08-19T12:47:03.960 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-08-19T12:47:03.961 INFO:tasks.ceph.mon.c:Sent signal 19 2024-08-19T12:47:03.961 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:47:04.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:04.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979367857}) 2024-08-19T12:47:04.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2440172244}) 2024-08-19T12:47:04.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:04.218 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979367857}) 2024-08-19T12:47:04.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3009465745}) 2024-08-19T12:47:04.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:04.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979367857}) 2024-08-19T12:47:04.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3261582847}) 2024-08-19T12:47:04.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:04.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979367857}) 2024-08-19T12:47:04.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3517408623}) 2024-08-19T12:47:04.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:04.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979367857}) 2024-08-19T12:47:04.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1078743638}) 2024-08-19T12:47:04.219 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:04.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979367857}) 2024-08-19T12:47:04.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3160532885}) 2024-08-19T12:47:04.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:04.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979367857}) 2024-08-19T12:47:04.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2859012364}) 2024-08-19T12:47:04.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:47:04.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979367857}) 2024-08-19T12:47:04.220 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:47:04.216+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1737922309}) 2024-08-19T12:47:18.961 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-08-19T12:47:18.962 INFO:tasks.ceph.mon.c:Sent signal 18 2024-08-19T12:47:18.962 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-08-19T12:47:18.963 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:47:19.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.073+0000 7fa8e63de640 10 monclient: get_monmap_and_config 2024-08-19T12:47:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.073+0000 7fa8e63de640 10 monclient: build_initial_monmap 2024-08-19T12:47:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.073+0000 7fa8e63de640 10 monclient: monmap: 2024-08-19T12:47:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:19.074918+0000 2024-08-19T12:47:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:19.074918+0000 2024-08-19T12:47:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:19.075 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:19.076 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:19.076 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:19.076 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:19.076 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:19.076 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:19.076 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:19.076 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:19.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.073+0000 7fa8e63de640 1 Processor -- start 2024-08-19T12:47:19.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.073+0000 7fa8e63de640 1 -- start start 2024-08-19T12:47:19.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.073+0000 7fa8e63de640 10 monclient: init 2024-08-19T12:47:19.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8e63de640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:19.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8e63de640 10 monclient: _add_conns ranks=[6,0,8,4,2,3,1,5,7] 2024-08-19T12:47:19.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8e63de640 10 monclient(hunting): picked mon.noname-d con 0x7fa8e0073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:19.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8e63de640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e00a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:19.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8e63de640 10 monclient(hunting): picked mon.noname-e con 0x7fa8e013f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:19.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8e63de640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e009dae0 con 0x7fa8e00a57a0 2024-08-19T12:47:19.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8e63de640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e0136910 con 0x7fa8e0073470 2024-08-19T12:47:19.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8e63de640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e009dd50 con 0x7fa8e013f2e0 2024-08-19T12:47:19.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8e63de640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:19.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8e63de640 10 monclient(hunting): authenticate will time out at 1233.162531s 2024-08-19T12:47:19.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8e497b640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7fa8e013f2e0 legacy=0x7fa8e01416d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:47:19.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8dffff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fa8e0073470 0x7fa8e0136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39044/0 (socket says 172.21.15.26:39044) 2024-08-19T12:47:19.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.074+0000 7fa8dffff640 1 -- 172.21.15.26:0/505469868 learned_addr learned my addr 172.21.15.26:0/505469868 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:19.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.076+0000 7fa8dffff640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.26:6792/0 conn(0x7fa8e0073470 legacy=0x7fa8e0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:19.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.076+0000 7fa8df7fe640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.26:6789/0 conn(0x7fa8e00a57a0 legacy=0x7fa8e00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:19.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.276+0000 7fa8e497b640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.26:6793/0 conn(0x7fa8e013f2e0 legacy=0x7fa8e01416d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:47:19.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:19.677+0000 7fa8e497b640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.26:6793/0 conn(0x7fa8e013f2e0 legacy=0x7fa8e01416d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:47:20.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:20.478+0000 7fa8e497b640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.26:6793/0 conn(0x7fa8e013f2e0 legacy=0x7fa8e01416d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:47:22.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 10 monclient(hunting): tick 2024-08-19T12:47:22.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:47:22.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 1 monclient(hunting): continuing hunt 2024-08-19T12:47:22.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:47:22.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.26:6793/0 conn(0x7fa8e013f2e0 legacy=0x7fa8e01416d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:47:22.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.26:6792/0 conn(0x7fa8e0073470 legacy=0x7fa8e0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:22.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.26:6789/0 conn(0x7fa8e00a57a0 legacy=0x7fa8e00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:22.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 10 monclient: _add_conns ranks=[4,5,2,7,1,3] 2024-08-19T12:47:22.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 10 monclient(hunting): picked mon.noname-c con 0x7fa8cc0038c0 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:22.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 10 monclient(hunting): picked mon.noname-h con 0x7fa8cc007550 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:22.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 10 monclient(hunting): picked mon.noname-b con 0x7fa8cc00b2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:22.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 1 -- 172.21.15.26:0/505469868 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e009e030 con 0x7fa8cc00b2e0 2024-08-19T12:47:22.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 1 -- 172.21.15.26:0/505469868 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e009dd50 con 0x7fa8cc0038c0 2024-08-19T12:47:22.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 1 -- 172.21.15.26:0/505469868 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8cc011110 con 0x7fa8cc007550 2024-08-19T12:47:22.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.074+0000 7fa8ddffb640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:22.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.077+0000 7fa8dffff640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.26:6791/0 conn(0x7fa8cc0038c0 legacy=0x7fa8cc005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:22.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.077+0000 7fa8e497b640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.26:6790/0 conn(0x7fa8cc00b2e0 legacy=0x7fa8cc00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:22.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:22.077+0000 7fa8df7fe640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.104:6791/0 conn(0x7fa8cc007550 legacy=0x7fa8cc009a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:23.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 1 -- 172.21.15.26:0/505469868 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2719419118 0 0) 0x7fa8cc011110 con 0x7fa8cc007550 2024-08-19T12:47:23.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:23.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:23.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:23.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient(hunting): my global_id is 26627 2024-08-19T12:47:23.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 1 -- 172.21.15.26:0/505469868 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8c8003520 con 0x7fa8cc007550 2024-08-19T12:47:23.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 1 -- 172.21.15.26:0/505469868 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3198033039 0 0) 0x7fa8e009e030 con 0x7fa8cc00b2e0 2024-08-19T12:47:23.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient(hunting): my global_id is 26597 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 1 -- 172.21.15.26:0/505469868 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8cc011110 con 0x7fa8cc00b2e0 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 1 -- 172.21.15.26:0/505469868 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1268947491 0 0) 0x7fa8c8003520 con 0x7fa8cc007550 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.26:6791/0 conn(0x7fa8cc0038c0 legacy=0x7fa8cc005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.26:6790/0 conn(0x7fa8cc00b2e0 legacy=0x7fa8cc00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient: _finish_hunting 0 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 1 monclient: found mon.noname-h 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8e497b640 1 -- 172.21.15.26:0/505469868 reap_dead start 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:47:23.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 1 -- 172.21.15.26:0/505469868 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8cc012410 con 0x7fa8cc007550 2024-08-19T12:47:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient: _finish_auth 0 2024-08-19T12:47:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient: _check_auth_tickets 2024-08-19T12:47:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 1 -- 172.21.15.26:0/505469868 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa8d4001ab0 con 0x7fa8cc007550 2024-08-19T12:47:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8e63de640 5 monclient: authenticate success, global_id 26627 2024-08-19T12:47:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8e63de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.981+0000 7fa8deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8deffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:47:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8deffd640 10 monclient: dump: 2024-08-19T12:47:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:23.987 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:23.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8e63de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8deffd640 1 -- 172.21.15.26:0/505469868 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8d4003610 con 0x7fa8cc007550 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8deffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8deffd640 1 -- 172.21.15.26:0/505469868 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa8d4003b80 con 0x7fa8cc007550 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8e63de640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8e63de640 4 set_mon_vals no callback set 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8deffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8deffd640 10 monclient: dump: 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:23.989 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:23.990 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:23.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8e63de640 10 monclient: shutdown 2024-08-19T12:47:23.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8e63de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:23.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.982+0000 7fa8e63de640 1 -- 172.21.15.26:0/505469868 >> v1:172.21.15.104:6791/0 conn(0x7fa8cc007550 legacy=0x7fa8cc009a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:23.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.983+0000 7fa8e63de640 1 -- 172.21.15.26:0/505469868 shutdown_connections 2024-08-19T12:47:23.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.983+0000 7fa8e63de640 1 -- 172.21.15.26:0/505469868 >> 172.21.15.26:0/505469868 conn(0x7fa8e00abff0 msgr2=0x7fa8e00ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:23.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.983+0000 7fa8e63de640 1 -- 172.21.15.26:0/505469868 shutdown_connections 2024-08-19T12:47:23.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.983+0000 7fa8e63de640 1 -- 172.21.15.26:0/505469868 wait complete. 2024-08-19T12:47:23.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.983+0000 7fa8e63de640 10 monclient: build_initial_monmap 2024-08-19T12:47:23.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.983+0000 7fa8e63de640 10 monclient: monmap: 2024-08-19T12:47:23.993 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:23.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:23.993 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:23.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:23.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:23.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:23.993 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:23.993 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:23.993 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.984+0000 7fa8e63de640 1 librados: starting msgr at 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.984+0000 7fa8e63de640 1 librados: starting objecter 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.984+0000 7fa8e63de640 1 Processor -- start 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 1 -- start start 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 1 librados: setting wanted keys 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 1 librados: calling monclient init 2024-08-19T12:47:23.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 10 monclient: init 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 10 monclient: _add_conns ranks=[2,6,7,4,8,1,5,0,3] 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 10 monclient(hunting): picked mon.noname-b con 0x7fa8cc007550 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 10 monclient(hunting): picked mon.noname-d con 0x7fa8e012e430 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 10 monclient(hunting): picked mon.noname-i con 0x7fa8e012e830 addr v1:172.21.15.104:6792/0 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e012f470 con 0x7fa8cc007550 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e0130ed0 con 0x7fa8e012e430 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e01311b0 con 0x7fa8e012e830 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8e63de640 10 monclient(hunting): authenticate will time out at 1238.073568s 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8df7fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fa8e012e430 0x7fa8e01ceb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39046/0 (socket says 172.21.15.26:39046) 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8dffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa8cc007550 0x7fa8e012dd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:32964/0 (socket says 172.21.15.26:32964) 2024-08-19T12:47:23.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.985+0000 7fa8df7fe640 1 -- 172.21.15.26:0/1914581339 learned_addr learned my addr 172.21.15.26:0/1914581339 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:23.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.987+0000 7fa8df7fe640 1 -- 172.21.15.26:0/1914581339 >> v1:172.21.15.26:6792/0 conn(0x7fa8e012e430 legacy=0x7fa8e01ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:23.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.987+0000 7fa8dffff640 1 -- 172.21.15.26:0/1914581339 >> v1:172.21.15.26:6790/0 conn(0x7fa8cc007550 legacy=0x7fa8e012dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:23.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.987+0000 7fa8dcff9640 10 client.?.objecter ms_handle_connect 0x7fa8e012e430 2024-08-19T12:47:23.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.987+0000 7fa8dcff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:23.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.987+0000 7fa8dcff9640 10 client.?.objecter ms_handle_connect 0x7fa8cc007550 2024-08-19T12:47:23.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.987+0000 7fa8dcff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:23.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.987+0000 7fa8dcff9640 10 client.?.objecter ms_handle_connect 0x7fa8e012e830 2024-08-19T12:47:23.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.987+0000 7fa8e497b640 1 -- 172.21.15.26:0/1914581339 >> v1:172.21.15.104:6792/0 conn(0x7fa8e012e830 legacy=0x7fa8e01d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:23.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:23.987+0000 7fa8dcff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:24.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2769463582 0 0) 0x7fa8e012f470 con 0x7fa8cc007550 2024-08-19T12:47:24.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): my global_id is 26606 2024-08-19T12:47:24.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8cc00f7d0 con 0x7fa8cc007550 2024-08-19T12:47:24.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3921525006 0 0) 0x7fa8e0130ed0 con 0x7fa8e012e430 2024-08-19T12:47:24.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): my global_id is 164247 2024-08-19T12:47:24.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8e012f470 con 0x7fa8e012e430 2024-08-19T12:47:24.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2082611921 0 0) 0x7fa8e012f470 con 0x7fa8e012e430 2024-08-19T12:47:24.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8e0130ed0 con 0x7fa8e012e430 2024-08-19T12:47:24.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa8d40037c0 con 0x7fa8e012e430 2024-08-19T12:47:24.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:24.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:47:24.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): dump: 2024-08-19T12:47:24.244 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.244 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.244 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.244 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.244 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.244 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.244 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.244 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.244 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.245 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.245 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.245 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.245 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.245 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.245 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.245 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:24.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 681489325 0 0) 0x7fa8cc00f7d0 con 0x7fa8cc007550 2024-08-19T12:47:24.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8e012f470 con 0x7fa8cc007550 2024-08-19T12:47:24.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa8d0001c40 con 0x7fa8cc007550 2024-08-19T12:47:24.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:24.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:24.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): dump: 2024-08-19T12:47:24.246 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.246 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.246 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.246 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.246 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.246 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.246 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.246 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.246 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.246 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.247 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.247 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.247 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.247 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.247 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.247 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2272031572 0 0) 0x7fa8e01311b0 con 0x7fa8e012e830 2024-08-19T12:47:24.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 10 monclient(hunting): my global_id is 135556 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8cc00f7d0 con 0x7fa8e012e830 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.237+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1003185787 0 0) 0x7fa8e0130ed0 con 0x7fa8e012e430 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 >> v1:172.21.15.104:6792/0 conn(0x7fa8e012e830 legacy=0x7fa8e01d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 >> v1:172.21.15.26:6790/0 conn(0x7fa8cc007550 legacy=0x7fa8e012dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 10 monclient: _finish_hunting 0 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 1 monclient: found mon.d 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e012ff70 con 0x7fa8e012e430 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 10 monclient: _finish_auth 0 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 10 monclient: _check_auth_tickets 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:24.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8e63de640 5 monclient: authenticate success, global_id 164247 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8e63de640 10 monclient: _renew_subs 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8e63de640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8e63de640 1 -- 172.21.15.26:0/1914581339 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e01335a0 con 0x7fa8e012e430 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8e63de640 10 client.164247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8e63de640 10 monclient: _renew_subs 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8e63de640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8e63de640 1 -- 172.21.15.26:0/1914581339 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa8e01d3ba0 con 0x7fa8e012e430 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8e63de640 1 librados: init done 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8d4003610 con 0x7fa8e012e430 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa8d4004110 con 0x7fa8e012e430 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:24.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8de7fc640 4 set_mon_vals no callback set 2024-08-19T12:47:24.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.238+0000 7fa8dcff9640 10 monclient: dump: 2024-08-19T12:47:24.250 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.250 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.250 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.250 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.250 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.250 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.251 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.251 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.251 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.251 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.251 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.251 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.251 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.251 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.251 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.239+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa8d40042b0 con 0x7fa8e012e430 2024-08-19T12:47:24.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.240+0000 7fa8e63de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:24.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.240+0000 7fa8e63de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:24.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.240+0000 7fa8e63de640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:24.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.240+0000 7fa8e63de640 1 -- 172.21.15.26:0/1914581339 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8c0005080 con 0x7fa8e012e430 2024-08-19T12:47:24.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.240+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(354..354 src has 1..354) ==== 4961+0+0 (unknown 3168247542 0 0) 0x7fa8d404dfb0 con 0x7fa8e012e430 2024-08-19T12:47:24.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.240+0000 7fa8dcff9640 10 client.164247.objecter ms_dispatch 0x7fa8e0000b90 osd_map(354..354 src has 1..354) 2024-08-19T12:47:24.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.240+0000 7fa8dcff9640 3 client.164247.objecter handle_osd_map got epochs [354,354] > 0 2024-08-19T12:47:24.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.240+0000 7fa8dcff9640 3 client.164247.objecter handle_osd_map decoding full epoch 354 2024-08-19T12:47:24.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.240+0000 7fa8dcff9640 20 client.164247.objecter dump_active .. 0 homeless 2024-08-19T12:47:24.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.240+0000 7fa8dffff640 10 monclient: get_auth_request con 0x7fa8cc042920 auth_method 0 2024-08-19T12:47:24.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.243+0000 7fa8dcff9640 10 client.164247.objecter ms_handle_connect 0x7fa8cc042920 2024-08-19T12:47:24.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.250+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fa8d4017590 con 0x7fa8e012e430 2024-08-19T12:47:24.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.250+0000 7fa8dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:24.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.250+0000 7fa8dcff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:24.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.509+0000 7fa8e63de640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:47:24.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.509+0000 7fa8e63de640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:47:24.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.509+0000 7fa8e63de640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:24.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.509+0000 7fa8e63de640 1 -- 172.21.15.26:0/1914581339 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa8c0005370 con 0x7fa8e012e430 2024-08-19T12:47:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.509+0000 7fa8dcff9640 1 -- 172.21.15.26:0/1914581339 <== mon.6 v1:172.21.15.26:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2723 (unknown 254746657 0 2838951375) 0x7fa8d4016e90 con 0x7fa8e012e430 2024-08-19T12:47:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.509+0000 7fa8dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:47:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.509+0000 7fa8dcff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:24.512 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":60,"quorum":[0,1,2,3,4,5,6,7],"quorum_names":["a","f","b","g","c","h","d","i"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:47:24.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.511+0000 7fa8e63de640 10 librados: watch_flush enter 2024-08-19T12:47:24.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.511+0000 7fa8e63de640 10 librados: watch_flush exit 2024-08-19T12:47:24.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.511+0000 7fa8e63de640 20 client.164247.objecter shutdown clearing up homeless session... 2024-08-19T12:47:24.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.511+0000 7fa8e63de640 10 client.164247.objecter successfully canceled tick 2024-08-19T12:47:24.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.511+0000 7fa8e63de640 1 -- 172.21.15.26:0/1914581339 >> v1:172.21.15.104:6800/899887506 conn(0x7fa8cc042920 legacy=0x7fa8cc044de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.511+0000 7fa8e63de640 10 monclient: shutdown 2024-08-19T12:47:24.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.511+0000 7fa8e63de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:24.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.511+0000 7fa8e63de640 1 -- 172.21.15.26:0/1914581339 >> v1:172.21.15.26:6792/0 conn(0x7fa8e012e430 legacy=0x7fa8e01ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.511+0000 7fa8e63de640 1 -- 172.21.15.26:0/1914581339 shutdown_connections 2024-08-19T12:47:24.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.511+0000 7fa8e63de640 1 -- 172.21.15.26:0/1914581339 >> 172.21.15.26:0/1914581339 conn(0x7fa8e00abff0 msgr2=0x7fa8e00a99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:24.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.512+0000 7fa8e63de640 1 -- 172.21.15.26:0/1914581339 shutdown_connections 2024-08-19T12:47:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.512+0000 7fa8e63de640 1 -- 172.21.15.26:0/1914581339 wait complete. 2024-08-19T12:47:24.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.512+0000 7fa8e63de640 1 librados: shutdown 2024-08-19T12:47:24.524 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-08-19T12:47:24.524 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-08-19T12:47:24.525 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:47:24.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.622+0000 7fc425521640 10 monclient: get_monmap_and_config 2024-08-19T12:47:24.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.622+0000 7fc425521640 10 monclient: build_initial_monmap 2024-08-19T12:47:24.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.622+0000 7fc425521640 10 monclient: monmap: 2024-08-19T12:47:24.625 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:24.625 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.625 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:24.624779+0000 2024-08-19T12:47:24.626 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:24.624779+0000 2024-08-19T12:47:24.626 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:24.626 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.626 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:24.626 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:24.626 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:24.626 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:24.626 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:24.626 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:24.626 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:24.626 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:24.627 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:24.627 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.623+0000 7fc425521640 1 Processor -- start 2024-08-19T12:47:24.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.623+0000 7fc425521640 1 -- start start 2024-08-19T12:47:24.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.623+0000 7fc425521640 10 monclient: init 2024-08-19T12:47:24.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.623+0000 7fc425521640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:24.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.623+0000 7fc425521640 10 monclient: _add_conns ranks=[0,8,3,7,4,2,1,6,5] 2024-08-19T12:47:24.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.624+0000 7fc425521640 10 monclient(hunting): picked mon.noname-a con 0x7fc42006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:24.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.624+0000 7fc425521640 10 monclient(hunting): picked mon.noname-e con 0x7fc4200a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:24.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.624+0000 7fc425521640 10 monclient(hunting): picked mon.noname-g con 0x7fc4200a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:24.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.624+0000 7fc425521640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc420136790 con 0x7fc42006e7b0 2024-08-19T12:47:24.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.624+0000 7fc425521640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc42009ddf0 con 0x7fc4200a5d50 2024-08-19T12:47:24.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.624+0000 7fc425521640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc42009e0d0 con 0x7fc4200a5920 2024-08-19T12:47:24.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.624+0000 7fc425521640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:24.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.624+0000 7fc425521640 10 monclient(hunting): authenticate will time out at 1238.712072s 2024-08-19T12:47:24.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.624+0000 7fc41e7fc640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7fc4200a5920 legacy=0x7fc42013e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:47:24.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.624+0000 7fc41effd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc42006e7b0 0x7fc420136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50258/0 (socket says 172.21.15.26:50258) 2024-08-19T12:47:24.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.624+0000 7fc41effd640 1 -- 172.21.15.26:0/3631207977 learned_addr learned my addr 172.21.15.26:0/3631207977 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41effd640 1 -- 172.21.15.26:0/3631207977 >> v1:172.21.15.26:6789/0 conn(0x7fc42006e7b0 legacy=0x7fc420136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 1 -- 172.21.15.26:0/3631207977 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2406133812 0 0) 0x7fc420136790 con 0x7fc42006e7b0 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient(hunting): my global_id is 144927 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 1 -- 172.21.15.26:0/3631207977 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc40c003520 con 0x7fc42006e7b0 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41f7fe640 1 -- 172.21.15.26:0/3631207977 >> v1:172.21.15.104:6790/0 conn(0x7fc4200a5d50 legacy=0x7fc420141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 1 -- 172.21.15.26:0/3631207977 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3868847177 0 0) 0x7fc40c003520 con 0x7fc42006e7b0 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 1 -- 172.21.15.26:0/3631207977 >> v1:172.21.15.26:6793/0 conn(0x7fc4200a5920 legacy=0x7fc42013e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 1 -- 172.21.15.26:0/3631207977 >> v1:172.21.15.104:6790/0 conn(0x7fc4200a5d50 legacy=0x7fc420141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 1 monclient: found mon.noname-a 2024-08-19T12:47:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:47:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 1 -- 172.21.15.26:0/3631207977 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc42009e3b0 con 0x7fc42006e7b0 2024-08-19T12:47:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient: _finish_auth 0 2024-08-19T12:47:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:47:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 1 -- 172.21.15.26:0/3631207977 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc410001e60 con 0x7fc42006e7b0 2024-08-19T12:47:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc425521640 5 monclient: authenticate success, global_id 144927 2024-08-19T12:47:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc425521640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:47:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient: dump: 2024-08-19T12:47:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc425521640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 1 -- 172.21.15.26:0/3631207977 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc410004c20 con 0x7fc42006e7b0 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 1 -- 172.21.15.26:0/3631207977 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc410005190 con 0x7fc42006e7b0 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc425521640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc425521640 4 set_mon_vals no callback set 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.626+0000 7fc41dffb640 10 monclient: dump: 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.627+0000 7fc425521640 10 monclient: shutdown 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.627+0000 7fc425521640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.627+0000 7fc425521640 1 -- 172.21.15.26:0/3631207977 >> v1:172.21.15.26:6789/0 conn(0x7fc42006e7b0 legacy=0x7fc420136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.627+0000 7fc425521640 1 -- 172.21.15.26:0/3631207977 shutdown_connections 2024-08-19T12:47:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.627+0000 7fc425521640 1 -- 172.21.15.26:0/3631207977 >> 172.21.15.26:0/3631207977 conn(0x7fc4200a8740 msgr2=0x7fc4200a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.627+0000 7fc425521640 1 -- 172.21.15.26:0/3631207977 shutdown_connections 2024-08-19T12:47:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.627+0000 7fc425521640 1 -- 172.21.15.26:0/3631207977 wait complete. 2024-08-19T12:47:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.627+0000 7fc425521640 10 monclient: build_initial_monmap 2024-08-19T12:47:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 10 monclient: monmap: 2024-08-19T12:47:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 1 librados: starting msgr at 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 1 librados: starting objecter 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 1 Processor -- start 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 1 -- start start 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 1 librados: setting wanted keys 2024-08-19T12:47:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 1 librados: calling monclient init 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 10 monclient: init 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 10 monclient: _add_conns ranks=[8,5,6,7,0,3,2,4,1] 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 10 monclient(hunting): picked mon.noname-e con 0x7fc42006e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 10 monclient(hunting): picked mon.noname-h con 0x7fc4200a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 10 monclient(hunting): picked mon.noname-d con 0x7fc4200a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc42012df10 con 0x7fc4200a5920 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc420124330 con 0x7fc4200a5d50 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc420126800 con 0x7fc42006e7b0 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc425521640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.628+0000 7fc41effd640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7fc42006e7b0 legacy=0x7fc420123510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.629+0000 7fc425521640 10 monclient(hunting): authenticate will time out at 1238.717006s 2024-08-19T12:47:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.629+0000 7fc41f7fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fc4200a5d50 0x7fc4201d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39054/0 (socket says 172.21.15.26:39054) 2024-08-19T12:47:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.629+0000 7fc41f7fe640 1 -- 172.21.15.26:0/277073816 learned_addr learned my addr 172.21.15.26:0/277073816 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc41f7fe640 1 -- 172.21.15.26:0/277073816 >> v1:172.21.15.26:6792/0 conn(0x7fc4200a5d50 legacy=0x7fc4201d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 client.?.objecter ms_handle_connect 0x7fc4200a5d50 2024-08-19T12:47:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 798071854 0 0) 0x7fc420124330 con 0x7fc4200a5d50 2024-08-19T12:47:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient(hunting): my global_id is 164256 2024-08-19T12:47:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc41e7fc640 1 -- 172.21.15.26:0/277073816 >> v1:172.21.15.104:6791/0 conn(0x7fc4200a5920 legacy=0x7fc420123c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc400004eb0 con 0x7fc4200a5d50 2024-08-19T12:47:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 client.?.objecter ms_handle_connect 0x7fc4200a5920 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 233315237 0 0) 0x7fc408003d30 con 0x7fc4200a5920 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient(hunting): my global_id is 26636 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc420124330 con 0x7fc4200a5920 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3587461404 0 0) 0x7fc400004eb0 con 0x7fc4200a5d50 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc408003d30 con 0x7fc4200a5d50 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc418003610 con 0x7fc4200a5d50 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:47:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient(hunting): dump: 2024-08-19T12:47:24.650 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.650 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.650 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.650 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.650 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.650 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.650 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.650 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.650 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.650 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.650 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.650 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.651 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.651 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.651 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.651 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:24.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2868515202 0 0) 0x7fc420124330 con 0x7fc4200a5920 2024-08-19T12:47:24.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc400004eb0 con 0x7fc4200a5920 2024-08-19T12:47:24.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3859179176 0 0) 0x7fc408003d30 con 0x7fc4200a5d50 2024-08-19T12:47:24.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 >> v1:172.21.15.26:6793/0 conn(0x7fc42006e7b0 legacy=0x7fc420123510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:47:24.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 >> v1:172.21.15.104:6791/0 conn(0x7fc4200a5920 legacy=0x7fc420123c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient: _finish_hunting 0 2024-08-19T12:47:24.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 monclient: found mon.d 2024-08-19T12:47:24.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:24.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:24.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc420133220 con 0x7fc4200a5d50 2024-08-19T12:47:24.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient: _finish_auth 0 2024-08-19T12:47:24.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 10 monclient: _check_auth_tickets 2024-08-19T12:47:24.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc4177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:24.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc425521640 5 monclient: authenticate success, global_id 164256 2024-08-19T12:47:24.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc425521640 10 monclient: _renew_subs 2024-08-19T12:47:24.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc425521640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:24.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc425521640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc420125b10 con 0x7fc4200a5d50 2024-08-19T12:47:24.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc425521640 10 client.164256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:24.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc425521640 10 monclient: _renew_subs 2024-08-19T12:47:24.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc425521640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:24.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc425521640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fc4201d79b0 con 0x7fc4200a5d50 2024-08-19T12:47:24.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.631+0000 7fc425521640 1 librados: init done 2024-08-19T12:47:24.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.632+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc418003b10 con 0x7fc4200a5d50 2024-08-19T12:47:24.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.632+0000 7fc4177fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:24.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.632+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc418004040 con 0x7fc4200a5d50 2024-08-19T12:47:24.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.632+0000 7fc4177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.632+0000 7fc41d7fa640 4 set_mon_vals no callback set 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.632+0000 7fc4177fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.632+0000 7fc4177fe640 10 monclient: dump: 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.654 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.632+0000 7fc425521640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.632+0000 7fc425521640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc4040065c0 con 0x7fc404002860 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.632+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fc4180041e0 con 0x7fc4200a5d50 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.633+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(355..355 src has 1..355) ==== 5333+0+0 (unknown 1574624872 0 0) 0x7fc41804e060 con 0x7fc4200a5d50 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.633+0000 7fc4177fe640 10 client.164256.objecter ms_dispatch 0x7fc420000b90 osd_map(355..355 src has 1..355) 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.633+0000 7fc4177fe640 3 client.164256.objecter handle_osd_map got epochs [355,355] > 0 2024-08-19T12:47:24.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.633+0000 7fc4177fe640 3 client.164256.objecter handle_osd_map decoding full epoch 355 2024-08-19T12:47:24.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.633+0000 7fc4177fe640 20 client.164256.objecter dump_active .. 0 homeless 2024-08-19T12:47:24.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.633+0000 7fc41e7fc640 10 monclient: get_auth_request con 0x7fc400041150 auth_method 0 2024-08-19T12:47:24.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 10 client.164256.objecter ms_handle_connect 0x7fc404002860 2024-08-19T12:47:24.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc41effd640 1 -- 172.21.15.26:0/277073816 >> v1:172.21.15.26:6789/0 conn(0x7fc404002860 legacy=0x7fc404004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 10 client.164256.objecter resend_mon_ops 2024-08-19T12:47:24.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1073487475 0 0) 0x7fc4040065c0 con 0x7fc404002860 2024-08-19T12:47:24.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 10 monclient(hunting): my global_id is 144936 2024-08-19T12:47:24.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc408003d30 con 0x7fc404002860 2024-08-19T12:47:24.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2105762811 0 0) 0x7fc408003d30 con 0x7fc404002860 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc4040065c0 con 0x7fc404002860 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc410004ea0 con 0x7fc404002860 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 10 monclient: dump: 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.657 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.658 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.658 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.658 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.658 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.658 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.658 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.658 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.658 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 563192806 0 0) 0x7fc4040065c0 con 0x7fc404002860 2024-08-19T12:47:24.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc40400a160 con 0x7fc404002860 2024-08-19T12:47:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.635+0000 7fc4177fe640 10 client.164256.objecter ms_handle_connect 0x7fc400041150 2024-08-19T12:47:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.636+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc4100016a0 con 0x7fc404002860 2024-08-19T12:47:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.636+0000 7fc4177fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.636+0000 7fc4177fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.636+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 >> v1:172.21.15.26:6789/0 conn(0x7fc404002860 legacy=0x7fc404004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.636+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 >> v1:172.21.15.26:6789/0 conn(0x7fc404002860 legacy=0x7fc404004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.642+0000 7fc425521640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.642+0000 7fc425521640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc404009e50 con 0x7fc404009110 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc41effd640 1 -- 172.21.15.26:0/277073816 >> v1:172.21.15.26:6789/0 conn(0x7fc404009110 legacy=0x7fc4040095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 10 client.164256.objecter ms_handle_connect 0x7fc404009110 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 10 client.164256.objecter resend_mon_ops 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1930659483 0 0) 0x7fc404009e50 con 0x7fc404009110 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 10 monclient(hunting): my global_id is 144945 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4040065c0 con 0x7fc404009110 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2266026831 0 0) 0x7fc4040065c0 con 0x7fc404009110 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc404009e50 con 0x7fc404009110 2024-08-19T12:47:24.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc4100054d0 con 0x7fc404009110 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 10 monclient: dump: 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.661 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.662 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 691248599 0 0) 0x7fc404009e50 con 0x7fc404009110 2024-08-19T12:47:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.644+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc40400c3f0 con 0x7fc404009110 2024-08-19T12:47:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7869 (unknown 0 0 2950895395) 0x7fc410005600 con 0x7fc404009110 2024-08-19T12:47:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc4177fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc4177fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 >> v1:172.21.15.26:6789/0 conn(0x7fc404009110 legacy=0x7fc4040095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc4177fe640 1 -- 172.21.15.26:0/277073816 >> v1:172.21.15.26:6789/0 conn(0x7fc404009110 legacy=0x7fc4040095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:24.663 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:47:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-08-19T12:47:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 296563, 2024-08-19T12:47:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:47:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:24.683 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 10 librados: watch_flush enter 2024-08-19T12:47:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 10 librados: watch_flush exit 2024-08-19T12:47:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 20 client.164256.objecter shutdown clearing up homeless session... 2024-08-19T12:47:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 10 client.164256.objecter successfully canceled tick 2024-08-19T12:47:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 1 -- 172.21.15.26:0/277073816 >> v1:172.21.15.104:6800/899887506 conn(0x7fc400041150 legacy=0x7fc400043610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 10 monclient: shutdown 2024-08-19T12:47:24.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 1 -- 172.21.15.26:0/277073816 >> v1:172.21.15.26:6792/0 conn(0x7fc4200a5d50 legacy=0x7fc4201d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc41f7fe640 1 -- 172.21.15.26:0/277073816 reap_dead start 2024-08-19T12:47:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc41f7fe640 1 -- 172.21.15.26:0/277073816 reap_dead start 2024-08-19T12:47:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 1 -- 172.21.15.26:0/277073816 shutdown_connections 2024-08-19T12:47:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 1 -- 172.21.15.26:0/277073816 >> 172.21.15.26:0/277073816 conn(0x7fc4200a8740 msgr2=0x7fc42006d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 1 -- 172.21.15.26:0/277073816 shutdown_connections 2024-08-19T12:47:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 1 -- 172.21.15.26:0/277073816 wait complete. 2024-08-19T12:47:24.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.645+0000 7fc425521640 1 librados: shutdown 2024-08-19T12:47:24.686 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:47:24.687 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-08-19T12:47:24.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:47:24.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.784+0000 7f48f6d9e640 10 monclient: get_monmap_and_config 2024-08-19T12:47:24.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.784+0000 7f48f6d9e640 10 monclient: build_initial_monmap 2024-08-19T12:47:24.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.784+0000 7f48f6d9e640 10 monclient: monmap: 2024-08-19T12:47:24.787 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:24.788 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.788 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:24.786737+0000 2024-08-19T12:47:24.788 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:24.786737+0000 2024-08-19T12:47:24.788 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:24.788 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.788 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:24.788 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:24.788 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:24.788 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:24.789 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:24.789 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:24.789 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:24.789 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:24.789 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:24.789 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.785+0000 7f48f6d9e640 1 Processor -- start 2024-08-19T12:47:24.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.785+0000 7f48f6d9e640 1 -- start start 2024-08-19T12:47:24.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.785+0000 7f48f6d9e640 10 monclient: init 2024-08-19T12:47:24.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f6d9e640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:24.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f6d9e640 10 monclient: _add_conns ranks=[8,1,5,6,3,4,7,2,0] 2024-08-19T12:47:24.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f6d9e640 10 monclient(hunting): picked mon.noname-e con 0x7f48f00a5660 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:24.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f6d9e640 10 monclient(hunting): picked mon.noname-f con 0x7f48f0136130 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:24.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f6d9e640 10 monclient(hunting): picked mon.noname-h con 0x7f48f013f7c0 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:24.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f6d9e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f009dc10 con 0x7f48f0136130 2024-08-19T12:47:24.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f6d9e640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f00a9830 con 0x7f48f013f7c0 2024-08-19T12:47:24.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f6d9e640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f009def0 con 0x7f48f00a5660 2024-08-19T12:47:24.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f6d9e640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:24.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f6d9e640 10 monclient(hunting): authenticate will time out at 1238.874199s 2024-08-19T12:47:24.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f5d9c640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f48f00a5660 legacy=0x7f48f00a5a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:47:24.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f559b640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f48f0136130 0x7f48f013e0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:44156/0 (socket says 172.21.15.26:44156) 2024-08-19T12:47:24.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.786+0000 7f48f559b640 1 -- 172.21.15.26:0/966090988 learned_addr learned my addr 172.21.15.26:0/966090988 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:24.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.788+0000 7f48f559b640 1 -- 172.21.15.26:0/966090988 >> v1:172.21.15.104:6789/0 conn(0x7f48f0136130 legacy=0x7f48f013e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.788+0000 7f48f659d640 1 -- 172.21.15.26:0/966090988 >> v1:172.21.15.104:6791/0 conn(0x7f48f013f7c0 legacy=0x7f48f0141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.788+0000 7f48f4d9a640 1 -- 172.21.15.26:0/966090988 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2818460853 0 0) 0x7f48e4001450 con 0x7f48f0136130 2024-08-19T12:47:24.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.788+0000 7f48f4d9a640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.788+0000 7f48f4d9a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.788+0000 7f48f4d9a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.788+0000 7f48f4d9a640 10 monclient(hunting): my global_id is 27415 2024-08-19T12:47:24.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.788+0000 7f48f4d9a640 1 -- 172.21.15.26:0/966090988 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48d8003520 con 0x7f48f0136130 2024-08-19T12:47:24.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.788+0000 7f48f4d9a640 1 -- 172.21.15.26:0/966090988 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3944054097 0 0) 0x7f48f00a9830 con 0x7f48f013f7c0 2024-08-19T12:47:24.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient(hunting): my global_id is 26645 2024-08-19T12:47:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 1 -- 172.21.15.26:0/966090988 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48e4001450 con 0x7f48f013f7c0 2024-08-19T12:47:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 1 -- 172.21.15.26:0/966090988 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 318372837 0 0) 0x7f48d8003520 con 0x7f48f0136130 2024-08-19T12:47:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 1 -- 172.21.15.26:0/966090988 >> v1:172.21.15.26:6793/0 conn(0x7f48f00a5660 legacy=0x7f48f00a5a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:47:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 1 -- 172.21.15.26:0/966090988 >> v1:172.21.15.104:6791/0 conn(0x7f48f013f7c0 legacy=0x7f48f0141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient: _finish_hunting 0 2024-08-19T12:47:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 1 monclient: found mon.noname-f 2024-08-19T12:47:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:24.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 1 -- 172.21.15.26:0/966090988 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f009e1d0 con 0x7f48f0136130 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient: _finish_auth 0 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient: _check_auth_tickets 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 1 -- 172.21.15.26:0/966090988 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48e4001ca0 con 0x7f48f0136130 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient: dump: 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.807 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.808 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.808 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.808 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.808 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.808 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.808 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.808 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.809 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.809 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.809 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f6d9e640 5 monclient: authenticate success, global_id 27415 2024-08-19T12:47:24.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f6d9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:24.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 1 -- 172.21.15.26:0/966090988 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48e4004950 con 0x7f48f0136130 2024-08-19T12:47:24.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:24.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f6d9e640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:24.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 1 -- 172.21.15.26:0/966090988 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48e4004ec0 con 0x7f48f0136130 2024-08-19T12:47:24.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f6d9e640 4 set_mon_vals no callback set 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f4d9a640 10 monclient: dump: 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f6d9e640 10 monclient: shutdown 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f6d9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f6d9e640 1 -- 172.21.15.26:0/966090988 >> v1:172.21.15.104:6789/0 conn(0x7f48f0136130 legacy=0x7f48f013e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f6d9e640 1 -- 172.21.15.26:0/966090988 shutdown_connections 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.789+0000 7f48f6d9e640 1 -- 172.21.15.26:0/966090988 >> 172.21.15.26:0/966090988 conn(0x7f48f00abe30 msgr2=0x7f48f00ac240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.790+0000 7f48f6d9e640 1 -- 172.21.15.26:0/966090988 shutdown_connections 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.790+0000 7f48f6d9e640 1 -- 172.21.15.26:0/966090988 wait complete. 2024-08-19T12:47:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.790+0000 7f48f6d9e640 10 monclient: build_initial_monmap 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.790+0000 7f48f6d9e640 10 monclient: monmap: 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:24.813 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.790+0000 7f48f6d9e640 1 librados: starting msgr at 2024-08-19T12:47:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.790+0000 7f48f6d9e640 1 librados: starting objecter 2024-08-19T12:47:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 1 Processor -- start 2024-08-19T12:47:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 1 -- start start 2024-08-19T12:47:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 1 librados: setting wanted keys 2024-08-19T12:47:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 1 librados: calling monclient init 2024-08-19T12:47:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 10 monclient: init 2024-08-19T12:47:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:24.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 10 monclient: _add_conns ranks=[2,1,6,7,5,4,8,3,0] 2024-08-19T12:47:24.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 10 monclient(hunting): picked mon.noname-b con 0x7f48f00a5660 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:24.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 10 monclient(hunting): picked mon.noname-f con 0x7f48f0136130 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:24.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 10 monclient(hunting): picked mon.noname-d con 0x7f48f013f7c0 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:24.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f013e920 con 0x7f48f0136130 2024-08-19T12:47:24.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f00aa3d0 con 0x7f48f00a5660 2024-08-19T12:47:24.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f013c5c0 con 0x7f48f013f7c0 2024-08-19T12:47:24.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:24.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f6d9e640 10 monclient(hunting): authenticate will time out at 1238.879559s 2024-08-19T12:47:24.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f5d9c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f48f00a5660 0x7f48f0132520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:32974/0 (socket says 172.21.15.26:32974) 2024-08-19T12:47:24.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.791+0000 7f48f5d9c640 1 -- 172.21.15.26:0/3750037611 learned_addr learned my addr 172.21.15.26:0/3750037611 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:24.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48f5d9c640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.26:6790/0 conn(0x7f48f00a5660 legacy=0x7f48f0132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48ee7fc640 10 client.?.objecter ms_handle_connect 0x7f48f00a5660 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48f659d640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.26:6792/0 conn(0x7f48f013f7c0 legacy=0x7f48f01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48ee7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48ee7fc640 10 client.?.objecter ms_handle_connect 0x7f48f013f7c0 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48ee7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48f559b640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.104:6789/0 conn(0x7f48f0136130 legacy=0x7f48f01d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48ee7fc640 10 client.?.objecter ms_handle_connect 0x7f48f0136130 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48ee7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1021261306 0 0) 0x7f48f013c5c0 con 0x7f48f013f7c0 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48ee7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48ee7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48ee7fc640 10 monclient(hunting): my global_id is 164265 2024-08-19T12:47:24.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.793+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48dc004ee0 con 0x7f48f013f7c0 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3908232789 0 0) 0x7f48f00aa3d0 con 0x7f48f00a5660 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): my global_id is 26615 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48f013c5c0 con 0x7f48f00a5660 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 908086623 0 0) 0x7f48f013e920 con 0x7f48f0136130 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): my global_id is 27424 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48f00aa3d0 con 0x7f48f0136130 2024-08-19T12:47:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1685145271 0 0) 0x7f48dc004ee0 con 0x7f48f013f7c0 2024-08-19T12:47:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48f013e920 con 0x7f48f013f7c0 2024-08-19T12:47:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48e0003990 con 0x7f48f013f7c0 2024-08-19T12:47:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:47:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): dump: 2024-08-19T12:47:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.817 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1176454918 0 0) 0x7f48f013c5c0 con 0x7f48f00a5660 2024-08-19T12:47:24.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48dc004ee0 con 0x7f48f00a5660 2024-08-19T12:47:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48e80046e0 con 0x7f48f00a5660 2024-08-19T12:47:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): dump: 2024-08-19T12:47:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.819 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1429589272 0 0) 0x7f48f00aa3d0 con 0x7f48f0136130 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48f013c5c0 con 0x7f48f0136130 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48e4001c20 con 0x7f48f0136130 2024-08-19T12:47:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient(hunting): dump: 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.821 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 384707935 0 0) 0x7f48f013e920 con 0x7f48f013f7c0 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.26:6790/0 conn(0x7f48f00a5660 legacy=0x7f48f0132520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.104:6789/0 conn(0x7f48f0136130 legacy=0x7f48f01d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 monclient: found mon.d 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:24.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f013e640 con 0x7f48f013f7c0 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient: _finish_auth 0 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48f6d9e640 5 monclient: authenticate success, global_id 164265 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48f6d9e640 10 monclient: _renew_subs 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48f6d9e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48f6d9e640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f48f013d460 con 0x7f48f013f7c0 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48f6d9e640 10 client.164265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48f6d9e640 10 monclient: _renew_subs 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48f6d9e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48f6d9e640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f48f013f0b0 con 0x7f48f013f7c0 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48f6d9e640 1 librados: init done 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48e0003ec0 con 0x7f48f013f7c0 2024-08-19T12:47:24.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48e00043f0 con 0x7f48f013f7c0 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ef7fe640 4 set_mon_vals no callback set 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.794+0000 7f48ee7fc640 10 monclient: dump: 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.824 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.795+0000 7f48f6d9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.795+0000 7f48f6d9e640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f48d00065c0 con 0x7f48d0002860 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.795+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f48e0004590 con 0x7f48f013f7c0 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.795+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(355..355 src has 1..355) ==== 5333+0+0 (unknown 1574624872 0 0) 0x7f48e004e4d0 con 0x7f48f013f7c0 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.795+0000 7f48ee7fc640 10 client.164265.objecter ms_dispatch 0x7f48f0000b90 osd_map(355..355 src has 1..355) 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.795+0000 7f48ee7fc640 3 client.164265.objecter handle_osd_map got epochs [355,355] > 0 2024-08-19T12:47:24.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.795+0000 7f48ee7fc640 3 client.164265.objecter handle_osd_map decoding full epoch 355 2024-08-19T12:47:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.795+0000 7f48ee7fc640 20 client.164265.objecter dump_active .. 0 homeless 2024-08-19T12:47:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.795+0000 7f48f559b640 10 monclient: get_auth_request con 0x7f48dc041990 auth_method 0 2024-08-19T12:47:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 10 client.164265.objecter ms_handle_connect 0x7f48d0002860 2024-08-19T12:47:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 10 client.164265.objecter resend_mon_ops 2024-08-19T12:47:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48f5d9c640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.26:6790/0 conn(0x7f48d0002860 legacy=0x7f48d0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 56105962 0 0) 0x7f48d00065c0 con 0x7f48d0002860 2024-08-19T12:47:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 10 monclient(hunting): my global_id is 26624 2024-08-19T12:47:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48f013e920 con 0x7f48d0002860 2024-08-19T12:47:24.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1996329135 0 0) 0x7f48f013e920 con 0x7f48d0002860 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48d00065c0 con 0x7f48d0002860 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48e8004e40 con 0x7f48d0002860 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.797+0000 7f48ee7fc640 10 monclient: dump: 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.827 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.798+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1131836250 0 0) 0x7f48d00065c0 con 0x7f48d0002860 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.798+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f48d000a160 con 0x7f48d0002860 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.798+0000 7f48ee7fc640 10 client.164265.objecter ms_handle_connect 0x7f48dc041990 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.798+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f48e80063e0 con 0x7f48d0002860 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.798+0000 7f48ee7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:24.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.798+0000 7f48ee7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.798+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.26:6790/0 conn(0x7f48d0002860 legacy=0x7f48d0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.798+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.26:6790/0 conn(0x7f48d0002860 legacy=0x7f48d0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.804+0000 7f48f6d9e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.804+0000 7f48f6d9e640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f48d0009e70 con 0x7f48d0009130 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.806+0000 7f48ee7fc640 10 client.164265.objecter ms_handle_connect 0x7f48d0009130 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.806+0000 7f48ee7fc640 10 client.164265.objecter resend_mon_ops 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.806+0000 7f48f5d9c640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.26:6790/0 conn(0x7f48d0009130 legacy=0x7f48d00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.806+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3213134219 0 0) 0x7f48d0009e70 con 0x7f48d0009130 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.806+0000 7f48ee7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.806+0000 7f48ee7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.806+0000 7f48ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.806+0000 7f48ee7fc640 10 monclient(hunting): my global_id is 26633 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48d00065c0 con 0x7f48d0009130 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2195202483 0 0) 0x7f48d00065c0 con 0x7f48d0009130 2024-08-19T12:47:24.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48d0009e70 con 0x7f48d0009130 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48e80053b0 con 0x7f48d0009130 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 10 monclient: dump: 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.832 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4166537021 0 0) 0x7f48d0009e70 con 0x7f48d0009130 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f48d000c3f0 con 0x7f48d0009130 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7700 (unknown 0 0 3523400619) 0x7f48e8005820 con 0x7f48d0009130 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.26:6790/0 conn(0x7f48d0009130 legacy=0x7f48d00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48ee7fc640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.26:6790/0 conn(0x7f48d0009130 legacy=0x7f48d00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:24.834 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-08-19T12:47:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 557862, 2024-08-19T12:47:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:24.834 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:24.835 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:24.836 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:24.837 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:24.838 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:24.839 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.841 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:24.845 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:24.846 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:24.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:24.851 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:24.852 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:47:24.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:47:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:24.854 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:24.855 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:24.855 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:24.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48f6d9e640 10 librados: watch_flush enter 2024-08-19T12:47:24.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48f6d9e640 10 librados: watch_flush exit 2024-08-19T12:47:24.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48f6d9e640 20 client.164265.objecter shutdown clearing up homeless session... 2024-08-19T12:47:24.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.807+0000 7f48f6d9e640 10 client.164265.objecter successfully canceled tick 2024-08-19T12:47:24.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.808+0000 7f48f6d9e640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.104:6800/899887506 conn(0x7f48dc041990 legacy=0x7f48dc043e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.808+0000 7f48f6d9e640 10 monclient: shutdown 2024-08-19T12:47:24.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.808+0000 7f48f6d9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:24.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.808+0000 7f48f6d9e640 1 -- 172.21.15.26:0/3750037611 >> v1:172.21.15.26:6792/0 conn(0x7f48f013f7c0 legacy=0x7f48f01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.808+0000 7f48f659d640 1 -- 172.21.15.26:0/3750037611 reap_dead start 2024-08-19T12:47:24.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.808+0000 7f48f659d640 1 -- 172.21.15.26:0/3750037611 reap_dead start 2024-08-19T12:47:24.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.808+0000 7f48f6d9e640 1 -- 172.21.15.26:0/3750037611 shutdown_connections 2024-08-19T12:47:24.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.808+0000 7f48f6d9e640 1 -- 172.21.15.26:0/3750037611 >> 172.21.15.26:0/3750037611 conn(0x7f48f00abe30 msgr2=0x7f48f0138900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:24.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.808+0000 7f48f6d9e640 1 -- 172.21.15.26:0/3750037611 shutdown_connections 2024-08-19T12:47:24.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.808+0000 7f48f6d9e640 1 -- 172.21.15.26:0/3750037611 wait complete. 2024-08-19T12:47:24.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.808+0000 7f48f6d9e640 1 librados: shutdown 2024-08-19T12:47:24.858 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:47:24.858 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-08-19T12:47:24.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:47:24.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.955+0000 7f089ef21640 10 monclient: get_monmap_and_config 2024-08-19T12:47:24.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.955+0000 7f089ef21640 10 monclient: build_initial_monmap 2024-08-19T12:47:24.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.955+0000 7f089ef21640 10 monclient: monmap: 2024-08-19T12:47:24.958 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:24.958 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.958 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:24.957462+0000 2024-08-19T12:47:24.958 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:24.957462+0000 2024-08-19T12:47:24.958 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:24.958 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.958 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:24.959 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:24.959 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:24.959 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:24.959 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:24.959 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:24.959 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:24.959 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:24.959 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:24.959 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 1 Processor -- start 2024-08-19T12:47:24.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 1 -- start start 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 10 monclient: init 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 10 monclient: _add_conns ranks=[0,8,1,4,2,6,5,3,7] 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 10 monclient(hunting): picked mon.noname-a con 0x7f089806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 10 monclient(hunting): picked mon.noname-e con 0x7f08980a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 10 monclient(hunting): picked mon.noname-f con 0x7f08980a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0898136790 con 0x7f089806e7b0 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f089809ddf0 con 0x7f08980a5d50 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f089809e0d0 con 0x7f08980a5920 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f089ef21640 10 monclient(hunting): authenticate will time out at 1239.044913s 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.956+0000 7f088ffff640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f08980a5920 legacy=0x7f089813e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:47:24.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.957+0000 7f089ccbd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f089806e7b0 0x7f0898136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50286/0 (socket says 172.21.15.26:50286) 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.957+0000 7f089ccbd640 1 -- 172.21.15.26:0/594602101 learned_addr learned my addr 172.21.15.26:0/594602101 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f089ccbd640 1 -- 172.21.15.26:0/594602101 >> v1:172.21.15.26:6789/0 conn(0x7f089806e7b0 legacy=0x7f0898136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f089d4be640 1 -- 172.21.15.26:0/594602101 >> v1:172.21.15.104:6789/0 conn(0x7f08980a5d50 legacy=0x7f0898141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 1 -- 172.21.15.26:0/594602101 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 742511941 0 0) 0x7f0898136790 con 0x7f089806e7b0 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient(hunting): my global_id is 144954 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 1 -- 172.21.15.26:0/594602101 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0880003520 con 0x7f089806e7b0 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 1 -- 172.21.15.26:0/594602101 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 556325529 0 0) 0x7f0890001820 con 0x7f08980a5d50 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient(hunting): my global_id is 27433 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 1 -- 172.21.15.26:0/594602101 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0898136790 con 0x7f08980a5d50 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 1 -- 172.21.15.26:0/594602101 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2844001623 0 0) 0x7f0880003520 con 0x7f089806e7b0 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 1 -- 172.21.15.26:0/594602101 >> v1:172.21.15.26:6793/0 conn(0x7f08980a5920 legacy=0x7f089813e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 1 -- 172.21.15.26:0/594602101 >> v1:172.21.15.104:6789/0 conn(0x7f08980a5d50 legacy=0x7f0898141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 1 monclient: found mon.noname-a 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 1 -- 172.21.15.26:0/594602101 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089809e3b0 con 0x7f089806e7b0 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:47:24.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 1 -- 172.21.15.26:0/594602101 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0894001ae0 con 0x7f089806e7b0 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f089ef21640 5 monclient: authenticate success, global_id 144954 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f089ef21640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient: dump: 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.976 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f089ef21640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 1 -- 172.21.15.26:0/594602101 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f08940048a0 con 0x7f089806e7b0 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 1 -- 172.21.15.26:0/594602101 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0894004e10 con 0x7f089806e7b0 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f089ef21640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:24.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f089ef21640 4 set_mon_vals no callback set 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.959+0000 7f088f7fe640 10 monclient: dump: 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.978 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.960+0000 7f089ef21640 10 monclient: shutdown 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.960+0000 7f089ef21640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.960+0000 7f089ef21640 1 -- 172.21.15.26:0/594602101 >> v1:172.21.15.26:6789/0 conn(0x7f089806e7b0 legacy=0x7f0898136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.960+0000 7f089ef21640 1 -- 172.21.15.26:0/594602101 shutdown_connections 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.960+0000 7f089ef21640 1 -- 172.21.15.26:0/594602101 >> 172.21.15.26:0/594602101 conn(0x7f08980a8740 msgr2=0x7f08980a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.960+0000 7f089ef21640 1 -- 172.21.15.26:0/594602101 shutdown_connections 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.960+0000 7f089ef21640 1 -- 172.21.15.26:0/594602101 wait complete. 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.960+0000 7f089ef21640 10 monclient: build_initial_monmap 2024-08-19T12:47:24.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.960+0000 7f089ef21640 10 monclient: monmap: 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:24.980 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 1 librados: starting msgr at 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 1 librados: starting objecter 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 1 Processor -- start 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 1 -- start start 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 1 librados: setting wanted keys 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 1 librados: calling monclient init 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 10 monclient: init 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 10 monclient: _add_conns ranks=[6,0,2,5,1,8,7,4,3] 2024-08-19T12:47:24.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 10 monclient(hunting): picked mon.noname-d con 0x7f089806e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 10 monclient(hunting): picked mon.noname-a con 0x7f08980a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 10 monclient(hunting): picked mon.noname-b con 0x7f08980a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08980a1c10 con 0x7f08980a5920 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08980a3820 con 0x7f08980a5d50 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0898134c80 con 0x7f089806e7b0 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ef21640 10 monclient(hunting): authenticate will time out at 1239.049782s 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f088ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f08980a5920 0x7f08981d3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50296/0 (socket says 172.21.15.26:50296) 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f088ffff640 1 -- 172.21.15.26:0/819413729 learned_addr learned my addr 172.21.15.26:0/819413729 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.961+0000 7f089ccbd640 1 --1- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6792/0 conn(0x7f089806e7b0 0x7f08980a4f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39070/0 (socket says 172.21.15.26:39070) 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f089d4be640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6790/0 conn(0x7f08980a5d50 legacy=0x7f08981d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088ffff640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6789/0 conn(0x7f08980a5920 legacy=0x7f08981d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 client.?.objecter ms_handle_connect 0x7f08980a5d50 2024-08-19T12:47:24.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 client.?.objecter ms_handle_connect 0x7f08980a5920 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 client.?.objecter ms_handle_connect 0x7f089806e7b0 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f089ccbd640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6792/0 conn(0x7f089806e7b0 legacy=0x7f08980a4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2403750847 0 0) 0x7f08980a3820 con 0x7f08980a5d50 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): my global_id is 26642 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0884004eb0 con 0x7f08980a5d50 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3871855716 0 0) 0x7f08980a1c10 con 0x7f08980a5920 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): my global_id is 144963 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08980a3820 con 0x7f08980a5920 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4010148695 0 0) 0x7f0898134c80 con 0x7f089806e7b0 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): my global_id is 164274 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08980a1c10 con 0x7f089806e7b0 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2247039359 0 0) 0x7f08980a3820 con 0x7f08980a5920 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0898134c80 con 0x7f08980a5920 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f08880045d0 con 0x7f08980a5920 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:24.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): dump: 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.985 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4230230307 0 0) 0x7f0884004eb0 con 0x7f08980a5d50 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08980a3820 con 0x7f08980a5d50 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2157931671 0 0) 0x7f08980a1c10 con 0x7f089806e7b0 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0884004eb0 con 0x7f089806e7b0 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f08900018a0 con 0x7f08980a5d50 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:24.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): dump: 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.987 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0894004d90 con 0x7f089806e7b0 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient(hunting): dump: 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.988 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2196803766 0 0) 0x7f0898134c80 con 0x7f08980a5920 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6792/0 conn(0x7f089806e7b0 legacy=0x7f08980a4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6790/0 conn(0x7f08980a5d50 legacy=0x7f08981d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient: _finish_hunting 0 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 monclient: found mon.a 2024-08-19T12:47:24.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0898142ec0 con 0x7f08980a5920 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient: _finish_auth 0 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 10 monclient: _check_auth_tickets 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f088d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f089ef21640 5 monclient: authenticate success, global_id 144963 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f089ef21640 10 monclient: _renew_subs 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f089ef21640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.964+0000 7f089ef21640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f08980a3540 con 0x7f08980a5920 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f089ef21640 10 client.144963.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f089ef21640 10 monclient: _renew_subs 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f089ef21640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:47:24.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f089ef21640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0898132de0 con 0x7f08980a5920 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f089ef21640 1 librados: init done 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0888004b50 con 0x7f08980a5920 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f088d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0888005080 con 0x7f08980a5920 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f088d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f088d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f088effd640 4 set_mon_vals no callback set 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f088d7fa640 10 monclient: dump: 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.991 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f089ef21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f089ef21640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f08780065c0 con 0x7f0878002860 2024-08-19T12:47:24.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.965+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f0888005220 con 0x7f08980a5920 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.966+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(355..355 src has 1..355) ==== 5333+0+0 (unknown 1574624872 0 0) 0x7f088804e170 con 0x7f08980a5920 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.966+0000 7f088d7fa640 10 client.144963.objecter ms_dispatch 0x7f0898000b90 osd_map(355..355 src has 1..355) 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.966+0000 7f088d7fa640 3 client.144963.objecter handle_osd_map got epochs [355,355] > 0 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.966+0000 7f088d7fa640 3 client.144963.objecter handle_osd_map decoding full epoch 355 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.966+0000 7f088d7fa640 20 client.144963.objecter dump_active .. 0 homeless 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.966+0000 7f089d4be640 10 monclient: get_auth_request con 0x7f0884041980 auth_method 0 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 10 client.144963.objecter ms_handle_connect 0x7f0878002860 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 10 client.144963.objecter resend_mon_ops 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f089ccbd640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6791/0 conn(0x7f0878002860 legacy=0x7f0878004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2116795725 0 0) 0x7f08780065c0 con 0x7f0878002860 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 10 monclient(hunting): my global_id is 26950 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0898134c80 con 0x7f0878002860 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2391554463 0 0) 0x7f0898134c80 con 0x7f0878002860 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08780065c0 con 0x7f0878002860 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0894001580 con 0x7f0878002860 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 10 monclient: dump: 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.994 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2279306823 0 0) 0x7f08780065c0 con 0x7f0878002860 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f087800a160 con 0x7f0878002860 2024-08-19T12:47:24.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 10 client.144963.objecter ms_handle_connect 0x7f0884041980 2024-08-19T12:47:24.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0894001d30 con 0x7f0878002860 2024-08-19T12:47:24.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:24.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:24.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6791/0 conn(0x7f0878002860 legacy=0x7f0878004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:24.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.968+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6791/0 conn(0x7f0878002860 legacy=0x7f0878004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:24.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.975+0000 7f089ef21640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.975+0000 7f089ef21640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0878009e70 con 0x7f0878009130 2024-08-19T12:47:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ccbd640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6791/0 conn(0x7f0878009130 legacy=0x7f08780095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 10 client.144963.objecter ms_handle_connect 0x7f0878009130 2024-08-19T12:47:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 10 client.144963.objecter resend_mon_ops 2024-08-19T12:47:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3208334155 0 0) 0x7f0878009e70 con 0x7f0878009130 2024-08-19T12:47:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:47:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 10 monclient(hunting): my global_id is 26959 2024-08-19T12:47:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08780065c0 con 0x7f0878009130 2024-08-19T12:47:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2074100629 0 0) 0x7f08780065c0 con 0x7f0878009130 2024-08-19T12:47:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0878009e70 con 0x7f0878009130 2024-08-19T12:47:24.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f08940059c0 con 0x7f0878009130 2024-08-19T12:47:24.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:24.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:24.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 10 monclient: dump: 2024-08-19T12:47:24.999 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:24.999 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:24.999 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.999 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:24.999 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:24.999 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:24.999 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:24.999 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.000 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.000 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.000 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.000 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.000 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.000 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.000 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.000 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1972492731 0 0) 0x7f0878009e70 con 0x7f0878009130 2024-08-19T12:47:25.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f087800c3f0 con 0x7f0878009130 2024-08-19T12:47:25.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 1239902286) 0x7f08940048a0 con 0x7f0878009130 2024-08-19T12:47:25.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:25.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:25.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6791/0 conn(0x7f0878009130 legacy=0x7f08780095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f088d7fa640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6791/0 conn(0x7f0878009130 legacy=0x7f08780095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.001 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:25.001 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:25.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:25.001 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:25.001 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-08-19T12:47:25.002 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 558008, 2024-08-19T12:47:25.002 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:25.002 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:25.002 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:25.002 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:25.002 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:25.002 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:25.002 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:25.002 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:25.002 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:47:25.002 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.002 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:47:25.003 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:25.003 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:25.003 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:25.003 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.003 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.003 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.003 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.003 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.003 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.003 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.003 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.004 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.004 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.004 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:25.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:25.004 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.004 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.004 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.004 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.004 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.004 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.005 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.005 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.005 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.005 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.005 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:25.005 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:25.005 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:25.005 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:25.006 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:25.006 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:25.006 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:25.006 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:25.006 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:25.006 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:25.006 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:25.006 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:25.006 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:25.006 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:25.006 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:25.007 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:25.007 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:25.007 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.007 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.007 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.007 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.007 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.007 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.007 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.007 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:25.009 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:25.013 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:25.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:25.013 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.013 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.013 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:25.014 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:25.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:25.014 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:25.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:25.015 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.015 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.015 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:25.016 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.018 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:25.018 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.018 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.018 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.018 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:25.018 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:25.018 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.018 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.018 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:25.019 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:25.019 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.019 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:25.019 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:25.020 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:25.020 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.020 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.020 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:25.020 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:25.020 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:25.021 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.021 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:25.021 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:25.021 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.021 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.021 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.022 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.022 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:25.022 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:25.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.022 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.022 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.022 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:25.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.022 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.022 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:25.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.023 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.023 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.023 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:25.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.023 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:25.023 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:25.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 10 librados: watch_flush enter 2024-08-19T12:47:25.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 10 librados: watch_flush exit 2024-08-19T12:47:25.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 20 client.144963.objecter shutdown clearing up homeless session... 2024-08-19T12:47:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 10 client.144963.objecter successfully canceled tick 2024-08-19T12:47:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.104:6800/899887506 conn(0x7f0884041980 legacy=0x7f0884043e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 10 monclient: shutdown 2024-08-19T12:47:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 1 -- 172.21.15.26:0/819413729 >> v1:172.21.15.26:6789/0 conn(0x7f08980a5920 legacy=0x7f08981d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089d4be640 1 -- 172.21.15.26:0/819413729 reap_dead start 2024-08-19T12:47:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089d4be640 1 -- 172.21.15.26:0/819413729 reap_dead start 2024-08-19T12:47:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 1 -- 172.21.15.26:0/819413729 shutdown_connections 2024-08-19T12:47:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 1 -- 172.21.15.26:0/819413729 >> 172.21.15.26:0/819413729 conn(0x7f08980a8740 msgr2=0x7f08980aa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 1 -- 172.21.15.26:0/819413729 shutdown_connections 2024-08-19T12:47:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 1 -- 172.21.15.26:0/819413729 wait complete. 2024-08-19T12:47:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:24.978+0000 7f089ef21640 1 librados: shutdown 2024-08-19T12:47:25.027 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:47:25.027 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon d 2024-08-19T12:47:25.027 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:47:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.125+0000 7f723e667640 10 monclient: get_monmap_and_config 2024-08-19T12:47:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.125+0000 7f723e667640 10 monclient: build_initial_monmap 2024-08-19T12:47:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.125+0000 7f723e667640 10 monclient: monmap: 2024-08-19T12:47:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:25.127237+0000 2024-08-19T12:47:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:25.127237+0000 2024-08-19T12:47:25.127 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:25.128 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:25.128 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 1 Processor -- start 2024-08-19T12:47:25.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 1 -- start start 2024-08-19T12:47:25.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 10 monclient: init 2024-08-19T12:47:25.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:25.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 10 monclient: _add_conns ranks=[5,1,8,7,4,6,2,0,3] 2024-08-19T12:47:25.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 10 monclient(hunting): picked mon.noname-h con 0x7f7238073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:25.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 10 monclient(hunting): picked mon.noname-f con 0x7f72380a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:25.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 10 monclient(hunting): picked mon.noname-e con 0x7f723813f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:25.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f723809dae0 con 0x7f72380a57a0 2024-08-19T12:47:25.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7238136910 con 0x7f7238073470 2024-08-19T12:47:25.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f723809dd50 con 0x7f723813f2e0 2024-08-19T12:47:25.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:25.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723e667640 10 monclient(hunting): authenticate will time out at 1239.214649s 2024-08-19T12:47:25.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f723cc04640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f723813f2e0 legacy=0x7f72381416d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:47:25.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f72377fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f72380a57a0 0x7f72380a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:44182/0 (socket says 172.21.15.26:44182) 2024-08-19T12:47:25.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.126+0000 7f72377fe640 1 -- 172.21.15.26:0/2798527245 learned_addr learned my addr 172.21.15.26:0/2798527245 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f72377fe640 1 -- 172.21.15.26:0/2798527245 >> v1:172.21.15.104:6789/0 conn(0x7f72380a57a0 legacy=0x7f72380a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7237fff640 1 -- 172.21.15.26:0/2798527245 >> v1:172.21.15.104:6791/0 conn(0x7f7238073470 legacy=0x7f7238136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 1 -- 172.21.15.26:0/2798527245 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 514162407 0 0) 0x7f723809dae0 con 0x7f72380a57a0 2024-08-19T12:47:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient(hunting): my global_id is 27442 2024-08-19T12:47:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 1 -- 172.21.15.26:0/2798527245 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7220003520 con 0x7f72380a57a0 2024-08-19T12:47:25.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 1 -- 172.21.15.26:0/2798527245 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 962719080 0 0) 0x7f7238136910 con 0x7f7238073470 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient(hunting): my global_id is 26654 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 1 -- 172.21.15.26:0/2798527245 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f723809dae0 con 0x7f7238073470 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 1 -- 172.21.15.26:0/2798527245 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1705454423 0 0) 0x7f7220003520 con 0x7f72380a57a0 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 1 -- 172.21.15.26:0/2798527245 >> v1:172.21.15.26:6793/0 conn(0x7f723813f2e0 legacy=0x7f72381416d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 1 -- 172.21.15.26:0/2798527245 >> v1:172.21.15.104:6791/0 conn(0x7f7238073470 legacy=0x7f7238136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 1 monclient: found mon.noname-f 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:47:25.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 1 -- 172.21.15.26:0/2798527245 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f723809e030 con 0x7f72380a57a0 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient: _finish_auth 0 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 1 -- 172.21.15.26:0/2798527245 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f722c001ae0 con 0x7f72380a57a0 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient: dump: 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.142 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f723e667640 5 monclient: authenticate success, global_id 27442 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f723e667640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:25.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 1 -- 172.21.15.26:0/2798527245 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f722c0048a0 con 0x7f72380a57a0 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f723e667640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f723e667640 4 set_mon_vals no callback set 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 1 -- 172.21.15.26:0/2798527245 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f722c004e50 con 0x7f72380a57a0 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.129+0000 7f7236ffd640 10 monclient: dump: 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.144 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.130+0000 7f723e667640 10 monclient: shutdown 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.130+0000 7f723e667640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.130+0000 7f723e667640 1 -- 172.21.15.26:0/2798527245 >> v1:172.21.15.104:6789/0 conn(0x7f72380a57a0 legacy=0x7f72380a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.130+0000 7f723e667640 1 -- 172.21.15.26:0/2798527245 shutdown_connections 2024-08-19T12:47:25.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.130+0000 7f723e667640 1 -- 172.21.15.26:0/2798527245 >> 172.21.15.26:0/2798527245 conn(0x7f72380abff0 msgr2=0x7f72380ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.130+0000 7f723e667640 1 -- 172.21.15.26:0/2798527245 shutdown_connections 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.130+0000 7f723e667640 1 -- 172.21.15.26:0/2798527245 wait complete. 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.130+0000 7f723e667640 10 monclient: build_initial_monmap 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.130+0000 7f723e667640 10 monclient: monmap: 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:25.146 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 1 librados: starting msgr at 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 1 librados: starting objecter 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 1 Processor -- start 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 1 -- start start 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 1 librados: setting wanted keys 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 1 librados: calling monclient init 2024-08-19T12:47:25.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 10 monclient: init 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 10 monclient: _add_conns ranks=[5,7,8,2,6,0,4,1,3] 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 10 monclient(hunting): picked mon.noname-h con 0x7f7238073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 10 monclient(hunting): picked mon.noname-i con 0x7f72380a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 10 monclient(hunting): picked mon.noname-e con 0x7f723813f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7238131200 con 0x7f7238073470 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f723812e9b0 con 0x7f72380a57a0 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723cc04640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f723813f2e0 legacy=0x7f72381d6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f723812f900 con 0x7f723813f2e0 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.131+0000 7f723e667640 10 monclient(hunting): authenticate will time out at 1239.219929s 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.132+0000 7f72377fe640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f72380a57a0 0x7f723812e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:53266/0 (socket says 172.21.15.26:53266) 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.132+0000 7f72377fe640 1 -- 172.21.15.26:0/2782545809 learned_addr learned my addr 172.21.15.26:0/2782545809 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:25.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7237fff640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.104:6791/0 conn(0x7f7238073470 legacy=0x7f723812db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 client.?.objecter ms_handle_connect 0x7f7238073470 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 client.?.objecter ms_handle_connect 0x7f72380a57a0 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f72377fe640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.104:6792/0 conn(0x7f72380a57a0 legacy=0x7f723812e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2357961206 0 0) 0x7f7238131200 con 0x7f7238073470 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): my global_id is 26663 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7224004eb0 con 0x7f7238073470 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2948295401 0 0) 0x7f723812e9b0 con 0x7f72380a57a0 2024-08-19T12:47:25.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): my global_id is 135565 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7238131200 con 0x7f72380a57a0 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4116822581 0 0) 0x7f7224004eb0 con 0x7f7238073470 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f723812e9b0 con 0x7f7238073470 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7228003950 con 0x7f7238073470 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): dump: 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.150 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.151 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2595681434 0 0) 0x7f7238131200 con 0x7f72380a57a0 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7224004eb0 con 0x7f72380a57a0 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f722c001a60 con 0x7f72380a57a0 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient(hunting): dump: 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.152 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 169015414 0 0) 0x7f723812e9b0 con 0x7f7238073470 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6793/0 conn(0x7f723813f2e0 legacy=0x7f72381d6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.104:6792/0 conn(0x7f72380a57a0 legacy=0x7f723812e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 monclient: found mon.h 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72381314e0 con 0x7f7238073470 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient: _finish_auth 0 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f7234ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f723e667640 5 monclient: authenticate success, global_id 26663 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f723e667640 10 monclient: _renew_subs 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f723e667640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f723e667640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7238132d90 con 0x7f7238073470 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f723e667640 10 client.26663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f723e667640 10 monclient: _renew_subs 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f723e667640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f723e667640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f72381d7870 con 0x7f7238073470 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.134+0000 7f723e667640 1 librados: init done 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.135+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7228003b70 con 0x7f7238073470 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.135+0000 7f7234ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.135+0000 7f72367fc640 4 set_mon_vals no callback set 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.135+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f72280041a0 con 0x7f7238073470 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.135+0000 7f7234ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.135+0000 7f7234ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.135+0000 7f7234ff9640 10 monclient: dump: 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.155 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.135+0000 7f723e667640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.135+0000 7f723e667640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f72180065c0 con 0x7f7218002860 2024-08-19T12:47:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.136+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f7228011a80 con 0x7f7238073470 2024-08-19T12:47:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.136+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(355..355 src has 1..355) ==== 5333+0+0 (unknown 1574624872 0 0) 0x7f722804e2a0 con 0x7f7238073470 2024-08-19T12:47:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.136+0000 7f7234ff9640 10 client.26663.objecter ms_dispatch 0x7f7238000b90 osd_map(355..355 src has 1..355) 2024-08-19T12:47:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.136+0000 7f7234ff9640 3 client.26663.objecter handle_osd_map got epochs [355,355] > 0 2024-08-19T12:47:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.136+0000 7f7234ff9640 3 client.26663.objecter handle_osd_map decoding full epoch 355 2024-08-19T12:47:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.136+0000 7f723cc04640 10 monclient: get_auth_request con 0x7f7224041440 auth_method 0 2024-08-19T12:47:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.136+0000 7f7234ff9640 20 client.26663.objecter dump_active .. 0 homeless 2024-08-19T12:47:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.137+0000 7f72377fe640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f7218002860 legacy=0x7f7218004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f7234ff9640 10 client.26663.objecter ms_handle_connect 0x7f7218002860 2024-08-19T12:47:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f7234ff9640 10 client.26663.objecter resend_mon_ops 2024-08-19T12:47:25.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 466375018 0 0) 0x7f72180065c0 con 0x7f7218002860 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f7234ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f7234ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f7234ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f7234ff9640 10 monclient(hunting): my global_id is 164283 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f723812e9b0 con 0x7f7218002860 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f72377fe640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f7218002860 legacy=0x7f7218004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f72377fe640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f7218002860 legacy=0x7f7218004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f72377fe640 1 --1- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f7218002860 0x7f7218004d00 :-1 s=OPENED pgs=3297 cs=1 l=1).handle_message read tag failed 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f72377fe640 1 --1- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f7218002860 0x7f7218004d00 :-1 s=OPENED pgs=3297 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f7218002860 legacy=0x7f7218004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f7218002860 legacy=0x7f7218004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f72180065c0 con 0x7f722404ac20 2024-08-19T12:47:25.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.138+0000 7f7234ff9640 10 client.26663.objecter ms_handle_connect 0x7f7224041440 2024-08-19T12:47:25.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.140+0000 7f72377fe640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f722404ac20 legacy=0x7f722404b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.140+0000 7f7234ff9640 10 client.26663.objecter ms_handle_connect 0x7f722404ac20 2024-08-19T12:47:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.140+0000 7f7234ff9640 10 client.26663.objecter resend_mon_ops 2024-08-19T12:47:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.140+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 656857229 0 0) 0x7f72180065c0 con 0x7f722404ac20 2024-08-19T12:47:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.140+0000 7f7234ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.140+0000 7f7234ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.140+0000 7f7234ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.140+0000 7f7234ff9640 10 monclient(hunting): my global_id is 164292 2024-08-19T12:47:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.140+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f723812f900 con 0x7f722404ac20 2024-08-19T12:47:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3272276070 0 0) 0x7f723812f900 con 0x7f722404ac20 2024-08-19T12:47:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72180065c0 con 0x7f722404ac20 2024-08-19T12:47:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f722c001d30 con 0x7f722404ac20 2024-08-19T12:47:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 10 monclient: dump: 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.163 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1033710013 0 0) 0x7f72180065c0 con 0x7f722404ac20 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f722404c8c0 con 0x7f722404ac20 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f722c0055c0 con 0x7f722404ac20 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f722404ac20 legacy=0x7f722404b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.141+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f722404ac20 legacy=0x7f722404b030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.147+0000 7f723e667640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.147+0000 7f723e667640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7218009d80 con 0x7f7218009140 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f72377fe640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f7218009140 legacy=0x7f7218009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 10 client.26663.objecter ms_handle_connect 0x7f7218009140 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 10 client.26663.objecter resend_mon_ops 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1365358863 0 0) 0x7f7218009d80 con 0x7f7218009140 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 10 monclient(hunting): my global_id is 164301 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72180065c0 con 0x7f7218009140 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 187659409 0 0) 0x7f72180065c0 con 0x7f7218009140 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7218009d80 con 0x7f7218009140 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f722c005ca0 con 0x7f7218009140 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:25.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 10 monclient: dump: 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.166 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.167 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.167 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4006644484 0 0) 0x7f7218009d80 con 0x7f7218009140 2024-08-19T12:47:25.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f721800c350 con 0x7f7218009140 2024-08-19T12:47:25.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7532 (unknown 0 0 2360788976) 0x7f722c006100 con 0x7f7218009140 2024-08-19T12:47:25.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:25.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:25.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f7218009140 legacy=0x7f7218009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f7234ff9640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.26:6792/0 conn(0x7f7218009140 legacy=0x7f7218009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.150+0000 7f723cc04640 1 -- 172.21.15.26:0/2782545809 reap_dead start 2024-08-19T12:47:25.167 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 28705, 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:25.168 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.169 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.170 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:25.171 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.172 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.173 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:25.174 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.177 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.178 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:25.181 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.182 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:25.185 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.187 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:25.187 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:25.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 10 librados: watch_flush enter 2024-08-19T12:47:25.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 10 librados: watch_flush exit 2024-08-19T12:47:25.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 20 client.26663.objecter shutdown clearing up homeless session... 2024-08-19T12:47:25.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 10 client.26663.objecter successfully canceled tick 2024-08-19T12:47:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.104:6800/899887506 conn(0x7f7224041440 legacy=0x7f7224043900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 10 monclient: shutdown 2024-08-19T12:47:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 1 -- 172.21.15.26:0/2782545809 >> v1:172.21.15.104:6791/0 conn(0x7f7238073470 legacy=0x7f723812db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 1 -- 172.21.15.26:0/2782545809 shutdown_connections 2024-08-19T12:47:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 1 -- 172.21.15.26:0/2782545809 >> 172.21.15.26:0/2782545809 conn(0x7f72380abff0 msgr2=0x7f72381345c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 1 -- 172.21.15.26:0/2782545809 shutdown_connections 2024-08-19T12:47:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 1 -- 172.21.15.26:0/2782545809 wait complete. 2024-08-19T12:47:25.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.151+0000 7f723e667640 1 librados: shutdown 2024-08-19T12:47:25.189 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:47:25.190 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-08-19T12:47:25.190 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:47:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.297+0000 7f7a0c586640 10 monclient: get_monmap_and_config 2024-08-19T12:47:25.299 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.297+0000 7f7a0c586640 10 monclient: build_initial_monmap 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.297+0000 7f7a0c586640 10 monclient: monmap: 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:25.299762+0000 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:25.299762+0000 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:25.300 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:25.301 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:25.301 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.298+0000 7f7a0c586640 1 Processor -- start 2024-08-19T12:47:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.298+0000 7f7a0c586640 1 -- start start 2024-08-19T12:47:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.298+0000 7f7a0c586640 10 monclient: init 2024-08-19T12:47:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.299+0000 7f7a0c586640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.299+0000 7f7a0c586640 10 monclient: _add_conns ranks=[6,5,4,7,3,2,8,1,0] 2024-08-19T12:47:25.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.299+0000 7f7a0c586640 10 monclient(hunting): picked mon.noname-d con 0x7f7a0406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.299+0000 7f7a0c586640 10 monclient(hunting): picked mon.noname-h con 0x7f7a040a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.299+0000 7f7a0c586640 10 monclient(hunting): picked mon.noname-c con 0x7f7a040a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.299+0000 7f7a0c586640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a04136790 con 0x7f7a040a5d50 2024-08-19T12:47:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.299+0000 7f7a0c586640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a0409ddf0 con 0x7f7a040a5920 2024-08-19T12:47:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.299+0000 7f7a0c586640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a0409e0d0 con 0x7f7a0406e7b0 2024-08-19T12:47:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.299+0000 7f7a0c586640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.299+0000 7f7a0c586640 10 monclient(hunting): authenticate will time out at 1239.387250s 2024-08-19T12:47:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.299+0000 7f7a0a322640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f7a0406e7b0 0x7f7a04136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39096/0 (socket says 172.21.15.26:39096) 2024-08-19T12:47:25.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.299+0000 7f7a0a322640 1 -- 172.21.15.26:0/3830581645 learned_addr learned my addr 172.21.15.26:0/3830581645 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a0ab23640 1 -- 172.21.15.26:0/3830581645 >> v1:172.21.15.26:6791/0 conn(0x7f7a040a5d50 legacy=0x7f7a04141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a0a322640 1 -- 172.21.15.26:0/3830581645 >> v1:172.21.15.26:6792/0 conn(0x7f7a0406e7b0 legacy=0x7f7a04136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a09b21640 1 -- 172.21.15.26:0/3830581645 >> v1:172.21.15.104:6791/0 conn(0x7f7a040a5920 legacy=0x7f7a0413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a09320640 1 -- 172.21.15.26:0/3830581645 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2090018792 0 0) 0x7f7a04136790 con 0x7f7a040a5d50 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a0a322640 1 -- 172.21.15.26:0/3830581645 >> v1:172.21.15.26:6792/0 conn(0x7f7a0406e7b0 legacy=0x7f7a04136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a09320640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a09320640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a09320640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a0a322640 1 -- 172.21.15.26:0/3830581645 >> v1:172.21.15.26:6792/0 conn(0x7f7a0406e7b0 legacy=0x7f7a04136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a09320640 10 monclient(hunting): my global_id is 26968 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a0a322640 1 --1- 172.21.15.26:0/3830581645 >> v1:172.21.15.26:6792/0 conn(0x7f7a0406e7b0 0x7f7a04136080 :-1 s=OPENED pgs=3301 cs=1 l=1).handle_message read tag failed 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a0a322640 1 --1- 172.21.15.26:0/3830581645 >> v1:172.21.15.26:6792/0 conn(0x7f7a0406e7b0 0x7f7a04136080 :-1 s=OPENED pgs=3301 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a09320640 1 -- 172.21.15.26:0/3830581645 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79f0003520 con 0x7f7a040a5d50 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.301+0000 7f7a09320640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6792/0 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 1 -- 172.21.15.26:0/3830581645 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1956959191 0 0) 0x7f79f40014a0 con 0x7f7a040a5920 2024-08-19T12:47:25.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient(hunting): my global_id is 26672 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 1 -- 172.21.15.26:0/3830581645 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a04136790 con 0x7f7a040a5920 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 1 -- 172.21.15.26:0/3830581645 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1504197466 0 0) 0x7f79f0003520 con 0x7f7a040a5d50 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 1 -- 172.21.15.26:0/3830581645 >> v1:172.21.15.26:6792/0 conn(0x7f7a0406e7b0 legacy=0x7f7a04136080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 1 -- 172.21.15.26:0/3830581645 >> v1:172.21.15.104:6791/0 conn(0x7f7a040a5920 legacy=0x7f7a0413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient: _finish_hunting 0 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 1 monclient: found mon.noname-c 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 1 -- 172.21.15.26:0/3830581645 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a0409e3b0 con 0x7f7a040a5d50 2024-08-19T12:47:25.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient: _finish_auth 0 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient: _check_auth_tickets 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a0c586640 5 monclient: authenticate success, global_id 26968 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 1 -- 172.21.15.26:0/3830581645 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79fc001eb0 con 0x7f7a040a5d50 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a0c586640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient: dump: 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.310 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a0c586640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 1 -- 172.21.15.26:0/3830581645 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79fc003990 con 0x7f7a040a5d50 2024-08-19T12:47:25.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a0c586640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 1 -- 172.21.15.26:0/3830581645 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79fc003f00 con 0x7f7a040a5d50 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a0c586640 4 set_mon_vals no callback set 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a09320640 10 monclient: dump: 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.312 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a0c586640 10 monclient: shutdown 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a0c586640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a0c586640 1 -- 172.21.15.26:0/3830581645 >> v1:172.21.15.26:6791/0 conn(0x7f7a040a5d50 legacy=0x7f7a04141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a0c586640 1 -- 172.21.15.26:0/3830581645 shutdown_connections 2024-08-19T12:47:25.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a0c586640 1 -- 172.21.15.26:0/3830581645 >> 172.21.15.26:0/3830581645 conn(0x7f7a040a8740 msgr2=0x7f7a040a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a0c586640 1 -- 172.21.15.26:0/3830581645 shutdown_connections 2024-08-19T12:47:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.302+0000 7f7a0c586640 1 -- 172.21.15.26:0/3830581645 wait complete. 2024-08-19T12:47:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.303+0000 7f7a0c586640 10 monclient: build_initial_monmap 2024-08-19T12:47:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.303+0000 7f7a0c586640 10 monclient: monmap: 2024-08-19T12:47:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:25.314 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.303+0000 7f7a0c586640 1 librados: starting msgr at 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.303+0000 7f7a0c586640 1 librados: starting objecter 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.303+0000 7f7a0c586640 1 Processor -- start 2024-08-19T12:47:25.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.303+0000 7f7a0c586640 1 -- start start 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.303+0000 7f7a0c586640 1 librados: setting wanted keys 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.303+0000 7f7a0c586640 1 librados: calling monclient init 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.303+0000 7f7a0c586640 10 monclient: init 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.304+0000 7f7a0c586640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.304+0000 7f7a0c586640 10 monclient: _add_conns ranks=[6,2,3,1,4,7,5,0,8] 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.304+0000 7f7a0c586640 10 monclient(hunting): picked mon.noname-d con 0x7f7a0406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.304+0000 7f7a0c586640 10 monclient(hunting): picked mon.noname-b con 0x7f7a040a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.304+0000 7f7a0c586640 10 monclient(hunting): picked mon.noname-g con 0x7f7a040a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.304+0000 7f7a0c586640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a040a1040 con 0x7f7a040a5920 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.304+0000 7f7a0c586640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a040a1ba0 con 0x7f7a040a5d50 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.304+0000 7f7a0c586640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a040a2b00 con 0x7f7a0406e7b0 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.304+0000 7f7a0c586640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:25.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.304+0000 7f7a0c586640 10 monclient(hunting): authenticate will time out at 1239.392411s 2024-08-19T12:47:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.304+0000 7f7a0a322640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f7a0406e7b0 0x7f7a041ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39112/0 (socket says 172.21.15.26:39112) 2024-08-19T12:47:25.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.304+0000 7f7a0a322640 1 -- 172.21.15.26:0/3812574052 learned_addr learned my addr 172.21.15.26:0/3812574052 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f7a0a322640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.26:6792/0 conn(0x7f7a0406e7b0 legacy=0x7f7a041ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 client.?.objecter ms_handle_connect 0x7f7a0406e7b0 2024-08-19T12:47:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 client.?.objecter ms_handle_connect 0x7f7a040a5920 2024-08-19T12:47:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f7a09b21640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.26:6790/0 conn(0x7f7a040a5920 legacy=0x7f7a041d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1982057367 0 0) 0x7f7a040a2b00 con 0x7f7a0406e7b0 2024-08-19T12:47:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 monclient(hunting): my global_id is 164319 2024-08-19T12:47:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79e8004eb0 con 0x7f7a0406e7b0 2024-08-19T12:47:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1706638305 0 0) 0x7f7a040a1040 con 0x7f7a040a5920 2024-08-19T12:47:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 monclient(hunting): my global_id is 26651 2024-08-19T12:47:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a040a2b00 con 0x7f7a040a5920 2024-08-19T12:47:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 client.?.objecter ms_handle_connect 0x7f7a040a5d50 2024-08-19T12:47:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f7a0ab23640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.104:6790/0 conn(0x7f7a040a5d50 legacy=0x7f7a041d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.306+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2560420106 0 0) 0x7f79e8004eb0 con 0x7f7a0406e7b0 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7a040a1040 con 0x7f7a0406e7b0 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7a000047e0 con 0x7f7a0406e7b0 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient(hunting): dump: 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.324 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3537862034 0 0) 0x7f7a040a2b00 con 0x7f7a040a5920 2024-08-19T12:47:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79e8004eb0 con 0x7f7a040a5920 2024-08-19T12:47:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79f4003950 con 0x7f7a040a5920 2024-08-19T12:47:25.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient(hunting): dump: 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.325 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2505983593 0 0) 0x7f7a040a1ba0 con 0x7f7a040a5d50 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient(hunting): my global_id is 154443 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a040a2b00 con 0x7f7a040a5d50 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2186303051 0 0) 0x7f7a040a1040 con 0x7f7a0406e7b0 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.104:6790/0 conn(0x7f7a040a5d50 legacy=0x7f7a041d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.26:6790/0 conn(0x7f7a040a5920 legacy=0x7f7a041d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient: _finish_hunting 0 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 monclient: found mon.d 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a040a3dd0 con 0x7f7a0406e7b0 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient: _finish_auth 0 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient: _check_auth_tickets 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f7a0c586640 5 monclient: authenticate success, global_id 164319 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f7a0c586640 10 monclient: _renew_subs 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f7a0c586640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f7a0c586640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7a040a4060 con 0x7f7a0406e7b0 2024-08-19T12:47:25.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f7a0c586640 10 client.164319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f7a0c586640 10 monclient: _renew_subs 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f7a0c586640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f7a0c586640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f7a040a0af0 con 0x7f7a0406e7b0 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f7a0c586640 1 librados: init done 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7a00004d00 con 0x7f7a0406e7b0 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7a00005270 con 0x7f7a0406e7b0 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79fbfff640 4 set_mon_vals no callback set 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.307+0000 7f79faffd640 10 monclient: dump: 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.328 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.308+0000 7f7a0c586640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:25.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.308+0000 7f7a0c586640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f79ec0065c0 con 0x7f79ec002860 2024-08-19T12:47:25.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.308+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f7a00005410 con 0x7f7a0406e7b0 2024-08-19T12:47:25.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.308+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(356..356 src has 1..356) ==== 5346+0+0 (unknown 2942184895 0 0) 0x7f7a0004e310 con 0x7f7a0406e7b0 2024-08-19T12:47:25.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.308+0000 7f79faffd640 10 client.164319.objecter ms_dispatch 0x7f7a04000b90 osd_map(356..356 src has 1..356) 2024-08-19T12:47:25.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.308+0000 7f79faffd640 3 client.164319.objecter handle_osd_map got epochs [356,356] > 0 2024-08-19T12:47:25.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.308+0000 7f79faffd640 3 client.164319.objecter handle_osd_map decoding full epoch 356 2024-08-19T12:47:25.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.308+0000 7f79faffd640 20 client.164319.objecter dump_active .. 0 homeless 2024-08-19T12:47:25.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.308+0000 7f7a0ab23640 10 monclient: get_auth_request con 0x7f79e8041580 auth_method 0 2024-08-19T12:47:25.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.310+0000 7f7a09b21640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.104:6789/0 conn(0x7f79ec002860 legacy=0x7f79ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.310+0000 7f79faffd640 10 client.164319.objecter ms_handle_connect 0x7f79ec002860 2024-08-19T12:47:25.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.310+0000 7f79faffd640 10 client.164319.objecter resend_mon_ops 2024-08-19T12:47:25.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.310+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 944184620 0 0) 0x7f79ec0065c0 con 0x7f79ec002860 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.310+0000 7f79faffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.310+0000 7f79faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.310+0000 7f79faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.310+0000 7f79faffd640 10 monclient(hunting): my global_id is 27451 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.310+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a040a1040 con 0x7f79ec002860 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.310+0000 7f79faffd640 10 client.164319.objecter ms_handle_connect 0x7f79e8041580 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2201866155 0 0) 0x7f7a040a1040 con 0x7f79ec002860 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79ec0065c0 con 0x7f79ec002860 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79f4003f30 con 0x7f79ec002860 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 10 monclient: dump: 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.331 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.332 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2872571284 0 0) 0x7f79ec0065c0 con 0x7f79ec002860 2024-08-19T12:47:25.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f79ec00a160 con 0x7f79ec002860 2024-08-19T12:47:25.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f79f4003fc0 con 0x7f79ec002860 2024-08-19T12:47:25.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:25.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:25.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.104:6789/0 conn(0x7f79ec002860 legacy=0x7f79ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.311+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.104:6789/0 conn(0x7f79ec002860 legacy=0x7f79ec004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.317+0000 7f7a0c586640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:25.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.317+0000 7f7a0c586640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f79ec009e70 con 0x7f79ec009130 2024-08-19T12:47:25.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f7a09b21640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.104:6789/0 conn(0x7f79ec009130 legacy=0x7f79ec0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 10 client.164319.objecter ms_handle_connect 0x7f79ec009130 2024-08-19T12:47:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 10 client.164319.objecter resend_mon_ops 2024-08-19T12:47:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2201804552 0 0) 0x7f79ec009e70 con 0x7f79ec009130 2024-08-19T12:47:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 10 monclient(hunting): my global_id is 27460 2024-08-19T12:47:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79ec0065c0 con 0x7f79ec009130 2024-08-19T12:47:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1364541901 0 0) 0x7f79ec0065c0 con 0x7f79ec009130 2024-08-19T12:47:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79ec009e70 con 0x7f79ec009130 2024-08-19T12:47:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79f4004460 con 0x7f79ec009130 2024-08-19T12:47:25.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 10 monclient: dump: 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.337 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3740336447 0 0) 0x7f79ec009e70 con 0x7f79ec009130 2024-08-19T12:47:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f79ec00c410 con 0x7f79ec009130 2024-08-19T12:47:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7700 (unknown 0 0 273141744) 0x7f79f4005cb0 con 0x7f79ec009130 2024-08-19T12:47:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.104:6789/0 conn(0x7f79ec009130 legacy=0x7f79ec0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.320+0000 7f79faffd640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.104:6789/0 conn(0x7f79ec009130 legacy=0x7f79ec0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.338 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-08-19T12:47:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 558326, 2024-08-19T12:47:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:25.338 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:47:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:47:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.339 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:25.340 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.341 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:25.342 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.343 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.345 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:25.346 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.347 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:25.350 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:25.351 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:25.354 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:25.355 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:25.356 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.357 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:47:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.358 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:25.358 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:25.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 10 librados: watch_flush enter 2024-08-19T12:47:25.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 10 librados: watch_flush exit 2024-08-19T12:47:25.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 20 client.164319.objecter shutdown clearing up homeless session... 2024-08-19T12:47:25.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 10 client.164319.objecter successfully canceled tick 2024-08-19T12:47:25.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.104:6800/899887506 conn(0x7f79e8041580 legacy=0x7f79e8043a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 10 monclient: shutdown 2024-08-19T12:47:25.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:25.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 1 -- 172.21.15.26:0/3812574052 >> v1:172.21.15.26:6792/0 conn(0x7f7a0406e7b0 legacy=0x7f7a041ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0ab23640 1 -- 172.21.15.26:0/3812574052 reap_dead start 2024-08-19T12:47:25.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0ab23640 1 -- 172.21.15.26:0/3812574052 reap_dead start 2024-08-19T12:47:25.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 1 -- 172.21.15.26:0/3812574052 shutdown_connections 2024-08-19T12:47:25.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 1 -- 172.21.15.26:0/3812574052 >> 172.21.15.26:0/3812574052 conn(0x7f7a040a8740 msgr2=0x7f7a040aa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:25.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 1 -- 172.21.15.26:0/3812574052 shutdown_connections 2024-08-19T12:47:25.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 1 -- 172.21.15.26:0/3812574052 wait complete. 2024-08-19T12:47:25.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.321+0000 7f7a0c586640 1 librados: shutdown 2024-08-19T12:47:25.361 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:47:25.362 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-08-19T12:47:25.362 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:47:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.459+0000 7f3557314640 10 monclient: get_monmap_and_config 2024-08-19T12:47:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.459+0000 7f3557314640 10 monclient: build_initial_monmap 2024-08-19T12:47:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.459+0000 7f3557314640 10 monclient: monmap: 2024-08-19T12:47:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.461 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:25.461619+0000 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:25.461619+0000 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:25.462 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.460+0000 7f3557314640 1 Processor -- start 2024-08-19T12:47:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.460+0000 7f3557314640 1 -- start start 2024-08-19T12:47:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.460+0000 7f3557314640 10 monclient: init 2024-08-19T12:47:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.461+0000 7f3557314640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.461+0000 7f3557314640 10 monclient: _add_conns ranks=[0,3,1,2,4,8,7,5,6] 2024-08-19T12:47:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.461+0000 7f3557314640 10 monclient(hunting): picked mon.noname-a con 0x7f355006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.461+0000 7f3557314640 10 monclient(hunting): picked mon.noname-g con 0x7f35500a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.461+0000 7f3557314640 10 monclient(hunting): picked mon.noname-f con 0x7f35500a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.461+0000 7f3557314640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3550136790 con 0x7f355006e7b0 2024-08-19T12:47:25.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.461+0000 7f3557314640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355009ddf0 con 0x7f35500a5d50 2024-08-19T12:47:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.461+0000 7f3557314640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355009e0d0 con 0x7f35500a5920 2024-08-19T12:47:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.461+0000 7f3557314640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.461+0000 7f3557314640 10 monclient(hunting): authenticate will time out at 1239.549108s 2024-08-19T12:47:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.461+0000 7f35550b0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f355006e7b0 0x7f3550136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50310/0 (socket says 172.21.15.26:50310) 2024-08-19T12:47:25.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.461+0000 7f35550b0640 1 -- 172.21.15.26:0/1792214159 learned_addr learned my addr 172.21.15.26:0/1792214159 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f35550b0640 1 -- 172.21.15.26:0/1792214159 >> v1:172.21.15.26:6789/0 conn(0x7f355006e7b0 legacy=0x7f3550136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 1 -- 172.21.15.26:0/1792214159 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3775418145 0 0) 0x7f3550136790 con 0x7f355006e7b0 2024-08-19T12:47:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient(hunting): my global_id is 144972 2024-08-19T12:47:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 1 -- 172.21.15.26:0/1792214159 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f353c003520 con 0x7f355006e7b0 2024-08-19T12:47:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f35548af640 1 -- 172.21.15.26:0/1792214159 >> v1:172.21.15.104:6790/0 conn(0x7f35500a5920 legacy=0x7f355013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f35558b1640 1 -- 172.21.15.26:0/1792214159 >> v1:172.21.15.104:6789/0 conn(0x7f35500a5d50 legacy=0x7f3550141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 1 -- 172.21.15.26:0/1792214159 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 947409133 0 0) 0x7f353c003520 con 0x7f355006e7b0 2024-08-19T12:47:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 1 -- 172.21.15.26:0/1792214159 >> v1:172.21.15.104:6790/0 conn(0x7f35500a5920 legacy=0x7f355013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 1 -- 172.21.15.26:0/1792214159 >> v1:172.21.15.104:6789/0 conn(0x7f35500a5d50 legacy=0x7f3550141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient: _finish_hunting 0 2024-08-19T12:47:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 1 monclient: found mon.noname-a 2024-08-19T12:47:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:47:25.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 1 -- 172.21.15.26:0/1792214159 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355009e3b0 con 0x7f355006e7b0 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient: _finish_auth 0 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient: _check_auth_tickets 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 1 -- 172.21.15.26:0/1792214159 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f354c001ae0 con 0x7f355006e7b0 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient: dump: 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.470 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3557314640 5 monclient: authenticate success, global_id 144972 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3557314640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:25.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 1 -- 172.21.15.26:0/1792214159 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f354c0048a0 con 0x7f355006e7b0 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3557314640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3557314640 4 set_mon_vals no callback set 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 1 -- 172.21.15.26:0/1792214159 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f354c004dd0 con 0x7f355006e7b0 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.463+0000 7f3547fff640 10 monclient: dump: 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.472 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.464+0000 7f3557314640 10 monclient: shutdown 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.464+0000 7f3557314640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.464+0000 7f3557314640 1 -- 172.21.15.26:0/1792214159 >> v1:172.21.15.26:6789/0 conn(0x7f355006e7b0 legacy=0x7f3550136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.464+0000 7f3557314640 1 -- 172.21.15.26:0/1792214159 shutdown_connections 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.464+0000 7f3557314640 1 -- 172.21.15.26:0/1792214159 >> 172.21.15.26:0/1792214159 conn(0x7f35500a8740 msgr2=0x7f35500a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.464+0000 7f3557314640 1 -- 172.21.15.26:0/1792214159 shutdown_connections 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.464+0000 7f3557314640 1 -- 172.21.15.26:0/1792214159 wait complete. 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 10 monclient: build_initial_monmap 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 10 monclient: monmap: 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:25.474 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 1 librados: starting msgr at 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 1 librados: starting objecter 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 1 Processor -- start 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 1 -- start start 2024-08-19T12:47:25.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 1 librados: setting wanted keys 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 1 librados: calling monclient init 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 10 monclient: init 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 10 monclient: _add_conns ranks=[0,6,4,8,1,5,3,7,2] 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 10 monclient(hunting): picked mon.noname-a con 0x7f355006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 10 monclient(hunting): picked mon.noname-d con 0x7f35500a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 10 monclient(hunting): picked mon.noname-c con 0x7f35500a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35501349d0 con 0x7f355006e7b0 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35500a2e00 con 0x7f35500a5d50 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35500a1bc0 con 0x7f35500a5920 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.465+0000 7f3557314640 10 monclient(hunting): authenticate will time out at 1239.553967s 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.466+0000 7f35548af640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f35500a5920 0x7f35501d3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:39114/0 (socket says 172.21.15.26:39114) 2024-08-19T12:47:25.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.466+0000 7f35558b1640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f35500a5d50 0x7f35501d69a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:42066/0 (socket says 172.21.15.26:42066) 2024-08-19T12:47:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.466+0000 7f35548af640 1 -- 172.21.15.26:0/3312461026 learned_addr learned my addr 172.21.15.26:0/3312461026 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:25.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f35548af640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.26:6792/0 conn(0x7f35500a5920 legacy=0x7f35501d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f35558b1640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.26:6791/0 conn(0x7f35500a5d50 legacy=0x7f35501d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 client.?.objecter ms_handle_connect 0x7f35500a5920 2024-08-19T12:47:25.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 client.?.objecter ms_handle_connect 0x7f35500a5d50 2024-08-19T12:47:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 client.?.objecter ms_handle_connect 0x7f355006e7b0 2024-08-19T12:47:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f35550b0640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.26:6789/0 conn(0x7f355006e7b0 legacy=0x7f35500a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1658228688 0 0) 0x7f35500a2e00 con 0x7f35500a5d50 2024-08-19T12:47:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): my global_id is 26977 2024-08-19T12:47:25.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3534004eb0 con 0x7f35500a5d50 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3086308529 0 0) 0x7f35501349d0 con 0x7f355006e7b0 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): my global_id is 144981 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35500a2e00 con 0x7f355006e7b0 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1452314894 0 0) 0x7f35500a1bc0 con 0x7f35500a5920 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): my global_id is 164328 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35501349d0 con 0x7f35500a5920 2024-08-19T12:47:25.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3924382587 0 0) 0x7f3534004eb0 con 0x7f35500a5d50 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35500a1bc0 con 0x7f35500a5d50 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3548002030 con 0x7f35500a5d50 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): dump: 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.484 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1423461727 0 0) 0x7f35500a2e00 con 0x7f355006e7b0 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3534004eb0 con 0x7f355006e7b0 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 88776732 0 0) 0x7f35501349d0 con 0x7f35500a5920 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35500a2e00 con 0x7f35500a5920 2024-08-19T12:47:25.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3540003610 con 0x7f35500a5920 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): dump: 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.486 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f354c004d90 con 0x7f355006e7b0 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient(hunting): dump: 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.487 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1528421207 0 0) 0x7f35500a1bc0 con 0x7f35500a5d50 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.26:6792/0 conn(0x7f35500a5920 legacy=0x7f35501d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.26:6789/0 conn(0x7f355006e7b0 legacy=0x7f35500a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 monclient: found mon.c 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35500a2b20 con 0x7f35500a5d50 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient: _finish_auth 0 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3545ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3557314640 5 monclient: authenticate success, global_id 26977 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3557314640 10 monclient: _renew_subs 2024-08-19T12:47:25.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3557314640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:47:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.468+0000 7f3557314640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f35500a2810 con 0x7f35500a5d50 2024-08-19T12:47:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.469+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3548003d50 con 0x7f35500a5d50 2024-08-19T12:47:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.469+0000 7f3545ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.469+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3548004280 con 0x7f35500a5d50 2024-08-19T12:47:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.469+0000 7f3545ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.469+0000 7f3545ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.469+0000 7f3545ffb640 10 monclient: dump: 2024-08-19T12:47:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.490 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.469+0000 7f35477fe640 4 set_mon_vals no callback set 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.469+0000 7f3557314640 10 client.26977.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:25.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.469+0000 7f3557314640 10 monclient: _renew_subs 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.469+0000 7f3557314640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.469+0000 7f3557314640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f35500a51e0 con 0x7f35500a5d50 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.469+0000 7f3557314640 1 librados: init done 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.470+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3548004420 con 0x7f35500a5d50 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.470+0000 7f3557314640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.470+0000 7f3557314640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3550136740 con 0x7f3550132a20 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.470+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(356..356 src has 1..356) ==== 5346+0+0 (unknown 2942184895 0 0) 0x7f354804e2b0 con 0x7f35500a5d50 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.470+0000 7f3545ffb640 10 client.26977.objecter ms_dispatch 0x7f3550000b90 osd_map(356..356 src has 1..356) 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.470+0000 7f3545ffb640 3 client.26977.objecter handle_osd_map got epochs [356,356] > 0 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.470+0000 7f3545ffb640 3 client.26977.objecter handle_osd_map decoding full epoch 356 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.470+0000 7f35548af640 10 monclient: get_auth_request con 0x7f35340419f0 auth_method 0 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.470+0000 7f3545ffb640 20 client.26977.objecter dump_active .. 0 homeless 2024-08-19T12:47:25.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.472+0000 7f3545ffb640 10 client.26977.objecter ms_handle_connect 0x7f3550132a20 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.472+0000 7f35550b0640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.104:6790/0 conn(0x7f3550132a20 legacy=0x7f35501253f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.472+0000 7f3545ffb640 10 client.26977.objecter resend_mon_ops 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.472+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2714452890 0 0) 0x7f3534004eb0 con 0x7f3550132a20 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.472+0000 7f3545ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.472+0000 7f3545ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.472+0000 7f3545ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.472+0000 7f3545ffb640 10 monclient(hunting): my global_id is 154461 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.472+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35500a1bc0 con 0x7f3550132a20 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.472+0000 7f3545ffb640 10 client.26977.objecter ms_handle_connect 0x7f35340419f0 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2916754710 0 0) 0x7f35500a1bc0 con 0x7f3550132a20 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3534004eb0 con 0x7f3550132a20 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f354c001580 con 0x7f3550132a20 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 10 monclient: dump: 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.494 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1297921257 0 0) 0x7f3534004eb0 con 0x7f3550132a20 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3550077810 con 0x7f3550132a20 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f354c0048a0 con 0x7f3550132a20 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.104:6790/0 conn(0x7f3550132a20 legacy=0x7f35501253f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.473+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.104:6790/0 conn(0x7f3550132a20 legacy=0x7f35501253f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.479+0000 7f3557314640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:25.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.479+0000 7f3557314640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f35501345a0 con 0x7f355002e1d0 2024-08-19T12:47:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 10 client.26977.objecter ms_handle_connect 0x7f355002e1d0 2024-08-19T12:47:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 10 client.26977.objecter resend_mon_ops 2024-08-19T12:47:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f35550b0640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.104:6790/0 conn(0x7f355002e1d0 legacy=0x7f3550125b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 514594169 0 0) 0x7f35501345a0 con 0x7f355002e1d0 2024-08-19T12:47:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 10 monclient(hunting): my global_id is 154470 2024-08-19T12:47:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3534004eb0 con 0x7f355002e1d0 2024-08-19T12:47:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 140388504 0 0) 0x7f3534004eb0 con 0x7f355002e1d0 2024-08-19T12:47:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35501345a0 con 0x7f355002e1d0 2024-08-19T12:47:25.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f354c005a00 con 0x7f355002e1d0 2024-08-19T12:47:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:47:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 10 monclient: dump: 2024-08-19T12:47:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.498 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2326872680 0 0) 0x7f35501345a0 con 0x7f355002e1d0 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.482+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3550132e00 con 0x7f355002e1d0 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 1174258989) 0x7f354c005d10 con 0x7f355002e1d0 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3545ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3545ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.104:6790/0 conn(0x7f355002e1d0 legacy=0x7f3550125b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3545ffb640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.104:6790/0 conn(0x7f355002e1d0 legacy=0x7f3550125b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.500 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:25.500 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:25.500 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:25.500 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:25.500 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-08-19T12:47:25.500 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 157213, 2024-08-19T12:47:25.500 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:25.500 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:25.500 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:25.500 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.501 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:25.502 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:25.503 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:25.504 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.505 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:25.506 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.507 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:25.509 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:25.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.511 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.513 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.514 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:25.515 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.515 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.515 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.515 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:25.515 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:25.515 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.515 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.515 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.516 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:25.516 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:25.516 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.516 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.516 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:25.516 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.518 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:25.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.520 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.520 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:25.520 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:25.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 10 librados: watch_flush enter 2024-08-19T12:47:25.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 10 librados: watch_flush exit 2024-08-19T12:47:25.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 20 client.26977.objecter shutdown clearing up homeless session... 2024-08-19T12:47:25.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 10 client.26977.objecter successfully canceled tick 2024-08-19T12:47:25.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.104:6800/899887506 conn(0x7f35340419f0 legacy=0x7f3534043eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 10 monclient: shutdown 2024-08-19T12:47:25.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:25.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 1 -- 172.21.15.26:0/3312461026 >> v1:172.21.15.26:6791/0 conn(0x7f35500a5d50 legacy=0x7f35501d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f35558b1640 1 -- 172.21.15.26:0/3312461026 reap_dead start 2024-08-19T12:47:25.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f35558b1640 1 -- 172.21.15.26:0/3312461026 reap_dead start 2024-08-19T12:47:25.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 1 -- 172.21.15.26:0/3312461026 shutdown_connections 2024-08-19T12:47:25.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 1 -- 172.21.15.26:0/3312461026 >> 172.21.15.26:0/3312461026 conn(0x7f35500a8740 msgr2=0x7f35500aa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:25.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 1 -- 172.21.15.26:0/3312461026 shutdown_connections 2024-08-19T12:47:25.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 1 -- 172.21.15.26:0/3312461026 wait complete. 2024-08-19T12:47:25.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.483+0000 7f3557314640 1 librados: shutdown 2024-08-19T12:47:25.523 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:47:25.523 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-08-19T12:47:25.523 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:47:25.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.620+0000 7fac71dd9640 10 monclient: get_monmap_and_config 2024-08-19T12:47:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.620+0000 7fac71dd9640 10 monclient: build_initial_monmap 2024-08-19T12:47:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.620+0000 7fac71dd9640 10 monclient: monmap: 2024-08-19T12:47:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:25.622704+0000 2024-08-19T12:47:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:25.622704+0000 2024-08-19T12:47:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:25.623 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:25.624 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:25.624 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.621+0000 7fac71dd9640 1 Processor -- start 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.621+0000 7fac71dd9640 1 -- start start 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.621+0000 7fac71dd9640 10 monclient: init 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac71dd9640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac71dd9640 10 monclient: _add_conns ranks=[8,7,0,1,3,5,4,6,2] 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac71dd9640 10 monclient(hunting): picked mon.noname-e con 0x7fac6c06e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac71dd9640 10 monclient(hunting): picked mon.noname-i con 0x7fac6c0a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac71dd9640 10 monclient(hunting): picked mon.noname-a con 0x7fac6c0a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac71dd9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac6c136790 con 0x7fac6c0a5d50 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac71dd9640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac6c09ddf0 con 0x7fac6c0a5920 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac71dd9640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac6c09e0d0 con 0x7fac6c06e7b0 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac71dd9640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac71dd9640 10 monclient(hunting): authenticate will time out at 1239.710165s 2024-08-19T12:47:25.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac6b7fe640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7fac6c06e7b0 legacy=0x7fac6c136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:47:25.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac6bfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fac6c0a5d50 0x7fac6c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50346/0 (socket says 172.21.15.26:50346) 2024-08-19T12:47:25.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.622+0000 7fac6bfff640 1 -- 172.21.15.26:0/1059946280 learned_addr learned my addr 172.21.15.26:0/1059946280 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:25.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6bfff640 1 -- 172.21.15.26:0/1059946280 >> v1:172.21.15.26:6789/0 conn(0x7fac6c0a5d50 legacy=0x7fac6c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6affd640 1 -- 172.21.15.26:0/1059946280 >> v1:172.21.15.104:6792/0 conn(0x7fac6c0a5920 legacy=0x7fac6c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 1 -- 172.21.15.26:0/1059946280 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2375987089 0 0) 0x7fac6c136790 con 0x7fac6c0a5d50 2024-08-19T12:47:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 10 monclient(hunting): my global_id is 144990 2024-08-19T12:47:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 1 -- 172.21.15.26:0/1059946280 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac54003520 con 0x7fac6c0a5d50 2024-08-19T12:47:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 1 -- 172.21.15.26:0/1059946280 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3316842350 0 0) 0x7fac54003520 con 0x7fac6c0a5d50 2024-08-19T12:47:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 1 -- 172.21.15.26:0/1059946280 >> v1:172.21.15.26:6793/0 conn(0x7fac6c06e7b0 legacy=0x7fac6c136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:47:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 1 -- 172.21.15.26:0/1059946280 >> v1:172.21.15.104:6792/0 conn(0x7fac6c0a5920 legacy=0x7fac6c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 1 monclient: found mon.noname-a 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 1 -- 172.21.15.26:0/1059946280 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac6c09e3b0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 1 -- 172.21.15.26:0/1059946280 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fac64001e60 con 0x7fac6c0a5d50 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac6a7fc640 10 monclient: dump: 2024-08-19T12:47:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.633 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.634 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac71dd9640 5 monclient: authenticate success, global_id 144990 2024-08-19T12:47:25.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.624+0000 7fac71dd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:25.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac6a7fc640 1 -- 172.21.15.26:0/1059946280 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fac64004c20 con 0x7fac6c0a5d50 2024-08-19T12:47:25.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac6a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:25.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac6a7fc640 1 -- 172.21.15.26:0/1059946280 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fac64005190 con 0x7fac6c0a5d50 2024-08-19T12:47:25.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac71dd9640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:25.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac71dd9640 4 set_mon_vals no callback set 2024-08-19T12:47:25.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac6a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:47:25.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac6a7fc640 10 monclient: dump: 2024-08-19T12:47:25.634 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.635 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac71dd9640 10 monclient: shutdown 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac71dd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac71dd9640 1 -- 172.21.15.26:0/1059946280 >> v1:172.21.15.26:6789/0 conn(0x7fac6c0a5d50 legacy=0x7fac6c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac71dd9640 1 -- 172.21.15.26:0/1059946280 shutdown_connections 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac71dd9640 1 -- 172.21.15.26:0/1059946280 >> 172.21.15.26:0/1059946280 conn(0x7fac6c0a8740 msgr2=0x7fac6c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac71dd9640 1 -- 172.21.15.26:0/1059946280 shutdown_connections 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.625+0000 7fac71dd9640 1 -- 172.21.15.26:0/1059946280 wait complete. 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.626+0000 7fac71dd9640 10 monclient: build_initial_monmap 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.626+0000 7fac71dd9640 10 monclient: monmap: 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:25.636 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.626+0000 7fac71dd9640 1 librados: starting msgr at 2024-08-19T12:47:25.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.626+0000 7fac71dd9640 1 librados: starting objecter 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.626+0000 7fac71dd9640 1 Processor -- start 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.626+0000 7fac71dd9640 1 -- start start 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.626+0000 7fac71dd9640 1 librados: setting wanted keys 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.626+0000 7fac71dd9640 1 librados: calling monclient init 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.626+0000 7fac71dd9640 10 monclient: init 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac71dd9640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac71dd9640 10 monclient: _add_conns ranks=[1,8,3,4,6,7,0,5,2] 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac71dd9640 10 monclient(hunting): picked mon.noname-f con 0x7fac6c06e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac71dd9640 10 monclient(hunting): picked mon.noname-e con 0x7fac6c0a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac71dd9640 10 monclient(hunting): picked mon.noname-g con 0x7fac6c0a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac71dd9640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac6c13b0b0 con 0x7fac6c06e7b0 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac6affd640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7fac6c0a5920 legacy=0x7fac6c137b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:47:25.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac71dd9640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac6c1429a0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac71dd9640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac6c138250 con 0x7fac6c0a5920 2024-08-19T12:47:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac71dd9640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac71dd9640 10 monclient(hunting): authenticate will time out at 1239.715257s 2024-08-19T12:47:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac6b7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fac6c06e7b0 0x7fac6c137430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:44228/0 (socket says 172.21.15.26:44228) 2024-08-19T12:47:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac6bfff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fac6c0a5d50 0x7fac6c1d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:51492/0 (socket says 172.21.15.26:51492) 2024-08-19T12:47:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.627+0000 7fac6b7fe640 1 -- 172.21.15.26:0/1442110284 learned_addr learned my addr 172.21.15.26:0/1442110284 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac6b7fe640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6789/0 conn(0x7fac6c06e7b0 legacy=0x7fac6c137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 10 client.?.objecter ms_handle_connect 0x7fac6c06e7b0 2024-08-19T12:47:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac6bfff640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6790/0 conn(0x7fac6c0a5d50 legacy=0x7fac6c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 10 client.?.objecter ms_handle_connect 0x7fac6c0a5d50 2024-08-19T12:47:25.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2142305128 0 0) 0x7fac6c13b0b0 con 0x7fac6c06e7b0 2024-08-19T12:47:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 10 monclient(hunting): my global_id is 27478 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac58004ee0 con 0x7fac6c06e7b0 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3743499962 0 0) 0x7fac6c1429a0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 10 monclient(hunting): my global_id is 154479 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.629+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac6c13b0b0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2663091272 0 0) 0x7fac6c13b0b0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac6c1429a0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fac640020b0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:47:25.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient(hunting): dump: 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2897193613 0 0) 0x7fac58004ee0 con 0x7fac6c06e7b0 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac6c13b0b0 con 0x7fac6c06e7b0 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fac5c004620 con 0x7fac6c06e7b0 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient(hunting): dump: 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.648 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4125689886 0 0) 0x7fac6c1429a0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.26:6793/0 conn(0x7fac6c0a5920 legacy=0x7fac6c137b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6789/0 conn(0x7fac6c06e7b0 legacy=0x7fac6c137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient: _finish_hunting 0 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 monclient: found mon.g 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac6c13add0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient: _finish_auth 0 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient: _check_auth_tickets 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac71dd9640 5 monclient: authenticate success, global_id 154479 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac71dd9640 10 monclient: _renew_subs 2024-08-19T12:47:25.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac71dd9640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac71dd9640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fac6c1398b0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac71dd9640 10 client.154479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac71dd9640 10 monclient: _renew_subs 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac71dd9640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac71dd9640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fac6c1d79b0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac71dd9640 1 librados: init done 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.3 v1:172.21.15.104:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fac64004c20 con 0x7fac6c0a5d50 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.3 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fac64001900 con 0x7fac6c0a5d50 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac69ffb640 4 set_mon_vals no callback set 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:47:25.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.630+0000 7fac53fff640 10 monclient: dump: 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.652 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.631+0000 7fac71dd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.631+0000 7fac71dd9640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fac4c0065c0 con 0x7fac4c002860 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.631+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.3 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fac640119c0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.631+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.3 v1:172.21.15.104:6790/0 8 ==== osd_map(356..356 src has 1..356) ==== 5346+0+0 (unknown 2942184895 0 0) 0x7fac6404e3d0 con 0x7fac6c0a5d50 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.631+0000 7fac53fff640 10 client.154479.objecter ms_dispatch 0x7fac6c000b90 osd_map(356..356 src has 1..356) 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.631+0000 7fac53fff640 3 client.154479.objecter handle_osd_map got epochs [356,356] > 0 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.631+0000 7fac53fff640 3 client.154479.objecter handle_osd_map decoding full epoch 356 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.631+0000 7fac53fff640 20 client.154479.objecter dump_active .. 0 homeless 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.632+0000 7fac6affd640 10 monclient: get_auth_request con 0x7fac58041690 auth_method 0 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.633+0000 7fac53fff640 10 client.154479.objecter ms_handle_connect 0x7fac4c002860 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.633+0000 7fac53fff640 10 client.154479.objecter resend_mon_ops 2024-08-19T12:47:25.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.633+0000 7fac6b7fe640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac4c002860 legacy=0x7fac4c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.633+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3796786527 0 0) 0x7fac4c0065c0 con 0x7fac4c002860 2024-08-19T12:47:25.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.633+0000 7fac53fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.633+0000 7fac53fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.633+0000 7fac53fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.633+0000 7fac53fff640 10 monclient(hunting): my global_id is 26681 2024-08-19T12:47:25.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.633+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac6c1429a0 con 0x7fac4c002860 2024-08-19T12:47:25.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 594113136 0 0) 0x7fac6c1429a0 con 0x7fac4c002860 2024-08-19T12:47:25.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac4c0065c0 con 0x7fac4c002860 2024-08-19T12:47:25.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fac5c004e70 con 0x7fac4c002860 2024-08-19T12:47:25.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac53fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:47:25.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac53fff640 10 monclient: dump: 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.655 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3546420942 0 0) 0x7fac4c0065c0 con 0x7fac4c002860 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac4c00a160 con 0x7fac4c002860 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac53fff640 10 client.154479.objecter ms_handle_connect 0x7fac58041690 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac6b7fe640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac4c002860 legacy=0x7fac4c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac6b7fe640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac4c002860 legacy=0x7fac4c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac6b7fe640 1 --1- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac4c002860 0x7fac4c004d00 :-1 s=OPENED pgs=3363 cs=1 l=1).handle_message read tag failed 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac6b7fe640 1 --1- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac4c002860 0x7fac4c004d00 :-1 s=OPENED pgs=3363 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac4c002860 legacy=0x7fac4c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac4c002860 legacy=0x7fac4c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.634+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fac4c0065c0 con 0x7fac5804c810 2024-08-19T12:47:25.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.636+0000 7fac6b7fe640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac5804c810 legacy=0x7fac5804ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.636+0000 7fac53fff640 10 client.154479.objecter ms_handle_connect 0x7fac5804c810 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.636+0000 7fac53fff640 10 client.154479.objecter resend_mon_ops 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.636+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1433371036 0 0) 0x7fac4c0065c0 con 0x7fac5804c810 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.636+0000 7fac53fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.636+0000 7fac53fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.636+0000 7fac53fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.636+0000 7fac53fff640 10 monclient(hunting): my global_id is 26690 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.636+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac6c1429a0 con 0x7fac5804c810 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1933481638 0 0) 0x7fac6c1429a0 con 0x7fac5804c810 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac4c0065c0 con 0x7fac5804c810 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fac5c005890 con 0x7fac5804c810 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:47:25.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 10 monclient: dump: 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.658 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.659 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.659 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.659 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2809282429 0 0) 0x7fac4c0065c0 con 0x7fac5804c810 2024-08-19T12:47:25.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac640020b0 con 0x7fac5804c810 2024-08-19T12:47:25.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fac5c006e10 con 0x7fac5804c810 2024-08-19T12:47:25.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:25.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:25.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac5804c810 legacy=0x7fac5804ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.637+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac5804c810 legacy=0x7fac5804ae50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.643+0000 7fac71dd9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:25.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.643+0000 7fac71dd9640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fac4c00c4f0 con 0x7fac4c0090e0 2024-08-19T12:47:25.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac6b7fe640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac4c0090e0 legacy=0x7fac4c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 10 client.154479.objecter ms_handle_connect 0x7fac4c0090e0 2024-08-19T12:47:25.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 10 client.154479.objecter resend_mon_ops 2024-08-19T12:47:25.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1662689135 0 0) 0x7fac4c00c4f0 con 0x7fac4c0090e0 2024-08-19T12:47:25.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 10 monclient(hunting): my global_id is 26699 2024-08-19T12:47:25.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac4c0065c0 con 0x7fac4c0090e0 2024-08-19T12:47:25.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 894140105 0 0) 0x7fac4c0065c0 con 0x7fac4c0090e0 2024-08-19T12:47:25.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac4c00c4f0 con 0x7fac4c0090e0 2024-08-19T12:47:25.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fac5c005e40 con 0x7fac4c0090e0 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 10 monclient: dump: 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.662 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4150684125 0 0) 0x7fac4c00c4f0 con 0x7fac4c0090e0 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.646+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac4c009c90 con 0x7fac4c0090e0 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 1760078660) 0x7fac5c009f40 con 0x7fac4c0090e0 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac53fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac53fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac4c0090e0 legacy=0x7fac4c009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac53fff640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6791/0 conn(0x7fac4c0090e0 legacy=0x7fac4c009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac6bfff640 1 -- 172.21.15.26:0/1442110284 reap_dead start 2024-08-19T12:47:25.664 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-08-19T12:47:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 558671, 2024-08-19T12:47:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:25.664 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:25.665 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.666 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.667 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:25.668 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.669 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.671 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.672 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.674 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:25.676 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:25.677 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:25.678 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.679 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:25.680 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.682 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:25.682 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:25.682 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.682 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.682 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.682 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.682 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:25.682 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.684 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:25.684 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:25.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac71dd9640 10 librados: watch_flush enter 2024-08-19T12:47:25.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac71dd9640 10 librados: watch_flush exit 2024-08-19T12:47:25.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac71dd9640 20 client.154479.objecter shutdown clearing up homeless session... 2024-08-19T12:47:25.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac71dd9640 10 client.154479.objecter successfully canceled tick 2024-08-19T12:47:25.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac71dd9640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6800/899887506 conn(0x7fac58041690 legacy=0x7fac58043b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac71dd9640 10 monclient: shutdown 2024-08-19T12:47:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac71dd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac71dd9640 1 -- 172.21.15.26:0/1442110284 >> v1:172.21.15.104:6790/0 conn(0x7fac6c0a5d50 legacy=0x7fac6c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac71dd9640 1 -- 172.21.15.26:0/1442110284 shutdown_connections 2024-08-19T12:47:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac71dd9640 1 -- 172.21.15.26:0/1442110284 >> 172.21.15.26:0/1442110284 conn(0x7fac6c0a8740 msgr2=0x7fac6c0aaff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac71dd9640 1 -- 172.21.15.26:0/1442110284 shutdown_connections 2024-08-19T12:47:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.647+0000 7fac71dd9640 1 -- 172.21.15.26:0/1442110284 wait complete. 2024-08-19T12:47:25.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.648+0000 7fac71dd9640 1 librados: shutdown 2024-08-19T12:47:25.686 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:47:25.687 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-08-19T12:47:25.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:47:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.785+0000 7f947c06e640 10 monclient: get_monmap_and_config 2024-08-19T12:47:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.785+0000 7f947c06e640 10 monclient: build_initial_monmap 2024-08-19T12:47:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.785+0000 7f947c06e640 10 monclient: monmap: 2024-08-19T12:47:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:47:25.787004+0000 2024-08-19T12:47:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:47:25.787004+0000 2024-08-19T12:47:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:25.787 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:25.788 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:25.788 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.785+0000 7f947c06e640 1 Processor -- start 2024-08-19T12:47:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.785+0000 7f947c06e640 1 -- start start 2024-08-19T12:47:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.785+0000 7f947c06e640 10 monclient: init 2024-08-19T12:47:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.786+0000 7f947c06e640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.786+0000 7f947c06e640 10 monclient: _add_conns ranks=[5,4,7,6,0,3,2,8,1] 2024-08-19T12:47:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.786+0000 7f947c06e640 10 monclient(hunting): picked mon.noname-h con 0x7f947406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.786+0000 7f947c06e640 10 monclient(hunting): picked mon.noname-c con 0x7f94740a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:47:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.786+0000 7f947c06e640 10 monclient(hunting): picked mon.noname-i con 0x7f94740a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:47:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.786+0000 7f947c06e640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9474136790 con 0x7f94740a5920 2024-08-19T12:47:25.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.786+0000 7f947c06e640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f947409ddf0 con 0x7f947406e7b0 2024-08-19T12:47:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.786+0000 7f947c06e640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f947409e0d0 con 0x7f94740a5d50 2024-08-19T12:47:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.786+0000 7f947c06e640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.786+0000 7f947c06e640 10 monclient(hunting): authenticate will time out at 1239.874480s 2024-08-19T12:47:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.786+0000 7f9479609640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f94740a5920 0x7f947413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:42082/0 (socket says 172.21.15.26:42082) 2024-08-19T12:47:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.786+0000 7f9479609640 1 -- 172.21.15.26:0/2137037015 learned_addr learned my addr 172.21.15.26:0/2137037015 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.788+0000 7f9479609640 1 -- 172.21.15.26:0/2137037015 >> v1:172.21.15.26:6791/0 conn(0x7f94740a5920 legacy=0x7f947413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.788+0000 7f947a60b640 1 -- 172.21.15.26:0/2137037015 >> v1:172.21.15.104:6792/0 conn(0x7f94740a5d50 legacy=0x7f9474141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.788+0000 7f9479e0a640 1 -- 172.21.15.26:0/2137037015 >> v1:172.21.15.104:6791/0 conn(0x7f947406e7b0 legacy=0x7f9474136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.788+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2781187735 0 0) 0x7f9474136790 con 0x7f94740a5920 2024-08-19T12:47:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.788+0000 7f9478e08640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.788+0000 7f9478e08640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.788+0000 7f9478e08640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.788+0000 7f9478e08640 10 monclient(hunting): my global_id is 26986 2024-08-19T12:47:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.788+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f945c003520 con 0x7f94740a5920 2024-08-19T12:47:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2179061767 0 0) 0x7f9464001820 con 0x7f947406e7b0 2024-08-19T12:47:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient(hunting): my global_id is 26708 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9474136790 con 0x7f947406e7b0 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4268418964 0 0) 0x7f947409e0d0 con 0x7f94740a5d50 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient(hunting): my global_id is 135583 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9464001820 con 0x7f94740a5d50 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3951956845 0 0) 0x7f945c003520 con 0x7f94740a5920 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 >> v1:172.21.15.104:6792/0 conn(0x7f94740a5d50 legacy=0x7f9474141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 >> v1:172.21.15.104:6791/0 conn(0x7f947406e7b0 legacy=0x7f9474136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient: _finish_hunting 0 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 1 monclient: found mon.noname-c 2024-08-19T12:47:25.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:47:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947409e3b0 con 0x7f94740a5920 2024-08-19T12:47:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient: _finish_auth 0 2024-08-19T12:47:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient: _check_auth_tickets 2024-08-19T12:47:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9468001ae0 con 0x7f94740a5920 2024-08-19T12:47:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f947c06e640 5 monclient: authenticate success, global_id 26986 2024-08-19T12:47:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f947c06e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:47:25.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient: dump: 2024-08-19T12:47:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.799 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.800 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f947c06e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:47:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f94680048a0 con 0x7f94740a5920 2024-08-19T12:47:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 1 -- 172.21.15.26:0/2137037015 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9468004e10 con 0x7f94740a5920 2024-08-19T12:47:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:47:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f9478e08640 10 monclient: dump: 2024-08-19T12:47:25.800 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.801 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.801 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.801 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.801 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.801 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.801 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.801 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.801 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.801 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.801 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.802 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.802 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.802 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.802 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.802 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f947c06e640 10 monclient: get_monmap_and_config success 2024-08-19T12:47:25.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f947c06e640 4 set_mon_vals no callback set 2024-08-19T12:47:25.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f947c06e640 10 monclient: shutdown 2024-08-19T12:47:25.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f947c06e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:25.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f947c06e640 1 -- 172.21.15.26:0/2137037015 >> v1:172.21.15.26:6791/0 conn(0x7f94740a5920 legacy=0x7f947413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f947c06e640 1 -- 172.21.15.26:0/2137037015 shutdown_connections 2024-08-19T12:47:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f947c06e640 1 -- 172.21.15.26:0/2137037015 >> 172.21.15.26:0/2137037015 conn(0x7f94740a8740 msgr2=0x7f94740a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.789+0000 7f947c06e640 1 -- 172.21.15.26:0/2137037015 shutdown_connections 2024-08-19T12:47:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.790+0000 7f947c06e640 1 -- 172.21.15.26:0/2137037015 wait complete. 2024-08-19T12:47:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.790+0000 7f947c06e640 10 monclient: build_initial_monmap 2024-08-19T12:47:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.790+0000 7f947c06e640 10 monclient: monmap: 2024-08-19T12:47:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:47:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:47:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:47:25.803 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:47:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:47:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:47:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:47:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:47:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:47:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:47:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:47:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:47:25.804 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:47:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:47:25.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.790+0000 7f947c06e640 1 librados: starting msgr at 2024-08-19T12:47:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.790+0000 7f947c06e640 1 librados: starting objecter 2024-08-19T12:47:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.790+0000 7f947c06e640 1 Processor -- start 2024-08-19T12:47:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.790+0000 7f947c06e640 1 -- start start 2024-08-19T12:47:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.790+0000 7f947c06e640 1 librados: setting wanted keys 2024-08-19T12:47:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.790+0000 7f947c06e640 1 librados: calling monclient init 2024-08-19T12:47:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.790+0000 7f947c06e640 10 monclient: init 2024-08-19T12:47:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.791+0000 7f947c06e640 10 monclient: _reopen_session rank -1 2024-08-19T12:47:25.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.791+0000 7f947c06e640 10 monclient: _add_conns ranks=[2,3,5,7,8,6,4,0,1] 2024-08-19T12:47:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.791+0000 7f947c06e640 10 monclient(hunting): picked mon.noname-b con 0x7f947406e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:47:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.791+0000 7f947c06e640 10 monclient(hunting): picked mon.noname-g con 0x7f94740a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:47:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.791+0000 7f947c06e640 10 monclient(hunting): picked mon.noname-h con 0x7f94740a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:47:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.791+0000 7f947c06e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9474138fd0 con 0x7f947406e7b0 2024-08-19T12:47:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.791+0000 7f947c06e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9474139b30 con 0x7f94740a5920 2024-08-19T12:47:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.791+0000 7f947c06e640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f947413aa90 con 0x7f94740a5d50 2024-08-19T12:47:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.791+0000 7f947c06e640 10 monclient(hunting): _renew_subs 2024-08-19T12:47:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.791+0000 7f947c06e640 10 monclient(hunting): authenticate will time out at 1239.879340s 2024-08-19T12:47:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.791+0000 7f9479e0a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f947406e7b0 0x7f9474137920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33024/0 (socket says 172.21.15.26:33024) 2024-08-19T12:47:25.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.791+0000 7f9479e0a640 1 -- 172.21.15.26:0/4236018455 learned_addr learned my addr 172.21.15.26:0/4236018455 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:47:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f9479e0a640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.26:6790/0 conn(0x7f947406e7b0 legacy=0x7f9474137920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 client.?.objecter ms_handle_connect 0x7f947406e7b0 2024-08-19T12:47:25.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f9479609640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.104:6790/0 conn(0x7f94740a5920 legacy=0x7f9474138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 client.?.objecter ms_handle_connect 0x7f94740a5920 2024-08-19T12:47:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 client.?.objecter ms_handle_connect 0x7f94740a5d50 2024-08-19T12:47:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:47:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f947a60b640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.104:6791/0 conn(0x7f94740a5d50 legacy=0x7f94741d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 192307565 0 0) 0x7f9474138fd0 con 0x7f947406e7b0 2024-08-19T12:47:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 monclient(hunting): my global_id is 26660 2024-08-19T12:47:25.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9460004eb0 con 0x7f947406e7b0 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1077119375 0 0) 0x7f9474139b30 con 0x7f94740a5920 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 monclient(hunting): my global_id is 154488 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9474138fd0 con 0x7f94740a5920 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2484900666 0 0) 0x7f947413aa90 con 0x7f94740a5d50 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 10 monclient(hunting): my global_id is 26717 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.793+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9474139b30 con 0x7f94740a5d50 2024-08-19T12:47:25.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3614524021 0 0) 0x7f9460004eb0 con 0x7f947406e7b0 2024-08-19T12:47:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f947413aa90 con 0x7f947406e7b0 2024-08-19T12:47:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9464003cd0 con 0x7f947406e7b0 2024-08-19T12:47:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:47:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient(hunting): dump: 2024-08-19T12:47:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.816 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.817 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.817 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1327493689 0 0) 0x7f9474138fd0 con 0x7f94740a5920 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9460004eb0 con 0x7f94740a5920 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9468004d90 con 0x7f94740a5920 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient(hunting): dump: 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.818 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3425488319 0 0) 0x7f9474139b30 con 0x7f94740a5d50 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9474138fd0 con 0x7f94740a5d50 2024-08-19T12:47:25.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f946c001cb0 con 0x7f94740a5d50 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient(hunting): dump: 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.820 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3638418182 0 0) 0x7f947413aa90 con 0x7f947406e7b0 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.104:6791/0 conn(0x7f94740a5d50 legacy=0x7f94741d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.104:6790/0 conn(0x7f94740a5920 legacy=0x7f9474138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient: _finish_hunting 0 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 monclient: found mon.b 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947413ad70 con 0x7f947406e7b0 2024-08-19T12:47:25.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient: _finish_auth 0 2024-08-19T12:47:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient: _check_auth_tickets 2024-08-19T12:47:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:47:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f947c06e640 5 monclient: authenticate success, global_id 26660 2024-08-19T12:47:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f947c06e640 10 monclient: _renew_subs 2024-08-19T12:47:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f947c06e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:47:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f947c06e640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f947413a6f0 con 0x7f947406e7b0 2024-08-19T12:47:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f947c06e640 10 client.26660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:47:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f947c06e640 10 monclient: _renew_subs 2024-08-19T12:47:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f947c06e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:47:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f947c06e640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9474138740 con 0x7f947406e7b0 2024-08-19T12:47:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f947c06e640 1 librados: init done 2024-08-19T12:47:25.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9464002020 con 0x7f947406e7b0 2024-08-19T12:47:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:47:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9464004520 con 0x7f947406e7b0 2024-08-19T12:47:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f9473fff640 4 set_mon_vals no callback set 2024-08-19T12:47:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:47:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.794+0000 7f94727fc640 10 monclient: dump: 2024-08-19T12:47:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.823 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.795+0000 7f947c06e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.795+0000 7f947c06e640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f94540065c0 con 0x7f9454002860 2024-08-19T12:47:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.795+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f9464011e00 con 0x7f947406e7b0 2024-08-19T12:47:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.795+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(356..356 src has 1..356) ==== 5346+0+0 (unknown 2942184895 0 0) 0x7f946404e830 con 0x7f947406e7b0 2024-08-19T12:47:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.795+0000 7f94727fc640 10 client.26660.objecter ms_dispatch 0x7f9474000b90 osd_map(356..356 src has 1..356) 2024-08-19T12:47:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.795+0000 7f94727fc640 3 client.26660.objecter handle_osd_map got epochs [356,356] > 0 2024-08-19T12:47:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.795+0000 7f94727fc640 3 client.26660.objecter handle_osd_map decoding full epoch 356 2024-08-19T12:47:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.795+0000 7f94727fc640 20 client.26660.objecter dump_active .. 0 homeless 2024-08-19T12:47:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.795+0000 7f947a60b640 10 monclient: get_auth_request con 0x7f94600418b0 auth_method 0 2024-08-19T12:47:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f9479609640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.104:6792/0 conn(0x7f9454002860 legacy=0x7f9454004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 10 client.26660.objecter ms_handle_connect 0x7f9454002860 2024-08-19T12:47:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 10 client.26660.objecter resend_mon_ops 2024-08-19T12:47:25.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3638866999 0 0) 0x7f94540065c0 con 0x7f9454002860 2024-08-19T12:47:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 10 monclient(hunting): my global_id is 135592 2024-08-19T12:47:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f947413aa90 con 0x7f9454002860 2024-08-19T12:47:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3188608308 0 0) 0x7f947413aa90 con 0x7f9454002860 2024-08-19T12:47:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94540065c0 con 0x7f9454002860 2024-08-19T12:47:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9468001580 con 0x7f9454002860 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 10 monclient: dump: 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.827 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.797+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 30429877 0 0) 0x7f94540065c0 con 0x7f9454002860 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.798+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f945400a160 con 0x7f9454002860 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.798+0000 7f94727fc640 10 client.26660.objecter ms_handle_connect 0x7f94600418b0 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.798+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f94680048a0 con 0x7f9454002860 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.798+0000 7f94727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.798+0000 7f94727fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.798+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.104:6792/0 conn(0x7f9454002860 legacy=0x7f9454004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.798+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.104:6792/0 conn(0x7f9454002860 legacy=0x7f9454004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.804+0000 7f947c06e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:47:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.804+0000 7f947c06e640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9454009e70 con 0x7f9454009130 2024-08-19T12:47:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f9479609640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.104:6792/0 conn(0x7f9454009130 legacy=0x7f94540095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:47:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 10 client.26660.objecter ms_handle_connect 0x7f9454009130 2024-08-19T12:47:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 10 client.26660.objecter resend_mon_ops 2024-08-19T12:47:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1421427349 0 0) 0x7f9454009e70 con 0x7f9454009130 2024-08-19T12:47:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 10 monclient(hunting): _negotiate 2024-08-19T12:47:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:47:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:47:25.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 10 monclient(hunting): my global_id is 135601 2024-08-19T12:47:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94540065c0 con 0x7f9454009130 2024-08-19T12:47:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3210986321 0 0) 0x7f94540065c0 con 0x7f9454009130 2024-08-19T12:47:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9454009e70 con 0x7f9454009130 2024-08-19T12:47:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9468005b80 con 0x7f9454009130 2024-08-19T12:47:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:47:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:47:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 10 monclient: dump: 2024-08-19T12:47:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:47:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:47:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:47:25.830 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1956424423 0 0) 0x7f9454009e70 con 0x7f9454009130 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f945400c410 con 0x7f9454009130 2024-08-19T12:47:25.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7700 (unknown 0 0 2654517513) 0x7f9468001320 con 0x7f9454009130 2024-08-19T12:47:25.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:47:25.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:47:25.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.104:6792/0 conn(0x7f9454009130 legacy=0x7f94540095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f94727fc640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.104:6792/0 conn(0x7f9454009130 legacy=0x7f94540095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:47:25.832 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:47:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:25.832 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:47:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 60, 2024-08-19T12:47:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 416166, 2024-08-19T12:47:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:47:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:47:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:47:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:47:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:47:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:47:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:47:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:47:25.833 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:47:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:47:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:47:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:47:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.834 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:47:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:47:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:47:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:47:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:47:25.836 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:47:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:47:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:47:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:47:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:47:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:47:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:47:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:47:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:47:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:47:25.837 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:47:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:47:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:47:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:47:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:47:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:47:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:47:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:47:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:47:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:47:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:47:25.838 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:47:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:47:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:47:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:47:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:47:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:47:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:47:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:47:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:47:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:47:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.840 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.841 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:47:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:47:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:47:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:47:25.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:47:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:47:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:47:25.843 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:47:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:47:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:47:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:47:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:47:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:47:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:47:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:47:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:47:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:47:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:47:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:47:25.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:47:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:47:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:47:25.848 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:47:25.852 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.853 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:47:25.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:47:25.855 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:47:25.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:47:25.855 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:47:25.855 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:47:25.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 10 librados: watch_flush enter 2024-08-19T12:47:25.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 10 librados: watch_flush exit 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 20 client.26660.objecter shutdown clearing up homeless session... 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 10 client.26660.objecter successfully canceled tick 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.104:6800/899887506 conn(0x7f94600418b0 legacy=0x7f9460043d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 10 monclient: shutdown 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 1 -- 172.21.15.26:0/4236018455 >> v1:172.21.15.26:6790/0 conn(0x7f947406e7b0 legacy=0x7f9474137920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947a60b640 1 -- 172.21.15.26:0/4236018455 reap_dead start 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947a60b640 1 -- 172.21.15.26:0/4236018455 reap_dead start 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 1 -- 172.21.15.26:0/4236018455 shutdown_connections 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 1 -- 172.21.15.26:0/4236018455 >> 172.21.15.26:0/4236018455 conn(0x7f94740a8740 msgr2=0x7f94740aa000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 1 -- 172.21.15.26:0/4236018455 shutdown_connections 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 1 -- 172.21.15.26:0/4236018455 wait complete. 2024-08-19T12:47:25.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:47:25.808+0000 7f947c06e640 1 librados: shutdown 2024-08-19T12:47:25.858 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:47:25.858 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T12:48:36.402 INFO:tasks.ceph.osd.5.smithi104.stderr:2024-08-19T12:48:36.400+0000 7f0365f7e640 -1 snap_mapper.add_oid found existing snaps mapped on 300:ee0c4edf:test-rados-api-smithi104-24748-80::foo:2, removing 2024-08-19T12:48:36.403 INFO:tasks.ceph.osd.1.smithi026.stderr:2024-08-19T12:48:36.400+0000 7fa69ef21640 -1 snap_mapper.add_oid found existing snaps mapped on 300:ee0c4edf:test-rados-api-smithi104-24748-80::foo:2, removing 2024-08-19T12:48:55.862 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-08-19T12:48:55.862 INFO:tasks.ceph.mon.e:Restarting daemon 2024-08-19T12:48:55.863 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-08-19T12:48:55.866 INFO:tasks.ceph.mon.e:Started 2024-08-19T12:48:55.866 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-08-19T12:48:55.866 INFO:tasks.ceph.mon.c:Sent signal 19 2024-08-19T12:48:55.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:49:10.867 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-08-19T12:49:10.868 INFO:tasks.ceph.mon.c:Sent signal 18 2024-08-19T12:49:10.868 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:49:10.868 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:49:10.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.984+0000 7fb0421aa640 10 monclient: get_monmap_and_config 2024-08-19T12:49:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.984+0000 7fb0421aa640 10 monclient: build_initial_monmap 2024-08-19T12:49:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.984+0000 7fb0421aa640 10 monclient: monmap: 2024-08-19T12:49:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:10.984924+0000 2024-08-19T12:49:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:10.984924+0000 2024-08-19T12:49:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:10.986 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:10.986 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:10.986 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:10.986 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:10.986 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:10.986 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:10.986 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:10.986 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:10.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 1 Processor -- start 2024-08-19T12:49:10.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 1 -- start start 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 10 monclient: init 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 10 monclient: _add_conns ranks=[0,1,6,2,4,3,7,8,5] 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 10 monclient(hunting): picked mon.noname-a con 0x7fb03c073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 10 monclient(hunting): picked mon.noname-f con 0x7fb03c0a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 10 monclient(hunting): picked mon.noname-d con 0x7fb03c13f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb03c09dae0 con 0x7fb03c073470 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb03c136910 con 0x7fb03c0a57a0 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb03c09dd50 con 0x7fb03c13f2e0 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb0421aa640 10 monclient(hunting): authenticate will time out at 1345.072632s 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb03b7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb03c073470 0x7fb03c136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39714/0 (socket says 172.21.15.26:39714) 2024-08-19T12:49:10.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.985+0000 7fb03b7fe640 1 -- 172.21.15.26:0/1272286492 learned_addr learned my addr 172.21.15.26:0/1272286492 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:10.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.987+0000 7fb03bfff640 1 -- 172.21.15.26:0/1272286492 >> v1:172.21.15.26:6792/0 conn(0x7fb03c13f2e0 legacy=0x7fb03c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:10.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.987+0000 7fb03b7fe640 1 -- 172.21.15.26:0/1272286492 >> v1:172.21.15.26:6789/0 conn(0x7fb03c073470 legacy=0x7fb03c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:10.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:10.987+0000 7fb033fff640 1 -- 172.21.15.26:0/1272286492 >> v1:172.21.15.104:6789/0 conn(0x7fb03c0a57a0 legacy=0x7fb03c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 1 -- 172.21.15.26:0/1272286492 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 958561367 0 0) 0x7fb03c136910 con 0x7fb03c0a57a0 2024-08-19T12:49:11.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient(hunting): my global_id is 27487 2024-08-19T12:49:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 1 -- 172.21.15.26:0/1272286492 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb028003520 con 0x7fb03c0a57a0 2024-08-19T12:49:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 1 -- 172.21.15.26:0/1272286492 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 710271003 0 0) 0x7fb03c09dd50 con 0x7fb03c13f2e0 2024-08-19T12:49:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient(hunting): my global_id is 164337 2024-08-19T12:49:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 1 -- 172.21.15.26:0/1272286492 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb03c136910 con 0x7fb03c13f2e0 2024-08-19T12:49:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 1 -- 172.21.15.26:0/1272286492 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1490851635 0 0) 0x7fb028003520 con 0x7fb03c0a57a0 2024-08-19T12:49:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 1 -- 172.21.15.26:0/1272286492 >> v1:172.21.15.26:6792/0 conn(0x7fb03c13f2e0 legacy=0x7fb03c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 1 -- 172.21.15.26:0/1272286492 >> v1:172.21.15.26:6789/0 conn(0x7fb03c073470 legacy=0x7fb03c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient: _finish_hunting 0 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 1 monclient: found mon.noname-f 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 1 -- 172.21.15.26:0/1272286492 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb03c09e030 con 0x7fb03c0a57a0 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient: _finish_auth 0 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient: _check_auth_tickets 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 1 -- 172.21.15.26:0/1272286492 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb024001b30 con 0x7fb03c0a57a0 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.143+0000 7fb03affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb03affd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb03affd640 10 monclient: dump: 2024-08-19T12:49:11.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.149 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb0421aa640 5 monclient: authenticate success, global_id 27487 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb0421aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb03affd640 1 -- 172.21.15.26:0/1272286492 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb024003610 con 0x7fb03c0a57a0 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb03affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb03affd640 1 -- 172.21.15.26:0/1272286492 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb024003b80 con 0x7fb03c0a57a0 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb03affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb03affd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb03affd640 10 monclient: dump: 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb0421aa640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb0421aa640 4 set_mon_vals no callback set 2024-08-19T12:49:11.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb0421aa640 10 monclient: shutdown 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb0421aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.144+0000 7fb0421aa640 1 -- 172.21.15.26:0/1272286492 >> v1:172.21.15.104:6789/0 conn(0x7fb03c0a57a0 legacy=0x7fb03c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.145+0000 7fb0421aa640 1 -- 172.21.15.26:0/1272286492 shutdown_connections 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.145+0000 7fb0421aa640 1 -- 172.21.15.26:0/1272286492 >> 172.21.15.26:0/1272286492 conn(0x7fb03c0abff0 msgr2=0x7fb03c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.145+0000 7fb0421aa640 1 -- 172.21.15.26:0/1272286492 shutdown_connections 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.145+0000 7fb0421aa640 1 -- 172.21.15.26:0/1272286492 wait complete. 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.145+0000 7fb0421aa640 10 monclient: build_initial_monmap 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.145+0000 7fb0421aa640 10 monclient: monmap: 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:11.152 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 1 librados: starting msgr at 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 1 librados: starting objecter 2024-08-19T12:49:11.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 1 Processor -- start 2024-08-19T12:49:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 1 -- start start 2024-08-19T12:49:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 1 librados: setting wanted keys 2024-08-19T12:49:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 1 librados: calling monclient init 2024-08-19T12:49:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 10 monclient: init 2024-08-19T12:49:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 10 monclient: _add_conns ranks=[1,7,5,2,8,6,4,3,0] 2024-08-19T12:49:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 10 monclient(hunting): picked mon.noname-f con 0x7fb03c073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:11.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 10 monclient(hunting): picked mon.noname-i con 0x7fb03c0a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 10 monclient(hunting): picked mon.noname-h con 0x7fb03c13f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.146+0000 7fb0421aa640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb03c1371d0 con 0x7fb03c073470 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.147+0000 7fb0421aa640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb03c1353b0 con 0x7fb03c13f2e0 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.147+0000 7fb0421aa640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb03c136c50 con 0x7fb03c0a57a0 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.147+0000 7fb0421aa640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.147+0000 7fb0421aa640 10 monclient(hunting): authenticate will time out at 1345.234060s 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.147+0000 7fb03b7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb03c073470 0x7fb03c0a2050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:57900/0 (socket says 172.21.15.26:57900) 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.147+0000 7fb03b7fe640 1 -- 172.21.15.26:0/173771397 learned_addr learned my addr 172.21.15.26:0/173771397 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.149+0000 7fb033fff640 1 -- 172.21.15.26:0/173771397 >> v1:172.21.15.104:6792/0 conn(0x7fb03c0a57a0 legacy=0x7fb03c0a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.149+0000 7fb03b7fe640 1 -- 172.21.15.26:0/173771397 >> v1:172.21.15.104:6789/0 conn(0x7fb03c073470 legacy=0x7fb03c0a2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.149+0000 7fb038ff9640 10 client.?.objecter ms_handle_connect 0x7fb03c0a57a0 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.149+0000 7fb038ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.149+0000 7fb038ff9640 10 client.?.objecter ms_handle_connect 0x7fb03c073470 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.149+0000 7fb038ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:11.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.149+0000 7fb038ff9640 10 client.?.objecter ms_handle_connect 0x7fb03c13f2e0 2024-08-19T12:49:11.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.149+0000 7fb038ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:11.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.149+0000 7fb03bfff640 1 -- 172.21.15.26:0/173771397 >> v1:172.21.15.104:6791/0 conn(0x7fb03c13f2e0 legacy=0x7fb03c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.163+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3718910090 0 0) 0x7fb03c136c50 con 0x7fb03c0a57a0 2024-08-19T12:49:11.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.163+0000 7fb038ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.163+0000 7fb038ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.163+0000 7fb038ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.163+0000 7fb038ff9640 10 monclient(hunting): my global_id is 135610 2024-08-19T12:49:11.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.163+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb01c004eb0 con 0x7fb03c0a57a0 2024-08-19T12:49:11.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.163+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2415612358 0 0) 0x7fb03c1371d0 con 0x7fb03c073470 2024-08-19T12:49:11.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.163+0000 7fb038ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.163+0000 7fb038ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.163+0000 7fb038ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.163+0000 7fb038ff9640 10 monclient(hunting): my global_id is 27496 2024-08-19T12:49:11.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.163+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb03c136c50 con 0x7fb03c073470 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3506707349 0 0) 0x7fb01c004eb0 con 0x7fb03c0a57a0 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb03c1371d0 con 0x7fb03c0a57a0 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb024003b00 con 0x7fb03c0a57a0 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient(hunting): dump: 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.168 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1243786773 0 0) 0x7fb03c136c50 con 0x7fb03c073470 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb01c004eb0 con 0x7fb03c073470 2024-08-19T12:49:11.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb02c001d00 con 0x7fb03c073470 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient(hunting): dump: 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.170 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2289478993 0 0) 0x7fb03c1353b0 con 0x7fb03c13f2e0 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient(hunting): my global_id is 26726 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb03c136c50 con 0x7fb03c13f2e0 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2940766997 0 0) 0x7fb03c1371d0 con 0x7fb03c0a57a0 2024-08-19T12:49:11.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 >> v1:172.21.15.104:6791/0 conn(0x7fb03c13f2e0 legacy=0x7fb03c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 >> v1:172.21.15.104:6789/0 conn(0x7fb03c073470 legacy=0x7fb03c0a2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 monclient: found mon.i 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb03c13eac0 con 0x7fb03c0a57a0 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient: _finish_auth 0 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb0421aa640 5 monclient: authenticate success, global_id 135610 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb0421aa640 10 monclient: _renew_subs 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb0421aa640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:49:11.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb0421aa640 1 -- 172.21.15.26:0/173771397 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb03c0a2e70 con 0x7fb03c0a57a0 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb0421aa640 10 client.135610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb0421aa640 10 monclient: _renew_subs 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb0421aa640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb0421aa640 1 -- 172.21.15.26:0/173771397 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fb03c0a4610 con 0x7fb03c0a57a0 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb0421aa640 1 librados: init done 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb024004170 con 0x7fb03c0a57a0 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb0240043a0 con 0x7fb03c0a57a0 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb03a7fc640 4 set_mon_vals no callback set 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.164+0000 7fb038ff9640 10 monclient: dump: 2024-08-19T12:49:11.173 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.174 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.175 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.165+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fb024011dc0 con 0x7fb03c0a57a0 2024-08-19T12:49:11.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.165+0000 7fb0421aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:11.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.165+0000 7fb0421aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:11.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.165+0000 7fb0421aa640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:49:11.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.165+0000 7fb0421aa640 1 -- 172.21.15.26:0/173771397 --> v1:172.21.15.104:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb020005080 con 0x7fb03c0a57a0 2024-08-19T12:49:11.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.165+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(404..404 src has 311..404) ==== 5397+0+0 (unknown 1383070081 0 0) 0x7fb02404e180 con 0x7fb03c0a57a0 2024-08-19T12:49:11.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.165+0000 7fb038ff9640 10 client.135610.objecter ms_dispatch 0x7fb03c000b90 osd_map(404..404 src has 311..404) 2024-08-19T12:49:11.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.165+0000 7fb038ff9640 3 client.135610.objecter handle_osd_map got epochs [404,404] > 0 2024-08-19T12:49:11.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.165+0000 7fb038ff9640 3 client.135610.objecter handle_osd_map decoding full epoch 404 2024-08-19T12:49:11.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.166+0000 7fb038ff9640 20 client.135610.objecter dump_active .. 0 homeless 2024-08-19T12:49:11.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.166+0000 7fb03b7fe640 10 monclient: get_auth_request con 0x7fb01c041690 auth_method 0 2024-08-19T12:49:11.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.168+0000 7fb038ff9640 10 client.135610.objecter ms_handle_connect 0x7fb01c041690 2024-08-19T12:49:11.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.170+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.7 v1:172.21.15.104:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fb0240176f0 con 0x7fb03c0a57a0 2024-08-19T12:49:11.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.170+0000 7fb038ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:11.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.170+0000 7fb038ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:11.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.443+0000 7fb0421aa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:49:11.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.443+0000 7fb0421aa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:49:11.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.443+0000 7fb0421aa640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:49:11.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.443+0000 7fb0421aa640 1 -- 172.21.15.26:0/173771397 --> v1:172.21.15.104:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb020005370 con 0x7fb03c0a57a0 2024-08-19T12:49:11.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.443+0000 7fb038ff9640 1 -- 172.21.15.26:0/173771397 <== mon.7 v1:172.21.15.104:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 3973919154) 0x7fb024016ff0 con 0x7fb03c0a57a0 2024-08-19T12:49:11.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.443+0000 7fb038ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:49:11.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.443+0000 7fb038ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:11.445 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":68,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:49:11.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 10 librados: watch_flush enter 2024-08-19T12:49:11.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 10 librados: watch_flush exit 2024-08-19T12:49:11.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 20 client.135610.objecter shutdown clearing up homeless session... 2024-08-19T12:49:11.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 10 client.135610.objecter successfully canceled tick 2024-08-19T12:49:11.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 1 -- 172.21.15.26:0/173771397 >> v1:172.21.15.104:6800/899887506 conn(0x7fb01c041690 legacy=0x7fb01c043b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 10 monclient: shutdown 2024-08-19T12:49:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 1 -- 172.21.15.26:0/173771397 >> v1:172.21.15.104:6792/0 conn(0x7fb03c0a57a0 legacy=0x7fb03c0a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 1 -- 172.21.15.26:0/173771397 shutdown_connections 2024-08-19T12:49:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 1 -- 172.21.15.26:0/173771397 >> 172.21.15.26:0/173771397 conn(0x7fb03c0abff0 msgr2=0x7fb03c134980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 1 -- 172.21.15.26:0/173771397 shutdown_connections 2024-08-19T12:49:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 1 -- 172.21.15.26:0/173771397 wait complete. 2024-08-19T12:49:11.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.446+0000 7fb0421aa640 1 librados: shutdown 2024-08-19T12:49:11.457 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:49:11.457 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:49:11.457 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:49:11.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.556+0000 7f5c18721640 10 monclient: get_monmap_and_config 2024-08-19T12:49:11.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.556+0000 7f5c18721640 10 monclient: build_initial_monmap 2024-08-19T12:49:11.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.556+0000 7f5c18721640 10 monclient: monmap: 2024-08-19T12:49:11.557 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:11.557 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.557 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:11.557586+0000 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:11.557586+0000 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:11.558 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.557+0000 7f5c18721640 1 Processor -- start 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.557+0000 7f5c18721640 1 -- start start 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.557+0000 7f5c18721640 10 monclient: init 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.557+0000 7f5c18721640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.557+0000 7f5c18721640 10 monclient: _add_conns ranks=[7,3,6,0,1,8,5,2,4] 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.557+0000 7f5c18721640 10 monclient(hunting): picked mon.noname-i con 0x7f5c1006e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.558+0000 7f5c18721640 10 monclient(hunting): picked mon.noname-g con 0x7f5c100a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.558+0000 7f5c18721640 10 monclient(hunting): picked mon.noname-d con 0x7f5c100a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.558+0000 7f5c18721640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c10136790 con 0x7f5c100a5920 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.558+0000 7f5c18721640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1009ddf0 con 0x7f5c100a5d50 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.558+0000 7f5c18721640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1009e0d0 con 0x7f5c1006e7b0 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.558+0000 7f5c18721640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:11.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.558+0000 7f5c18721640 10 monclient(hunting): authenticate will time out at 1345.645064s 2024-08-19T12:49:11.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.558+0000 7f5c16cbe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f5c100a5d50 0x7f5c10141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:50910/0 (socket says 172.21.15.26:50910) 2024-08-19T12:49:11.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.558+0000 7f5c16cbe640 1 -- 172.21.15.26:0/997618138 learned_addr learned my addr 172.21.15.26:0/997618138 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:11.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c16cbe640 1 -- 172.21.15.26:0/997618138 >> v1:172.21.15.26:6792/0 conn(0x7f5c100a5d50 legacy=0x7f5c10141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c15cbc640 1 -- 172.21.15.26:0/997618138 >> v1:172.21.15.104:6790/0 conn(0x7f5c100a5920 legacy=0x7f5c1013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c164bd640 1 -- 172.21.15.26:0/997618138 >> v1:172.21.15.104:6792/0 conn(0x7f5c1006e7b0 legacy=0x7f5c10136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 1 -- 172.21.15.26:0/997618138 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3996493171 0 0) 0x7f5c080017d0 con 0x7f5c100a5d50 2024-08-19T12:49:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 10 monclient(hunting): my global_id is 164346 2024-08-19T12:49:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 1 -- 172.21.15.26:0/997618138 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bfc003520 con 0x7f5c100a5d50 2024-08-19T12:49:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 1 -- 172.21.15.26:0/997618138 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 310695082 0 0) 0x7f5bfc003520 con 0x7f5c100a5d50 2024-08-19T12:49:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 1 -- 172.21.15.26:0/997618138 >> v1:172.21.15.104:6792/0 conn(0x7f5c1006e7b0 legacy=0x7f5c10136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 1 -- 172.21.15.26:0/997618138 >> v1:172.21.15.104:6790/0 conn(0x7f5c100a5920 legacy=0x7f5c1013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 10 monclient: _finish_hunting 0 2024-08-19T12:49:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 1 monclient: found mon.noname-d 2024-08-19T12:49:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:11.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:49:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 1 -- 172.21.15.26:0/997618138 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1009e3b0 con 0x7f5c100a5d50 2024-08-19T12:49:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 10 monclient: _finish_auth 0 2024-08-19T12:49:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 10 monclient: _check_auth_tickets 2024-08-19T12:49:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 1 -- 172.21.15.26:0/997618138 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c08002030 con 0x7f5c100a5d50 2024-08-19T12:49:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c18721640 5 monclient: authenticate success, global_id 164346 2024-08-19T12:49:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c18721640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:11.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.560+0000 7f5c154bb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c154bb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c154bb640 10 monclient: dump: 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.567 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c18721640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c154bb640 1 -- 172.21.15.26:0/997618138 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c08004d10 con 0x7f5c100a5d50 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c154bb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c154bb640 1 -- 172.21.15.26:0/997618138 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c08005280 con 0x7f5c100a5d50 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c18721640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c18721640 4 set_mon_vals no callback set 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c154bb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c154bb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:11.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c154bb640 10 monclient: dump: 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.569 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c18721640 10 monclient: shutdown 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c18721640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c18721640 1 -- 172.21.15.26:0/997618138 >> v1:172.21.15.26:6792/0 conn(0x7f5c100a5d50 legacy=0x7f5c10141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c18721640 1 -- 172.21.15.26:0/997618138 shutdown_connections 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c18721640 1 -- 172.21.15.26:0/997618138 >> 172.21.15.26:0/997618138 conn(0x7f5c100a8740 msgr2=0x7f5c100a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c18721640 1 -- 172.21.15.26:0/997618138 shutdown_connections 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.561+0000 7f5c18721640 1 -- 172.21.15.26:0/997618138 wait complete. 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.562+0000 7f5c18721640 10 monclient: build_initial_monmap 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.562+0000 7f5c18721640 10 monclient: monmap: 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:11.570 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:11.571 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.562+0000 7f5c18721640 1 librados: starting msgr at 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.562+0000 7f5c18721640 1 librados: starting objecter 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.562+0000 7f5c18721640 1 Processor -- start 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.562+0000 7f5c18721640 1 -- start start 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.562+0000 7f5c18721640 1 librados: setting wanted keys 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.562+0000 7f5c18721640 1 librados: calling monclient init 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.562+0000 7f5c18721640 10 monclient: init 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c18721640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c18721640 10 monclient: _add_conns ranks=[1,0,4,2,5,8,6,7,3] 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c18721640 10 monclient(hunting): picked mon.noname-f con 0x7f5c1006e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c18721640 10 monclient(hunting): picked mon.noname-a con 0x7f5c100a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c18721640 10 monclient(hunting): picked mon.noname-c con 0x7f5c100a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c18721640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c100a1370 con 0x7f5c100a5920 2024-08-19T12:49:11.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c18721640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c100a1ed0 con 0x7f5c1006e7b0 2024-08-19T12:49:11.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c18721640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c10134970 con 0x7f5c100a5d50 2024-08-19T12:49:11.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c18721640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:11.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c18721640 10 monclient(hunting): authenticate will time out at 1345.650184s 2024-08-19T12:49:11.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c15cbc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5c100a5920 0x7f5c100a4e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39738/0 (socket says 172.21.15.26:39738) 2024-08-19T12:49:11.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c16cbe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f5c100a5d50 0x7f5c101d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57420/0 (socket says 172.21.15.26:57420) 2024-08-19T12:49:11.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.563+0000 7f5c15cbc640 1 -- 172.21.15.26:0/123409848 learned_addr learned my addr 172.21.15.26:0/123409848 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:11.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c15cbc640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.26:6789/0 conn(0x7f5c100a5920 legacy=0x7f5c100a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c16cbe640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.26:6791/0 conn(0x7f5c100a5d50 legacy=0x7f5c101d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 client.?.objecter ms_handle_connect 0x7f5c100a5920 2024-08-19T12:49:11.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:11.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 client.?.objecter ms_handle_connect 0x7f5c100a5d50 2024-08-19T12:49:11.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:11.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 client.?.objecter ms_handle_connect 0x7f5c1006e7b0 2024-08-19T12:49:11.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c164bd640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.104:6789/0 conn(0x7f5c1006e7b0 legacy=0x7f5c10132b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1647933522 0 0) 0x7f5c10134970 con 0x7f5c100a5d50 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): my global_id is 26995 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bf4004eb0 con 0x7f5c100a5d50 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3644072829 0 0) 0x7f5c100a1370 con 0x7f5c100a5920 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): my global_id is 144999 2024-08-19T12:49:11.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c10134970 con 0x7f5c100a5920 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3972289266 0 0) 0x7f5c100a1ed0 con 0x7f5c1006e7b0 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): my global_id is 27505 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c100a1370 con 0x7f5c1006e7b0 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2645865061 0 0) 0x7f5bf4004eb0 con 0x7f5c100a5d50 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c100a1ed0 con 0x7f5c100a5d50 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c08002070 con 0x7f5c100a5d50 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): dump: 2024-08-19T12:49:11.580 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.581 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3382542631 0 0) 0x7f5c10134970 con 0x7f5c100a5920 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.565+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5bf4004eb0 con 0x7f5c100a5920 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2094361010 0 0) 0x7f5c100a1370 con 0x7f5c1006e7b0 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c10134970 con 0x7f5c1006e7b0 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c00003610 con 0x7f5c100a5920 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient(hunting): dump: 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.582 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c0c001ca0 con 0x7f5c1006e7b0 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient(hunting): dump: 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.584 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1388924401 0 0) 0x7f5c100a1ed0 con 0x7f5c100a5d50 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.104:6789/0 conn(0x7f5c1006e7b0 legacy=0x7f5c10132b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.26:6789/0 conn(0x7f5c100a5920 legacy=0x7f5c100a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 1 monclient: found mon.c 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:11.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c100a2e30 con 0x7f5c100a5d50 2024-08-19T12:49:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient: _finish_auth 0 2024-08-19T12:49:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:49:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c18721640 5 monclient: authenticate success, global_id 26995 2024-08-19T12:49:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c18721640 10 monclient: _renew_subs 2024-08-19T12:49:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c18721640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c18721640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5c100a5540 con 0x7f5c100a5d50 2024-08-19T12:49:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c18721640 10 client.26995.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c18721640 10 monclient: _renew_subs 2024-08-19T12:49:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c18721640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c18721640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f5c101d7b90 con 0x7f5c100a5d50 2024-08-19T12:49:11.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c18721640 1 librados: init done 2024-08-19T12:49:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c08004cd0 con 0x7f5c100a5d50 2024-08-19T12:49:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c080054b0 con 0x7f5c100a5d50 2024-08-19T12:49:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c14cba640 4 set_mon_vals no callback set 2024-08-19T12:49:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.566+0000 7f5c06ffd640 10 monclient: dump: 2024-08-19T12:49:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.587 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.567+0000 7f5c18721640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:11.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.567+0000 7f5c18721640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5bf80065c0 con 0x7f5bf8002860 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.567+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5c08005650 con 0x7f5c100a5d50 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.567+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(405..405 src has 311..405) ==== 5413+0+0 (unknown 3147633577 0 0) 0x7f5c0804e2e0 con 0x7f5c100a5d50 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.567+0000 7f5c06ffd640 10 client.26995.objecter ms_dispatch 0x7f5c10000b90 osd_map(405..405 src has 311..405) 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.567+0000 7f5c06ffd640 3 client.26995.objecter handle_osd_map got epochs [405,405] > 0 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.567+0000 7f5c06ffd640 3 client.26995.objecter handle_osd_map decoding full epoch 405 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.567+0000 7f5c06ffd640 20 client.26995.objecter dump_active .. 0 homeless 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.567+0000 7f5c15cbc640 10 monclient: get_auth_request con 0x7f5bf4041ab0 auth_method 0 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 10 client.26995.objecter ms_handle_connect 0x7f5bf8002860 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 10 client.26995.objecter resend_mon_ops 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c164bd640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.26:6789/0 conn(0x7f5bf8002860 legacy=0x7f5bf8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2864692785 0 0) 0x7f5bf80065c0 con 0x7f5bf8002860 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 10 monclient(hunting): my global_id is 145008 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c100a1ed0 con 0x7f5bf8002860 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2467200949 0 0) 0x7f5c100a1ed0 con 0x7f5bf8002860 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5bf80065c0 con 0x7f5bf8002860 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c0c003c00 con 0x7f5bf8002860 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 10 monclient: dump: 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.590 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2335635453 0 0) 0x7f5bf80065c0 con 0x7f5bf8002860 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5bf800a160 con 0x7f5bf8002860 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.569+0000 7f5c06ffd640 10 client.26995.objecter ms_handle_connect 0x7f5bf4041ab0 2024-08-19T12:49:11.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.570+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5c0c003e20 con 0x7f5bf8002860 2024-08-19T12:49:11.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.570+0000 7f5c06ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:11.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.570+0000 7f5c06ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:11.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.570+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.26:6789/0 conn(0x7f5bf8002860 legacy=0x7f5bf8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.570+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.26:6789/0 conn(0x7f5bf8002860 legacy=0x7f5bf8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:11.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.576+0000 7f5c18721640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:11.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.576+0000 7f5c18721640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5bf8009e70 con 0x7f5bf8009130 2024-08-19T12:49:11.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 10 client.26995.objecter ms_handle_connect 0x7f5bf8009130 2024-08-19T12:49:11.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c164bd640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.26:6789/0 conn(0x7f5bf8009130 legacy=0x7f5bf80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 10 client.26995.objecter resend_mon_ops 2024-08-19T12:49:11.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1318919402 0 0) 0x7f5bf8009e70 con 0x7f5bf8009130 2024-08-19T12:49:11.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 10 monclient(hunting): my global_id is 145017 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bf80065c0 con 0x7f5bf8009130 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 916677443 0 0) 0x7f5bf80065c0 con 0x7f5bf8009130 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5bf8009e70 con 0x7f5bf8009130 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c0c004410 con 0x7f5bf8009130 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 10 monclient: dump: 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.594 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1341497906 0 0) 0x7f5bf8009e70 con 0x7f5bf8009130 2024-08-19T12:49:11.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.578+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5bf800c410 con 0x7f5bf8009130 2024-08-19T12:49:11.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7546 (unknown 0 0 2179718257) 0x7f5c0c005d00 con 0x7f5bf8009130 2024-08-19T12:49:11.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c06ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:11.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c06ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:11.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.26:6789/0 conn(0x7f5bf8009130 legacy=0x7f5bf80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c06ffd640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.26:6789/0 conn(0x7f5bf8009130 legacy=0x7f5bf80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:11.596 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:11.596 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:11.596 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:11.596 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:49:11.597 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:11.597 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 403497, 2024-08-19T12:49:11.597 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:11.597 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:11.597 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:11.597 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:11.597 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:11.597 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:11.597 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:11.597 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:11.597 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:11.597 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:11.598 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:11.599 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:11.600 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:11.601 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:11.602 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:11.602 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:11.602 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:11.602 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:11.602 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:11.602 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.602 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:11.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.602 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:11.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.602 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:11.602 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.603 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.605 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.606 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:11.607 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:11.607 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:11.607 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.607 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:11.607 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.608 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:11.611 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:11.612 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:11.613 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:11.615 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:49:11.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.616 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.616 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:11.616 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:11.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 10 librados: watch_flush enter 2024-08-19T12:49:11.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 10 librados: watch_flush exit 2024-08-19T12:49:11.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 20 client.26995.objecter shutdown clearing up homeless session... 2024-08-19T12:49:11.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 10 client.26995.objecter successfully canceled tick 2024-08-19T12:49:11.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.104:6800/899887506 conn(0x7f5bf4041ab0 legacy=0x7f5bf4043f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 10 monclient: shutdown 2024-08-19T12:49:11.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:11.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 1 -- 172.21.15.26:0/123409848 >> v1:172.21.15.26:6791/0 conn(0x7f5c100a5d50 legacy=0x7f5c101d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c16cbe640 1 -- 172.21.15.26:0/123409848 reap_dead start 2024-08-19T12:49:11.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c16cbe640 1 -- 172.21.15.26:0/123409848 reap_dead start 2024-08-19T12:49:11.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 1 -- 172.21.15.26:0/123409848 shutdown_connections 2024-08-19T12:49:11.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 1 -- 172.21.15.26:0/123409848 >> 172.21.15.26:0/123409848 conn(0x7f5c100a8740 msgr2=0x7f5c100aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:11.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 1 -- 172.21.15.26:0/123409848 shutdown_connections 2024-08-19T12:49:11.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 1 -- 172.21.15.26:0/123409848 wait complete. 2024-08-19T12:49:11.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.579+0000 7f5c18721640 1 librados: shutdown 2024-08-19T12:49:11.619 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:11.619 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:49:11.620 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:49:11.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.718+0000 7f0b0ff55640 10 monclient: get_monmap_and_config 2024-08-19T12:49:11.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.718+0000 7f0b0ff55640 10 monclient: build_initial_monmap 2024-08-19T12:49:11.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.718+0000 7f0b0ff55640 10 monclient: monmap: 2024-08-19T12:49:11.719 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:11.719 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.719 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:11.719621+0000 2024-08-19T12:49:11.720 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:11.719621+0000 2024-08-19T12:49:11.720 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:11.720 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.720 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:11.720 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:11.720 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:11.720 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:11.720 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:11.720 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:11.720 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:11.720 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:11.720 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:11.721 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.719+0000 7f0b0ff55640 1 Processor -- start 2024-08-19T12:49:11.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.719+0000 7f0b0ff55640 1 -- start start 2024-08-19T12:49:11.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.719+0000 7f0b0ff55640 10 monclient: init 2024-08-19T12:49:11.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.719+0000 7f0b0ff55640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.720+0000 7f0b0ff55640 10 monclient: _add_conns ranks=[7,1,4,3,0,6,8,5,2] 2024-08-19T12:49:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.720+0000 7f0b0ff55640 10 monclient(hunting): picked mon.noname-i con 0x7f0b0806e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.720+0000 7f0b0ff55640 10 monclient(hunting): picked mon.noname-f con 0x7f0b080a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.720+0000 7f0b0ff55640 10 monclient(hunting): picked mon.noname-c con 0x7f0b080a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.720+0000 7f0b0ff55640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b08136790 con 0x7f0b080a5920 2024-08-19T12:49:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.720+0000 7f0b0ff55640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b0809ddf0 con 0x7f0b080a5d50 2024-08-19T12:49:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.720+0000 7f0b0ff55640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b0809e0d0 con 0x7f0b0806e7b0 2024-08-19T12:49:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.720+0000 7f0b0ff55640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.720+0000 7f0b0ff55640 10 monclient(hunting): authenticate will time out at 1345.807103s 2024-08-19T12:49:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.720+0000 7f0b0e4f2640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f0b080a5d50 0x7f0b08141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57428/0 (socket says 172.21.15.26:57428) 2024-08-19T12:49:11.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.720+0000 7f0b0e4f2640 1 -- 172.21.15.26:0/3797492841 learned_addr learned my addr 172.21.15.26:0/3797492841 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:11.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0e4f2640 1 -- 172.21.15.26:0/3797492841 >> v1:172.21.15.26:6791/0 conn(0x7f0b080a5d50 legacy=0x7f0b08141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 1 -- 172.21.15.26:0/3797492841 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3031851352 0 0) 0x7f0af80017d0 con 0x7f0b080a5d50 2024-08-19T12:49:11.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient(hunting): my global_id is 27004 2024-08-19T12:49:11.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0d4f0640 1 -- 172.21.15.26:0/3797492841 >> v1:172.21.15.104:6789/0 conn(0x7f0b080a5920 legacy=0x7f0b0813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 1 -- 172.21.15.26:0/3797492841 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0af0003520 con 0x7f0b080a5d50 2024-08-19T12:49:11.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0dcf1640 1 -- 172.21.15.26:0/3797492841 >> v1:172.21.15.104:6792/0 conn(0x7f0b0806e7b0 legacy=0x7f0b08136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 1 -- 172.21.15.26:0/3797492841 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1604323878 0 0) 0x7f0af0003520 con 0x7f0b080a5d50 2024-08-19T12:49:11.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 1 -- 172.21.15.26:0/3797492841 >> v1:172.21.15.104:6792/0 conn(0x7f0b0806e7b0 legacy=0x7f0b08136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 1 -- 172.21.15.26:0/3797492841 >> v1:172.21.15.104:6789/0 conn(0x7f0b080a5920 legacy=0x7f0b0813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient: _finish_hunting 0 2024-08-19T12:49:11.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 1 monclient: found mon.noname-c 2024-08-19T12:49:11.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:11.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:49:11.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 1 -- 172.21.15.26:0/3797492841 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b0809e3b0 con 0x7f0b080a5d50 2024-08-19T12:49:11.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient: _finish_auth 0 2024-08-19T12:49:11.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient: _check_auth_tickets 2024-08-19T12:49:11.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:11.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 1 -- 172.21.15.26:0/3797492841 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0af8002030 con 0x7f0b080a5d50 2024-08-19T12:49:11.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:49:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient: dump: 2024-08-19T12:49:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.730 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ff55640 5 monclient: authenticate success, global_id 27004 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ff55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 1 -- 172.21.15.26:0/3797492841 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0af8004d10 con 0x7f0b080a5d50 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 1 -- 172.21.15.26:0/3797492841 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0af8005240 con 0x7f0b080a5d50 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ff55640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ff55640 4 set_mon_vals no callback set 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.722+0000 7f0b0ccef640 10 monclient: dump: 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.732 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.723+0000 7f0b0ff55640 10 monclient: shutdown 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.723+0000 7f0b0ff55640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.723+0000 7f0b0ff55640 1 -- 172.21.15.26:0/3797492841 >> v1:172.21.15.26:6791/0 conn(0x7f0b080a5d50 legacy=0x7f0b08141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.723+0000 7f0b0ff55640 1 -- 172.21.15.26:0/3797492841 shutdown_connections 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.723+0000 7f0b0ff55640 1 -- 172.21.15.26:0/3797492841 >> 172.21.15.26:0/3797492841 conn(0x7f0b080a8740 msgr2=0x7f0b080a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.723+0000 7f0b0ff55640 1 -- 172.21.15.26:0/3797492841 shutdown_connections 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.723+0000 7f0b0ff55640 1 -- 172.21.15.26:0/3797492841 wait complete. 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 10 monclient: build_initial_monmap 2024-08-19T12:49:11.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 10 monclient: monmap: 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 1 librados: starting msgr at 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 1 librados: starting objecter 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 1 Processor -- start 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 1 -- start start 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 1 librados: setting wanted keys 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 1 librados: calling monclient init 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 10 monclient: init 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 10 monclient: _add_conns ranks=[7,3,1,2,8,0,4,6,5] 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 10 monclient(hunting): picked mon.noname-i con 0x7f0b0806e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:11.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.724+0000 7f0b0ff55640 10 monclient(hunting): picked mon.noname-g con 0x7f0b080a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:11.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.725+0000 7f0b0ff55640 10 monclient(hunting): picked mon.noname-f con 0x7f0b080a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:11.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.725+0000 7f0b0ff55640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b080a1010 con 0x7f0b080a5d50 2024-08-19T12:49:11.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.725+0000 7f0b0ff55640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b080a1b70 con 0x7f0b080a5920 2024-08-19T12:49:11.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.725+0000 7f0b0ff55640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b080a2ad0 con 0x7f0b0806e7b0 2024-08-19T12:49:11.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.725+0000 7f0b0ff55640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:11.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.725+0000 7f0b0ff55640 10 monclient(hunting): authenticate will time out at 1345.812072s 2024-08-19T12:49:11.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.725+0000 7f0b0e4f2640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0b080a5d50 0x7f0b081d6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:57926/0 (socket says 172.21.15.26:57926) 2024-08-19T12:49:11.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.725+0000 7f0b0dcf1640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f0b0806e7b0 0x7f0b081ce580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:53878/0 (socket says 172.21.15.26:53878) 2024-08-19T12:49:11.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.725+0000 7f0b0e4f2640 1 -- 172.21.15.26:0/2841243829 learned_addr learned my addr 172.21.15.26:0/2841243829 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:11.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b0dcf1640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.104:6792/0 conn(0x7f0b0806e7b0 legacy=0x7f0b081ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b0e4f2640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.104:6789/0 conn(0x7f0b080a5d50 legacy=0x7f0b081d6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 client.?.objecter ms_handle_connect 0x7f0b0806e7b0 2024-08-19T12:49:11.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 client.?.objecter ms_handle_connect 0x7f0b080a5d50 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 client.?.objecter ms_handle_connect 0x7f0b080a5920 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b0d4f0640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.104:6790/0 conn(0x7f0b080a5920 legacy=0x7f0b081d3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3562065150 0 0) 0x7f0b080a2ad0 con 0x7f0b0806e7b0 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): my global_id is 135637 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0af4004eb0 con 0x7f0b0806e7b0 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b0d4f0640 1 --1- 172.21.15.26:0/2841243829 >> v1:172.21.15.104:6790/0 conn(0x7f0b080a5920 0x7f0b081d3560 :-1 s=OPENED pgs=3355 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f0b080a1b70 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:49:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4249448118 0 0) 0x7f0b080a1010 con 0x7f0b080a5d50 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): my global_id is 27523 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b080a2ad0 con 0x7f0b080a5d50 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1212347337 0 0) 0x7f0af4004eb0 con 0x7f0b0806e7b0 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b080a1010 con 0x7f0b0806e7b0 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0b00003610 con 0x7f0b0806e7b0 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): dump: 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.745 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2589373772 0 0) 0x7f0b080a2ad0 con 0x7f0b080a5d50 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0af4004eb0 con 0x7f0b080a5d50 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0af8002030 con 0x7f0b080a5d50 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.727+0000 7f0b067fc640 10 monclient(hunting): dump: 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.747 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4167999322 0 0) 0x7f0b080a1010 con 0x7f0b0806e7b0 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.104:6790/0 conn(0x7f0b080a5920 legacy=0x7f0b081d3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.104:6789/0 conn(0x7f0b080a5d50 legacy=0x7f0b081d6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 10 monclient: _finish_hunting 0 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 1 monclient: found mon.i 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b080a2db0 con 0x7f0b0806e7b0 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 10 monclient: _finish_auth 0 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 10 monclient: _check_auth_tickets 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b0ff55640 5 monclient: authenticate success, global_id 135637 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b0ff55640 10 monclient: _renew_subs 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b0ff55640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692731134 0 0) 0x7f0b080a1b70 con 0x7f0b080a5920 2024-08-19T12:49:11.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b0ff55640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0b080a3d50 con 0x7f0b0806e7b0 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b0ff55640 10 client.135637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b0ff55640 10 monclient: _renew_subs 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b0ff55640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b0ff55640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f0b080a0910 con 0x7f0b0806e7b0 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b0ff55640 1 librados: init done 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b00003b00 con 0x7f0b0806e7b0 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0b00004030 con 0x7f0b0806e7b0 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b07fff640 4 set_mon_vals no callback set 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:11.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b067fc640 10 monclient: dump: 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.751 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b0ff55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.728+0000 7f0b0ff55640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0ae80065c0 con 0x7f0ae8002860 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.729+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f0b000041d0 con 0x7f0b0806e7b0 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.729+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(405..405 src has 311..405) ==== 5413+0+0 (unknown 3147633577 0 0) 0x7f0b0004e280 con 0x7f0b0806e7b0 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.729+0000 7f0b067fc640 10 client.135637.objecter ms_dispatch 0x7f0b08000b90 osd_map(405..405 src has 311..405) 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.729+0000 7f0b067fc640 3 client.135637.objecter handle_osd_map got epochs [405,405] > 0 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.729+0000 7f0b067fc640 3 client.135637.objecter handle_osd_map decoding full epoch 405 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.729+0000 7f0b067fc640 20 client.135637.objecter dump_active .. 0 homeless 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.729+0000 7f0b0e4f2640 10 monclient: get_auth_request con 0x7f0af4041580 auth_method 0 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 10 client.135637.objecter ms_handle_connect 0x7f0ae8002860 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b0d4f0640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.26:6790/0 conn(0x7f0ae8002860 legacy=0x7f0ae8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 10 client.135637.objecter resend_mon_ops 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282138547 0 0) 0x7f0ae80065c0 con 0x7f0ae8002860 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 10 monclient(hunting): my global_id is 26669 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b080a1b70 con 0x7f0ae8002860 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3249549578 0 0) 0x7f0b080a1b70 con 0x7f0ae8002860 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ae80065c0 con 0x7f0ae8002860 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0afc001c30 con 0x7f0ae8002860 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 10 monclient: dump: 2024-08-19T12:49:11.753 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 891068950 0 0) 0x7f0ae80065c0 con 0x7f0ae8002860 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ae800a160 con 0x7f0ae8002860 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 10 client.135637.objecter ms_handle_connect 0x7f0af4041580 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0afc0057c0 con 0x7f0ae8002860 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.26:6790/0 conn(0x7f0ae8002860 legacy=0x7f0ae8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.731+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.26:6790/0 conn(0x7f0ae8002860 legacy=0x7f0ae8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.737+0000 7f0b0ff55640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.738+0000 7f0b0ff55640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0ae8009e70 con 0x7f0ae8009130 2024-08-19T12:49:11.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b0d4f0640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.26:6790/0 conn(0x7f0ae8009130 legacy=0x7f0ae80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 10 client.135637.objecter ms_handle_connect 0x7f0ae8009130 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 10 client.135637.objecter resend_mon_ops 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1352467248 0 0) 0x7f0ae8009e70 con 0x7f0ae8009130 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 10 monclient(hunting): my global_id is 26678 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ae80065c0 con 0x7f0ae8009130 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3515571143 0 0) 0x7f0ae80065c0 con 0x7f0ae8009130 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ae8009e70 con 0x7f0ae8009130 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0afc003d80 con 0x7f0ae8009130 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:11.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 10 monclient: dump: 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.757 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.758 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.758 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2571360793 0 0) 0x7f0ae8009e70 con 0x7f0ae8009130 2024-08-19T12:49:11.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ae800c410 con 0x7f0ae8009130 2024-08-19T12:49:11.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1710625572) 0x7f0afc004020 con 0x7f0ae8009130 2024-08-19T12:49:11.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:11.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:11.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.26:6790/0 conn(0x7f0ae8009130 legacy=0x7f0ae80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.740+0000 7f0b067fc640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.26:6790/0 conn(0x7f0ae8009130 legacy=0x7f0ae80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 664795, 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:11.759 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:11.760 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:11.761 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:11.762 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:11.763 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.764 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:11.765 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.766 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.766 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.766 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:11.766 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:11.766 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.766 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.766 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.766 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:11.766 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.767 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:11.769 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.770 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.772 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:11.773 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:11.777 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:49:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.778 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:11.778 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:11.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 10 librados: watch_flush enter 2024-08-19T12:49:11.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 10 librados: watch_flush exit 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 20 client.135637.objecter shutdown clearing up homeless session... 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 10 client.135637.objecter successfully canceled tick 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.104:6800/899887506 conn(0x7f0af4041580 legacy=0x7f0af4043a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 10 monclient: shutdown 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 1 -- 172.21.15.26:0/2841243829 >> v1:172.21.15.104:6792/0 conn(0x7f0b0806e7b0 legacy=0x7f0b081ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0e4f2640 1 -- 172.21.15.26:0/2841243829 reap_dead start 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0e4f2640 1 -- 172.21.15.26:0/2841243829 reap_dead start 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 1 -- 172.21.15.26:0/2841243829 shutdown_connections 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 1 -- 172.21.15.26:0/2841243829 >> 172.21.15.26:0/2841243829 conn(0x7f0b080a8740 msgr2=0x7f0b080aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 1 -- 172.21.15.26:0/2841243829 shutdown_connections 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 1 -- 172.21.15.26:0/2841243829 wait complete. 2024-08-19T12:49:11.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.741+0000 7f0b0ff55640 1 librados: shutdown 2024-08-19T12:49:11.782 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:11.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:49:11.782 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:49:11.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.881+0000 7f0600817640 10 monclient: get_monmap_and_config 2024-08-19T12:49:11.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.881+0000 7f0600817640 10 monclient: build_initial_monmap 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.881+0000 7f0600817640 10 monclient: monmap: 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:11.882724+0000 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:11.882724+0000 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:11.883 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:11.884 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:11.884 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:11.884 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:11.884 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.882+0000 7f0600817640 1 Processor -- start 2024-08-19T12:49:11.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.882+0000 7f0600817640 1 -- start start 2024-08-19T12:49:11.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.882+0000 7f0600817640 10 monclient: init 2024-08-19T12:49:11.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.883+0000 7f0600817640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:11.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.883+0000 7f0600817640 10 monclient: _add_conns ranks=[5,2,4,8,0,3,7,6,1] 2024-08-19T12:49:11.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.883+0000 7f0600817640 10 monclient(hunting): picked mon.noname-h con 0x7f05fc073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:11.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.883+0000 7f0600817640 10 monclient(hunting): picked mon.noname-b con 0x7f05fc0a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:11.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.883+0000 7f0600817640 10 monclient(hunting): picked mon.noname-c con 0x7f05fc13f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:11.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.883+0000 7f0600817640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05fc09dae0 con 0x7f05fc0a57a0 2024-08-19T12:49:11.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.883+0000 7f0600817640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05fc136910 con 0x7f05fc13f2e0 2024-08-19T12:49:11.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.883+0000 7f0600817640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05fc09dd50 con 0x7f05fc073470 2024-08-19T12:49:11.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.883+0000 7f0600817640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:11.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.883+0000 7f0600817640 10 monclient(hunting): authenticate will time out at 1345.970232s 2024-08-19T12:49:11.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.883+0000 7f05f9d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f05fc0a57a0 0x7f05fc0a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59266/0 (socket says 172.21.15.26:59266) 2024-08-19T12:49:11.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.883+0000 7f05f9d9b640 1 -- 172.21.15.26:0/3035693487 learned_addr learned my addr 172.21.15.26:0/3035693487 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:11.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f9d9b640 1 -- 172.21.15.26:0/3035693487 >> v1:172.21.15.26:6790/0 conn(0x7f05fc0a57a0 legacy=0x7f05fc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 1 -- 172.21.15.26:0/3035693487 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1885853867 0 0) 0x7f05fc09dae0 con 0x7f05fc0a57a0 2024-08-19T12:49:11.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05fa59c640 1 -- 172.21.15.26:0/3035693487 >> v1:172.21.15.104:6791/0 conn(0x7f05fc073470 legacy=0x7f05fc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient(hunting): my global_id is 26687 2024-08-19T12:49:11.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 1 -- 172.21.15.26:0/3035693487 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05e4003520 con 0x7f05fc0a57a0 2024-08-19T12:49:11.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 1 -- 172.21.15.26:0/3035693487 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 922569967 0 0) 0x7f05fc09dd50 con 0x7f05fc073470 2024-08-19T12:49:11.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient(hunting): my global_id is 26735 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 1 -- 172.21.15.26:0/3035693487 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05fc09dae0 con 0x7f05fc073470 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 1 -- 172.21.15.26:0/3035693487 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1523238237 0 0) 0x7f05e4003520 con 0x7f05fc0a57a0 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 1 -- 172.21.15.26:0/3035693487 >> v1:172.21.15.104:6791/0 conn(0x7f05fc073470 legacy=0x7f05fc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 1 -- 172.21.15.26:0/3035693487 >> v1:172.21.15.26:6791/0 conn(0x7f05fc13f2e0 legacy=0x7f05fc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient: _finish_hunting 0 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 1 monclient: found mon.noname-b 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 1 -- 172.21.15.26:0/3035693487 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05fc09e030 con 0x7f05fc0a57a0 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient: _finish_auth 0 2024-08-19T12:49:11.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient: _check_auth_tickets 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 1 -- 172.21.15.26:0/3035693487 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f05f0001ae0 con 0x7f05fc0a57a0 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f0600817640 5 monclient: authenticate success, global_id 26687 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f0600817640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient: dump: 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.892 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f0600817640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 1 -- 172.21.15.26:0/3035693487 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f05f00048a0 con 0x7f05fc0a57a0 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:11.893 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.885+0000 7f05f959a640 1 -- 172.21.15.26:0/3035693487 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f05f0004e10 con 0x7f05fc0a57a0 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.886+0000 7f0600817640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.886+0000 7f0600817640 4 set_mon_vals no callback set 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.886+0000 7f05f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.886+0000 7f05f959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.886+0000 7f05f959a640 10 monclient: dump: 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.894 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.886+0000 7f0600817640 10 monclient: shutdown 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.886+0000 7f0600817640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.886+0000 7f0600817640 1 -- 172.21.15.26:0/3035693487 >> v1:172.21.15.26:6790/0 conn(0x7f05fc0a57a0 legacy=0x7f05fc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.886+0000 7f0600817640 1 -- 172.21.15.26:0/3035693487 shutdown_connections 2024-08-19T12:49:11.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.886+0000 7f0600817640 1 -- 172.21.15.26:0/3035693487 >> 172.21.15.26:0/3035693487 conn(0x7f05fc0abff0 msgr2=0x7f05fc0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.886+0000 7f0600817640 1 -- 172.21.15.26:0/3035693487 shutdown_connections 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.886+0000 7f0600817640 1 -- 172.21.15.26:0/3035693487 wait complete. 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.887+0000 7f0600817640 10 monclient: build_initial_monmap 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.887+0000 7f0600817640 10 monclient: monmap: 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.887+0000 7f0600817640 1 librados: starting msgr at 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.887+0000 7f0600817640 1 librados: starting objecter 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.887+0000 7f0600817640 1 Processor -- start 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.887+0000 7f0600817640 1 -- start start 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.887+0000 7f0600817640 1 librados: setting wanted keys 2024-08-19T12:49:11.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.887+0000 7f0600817640 1 librados: calling monclient init 2024-08-19T12:49:11.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.887+0000 7f0600817640 10 monclient: init 2024-08-19T12:49:11.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.888+0000 7f0600817640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:11.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.888+0000 7f0600817640 10 monclient: _add_conns ranks=[1,0,4,2,5,7,3,8,6] 2024-08-19T12:49:11.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.888+0000 7f0600817640 10 monclient(hunting): picked mon.noname-f con 0x7f05fc073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:11.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.888+0000 7f0600817640 10 monclient(hunting): picked mon.noname-a con 0x7f05fc0a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:11.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.888+0000 7f0600817640 10 monclient(hunting): picked mon.noname-c con 0x7f05fc13f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:11.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.888+0000 7f0600817640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05fc130330 con 0x7f05fc0a57a0 2024-08-19T12:49:11.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.888+0000 7f0600817640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05fc130610 con 0x7f05fc073470 2024-08-19T12:49:11.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.888+0000 7f0600817640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05fc12ea00 con 0x7f05fc13f2e0 2024-08-19T12:49:11.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.888+0000 7f0600817640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:11.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.888+0000 7f0600817640 10 monclient(hunting): authenticate will time out at 1345.975210s 2024-08-19T12:49:11.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.888+0000 7f05f9d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f05fc0a57a0 0x7f05fc12e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39766/0 (socket says 172.21.15.26:39766) 2024-08-19T12:49:11.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.888+0000 7f05f9d9b640 1 -- 172.21.15.26:0/2186869289 learned_addr learned my addr 172.21.15.26:0/2186869289 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05f9d9b640 1 -- 172.21.15.26:0/2186869289 >> v1:172.21.15.26:6789/0 conn(0x7f05fc0a57a0 legacy=0x7f05fc12e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 client.?.objecter ms_handle_connect 0x7f05fc0a57a0 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05fa59c640 1 -- 172.21.15.26:0/2186869289 >> v1:172.21.15.104:6789/0 conn(0x7f05fc073470 legacy=0x7f05fc12dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 client.?.objecter ms_handle_connect 0x7f05fc073470 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1713908573 0 0) 0x7f05fc130330 con 0x7f05fc0a57a0 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 monclient(hunting): my global_id is 145026 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05dc004eb0 con 0x7f05fc0a57a0 2024-08-19T12:49:11.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3851503491 0 0) 0x7f05fc130610 con 0x7f05fc073470 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 monclient(hunting): my global_id is 27532 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05fc130330 con 0x7f05fc073470 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2056202974 0 0) 0x7f05dc004eb0 con 0x7f05fc0a57a0 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05fc130610 con 0x7f05fc0a57a0 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f05f0004d90 con 0x7f05fc0a57a0 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 monclient(hunting): dump: 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.906 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05eaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.890+0000 7f05f9d9b640 1 --1- 172.21.15.26:0/2186869289 >> v1:172.21.15.26:6789/0 conn(0x7f05fc0a57a0 0x7f05fc12e2f0 :-1 s=OPENED pgs=3614 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f05fc130610 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4153766111 0 0) 0x7f05fc130330 con 0x7f05fc073470 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05dc004eb0 con 0x7f05fc073470 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f05f4003990 con 0x7f05fc073470 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 10 monclient(hunting): dump: 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.908 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 176341670 0 0) 0x7f05dc004eb0 con 0x7f05fc073470 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 >> v1:172.21.15.26:6791/0 conn(0x7f05fc13f2e0 legacy=0x7f05fc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 >> v1:172.21.15.26:6789/0 conn(0x7f05fc0a57a0 legacy=0x7f05fc12e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 10 monclient: _finish_hunting 0 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 1 monclient: found mon.f 2024-08-19T12:49:11.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05fc12ed10 con 0x7f05fc073470 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 10 monclient: _finish_auth 0 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 10 monclient: _check_auth_tickets 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f0600817640 5 monclient: authenticate success, global_id 27532 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1337106787 0 0) 0x7f05fc130610 con 0x7f05fc0a57a0 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f0600817640 10 monclient: _renew_subs 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f0600817640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f0600817640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f05fc13f0b0 con 0x7f05fc073470 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f0600817640 10 client.27532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:11.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f0600817640 10 monclient: _renew_subs 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f0600817640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f0600817640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f05fc141e00 con 0x7f05fc073470 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f0600817640 1 librados: init done 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f05f4003e90 con 0x7f05fc073470 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f05f4004400 con 0x7f05fc073470 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05f8d99640 4 set_mon_vals no callback set 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.891+0000 7f05eaffd640 10 monclient: dump: 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.911 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.892+0000 7f0600817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:11.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.892+0000 7f0600817640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f05e00065c0 con 0x7f05e0002860 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.892+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f05f40045a0 con 0x7f05fc073470 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.892+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(405..405 src has 311..405) ==== 5413+0+0 (unknown 3147633577 0 0) 0x7f05f404e590 con 0x7f05fc073470 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.892+0000 7f05eaffd640 10 client.27532.objecter ms_dispatch 0x7f05fc000b90 osd_map(405..405 src has 311..405) 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.892+0000 7f05eaffd640 3 client.27532.objecter handle_osd_map got epochs [405,405] > 0 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.892+0000 7f05eaffd640 3 client.27532.objecter handle_osd_map decoding full epoch 405 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.892+0000 7f05eaffd640 20 client.27532.objecter dump_active .. 0 homeless 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.892+0000 7f05fad9d640 10 monclient: get_auth_request con 0x7f05dc0414c0 auth_method 0 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05f9d9b640 1 -- 172.21.15.26:0/2186869289 >> v1:172.21.15.26:6791/0 conn(0x7f05e0002860 legacy=0x7f05e0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 10 client.27532.objecter ms_handle_connect 0x7f05e0002860 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 10 client.27532.objecter resend_mon_ops 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 789559463 0 0) 0x7f05e00065c0 con 0x7f05e0002860 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 10 monclient(hunting): my global_id is 27013 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05fc130610 con 0x7f05e0002860 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3200944238 0 0) 0x7f05fc130610 con 0x7f05e0002860 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05e00065c0 con 0x7f05e0002860 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f05f0001d30 con 0x7f05e0002860 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 10 monclient: dump: 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.915 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.915 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.915 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.915 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.915 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.915 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.915 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.915 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.915 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.915 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2282097663 0 0) 0x7f05e00065c0 con 0x7f05e0002860 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.894+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05e000a160 con 0x7f05e0002860 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.895+0000 7f05eaffd640 10 client.27532.objecter ms_handle_connect 0x7f05dc0414c0 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.895+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f05f0005740 con 0x7f05e0002860 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.895+0000 7f05eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.895+0000 7f05eaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.895+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 >> v1:172.21.15.26:6791/0 conn(0x7f05e0002860 legacy=0x7f05e0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.895+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 >> v1:172.21.15.26:6791/0 conn(0x7f05e0002860 legacy=0x7f05e0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.901+0000 7f0600817640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.901+0000 7f0600817640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f05e0009e70 con 0x7f05e0009130 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05f9d9b640 1 -- 172.21.15.26:0/2186869289 >> v1:172.21.15.26:6791/0 conn(0x7f05e0009130 legacy=0x7f05e00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 10 client.27532.objecter ms_handle_connect 0x7f05e0009130 2024-08-19T12:49:11.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 10 client.27532.objecter resend_mon_ops 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2185383751 0 0) 0x7f05e0009e70 con 0x7f05e0009130 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 10 monclient(hunting): my global_id is 27022 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05e00065c0 con 0x7f05e0009130 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3281802815 0 0) 0x7f05e00065c0 con 0x7f05e0009130 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f05e0009e70 con 0x7f05e0009130 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f05f0005ee0 con 0x7f05e0009130 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.903+0000 7f05eaffd640 10 monclient: dump: 2024-08-19T12:49:11.917 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:11.918 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:11.919 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:11.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1826688065 0 0) 0x7f05e0009e70 con 0x7f05e0009130 2024-08-19T12:49:11.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05e000c3f0 con 0x7f05e0009130 2024-08-19T12:49:11.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3534505494) 0x7f05f0008280 con 0x7f05e0009130 2024-08-19T12:49:11.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f05eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:11.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f05eaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:11.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 >> v1:172.21.15.26:6791/0 conn(0x7f05e0009130 legacy=0x7f05e00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f05eaffd640 1 -- 172.21.15.26:0/2186869289 >> v1:172.21.15.26:6791/0 conn(0x7f05e0009130 legacy=0x7f05e00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:11.919 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:11.919 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:11.919 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 664934, 2024-08-19T12:49:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:11.920 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:11.921 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:11.922 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:11.923 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:11.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:11.925 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.926 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:11.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.930 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:11.931 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.934 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.935 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:11.936 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:11.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.938 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:11.938 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:11.938 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:11.938 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.938 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:11.938 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:11.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:11.938 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:11.938 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:11.939 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:11.939 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:11.939 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:11.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:11.939 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:11.939 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:11.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 10 librados: watch_flush enter 2024-08-19T12:49:11.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 10 librados: watch_flush exit 2024-08-19T12:49:11.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 20 client.27532.objecter shutdown clearing up homeless session... 2024-08-19T12:49:11.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 10 client.27532.objecter successfully canceled tick 2024-08-19T12:49:11.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 1 -- 172.21.15.26:0/2186869289 >> v1:172.21.15.104:6800/899887506 conn(0x7f05dc0414c0 legacy=0x7f05dc043980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 10 monclient: shutdown 2024-08-19T12:49:11.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:11.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 1 -- 172.21.15.26:0/2186869289 >> v1:172.21.15.104:6789/0 conn(0x7f05fc073470 legacy=0x7f05fc12dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:11.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f05fad9d640 1 -- 172.21.15.26:0/2186869289 reap_dead start 2024-08-19T12:49:11.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f05fad9d640 1 -- 172.21.15.26:0/2186869289 reap_dead start 2024-08-19T12:49:11.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 1 -- 172.21.15.26:0/2186869289 shutdown_connections 2024-08-19T12:49:11.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 1 -- 172.21.15.26:0/2186869289 >> 172.21.15.26:0/2186869289 conn(0x7f05fc0abff0 msgr2=0x7f05fc1354a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:11.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 1 -- 172.21.15.26:0/2186869289 shutdown_connections 2024-08-19T12:49:11.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 1 -- 172.21.15.26:0/2186869289 wait complete. 2024-08-19T12:49:11.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:11.904+0000 7f0600817640 1 librados: shutdown 2024-08-19T12:49:11.942 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:11.942 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:49:11.943 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:49:12.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.042+0000 7ff7a8533640 10 monclient: get_monmap_and_config 2024-08-19T12:49:12.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.042+0000 7ff7a8533640 10 monclient: build_initial_monmap 2024-08-19T12:49:12.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.042+0000 7ff7a8533640 10 monclient: monmap: 2024-08-19T12:49:12.043 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:12.043 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.043 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:12.043385+0000 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:12.043385+0000 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:12.044 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 1 Processor -- start 2024-08-19T12:49:12.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 1 -- start start 2024-08-19T12:49:12.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 10 monclient: init 2024-08-19T12:49:12.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:12.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 10 monclient: _add_conns ranks=[8,6,5,7,4,2,1,3,0] 2024-08-19T12:49:12.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 10 monclient(hunting): picked mon.noname-e con 0x7ff7a006e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:12.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 10 monclient(hunting): picked mon.noname-d con 0x7ff7a00a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:12.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 10 monclient(hunting): picked mon.noname-h con 0x7ff7a00a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:12.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a0136790 con 0x7ff7a00a5d50 2024-08-19T12:49:12.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a009ddf0 con 0x7ff7a00a5920 2024-08-19T12:49:12.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a009e0d0 con 0x7ff7a006e7b0 2024-08-19T12:49:12.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:12.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a8533640 10 monclient(hunting): authenticate will time out at 1346.130953s 2024-08-19T12:49:12.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a62cf640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7ff7a006e7b0 0x7ff7a0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:35674/0 (socket says 172.21.15.26:35674) 2024-08-19T12:49:12.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.043+0000 7ff7a5ace640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7ff7a00a5920 0x7ff7a013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:50922/0 (socket says 172.21.15.26:50922) 2024-08-19T12:49:12.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.044+0000 7ff7a62cf640 1 -- 172.21.15.26:0/712325966 learned_addr learned my addr 172.21.15.26:0/712325966 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:12.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a5ace640 1 -- 172.21.15.26:0/712325966 >> v1:172.21.15.26:6792/0 conn(0x7ff7a00a5920 legacy=0x7ff7a013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a62cf640 1 -- 172.21.15.26:0/712325966 >> v1:172.21.15.26:6793/0 conn(0x7ff7a006e7b0 legacy=0x7ff7a0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 1 -- 172.21.15.26:0/712325966 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4105177605 0 0) 0x7ff7a009e0d0 con 0x7ff7a006e7b0 2024-08-19T12:49:12.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient(hunting): my global_id is 174113 2024-08-19T12:49:12.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 1 -- 172.21.15.26:0/712325966 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff78c003520 con 0x7ff7a006e7b0 2024-08-19T12:49:12.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 1 -- 172.21.15.26:0/712325966 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 593825730 0 0) 0x7ff790001450 con 0x7ff7a00a5920 2024-08-19T12:49:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient(hunting): my global_id is 164355 2024-08-19T12:49:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 1 -- 172.21.15.26:0/712325966 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7a009e0d0 con 0x7ff7a00a5920 2024-08-19T12:49:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a6ad0640 1 -- 172.21.15.26:0/712325966 >> v1:172.21.15.104:6791/0 conn(0x7ff7a00a5d50 legacy=0x7ff7a0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 1 -- 172.21.15.26:0/712325966 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2001698030 0 0) 0x7ff78c003520 con 0x7ff7a006e7b0 2024-08-19T12:49:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 1 -- 172.21.15.26:0/712325966 >> v1:172.21.15.26:6792/0 conn(0x7ff7a00a5920 legacy=0x7ff7a013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 1 -- 172.21.15.26:0/712325966 >> v1:172.21.15.104:6791/0 conn(0x7ff7a00a5d50 legacy=0x7ff7a0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient: _finish_hunting 0 2024-08-19T12:49:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 1 monclient: found mon.noname-e 2024-08-19T12:49:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 1 -- 172.21.15.26:0/712325966 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a009e3b0 con 0x7ff7a006e7b0 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient: _finish_auth 0 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient: _check_auth_tickets 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 1 -- 172.21.15.26:0/712325966 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff79c001ae0 con 0x7ff7a006e7b0 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient: dump: 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.056 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a8533640 5 monclient: authenticate success, global_id 174113 2024-08-19T12:49:12.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a8533640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 1 -- 172.21.15.26:0/712325966 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff79c0048a0 con 0x7ff7a006e7b0 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 1 -- 172.21.15.26:0/712325966 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff79c004d90 con 0x7ff7a006e7b0 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a52cd640 10 monclient: dump: 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.058 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a8533640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.046+0000 7ff7a8533640 4 set_mon_vals no callback set 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.047+0000 7ff7a8533640 10 monclient: shutdown 2024-08-19T12:49:12.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.047+0000 7ff7a8533640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.047+0000 7ff7a8533640 1 -- 172.21.15.26:0/712325966 >> v1:172.21.15.26:6793/0 conn(0x7ff7a006e7b0 legacy=0x7ff7a0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.047+0000 7ff7a8533640 1 -- 172.21.15.26:0/712325966 shutdown_connections 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.047+0000 7ff7a8533640 1 -- 172.21.15.26:0/712325966 >> 172.21.15.26:0/712325966 conn(0x7ff7a00a8740 msgr2=0x7ff7a00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.047+0000 7ff7a8533640 1 -- 172.21.15.26:0/712325966 shutdown_connections 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.047+0000 7ff7a8533640 1 -- 172.21.15.26:0/712325966 wait complete. 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.047+0000 7ff7a8533640 10 monclient: build_initial_monmap 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.047+0000 7ff7a8533640 10 monclient: monmap: 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:12.060 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 1 librados: starting msgr at 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 1 librados: starting objecter 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 1 Processor -- start 2024-08-19T12:49:12.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 1 -- start start 2024-08-19T12:49:12.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 1 librados: setting wanted keys 2024-08-19T12:49:12.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 1 librados: calling monclient init 2024-08-19T12:49:12.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 10 monclient: init 2024-08-19T12:49:12.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:12.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 10 monclient: _add_conns ranks=[4,7,3,5,0,2,1,8,6] 2024-08-19T12:49:12.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 10 monclient(hunting): picked mon.noname-c con 0x7ff7a006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:12.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 10 monclient(hunting): picked mon.noname-i con 0x7ff7a00a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:12.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 10 monclient(hunting): picked mon.noname-g con 0x7ff7a00a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:12.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a0142a70 con 0x7ff7a00a5d50 2024-08-19T12:49:12.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a013b530 con 0x7ff7a006e7b0 2024-08-19T12:49:12.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a0134580 con 0x7ff7a00a5920 2024-08-19T12:49:12.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a8533640 10 monclient(hunting): authenticate will time out at 1346.135887s 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a62cf640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7ff7a006e7b0 0x7ff7a01ce6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57464/0 (socket says 172.21.15.26:57464) 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.048+0000 7ff7a62cf640 1 -- 172.21.15.26:0/530150126 learned_addr learned my addr 172.21.15.26:0/530150126 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a62cf640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.26:6791/0 conn(0x7ff7a006e7b0 legacy=0x7ff7a01ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 client.?.objecter ms_handle_connect 0x7ff7a006e7b0 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1148997519 0 0) 0x7ff7a013b530 con 0x7ff7a006e7b0 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): my global_id is 27031 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff784004eb0 con 0x7ff7a006e7b0 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 client.?.objecter ms_handle_connect 0x7ff7a00a5d50 2024-08-19T12:49:12.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a6ad0640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.104:6790/0 conn(0x7ff7a00a5d50 legacy=0x7ff7a01d6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 client.?.objecter ms_handle_connect 0x7ff7a00a5920 2024-08-19T12:49:12.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a5ace640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.104:6792/0 conn(0x7ff7a00a5920 legacy=0x7ff7a01d36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3457240851 0 0) 0x7ff784004eb0 con 0x7ff7a006e7b0 2024-08-19T12:49:12.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff7a013b530 con 0x7ff7a006e7b0 2024-08-19T12:49:12.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff79c004d90 con 0x7ff7a006e7b0 2024-08-19T12:49:12.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:12.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:49:12.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): dump: 2024-08-19T12:49:12.064 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.064 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.065 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4194971376 0 0) 0x7ff7a0142a70 con 0x7ff7a00a5d50 2024-08-19T12:49:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): my global_id is 154506 2024-08-19T12:49:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff784004eb0 con 0x7ff7a00a5d50 2024-08-19T12:49:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 810299608 0 0) 0x7ff7a0134580 con 0x7ff7a00a5920 2024-08-19T12:49:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient(hunting): my global_id is 135646 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7a0142a70 con 0x7ff7a00a5920 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3198125892 0 0) 0x7ff7a013b530 con 0x7ff7a006e7b0 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.104:6792/0 conn(0x7ff7a00a5920 legacy=0x7ff7a01d36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.104:6790/0 conn(0x7ff7a00a5d50 legacy=0x7ff7a01d6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 monclient: found mon.c 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a01391d0 con 0x7ff7a006e7b0 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient: _finish_auth 0 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:12.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a8533640 5 monclient: authenticate success, global_id 27031 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a8533640 10 monclient: _renew_subs 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a8533640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a8533640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff7a013b250 con 0x7ff7a006e7b0 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a8533640 10 client.27031.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a8533640 10 monclient: _renew_subs 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a8533640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a8533640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7ff7a0138830 con 0x7ff7a006e7b0 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a8533640 1 librados: init done 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff79c001b40 con 0x7ff7a006e7b0 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff79c0056c0 con 0x7ff7a006e7b0 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff7a4acc640 4 set_mon_vals no callback set 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.051+0000 7ff796ffd640 10 monclient: dump: 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.069 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.052+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff79c011b20 con 0x7ff7a006e7b0 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.052+0000 7ff7a8533640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.052+0000 7ff7a8533640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff7880065c0 con 0x7ff788002860 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.052+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(405..405 src has 311..405) ==== 5413+0+0 (unknown 3147633577 0 0) 0x7ff79c04e2b0 con 0x7ff7a006e7b0 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.052+0000 7ff796ffd640 10 client.27031.objecter ms_dispatch 0x7ff7a0000b90 osd_map(405..405 src has 311..405) 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.052+0000 7ff796ffd640 3 client.27031.objecter handle_osd_map got epochs [405,405] > 0 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.052+0000 7ff796ffd640 3 client.27031.objecter handle_osd_map decoding full epoch 405 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.052+0000 7ff796ffd640 20 client.27031.objecter dump_active .. 0 homeless 2024-08-19T12:49:12.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.053+0000 7ff7a6ad0640 10 monclient: get_auth_request con 0x7ff784041170 auth_method 0 2024-08-19T12:49:12.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.054+0000 7ff7a5ace640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.26:6792/0 conn(0x7ff788002860 legacy=0x7ff788004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.054+0000 7ff796ffd640 10 client.27031.objecter ms_handle_connect 0x7ff788002860 2024-08-19T12:49:12.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.054+0000 7ff796ffd640 10 client.27031.objecter resend_mon_ops 2024-08-19T12:49:12.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 772588797 0 0) 0x7ff7880065c0 con 0x7ff788002860 2024-08-19T12:49:12.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 10 monclient(hunting): my global_id is 164364 2024-08-19T12:49:12.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7a013b530 con 0x7ff788002860 2024-08-19T12:49:12.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1088653206 0 0) 0x7ff7a013b530 con 0x7ff788002860 2024-08-19T12:49:12.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff7880065c0 con 0x7ff788002860 2024-08-19T12:49:12.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff790001a50 con 0x7ff788002860 2024-08-19T12:49:12.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:12.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 10 monclient: dump: 2024-08-19T12:49:12.074 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.074 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.074 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.074 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.074 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.074 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 10 client.27031.objecter ms_handle_connect 0x7ff784041170 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2438966392 0 0) 0x7ff7880065c0 con 0x7ff788002860 2024-08-19T12:49:12.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff78800a160 con 0x7ff788002860 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff790004880 con 0x7ff788002860 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.26:6792/0 conn(0x7ff788002860 legacy=0x7ff788004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.055+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.26:6792/0 conn(0x7ff788002860 legacy=0x7ff788004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.061+0000 7ff7a8533640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.061+0000 7ff7a8533640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff788009e70 con 0x7ff788009130 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff7a5ace640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.26:6792/0 conn(0x7ff788009130 legacy=0x7ff7880095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 10 client.27031.objecter ms_handle_connect 0x7ff788009130 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 10 client.27031.objecter resend_mon_ops 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1354124598 0 0) 0x7ff788009e70 con 0x7ff788009130 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 10 monclient(hunting): my global_id is 164373 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7880065c0 con 0x7ff788009130 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2341831909 0 0) 0x7ff7880065c0 con 0x7ff788009130 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff788009e70 con 0x7ff788009130 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff790005030 con 0x7ff788009130 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 10 monclient: dump: 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.077 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 377578707 0 0) 0x7ff788009e70 con 0x7ff788009130 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff78800c3f0 con 0x7ff788009130 2024-08-19T12:49:12.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 2346634909) 0x7ff790007390 con 0x7ff788009130 2024-08-19T12:49:12.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:12.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:12.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.064+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.26:6792/0 conn(0x7ff788009130 legacy=0x7ff7880095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff796ffd640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.26:6792/0 conn(0x7ff788009130 legacy=0x7ff7880095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.079 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:12.079 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:12.079 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:12.079 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:12.079 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:12.079 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 135619, 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:12.080 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.081 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.082 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.082 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:12.082 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:12.082 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.082 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.082 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.082 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.082 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.082 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.082 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.082 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.082 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:12.083 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.084 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:12.085 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.086 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:12.089 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.090 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.093 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:12.094 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.095 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.098 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.099 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.099 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.099 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:12.099 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:12.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 10 librados: watch_flush enter 2024-08-19T12:49:12.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 10 librados: watch_flush exit 2024-08-19T12:49:12.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 20 client.27031.objecter shutdown clearing up homeless session... 2024-08-19T12:49:12.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 10 client.27031.objecter successfully canceled tick 2024-08-19T12:49:12.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.104:6800/899887506 conn(0x7ff784041170 legacy=0x7ff784043630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 10 monclient: shutdown 2024-08-19T12:49:12.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:12.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 1 -- 172.21.15.26:0/530150126 >> v1:172.21.15.26:6791/0 conn(0x7ff7a006e7b0 legacy=0x7ff7a01ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a6ad0640 1 -- 172.21.15.26:0/530150126 reap_dead start 2024-08-19T12:49:12.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a6ad0640 1 -- 172.21.15.26:0/530150126 reap_dead start 2024-08-19T12:49:12.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 1 -- 172.21.15.26:0/530150126 shutdown_connections 2024-08-19T12:49:12.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 1 -- 172.21.15.26:0/530150126 >> 172.21.15.26:0/530150126 conn(0x7ff7a00a8740 msgr2=0x7ff7a006dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:12.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 1 -- 172.21.15.26:0/530150126 shutdown_connections 2024-08-19T12:49:12.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 1 -- 172.21.15.26:0/530150126 wait complete. 2024-08-19T12:49:12.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.065+0000 7ff7a8533640 1 librados: shutdown 2024-08-19T12:49:12.102 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:12.102 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:49:12.102 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:49:12.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.201+0000 7faaee88b640 10 monclient: get_monmap_and_config 2024-08-19T12:49:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.201+0000 7faaee88b640 10 monclient: build_initial_monmap 2024-08-19T12:49:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.201+0000 7faaee88b640 10 monclient: monmap: 2024-08-19T12:49:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:12.201950+0000 2024-08-19T12:49:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:12.201950+0000 2024-08-19T12:49:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:12.202 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:12.203 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:12.203 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:12.203 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:12.203 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:12.203 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:12.203 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.201+0000 7faaee88b640 1 Processor -- start 2024-08-19T12:49:12.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.201+0000 7faaee88b640 1 -- start start 2024-08-19T12:49:12.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.201+0000 7faaee88b640 10 monclient: init 2024-08-19T12:49:12.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.202+0000 7faaee88b640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:12.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.202+0000 7faaee88b640 10 monclient: _add_conns ranks=[2,3,1,7,0,4,8,6,5] 2024-08-19T12:49:12.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.202+0000 7faaee88b640 10 monclient(hunting): picked mon.noname-b con 0x7faae806e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:12.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.202+0000 7faaee88b640 10 monclient(hunting): picked mon.noname-g con 0x7faae80a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:12.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.202+0000 7faaee88b640 10 monclient(hunting): picked mon.noname-f con 0x7faae80a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:12.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.202+0000 7faaee88b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faae8136790 con 0x7faae80a5d50 2024-08-19T12:49:12.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.202+0000 7faaee88b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faae809ddf0 con 0x7faae806e7b0 2024-08-19T12:49:12.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.202+0000 7faaee88b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faae809e0d0 con 0x7faae80a5920 2024-08-19T12:49:12.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.202+0000 7faaee88b640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:12.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.202+0000 7faaee88b640 10 monclient(hunting): authenticate will time out at 1346.289511s 2024-08-19T12:49:12.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.202+0000 7faae7fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7faae806e7b0 0x7faae8136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59268/0 (socket says 172.21.15.26:59268) 2024-08-19T12:49:12.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.202+0000 7faae7fff640 1 -- 172.21.15.26:0/1848282782 learned_addr learned my addr 172.21.15.26:0/1848282782 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.204+0000 7faae7fff640 1 -- 172.21.15.26:0/1848282782 >> v1:172.21.15.26:6790/0 conn(0x7faae806e7b0 legacy=0x7faae8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 1 -- 172.21.15.26:0/1848282782 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3212316465 0 0) 0x7faad8001450 con 0x7faae806e7b0 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient(hunting): my global_id is 26696 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faaece28640 1 -- 172.21.15.26:0/1848282782 >> v1:172.21.15.104:6789/0 conn(0x7faae80a5d50 legacy=0x7faae8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 1 -- 172.21.15.26:0/1848282782 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faad4003520 con 0x7faae806e7b0 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faadffff640 1 -- 172.21.15.26:0/1848282782 >> v1:172.21.15.104:6790/0 conn(0x7faae80a5920 legacy=0x7faae813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 1 -- 172.21.15.26:0/1848282782 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1059653069 0 0) 0x7faae8136790 con 0x7faae80a5d50 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient(hunting): my global_id is 27541 2024-08-19T12:49:12.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 1 -- 172.21.15.26:0/1848282782 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faad8001450 con 0x7faae80a5d50 2024-08-19T12:49:12.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 1 -- 172.21.15.26:0/1848282782 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3556867375 0 0) 0x7faad4003520 con 0x7faae806e7b0 2024-08-19T12:49:12.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 1 -- 172.21.15.26:0/1848282782 >> v1:172.21.15.104:6790/0 conn(0x7faae80a5920 legacy=0x7faae813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 1 -- 172.21.15.26:0/1848282782 >> v1:172.21.15.104:6789/0 conn(0x7faae80a5d50 legacy=0x7faae8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient: _finish_hunting 0 2024-08-19T12:49:12.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 1 monclient: found mon.noname-b 2024-08-19T12:49:12.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:12.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:49:12.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 1 -- 172.21.15.26:0/1848282782 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae809e3b0 con 0x7faae806e7b0 2024-08-19T12:49:12.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient: _finish_auth 0 2024-08-19T12:49:12.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient: _check_auth_tickets 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 1 -- 172.21.15.26:0/1848282782 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faad8001cb0 con 0x7faae806e7b0 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faaee88b640 5 monclient: authenticate success, global_id 26696 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faaee88b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient: dump: 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.211 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faaee88b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 1 -- 172.21.15.26:0/1848282782 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faad8004950 con 0x7faae806e7b0 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:12.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 1 -- 172.21.15.26:0/1848282782 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faad8004ec0 con 0x7faae806e7b0 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faaee88b640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faaee88b640 4 set_mon_vals no callback set 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faae77fe640 10 monclient: dump: 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faaee88b640 10 monclient: shutdown 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faaee88b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.205+0000 7faaee88b640 1 -- 172.21.15.26:0/1848282782 >> v1:172.21.15.26:6790/0 conn(0x7faae806e7b0 legacy=0x7faae8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 1 -- 172.21.15.26:0/1848282782 shutdown_connections 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 1 -- 172.21.15.26:0/1848282782 >> 172.21.15.26:0/1848282782 conn(0x7faae80a8740 msgr2=0x7faae80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:12.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 1 -- 172.21.15.26:0/1848282782 shutdown_connections 2024-08-19T12:49:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 1 -- 172.21.15.26:0/1848282782 wait complete. 2024-08-19T12:49:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 10 monclient: build_initial_monmap 2024-08-19T12:49:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 10 monclient: monmap: 2024-08-19T12:49:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.215 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:12.218 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 1 librados: starting msgr at 2024-08-19T12:49:12.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 1 librados: starting objecter 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 1 Processor -- start 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 1 -- start start 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 1 librados: setting wanted keys 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 1 librados: calling monclient init 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.206+0000 7faaee88b640 10 monclient: init 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.207+0000 7faaee88b640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.207+0000 7faaee88b640 10 monclient: _add_conns ranks=[3,5,7,0,4,1,8,2,6] 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.207+0000 7faaee88b640 10 monclient(hunting): picked mon.noname-g con 0x7faae806e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.207+0000 7faaee88b640 10 monclient(hunting): picked mon.noname-h con 0x7faae80a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.207+0000 7faaee88b640 10 monclient(hunting): picked mon.noname-i con 0x7faae80a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.207+0000 7faaee88b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faae8138fb0 con 0x7faae806e7b0 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.207+0000 7faaee88b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faae81cf060 con 0x7faae80a5920 2024-08-19T12:49:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.207+0000 7faaee88b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faae8139b10 con 0x7faae80a5d50 2024-08-19T12:49:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.207+0000 7faaee88b640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.207+0000 7faaee88b640 10 monclient(hunting): authenticate will time out at 1346.294490s 2024-08-19T12:49:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.207+0000 7faae7fff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7faae806e7b0 0x7faae81ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:52552/0 (socket says 172.21.15.26:52552) 2024-08-19T12:49:12.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.207+0000 7faae7fff640 1 -- 172.21.15.26:0/566672162 learned_addr learned my addr 172.21.15.26:0/566672162 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:12.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae7fff640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.104:6790/0 conn(0x7faae806e7b0 legacy=0x7faae81ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faaece28640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.104:6792/0 conn(0x7faae80a5d50 legacy=0x7faae81d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 client.?.objecter ms_handle_connect 0x7faae806e7b0 2024-08-19T12:49:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 client.?.objecter ms_handle_connect 0x7faae80a5d50 2024-08-19T12:49:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 client.?.objecter ms_handle_connect 0x7faae80a5920 2024-08-19T12:49:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faadffff640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.104:6791/0 conn(0x7faae80a5920 legacy=0x7faae81ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 589701064 0 0) 0x7faae8138fb0 con 0x7faae806e7b0 2024-08-19T12:49:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 monclient(hunting): my global_id is 154524 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac8004eb0 con 0x7faae806e7b0 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2123079712 0 0) 0x7faae8139b10 con 0x7faae80a5d50 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 monclient(hunting): my global_id is 135655 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faae8138fb0 con 0x7faae80a5d50 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2548978382 0 0) 0x7faae81cf060 con 0x7faae80a5920 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.209+0000 7faae57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient(hunting): my global_id is 26753 2024-08-19T12:49:12.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faae8139b10 con 0x7faae80a5920 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3446849863 0 0) 0x7faae8138fb0 con 0x7faae80a5d50 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faae81cf060 con 0x7faae80a5d50 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2747814372 0 0) 0x7faac8004eb0 con 0x7faae806e7b0 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faae8138fb0 con 0x7faae806e7b0 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faae00020b0 con 0x7faae80a5d50 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient(hunting): dump: 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.228 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faad8004e40 con 0x7faae806e7b0 2024-08-19T12:49:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient(hunting): dump: 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.230 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4038108834 0 0) 0x7faae8139b10 con 0x7faae80a5920 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faac8004eb0 con 0x7faae80a5920 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faad0001c80 con 0x7faae80a5920 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient(hunting): dump: 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.231 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.232 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 139561732 0 0) 0x7faae81cf060 con 0x7faae80a5d50 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.104:6791/0 conn(0x7faae80a5920 legacy=0x7faae81ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.104:6790/0 conn(0x7faae806e7b0 legacy=0x7faae81ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient: _finish_hunting 0 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 monclient: found mon.i 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae813aa70 con 0x7faae80a5d50 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient: _finish_auth 0 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient: _check_auth_tickets 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:12.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faaee88b640 5 monclient: authenticate success, global_id 135655 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faaee88b640 10 monclient: _renew_subs 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faaee88b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faaee88b640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faae813a6d0 con 0x7faae80a5d50 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faaee88b640 10 client.135655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faaee88b640 10 monclient: _renew_subs 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faaee88b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faaee88b640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7faae8138610 con 0x7faae80a5d50 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faaee88b640 1 librados: init done 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faae0003c40 con 0x7faae80a5d50 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faae0004230 con 0x7faae80a5d50 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae6ffd640 4 set_mon_vals no callback set 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.210+0000 7faae57fa640 10 monclient: dump: 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.235 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.211+0000 7faaee88b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.211+0000 7faaee88b640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7faacc0065c0 con 0x7faacc002860 2024-08-19T12:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.211+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7faae0004450 con 0x7faae80a5d50 2024-08-19T12:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.211+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(405..405 src has 311..405) ==== 5413+0+0 (unknown 3147633577 0 0) 0x7faae004e440 con 0x7faae80a5d50 2024-08-19T12:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.211+0000 7faae57fa640 10 client.135655.objecter ms_dispatch 0x7faae8000b90 osd_map(405..405 src has 311..405) 2024-08-19T12:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.211+0000 7faae57fa640 3 client.135655.objecter handle_osd_map got epochs [405,405] > 0 2024-08-19T12:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.211+0000 7faae57fa640 3 client.135655.objecter handle_osd_map decoding full epoch 405 2024-08-19T12:49:12.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.212+0000 7faae57fa640 20 client.135655.objecter dump_active .. 0 homeless 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.212+0000 7faadffff640 10 monclient: get_auth_request con 0x7faac80419b0 auth_method 0 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 10 client.135655.objecter ms_handle_connect 0x7faacc002860 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae7fff640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.26:6793/0 conn(0x7faacc002860 legacy=0x7faacc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 10 client.135655.objecter resend_mon_ops 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2794271566 0 0) 0x7faacc0065c0 con 0x7faacc002860 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 10 monclient(hunting): my global_id is 174122 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faae81cf060 con 0x7faacc002860 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 245884225 0 0) 0x7faae81cf060 con 0x7faacc002860 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faacc0065c0 con 0x7faacc002860 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faad8001c50 con 0x7faacc002860 2024-08-19T12:49:12.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:49:12.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.213+0000 7faae57fa640 10 monclient: dump: 2024-08-19T12:49:12.238 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.238 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.238 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.238 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.238 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.238 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.238 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.238 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.239 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.214+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3358130096 0 0) 0x7faacc0065c0 con 0x7faacc002860 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.214+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faacc00a160 con 0x7faacc002860 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.214+0000 7faae57fa640 10 client.135655.objecter ms_handle_connect 0x7faac80419b0 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.214+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7faad8005880 con 0x7faacc002860 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.214+0000 7faae57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.214+0000 7faae57fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.214+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.26:6793/0 conn(0x7faacc002860 legacy=0x7faacc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.214+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.26:6793/0 conn(0x7faacc002860 legacy=0x7faacc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.220+0000 7faaee88b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.220+0000 7faaee88b640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7faacc009e70 con 0x7faacc009130 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.222+0000 7faae7fff640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.26:6793/0 conn(0x7faacc009130 legacy=0x7faacc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.222+0000 7faae57fa640 10 client.135655.objecter ms_handle_connect 0x7faacc009130 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.222+0000 7faae57fa640 10 client.135655.objecter resend_mon_ops 2024-08-19T12:49:12.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.222+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1480446768 0 0) 0x7faacc009e70 con 0x7faacc009130 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.222+0000 7faae57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.222+0000 7faae57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.222+0000 7faae57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.222+0000 7faae57fa640 10 monclient(hunting): my global_id is 174131 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.222+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faacc0065c0 con 0x7faacc009130 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1743605222 0 0) 0x7faacc0065c0 con 0x7faacc009130 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faacc009e70 con 0x7faacc009130 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faad8005f60 con 0x7faacc009130 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 10 monclient: dump: 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.241 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.242 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.243 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2289190545 0 0) 0x7faacc009e70 con 0x7faacc009130 2024-08-19T12:49:12.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faacc00c410 con 0x7faacc009130 2024-08-19T12:49:12.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 1086911994) 0x7faad80082e0 con 0x7faacc009130 2024-08-19T12:49:12.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:12.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:12.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.26:6793/0 conn(0x7faacc009130 legacy=0x7faacc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faae57fa640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.26:6793/0 conn(0x7faacc009130 legacy=0x7faacc0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.243 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:12.243 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 16267, 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:12.244 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.245 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.246 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:12.247 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.248 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.249 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:12.250 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.251 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.252 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:12.253 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.254 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.255 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.257 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.258 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.261 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.263 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:12.263 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:12.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faaee88b640 10 librados: watch_flush enter 2024-08-19T12:49:12.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faaee88b640 10 librados: watch_flush exit 2024-08-19T12:49:12.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faaee88b640 20 client.135655.objecter shutdown clearing up homeless session... 2024-08-19T12:49:12.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faaee88b640 10 client.135655.objecter successfully canceled tick 2024-08-19T12:49:12.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faaee88b640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.104:6800/899887506 conn(0x7faac80419b0 legacy=0x7faac8043e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faaee88b640 10 monclient: shutdown 2024-08-19T12:49:12.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faaee88b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:12.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faaee88b640 1 -- 172.21.15.26:0/566672162 >> v1:172.21.15.104:6792/0 conn(0x7faae80a5d50 legacy=0x7faae81d64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faaece28640 1 -- 172.21.15.26:0/566672162 reap_dead start 2024-08-19T12:49:12.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faaece28640 1 -- 172.21.15.26:0/566672162 reap_dead start 2024-08-19T12:49:12.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faaee88b640 1 -- 172.21.15.26:0/566672162 shutdown_connections 2024-08-19T12:49:12.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.223+0000 7faaee88b640 1 -- 172.21.15.26:0/566672162 >> 172.21.15.26:0/566672162 conn(0x7faae80a8740 msgr2=0x7faae80aaee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:12.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.224+0000 7faaee88b640 1 -- 172.21.15.26:0/566672162 shutdown_connections 2024-08-19T12:49:12.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.224+0000 7faaee88b640 1 -- 172.21.15.26:0/566672162 wait complete. 2024-08-19T12:49:12.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.224+0000 7faaee88b640 1 librados: shutdown 2024-08-19T12:49:12.266 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:12.266 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:49:12.266 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:49:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.365+0000 7f83ef772640 10 monclient: get_monmap_and_config 2024-08-19T12:49:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.365+0000 7f83ef772640 10 monclient: build_initial_monmap 2024-08-19T12:49:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.365+0000 7f83ef772640 10 monclient: monmap: 2024-08-19T12:49:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:12.366180+0000 2024-08-19T12:49:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:12.366180+0000 2024-08-19T12:49:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:12.366 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:12.367 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:12.367 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.365+0000 7f83ef772640 1 Processor -- start 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ef772640 1 -- start start 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ef772640 10 monclient: init 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ef772640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ef772640 10 monclient: _add_conns ranks=[4,3,5,2,6,8,7,0,1] 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ef772640 10 monclient(hunting): picked mon.noname-c con 0x7f83e806e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ef772640 10 monclient(hunting): picked mon.noname-g con 0x7f83e80a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ef772640 10 monclient(hunting): picked mon.noname-h con 0x7f83e80a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ef772640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e8136790 con 0x7f83e80a5920 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ef772640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e809ddf0 con 0x7f83e806e7b0 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ef772640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e809e0d0 con 0x7f83e80a5d50 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ef772640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ef772640 10 monclient(hunting): authenticate will time out at 1346.453611s 2024-08-19T12:49:12.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ed50e640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f83e806e7b0 0x7f83e8136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57470/0 (socket says 172.21.15.26:57470) 2024-08-19T12:49:12.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.366+0000 7f83ed50e640 1 -- 172.21.15.26:0/947134538 learned_addr learned my addr 172.21.15.26:0/947134538 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:12.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.368+0000 7f83ed50e640 1 -- 172.21.15.26:0/947134538 >> v1:172.21.15.26:6791/0 conn(0x7f83e806e7b0 legacy=0x7f83e8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 1 -- 172.21.15.26:0/947134538 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3074246860 0 0) 0x7f83e4001450 con 0x7f83e806e7b0 2024-08-19T12:49:12.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient(hunting): my global_id is 27040 2024-08-19T12:49:12.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 1 -- 172.21.15.26:0/947134538 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83d4003520 con 0x7f83e806e7b0 2024-08-19T12:49:12.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83ecd0d640 1 -- 172.21.15.26:0/947134538 >> v1:172.21.15.104:6790/0 conn(0x7f83e80a5920 legacy=0x7f83e813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83edd0f640 1 -- 172.21.15.26:0/947134538 >> v1:172.21.15.104:6791/0 conn(0x7f83e80a5d50 legacy=0x7f83e8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 1 -- 172.21.15.26:0/947134538 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4072355718 0 0) 0x7f83d4003520 con 0x7f83e806e7b0 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 1 -- 172.21.15.26:0/947134538 >> v1:172.21.15.104:6791/0 conn(0x7f83e80a5d50 legacy=0x7f83e8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 1 -- 172.21.15.26:0/947134538 >> v1:172.21.15.104:6790/0 conn(0x7f83e80a5920 legacy=0x7f83e813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient: _finish_hunting 0 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 1 monclient: found mon.noname-c 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 1 -- 172.21.15.26:0/947134538 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e809e3b0 con 0x7f83e806e7b0 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient: _finish_auth 0 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient: _check_auth_tickets 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:12.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 1 -- 172.21.15.26:0/947134538 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f83e4001cb0 con 0x7f83e806e7b0 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient: dump: 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.375 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83ef772640 5 monclient: authenticate success, global_id 27040 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83ef772640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 1 -- 172.21.15.26:0/947134538 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f83e4004990 con 0x7f83e806e7b0 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 1 -- 172.21.15.26:0/947134538 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f83e4004f00 con 0x7f83e806e7b0 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83ef772640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83ef772640 4 set_mon_vals no callback set 2024-08-19T12:49:12.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83dffff640 10 monclient: dump: 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83ef772640 10 monclient: shutdown 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83ef772640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83ef772640 1 -- 172.21.15.26:0/947134538 >> v1:172.21.15.26:6791/0 conn(0x7f83e806e7b0 legacy=0x7f83e8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83ef772640 1 -- 172.21.15.26:0/947134538 shutdown_connections 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.369+0000 7f83ef772640 1 -- 172.21.15.26:0/947134538 >> 172.21.15.26:0/947134538 conn(0x7f83e80a8740 msgr2=0x7f83e80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.370+0000 7f83ef772640 1 -- 172.21.15.26:0/947134538 shutdown_connections 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.370+0000 7f83ef772640 1 -- 172.21.15.26:0/947134538 wait complete. 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.370+0000 7f83ef772640 10 monclient: build_initial_monmap 2024-08-19T12:49:12.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.370+0000 7f83ef772640 10 monclient: monmap: 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:12.379 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.370+0000 7f83ef772640 1 librados: starting msgr at 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.370+0000 7f83ef772640 1 librados: starting objecter 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.370+0000 7f83ef772640 1 Processor -- start 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 1 -- start start 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 1 librados: setting wanted keys 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 1 librados: calling monclient init 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 10 monclient: init 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 10 monclient: _add_conns ranks=[0,3,1,4,6,5,2,7,8] 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 10 monclient(hunting): picked mon.noname-a con 0x7f83e806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 10 monclient(hunting): picked mon.noname-g con 0x7f83e80a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:12.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 10 monclient(hunting): picked mon.noname-f con 0x7f83e80a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:12.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e80a50a0 con 0x7f83e806e7b0 2024-08-19T12:49:12.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e80a1990 con 0x7f83e80a5d50 2024-08-19T12:49:12.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e8134a30 con 0x7f83e80a5920 2024-08-19T12:49:12.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:12.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ef772640 10 monclient(hunting): authenticate will time out at 1346.458579s 2024-08-19T12:49:12.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ed50e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f83e806e7b0 0x7f83e81ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39772/0 (socket says 172.21.15.26:39772) 2024-08-19T12:49:12.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.371+0000 7f83ed50e640 1 -- 172.21.15.26:0/2606834722 learned_addr learned my addr 172.21.15.26:0/2606834722 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ed50e640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.26:6789/0 conn(0x7f83e806e7b0 legacy=0x7f83e81ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ddffb640 10 client.?.objecter ms_handle_connect 0x7f83e806e7b0 2024-08-19T12:49:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ddffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ddffb640 10 client.?.objecter ms_handle_connect 0x7f83e80a5920 2024-08-19T12:49:12.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ecd0d640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.104:6790/0 conn(0x7f83e80a5920 legacy=0x7f83e81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ddffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ddffb640 10 client.?.objecter ms_handle_connect 0x7f83e80a5d50 2024-08-19T12:49:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ddffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83edd0f640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.104:6789/0 conn(0x7f83e80a5d50 legacy=0x7f83e81d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 122318374 0 0) 0x7f83e80a50a0 con 0x7f83e806e7b0 2024-08-19T12:49:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ddffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ddffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ddffb640 10 monclient(hunting): my global_id is 145035 2024-08-19T12:49:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.373+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83cc004eb0 con 0x7f83e806e7b0 2024-08-19T12:49:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2502540312 0 0) 0x7f83e8134a30 con 0x7f83e80a5920 2024-08-19T12:49:12.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient(hunting): my global_id is 154542 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83e80a50a0 con 0x7f83e80a5920 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2584874860 0 0) 0x7f83e80a1990 con 0x7f83e80a5d50 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient(hunting): my global_id is 27550 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83e8134a30 con 0x7f83e80a5d50 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1640252692 0 0) 0x7f83cc004eb0 con 0x7f83e806e7b0 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83e80a1990 con 0x7f83e806e7b0 2024-08-19T12:49:12.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f83e4004e80 con 0x7f83e806e7b0 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient(hunting): dump: 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.389 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1895821507 0 0) 0x7f83e8134a30 con 0x7f83e80a5d50 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83cc004eb0 con 0x7f83e80a5d50 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3821152684 0 0) 0x7f83e80a1990 con 0x7f83e806e7b0 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.104:6790/0 conn(0x7f83e80a5920 legacy=0x7f83e81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.104:6789/0 conn(0x7f83e80a5d50 legacy=0x7f83e81d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient: _finish_hunting 0 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 monclient: found mon.a 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e80a0eb0 con 0x7f83e806e7b0 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient: _finish_auth 0 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient: _check_auth_tickets 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ef772640 5 monclient: authenticate success, global_id 145035 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ef772640 10 monclient: _renew_subs 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ef772640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ef772640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f83e80a28f0 con 0x7f83e806e7b0 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f83e4005240 con 0x7f83e806e7b0 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f83e4005730 con 0x7f83e806e7b0 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ddffb640 10 monclient: dump: 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.392 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83df7fe640 4 set_mon_vals no callback set 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ef772640 10 client.145035.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ef772640 10 monclient: _renew_subs 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ef772640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ef772640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f83e80a07f0 con 0x7f83e806e7b0 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.374+0000 7f83ef772640 1 librados: init done 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.375+0000 7f83ef772640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:12.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.375+0000 7f83ef772640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f83e80a0b30 con 0x7f83e802e340 2024-08-19T12:49:12.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.375+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f83e4011b90 con 0x7f83e806e7b0 2024-08-19T12:49:12.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.375+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(406..406 src has 311..406) ==== 5413+0+0 (unknown 2461991952 0 0) 0x7f83e404e260 con 0x7f83e806e7b0 2024-08-19T12:49:12.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.375+0000 7f83ddffb640 10 client.145035.objecter ms_dispatch 0x7f83e8000b90 osd_map(406..406 src has 311..406) 2024-08-19T12:49:12.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.375+0000 7f83ddffb640 3 client.145035.objecter handle_osd_map got epochs [406,406] > 0 2024-08-19T12:49:12.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.375+0000 7f83ddffb640 3 client.145035.objecter handle_osd_map decoding full epoch 406 2024-08-19T12:49:12.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.375+0000 7f83ddffb640 20 client.145035.objecter dump_active .. 0 homeless 2024-08-19T12:49:12.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.375+0000 7f83edd0f640 10 monclient: get_auth_request con 0x7f83cc041100 auth_method 0 2024-08-19T12:49:12.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.377+0000 7f83ecd0d640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.104:6789/0 conn(0x7f83e802e340 legacy=0x7f83e8063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.377+0000 7f83ddffb640 10 client.145035.objecter ms_handle_connect 0x7f83e802e340 2024-08-19T12:49:12.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.377+0000 7f83ddffb640 10 client.145035.objecter resend_mon_ops 2024-08-19T12:49:12.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.377+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1654590324 0 0) 0x7f83e80a0b30 con 0x7f83e802e340 2024-08-19T12:49:12.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.377+0000 7f83ddffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.377+0000 7f83ddffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.377+0000 7f83ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.377+0000 7f83ddffb640 10 monclient(hunting): my global_id is 27559 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.377+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83e80a1990 con 0x7f83e802e340 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2459735860 0 0) 0x7f83e80a1990 con 0x7f83e802e340 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83e80a0b30 con 0x7f83e802e340 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f83d8003e10 con 0x7f83e802e340 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 10 monclient: dump: 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.395 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 10 client.145035.objecter ms_handle_connect 0x7f83cc041100 2024-08-19T12:49:12.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1611726781 0 0) 0x7f83e80a0b30 con 0x7f83e802e340 2024-08-19T12:49:12.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f83e80a4ed0 con 0x7f83e802e340 2024-08-19T12:49:12.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f83d8004010 con 0x7f83e802e340 2024-08-19T12:49:12.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:12.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:12.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.104:6789/0 conn(0x7f83e802e340 legacy=0x7f83e8063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.378+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.104:6789/0 conn(0x7f83e802e340 legacy=0x7f83e8063a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.384+0000 7f83ef772640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:12.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.384+0000 7f83ef772640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f83e8125a20 con 0x7f83e81cebc0 2024-08-19T12:49:12.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ecd0d640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.104:6789/0 conn(0x7f83e81cebc0 legacy=0x7f83e8125310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 10 client.145035.objecter ms_handle_connect 0x7f83e81cebc0 2024-08-19T12:49:12.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 10 client.145035.objecter resend_mon_ops 2024-08-19T12:49:12.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3959245018 0 0) 0x7f83e8125a20 con 0x7f83e81cebc0 2024-08-19T12:49:12.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 10 monclient(hunting): my global_id is 27568 2024-08-19T12:49:12.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83e80a0b30 con 0x7f83e81cebc0 2024-08-19T12:49:12.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2833237558 0 0) 0x7f83e80a0b30 con 0x7f83e81cebc0 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83e8125a20 con 0x7f83e81cebc0 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f83d8005b00 con 0x7f83e81cebc0 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.387+0000 7f83ddffb640 10 monclient: dump: 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.400 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4103154191 0 0) 0x7f83e8125a20 con 0x7f83e81cebc0 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f83e802e720 con 0x7f83e81cebc0 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 4186386359) 0x7f83d8007e30 con 0x7f83e81cebc0 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ddffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ddffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:12.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.104:6789/0 conn(0x7f83e81cebc0 legacy=0x7f83e8125310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ddffb640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.104:6789/0 conn(0x7f83e81cebc0 legacy=0x7f83e8125310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.402 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 665392, 2024-08-19T12:49:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:12.402 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.403 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:12.404 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.405 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:12.406 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.407 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:12.408 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.410 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.412 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:12.413 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.414 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.416 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:12.417 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:12.419 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:12.419 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.419 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.419 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.423 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.424 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:49:12.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.425 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:12.425 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ef772640 10 librados: watch_flush enter 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ef772640 10 librados: watch_flush exit 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ef772640 20 client.145035.objecter shutdown clearing up homeless session... 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ef772640 10 client.145035.objecter successfully canceled tick 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ef772640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.104:6800/899887506 conn(0x7f83cc041100 legacy=0x7f83cc0435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ef772640 10 monclient: shutdown 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ef772640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ef772640 1 -- 172.21.15.26:0/2606834722 >> v1:172.21.15.26:6789/0 conn(0x7f83e806e7b0 legacy=0x7f83e81ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83edd0f640 1 -- 172.21.15.26:0/2606834722 reap_dead start 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83edd0f640 1 -- 172.21.15.26:0/2606834722 reap_dead start 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ef772640 1 -- 172.21.15.26:0/2606834722 shutdown_connections 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ef772640 1 -- 172.21.15.26:0/2606834722 >> 172.21.15.26:0/2606834722 conn(0x7f83e80a8740 msgr2=0x7f83e806dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:12.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ef772640 1 -- 172.21.15.26:0/2606834722 shutdown_connections 2024-08-19T12:49:12.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.388+0000 7f83ef772640 1 -- 172.21.15.26:0/2606834722 wait complete. 2024-08-19T12:49:12.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.389+0000 7f83ef772640 1 librados: shutdown 2024-08-19T12:49:12.428 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:12.428 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:49:12.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:49:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.528+0000 7fc59075c640 10 monclient: get_monmap_and_config 2024-08-19T12:49:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.528+0000 7fc59075c640 10 monclient: build_initial_monmap 2024-08-19T12:49:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.528+0000 7fc59075c640 10 monclient: monmap: 2024-08-19T12:49:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:12.529137+0000 2024-08-19T12:49:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:12.529137+0000 2024-08-19T12:49:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.529 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:12.530 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:12.530 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:12.530 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:12.530 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:12.530 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:12.530 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:12.530 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:12.530 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:12.530 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.528+0000 7fc59075c640 1 Processor -- start 2024-08-19T12:49:12.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.528+0000 7fc59075c640 1 -- start start 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.528+0000 7fc59075c640 10 monclient: init 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc59075c640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc59075c640 10 monclient: _add_conns ranks=[2,0,8,4,1,3,7,5,6] 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc59075c640 10 monclient(hunting): picked mon.noname-b con 0x7fc588073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc59075c640 10 monclient(hunting): picked mon.noname-a con 0x7fc5880a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc59075c640 10 monclient(hunting): picked mon.noname-e con 0x7fc58813f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc59075c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc58809dae0 con 0x7fc5880a57a0 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc59075c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc588136910 con 0x7fc588073470 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc59075c640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc58809dd50 con 0x7fc58813f2e0 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc59075c640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc59075c640 10 monclient(hunting): authenticate will time out at 1346.616569s 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc58e4f8640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fc588073470 0x7fc588136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59292/0 (socket says 172.21.15.26:59292) 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc58e4f8640 1 -- 172.21.15.26:0/87969343 learned_addr learned my addr 172.21.15.26:0/87969343 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:12.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.529+0000 7fc58ecf9640 1 --1- 172.21.15.26:0/87969343 >> v1:172.21.15.26:6793/0 conn(0x7fc58813f2e0 0x7fc5881416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:35690/0 (socket says 172.21.15.26:35690) 2024-08-19T12:49:12.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.531+0000 7fc58dcf7640 1 -- 172.21.15.26:0/87969343 >> v1:172.21.15.26:6789/0 conn(0x7fc5880a57a0 legacy=0x7fc5880a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.531+0000 7fc58e4f8640 1 -- 172.21.15.26:0/87969343 >> v1:172.21.15.26:6790/0 conn(0x7fc588073470 legacy=0x7fc588136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.531+0000 7fc58ecf9640 1 -- 172.21.15.26:0/87969343 >> v1:172.21.15.26:6793/0 conn(0x7fc58813f2e0 legacy=0x7fc5881416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 829644015 0 0) 0x7fc58809dae0 con 0x7fc5880a57a0 2024-08-19T12:49:12.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient(hunting): my global_id is 145044 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc574003520 con 0x7fc5880a57a0 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 872744477 0 0) 0x7fc588136910 con 0x7fc588073470 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient(hunting): my global_id is 26705 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc58809dae0 con 0x7fc588073470 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 101575099 0 0) 0x7fc58809dd50 con 0x7fc58813f2e0 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient(hunting): my global_id is 174140 2024-08-19T12:49:12.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc588136910 con 0x7fc58813f2e0 2024-08-19T12:49:12.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1454765443 0 0) 0x7fc588136910 con 0x7fc58813f2e0 2024-08-19T12:49:12.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 >> v1:172.21.15.26:6790/0 conn(0x7fc588073470 legacy=0x7fc588136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 >> v1:172.21.15.26:6789/0 conn(0x7fc5880a57a0 legacy=0x7fc5880a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient: _finish_hunting 0 2024-08-19T12:49:12.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 monclient: found mon.noname-e 2024-08-19T12:49:12.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:12.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:49:12.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58809e030 con 0x7fc58813f2e0 2024-08-19T12:49:12.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient: _finish_auth 0 2024-08-19T12:49:12.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient: _check_auth_tickets 2024-08-19T12:49:12.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:12.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc580001d80 con 0x7fc58813f2e0 2024-08-19T12:49:12.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:49:12.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient: dump: 2024-08-19T12:49:12.539 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.539 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.539 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.539 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.539 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.539 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.539 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.539 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.539 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc59075c640 5 monclient: authenticate success, global_id 174140 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc59075c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc580001fd0 con 0x7fc58813f2e0 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 1 -- 172.21.15.26:0/87969343 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc580004ea0 con 0x7fc58813f2e0 2024-08-19T12:49:12.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc59075c640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc59075c640 4 set_mon_vals no callback set 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc58d4f6640 10 monclient: dump: 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.541 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.532+0000 7fc59075c640 10 monclient: shutdown 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.533+0000 7fc59075c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.533+0000 7fc59075c640 1 -- 172.21.15.26:0/87969343 >> v1:172.21.15.26:6793/0 conn(0x7fc58813f2e0 legacy=0x7fc5881416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.533+0000 7fc59075c640 1 -- 172.21.15.26:0/87969343 shutdown_connections 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.533+0000 7fc59075c640 1 -- 172.21.15.26:0/87969343 >> 172.21.15.26:0/87969343 conn(0x7fc5880abff0 msgr2=0x7fc5880ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.533+0000 7fc59075c640 1 -- 172.21.15.26:0/87969343 shutdown_connections 2024-08-19T12:49:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.533+0000 7fc59075c640 1 -- 172.21.15.26:0/87969343 wait complete. 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.533+0000 7fc59075c640 10 monclient: build_initial_monmap 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.533+0000 7fc59075c640 10 monclient: monmap: 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:12.543 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.533+0000 7fc59075c640 1 librados: starting msgr at 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.533+0000 7fc59075c640 1 librados: starting objecter 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 1 Processor -- start 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 1 -- start start 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 1 librados: setting wanted keys 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 1 librados: calling monclient init 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 10 monclient: init 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:12.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 10 monclient: _add_conns ranks=[7,0,5,3,1,4,6,8,2] 2024-08-19T12:49:12.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 10 monclient(hunting): picked mon.noname-i con 0x7fc588073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:12.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 10 monclient(hunting): picked mon.noname-a con 0x7fc5880a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:12.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 10 monclient(hunting): picked mon.noname-h con 0x7fc58813f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:12.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc588142b10 con 0x7fc5880a57a0 2024-08-19T12:49:12.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc58812f790 con 0x7fc58813f2e0 2024-08-19T12:49:12.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc58812ec50 con 0x7fc588073470 2024-08-19T12:49:12.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:12.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc59075c640 10 monclient(hunting): authenticate will time out at 1346.621578s 2024-08-19T12:49:12.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc58dcf7640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc5880a57a0 0x7fc58812e540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39788/0 (socket says 172.21.15.26:39788) 2024-08-19T12:49:12.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.534+0000 7fc58dcf7640 1 -- 172.21.15.26:0/1115405533 learned_addr learned my addr 172.21.15.26:0/1115405533 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:12.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.536+0000 7fc58dcf7640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.26:6789/0 conn(0x7fc5880a57a0 legacy=0x7fc58812e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.536+0000 7fc57effd640 10 client.?.objecter ms_handle_connect 0x7fc5880a57a0 2024-08-19T12:49:12.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.536+0000 7fc57effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.536+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2341696217 0 0) 0x7fc588142b10 con 0x7fc5880a57a0 2024-08-19T12:49:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.536+0000 7fc57effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.536+0000 7fc57effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.536+0000 7fc57effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.536+0000 7fc57effd640 10 monclient(hunting): my global_id is 145053 2024-08-19T12:49:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.536+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc56c004ee0 con 0x7fc5880a57a0 2024-08-19T12:49:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 client.?.objecter ms_handle_connect 0x7fc58813f2e0 2024-08-19T12:49:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc58ecf9640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.104:6791/0 conn(0x7fc58813f2e0 legacy=0x7fc5881d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 client.?.objecter ms_handle_connect 0x7fc588073470 2024-08-19T12:49:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc58e4f8640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.104:6792/0 conn(0x7fc588073470 legacy=0x7fc58812de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 801659651 0 0) 0x7fc56c004ee0 con 0x7fc5880a57a0 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc588142b10 con 0x7fc5880a57a0 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc578003610 con 0x7fc5880a57a0 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient(hunting): dump: 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.551 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2101574961 0 0) 0x7fc58812ec50 con 0x7fc588073470 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient(hunting): my global_id is 135664 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc56c004ee0 con 0x7fc588073470 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378669243 0 0) 0x7fc58812f790 con 0x7fc58813f2e0 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient(hunting): my global_id is 26771 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc58812ec50 con 0x7fc58813f2e0 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3238485568 0 0) 0x7fc588142b10 con 0x7fc5880a57a0 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.104:6792/0 conn(0x7fc588073470 legacy=0x7fc58812de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.104:6791/0 conn(0x7fc58813f2e0 legacy=0x7fc5881d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient: _finish_hunting 0 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 monclient: found mon.a 2024-08-19T12:49:12.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:12.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:12.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc588131810 con 0x7fc5880a57a0 2024-08-19T12:49:12.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient: _finish_auth 0 2024-08-19T12:49:12.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient: _check_auth_tickets 2024-08-19T12:49:12.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:12.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc59075c640 5 monclient: authenticate success, global_id 145053 2024-08-19T12:49:12.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc59075c640 10 monclient: _renew_subs 2024-08-19T12:49:12.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc59075c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:12.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc59075c640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc588131af0 con 0x7fc5880a57a0 2024-08-19T12:49:12.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc59075c640 10 client.145053.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:12.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc59075c640 10 monclient: _renew_subs 2024-08-19T12:49:12.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc59075c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc59075c640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc5881302f0 con 0x7fc5880a57a0 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc59075c640 1 librados: init done 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc578003bd0 con 0x7fc5880a57a0 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc578004140 con 0x7fc5880a57a0 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc58ccf5640 4 set_mon_vals no callback set 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.537+0000 7fc57effd640 10 monclient: dump: 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.555 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.538+0000 7fc59075c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.538+0000 7fc59075c640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc5700065c0 con 0x7fc570002860 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.538+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fc5780042e0 con 0x7fc5880a57a0 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.538+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(406..406 src has 311..406) ==== 5413+0+0 (unknown 2461991952 0 0) 0x7fc57804e2b0 con 0x7fc5880a57a0 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.538+0000 7fc57effd640 10 client.145053.objecter ms_dispatch 0x7fc588000b90 osd_map(406..406 src has 311..406) 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.538+0000 7fc57effd640 3 client.145053.objecter handle_osd_map got epochs [406,406] > 0 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.538+0000 7fc57effd640 3 client.145053.objecter handle_osd_map decoding full epoch 406 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.538+0000 7fc57effd640 20 client.145053.objecter dump_active .. 0 homeless 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.538+0000 7fc58ecf9640 10 monclient: get_auth_request con 0x7fc56c041250 auth_method 0 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc58e4f8640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.104:6790/0 conn(0x7fc570002860 legacy=0x7fc570004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 10 client.145053.objecter ms_handle_connect 0x7fc570002860 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 10 client.145053.objecter resend_mon_ops 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2642290232 0 0) 0x7fc5700065c0 con 0x7fc570002860 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 10 monclient(hunting): my global_id is 154551 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc588142b10 con 0x7fc570002860 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3992966379 0 0) 0x7fc588142b10 con 0x7fc570002860 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5700065c0 con 0x7fc570002860 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc584001b30 con 0x7fc570002860 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 10 monclient: dump: 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.558 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.540+0000 7fc57effd640 10 client.145053.objecter ms_handle_connect 0x7fc56c041250 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.541+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2656758831 0 0) 0x7fc5700065c0 con 0x7fc570002860 2024-08-19T12:49:12.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.541+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc57000a160 con 0x7fc570002860 2024-08-19T12:49:12.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.541+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc584004a90 con 0x7fc570002860 2024-08-19T12:49:12.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.541+0000 7fc57effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:12.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.541+0000 7fc57effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:12.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.541+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.104:6790/0 conn(0x7fc570002860 legacy=0x7fc570004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.541+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.104:6790/0 conn(0x7fc570002860 legacy=0x7fc570004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.547+0000 7fc59075c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:12.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.547+0000 7fc59075c640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc570009e30 con 0x7fc570009110 2024-08-19T12:49:12.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.550+0000 7fc57effd640 10 client.145053.objecter ms_handle_connect 0x7fc570009110 2024-08-19T12:49:12.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.550+0000 7fc57effd640 10 client.145053.objecter resend_mon_ops 2024-08-19T12:49:12.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.550+0000 7fc58e4f8640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.104:6790/0 conn(0x7fc570009110 legacy=0x7fc570009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1491222749 0 0) 0x7fc570009e30 con 0x7fc570009110 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 10 monclient(hunting): my global_id is 154560 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5700065c0 con 0x7fc570009110 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1058973417 0 0) 0x7fc5700065c0 con 0x7fc570009110 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc570009e30 con 0x7fc570009110 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc584005280 con 0x7fc570009110 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 10 monclient: dump: 2024-08-19T12:49:12.562 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.563 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.564 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3376951246 0 0) 0x7fc570009e30 con 0x7fc570009110 2024-08-19T12:49:12.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc57000c3f0 con 0x7fc570009110 2024-08-19T12:49:12.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1194415631) 0x7fc5840075e0 con 0x7fc570009110 2024-08-19T12:49:12.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:12.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:12.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.104:6790/0 conn(0x7fc570009110 legacy=0x7fc570009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc57effd640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.104:6790/0 conn(0x7fc570009110 legacy=0x7fc570009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:12.564 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 264283, 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:12.565 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.566 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.567 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:12.568 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.569 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.570 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.571 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.572 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.573 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:12.575 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:12.576 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:12.577 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.578 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.579 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:12.580 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.583 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.584 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.584 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:12.584 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:12.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc59075c640 10 librados: watch_flush enter 2024-08-19T12:49:12.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc59075c640 10 librados: watch_flush exit 2024-08-19T12:49:12.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc59075c640 20 client.145053.objecter shutdown clearing up homeless session... 2024-08-19T12:49:12.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc59075c640 10 client.145053.objecter successfully canceled tick 2024-08-19T12:49:12.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc59075c640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.104:6800/899887506 conn(0x7fc56c041250 legacy=0x7fc56c043710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc59075c640 10 monclient: shutdown 2024-08-19T12:49:12.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc59075c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:12.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc59075c640 1 -- 172.21.15.26:0/1115405533 >> v1:172.21.15.26:6789/0 conn(0x7fc5880a57a0 legacy=0x7fc58812e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc58ecf9640 1 -- 172.21.15.26:0/1115405533 reap_dead start 2024-08-19T12:49:12.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc58ecf9640 1 -- 172.21.15.26:0/1115405533 reap_dead start 2024-08-19T12:49:12.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc59075c640 1 -- 172.21.15.26:0/1115405533 shutdown_connections 2024-08-19T12:49:12.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc59075c640 1 -- 172.21.15.26:0/1115405533 >> 172.21.15.26:0/1115405533 conn(0x7fc5880abff0 msgr2=0x7fc5880a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:12.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.553+0000 7fc59075c640 1 -- 172.21.15.26:0/1115405533 shutdown_connections 2024-08-19T12:49:12.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.554+0000 7fc59075c640 1 -- 172.21.15.26:0/1115405533 wait complete. 2024-08-19T12:49:12.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.554+0000 7fc59075c640 1 librados: shutdown 2024-08-19T12:49:12.587 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:12.587 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:49:12.587 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:49:12.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.686+0000 7fecda92f640 10 monclient: get_monmap_and_config 2024-08-19T12:49:12.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.686+0000 7fecda92f640 10 monclient: build_initial_monmap 2024-08-19T12:49:12.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.686+0000 7fecda92f640 10 monclient: monmap: 2024-08-19T12:49:12.687 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:12.687 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.687 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:12.687437+0000 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:12.687437+0000 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:12.688 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 1 Processor -- start 2024-08-19T12:49:12.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 1 -- start start 2024-08-19T12:49:12.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 10 monclient: init 2024-08-19T12:49:12.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:12.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 10 monclient: _add_conns ranks=[8,3,2,1,6,5,4,7,0] 2024-08-19T12:49:12.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 10 monclient(hunting): picked mon.noname-e con 0x7fecd406e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:12.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 10 monclient(hunting): picked mon.noname-g con 0x7fecd40a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:12.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 10 monclient(hunting): picked mon.noname-b con 0x7fecd40a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:12.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecd4136790 con 0x7fecd40a5d50 2024-08-19T12:49:12.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecd409ddf0 con 0x7fecd40a5920 2024-08-19T12:49:12.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecd409e0d0 con 0x7fecd406e7b0 2024-08-19T12:49:12.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:12.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.687+0000 7fecda92f640 10 monclient(hunting): authenticate will time out at 1346.774898s 2024-08-19T12:49:12.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.688+0000 7fecd3fff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fecd406e7b0 0x7fecd4136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:35704/0 (socket says 172.21.15.26:35704) 2024-08-19T12:49:12.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.688+0000 7fecd3fff640 1 -- 172.21.15.26:0/3506664217 learned_addr learned my addr 172.21.15.26:0/3506664217 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:12.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.688+0000 7fecd8ecc640 1 --1- 172.21.15.26:0/3506664217 >> v1:172.21.15.26:6790/0 conn(0x7fecd40a5d50 0x7fecd4141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59306/0 (socket says 172.21.15.26:59306) 2024-08-19T12:49:12.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd3fff640 1 -- 172.21.15.26:0/3506664217 >> v1:172.21.15.26:6793/0 conn(0x7fecd406e7b0 legacy=0x7fecd4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd8ecc640 1 -- 172.21.15.26:0/3506664217 >> v1:172.21.15.26:6790/0 conn(0x7fecd40a5d50 legacy=0x7fecd4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 1 -- 172.21.15.26:0/3506664217 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4279506320 0 0) 0x7fecd409e0d0 con 0x7fecd406e7b0 2024-08-19T12:49:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient(hunting): my global_id is 174149 2024-08-19T12:49:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 1 -- 172.21.15.26:0/3506664217 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecbc003520 con 0x7fecd406e7b0 2024-08-19T12:49:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 1 -- 172.21.15.26:0/3506664217 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1858707247 0 0) 0x7fecd4136790 con 0x7fecd40a5d50 2024-08-19T12:49:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient(hunting): my global_id is 26714 2024-08-19T12:49:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 1 -- 172.21.15.26:0/3506664217 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecd409e0d0 con 0x7fecd40a5d50 2024-08-19T12:49:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd37fe640 1 -- 172.21.15.26:0/3506664217 >> v1:172.21.15.104:6790/0 conn(0x7fecd40a5920 legacy=0x7fecd413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 1 -- 172.21.15.26:0/3506664217 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3028524209 0 0) 0x7fecbc003520 con 0x7fecd406e7b0 2024-08-19T12:49:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 1 -- 172.21.15.26:0/3506664217 >> v1:172.21.15.104:6790/0 conn(0x7fecd40a5920 legacy=0x7fecd413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 1 -- 172.21.15.26:0/3506664217 >> v1:172.21.15.26:6790/0 conn(0x7fecd40a5d50 legacy=0x7fecd4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:49:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 1 monclient: found mon.noname-e 2024-08-19T12:49:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:49:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 1 -- 172.21.15.26:0/3506664217 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd409e3b0 con 0x7fecd406e7b0 2024-08-19T12:49:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient: _finish_auth 0 2024-08-19T12:49:12.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 1 -- 172.21.15.26:0/3506664217 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fecc4001ae0 con 0x7fecd406e7b0 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient: dump: 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.697 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecda92f640 5 monclient: authenticate success, global_id 174149 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecda92f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 1 -- 172.21.15.26:0/3506664217 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fecc40048a0 con 0x7fecd406e7b0 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:12.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 1 -- 172.21.15.26:0/3506664217 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fecc4004e10 con 0x7fecd406e7b0 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecda92f640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecda92f640 4 set_mon_vals no callback set 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.690+0000 7fecd2ffd640 10 monclient: dump: 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.699 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.691+0000 7fecda92f640 10 monclient: shutdown 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.691+0000 7fecda92f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.691+0000 7fecda92f640 1 -- 172.21.15.26:0/3506664217 >> v1:172.21.15.26:6793/0 conn(0x7fecd406e7b0 legacy=0x7fecd4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.691+0000 7fecda92f640 1 -- 172.21.15.26:0/3506664217 shutdown_connections 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.691+0000 7fecda92f640 1 -- 172.21.15.26:0/3506664217 >> 172.21.15.26:0/3506664217 conn(0x7fecd40a8740 msgr2=0x7fecd40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:12.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.691+0000 7fecda92f640 1 -- 172.21.15.26:0/3506664217 shutdown_connections 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.691+0000 7fecda92f640 1 -- 172.21.15.26:0/3506664217 wait complete. 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.691+0000 7fecda92f640 10 monclient: build_initial_monmap 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.691+0000 7fecda92f640 10 monclient: monmap: 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:12.701 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 1 librados: starting msgr at 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 1 librados: starting objecter 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 1 Processor -- start 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 1 -- start start 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 1 librados: setting wanted keys 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 1 librados: calling monclient init 2024-08-19T12:49:12.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 10 monclient: init 2024-08-19T12:49:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 10 monclient: _add_conns ranks=[2,4,8,0,1,3,5,6,7] 2024-08-19T12:49:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 10 monclient(hunting): picked mon.noname-b con 0x7fecd406e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 10 monclient(hunting): picked mon.noname-c con 0x7fecd40a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 10 monclient(hunting): picked mon.noname-e con 0x7fecd40a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecd40a0db0 con 0x7fecd406e7b0 2024-08-19T12:49:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecd4134740 con 0x7fecd40a5920 2024-08-19T12:49:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecd4134c90 con 0x7fecd40a5d50 2024-08-19T12:49:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecda92f640 10 monclient(hunting): authenticate will time out at 1346.779853s 2024-08-19T12:49:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecd3fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fecd406e7b0 0x7fecd4132030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59322/0 (socket says 172.21.15.26:59322) 2024-08-19T12:49:12.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.692+0000 7fecd3fff640 1 -- 172.21.15.26:0/2538323709 learned_addr learned my addr 172.21.15.26:0/2538323709 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:12.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd3fff640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.26:6790/0 conn(0x7fecd406e7b0 legacy=0x7fecd4132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd37fe640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.26:6791/0 conn(0x7fecd40a5920 legacy=0x7fecd4132740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 client.?.objecter ms_handle_connect 0x7fecd406e7b0 2024-08-19T12:49:12.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 client.?.objecter ms_handle_connect 0x7fecd40a5920 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 client.?.objecter ms_handle_connect 0x7fecd40a5d50 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd8ecc640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.26:6793/0 conn(0x7fecd40a5d50 legacy=0x7fecd41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3356914505 0 0) 0x7fecd40a0db0 con 0x7fecd406e7b0 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): my global_id is 26723 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecc0004eb0 con 0x7fecd406e7b0 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1137653470 0 0) 0x7fecd4134740 con 0x7fecd40a5920 2024-08-19T12:49:12.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): my global_id is 27049 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecd40a0db0 con 0x7fecd40a5920 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1450472304 0 0) 0x7fecd4134c90 con 0x7fecd40a5d50 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): my global_id is 174158 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecd4134740 con 0x7fecd40a5d50 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 896303660 0 0) 0x7fecc0004eb0 con 0x7fecd406e7b0 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecd4134c90 con 0x7fecd406e7b0 2024-08-19T12:49:12.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2477292167 0 0) 0x7fecd40a0db0 con 0x7fecd40a5920 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecc0004eb0 con 0x7fecd40a5920 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2914919113 0 0) 0x7fecd4134740 con 0x7fecd40a5d50 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecd40a0db0 con 0x7fecd40a5d50 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fecc4004d90 con 0x7fecd406e7b0 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): dump: 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.711 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fecc8001cd0 con 0x7fecd40a5920 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:12.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): dump: 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.713 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7feccc001fc0 con 0x7fecd40a5d50 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient(hunting): dump: 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.714 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 972718126 0 0) 0x7fecd4134c90 con 0x7fecd406e7b0 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.26:6793/0 conn(0x7fecd40a5d50 legacy=0x7fecd41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.26:6791/0 conn(0x7fecd40a5920 legacy=0x7fecd4132740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 monclient: found mon.b 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd40a07f0 con 0x7fecd406e7b0 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient: _finish_auth 0 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecda92f640 5 monclient: authenticate success, global_id 26723 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecda92f640 10 monclient: _renew_subs 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecda92f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecda92f640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fecd40a0ad0 con 0x7fecd406e7b0 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecda92f640 10 client.26723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:12.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecda92f640 10 monclient: _renew_subs 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecda92f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecda92f640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fecd4132e50 con 0x7fecd406e7b0 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecda92f640 1 librados: init done 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fecc4001d30 con 0x7fecd406e7b0 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fecc40048a0 con 0x7fecd406e7b0 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd1ffb640 4 set_mon_vals no callback set 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.695+0000 7fecd0ff9640 10 monclient: dump: 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.717 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.718 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.696+0000 7fecda92f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.696+0000 7fecda92f640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fecb40065c0 con 0x7fecb4002860 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.697+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fecc4011a00 con 0x7fecd406e7b0 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.697+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(406..406 src has 311..406) ==== 5413+0+0 (unknown 2461991952 0 0) 0x7fecc404e290 con 0x7fecd406e7b0 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.697+0000 7fecd0ff9640 10 client.26723.objecter ms_dispatch 0x7fecd4000b90 osd_map(406..406 src has 311..406) 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.697+0000 7fecd0ff9640 3 client.26723.objecter handle_osd_map got epochs [406,406] > 0 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.697+0000 7fecd0ff9640 3 client.26723.objecter handle_osd_map decoding full epoch 406 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.697+0000 7fecd0ff9640 20 client.26723.objecter dump_active .. 0 homeless 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.697+0000 7fecd8ecc640 10 monclient: get_auth_request con 0x7fecc0041be0 auth_method 0 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 10 client.26723.objecter ms_handle_connect 0x7fecb4002860 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 10 client.26723.objecter resend_mon_ops 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd37fe640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.104:6791/0 conn(0x7fecb4002860 legacy=0x7fecb4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2453021404 0 0) 0x7fecb40065c0 con 0x7fecb4002860 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 10 monclient(hunting): my global_id is 26780 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecd4134c90 con 0x7fecb4002860 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3475673585 0 0) 0x7fecd4134c90 con 0x7fecb4002860 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecb40065c0 con 0x7fecb4002860 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fecc8004070 con 0x7fecb4002860 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 10 monclient: dump: 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.720 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.721 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 10 client.26723.objecter ms_handle_connect 0x7fecc0041be0 2024-08-19T12:49:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3949419377 0 0) 0x7fecb40065c0 con 0x7fecb4002860 2024-08-19T12:49:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.699+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fecb400a160 con 0x7fecb4002860 2024-08-19T12:49:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.700+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fecc8004290 con 0x7fecb4002860 2024-08-19T12:49:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.700+0000 7fecd0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.700+0000 7fecd0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.700+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.104:6791/0 conn(0x7fecb4002860 legacy=0x7fecb4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.700+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.104:6791/0 conn(0x7fecb4002860 legacy=0x7fecb4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.706+0000 7fecda92f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:12.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.706+0000 7fecda92f640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fecb4009e50 con 0x7fecb4009110 2024-08-19T12:49:12.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd37fe640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.104:6791/0 conn(0x7fecb4009110 legacy=0x7fecb40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 10 client.26723.objecter ms_handle_connect 0x7fecb4009110 2024-08-19T12:49:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 10 client.26723.objecter resend_mon_ops 2024-08-19T12:49:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3993620519 0 0) 0x7fecb4009e50 con 0x7fecb4009110 2024-08-19T12:49:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 10 monclient(hunting): my global_id is 26789 2024-08-19T12:49:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecb40065c0 con 0x7fecb4009110 2024-08-19T12:49:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3729048813 0 0) 0x7fecb40065c0 con 0x7fecb4009110 2024-08-19T12:49:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecb4009e50 con 0x7fecb4009110 2024-08-19T12:49:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fecc8005cf0 con 0x7fecb4009110 2024-08-19T12:49:12.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 10 monclient: dump: 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.725 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.726 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1414766792 0 0) 0x7fecb4009e50 con 0x7fecb4009110 2024-08-19T12:49:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.709+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fecb400c3f0 con 0x7fecb4009110 2024-08-19T12:49:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7878 (unknown 0 0 701679281) 0x7fecc8007260 con 0x7fecb4009110 2024-08-19T12:49:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecd0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecd0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.104:6791/0 conn(0x7fecb4009110 legacy=0x7fecb40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecd0ff9640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.104:6791/0 conn(0x7fecb4009110 legacy=0x7fecb40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.727 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 665732, 2024-08-19T12:49:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:12.727 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.728 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.729 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:12.730 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:12.731 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.732 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.734 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.735 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.736 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.737 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.739 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:12.740 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.741 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.742 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:12.743 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.744 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.745 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.746 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:49:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.747 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:12.747 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 10 librados: watch_flush enter 2024-08-19T12:49:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 10 librados: watch_flush exit 2024-08-19T12:49:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 20 client.26723.objecter shutdown clearing up homeless session... 2024-08-19T12:49:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 10 client.26723.objecter successfully canceled tick 2024-08-19T12:49:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.104:6800/899887506 conn(0x7fecc0041be0 legacy=0x7fecc00440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 10 monclient: shutdown 2024-08-19T12:49:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 1 -- 172.21.15.26:0/2538323709 >> v1:172.21.15.26:6790/0 conn(0x7fecd406e7b0 legacy=0x7fecd4132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecd8ecc640 1 -- 172.21.15.26:0/2538323709 reap_dead start 2024-08-19T12:49:12.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecd8ecc640 1 -- 172.21.15.26:0/2538323709 reap_dead start 2024-08-19T12:49:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 1 -- 172.21.15.26:0/2538323709 shutdown_connections 2024-08-19T12:49:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 1 -- 172.21.15.26:0/2538323709 >> 172.21.15.26:0/2538323709 conn(0x7fecd40a8740 msgr2=0x7fecd406d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 1 -- 172.21.15.26:0/2538323709 shutdown_connections 2024-08-19T12:49:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 1 -- 172.21.15.26:0/2538323709 wait complete. 2024-08-19T12:49:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.710+0000 7fecda92f640 1 librados: shutdown 2024-08-19T12:49:12.750 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:12.750 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:49:12.751 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:49:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.850+0000 7fe1af8a1640 10 monclient: get_monmap_and_config 2024-08-19T12:49:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.850+0000 7fe1af8a1640 10 monclient: build_initial_monmap 2024-08-19T12:49:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.850+0000 7fe1af8a1640 10 monclient: monmap: 2024-08-19T12:49:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:12.851215+0000 2024-08-19T12:49:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:12.851215+0000 2024-08-19T12:49:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:12.851 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:12.852 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:12.852 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.850+0000 7fe1af8a1640 1 Processor -- start 2024-08-19T12:49:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1af8a1640 1 -- start start 2024-08-19T12:49:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1af8a1640 10 monclient: init 2024-08-19T12:49:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1af8a1640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1af8a1640 10 monclient: _add_conns ranks=[2,1,5,3,6,4,0,8,7] 2024-08-19T12:49:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1af8a1640 10 monclient(hunting): picked mon.noname-b con 0x7fe1a806e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1af8a1640 10 monclient(hunting): picked mon.noname-f con 0x7fe1a80a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1af8a1640 10 monclient(hunting): picked mon.noname-h con 0x7fe1a80a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1af8a1640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a8136790 con 0x7fe1a80a5920 2024-08-19T12:49:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1af8a1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a809ddf0 con 0x7fe1a806e7b0 2024-08-19T12:49:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1af8a1640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a809e0d0 con 0x7fe1a80a5d50 2024-08-19T12:49:12.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1af8a1640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:12.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1af8a1640 10 monclient(hunting): authenticate will time out at 1346.938802s 2024-08-19T12:49:12.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1ad63d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe1a806e7b0 0x7fe1a8136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59334/0 (socket says 172.21.15.26:59334) 2024-08-19T12:49:12.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.851+0000 7fe1ad63d640 1 -- 172.21.15.26:0/2587960926 learned_addr learned my addr 172.21.15.26:0/2587960926 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:12.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.853+0000 7fe1ad63d640 1 -- 172.21.15.26:0/2587960926 >> v1:172.21.15.26:6790/0 conn(0x7fe1a806e7b0 legacy=0x7fe1a8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 1 -- 172.21.15.26:0/2587960926 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3225174136 0 0) 0x7fe1a4001450 con 0x7fe1a806e7b0 2024-08-19T12:49:12.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient(hunting): my global_id is 26732 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 1 -- 172.21.15.26:0/2587960926 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe194003520 con 0x7fe1a806e7b0 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe1ade3e640 1 -- 172.21.15.26:0/2587960926 >> v1:172.21.15.104:6791/0 conn(0x7fe1a80a5d50 legacy=0x7fe1a8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe1ace3c640 1 -- 172.21.15.26:0/2587960926 >> v1:172.21.15.104:6789/0 conn(0x7fe1a80a5920 legacy=0x7fe1a813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 1 -- 172.21.15.26:0/2587960926 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 522532092 0 0) 0x7fe194003520 con 0x7fe1a806e7b0 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 1 -- 172.21.15.26:0/2587960926 >> v1:172.21.15.104:6791/0 conn(0x7fe1a80a5d50 legacy=0x7fe1a8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 1 -- 172.21.15.26:0/2587960926 >> v1:172.21.15.104:6789/0 conn(0x7fe1a80a5920 legacy=0x7fe1a813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient: _finish_hunting 0 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 1 monclient: found mon.noname-b 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 1 -- 172.21.15.26:0/2587960926 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a809e3b0 con 0x7fe1a806e7b0 2024-08-19T12:49:12.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient: _finish_auth 0 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient: _check_auth_tickets 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe1af8a1640 5 monclient: authenticate success, global_id 26732 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 1 -- 172.21.15.26:0/2587960926 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a4001cb0 con 0x7fe1a806e7b0 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient: dump: 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.860 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe1af8a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 1 -- 172.21.15.26:0/2587960926 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1a4004990 con 0x7fe1a806e7b0 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:12.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 1 -- 172.21.15.26:0/2587960926 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a4004f00 con 0x7fe1a806e7b0 2024-08-19T12:49:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe19ffff640 10 monclient: dump: 2024-08-19T12:49:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.862 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe1af8a1640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.854+0000 7fe1af8a1640 4 set_mon_vals no callback set 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.855+0000 7fe1af8a1640 10 monclient: shutdown 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.855+0000 7fe1af8a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.855+0000 7fe1af8a1640 1 -- 172.21.15.26:0/2587960926 >> v1:172.21.15.26:6790/0 conn(0x7fe1a806e7b0 legacy=0x7fe1a8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.855+0000 7fe1af8a1640 1 -- 172.21.15.26:0/2587960926 shutdown_connections 2024-08-19T12:49:12.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.855+0000 7fe1af8a1640 1 -- 172.21.15.26:0/2587960926 >> 172.21.15.26:0/2587960926 conn(0x7fe1a80a8740 msgr2=0x7fe1a80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.855+0000 7fe1af8a1640 1 -- 172.21.15.26:0/2587960926 shutdown_connections 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.855+0000 7fe1af8a1640 1 -- 172.21.15.26:0/2587960926 wait complete. 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.855+0000 7fe1af8a1640 10 monclient: build_initial_monmap 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.855+0000 7fe1af8a1640 10 monclient: monmap: 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:12.864 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.855+0000 7fe1af8a1640 1 librados: starting msgr at 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.855+0000 7fe1af8a1640 1 librados: starting objecter 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 1 Processor -- start 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 1 -- start start 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 1 librados: setting wanted keys 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 1 librados: calling monclient init 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 10 monclient: init 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:12.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 10 monclient: _add_conns ranks=[2,5,6,4,1,3,0,7,8] 2024-08-19T12:49:12.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 10 monclient(hunting): picked mon.noname-b con 0x7fe1a806e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:12.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 10 monclient(hunting): picked mon.noname-h con 0x7fe1a80a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:12.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 10 monclient(hunting): picked mon.noname-d con 0x7fe1a80a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:12.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a80a50a0 con 0x7fe1a806e7b0 2024-08-19T12:49:12.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a80a1990 con 0x7fe1a80a5920 2024-08-19T12:49:12.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a8134a30 con 0x7fe1a80a5d50 2024-08-19T12:49:12.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:12.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1af8a1640 10 monclient(hunting): authenticate will time out at 1346.943573s 2024-08-19T12:49:12.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1ad63d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe1a806e7b0 0x7fe1a81ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59344/0 (socket says 172.21.15.26:59344) 2024-08-19T12:49:12.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.856+0000 7fe1ad63d640 1 -- 172.21.15.26:0/62597625 learned_addr learned my addr 172.21.15.26:0/62597625 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:12.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe1ade3e640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.26:6792/0 conn(0x7fe1a80a5d50 legacy=0x7fe1a81d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe19dffb640 10 client.?.objecter ms_handle_connect 0x7fe1a80a5d50 2024-08-19T12:49:12.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe19dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe19dffb640 10 client.?.objecter ms_handle_connect 0x7fe1a806e7b0 2024-08-19T12:49:12.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe19dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe1ad63d640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.26:6790/0 conn(0x7fe1a806e7b0 legacy=0x7fe1a81ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe1ad63d640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.26:6790/0 conn(0x7fe1a806e7b0 legacy=0x7fe1a81ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T12:49:12.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe1ad63d640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.26:6790/0 conn(0x7fe1a806e7b0 legacy=0x7fe1a81ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe1ad63d640 1 --1- 172.21.15.26:0/62597625 >> v1:172.21.15.26:6790/0 conn(0x7fe1a806e7b0 0x7fe1a81ce4b0 :-1 s=OPENED pgs=3474 cs=1 l=1).handle_message read tag failed 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe1ad63d640 1 --1- 172.21.15.26:0/62597625 >> v1:172.21.15.26:6790/0 conn(0x7fe1a806e7b0 0x7fe1a81ce4b0 :-1 s=OPENED pgs=3474 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe19dffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6790/0 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe1ace3c640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.104:6791/0 conn(0x7fe1a80a5920 legacy=0x7fe1a81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe19dffb640 10 client.?.objecter ms_handle_connect 0x7fe1a80a5920 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe19dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.858+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3301890721 0 0) 0x7fe1a8134a30 con 0x7fe1a80a5d50 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient(hunting): my global_id is 164382 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe18c004eb0 con 0x7fe1a80a5d50 2024-08-19T12:49:12.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 763783185 0 0) 0x7fe18c004eb0 con 0x7fe1a80a5d50 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1a8134a30 con 0x7fe1a80a5d50 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a0002030 con 0x7fe1a80a5d50 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient(hunting): dump: 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.873 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1537784389 0 0) 0x7fe1a80a1990 con 0x7fe1a80a5920 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient(hunting): my global_id is 26807 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe18c004eb0 con 0x7fe1a80a5920 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1971354930 0 0) 0x7fe1a8134a30 con 0x7fe1a80a5d50 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.104:6791/0 conn(0x7fe1a80a5920 legacy=0x7fe1a81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.26:6790/0 conn(0x7fe1a806e7b0 legacy=0x7fe1a81ce4b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 monclient: found mon.d 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a80a0eb0 con 0x7fe1a80a5d50 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient: _finish_auth 0 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:12.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe1af8a1640 5 monclient: authenticate success, global_id 164382 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe1af8a1640 10 monclient: _renew_subs 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe1af8a1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe1af8a1640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe1a80a28f0 con 0x7fe1a80a5d50 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1a0003d50 con 0x7fe1a80a5d50 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a0004280 con 0x7fe1a80a5d50 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19dffb640 10 monclient: dump: 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.876 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.877 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.877 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.877 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.877 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.877 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.877 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.877 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.877 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.877 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.877 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.878 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe19f7fe640 4 set_mon_vals no callback set 2024-08-19T12:49:12.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe1af8a1640 10 client.164382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:12.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe1af8a1640 10 monclient: _renew_subs 2024-08-19T12:49:12.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe1af8a1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:12.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe1af8a1640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1a80a07f0 con 0x7fe1a80a5d50 2024-08-19T12:49:12.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.859+0000 7fe1af8a1640 1 librados: init done 2024-08-19T12:49:12.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.860+0000 7fe1af8a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:12.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.860+0000 7fe1af8a1640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1a80a4eb0 con 0x7fe1a802e340 2024-08-19T12:49:12.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.860+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe1a0004420 con 0x7fe1a80a5d50 2024-08-19T12:49:12.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.860+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(406..406 src has 311..406) ==== 5413+0+0 (unknown 2461991952 0 0) 0x7fe1a004e350 con 0x7fe1a80a5d50 2024-08-19T12:49:12.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.860+0000 7fe19dffb640 10 client.164382.objecter ms_dispatch 0x7fe1a8000b90 osd_map(406..406 src has 311..406) 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.860+0000 7fe19dffb640 3 client.164382.objecter handle_osd_map got epochs [406,406] > 0 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.860+0000 7fe19dffb640 3 client.164382.objecter handle_osd_map decoding full epoch 406 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.860+0000 7fe19dffb640 20 client.164382.objecter dump_active .. 0 homeless 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.861+0000 7fe1ace3c640 10 monclient: get_auth_request con 0x7fe18c040f90 auth_method 0 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe1ad63d640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.104:6792/0 conn(0x7fe1a802e340 legacy=0x7fe1a8063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 10 client.164382.objecter ms_handle_connect 0x7fe1a802e340 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 10 client.164382.objecter resend_mon_ops 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4124773892 0 0) 0x7fe1a80a4eb0 con 0x7fe1a802e340 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 10 monclient(hunting): my global_id is 135673 2024-08-19T12:49:12.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1a8134a30 con 0x7fe1a802e340 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 10 client.164382.objecter ms_handle_connect 0x7fe18c040f90 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1130169515 0 0) 0x7fe1a8134a30 con 0x7fe1a802e340 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1a80a4eb0 con 0x7fe1a802e340 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a4001d20 con 0x7fe1a802e340 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 10 monclient: dump: 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.880 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1599548391 0 0) 0x7fe1a80a4eb0 con 0x7fe1a802e340 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.863+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe1a802e720 con 0x7fe1a802e340 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.864+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe1a4004a00 con 0x7fe1a802e340 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.864+0000 7fe19dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:12.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.864+0000 7fe19dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:12.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.864+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.104:6792/0 conn(0x7fe1a802e340 legacy=0x7fe1a8063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.864+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.104:6792/0 conn(0x7fe1a802e340 legacy=0x7fe1a8063a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.870+0000 7fe1af8a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:12.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.870+0000 7fe1af8a1640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1a8125a20 con 0x7fe1a81cebc0 2024-08-19T12:49:12.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.872+0000 7fe1ad63d640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.104:6792/0 conn(0x7fe1a81cebc0 legacy=0x7fe1a8125310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:12.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.872+0000 7fe19dffb640 10 client.164382.objecter ms_handle_connect 0x7fe1a81cebc0 2024-08-19T12:49:12.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.872+0000 7fe19dffb640 10 client.164382.objecter resend_mon_ops 2024-08-19T12:49:12.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3378830280 0 0) 0x7fe1a8125a20 con 0x7fe1a81cebc0 2024-08-19T12:49:12.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:12.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 10 monclient(hunting): my global_id is 135682 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1a80a4eb0 con 0x7fe1a81cebc0 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3576626390 0 0) 0x7fe1a80a4eb0 con 0x7fe1a81cebc0 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1a8125a20 con 0x7fe1a81cebc0 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a4005240 con 0x7fe1a81cebc0 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 10 monclient: dump: 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:12.884 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2541127648 0 0) 0x7fe1a8125a20 con 0x7fe1a81cebc0 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe1a80a3040 con 0x7fe1a81cebc0 2024-08-19T12:49:12.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3361043512) 0x7fe1a40086f0 con 0x7fe1a81cebc0 2024-08-19T12:49:12.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:12.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:12.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.104:6792/0 conn(0x7fe1a81cebc0 legacy=0x7fe1a8125310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.873+0000 7fe19dffb640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.104:6792/0 conn(0x7fe1a81cebc0 legacy=0x7fe1a8125310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:12.886 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:12.886 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:12.886 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:12.886 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:12.886 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:12.886 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 523231, 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:49:12.887 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.888 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.889 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:12.890 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:12.891 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:12.892 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:12.892 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:12.892 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.892 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:12.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:12.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.892 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:12.892 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:12.892 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:12.894 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:12.896 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:12.896 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.896 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.896 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.897 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:12.897 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:12.897 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:12.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:12.899 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.900 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.903 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:12.904 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.905 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:12.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:12.907 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:12.907 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:12.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 10 librados: watch_flush enter 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 10 librados: watch_flush exit 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 20 client.164382.objecter shutdown clearing up homeless session... 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 10 client.164382.objecter successfully canceled tick 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.104:6800/899887506 conn(0x7fe18c040f90 legacy=0x7fe18c043450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 10 monclient: shutdown 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 1 -- 172.21.15.26:0/62597625 >> v1:172.21.15.26:6792/0 conn(0x7fe1a80a5d50 legacy=0x7fe1a81d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1ade3e640 1 -- 172.21.15.26:0/62597625 reap_dead start 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1ade3e640 1 -- 172.21.15.26:0/62597625 reap_dead start 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 1 -- 172.21.15.26:0/62597625 shutdown_connections 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 1 -- 172.21.15.26:0/62597625 >> 172.21.15.26:0/62597625 conn(0x7fe1a80a8740 msgr2=0x7fe1a806dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 1 -- 172.21.15.26:0/62597625 shutdown_connections 2024-08-19T12:49:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 1 -- 172.21.15.26:0/62597625 wait complete. 2024-08-19T12:49:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:12.874+0000 7fe1af8a1640 1 librados: shutdown 2024-08-19T12:49:12.910 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:12.910 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T12:49:12.910 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T12:49:13.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.008+0000 7f4826731640 10 monclient: get_monmap_and_config 2024-08-19T12:49:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.008+0000 7f4826731640 10 monclient: build_initial_monmap 2024-08-19T12:49:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.008+0000 7f4826731640 10 monclient: monmap: 2024-08-19T12:49:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:13.009697+0000 2024-08-19T12:49:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:13.009697+0000 2024-08-19T12:49:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:13.010 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:13.011 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:13.011 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:13.011 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:13.011 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:13.011 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:13.011 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.009+0000 7f4826731640 1 Processor -- start 2024-08-19T12:49:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.009+0000 7f4826731640 1 -- start start 2024-08-19T12:49:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.009+0000 7f4826731640 10 monclient: init 2024-08-19T12:49:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f4826731640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f4826731640 10 monclient: _add_conns ranks=[6,1,4,3,5,7,0,2,8] 2024-08-19T12:49:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f4826731640 10 monclient(hunting): picked mon.noname-d con 0x7f48200a2a60 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:13.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f4826731640 10 monclient(hunting): picked mon.noname-f con 0x7f48200a2e40 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f4826731640 10 monclient(hunting): picked mon.noname-c con 0x7f482013f590 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f4826731640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48200a4600 con 0x7f48200a2e40 2024-08-19T12:49:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f4826731640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4820074090 con 0x7f482013f590 2024-08-19T12:49:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f4826731640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4820074370 con 0x7f48200a2a60 2024-08-19T12:49:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f4826731640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f4826731640 10 monclient(hunting): authenticate will time out at 1347.097182s 2024-08-19T12:49:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f4824cce640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f482013f590 0x7f4820141980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57484/0 (socket says 172.21.15.26:57484) 2024-08-19T12:49:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f481ffff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f48200a2a60 0x7f48200a3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:50962/0 (socket says 172.21.15.26:50962) 2024-08-19T12:49:13.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.010+0000 7f4824cce640 1 -- 172.21.15.26:0/3686743177 learned_addr learned my addr 172.21.15.26:0/3686743177 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:13.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481ffff640 1 -- 172.21.15.26:0/3686743177 >> v1:172.21.15.26:6792/0 conn(0x7f48200a2a60 legacy=0x7f48200a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:13.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f4824cce640 1 -- 172.21.15.26:0/3686743177 >> v1:172.21.15.26:6791/0 conn(0x7f482013f590 legacy=0x7f4820141980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:13.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481f7fe640 1 -- 172.21.15.26:0/3686743177 >> v1:172.21.15.104:6789/0 conn(0x7f48200a2e40 legacy=0x7f48200a3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:13.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 1 -- 172.21.15.26:0/3686743177 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1255443515 0 0) 0x7f4820074370 con 0x7f48200a2a60 2024-08-19T12:49:13.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:13.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:13.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:13.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient(hunting): my global_id is 164391 2024-08-19T12:49:13.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 1 -- 172.21.15.26:0/3686743177 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4808003520 con 0x7f48200a2a60 2024-08-19T12:49:13.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 1 -- 172.21.15.26:0/3686743177 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3041259013 0 0) 0x7f4820074090 con 0x7f482013f590 2024-08-19T12:49:13.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:13.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:13.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:13.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient(hunting): my global_id is 27058 2024-08-19T12:49:13.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 1 -- 172.21.15.26:0/3686743177 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4820074370 con 0x7f482013f590 2024-08-19T12:49:13.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 1 -- 172.21.15.26:0/3686743177 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2364567251 0 0) 0x7f4808003520 con 0x7f48200a2a60 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 1 -- 172.21.15.26:0/3686743177 >> v1:172.21.15.26:6791/0 conn(0x7f482013f590 legacy=0x7f4820141980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 1 -- 172.21.15.26:0/3686743177 >> v1:172.21.15.104:6789/0 conn(0x7f48200a2e40 legacy=0x7f48200a3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient: _finish_hunting 0 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 1 monclient: found mon.noname-d 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 1 -- 172.21.15.26:0/3686743177 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48200a5980 con 0x7f48200a2a60 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient: _finish_auth 0 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient: _check_auth_tickets 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 1 -- 172.21.15.26:0/3686743177 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4810001ae0 con 0x7f48200a2a60 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f4826731640 5 monclient: authenticate success, global_id 164391 2024-08-19T12:49:13.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f4826731640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f481effd640 10 monclient: dump: 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:13.020 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.012+0000 7f4826731640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f481effd640 1 -- 172.21.15.26:0/3686743177 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48100047e0 con 0x7f48200a2a60 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f481effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f481effd640 1 -- 172.21.15.26:0/3686743177 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4810004d50 con 0x7f48200a2a60 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f481effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f481effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f481effd640 10 monclient: dump: 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:13.022 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:13.023 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f4826731640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f4826731640 4 set_mon_vals no callback set 2024-08-19T12:49:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f4826731640 10 monclient: shutdown 2024-08-19T12:49:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f4826731640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f4826731640 1 -- 172.21.15.26:0/3686743177 >> v1:172.21.15.26:6792/0 conn(0x7f48200a2a60 legacy=0x7f48200a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f4826731640 1 -- 172.21.15.26:0/3686743177 shutdown_connections 2024-08-19T12:49:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f4826731640 1 -- 172.21.15.26:0/3686743177 >> 172.21.15.26:0/3686743177 conn(0x7f48200a4b60 msgr2=0x7f48200a4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f4826731640 1 -- 172.21.15.26:0/3686743177 shutdown_connections 2024-08-19T12:49:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.013+0000 7f4826731640 1 -- 172.21.15.26:0/3686743177 wait complete. 2024-08-19T12:49:13.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.014+0000 7f4826731640 10 monclient: build_initial_monmap 2024-08-19T12:49:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.014+0000 7f4826731640 10 monclient: monmap: 2024-08-19T12:49:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:13.024 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.014+0000 7f4826731640 1 librados: starting msgr at 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.014+0000 7f4826731640 1 librados: starting objecter 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.014+0000 7f4826731640 1 Processor -- start 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.014+0000 7f4826731640 1 -- start start 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.014+0000 7f4826731640 1 librados: setting wanted keys 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.014+0000 7f4826731640 1 librados: calling monclient init 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.014+0000 7f4826731640 10 monclient: init 2024-08-19T12:49:13.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.015+0000 7f4826731640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.015+0000 7f4826731640 10 monclient: _add_conns ranks=[7,1,6,4,5,0,8,3,2] 2024-08-19T12:49:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.015+0000 7f4826731640 10 monclient(hunting): picked mon.noname-i con 0x7f48200a2a60 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.015+0000 7f4826731640 10 monclient(hunting): picked mon.noname-f con 0x7f48200a2e40 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.015+0000 7f4826731640 10 monclient(hunting): picked mon.noname-d con 0x7f482013f590 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.015+0000 7f4826731640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4820142dd0 con 0x7f48200a2e40 2024-08-19T12:49:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.015+0000 7f4826731640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4820139460 con 0x7f482013f590 2024-08-19T12:49:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.015+0000 7f4826731640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4820138790 con 0x7f48200a2a60 2024-08-19T12:49:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.015+0000 7f4826731640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.015+0000 7f4826731640 10 monclient(hunting): authenticate will time out at 1347.102169s 2024-08-19T12:49:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.015+0000 7f4824cce640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f482013f590 0x7f48201d6370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:50972/0 (socket says 172.21.15.26:50972) 2024-08-19T12:49:13.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.015+0000 7f4824cce640 1 -- 172.21.15.26:0/2569069297 learned_addr learned my addr 172.21.15.26:0/2569069297 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f4824cce640 1 -- 172.21.15.26:0/2569069297 >> v1:172.21.15.26:6792/0 conn(0x7f482013f590 legacy=0x7f48201d6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 client.?.objecter ms_handle_connect 0x7f482013f590 2024-08-19T12:49:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 client.?.objecter ms_handle_connect 0x7f48200a2e40 2024-08-19T12:49:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481f7fe640 1 -- 172.21.15.26:0/2569069297 >> v1:172.21.15.104:6789/0 conn(0x7f48200a2e40 legacy=0x7f4820138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 client.?.objecter ms_handle_connect 0x7f48200a2a60 2024-08-19T12:49:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481ffff640 1 -- 172.21.15.26:0/2569069297 >> v1:172.21.15.104:6792/0 conn(0x7f48200a2a60 legacy=0x7f4820137970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866755318 0 0) 0x7f4820139460 con 0x7f482013f590 2024-08-19T12:49:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:13.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 monclient(hunting): my global_id is 164400 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f480c004eb0 con 0x7f482013f590 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3257422116 0 0) 0x7f4820142dd0 con 0x7f48200a2e40 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 monclient(hunting): my global_id is 27595 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4820139460 con 0x7f48200a2e40 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3354147194 0 0) 0x7f4820138790 con 0x7f48200a2a60 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:13.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 10 monclient(hunting): my global_id is 135691 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.017+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4820142dd0 con 0x7f48200a2a60 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2066485290 0 0) 0x7f480c004eb0 con 0x7f482013f590 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4820138790 con 0x7f482013f590 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4818004b60 con 0x7f482013f590 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient(hunting): dump: 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3243927683 0 0) 0x7f4820139460 con 0x7f48200a2e40 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f480c004eb0 con 0x7f48200a2e40 2024-08-19T12:49:13.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4814003610 con 0x7f48200a2e40 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient(hunting): dump: 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:13.035 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 578788820 0 0) 0x7f4820142dd0 con 0x7f48200a2a60 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4820139460 con 0x7f48200a2a60 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4810004cd0 con 0x7f48200a2a60 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient(hunting): dump: 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:13.036 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:13.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 880285956 0 0) 0x7f4820138790 con 0x7f482013f590 2024-08-19T12:49:13.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 >> v1:172.21.15.104:6792/0 conn(0x7f48200a2a60 legacy=0x7f4820137970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:13.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 >> v1:172.21.15.104:6789/0 conn(0x7f48200a2e40 legacy=0x7f4820138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:13.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient: _finish_hunting 0 2024-08-19T12:49:13.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 monclient: found mon.d 2024-08-19T12:49:13.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:13.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:13.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482013bc00 con 0x7f482013f590 2024-08-19T12:49:13.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient: _finish_auth 0 2024-08-19T12:49:13.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient: _check_auth_tickets 2024-08-19T12:49:13.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:13.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f4826731640 5 monclient: authenticate success, global_id 164400 2024-08-19T12:49:13.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f4826731640 10 monclient: _renew_subs 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f4826731640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f4826731640 1 -- 172.21.15.26:0/2569069297 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f482013a760 con 0x7f482013f590 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f4826731640 10 client.164400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f4826731640 10 monclient: _renew_subs 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f4826731640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f4826731640 1 -- 172.21.15.26:0/2569069297 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f4820139d40 con 0x7f482013f590 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f4826731640 1 librados: init done 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4818004fb0 con 0x7f482013f590 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f48180054a0 con 0x7f482013f590 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481e7fc640 4 set_mon_vals no callback set 2024-08-19T12:49:13.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.018+0000 7f481cff9640 10 monclient: dump: 2024-08-19T12:49:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:13.040 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.019+0000 7f4826731640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.019+0000 7f4826731640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.019+0000 7f4826731640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.019+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f4818011900 con 0x7f482013f590 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.019+0000 7f4826731640 1 -- 172.21.15.26:0/2569069297 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4800005080 con 0x7f482013f590 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.019+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(406..406 src has 311..406) ==== 5413+0+0 (unknown 2461991952 0 0) 0x7f481804e4b0 con 0x7f482013f590 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.019+0000 7f481cff9640 10 client.164400.objecter ms_dispatch 0x7f4820000b90 osd_map(406..406 src has 311..406) 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.019+0000 7f481cff9640 3 client.164400.objecter handle_osd_map got epochs [406,406] > 0 2024-08-19T12:49:13.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.019+0000 7f481cff9640 3 client.164400.objecter handle_osd_map decoding full epoch 406 2024-08-19T12:49:13.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.019+0000 7f481cff9640 20 client.164400.objecter dump_active .. 0 homeless 2024-08-19T12:49:13.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.019+0000 7f481ffff640 10 monclient: get_auth_request con 0x7f480c041920 auth_method 0 2024-08-19T12:49:13.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.022+0000 7f481cff9640 10 client.164400.objecter ms_handle_connect 0x7f480c041920 2024-08-19T12:49:13.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.024+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f4818017870 con 0x7f482013f590 2024-08-19T12:49:13.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.024+0000 7f481cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:13.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.024+0000 7f481cff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:13.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.288+0000 7f4826731640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T12:49:13.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.288+0000 7f4826731640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T12:49:13.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.288+0000 7f4826731640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:13.289 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.288+0000 7f4826731640 1 -- 172.21.15.26:0/2569069297 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f4800005c40 con 0x7f482013f590 2024-08-19T12:49:13.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.290+0000 7f481cff9640 1 -- 172.21.15.26:0/2569069297 <== mon.6 v1:172.21.15.26:6792/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f4818017170 con 0x7f482013f590 2024-08-19T12:49:13.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.290+0000 7f481cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T12:49:13.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.290+0000 7f481cff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:13.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 10 librados: watch_flush enter 2024-08-19T12:49:13.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 10 librados: watch_flush exit 2024-08-19T12:49:13.293 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 20 client.164400.objecter shutdown clearing up homeless session... 2024-08-19T12:49:13.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 10 client.164400.objecter successfully canceled tick 2024-08-19T12:49:13.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 1 -- 172.21.15.26:0/2569069297 >> v1:172.21.15.104:6800/899887506 conn(0x7f480c041920 legacy=0x7f480c043de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:13.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 10 monclient: shutdown 2024-08-19T12:49:13.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:13.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 1 -- 172.21.15.26:0/2569069297 >> v1:172.21.15.26:6792/0 conn(0x7f482013f590 legacy=0x7f48201d6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:13.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 1 -- 172.21.15.26:0/2569069297 shutdown_connections 2024-08-19T12:49:13.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 1 -- 172.21.15.26:0/2569069297 >> 172.21.15.26:0/2569069297 conn(0x7f48200a4b60 msgr2=0x7f4820136320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:13.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 1 -- 172.21.15.26:0/2569069297 shutdown_connections 2024-08-19T12:49:13.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 1 -- 172.21.15.26:0/2569069297 wait complete. 2024-08-19T12:49:13.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:13.292+0000 7f4826731640 1 librados: shutdown 2024-08-19T12:49:13.304 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T12:49:13.726 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.726 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=13,logm=50} crc {auth=1991965717,config=1097430493,health=1449040904,kv=974608666,logm=3306117699}) 2024-08-19T12:49:13.726 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=12,kv=12,logm=52} crc {auth=1952805515,config=1097430493,health=3397957174,kv=1466468801,logm=1400280209}) 2024-08-19T12:49:13.726 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.726 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=13,logm=50} crc {auth=1991965717,config=1097430493,health=1449040904,kv=974608666,logm=3306117699}) 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=3364029597,config=1097430493,health=1145329893,kv=1344821656,logm=239964398}) 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=13,logm=50} crc {auth=1991965717,config=1097430493,health=1449040904,kv=974608666,logm=3306117699}) 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=25,config=2,health=9,kv=11,logm=53} crc {auth=224120053,config=4178277379,health=1886278603,kv=568943783,logm=1112638244}) 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=13,logm=50} crc {auth=1991965717,config=1097430493,health=1449040904,kv=974608666,logm=3306117699}) 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=20,config=2,health=10,kv=10,logm=58} crc {auth=1283210020,config=1097430493,health=1574669361,kv=2443582623,logm=3837680839}) 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=13,logm=50} crc {auth=1991965717,config=1097430493,health=1449040904,kv=974608666,logm=3306117699}) 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=25,config=2,health=12,kv=13,logm=48} crc {auth=224120053,config=1097430493,health=3397957174,kv=1344821656,logm=3095067589}) 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=13,logm=50} crc {auth=1991965717,config=1097430493,health=1449040904,kv=974608666,logm=3306117699}) 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=23,config=2,health=12,kv=13,logm=50} crc {auth=1663458551,config=1097430493,health=399318158,kv=1344821656,logm=1733175229}) 2024-08-19T12:49:13.727 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.728 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=13,logm=50} crc {auth=1991965717,config=1097430493,health=1449040904,kv=974608666,logm=3306117699}) 2024-08-19T12:49:13.728 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=25,config=1,health=12,kv=13,logm=49} crc {auth=224120053,config=1228700967,health=3397957174,kv=1344821656,logm=304702966}) 2024-08-19T12:49:13.728 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.728 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=13,logm=50} crc {auth=1991965717,config=1097430493,health=1449040904,kv=974608666,logm=3306117699}) 2024-08-19T12:49:13.728 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.724+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=224120053,config=1097430493,health=3397957174,kv=956038015,logm=3582114416}) 2024-08-19T12:49:13.734 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.734 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=848341110}) 2024-08-19T12:49:13.734 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2385760064}) 2024-08-19T12:49:13.734 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.734 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=848341110}) 2024-08-19T12:49:13.734 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=58787330}) 2024-08-19T12:49:13.734 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.734 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=848341110}) 2024-08-19T12:49:13.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2858962192}) 2024-08-19T12:49:13.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=848341110}) 2024-08-19T12:49:13.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2682082165}) 2024-08-19T12:49:13.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=848341110}) 2024-08-19T12:49:13.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1668795446}) 2024-08-19T12:49:13.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=848341110}) 2024-08-19T12:49:13.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=512348009}) 2024-08-19T12:49:13.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=848341110}) 2024-08-19T12:49:13.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1900242969}) 2024-08-19T12:49:13.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=848341110}) 2024-08-19T12:49:13.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.726+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3582685410}) 2024-08-19T12:49:13.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3390857142}) 2024-08-19T12:49:13.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1302087101}) 2024-08-19T12:49:13.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3390857142}) 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1674975870}) 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3390857142}) 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2398629708}) 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3390857142}) 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2524983204}) 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3390857142}) 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1591922574}) 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3390857142}) 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3101897137}) 2024-08-19T12:49:13.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3390857142}) 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1960555689}) 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3390857142}) 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.728+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=857833215}) 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184051872}) 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2979796042}) 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184051872}) 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1848000460}) 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184051872}) 2024-08-19T12:49:13.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1749988839}) 2024-08-19T12:49:13.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184051872}) 2024-08-19T12:49:13.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1762600915}) 2024-08-19T12:49:13.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184051872}) 2024-08-19T12:49:13.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3727690031}) 2024-08-19T12:49:13.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184051872}) 2024-08-19T12:49:13.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=443428257}) 2024-08-19T12:49:13.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184051872}) 2024-08-19T12:49:13.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=498326796}) 2024-08-19T12:49:13.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184051872}) 2024-08-19T12:49:13.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.730+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1178919855}) 2024-08-19T12:49:13.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543704775}) 2024-08-19T12:49:13.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603049699}) 2024-08-19T12:49:13.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543704775}) 2024-08-19T12:49:13.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4238416640}) 2024-08-19T12:49:13.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543704775}) 2024-08-19T12:49:13.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3846784523}) 2024-08-19T12:49:13.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543704775}) 2024-08-19T12:49:13.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2333141273}) 2024-08-19T12:49:13.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543704775}) 2024-08-19T12:49:13.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=219932444}) 2024-08-19T12:49:13.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543704775}) 2024-08-19T12:49:13.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=726202563}) 2024-08-19T12:49:13.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543704775}) 2024-08-19T12:49:13.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1826350105}) 2024-08-19T12:49:13.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543704775}) 2024-08-19T12:49:13.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.732+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1766483064}) 2024-08-19T12:49:13.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298334327}) 2024-08-19T12:49:13.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1367147718}) 2024-08-19T12:49:13.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298334327}) 2024-08-19T12:49:13.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480585496}) 2024-08-19T12:49:13.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298334327}) 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4279911202}) 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298334327}) 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1025405981}) 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298334327}) 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4289371190}) 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298334327}) 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3588206524}) 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298334327}) 2024-08-19T12:49:13.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2276505871}) 2024-08-19T12:49:13.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298334327}) 2024-08-19T12:49:13.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.741+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1721036981}) 2024-08-19T12:49:13.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398009381}) 2024-08-19T12:49:13.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2005819091}) 2024-08-19T12:49:13.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398009381}) 2024-08-19T12:49:13.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752556065}) 2024-08-19T12:49:13.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398009381}) 2024-08-19T12:49:13.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=542380195}) 2024-08-19T12:49:13.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398009381}) 2024-08-19T12:49:13.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3684368167}) 2024-08-19T12:49:13.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398009381}) 2024-08-19T12:49:13.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1940512324}) 2024-08-19T12:49:13.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398009381}) 2024-08-19T12:49:13.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=495228967}) 2024-08-19T12:49:13.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398009381}) 2024-08-19T12:49:13.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2282631015}) 2024-08-19T12:49:13.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398009381}) 2024-08-19T12:49:13.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.743+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=626519035}) 2024-08-19T12:49:13.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396963710}) 2024-08-19T12:49:13.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1708827500}) 2024-08-19T12:49:13.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396963710}) 2024-08-19T12:49:13.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2652480402}) 2024-08-19T12:49:13.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396963710}) 2024-08-19T12:49:13.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=96477005}) 2024-08-19T12:49:13.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396963710}) 2024-08-19T12:49:13.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1426459719}) 2024-08-19T12:49:13.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396963710}) 2024-08-19T12:49:13.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1560249531}) 2024-08-19T12:49:13.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396963710}) 2024-08-19T12:49:13.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=801169899}) 2024-08-19T12:49:13.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396963710}) 2024-08-19T12:49:13.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2233541339}) 2024-08-19T12:49:13.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396963710}) 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.744+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1581227054}) 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.745+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.745+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788021001}) 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.745+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3215368051}) 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.745+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.745+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788021001}) 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.745+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3464370316}) 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.745+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788021001}) 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3912451480}) 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788021001}) 2024-08-19T12:49:13.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=714163565}) 2024-08-19T12:49:13.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788021001}) 2024-08-19T12:49:13.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3631885467}) 2024-08-19T12:49:13.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788021001}) 2024-08-19T12:49:13.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=308188339}) 2024-08-19T12:49:13.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788021001}) 2024-08-19T12:49:13.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3203047808}) 2024-08-19T12:49:13.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788021001}) 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.746+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1364118368}) 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007910804}) 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3328261789}) 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007910804}) 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3786835427}) 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007910804}) 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4084404901}) 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.761 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007910804}) 2024-08-19T12:49:13.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1115464076}) 2024-08-19T12:49:13.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007910804}) 2024-08-19T12:49:13.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=986997050}) 2024-08-19T12:49:13.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007910804}) 2024-08-19T12:49:13.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1760525099}) 2024-08-19T12:49:13.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007910804}) 2024-08-19T12:49:13.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2567627582}) 2024-08-19T12:49:13.762 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007910804}) 2024-08-19T12:49:13.763 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.747+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=118577303}) 2024-08-19T12:49:13.772 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.756+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.772 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.756+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892487724}) 2024-08-19T12:49:13.772 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.756+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4232731049}) 2024-08-19T12:49:13.773 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.773 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892487724}) 2024-08-19T12:49:13.773 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1522209981}) 2024-08-19T12:49:13.773 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.773 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892487724}) 2024-08-19T12:49:13.773 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1879815124}) 2024-08-19T12:49:13.773 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.773 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892487724}) 2024-08-19T12:49:13.773 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1079100598}) 2024-08-19T12:49:13.773 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.773 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892487724}) 2024-08-19T12:49:13.773 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=181201118}) 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892487724}) 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=397647179}) 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892487724}) 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4162761819}) 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892487724}) 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.757+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1790601988}) 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896907557}) 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750227467}) 2024-08-19T12:49:13.774 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896907557}) 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3613511973}) 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896907557}) 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=13713229}) 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896907557}) 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2028518227}) 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896907557}) 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3807131905}) 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.775 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896907557}) 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=371055228}) 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896907557}) 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1120915050}) 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896907557}) 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.758+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3018266679}) 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864522930}) 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3232493322}) 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864522930}) 2024-08-19T12:49:13.776 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4264794951}) 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864522930}) 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1173291428}) 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864522930}) 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3994404916}) 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864522930}) 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2871326347}) 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864522930}) 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1842434249}) 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.759+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.777 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864522930}) 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3251077996}) 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864522930}) 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.760+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2846879568}) 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471376156}) 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1897944866}) 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471376156}) 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2519701838}) 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471376156}) 2024-08-19T12:49:13.778 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=284251109}) 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471376156}) 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1562531460}) 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471376156}) 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1561728703}) 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471376156}) 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1575607961}) 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471376156}) 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2794909265}) 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.779 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471376156}) 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.761+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1968672892}) 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476287282}) 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316807986}) 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476287282}) 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3294611088}) 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476287282}) 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3014266677}) 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476287282}) 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1949834946}) 2024-08-19T12:49:13.780 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.781 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476287282}) 2024-08-19T12:49:13.781 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1716060806}) 2024-08-19T12:49:13.781 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.781 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476287282}) 2024-08-19T12:49:13.781 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3744086266}) 2024-08-19T12:49:13.781 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.781 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476287282}) 2024-08-19T12:49:13.781 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1401336410}) 2024-08-19T12:49:13.781 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.781 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476287282}) 2024-08-19T12:49:13.781 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.762+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=591854155}) 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988199826}) 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4263412877}) 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988199826}) 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1757309222}) 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988199826}) 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3286496592}) 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988199826}) 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4149087738}) 2024-08-19T12:49:13.789 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988199826}) 2024-08-19T12:49:13.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3364382499}) 2024-08-19T12:49:13.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988199826}) 2024-08-19T12:49:13.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1535786382}) 2024-08-19T12:49:13.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988199826}) 2024-08-19T12:49:13.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.772+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2044888691}) 2024-08-19T12:49:13.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988199826}) 2024-08-19T12:49:13.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.773+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3025536166}) 2024-08-19T12:49:13.790 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046167505}) 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2450444799}) 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046167505}) 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1741489100}) 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046167505}) 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3648462240}) 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046167505}) 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3337414904}) 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046167505}) 2024-08-19T12:49:13.791 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1513697795}) 2024-08-19T12:49:13.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046167505}) 2024-08-19T12:49:13.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=513641455}) 2024-08-19T12:49:13.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046167505}) 2024-08-19T12:49:13.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1552568518}) 2024-08-19T12:49:13.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046167505}) 2024-08-19T12:49:13.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.774+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=90785786}) 2024-08-19T12:49:13.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350283314}) 2024-08-19T12:49:13.792 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3566865397}) 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350283314}) 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1003388480}) 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350283314}) 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2755654907}) 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350283314}) 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1870643125}) 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350283314}) 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2501535261}) 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.775+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.793 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350283314}) 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1776129598}) 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350283314}) 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3578869152}) 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350283314}) 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.776+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3854754763}) 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765032802}) 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3370460187}) 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765032802}) 2024-08-19T12:49:13.794 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=587825638}) 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765032802}) 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2246790902}) 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765032802}) 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3686071708}) 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765032802}) 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2310085918}) 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765032802}) 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1191973979}) 2024-08-19T12:49:13.795 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765032802}) 2024-08-19T12:49:13.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2675365931}) 2024-08-19T12:49:13.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765032802}) 2024-08-19T12:49:13.796 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.777+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3272705314}) 2024-08-19T12:49:13.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.982 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946322847}) 2024-08-19T12:49:13.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2078160849}) 2024-08-19T12:49:13.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946322847}) 2024-08-19T12:49:13.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2477501067}) 2024-08-19T12:49:13.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946322847}) 2024-08-19T12:49:13.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4172951323}) 2024-08-19T12:49:13.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946322847}) 2024-08-19T12:49:13.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3011596472}) 2024-08-19T12:49:13.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.983 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946322847}) 2024-08-19T12:49:13.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1362623278}) 2024-08-19T12:49:13.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946322847}) 2024-08-19T12:49:13.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2158420585}) 2024-08-19T12:49:13.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946322847}) 2024-08-19T12:49:13.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=632150795}) 2024-08-19T12:49:13.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:13.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946322847}) 2024-08-19T12:49:13.984 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:13.980+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1712213423}) 2024-08-19T12:49:14.221 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.219+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.219+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322702498}) 2024-08-19T12:49:14.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.219+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1272219187}) 2024-08-19T12:49:14.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.219+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.219+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322702498}) 2024-08-19T12:49:14.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.219+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1286360539}) 2024-08-19T12:49:14.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.219+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.219+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322702498}) 2024-08-19T12:49:14.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.219+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1338066175}) 2024-08-19T12:49:14.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.219+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.219+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322702498}) 2024-08-19T12:49:14.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.219+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=735892368}) 2024-08-19T12:49:14.222 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.220+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.220+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322702498}) 2024-08-19T12:49:14.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.220+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1483370376}) 2024-08-19T12:49:14.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.220+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.220+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322702498}) 2024-08-19T12:49:14.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.220+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2988659092}) 2024-08-19T12:49:14.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.220+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.220+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322702498}) 2024-08-19T12:49:14.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.220+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3161709546}) 2024-08-19T12:49:14.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.220+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.220+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322702498}) 2024-08-19T12:49:14.223 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.220+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2951024262}) 2024-08-19T12:49:14.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.225 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465262818}) 2024-08-19T12:49:14.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3990519069}) 2024-08-19T12:49:14.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465262818}) 2024-08-19T12:49:14.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316962382}) 2024-08-19T12:49:14.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465262818}) 2024-08-19T12:49:14.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2979088699}) 2024-08-19T12:49:14.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465262818}) 2024-08-19T12:49:14.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3194547131}) 2024-08-19T12:49:14.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.226 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465262818}) 2024-08-19T12:49:14.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=791371880}) 2024-08-19T12:49:14.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465262818}) 2024-08-19T12:49:14.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=574212735}) 2024-08-19T12:49:14.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465262818}) 2024-08-19T12:49:14.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.221+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2345769145}) 2024-08-19T12:49:14.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465262818}) 2024-08-19T12:49:14.227 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.222+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3563894590}) 2024-08-19T12:49:14.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180567005}) 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3671329597}) 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180567005}) 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2302223220}) 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180567005}) 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3585508898}) 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180567005}) 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3036828614}) 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180567005}) 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3620786290}) 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.232+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180567005}) 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2766085221}) 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180567005}) 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2195675352}) 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180567005}) 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.233+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1323195557}) 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367990853}) 2024-08-19T12:49:14.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984325714}) 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367990853}) 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3081246789}) 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367990853}) 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3557238628}) 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367990853}) 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2914840436}) 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367990853}) 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3985071294}) 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367990853}) 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2938643964}) 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367990853}) 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1866990970}) 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367990853}) 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.234+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3529223102}) 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177818122}) 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=993418205}) 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177818122}) 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3900627760}) 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177818122}) 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2474233799}) 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177818122}) 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2665948428}) 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.235+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.236+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177818122}) 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.236+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1469266618}) 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.236+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.236+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177818122}) 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.236+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2423149072}) 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.236+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.236+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177818122}) 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.236+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1342653989}) 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.236+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.236+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177818122}) 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.236+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3734620656}) 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965499960}) 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2687024648}) 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965499960}) 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835991564}) 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965499960}) 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3212839468}) 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965499960}) 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1587303258}) 2024-08-19T12:49:14.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965499960}) 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3693373899}) 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965499960}) 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3150781747}) 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965499960}) 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2241050791}) 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965499960}) 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.237+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4195194494}) 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367249652}) 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3075570948}) 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367249652}) 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3049732470}) 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367249652}) 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1474414184}) 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367249652}) 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1189826630}) 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367249652}) 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2106567866}) 2024-08-19T12:49:14.255 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367249652}) 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3940189359}) 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367249652}) 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1636581686}) 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367249652}) 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.238+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2136209259}) 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2238471326}) 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787190252}) 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.256 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2238471326}) 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3369471156}) 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2238471326}) 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2449311434}) 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2238471326}) 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1597587715}) 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2238471326}) 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3508776630}) 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2238471326}) 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1445949476}) 2024-08-19T12:49:14.257 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2238471326}) 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2493584091}) 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2238471326}) 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.240+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2168743624}) 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696519440}) 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=865101322}) 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696519440}) 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4190060089}) 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696519440}) 2024-08-19T12:49:14.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1514112525}) 2024-08-19T12:49:14.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696519440}) 2024-08-19T12:49:14.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4280474128}) 2024-08-19T12:49:14.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696519440}) 2024-08-19T12:49:14.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3653043756}) 2024-08-19T12:49:14.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696519440}) 2024-08-19T12:49:14.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=485952960}) 2024-08-19T12:49:14.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696519440}) 2024-08-19T12:49:14.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=718013391}) 2024-08-19T12:49:14.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696519440}) 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.241+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3379122653}) 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.244+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.244+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705252715}) 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.244+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2570448269}) 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.244+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.244+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705252715}) 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.244+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752572018}) 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705252715}) 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=595149320}) 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705252715}) 2024-08-19T12:49:14.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1329814147}) 2024-08-19T12:49:14.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705252715}) 2024-08-19T12:49:14.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=648655427}) 2024-08-19T12:49:14.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705252715}) 2024-08-19T12:49:14.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3411363217}) 2024-08-19T12:49:14.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705252715}) 2024-08-19T12:49:14.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1521778009}) 2024-08-19T12:49:14.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705252715}) 2024-08-19T12:49:14.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.245+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=717998607}) 2024-08-19T12:49:14.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799871570}) 2024-08-19T12:49:14.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1077825515}) 2024-08-19T12:49:14.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799871570}) 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2112233222}) 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799871570}) 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986010801}) 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799871570}) 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3251665049}) 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799871570}) 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2872682686}) 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799871570}) 2024-08-19T12:49:14.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2190883207}) 2024-08-19T12:49:14.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799871570}) 2024-08-19T12:49:14.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3464413879}) 2024-08-19T12:49:14.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799871570}) 2024-08-19T12:49:14.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.257+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1860931221}) 2024-08-19T12:49:14.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048478745}) 2024-08-19T12:49:14.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2804075716}) 2024-08-19T12:49:14.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048478745}) 2024-08-19T12:49:14.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2868924148}) 2024-08-19T12:49:14.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048478745}) 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=270539446}) 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048478745}) 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.282+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2344770322}) 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048478745}) 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2365627823}) 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048478745}) 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4289920461}) 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048478745}) 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4204530318}) 2024-08-19T12:49:14.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2048478745}) 2024-08-19T12:49:14.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.283+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=6896719}) 2024-08-19T12:49:14.305 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:49:14.305 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:49:14.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.402+0000 7f25779de640 10 monclient: get_monmap_and_config 2024-08-19T12:49:14.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.402+0000 7f25779de640 10 monclient: build_initial_monmap 2024-08-19T12:49:14.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.402+0000 7f25779de640 10 monclient: monmap: 2024-08-19T12:49:14.403 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:14.403 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.403 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:14.403251+0000 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:14.403251+0000 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:14.404 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.402+0000 7f25779de640 1 Processor -- start 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f25779de640 1 -- start start 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f25779de640 10 monclient: init 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f25779de640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f25779de640 10 monclient: _add_conns ranks=[7,8,6,1,0,5,4,2,3] 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f25779de640 10 monclient(hunting): picked mon.noname-i con 0x7f25700a19f0 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f25779de640 10 monclient(hunting): picked mon.noname-e con 0x7f257009e180 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f25779de640 10 monclient(hunting): picked mon.noname-d con 0x7f257009e5d0 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f25779de640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25700a7300 con 0x7f257009e5d0 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f25779de640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25700a0970 con 0x7f25700a19f0 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f25779de640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f257009fbd0 con 0x7f257009e180 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f25779de640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f25779de640 10 monclient(hunting): authenticate will time out at 1348.490697s 2024-08-19T12:49:14.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f2574f79640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f257009e180 0x7f257013b1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:35720/0 (socket says 172.21.15.26:35720) 2024-08-19T12:49:14.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f2575f7b640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f257009e5d0 0x7f257013e900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:50998/0 (socket says 172.21.15.26:50998) 2024-08-19T12:49:14.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.403+0000 7f2574f79640 1 -- 172.21.15.26:0/4111337544 learned_addr learned my addr 172.21.15.26:0/4111337544 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:14.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.405+0000 7f2574f79640 1 -- 172.21.15.26:0/4111337544 >> v1:172.21.15.26:6793/0 conn(0x7f257009e180 legacy=0x7f257013b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2575f7b640 1 -- 172.21.15.26:0/4111337544 >> v1:172.21.15.26:6792/0 conn(0x7f257009e5d0 legacy=0x7f257013e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 1 -- 172.21.15.26:0/4111337544 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1991048318 0 0) 0x7f25700a7300 con 0x7f257009e5d0 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient(hunting): my global_id is 164409 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 1 -- 172.21.15.26:0/4111337544 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f255c004630 con 0x7f257009e5d0 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 1 -- 172.21.15.26:0/4111337544 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2087989277 0 0) 0x7f257009fbd0 con 0x7f257009e180 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient(hunting): my global_id is 174167 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 1 -- 172.21.15.26:0/4111337544 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25700a7300 con 0x7f257009e180 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f257577a640 1 -- 172.21.15.26:0/4111337544 >> v1:172.21.15.104:6792/0 conn(0x7f25700a19f0 legacy=0x7f257009da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 1 -- 172.21.15.26:0/4111337544 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 162029818 0 0) 0x7f255c004630 con 0x7f257009e5d0 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 1 -- 172.21.15.26:0/4111337544 >> v1:172.21.15.26:6793/0 conn(0x7f257009e180 legacy=0x7f257013b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 1 -- 172.21.15.26:0/4111337544 >> v1:172.21.15.104:6792/0 conn(0x7f25700a19f0 legacy=0x7f257009da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient: _finish_hunting 0 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 1 monclient: found mon.noname-d 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 1 -- 172.21.15.26:0/4111337544 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257009feb0 con 0x7f257009e5d0 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient: _finish_auth 0 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient: _check_auth_tickets 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 1 -- 172.21.15.26:0/4111337544 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2568001e60 con 0x7f257009e5d0 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:14.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient: dump: 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.413 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.414 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f25779de640 5 monclient: authenticate success, global_id 164409 2024-08-19T12:49:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f25779de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 1 -- 172.21.15.26:0/4111337544 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2568004b60 con 0x7f257009e5d0 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 1 -- 172.21.15.26:0/4111337544 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f25680050d0 con 0x7f257009e5d0 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f25779de640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f25779de640 4 set_mon_vals no callback set 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.406+0000 7f2563fff640 10 monclient: dump: 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.415 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.407+0000 7f25779de640 10 monclient: shutdown 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.407+0000 7f25779de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.407+0000 7f25779de640 1 -- 172.21.15.26:0/4111337544 >> v1:172.21.15.26:6792/0 conn(0x7f257009e5d0 legacy=0x7f257013e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.407+0000 7f25779de640 1 -- 172.21.15.26:0/4111337544 shutdown_connections 2024-08-19T12:49:14.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.407+0000 7f25779de640 1 -- 172.21.15.26:0/4111337544 >> 172.21.15.26:0/4111337544 conn(0x7f25700a44d0 msgr2=0x7f25700a48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.407+0000 7f25779de640 1 -- 172.21.15.26:0/4111337544 shutdown_connections 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.407+0000 7f25779de640 1 -- 172.21.15.26:0/4111337544 wait complete. 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.407+0000 7f25779de640 10 monclient: build_initial_monmap 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.407+0000 7f25779de640 10 monclient: monmap: 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:14.417 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 1 librados: starting msgr at 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 1 librados: starting objecter 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 1 Processor -- start 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 1 -- start start 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 1 librados: setting wanted keys 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 1 librados: calling monclient init 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 10 monclient: init 2024-08-19T12:49:14.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 10 monclient: _add_conns ranks=[3,2,5,8,7,6,4,1,0] 2024-08-19T12:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 10 monclient(hunting): picked mon.noname-g con 0x7f257009e180 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 10 monclient(hunting): picked mon.noname-b con 0x7f257009e5d0 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 10 monclient(hunting): picked mon.noname-h con 0x7f25700a19f0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25701a9850 con 0x7f257009e5d0 2024-08-19T12:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25701a9b30 con 0x7f257009e180 2024-08-19T12:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25700730d0 con 0x7f25700a19f0 2024-08-19T12:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f25779de640 10 monclient(hunting): authenticate will time out at 1348.495732s 2024-08-19T12:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f2574f79640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f257009e5d0 0x7f25701ce8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59354/0 (socket says 172.21.15.26:59354) 2024-08-19T12:49:14.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.408+0000 7f2574f79640 1 -- 172.21.15.26:0/638703184 learned_addr learned my addr 172.21.15.26:0/638703184 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.410+0000 7f2574f79640 1 -- 172.21.15.26:0/638703184 >> v1:172.21.15.26:6790/0 conn(0x7f257009e5d0 legacy=0x7f25701ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.410+0000 7f2561ffb640 10 client.?.objecter ms_handle_connect 0x7f257009e5d0 2024-08-19T12:49:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.410+0000 7f2561ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 464737899 0 0) 0x7f25701a9850 con 0x7f257009e5d0 2024-08-19T12:49:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): my global_id is 26741 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2554004eb0 con 0x7f257009e5d0 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 client.?.objecter ms_handle_connect 0x7f257009e180 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f257577a640 1 -- 172.21.15.26:0/638703184 >> v1:172.21.15.104:6790/0 conn(0x7f257009e180 legacy=0x7f25701bc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 client.?.objecter ms_handle_connect 0x7f25700a19f0 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2575f7b640 1 -- 172.21.15.26:0/638703184 >> v1:172.21.15.104:6791/0 conn(0x7f25700a19f0 legacy=0x7f25701d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3664463412 0 0) 0x7f2554004eb0 con 0x7f257009e5d0 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25701a9850 con 0x7f257009e5d0 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2564004b20 con 0x7f257009e5d0 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): dump: 2024-08-19T12:49:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.425 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.425 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.425 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.425 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.425 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.425 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.425 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.425 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.425 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.425 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.426 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.426 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.426 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.426 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.426 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:14.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3092319955 0 0) 0x7f25700730d0 con 0x7f25700a19f0 2024-08-19T12:49:14.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): my global_id is 26816 2024-08-19T12:49:14.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2554004eb0 con 0x7f25700a19f0 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1971342491 0 0) 0x7f25701a9b30 con 0x7f257009e180 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient(hunting): my global_id is 154578 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25700730d0 con 0x7f257009e180 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2324449605 0 0) 0x7f25701a9850 con 0x7f257009e5d0 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 >> v1:172.21.15.104:6791/0 conn(0x7f25700a19f0 legacy=0x7f25701d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 >> v1:172.21.15.104:6790/0 conn(0x7f257009e180 legacy=0x7f25701bc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 monclient: found mon.b 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:49:14.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25701a9570 con 0x7f257009e5d0 2024-08-19T12:49:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient: _finish_auth 0 2024-08-19T12:49:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:49:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f25779de640 5 monclient: authenticate success, global_id 26741 2024-08-19T12:49:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f25779de640 10 monclient: _renew_subs 2024-08-19T12:49:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f25779de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:49:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f25779de640 1 -- 172.21.15.26:0/638703184 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f25701bca70 con 0x7f257009e5d0 2024-08-19T12:49:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2564004e30 con 0x7f257009e5d0 2024-08-19T12:49:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2564005320 con 0x7f257009e5d0 2024-08-19T12:49:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:14.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f2561ffb640 10 monclient: dump: 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.429 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f25637fe640 4 set_mon_vals no callback set 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f25779de640 10 client.26741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f25779de640 10 monclient: _renew_subs 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f25779de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f25779de640 1 -- 172.21.15.26:0/638703184 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f25701c9570 con 0x7f257009e5d0 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.411+0000 7f25779de640 1 librados: init done 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.412+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2564011780 con 0x7f257009e5d0 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.412+0000 7f25779de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:14.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.412+0000 7f25779de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.412+0000 7f25779de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:49:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.412+0000 7f25779de640 1 -- 172.21.15.26:0/638703184 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2570071f50 con 0x7f257009e5d0 2024-08-19T12:49:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.412+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(408..408 src has 311..408) ==== 4961+0+0 (unknown 1193793308 0 0) 0x7f256404e080 con 0x7f257009e5d0 2024-08-19T12:49:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.412+0000 7f2561ffb640 10 client.26741.objecter ms_dispatch 0x7f2570000b90 osd_map(408..408 src has 311..408) 2024-08-19T12:49:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.412+0000 7f2561ffb640 3 client.26741.objecter handle_osd_map got epochs [408,408] > 0 2024-08-19T12:49:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.412+0000 7f2561ffb640 3 client.26741.objecter handle_osd_map decoding full epoch 408 2024-08-19T12:49:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.412+0000 7f2561ffb640 20 client.26741.objecter dump_active .. 0 homeless 2024-08-19T12:49:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.413+0000 7f257577a640 10 monclient: get_auth_request con 0x7f25540411e0 auth_method 0 2024-08-19T12:49:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.415+0000 7f2561ffb640 10 client.26741.objecter ms_handle_connect 0x7f25540411e0 2024-08-19T12:49:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.417+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f2564017540 con 0x7f257009e5d0 2024-08-19T12:49:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.417+0000 7f2561ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:14.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.417+0000 7f2561ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:14.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.670+0000 7f25779de640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:49:14.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.670+0000 7f25779de640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:49:14.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.670+0000 7f25779de640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:49:14.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.670+0000 7f25779de640 1 -- 172.21.15.26:0/638703184 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f25701c98b0 con 0x7f257009e5d0 2024-08-19T12:49:14.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.671+0000 7f2561ffb640 1 -- 172.21.15.26:0/638703184 <== mon.2 v1:172.21.15.26:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 2705320996) 0x7f256401b440 con 0x7f257009e5d0 2024-08-19T12:49:14.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.671+0000 7f2561ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:49:14.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.671+0000 7f2561ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:14.672 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":68,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:49:14.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 10 librados: watch_flush enter 2024-08-19T12:49:14.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 10 librados: watch_flush exit 2024-08-19T12:49:14.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 20 client.26741.objecter shutdown clearing up homeless session... 2024-08-19T12:49:14.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 10 client.26741.objecter successfully canceled tick 2024-08-19T12:49:14.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 1 -- 172.21.15.26:0/638703184 >> v1:172.21.15.104:6800/899887506 conn(0x7f25540411e0 legacy=0x7f25540436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 10 monclient: shutdown 2024-08-19T12:49:14.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:14.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 1 -- 172.21.15.26:0/638703184 >> v1:172.21.15.26:6790/0 conn(0x7f257009e5d0 legacy=0x7f25701ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 1 -- 172.21.15.26:0/638703184 shutdown_connections 2024-08-19T12:49:14.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 1 -- 172.21.15.26:0/638703184 >> 172.21.15.26:0/638703184 conn(0x7f25700a44d0 msgr2=0x7f25700a3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:14.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 1 -- 172.21.15.26:0/638703184 shutdown_connections 2024-08-19T12:49:14.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 1 -- 172.21.15.26:0/638703184 wait complete. 2024-08-19T12:49:14.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.673+0000 7f25779de640 1 librados: shutdown 2024-08-19T12:49:14.684 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.684 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291355590}) 2024-08-19T12:49:14.684 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=510520503}) 2024-08-19T12:49:14.684 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.684 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291355590}) 2024-08-19T12:49:14.684 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=407765411}) 2024-08-19T12:49:14.684 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.684 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291355590}) 2024-08-19T12:49:14.684 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=812300200}) 2024-08-19T12:49:14.684 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.684 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291355590}) 2024-08-19T12:49:14.685 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4164188198}) 2024-08-19T12:49:14.685 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.685 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291355590}) 2024-08-19T12:49:14.685 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1119617752}) 2024-08-19T12:49:14.685 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.685 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291355590}) 2024-08-19T12:49:14.685 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1091606581}) 2024-08-19T12:49:14.685 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.686 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291355590}) 2024-08-19T12:49:14.686 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3451210306}) 2024-08-19T12:49:14.686 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.686 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291355590}) 2024-08-19T12:49:14.686 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.682+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4097375202}) 2024-08-19T12:49:14.687 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:49:14.687 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T12:49:14.687 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:49:14.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:49:14.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477734841}) 2024-08-19T12:49:14.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4138473009}) 2024-08-19T12:49:14.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477734841}) 2024-08-19T12:49:14.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3084494546}) 2024-08-19T12:49:14.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477734841}) 2024-08-19T12:49:14.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1168799040}) 2024-08-19T12:49:14.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477734841}) 2024-08-19T12:49:14.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1130921703}) 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477734841}) 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3952715510}) 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477734841}) 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3955798231}) 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477734841}) 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=667036672}) 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477734841}) 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.683+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1464137498}) 2024-08-19T12:49:14.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.684+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.684+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858083445}) 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.684+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3048508703}) 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.684+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.684+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858083445}) 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.684+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3829559483}) 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.684+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.684+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858083445}) 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.684+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1148905376}) 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858083445}) 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3441061409}) 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858083445}) 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1268920837}) 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858083445}) 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=145966813}) 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858083445}) 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1747312673}) 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858083445}) 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.685+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2672432945}) 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204057689}) 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2548845921}) 2024-08-19T12:49:14.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204057689}) 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=56345885}) 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204057689}) 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1670550655}) 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204057689}) 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3874839485}) 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204057689}) 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=209711445}) 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204057689}) 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=624932196}) 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204057689}) 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=892278900}) 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204057689}) 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.686+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=124187102}) 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46688880}) 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=110373996}) 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46688880}) 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4013197093}) 2024-08-19T12:49:14.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46688880}) 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1289971036}) 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46688880}) 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4289098322}) 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46688880}) 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3405729233}) 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46688880}) 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1690006141}) 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46688880}) 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=810340128}) 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46688880}) 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.687+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3242879581}) 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822191537}) 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1612543459}) 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822191537}) 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1078011871}) 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822191537}) 2024-08-19T12:49:14.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=181301759}) 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822191537}) 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=870468122}) 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822191537}) 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3887749192}) 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822191537}) 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1705377323}) 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822191537}) 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1535369557}) 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822191537}) 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.688+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2958061408}) 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236950322}) 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3911573160}) 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236950322}) 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3756573011}) 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236950322}) 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=972979260}) 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236950322}) 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3515901455}) 2024-08-19T12:49:14.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236950322}) 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.689+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=911790923}) 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.690+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.690+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236950322}) 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.690+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=768392024}) 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.690+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.690+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236950322}) 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.690+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=259247018}) 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.690+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.690+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236950322}) 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.690+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=77767166}) 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339391338}) 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3731676205}) 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339391338}) 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3120572395}) 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339391338}) 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3380359650}) 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339391338}) 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1314773061}) 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339391338}) 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=198763787}) 2024-08-19T12:49:14.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339391338}) 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3064325581}) 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339391338}) 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2057304532}) 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339391338}) 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.691+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1359837694}) 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519830364}) 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3711985212}) 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519830364}) 2024-08-19T12:49:14.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1961083882}) 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519830364}) 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=260003669}) 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519830364}) 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=80825234}) 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519830364}) 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3225058194}) 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519830364}) 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2942046018}) 2024-08-19T12:49:14.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519830364}) 2024-08-19T12:49:14.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1432645820}) 2024-08-19T12:49:14.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519830364}) 2024-08-19T12:49:14.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.692+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=447186997}) 2024-08-19T12:49:14.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196171800}) 2024-08-19T12:49:14.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=317870943}) 2024-08-19T12:49:14.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.716 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196171800}) 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660685032}) 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196171800}) 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3027192521}) 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196171800}) 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2680851660}) 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196171800}) 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1960962816}) 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196171800}) 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1627031948}) 2024-08-19T12:49:14.717 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.718 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196171800}) 2024-08-19T12:49:14.718 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1745637947}) 2024-08-19T12:49:14.718 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.718 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196171800}) 2024-08-19T12:49:14.718 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.693+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=411832157}) 2024-08-19T12:49:14.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.694+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.694+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145247373}) 2024-08-19T12:49:14.735 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.694+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1097617941}) 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.694+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.694+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145247373}) 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.694+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3826481225}) 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.694+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.694+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145247373}) 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.694+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=439097712}) 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.694+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145247373}) 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4033411040}) 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145247373}) 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3023736816}) 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.736 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145247373}) 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1221274555}) 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145247373}) 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2810302720}) 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145247373}) 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.695+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2135166190}) 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512046529}) 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=943654102}) 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512046529}) 2024-08-19T12:49:14.737 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=6223223}) 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512046529}) 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4051128250}) 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512046529}) 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=19451327}) 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512046529}) 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=247123215}) 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512046529}) 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3643608780}) 2024-08-19T12:49:14.738 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512046529}) 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1393921518}) 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512046529}) 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.696+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3412240907}) 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=475372897}) 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=266391998}) 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=475372897}) 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3644366106}) 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=475372897}) 2024-08-19T12:49:14.739 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2279026181}) 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=475372897}) 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2029853168}) 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=475372897}) 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=814497818}) 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=475372897}) 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1273853039}) 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=475372897}) 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=689510355}) 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.740 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=475372897}) 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.697+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2594422790}) 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660164894}) 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=176551149}) 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660164894}) 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1234572472}) 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660164894}) 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3825549072}) 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660164894}) 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2611709237}) 2024-08-19T12:49:14.741 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660164894}) 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2384109614}) 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660164894}) 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4245520949}) 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660164894}) 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1686043862}) 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660164894}) 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.698+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3691635650}) 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.742 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263489528}) 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3757539539}) 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263489528}) 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3662959575}) 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263489528}) 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=902319450}) 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263489528}) 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3878812068}) 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263489528}) 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=967933500}) 2024-08-19T12:49:14.743 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.699+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.700+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263489528}) 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.700+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1231375174}) 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.700+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.700+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263489528}) 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.700+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=926763715}) 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.700+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.700+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2263489528}) 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.700+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4189644518}) 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703367733}) 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=670637750}) 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703367733}) 2024-08-19T12:49:14.744 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2433224620}) 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703367733}) 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3851756541}) 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703367733}) 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2214535511}) 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703367733}) 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2981266548}) 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703367733}) 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4145984358}) 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.745 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703367733}) 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1500946479}) 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703367733}) 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.701+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3587327246}) 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921515338}) 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4143418629}) 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921515338}) 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867899785}) 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921515338}) 2024-08-19T12:49:14.746 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3824903091}) 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921515338}) 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2722801393}) 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921515338}) 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2489139693}) 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921515338}) 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=962286213}) 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921515338}) 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1226766884}) 2024-08-19T12:49:14.747 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921515338}) 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.703+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2451649983}) 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068739746}) 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3799844665}) 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068739746}) 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1065751918}) 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068739746}) 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=473581069}) 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068739746}) 2024-08-19T12:49:14.748 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2591298908}) 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068739746}) 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2688785287}) 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068739746}) 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=202408158}) 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068739746}) 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1450537057}) 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068739746}) 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.704+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3854786616}) 2024-08-19T12:49:14.749 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116901844}) 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1494664848}) 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116901844}) 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2017226796}) 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116901844}) 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3898431609}) 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116901844}) 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=659557253}) 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116901844}) 2024-08-19T12:49:14.750 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1227439090}) 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116901844}) 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1888544007}) 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116901844}) 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2112361626}) 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.705+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.706+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116901844}) 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.706+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1839842016}) 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980833147}) 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3925416664}) 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.751 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980833147}) 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2902651863}) 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980833147}) 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3655800013}) 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980833147}) 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2894692808}) 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980833147}) 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2936987924}) 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980833147}) 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=444516056}) 2024-08-19T12:49:14.752 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980833147}) 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3513314107}) 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980833147}) 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.707+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=707533440}) 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165371719}) 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755366321}) 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165371719}) 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=257602664}) 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165371719}) 2024-08-19T12:49:14.753 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2319977834}) 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165371719}) 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1490261651}) 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165371719}) 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=290727525}) 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165371719}) 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=411805782}) 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165371719}) 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3299036967}) 2024-08-19T12:49:14.754 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165371719}) 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.708+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2970841216}) 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.709+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.709+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371158211}) 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.709+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=431663135}) 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.709+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.709+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371158211}) 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.709+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=427484128}) 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371158211}) 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1986221459}) 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.755 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371158211}) 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1733567712}) 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371158211}) 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2343341627}) 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371158211}) 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=816478900}) 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371158211}) 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1391379955}) 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371158211}) 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.710+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3500671122}) 2024-08-19T12:49:14.756 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412046242}) 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1081411836}) 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412046242}) 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=300323431}) 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412046242}) 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2475665059}) 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412046242}) 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1745884138}) 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412046242}) 2024-08-19T12:49:14.757 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2355299531}) 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412046242}) 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=41520525}) 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412046242}) 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3237749659}) 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412046242}) 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.711+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=490218022}) 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275718499}) 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1664946436}) 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.758 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275718499}) 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3347704209}) 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275718499}) 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2124275574}) 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275718499}) 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3662461704}) 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275718499}) 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1922046395}) 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275718499}) 2024-08-19T12:49:14.759 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3188826401}) 2024-08-19T12:49:14.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275718499}) 2024-08-19T12:49:14.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3386655527}) 2024-08-19T12:49:14.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:14.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275718499}) 2024-08-19T12:49:14.760 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:14.712+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3985887114}) 2024-08-19T12:49:14.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.797+0000 7fa82128b640 10 monclient: get_monmap_and_config 2024-08-19T12:49:14.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.797+0000 7fa82128b640 10 monclient: build_initial_monmap 2024-08-19T12:49:14.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.797+0000 7fa82128b640 10 monclient: monmap: 2024-08-19T12:49:14.798 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:14.798682+0000 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:14.798682+0000 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:14.800 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:14.800 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.798+0000 7fa82128b640 1 Processor -- start 2024-08-19T12:49:14.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.798+0000 7fa82128b640 1 -- start start 2024-08-19T12:49:14.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.798+0000 7fa82128b640 10 monclient: init 2024-08-19T12:49:14.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.798+0000 7fa82128b640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:14.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.798+0000 7fa82128b640 10 monclient: _add_conns ranks=[8,6,5,2,3,1,0,7,4] 2024-08-19T12:49:14.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.799+0000 7fa82128b640 10 monclient(hunting): picked mon.noname-e con 0x7fa81c06e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:14.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.799+0000 7fa82128b640 10 monclient(hunting): picked mon.noname-d con 0x7fa81c0a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:14.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.799+0000 7fa82128b640 10 monclient(hunting): picked mon.noname-h con 0x7fa81c0a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:14.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.799+0000 7fa82128b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa81c136790 con 0x7fa81c0a5d50 2024-08-19T12:49:14.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.799+0000 7fa82128b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa81c09ddf0 con 0x7fa81c0a5920 2024-08-19T12:49:14.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.799+0000 7fa82128b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa81c09e0d0 con 0x7fa81c06e7b0 2024-08-19T12:49:14.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.799+0000 7fa82128b640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:14.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.799+0000 7fa82128b640 10 monclient(hunting): authenticate will time out at 1348.886111s 2024-08-19T12:49:14.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.799+0000 7fa81a7fc640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fa81c0a5920 0x7fa81c13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51008/0 (socket says 172.21.15.26:51008) 2024-08-19T12:49:14.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.799+0000 7fa81a7fc640 1 -- 172.21.15.26:0/1709049066 learned_addr learned my addr 172.21.15.26:0/1709049066 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:14.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.799+0000 7fa81affd640 1 --1- 172.21.15.26:0/1709049066 >> v1:172.21.15.26:6793/0 conn(0x7fa81c06e7b0 0x7fa81c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:35734/0 (socket says 172.21.15.26:35734) 2024-08-19T12:49:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa81affd640 1 -- 172.21.15.26:0/1709049066 >> v1:172.21.15.26:6793/0 conn(0x7fa81c06e7b0 legacy=0x7fa81c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa81a7fc640 1 -- 172.21.15.26:0/1709049066 >> v1:172.21.15.26:6792/0 conn(0x7fa81c0a5920 legacy=0x7fa81c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 1 -- 172.21.15.26:0/1709049066 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2260168016 0 0) 0x7fa804001380 con 0x7fa81c0a5920 2024-08-19T12:49:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa81b7fe640 1 -- 172.21.15.26:0/1709049066 >> v1:172.21.15.104:6791/0 conn(0x7fa81c0a5d50 legacy=0x7fa81c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient(hunting): my global_id is 164418 2024-08-19T12:49:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 1 -- 172.21.15.26:0/1709049066 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa808003520 con 0x7fa81c0a5920 2024-08-19T12:49:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 1 -- 172.21.15.26:0/1709049066 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1973336944 0 0) 0x7fa81c09e0d0 con 0x7fa81c06e7b0 2024-08-19T12:49:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient(hunting): my global_id is 174176 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 1 -- 172.21.15.26:0/1709049066 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa804001380 con 0x7fa81c06e7b0 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 1 -- 172.21.15.26:0/1709049066 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1248391093 0 0) 0x7fa808003520 con 0x7fa81c0a5920 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 1 -- 172.21.15.26:0/1709049066 >> v1:172.21.15.26:6793/0 conn(0x7fa81c06e7b0 legacy=0x7fa81c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 1 -- 172.21.15.26:0/1709049066 >> v1:172.21.15.104:6791/0 conn(0x7fa81c0a5d50 legacy=0x7fa81c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 1 monclient: found mon.noname-d 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 1 -- 172.21.15.26:0/1709049066 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c09e3b0 con 0x7fa81c0a5920 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient: _finish_auth 0 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:49:14.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 1 -- 172.21.15.26:0/1709049066 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa804001be0 con 0x7fa81c0a5920 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa82128b640 5 monclient: authenticate success, global_id 164418 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa82128b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient: dump: 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.808 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa82128b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 1 -- 172.21.15.26:0/1709049066 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8040048a0 con 0x7fa81c0a5920 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 1 -- 172.21.15.26:0/1709049066 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa804004dd0 con 0x7fa81c0a5920 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa82128b640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:14.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa82128b640 4 set_mon_vals no callback set 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.801+0000 7fa819ffb640 10 monclient: dump: 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.802+0000 7fa82128b640 10 monclient: shutdown 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.802+0000 7fa82128b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.802+0000 7fa82128b640 1 -- 172.21.15.26:0/1709049066 >> v1:172.21.15.26:6792/0 conn(0x7fa81c0a5920 legacy=0x7fa81c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.802+0000 7fa82128b640 1 -- 172.21.15.26:0/1709049066 shutdown_connections 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.802+0000 7fa82128b640 1 -- 172.21.15.26:0/1709049066 >> 172.21.15.26:0/1709049066 conn(0x7fa81c0a8740 msgr2=0x7fa81c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.802+0000 7fa82128b640 1 -- 172.21.15.26:0/1709049066 shutdown_connections 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.802+0000 7fa82128b640 1 -- 172.21.15.26:0/1709049066 wait complete. 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.803+0000 7fa82128b640 10 monclient: build_initial_monmap 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.803+0000 7fa82128b640 10 monclient: monmap: 2024-08-19T12:49:14.811 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:14.812 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.803+0000 7fa82128b640 1 librados: starting msgr at 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.803+0000 7fa82128b640 1 librados: starting objecter 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.803+0000 7fa82128b640 1 Processor -- start 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.803+0000 7fa82128b640 1 -- start start 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.803+0000 7fa82128b640 1 librados: setting wanted keys 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.803+0000 7fa82128b640 1 librados: calling monclient init 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.803+0000 7fa82128b640 10 monclient: init 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.804+0000 7fa82128b640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.804+0000 7fa82128b640 10 monclient: _add_conns ranks=[0,1,4,6,3,5,7,2,8] 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.804+0000 7fa82128b640 10 monclient(hunting): picked mon.noname-a con 0x7fa81c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.804+0000 7fa82128b640 10 monclient(hunting): picked mon.noname-f con 0x7fa81c0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.804+0000 7fa82128b640 10 monclient(hunting): picked mon.noname-c con 0x7fa81c0a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.804+0000 7fa82128b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa81c142ae0 con 0x7fa81c06e7b0 2024-08-19T12:49:14.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.804+0000 7fa82128b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa81c142e30 con 0x7fa81c0a5920 2024-08-19T12:49:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.804+0000 7fa82128b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa81c138560 con 0x7fa81c0a5d50 2024-08-19T12:49:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.804+0000 7fa82128b640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.804+0000 7fa82128b640 10 monclient(hunting): authenticate will time out at 1348.891177s 2024-08-19T12:49:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.804+0000 7fa81affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa81c06e7b0 0x7fa81c137740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39812/0 (socket says 172.21.15.26:39812) 2024-08-19T12:49:14.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.804+0000 7fa81affd640 1 -- 172.21.15.26:0/1778053708 learned_addr learned my addr 172.21.15.26:0/1778053708 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa81b7fe640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.26:6791/0 conn(0x7fa81c0a5d50 legacy=0x7fa81c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa81affd640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.26:6789/0 conn(0x7fa81c06e7b0 legacy=0x7fa81c137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 client.?.objecter ms_handle_connect 0x7fa81c0a5d50 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 client.?.objecter ms_handle_connect 0x7fa81c06e7b0 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3026054378 0 0) 0x7fa81c138560 con 0x7fa81c0a5d50 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): my global_id is 27067 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7fc004eb0 con 0x7fa81c0a5d50 2024-08-19T12:49:14.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3504155220 0 0) 0x7fa81c142ae0 con 0x7fa81c06e7b0 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): my global_id is 145062 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa81c138560 con 0x7fa81c06e7b0 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa81a7fc640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.104:6789/0 conn(0x7fa81c0a5920 legacy=0x7fa81c137e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 client.?.objecter ms_handle_connect 0x7fa81c0a5920 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4119666349 0 0) 0x7fa7fc004eb0 con 0x7fa81c0a5d50 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa81c142ae0 con 0x7fa81c0a5d50 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa814001cb0 con 0x7fa81c0a5d50 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:49:14.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): dump: 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.821 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 389054527 0 0) 0x7fa81c142e30 con 0x7fa81c0a5920 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): my global_id is 27604 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7fc004eb0 con 0x7fa81c0a5920 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1458999349 0 0) 0x7fa81c138560 con 0x7fa81c06e7b0 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa81c142e30 con 0x7fa81c06e7b0 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa80c004b60 con 0x7fa81c06e7b0 2024-08-19T12:49:14.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 10 monclient(hunting): dump: 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.823 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.806+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 957416441 0 0) 0x7fa81c142ae0 con 0x7fa81c0a5d50 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.104:6789/0 conn(0x7fa81c0a5920 legacy=0x7fa81c137e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.26:6789/0 conn(0x7fa81c06e7b0 legacy=0x7fa81c137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 10 monclient: _finish_hunting 0 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 1 monclient: found mon.c 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c13aad0 con 0x7fa81c0a5d50 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 10 monclient: _finish_auth 0 2024-08-19T12:49:14.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 10 monclient: _check_auth_tickets 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa82128b640 5 monclient: authenticate success, global_id 27067 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa82128b640 10 monclient: _renew_subs 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa82128b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa82128b640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa81c13b790 con 0x7fa81c0a5d50 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa82128b640 10 client.27067.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa82128b640 10 monclient: _renew_subs 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa82128b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa82128b640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fa81c139090 con 0x7fa81c0a5d50 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa82128b640 1 librados: init done 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8140038c0 con 0x7fa81c0a5d50 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa814003e30 con 0x7fa81c0a5d50 2024-08-19T12:49:14.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8197fa640 4 set_mon_vals no callback set 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa8137fe640 10 monclient: dump: 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.826 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa82128b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.807+0000 7fa82128b640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa8000065c0 con 0x7fa800002860 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.808+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa8140040d0 con 0x7fa81c0a5d50 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.808+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(408..408 src has 311..408) ==== 4961+0+0 (unknown 1193793308 0 0) 0x7fa81404de20 con 0x7fa81c0a5d50 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.808+0000 7fa8137fe640 10 client.27067.objecter ms_dispatch 0x7fa81c000b90 osd_map(408..408 src has 311..408) 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.808+0000 7fa8137fe640 3 client.27067.objecter handle_osd_map got epochs [408,408] > 0 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.808+0000 7fa8137fe640 3 client.27067.objecter handle_osd_map decoding full epoch 408 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.808+0000 7fa8137fe640 20 client.27067.objecter dump_active .. 0 homeless 2024-08-19T12:49:14.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.808+0000 7fa81a7fc640 10 monclient: get_auth_request con 0x7fa7fc0416c0 auth_method 0 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 10 client.27067.objecter ms_handle_connect 0x7fa800002860 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 10 client.27067.objecter resend_mon_ops 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa81affd640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.26:6789/0 conn(0x7fa800002860 legacy=0x7fa800004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2795834388 0 0) 0x7fa8000065c0 con 0x7fa800002860 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 10 monclient(hunting): my global_id is 145071 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa81c142ae0 con 0x7fa800002860 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1735758866 0 0) 0x7fa81c142ae0 con 0x7fa800002860 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8000065c0 con 0x7fa800002860 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa80c005230 con 0x7fa800002860 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:14.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 10 monclient: dump: 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.829 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.830 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.830 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 822643041 0 0) 0x7fa8000065c0 con 0x7fa800002860 2024-08-19T12:49:14.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa80000a160 con 0x7fa800002860 2024-08-19T12:49:14.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.810+0000 7fa8137fe640 10 client.27067.objecter ms_handle_connect 0x7fa7fc0416c0 2024-08-19T12:49:14.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.811+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa80c007550 con 0x7fa800002860 2024-08-19T12:49:14.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.811+0000 7fa8137fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:14.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.811+0000 7fa8137fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:14.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.811+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.26:6789/0 conn(0x7fa800002860 legacy=0x7fa800004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.811+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.26:6789/0 conn(0x7fa800002860 legacy=0x7fa800004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:14.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.817+0000 7fa82128b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:14.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.817+0000 7fa82128b640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa800009e70 con 0x7fa800009130 2024-08-19T12:49:14.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa81affd640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.26:6789/0 conn(0x7fa800009130 legacy=0x7fa8000095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 10 client.27067.objecter ms_handle_connect 0x7fa800009130 2024-08-19T12:49:14.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 10 client.27067.objecter resend_mon_ops 2024-08-19T12:49:14.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2745847290 0 0) 0x7fa800009e70 con 0x7fa800009130 2024-08-19T12:49:14.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 10 monclient(hunting): my global_id is 145080 2024-08-19T12:49:14.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8000065c0 con 0x7fa800009130 2024-08-19T12:49:14.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2776059097 0 0) 0x7fa8000065c0 con 0x7fa800009130 2024-08-19T12:49:14.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa800009e70 con 0x7fa800009130 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa80c006c10 con 0x7fa800009130 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 10 monclient: dump: 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.833 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.834 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3388671948 0 0) 0x7fa800009e70 con 0x7fa800009130 2024-08-19T12:49:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.819+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa80000c410 con 0x7fa800009130 2024-08-19T12:49:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7546 (unknown 0 0 2258790079) 0x7fa80c007020 con 0x7fa800009130 2024-08-19T12:49:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa8137fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa8137fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.26:6789/0 conn(0x7fa800009130 legacy=0x7fa8000095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa8137fe640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.26:6789/0 conn(0x7fa800009130 legacy=0x7fa8000095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:14.835 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:49:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 406737, 2024-08-19T12:49:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:14.835 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:14.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:14.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:14.838 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:14.839 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:14.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:14.841 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:14.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.844 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:14.845 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:14.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:14.848 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:14.849 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:14.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:14.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:14.852 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:14.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:14.854 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:14.854 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:14.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 10 librados: watch_flush enter 2024-08-19T12:49:14.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 10 librados: watch_flush exit 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 20 client.27067.objecter shutdown clearing up homeless session... 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 10 client.27067.objecter successfully canceled tick 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.104:6800/899887506 conn(0x7fa7fc0416c0 legacy=0x7fa7fc043b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 10 monclient: shutdown 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 1 -- 172.21.15.26:0/1778053708 >> v1:172.21.15.26:6791/0 conn(0x7fa81c0a5d50 legacy=0x7fa81c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa81b7fe640 1 -- 172.21.15.26:0/1778053708 reap_dead start 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa81b7fe640 1 -- 172.21.15.26:0/1778053708 reap_dead start 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 1 -- 172.21.15.26:0/1778053708 shutdown_connections 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 1 -- 172.21.15.26:0/1778053708 >> 172.21.15.26:0/1778053708 conn(0x7fa81c0a8740 msgr2=0x7fa81c0aa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 1 -- 172.21.15.26:0/1778053708 shutdown_connections 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 1 -- 172.21.15.26:0/1778053708 wait complete. 2024-08-19T12:49:14.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.820+0000 7fa82128b640 1 librados: shutdown 2024-08-19T12:49:14.857 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:14.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:49:14.857 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:49:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.956+0000 7f077056b640 10 monclient: get_monmap_and_config 2024-08-19T12:49:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.956+0000 7f077056b640 10 monclient: build_initial_monmap 2024-08-19T12:49:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.956+0000 7f077056b640 10 monclient: monmap: 2024-08-19T12:49:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:14.957101+0000 2024-08-19T12:49:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:14.957101+0000 2024-08-19T12:49:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:14.957 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:14.958 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:14.958 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:14.958 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:14.958 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:14.958 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:14.958 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:14.958 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.956+0000 7f077056b640 1 Processor -- start 2024-08-19T12:49:14.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.956+0000 7f077056b640 1 -- start start 2024-08-19T12:49:14.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.956+0000 7f077056b640 10 monclient: init 2024-08-19T12:49:14.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f077056b640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:14.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f077056b640 10 monclient: _add_conns ranks=[4,7,0,6,1,3,8,2,5] 2024-08-19T12:49:14.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f077056b640 10 monclient(hunting): picked mon.noname-c con 0x7f0768073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:14.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f077056b640 10 monclient(hunting): picked mon.noname-i con 0x7f07680a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:14.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f077056b640 10 monclient(hunting): picked mon.noname-a con 0x7f076813f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:14.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f077056b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f076809dae0 con 0x7f076813f2e0 2024-08-19T12:49:14.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f077056b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0768136910 con 0x7f0768073470 2024-08-19T12:49:14.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f077056b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f076809dd50 con 0x7f07680a57a0 2024-08-19T12:49:14.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f077056b640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:14.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f077056b640 10 monclient(hunting): authenticate will time out at 1349.044540s 2024-08-19T12:49:14.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f076e307640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f0768073470 0x7f0768136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57510/0 (socket says 172.21.15.26:57510) 2024-08-19T12:49:14.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f076e307640 1 -- 172.21.15.26:0/2969106148 learned_addr learned my addr 172.21.15.26:0/2969106148 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:14.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.957+0000 7f076eb08640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f076813f2e0 0x7f07681416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39836/0 (socket says 172.21.15.26:39836) 2024-08-19T12:49:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.959+0000 7f076eb08640 1 -- 172.21.15.26:0/2969106148 >> v1:172.21.15.26:6789/0 conn(0x7f076813f2e0 legacy=0x7f07681416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.959+0000 7f076e307640 1 -- 172.21.15.26:0/2969106148 >> v1:172.21.15.26:6791/0 conn(0x7f0768073470 legacy=0x7f0768136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076db06640 1 -- 172.21.15.26:0/2969106148 >> v1:172.21.15.104:6792/0 conn(0x7f07680a57a0 legacy=0x7f07680a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1945797118 0 0) 0x7f076809dae0 con 0x7f076813f2e0 2024-08-19T12:49:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient(hunting): my global_id is 145089 2024-08-19T12:49:14.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0754003520 con 0x7f076813f2e0 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3523596193 0 0) 0x7f0768136910 con 0x7f0768073470 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient(hunting): my global_id is 27076 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f076809dae0 con 0x7f0768073470 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3241027682 0 0) 0x7f076809dd50 con 0x7f07680a57a0 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient(hunting): my global_id is 135709 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0768136910 con 0x7f07680a57a0 2024-08-19T12:49:14.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1201134654 0 0) 0x7f0754003520 con 0x7f076813f2e0 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 >> v1:172.21.15.104:6792/0 conn(0x7f07680a57a0 legacy=0x7f07680a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 >> v1:172.21.15.26:6791/0 conn(0x7f0768073470 legacy=0x7f0768136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient: _finish_hunting 0 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 monclient: found mon.noname-a 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076809e030 con 0x7f076813f2e0 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient: _finish_auth 0 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient: _check_auth_tickets 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0760001e60 con 0x7f076813f2e0 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f077056b640 5 monclient: authenticate success, global_id 145089 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f077056b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:14.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient: dump: 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.969 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.970 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f077056b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0760004b60 con 0x7f076813f2e0 2024-08-19T12:49:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 1 -- 172.21.15.26:0/2969106148 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f07600050d0 con 0x7f076813f2e0 2024-08-19T12:49:14.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f076d305640 10 monclient: dump: 2024-08-19T12:49:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f077056b640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.960+0000 7f077056b640 4 set_mon_vals no callback set 2024-08-19T12:49:14.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.961+0000 7f077056b640 10 monclient: shutdown 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.961+0000 7f077056b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.961+0000 7f077056b640 1 -- 172.21.15.26:0/2969106148 >> v1:172.21.15.26:6789/0 conn(0x7f076813f2e0 legacy=0x7f07681416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.961+0000 7f077056b640 1 -- 172.21.15.26:0/2969106148 shutdown_connections 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.961+0000 7f077056b640 1 -- 172.21.15.26:0/2969106148 >> 172.21.15.26:0/2969106148 conn(0x7f07680abff0 msgr2=0x7f07680ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.961+0000 7f077056b640 1 -- 172.21.15.26:0/2969106148 shutdown_connections 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.961+0000 7f077056b640 1 -- 172.21.15.26:0/2969106148 wait complete. 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.961+0000 7f077056b640 10 monclient: build_initial_monmap 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.961+0000 7f077056b640 10 monclient: monmap: 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.973 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:14.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.961+0000 7f077056b640 1 librados: starting msgr at 2024-08-19T12:49:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.961+0000 7f077056b640 1 librados: starting objecter 2024-08-19T12:49:14.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 1 Processor -- start 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 1 -- start start 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 1 librados: setting wanted keys 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 1 librados: calling monclient init 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 10 monclient: init 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 10 monclient: _add_conns ranks=[6,8,4,2,1,3,7,5,0] 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 10 monclient(hunting): picked mon.noname-d con 0x7f0768073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 10 monclient(hunting): picked mon.noname-e con 0x7f07680a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 10 monclient(hunting): picked mon.noname-c con 0x7f076813f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0768135250 con 0x7f076813f2e0 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0768142b60 con 0x7f0768073470 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0768136f30 con 0x7f07680a57a0 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:14.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f077056b640 10 monclient(hunting): authenticate will time out at 1349.049651s 2024-08-19T12:49:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f076eb08640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f076813f2e0 0x7f07680a2c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57512/0 (socket says 172.21.15.26:57512) 2024-08-19T12:49:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f076e307640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f0768073470 0x7f07680a1e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51012/0 (socket says 172.21.15.26:51012) 2024-08-19T12:49:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.962+0000 7f076eb08640 1 -- 172.21.15.26:0/74605280 learned_addr learned my addr 172.21.15.26:0/74605280 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.964+0000 7f076eb08640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6791/0 conn(0x7f076813f2e0 legacy=0x7f07680a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.964+0000 7f076db06640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6793/0 conn(0x7f07680a57a0 legacy=0x7f07680a2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.964+0000 7f075effd640 10 client.?.objecter ms_handle_connect 0x7f076813f2e0 2024-08-19T12:49:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.964+0000 7f075effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.964+0000 7f076e307640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6792/0 conn(0x7f0768073470 legacy=0x7f07680a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.964+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:49:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 client.?.objecter ms_handle_connect 0x7f07680a57a0 2024-08-19T12:49:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 client.?.objecter ms_handle_connect 0x7f0768073470 2024-08-19T12:49:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4204045764 0 0) 0x7f0768135250 con 0x7f076813f2e0 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): my global_id is 27085 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f074c004eb0 con 0x7f076813f2e0 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2231886066 0 0) 0x7f0768142b60 con 0x7f0768073470 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): my global_id is 164427 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0768135250 con 0x7f0768073470 2024-08-19T12:49:14.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3632471689 0 0) 0x7f0768136f30 con 0x7f07680a57a0 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): my global_id is 174185 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0768142b60 con 0x7f07680a57a0 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3276623142 0 0) 0x7f0768135250 con 0x7f0768073470 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0768136f30 con 0x7f0768073470 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f07640047e0 con 0x7f0768073470 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): dump: 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.984 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 637353347 0 0) 0x7f074c004eb0 con 0x7f076813f2e0 2024-08-19T12:49:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0768135250 con 0x7f076813f2e0 2024-08-19T12:49:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1546909451 0 0) 0x7f0768142b60 con 0x7f07680a57a0 2024-08-19T12:49:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f074c004eb0 con 0x7f07680a57a0 2024-08-19T12:49:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f07600020b0 con 0x7f076813f2e0 2024-08-19T12:49:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): dump: 2024-08-19T12:49:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0758003610 con 0x7f07680a57a0 2024-08-19T12:49:14.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient(hunting): dump: 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.988 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.989 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.989 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.989 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.989 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 515450857 0 0) 0x7f0768136f30 con 0x7f0768073470 2024-08-19T12:49:14.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6793/0 conn(0x7f07680a57a0 legacy=0x7f07680a2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6791/0 conn(0x7f076813f2e0 legacy=0x7f07680a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient: _finish_hunting 0 2024-08-19T12:49:14.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 monclient: found mon.d 2024-08-19T12:49:14.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:14.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:14.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07680a42e0 con 0x7f0768073470 2024-08-19T12:49:14.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient: _finish_auth 0 2024-08-19T12:49:14.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient: _check_auth_tickets 2024-08-19T12:49:14.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:14.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f077056b640 5 monclient: authenticate success, global_id 164427 2024-08-19T12:49:14.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f077056b640 10 monclient: _renew_subs 2024-08-19T12:49:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f077056b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f077056b640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f07680a4570 con 0x7f0768073470 2024-08-19T12:49:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f077056b640 10 client.164427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f077056b640 10 monclient: _renew_subs 2024-08-19T12:49:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f077056b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f077056b640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f07681d7340 con 0x7f0768073470 2024-08-19T12:49:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f077056b640 1 librados: init done 2024-08-19T12:49:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0764004c30 con 0x7f0768073470 2024-08-19T12:49:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0764005120 con 0x7f0768073470 2024-08-19T12:49:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:14.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075ffff640 4 set_mon_vals no callback set 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.967+0000 7f075effd640 10 monclient: dump: 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.993 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.968+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f0764011580 con 0x7f0768073470 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.968+0000 7f077056b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.968+0000 7f077056b640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f07500065c0 con 0x7f0750002860 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.968+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(408..408 src has 311..408) ==== 4961+0+0 (unknown 1193793308 0 0) 0x7f076404dea0 con 0x7f0768073470 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.968+0000 7f075effd640 10 client.164427.objecter ms_dispatch 0x7f0768000b90 osd_map(408..408 src has 311..408) 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.968+0000 7f075effd640 3 client.164427.objecter handle_osd_map got epochs [408,408] > 0 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.968+0000 7f075effd640 3 client.164427.objecter handle_osd_map decoding full epoch 408 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.969+0000 7f075effd640 20 client.164427.objecter dump_active .. 0 homeless 2024-08-19T12:49:14.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.969+0000 7f076eb08640 10 monclient: get_auth_request con 0x7f074c041c80 auth_method 0 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 10 client.164427.objecter ms_handle_connect 0x7f0750002860 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 10 client.164427.objecter resend_mon_ops 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f076db06640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f0750002860 legacy=0x7f0750004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1157467064 0 0) 0x7f07500065c0 con 0x7f0750002860 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 10 monclient(hunting): my global_id is 26750 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0768136f30 con 0x7f0750002860 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 946949720 0 0) 0x7f0768136f30 con 0x7f0750002860 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07500065c0 con 0x7f0750002860 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0758003e50 con 0x7f0750002860 2024-08-19T12:49:14.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 10 monclient: dump: 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:14.996 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 10 client.164427.objecter ms_handle_connect 0x7f074c041c80 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3824952062 0 0) 0x7f07500065c0 con 0x7f0750002860 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f075000a160 con 0x7f0750002860 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0758003ee0 con 0x7f0750002860 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f0750002860 legacy=0x7f0750004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.971+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f0750002860 legacy=0x7f0750004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.978+0000 7f077056b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:14.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.978+0000 7f077056b640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0750009e50 con 0x7f0750009110 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f076db06640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f0750009110 legacy=0x7f07500095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f075effd640 10 client.164427.objecter ms_handle_connect 0x7f0750009110 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f075effd640 10 client.164427.objecter resend_mon_ops 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 498853381 0 0) 0x7f0750009e50 con 0x7f0750009110 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f075effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f075effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f075effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f075effd640 10 monclient(hunting): my global_id is 26759 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07500065c0 con 0x7f0750009110 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f076db06640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f0750009110 legacy=0x7f07500095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f076db06640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f0750009110 legacy=0x7f07500095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f076db06640 1 --1- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f0750009110 0x7f07500095a0 :-1 s=OPENED pgs=3478 cs=1 l=1).handle_message read tag failed 2024-08-19T12:49:14.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f076db06640 1 --1- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f0750009110 0x7f07500095a0 :-1 s=OPENED pgs=3478 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:49:14.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f0750009110 legacy=0x7f07500095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:14.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f0750009110 legacy=0x7f07500095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:14.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.980+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0750009e50 con 0x7f074c04bce0 2024-08-19T12:49:15.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f076db06640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f074c04bce0 legacy=0x7f074c04a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 10 client.164427.objecter ms_handle_connect 0x7f074c04bce0 2024-08-19T12:49:15.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 10 client.164427.objecter resend_mon_ops 2024-08-19T12:49:15.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1708083109 0 0) 0x7f0750009e50 con 0x7f074c04bce0 2024-08-19T12:49:15.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 10 monclient(hunting): my global_id is 26768 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0768136f30 con 0x7f074c04bce0 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 448506764 0 0) 0x7f0768136f30 con 0x7f074c04bce0 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0750009e50 con 0x7f074c04bce0 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0758004490 con 0x7f074c04bce0 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 10 monclient: dump: 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.001 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3828043181 0 0) 0x7f0750009e50 con 0x7f074c04bce0 2024-08-19T12:49:15.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0758003ee0 con 0x7f074c04bce0 2024-08-19T12:49:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 405378877) 0x7f0758005de0 con 0x7f074c04bce0 2024-08-19T12:49:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f074c04bce0 legacy=0x7f074c04a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f075effd640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6790/0 conn(0x7f074c04bce0 legacy=0x7f074c04a710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.983+0000 7f076eb08640 1 -- 172.21.15.26:0/74605280 reap_dead start 2024-08-19T12:49:15.003 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:15.003 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:15.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:15.003 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 668038, 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:15.004 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.006 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:15.007 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:15.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.010 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:15.014 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:15.016 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.017 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:15.021 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.022 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.024 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:15.024 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:15.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 10 librados: watch_flush enter 2024-08-19T12:49:15.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 10 librados: watch_flush exit 2024-08-19T12:49:15.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 20 client.164427.objecter shutdown clearing up homeless session... 2024-08-19T12:49:15.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 10 client.164427.objecter successfully canceled tick 2024-08-19T12:49:15.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.104:6800/899887506 conn(0x7f074c041c80 legacy=0x7f074c044140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 10 monclient: shutdown 2024-08-19T12:49:15.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:15.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 1 -- 172.21.15.26:0/74605280 >> v1:172.21.15.26:6792/0 conn(0x7f0768073470 legacy=0x7f07680a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 1 -- 172.21.15.26:0/74605280 shutdown_connections 2024-08-19T12:49:15.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 1 -- 172.21.15.26:0/74605280 >> 172.21.15.26:0/74605280 conn(0x7f07680abff0 msgr2=0x7f0768134850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:15.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 1 -- 172.21.15.26:0/74605280 shutdown_connections 2024-08-19T12:49:15.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 1 -- 172.21.15.26:0/74605280 wait complete. 2024-08-19T12:49:15.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:14.984+0000 7f077056b640 1 librados: shutdown 2024-08-19T12:49:15.026 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:15.027 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:49:15.027 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:49:15.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.126+0000 7f26d9d86640 10 monclient: get_monmap_and_config 2024-08-19T12:49:15.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.126+0000 7f26d9d86640 10 monclient: build_initial_monmap 2024-08-19T12:49:15.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.126+0000 7f26d9d86640 10 monclient: monmap: 2024-08-19T12:49:15.127 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:15.127 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.128 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:15.127462+0000 2024-08-19T12:49:15.128 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:15.127462+0000 2024-08-19T12:49:15.128 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:15.128 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:15.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:15.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:15.128 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:15.128 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:15.129 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:15.129 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:15.129 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:15.129 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:15.129 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.127+0000 7f26d9d86640 1 Processor -- start 2024-08-19T12:49:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.127+0000 7f26d9d86640 1 -- start start 2024-08-19T12:49:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.127+0000 7f26d9d86640 10 monclient: init 2024-08-19T12:49:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.127+0000 7f26d9d86640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.127+0000 7f26d9d86640 10 monclient: _add_conns ranks=[8,1,0,7,4,2,5,6,3] 2024-08-19T12:49:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.127+0000 7f26d9d86640 10 monclient(hunting): picked mon.noname-e con 0x7f26d406e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.127+0000 7f26d9d86640 10 monclient(hunting): picked mon.noname-f con 0x7f26d40a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.127+0000 7f26d9d86640 10 monclient(hunting): picked mon.noname-a con 0x7f26d40a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.127+0000 7f26d9d86640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d4136790 con 0x7f26d40a5d50 2024-08-19T12:49:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.127+0000 7f26d9d86640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d409ddf0 con 0x7f26d40a5920 2024-08-19T12:49:15.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.127+0000 7f26d9d86640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d409e0d0 con 0x7f26d406e7b0 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.128+0000 7f26d9d86640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.128+0000 7f26d9d86640 10 monclient(hunting): authenticate will time out at 1349.215015s 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.128+0000 7f26d37fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f26d406e7b0 0x7f26d4136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:35756/0 (socket says 172.21.15.26:35756) 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.128+0000 7f26d37fe640 1 -- 172.21.15.26:0/2646408776 learned_addr learned my addr 172.21.15.26:0/2646408776 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d3fff640 1 -- 172.21.15.26:0/2646408776 >> v1:172.21.15.26:6789/0 conn(0x7f26d40a5d50 legacy=0x7f26d4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d37fe640 1 -- 172.21.15.26:0/2646408776 >> v1:172.21.15.26:6793/0 conn(0x7f26d406e7b0 legacy=0x7f26d4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d2ffd640 1 -- 172.21.15.26:0/2646408776 >> v1:172.21.15.104:6789/0 conn(0x7f26d40a5920 legacy=0x7f26d413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1786582508 0 0) 0x7f26d4136790 con 0x7f26d40a5d50 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient(hunting): my global_id is 145098 2024-08-19T12:49:15.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26bc003520 con 0x7f26d40a5d50 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 637333795 0 0) 0x7f26d409e0d0 con 0x7f26d406e7b0 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient(hunting): my global_id is 174194 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d4136790 con 0x7f26d406e7b0 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2355449054 0 0) 0x7f26c80014a0 con 0x7f26d40a5920 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient(hunting): my global_id is 27613 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d409e0d0 con 0x7f26d40a5920 2024-08-19T12:49:15.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 847259481 0 0) 0x7f26bc003520 con 0x7f26d40a5d50 2024-08-19T12:49:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 >> v1:172.21.15.26:6793/0 conn(0x7f26d406e7b0 legacy=0x7f26d4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 >> v1:172.21.15.104:6789/0 conn(0x7f26d40a5920 legacy=0x7f26d413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 monclient: found mon.noname-a 2024-08-19T12:49:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:49:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d409e3b0 con 0x7f26d40a5d50 2024-08-19T12:49:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f26cc001eb0 con 0x7f26d40a5d50 2024-08-19T12:49:15.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d9d86640 5 monclient: authenticate success, global_id 145098 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d9d86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient: dump: 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.138 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d9d86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26cc003990 con 0x7f26d40a5d50 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 1 -- 172.21.15.26:0/2646408776 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f26cc003f00 con 0x7f26d40a5d50 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d27fc640 10 monclient: dump: 2024-08-19T12:49:15.139 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.140 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d9d86640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.130+0000 7f26d9d86640 4 set_mon_vals no callback set 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.131+0000 7f26d9d86640 10 monclient: shutdown 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.131+0000 7f26d9d86640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.131+0000 7f26d9d86640 1 -- 172.21.15.26:0/2646408776 >> v1:172.21.15.26:6789/0 conn(0x7f26d40a5d50 legacy=0x7f26d4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.131+0000 7f26d9d86640 1 -- 172.21.15.26:0/2646408776 shutdown_connections 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.131+0000 7f26d9d86640 1 -- 172.21.15.26:0/2646408776 >> 172.21.15.26:0/2646408776 conn(0x7f26d40a8740 msgr2=0x7f26d40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.131+0000 7f26d9d86640 1 -- 172.21.15.26:0/2646408776 shutdown_connections 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.131+0000 7f26d9d86640 1 -- 172.21.15.26:0/2646408776 wait complete. 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.131+0000 7f26d9d86640 10 monclient: build_initial_monmap 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.131+0000 7f26d9d86640 10 monclient: monmap: 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:15.141 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:15.142 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 1 librados: starting msgr at 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 1 librados: starting objecter 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 1 Processor -- start 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 1 -- start start 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 1 librados: setting wanted keys 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 1 librados: calling monclient init 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 10 monclient: init 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 10 monclient: _add_conns ranks=[8,5,1,2,6,0,4,3,7] 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 10 monclient(hunting): picked mon.noname-e con 0x7f26d406e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 10 monclient(hunting): picked mon.noname-h con 0x7f26d40a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:15.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 10 monclient(hunting): picked mon.noname-f con 0x7f26d40a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:15.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d40a31f0 con 0x7f26d40a5d50 2024-08-19T12:49:15.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d40a19f0 con 0x7f26d40a5920 2024-08-19T12:49:15.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d40a25b0 con 0x7f26d406e7b0 2024-08-19T12:49:15.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d9d86640 10 monclient(hunting): authenticate will time out at 1349.219925s 2024-08-19T12:49:15.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.132+0000 7f26d37fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f26d406e7b0 0x7f26d409fb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:35770/0 (socket says 172.21.15.26:35770) 2024-08-19T12:49:15.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.133+0000 7f26d37fe640 1 -- 172.21.15.26:0/1202604298 learned_addr learned my addr 172.21.15.26:0/1202604298 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:15.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26d37fe640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.26:6793/0 conn(0x7f26d406e7b0 legacy=0x7f26d409fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 client.?.objecter ms_handle_connect 0x7f26d406e7b0 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 client.?.objecter ms_handle_connect 0x7f26d40a5d50 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 client.?.objecter ms_handle_connect 0x7f26d40a5920 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26d3fff640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.104:6789/0 conn(0x7f26d40a5d50 legacy=0x7f26d41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26d2ffd640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.104:6791/0 conn(0x7f26d40a5920 legacy=0x7f26d40a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3327458423 0 0) 0x7f26d40a25b0 con 0x7f26d406e7b0 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): my global_id is 174203 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26c0004eb0 con 0x7f26d406e7b0 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1583939004 0 0) 0x7f26d40a31f0 con 0x7f26d40a5d50 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): my global_id is 27622 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d40a25b0 con 0x7f26d40a5d50 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 720139389 0 0) 0x7f26d40a19f0 con 0x7f26d40a5920 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): my global_id is 26834 2024-08-19T12:49:15.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d40a31f0 con 0x7f26d40a5920 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4116600326 0 0) 0x7f26c0004eb0 con 0x7f26d406e7b0 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26d40a19f0 con 0x7f26d406e7b0 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f26c40047e0 con 0x7f26d406e7b0 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): dump: 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.153 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.154 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4269078656 0 0) 0x7f26d40a25b0 con 0x7f26d40a5d50 2024-08-19T12:49:15.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26c0004eb0 con 0x7f26d40a5d50 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f26cc003e80 con 0x7f26d40a5d50 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient(hunting): dump: 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.155 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2946879539 0 0) 0x7f26d40a19f0 con 0x7f26d406e7b0 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.104:6791/0 conn(0x7f26d40a5920 legacy=0x7f26d40a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.104:6789/0 conn(0x7f26d40a5d50 legacy=0x7f26d41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 monclient: found mon.e 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d40a0990 con 0x7f26d406e7b0 2024-08-19T12:49:15.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26d9d86640 5 monclient: authenticate success, global_id 174203 2024-08-19T12:49:15.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26d9d86640 10 monclient: _renew_subs 2024-08-19T12:49:15.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26d9d86640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:49:15.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26d9d86640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f26d40a0c20 con 0x7f26d406e7b0 2024-08-19T12:49:15.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.135+0000 7f26d9d86640 10 client.174203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:15.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26d9d86640 10 monclient: _renew_subs 2024-08-19T12:49:15.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26d9d86640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:49:15.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26d9d86640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f26d41d7b90 con 0x7f26d406e7b0 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26d9d86640 1 librados: init done 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26c4004c30 con 0x7f26d406e7b0 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26bbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f26c4005120 con 0x7f26d406e7b0 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26d1ffb640 4 set_mon_vals no callback set 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26bbfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26bbfff640 10 monclient: dump: 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.158 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26d9d86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26d9d86640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f26b40065c0 con 0x7f26b4002860 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.136+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f26c40052c0 con 0x7f26d406e7b0 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.137+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(408..408 src has 311..408) ==== 4961+0+0 (unknown 1193793308 0 0) 0x7f26c404e020 con 0x7f26d406e7b0 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.137+0000 7f26bbfff640 10 client.174203.objecter ms_dispatch 0x7f26d4000b90 osd_map(408..408 src has 311..408) 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.137+0000 7f26bbfff640 3 client.174203.objecter handle_osd_map got epochs [408,408] > 0 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.137+0000 7f26bbfff640 3 client.174203.objecter handle_osd_map decoding full epoch 408 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.137+0000 7f26bbfff640 20 client.174203.objecter dump_active .. 0 homeless 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.137+0000 7f26d3fff640 10 monclient: get_auth_request con 0x7f26c00414f0 auth_method 0 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 10 client.174203.objecter ms_handle_connect 0x7f26b4002860 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 10 client.174203.objecter resend_mon_ops 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26d2ffd640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.26:6791/0 conn(0x7f26b4002860 legacy=0x7f26b4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2753831856 0 0) 0x7f26b40065c0 con 0x7f26b4002860 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 10 monclient(hunting): my global_id is 27094 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d40a19f0 con 0x7f26b4002860 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1554064917 0 0) 0x7f26d40a19f0 con 0x7f26b4002860 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26b40065c0 con 0x7f26b4002860 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f26c8003e30 con 0x7f26b4002860 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 10 monclient: dump: 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.161 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3318505332 0 0) 0x7f26b40065c0 con 0x7f26b4002860 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26b400a160 con 0x7f26b4002860 2024-08-19T12:49:15.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 10 client.174203.objecter ms_handle_connect 0x7f26c00414f0 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f26c8003e30 con 0x7f26b4002860 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.26:6791/0 conn(0x7f26b4002860 legacy=0x7f26b4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.139+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.26:6791/0 conn(0x7f26b4002860 legacy=0x7f26b4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.146+0000 7f26d9d86640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.146+0000 7f26d9d86640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f26b4009e50 con 0x7f26b4009110 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 10 client.174203.objecter ms_handle_connect 0x7f26b4009110 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 10 client.174203.objecter resend_mon_ops 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26d2ffd640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.26:6791/0 conn(0x7f26b4009110 legacy=0x7f26b40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1242118978 0 0) 0x7f26b4009e50 con 0x7f26b4009110 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 10 monclient(hunting): my global_id is 27103 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26b40065c0 con 0x7f26b4009110 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3214125918 0 0) 0x7f26b40065c0 con 0x7f26b4009110 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26b4009e50 con 0x7f26b4009110 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f26c8004440 con 0x7f26b4009110 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 10 monclient: dump: 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.164 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1345503400 0 0) 0x7f26b4009e50 con 0x7f26b4009110 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f26b400c3a0 con 0x7f26b4009110 2024-08-19T12:49:15.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1172418711) 0x7f26c8007b80 con 0x7f26b4009110 2024-08-19T12:49:15.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:15.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.148+0000 7f26bbfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:15.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.26:6791/0 conn(0x7f26b4009110 legacy=0x7f26b40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26bbfff640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.26:6791/0 conn(0x7f26b4009110 legacy=0x7f26b40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.166 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 668178, 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:49:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:15.186 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 10 librados: watch_flush enter 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 10 librados: watch_flush exit 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 20 client.174203.objecter shutdown clearing up homeless session... 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 10 client.174203.objecter successfully canceled tick 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.104:6800/899887506 conn(0x7f26c00414f0 legacy=0x7f26c00439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 10 monclient: shutdown 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 1 -- 172.21.15.26:0/1202604298 >> v1:172.21.15.26:6793/0 conn(0x7f26d406e7b0 legacy=0x7f26d409fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d3fff640 1 -- 172.21.15.26:0/1202604298 reap_dead start 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d3fff640 1 -- 172.21.15.26:0/1202604298 reap_dead start 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 1 -- 172.21.15.26:0/1202604298 shutdown_connections 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 1 -- 172.21.15.26:0/1202604298 >> 172.21.15.26:0/1202604298 conn(0x7f26d40a8740 msgr2=0x7f26d4135150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 1 -- 172.21.15.26:0/1202604298 shutdown_connections 2024-08-19T12:49:15.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 1 -- 172.21.15.26:0/1202604298 wait complete. 2024-08-19T12:49:15.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.149+0000 7f26d9d86640 1 librados: shutdown 2024-08-19T12:49:15.189 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:15.189 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:49:15.189 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:49:15.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106370386}) 2024-08-19T12:49:15.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1803050486}) 2024-08-19T12:49:15.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106370386}) 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078739706}) 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106370386}) 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3381261686}) 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106370386}) 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3986320572}) 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106370386}) 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1466408369}) 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106370386}) 2024-08-19T12:49:15.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3919885088}) 2024-08-19T12:49:15.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106370386}) 2024-08-19T12:49:15.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3602803916}) 2024-08-19T12:49:15.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106370386}) 2024-08-19T12:49:15.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1088544213}) 2024-08-19T12:49:15.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800862380}) 2024-08-19T12:49:15.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3939234989}) 2024-08-19T12:49:15.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800862380}) 2024-08-19T12:49:15.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1321947048}) 2024-08-19T12:49:15.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800862380}) 2024-08-19T12:49:15.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1655939769}) 2024-08-19T12:49:15.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800862380}) 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=23804369}) 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800862380}) 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=990554371}) 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800862380}) 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3838319949}) 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800862380}) 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=642741780}) 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800862380}) 2024-08-19T12:49:15.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.264+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1601204640}) 2024-08-19T12:49:15.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=759324201}) 2024-08-19T12:49:15.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4115756781}) 2024-08-19T12:49:15.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=759324201}) 2024-08-19T12:49:15.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1164452697}) 2024-08-19T12:49:15.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=759324201}) 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2901139052}) 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=759324201}) 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=903644615}) 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=759324201}) 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3880619734}) 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=759324201}) 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1257902071}) 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=759324201}) 2024-08-19T12:49:15.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1015324009}) 2024-08-19T12:49:15.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=759324201}) 2024-08-19T12:49:15.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.272+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2861255699}) 2024-08-19T12:49:15.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.301+0000 7fe61fa6e640 10 monclient: get_monmap_and_config 2024-08-19T12:49:15.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.301+0000 7fe61fa6e640 10 monclient: build_initial_monmap 2024-08-19T12:49:15.303 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.301+0000 7fe61fa6e640 10 monclient: monmap: 2024-08-19T12:49:15.303 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:15.303 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.303 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:15.302793+0000 2024-08-19T12:49:15.303 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:15.302793+0000 2024-08-19T12:49:15.303 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:15.303 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.303 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:15.303 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:15.303 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:15.304 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:15.304 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:15.304 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:15.304 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:15.304 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:15.304 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:15.304 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.304 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.303+0000 7fe61fa6e640 1 Processor -- start 2024-08-19T12:49:15.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.303+0000 7fe61fa6e640 1 -- start start 2024-08-19T12:49:15.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.303+0000 7fe61fa6e640 10 monclient: init 2024-08-19T12:49:15.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.304+0000 7fe61fa6e640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.304+0000 7fe61fa6e640 10 monclient: _add_conns ranks=[4,8,0,7,3,1,6,2,5] 2024-08-19T12:49:15.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.304+0000 7fe61fa6e640 10 monclient(hunting): picked mon.noname-c con 0x7fe6180a68e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:15.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.304+0000 7fe61fa6e640 10 monclient(hunting): picked mon.noname-e con 0x7fe6180a73d0 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:15.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.305+0000 7fe61fa6e640 10 monclient(hunting): picked mon.noname-a con 0x7fe61813c8f0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:15.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.305+0000 7fe61fa6e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6180a0390 con 0x7fe61813c8f0 2024-08-19T12:49:15.306 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.305+0000 7fe61e00b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe61813c8f0 0x7fe61813ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39870/0 (socket says 172.21.15.26:39870) 2024-08-19T12:49:15.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.305+0000 7fe61e00b640 1 -- 172.21.15.26:0/232621652 learned_addr learned my addr 172.21.15.26:0/232621652 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:15.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.305+0000 7fe61fa6e640 1 -- 172.21.15.26:0/232621652 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6180a0670 con 0x7fe6180a68e0 2024-08-19T12:49:15.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.305+0000 7fe61fa6e640 1 -- 172.21.15.26:0/232621652 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6180a0950 con 0x7fe6180a73d0 2024-08-19T12:49:15.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.305+0000 7fe61fa6e640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.305+0000 7fe61fa6e640 10 monclient(hunting): authenticate will time out at 1349.392210s 2024-08-19T12:49:15.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.307+0000 7fe61e00b640 1 -- 172.21.15.26:0/232621652 >> v1:172.21.15.26:6789/0 conn(0x7fe61813c8f0 legacy=0x7fe61813ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.307+0000 7fe61d80a640 1 -- 172.21.15.26:0/232621652 >> v1:172.21.15.26:6791/0 conn(0x7fe6180a68e0 legacy=0x7fe6180a6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.307+0000 7fe61d009640 1 -- 172.21.15.26:0/232621652 >> v1:172.21.15.26:6793/0 conn(0x7fe6180a73d0 legacy=0x7fe61813b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 1 -- 172.21.15.26:0/232621652 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1869246428 0 0) 0x7fe6180a0950 con 0x7fe6180a73d0 2024-08-19T12:49:15.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient(hunting): my global_id is 174212 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 1 -- 172.21.15.26:0/232621652 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe604003520 con 0x7fe6180a73d0 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 1 -- 172.21.15.26:0/232621652 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 552109787 0 0) 0x7fe6180a0670 con 0x7fe6180a68e0 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient(hunting): my global_id is 27112 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 1 -- 172.21.15.26:0/232621652 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6180a0950 con 0x7fe6180a68e0 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 1 -- 172.21.15.26:0/232621652 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1319356444 0 0) 0x7fe604003520 con 0x7fe6180a73d0 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 1 -- 172.21.15.26:0/232621652 >> v1:172.21.15.26:6791/0 conn(0x7fe6180a68e0 legacy=0x7fe6180a6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 1 -- 172.21.15.26:0/232621652 >> v1:172.21.15.26:6789/0 conn(0x7fe61813c8f0 legacy=0x7fe61813ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 1 monclient: found mon.noname-e 2024-08-19T12:49:15.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:49:15.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 1 -- 172.21.15.26:0/232621652 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6180742e0 con 0x7fe6180a73d0 2024-08-19T12:49:15.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 1 -- 172.21.15.26:0/232621652 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe608001b30 con 0x7fe6180a73d0 2024-08-19T12:49:15.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:49:15.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient: dump: 2024-08-19T12:49:15.312 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.312 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.313 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61fa6e640 5 monclient: authenticate success, global_id 174212 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61fa6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61d009640 1 --1- 172.21.15.26:0/232621652 >> v1:172.21.15.26:6793/0 conn(0x7fe6180a73d0 0x7fe61813b1d0 :-1 s=OPENED pgs=3431 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fe608003610 config(0 keys) 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 1 -- 172.21.15.26:0/232621652 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe608003610 con 0x7fe6180a73d0 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61c808640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61fa6e640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.308+0000 7fe61fa6e640 4 set_mon_vals no callback set 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.309+0000 7fe61fa6e640 10 monclient: shutdown 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.309+0000 7fe61fa6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.309+0000 7fe61fa6e640 1 -- 172.21.15.26:0/232621652 >> v1:172.21.15.26:6793/0 conn(0x7fe6180a73d0 legacy=0x7fe61813b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.309+0000 7fe61c808640 1 -- 172.21.15.26:0/232621652 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe608003b80 con 0x7fe6180a73d0 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.309+0000 7fe61c808640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.309+0000 7fe61fa6e640 1 -- 172.21.15.26:0/232621652 shutdown_connections 2024-08-19T12:49:15.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.309+0000 7fe61fa6e640 1 -- 172.21.15.26:0/232621652 >> 172.21.15.26:0/232621652 conn(0x7fe6180a44d0 msgr2=0x7fe6180a48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:15.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.309+0000 7fe61fa6e640 1 -- 172.21.15.26:0/232621652 shutdown_connections 2024-08-19T12:49:15.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.309+0000 7fe61fa6e640 1 -- 172.21.15.26:0/232621652 wait complete. 2024-08-19T12:49:15.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265711159}) 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2000926979}) 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265711159}) 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2122287850}) 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265711159}) 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4120026738}) 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265711159}) 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=262491534}) 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265711159}) 2024-08-19T12:49:15.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3139005312}) 2024-08-19T12:49:15.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265711159}) 2024-08-19T12:49:15.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1156461701}) 2024-08-19T12:49:15.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265711159}) 2024-08-19T12:49:15.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2201803218}) 2024-08-19T12:49:15.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:15.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265711159}) 2024-08-19T12:49:15.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1522463619}) 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fe61fa6e640 10 monclient: build_initial_monmap 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fe61fa6e640 10 monclient: monmap: 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:15.324 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fe61fa6e640 1 librados: starting msgr at 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fe61fa6e640 1 librados: starting objecter 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.310+0000 7fe61fa6e640 1 Processor -- start 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 1 -- start start 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 1 librados: setting wanted keys 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 1 librados: calling monclient init 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 10 monclient: init 2024-08-19T12:49:15.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 10 monclient: _add_conns ranks=[1,5,0,3,7,4,6,8,2] 2024-08-19T12:49:15.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 10 monclient(hunting): picked mon.noname-f con 0x7fe6180a68e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:15.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 10 monclient(hunting): picked mon.noname-h con 0x7fe6180a73d0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:15.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 10 monclient(hunting): picked mon.noname-a con 0x7fe61813c8f0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:15.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe618126a00 con 0x7fe61813c8f0 2024-08-19T12:49:15.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6180a6410 con 0x7fe6180a68e0 2024-08-19T12:49:15.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe61813ae60 con 0x7fe6180a73d0 2024-08-19T12:49:15.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61fa6e640 10 monclient(hunting): authenticate will time out at 1349.398688s 2024-08-19T12:49:15.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61e00b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe61813c8f0 0x7fe6181d43f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39892/0 (socket says 172.21.15.26:39892) 2024-08-19T12:49:15.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.311+0000 7fe61e00b640 1 -- 172.21.15.26:0/3287742242 learned_addr learned my addr 172.21.15.26:0/3287742242 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.313+0000 7fe61e00b640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.26:6789/0 conn(0x7fe61813c8f0 legacy=0x7fe6181d43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 client.?.objecter ms_handle_connect 0x7fe61813c8f0 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe61d80a640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.104:6789/0 conn(0x7fe6180a68e0 legacy=0x7fe61813a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 client.?.objecter ms_handle_connect 0x7fe6180a68e0 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 client.?.objecter ms_handle_connect 0x7fe6180a73d0 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe61d009640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.104:6791/0 conn(0x7fe6180a73d0 legacy=0x7fe6181d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1461307715 0 0) 0x7fe61813ae60 con 0x7fe6180a73d0 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): my global_id is 26843 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5fc004eb0 con 0x7fe6180a73d0 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2910911193 0 0) 0x7fe6180a6410 con 0x7fe6180a68e0 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): my global_id is 27631 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe61813ae60 con 0x7fe6180a68e0 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 814873946 0 0) 0x7fe5fc004eb0 con 0x7fe6180a73d0 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6180a6410 con 0x7fe6180a73d0 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe608001ab0 con 0x7fe6180a73d0 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:15.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): dump: 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.329 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 948488951 0 0) 0x7fe61813ae60 con 0x7fe6180a68e0 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5fc004eb0 con 0x7fe6180a68e0 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe614001280 con 0x7fe6180a68e0 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient(hunting): dump: 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.330 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.331 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1448471433 0 0) 0x7fe6180a6410 con 0x7fe6180a73d0 2024-08-19T12:49:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.104:6789/0 conn(0x7fe6180a68e0 legacy=0x7fe61813a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.26:6789/0 conn(0x7fe61813c8f0 legacy=0x7fe6181d43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 monclient: found mon.h 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6181246a0 con 0x7fe6180a73d0 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe60e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe61fa6e640 5 monclient: authenticate success, global_id 26843 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe61fa6e640 10 monclient: _renew_subs 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe61fa6e640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.314+0000 7fe61fa6e640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe618126720 con 0x7fe6180a73d0 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe61fa6e640 10 client.26843.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe61fa6e640 10 monclient: _renew_subs 2024-08-19T12:49:15.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe61fa6e640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe61fa6e640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fe618125200 con 0x7fe6180a73d0 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe61fa6e640 1 librados: init done 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe608003610 con 0x7fe6180a73d0 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe60e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe6080015d0 con 0x7fe6180a73d0 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe60e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe60e7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe60f7fe640 4 set_mon_vals no callback set 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe60e7fc640 10 monclient: dump: 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.334 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.335 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.336 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe61fa6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.315+0000 7fe61fa6e640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe6000065c0 con 0x7fe600002860 2024-08-19T12:49:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.316+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe608011a80 con 0x7fe6180a73d0 2024-08-19T12:49:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.316+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(409..409 src has 311..409) ==== 5333+0+0 (unknown 2618201168 0 0) 0x7fe60804e120 con 0x7fe6180a73d0 2024-08-19T12:49:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.316+0000 7fe60e7fc640 10 client.26843.objecter ms_dispatch 0x7fe618000b90 osd_map(409..409 src has 311..409) 2024-08-19T12:49:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.316+0000 7fe60e7fc640 3 client.26843.objecter handle_osd_map got epochs [409,409] > 0 2024-08-19T12:49:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.316+0000 7fe60e7fc640 3 client.26843.objecter handle_osd_map decoding full epoch 409 2024-08-19T12:49:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.316+0000 7fe60e7fc640 20 client.26843.objecter dump_active .. 0 homeless 2024-08-19T12:49:15.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.316+0000 7fe61e00b640 10 monclient: get_auth_request con 0x7fe5fc041500 auth_method 0 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe61d80a640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.26:6792/0 conn(0x7fe600002860 legacy=0x7fe600004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 10 client.26843.objecter ms_handle_connect 0x7fe600002860 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 10 client.26843.objecter resend_mon_ops 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2622823886 0 0) 0x7fe6000065c0 con 0x7fe600002860 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 10 monclient(hunting): my global_id is 164436 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6180a6410 con 0x7fe600002860 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2215350758 0 0) 0x7fe6180a6410 con 0x7fe600002860 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6000065c0 con 0x7fe600002860 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe614003bc0 con 0x7fe600002860 2024-08-19T12:49:15.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 10 monclient: dump: 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.338 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 666193256 0 0) 0x7fe6000065c0 con 0x7fe600002860 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe60000a160 con 0x7fe600002860 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.318+0000 7fe60e7fc640 10 client.26843.objecter ms_handle_connect 0x7fe5fc041500 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.319+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe614003ed0 con 0x7fe600002860 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.319+0000 7fe60e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.319+0000 7fe60e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.319+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.26:6792/0 conn(0x7fe600002860 legacy=0x7fe600004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.319+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.26:6792/0 conn(0x7fe600002860 legacy=0x7fe600004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.325+0000 7fe61fa6e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.325+0000 7fe61fa6e640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe600009e70 con 0x7fe600009130 2024-08-19T12:49:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.327+0000 7fe60e7fc640 10 client.26843.objecter ms_handle_connect 0x7fe600009130 2024-08-19T12:49:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.327+0000 7fe61d80a640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.26:6792/0 conn(0x7fe600009130 legacy=0x7fe6000095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.327+0000 7fe60e7fc640 10 client.26843.objecter resend_mon_ops 2024-08-19T12:49:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.327+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3422777825 0 0) 0x7fe600009e70 con 0x7fe600009130 2024-08-19T12:49:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.327+0000 7fe60e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.327+0000 7fe60e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.327+0000 7fe60e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.327+0000 7fe60e7fc640 10 monclient(hunting): my global_id is 164445 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.327+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6000065c0 con 0x7fe600009130 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3596843853 0 0) 0x7fe6000065c0 con 0x7fe600009130 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe600009e70 con 0x7fe600009130 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe6140043f0 con 0x7fe600009130 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 10 monclient: dump: 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.342 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 468897615 0 0) 0x7fe600009e70 con 0x7fe600009130 2024-08-19T12:49:15.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe60000c410 con 0x7fe600009130 2024-08-19T12:49:15.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 2725899553) 0x7fe614007b80 con 0x7fe600009130 2024-08-19T12:49:15.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:15.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:15.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.26:6792/0 conn(0x7fe600009130 legacy=0x7fe6000095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe60e7fc640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.26:6792/0 conn(0x7fe600009130 legacy=0x7fe6000095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.344 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:15.344 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 138882, 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:15.345 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.346 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.347 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:15.348 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.349 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.350 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.351 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.352 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:15.355 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:15.356 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:15.357 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.358 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:15.362 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.363 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.364 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:15.364 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:15.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe61fa6e640 10 librados: watch_flush enter 2024-08-19T12:49:15.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe61fa6e640 10 librados: watch_flush exit 2024-08-19T12:49:15.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe61fa6e640 20 client.26843.objecter shutdown clearing up homeless session... 2024-08-19T12:49:15.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe61fa6e640 10 client.26843.objecter successfully canceled tick 2024-08-19T12:49:15.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe61fa6e640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.104:6800/899887506 conn(0x7fe5fc041500 legacy=0x7fe5fc0439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe61fa6e640 10 monclient: shutdown 2024-08-19T12:49:15.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe61fa6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:15.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe61fa6e640 1 -- 172.21.15.26:0/3287742242 >> v1:172.21.15.104:6791/0 conn(0x7fe6180a73d0 legacy=0x7fe6181d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe61e00b640 1 -- 172.21.15.26:0/3287742242 reap_dead start 2024-08-19T12:49:15.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.328+0000 7fe61e00b640 1 -- 172.21.15.26:0/3287742242 reap_dead start 2024-08-19T12:49:15.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.329+0000 7fe61fa6e640 1 -- 172.21.15.26:0/3287742242 shutdown_connections 2024-08-19T12:49:15.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.329+0000 7fe61fa6e640 1 -- 172.21.15.26:0/3287742242 >> 172.21.15.26:0/3287742242 conn(0x7fe6180a44d0 msgr2=0x7fe6180a3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:15.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.329+0000 7fe61fa6e640 1 -- 172.21.15.26:0/3287742242 shutdown_connections 2024-08-19T12:49:15.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.329+0000 7fe61fa6e640 1 -- 172.21.15.26:0/3287742242 wait complete. 2024-08-19T12:49:15.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.329+0000 7fe61fa6e640 1 librados: shutdown 2024-08-19T12:49:15.367 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:15.368 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:49:15.368 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:49:15.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.466+0000 7f9883cf6640 10 monclient: get_monmap_and_config 2024-08-19T12:49:15.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.466+0000 7f9883cf6640 10 monclient: build_initial_monmap 2024-08-19T12:49:15.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.466+0000 7f9883cf6640 10 monclient: monmap: 2024-08-19T12:49:15.467 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:15.467 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.467 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:15.467547+0000 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:15.467547+0000 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:15.468 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 1 Processor -- start 2024-08-19T12:49:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 1 -- start start 2024-08-19T12:49:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 10 monclient: init 2024-08-19T12:49:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 10 monclient: _add_conns ranks=[7,0,2,1,3,4,6,8,5] 2024-08-19T12:49:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 10 monclient(hunting): picked mon.noname-i con 0x7f987c06e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 10 monclient(hunting): picked mon.noname-a con 0x7f987c0a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 10 monclient(hunting): picked mon.noname-b con 0x7f987c0a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987c136790 con 0x7f987c0a5920 2024-08-19T12:49:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987c09ddf0 con 0x7f987c0a5d50 2024-08-19T12:49:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987c09e0d0 con 0x7f987c06e7b0 2024-08-19T12:49:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.467+0000 7f9883cf6640 10 monclient(hunting): authenticate will time out at 1349.554957s 2024-08-19T12:49:15.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.468+0000 7f9881291640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f987c0a5920 0x7f987c13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39894/0 (socket says 172.21.15.26:39894) 2024-08-19T12:49:15.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.468+0000 7f9881291640 1 -- 172.21.15.26:0/3223886733 learned_addr learned my addr 172.21.15.26:0/3223886733 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:15.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9881291640 1 -- 172.21.15.26:0/3223886733 >> v1:172.21.15.26:6789/0 conn(0x7f987c0a5920 legacy=0x7f987c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9882293640 1 -- 172.21.15.26:0/3223886733 >> v1:172.21.15.26:6790/0 conn(0x7f987c0a5d50 legacy=0x7f987c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247866125 0 0) 0x7f987c136790 con 0x7f987c0a5920 2024-08-19T12:49:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient(hunting): my global_id is 145107 2024-08-19T12:49:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9868003520 con 0x7f987c0a5920 2024-08-19T12:49:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2892217165 0 0) 0x7f9874001750 con 0x7f987c0a5d50 2024-08-19T12:49:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.475 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient(hunting): my global_id is 26777 2024-08-19T12:49:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9881a92640 1 -- 172.21.15.26:0/3223886733 >> v1:172.21.15.104:6792/0 conn(0x7f987c06e7b0 legacy=0x7f987c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f987c136790 con 0x7f987c0a5d50 2024-08-19T12:49:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2120714243 0 0) 0x7f987c09e0d0 con 0x7f987c06e7b0 2024-08-19T12:49:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient(hunting): my global_id is 135718 2024-08-19T12:49:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9874001750 con 0x7f987c06e7b0 2024-08-19T12:49:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3042167771 0 0) 0x7f9868003520 con 0x7f987c0a5920 2024-08-19T12:49:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 >> v1:172.21.15.104:6792/0 conn(0x7f987c06e7b0 legacy=0x7f987c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 >> v1:172.21.15.26:6790/0 conn(0x7f987c0a5d50 legacy=0x7f987c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 monclient: found mon.noname-a 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987c09e3b0 con 0x7f987c0a5920 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f986c001ae0 con 0x7f987c0a5920 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9883cf6640 5 monclient: authenticate success, global_id 145107 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9883cf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:49:15.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient: dump: 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.478 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9883cf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f986c0048a0 con 0x7f987c0a5920 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9883cf6640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 1 -- 172.21.15.26:0/3223886733 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f986c004e10 con 0x7f987c0a5920 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9883cf6640 4 set_mon_vals no callback set 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.470+0000 7f9880a90640 10 monclient: dump: 2024-08-19T12:49:15.479 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.480 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.471+0000 7f9883cf6640 10 monclient: shutdown 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.471+0000 7f9883cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.471+0000 7f9883cf6640 1 -- 172.21.15.26:0/3223886733 >> v1:172.21.15.26:6789/0 conn(0x7f987c0a5920 legacy=0x7f987c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.471+0000 7f9883cf6640 1 -- 172.21.15.26:0/3223886733 shutdown_connections 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.471+0000 7f9883cf6640 1 -- 172.21.15.26:0/3223886733 >> 172.21.15.26:0/3223886733 conn(0x7f987c0a8740 msgr2=0x7f987c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.471+0000 7f9883cf6640 1 -- 172.21.15.26:0/3223886733 shutdown_connections 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.471+0000 7f9883cf6640 1 -- 172.21.15.26:0/3223886733 wait complete. 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.471+0000 7f9883cf6640 10 monclient: build_initial_monmap 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 10 monclient: monmap: 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:15.481 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 1 librados: starting msgr at 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 1 librados: starting objecter 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 1 Processor -- start 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 1 -- start start 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 1 librados: setting wanted keys 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 1 librados: calling monclient init 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 10 monclient: init 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 10 monclient: _add_conns ranks=[2,0,8,6,3,1,5,4,7] 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 10 monclient(hunting): picked mon.noname-b con 0x7f987c06e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 10 monclient(hunting): picked mon.noname-a con 0x7f987c0a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 10 monclient(hunting): picked mon.noname-e con 0x7f987c0a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987c138e10 con 0x7f987c0a5920 2024-08-19T12:49:15.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987c1382c0 con 0x7f987c06e7b0 2024-08-19T12:49:15.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987c13b170 con 0x7f987c0a5d50 2024-08-19T12:49:15.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9881291640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f987c0a5920 0x7f987c137bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39904/0 (socket says 172.21.15.26:39904) 2024-08-19T12:49:15.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9881a92640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f987c06e7b0 0x7f987c1374a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59420/0 (socket says 172.21.15.26:59420) 2024-08-19T12:49:15.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9881291640 1 -- 172.21.15.26:0/2308815790 learned_addr learned my addr 172.21.15.26:0/2308815790 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:15.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.472+0000 7f9883cf6640 10 monclient(hunting): authenticate will time out at 1349.559971s 2024-08-19T12:49:15.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 client.?.objecter ms_handle_connect 0x7f987c0a5920 2024-08-19T12:49:15.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 client.?.objecter ms_handle_connect 0x7f987c06e7b0 2024-08-19T12:49:15.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f9881291640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.26:6789/0 conn(0x7f987c0a5920 legacy=0x7f987c137bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 client.?.objecter ms_handle_connect 0x7f987c0a5d50 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f9881a92640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.26:6790/0 conn(0x7f987c06e7b0 legacy=0x7f987c1374a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f9882293640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.26:6793/0 conn(0x7f987c0a5d50 legacy=0x7f987c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4081315869 0 0) 0x7f987c138e10 con 0x7f987c0a5920 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): my global_id is 145116 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9860004eb0 con 0x7f987c0a5920 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4096331575 0 0) 0x7f987c1382c0 con 0x7f987c06e7b0 2024-08-19T12:49:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): my global_id is 26786 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f987c138e10 con 0x7f987c06e7b0 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3551959269 0 0) 0x7f987c13b170 con 0x7f987c0a5d50 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): my global_id is 174221 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f987c1382c0 con 0x7f987c0a5d50 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2854282411 0 0) 0x7f9860004eb0 con 0x7f987c0a5920 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f987c13b170 con 0x7f987c0a5920 2024-08-19T12:49:15.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f986c004d90 con 0x7f987c0a5920 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): dump: 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1459973965 0 0) 0x7f987c138e10 con 0x7f987c06e7b0 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9860004eb0 con 0x7f987c06e7b0 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9878003610 con 0x7f987c06e7b0 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): dump: 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.496 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2927620581 0 0) 0x7f987c1382c0 con 0x7f987c0a5d50 2024-08-19T12:49:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f987c138e10 con 0x7f987c0a5d50 2024-08-19T12:49:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9874003990 con 0x7f987c0a5d50 2024-08-19T12:49:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:49:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient(hunting): dump: 2024-08-19T12:49:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1273106371 0 0) 0x7f987c13b170 con 0x7f987c0a5920 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.26:6793/0 conn(0x7f987c0a5d50 legacy=0x7f987c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.26:6790/0 conn(0x7f987c06e7b0 legacy=0x7f987c1374a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 monclient: found mon.a 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987c13a8d0 con 0x7f987c0a5920 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f98727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f9883cf6640 5 monclient: authenticate success, global_id 145116 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f9883cf6640 10 monclient: _renew_subs 2024-08-19T12:49:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f9883cf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f9883cf6640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f987c13b590 con 0x7f987c0a5920 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f9883cf6640 10 client.145116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f9883cf6640 10 monclient: _renew_subs 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f9883cf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f9883cf6640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f987c1428c0 con 0x7f987c0a5920 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.475+0000 7f9883cf6640 1 librados: init done 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.476+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f986c0048a0 con 0x7f987c0a5920 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.476+0000 7f98727fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.476+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f986c001b00 con 0x7f987c0a5920 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.476+0000 7f98727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.476+0000 7f9873fff640 4 set_mon_vals no callback set 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.476+0000 7f98727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.476+0000 7f98727fc640 10 monclient: dump: 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.476+0000 7f9883cf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.476+0000 7f9883cf6640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f98640065c0 con 0x7f9864002860 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.476+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f986c011a00 con 0x7f987c0a5920 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.477+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(409..409 src has 311..409) ==== 5333+0+0 (unknown 2618201168 0 0) 0x7f986c04e270 con 0x7f987c0a5920 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.477+0000 7f98727fc640 10 client.145116.objecter ms_dispatch 0x7f987c000b90 osd_map(409..409 src has 311..409) 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.477+0000 7f98727fc640 3 client.145116.objecter handle_osd_map got epochs [409,409] > 0 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.477+0000 7f98727fc640 3 client.145116.objecter handle_osd_map decoding full epoch 409 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.477+0000 7f98727fc640 20 client.145116.objecter dump_active .. 0 homeless 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.477+0000 7f9882293640 10 monclient: get_auth_request con 0x7f9860041a90 auth_method 0 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.478+0000 7f98727fc640 10 client.145116.objecter ms_handle_connect 0x7f9864002860 2024-08-19T12:49:15.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.478+0000 7f98727fc640 10 client.145116.objecter resend_mon_ops 2024-08-19T12:49:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.478+0000 7f9881a92640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.26:6793/0 conn(0x7f9864002860 legacy=0x7f9864004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3077790187 0 0) 0x7f98640065c0 con 0x7f9864002860 2024-08-19T12:49:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 10 monclient(hunting): my global_id is 174230 2024-08-19T12:49:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f987c13b170 con 0x7f9864002860 2024-08-19T12:49:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2430894201 0 0) 0x7f987c13b170 con 0x7f9864002860 2024-08-19T12:49:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98640065c0 con 0x7f9864002860 2024-08-19T12:49:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9878003e50 con 0x7f9864002860 2024-08-19T12:49:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 10 monclient: dump: 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3555459107 0 0) 0x7f98640065c0 con 0x7f9864002860 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f986400a160 con 0x7f9864002860 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9878004110 con 0x7f9864002860 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.26:6793/0 conn(0x7f9864002860 legacy=0x7f9864004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.26:6793/0 conn(0x7f9864002860 legacy=0x7f9864004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.479+0000 7f98727fc640 10 client.145116.objecter ms_handle_connect 0x7f9860041a90 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.485+0000 7f9883cf6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.485+0000 7f9883cf6640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9864009e30 con 0x7f9864009110 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f9881a92640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.26:6793/0 conn(0x7f9864009110 legacy=0x7f9864009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 10 client.145116.objecter ms_handle_connect 0x7f9864009110 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 10 client.145116.objecter resend_mon_ops 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2767857844 0 0) 0x7f9864009e30 con 0x7f9864009110 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 10 monclient(hunting): my global_id is 174239 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98640065c0 con 0x7f9864009110 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2808497433 0 0) 0x7f98640065c0 con 0x7f9864009110 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9864009e30 con 0x7f9864009110 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9878005b00 con 0x7f9864009110 2024-08-19T12:49:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 10 monclient: dump: 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.507 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.507 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.507 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.507 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.507 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1777193642 0 0) 0x7f9864009e30 con 0x7f9864009110 2024-08-19T12:49:15.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f986400c3f0 con 0x7f9864009110 2024-08-19T12:49:15.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.488+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 3564291797) 0x7f9878006f50 con 0x7f9864009110 2024-08-19T12:49:15.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f98727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:15.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f98727fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:15.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.26:6793/0 conn(0x7f9864009110 legacy=0x7f9864009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f98727fc640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.26:6793/0 conn(0x7f9864009110 legacy=0x7f9864009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.508 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 19532, 2024-08-19T12:49:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:15.508 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:49:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.509 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.510 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:15.511 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:15.512 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.513 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.514 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.516 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:15.517 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.518 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:15.521 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:15.522 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:15.523 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.526 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.527 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.528 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:15.528 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 10 librados: watch_flush enter 2024-08-19T12:49:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 10 librados: watch_flush exit 2024-08-19T12:49:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 20 client.145116.objecter shutdown clearing up homeless session... 2024-08-19T12:49:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 10 client.145116.objecter successfully canceled tick 2024-08-19T12:49:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.104:6800/899887506 conn(0x7f9860041a90 legacy=0x7f9860043f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 10 monclient: shutdown 2024-08-19T12:49:15.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:15.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 1 -- 172.21.15.26:0/2308815790 >> v1:172.21.15.26:6789/0 conn(0x7f987c0a5920 legacy=0x7f987c137bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9882293640 1 -- 172.21.15.26:0/2308815790 reap_dead start 2024-08-19T12:49:15.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9882293640 1 -- 172.21.15.26:0/2308815790 reap_dead start 2024-08-19T12:49:15.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 1 -- 172.21.15.26:0/2308815790 shutdown_connections 2024-08-19T12:49:15.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 1 -- 172.21.15.26:0/2308815790 >> 172.21.15.26:0/2308815790 conn(0x7f987c0a8740 msgr2=0x7f987c0aa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:15.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 1 -- 172.21.15.26:0/2308815790 shutdown_connections 2024-08-19T12:49:15.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 1 -- 172.21.15.26:0/2308815790 wait complete. 2024-08-19T12:49:15.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.489+0000 7f9883cf6640 1 librados: shutdown 2024-08-19T12:49:15.531 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:15.532 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:49:15.532 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:49:15.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.630+0000 7fd37d038640 10 monclient: get_monmap_and_config 2024-08-19T12:49:15.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.630+0000 7fd37d038640 10 monclient: build_initial_monmap 2024-08-19T12:49:15.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.630+0000 7fd37d038640 10 monclient: monmap: 2024-08-19T12:49:15.631 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:15.631 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.631 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:15.631516+0000 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:15.631516+0000 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:15.632 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 1 Processor -- start 2024-08-19T12:49:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 1 -- start start 2024-08-19T12:49:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 10 monclient: init 2024-08-19T12:49:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 10 monclient: _add_conns ranks=[0,5,7,1,6,8,4,3,2] 2024-08-19T12:49:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 10 monclient(hunting): picked mon.noname-a con 0x7fd378073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 10 monclient(hunting): picked mon.noname-h con 0x7fd3780a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 10 monclient(hunting): picked mon.noname-i con 0x7fd37813f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37809dae0 con 0x7fd378073470 2024-08-19T12:49:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd378136910 con 0x7fd3780a57a0 2024-08-19T12:49:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37809dd50 con 0x7fd37813f2e0 2024-08-19T12:49:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd37d038640 10 monclient(hunting): authenticate will time out at 1349.718791s 2024-08-19T12:49:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd376d9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd378073470 0x7fd378136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39908/0 (socket says 172.21.15.26:39908) 2024-08-19T12:49:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.631+0000 7fd376d9d640 1 -- 172.21.15.26:0/1325072478 learned_addr learned my addr 172.21.15.26:0/1325072478 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd376d9d640 1 -- 172.21.15.26:0/1325072478 >> v1:172.21.15.26:6789/0 conn(0x7fd378073470 legacy=0x7fd378136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd37659c640 1 -- 172.21.15.26:0/1325072478 >> v1:172.21.15.104:6791/0 conn(0x7fd3780a57a0 legacy=0x7fd3780a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd37759e640 1 -- 172.21.15.26:0/1325072478 >> v1:172.21.15.104:6792/0 conn(0x7fd37813f2e0 legacy=0x7fd3781416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 1 -- 172.21.15.26:0/1325072478 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3273550068 0 0) 0x7fd37809dae0 con 0x7fd378073470 2024-08-19T12:49:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient(hunting): my global_id is 145125 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 1 -- 172.21.15.26:0/1325072478 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd360003520 con 0x7fd378073470 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 1 -- 172.21.15.26:0/1325072478 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2790377852 0 0) 0x7fd360003520 con 0x7fd378073470 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 1 -- 172.21.15.26:0/1325072478 >> v1:172.21.15.104:6792/0 conn(0x7fd37813f2e0 legacy=0x7fd3781416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 1 -- 172.21.15.26:0/1325072478 >> v1:172.21.15.104:6791/0 conn(0x7fd3780a57a0 legacy=0x7fd3780a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 1 monclient: found mon.noname-a 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 1 -- 172.21.15.26:0/1325072478 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd37809e030 con 0x7fd378073470 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 1 -- 172.21.15.26:0/1325072478 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd368001ae0 con 0x7fd378073470 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient: dump: 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.641 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd37d038640 5 monclient: authenticate success, global_id 145125 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd37d038640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 1 -- 172.21.15.26:0/1325072478 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd3680048a0 con 0x7fd378073470 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 1 -- 172.21.15.26:0/1325072478 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd368004e10 con 0x7fd378073470 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd37d038640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd37d038640 4 set_mon_vals no callback set 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.634+0000 7fd375d9b640 10 monclient: dump: 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.635+0000 7fd37d038640 10 monclient: shutdown 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.635+0000 7fd37d038640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.635+0000 7fd37d038640 1 -- 172.21.15.26:0/1325072478 >> v1:172.21.15.26:6789/0 conn(0x7fd378073470 legacy=0x7fd378136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.635+0000 7fd37d038640 1 -- 172.21.15.26:0/1325072478 shutdown_connections 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.635+0000 7fd37d038640 1 -- 172.21.15.26:0/1325072478 >> 172.21.15.26:0/1325072478 conn(0x7fd3780abff0 msgr2=0x7fd3780ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.635+0000 7fd37d038640 1 -- 172.21.15.26:0/1325072478 shutdown_connections 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.635+0000 7fd37d038640 1 -- 172.21.15.26:0/1325072478 wait complete. 2024-08-19T12:49:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.636+0000 7fd37d038640 10 monclient: build_initial_monmap 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.636+0000 7fd37d038640 10 monclient: monmap: 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:15.646 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.636+0000 7fd37d038640 1 librados: starting msgr at 2024-08-19T12:49:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.636+0000 7fd37d038640 1 librados: starting objecter 2024-08-19T12:49:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.636+0000 7fd37d038640 1 Processor -- start 2024-08-19T12:49:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.636+0000 7fd37d038640 1 -- start start 2024-08-19T12:49:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.636+0000 7fd37d038640 1 librados: setting wanted keys 2024-08-19T12:49:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.636+0000 7fd37d038640 1 librados: calling monclient init 2024-08-19T12:49:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.636+0000 7fd37d038640 10 monclient: init 2024-08-19T12:49:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.636+0000 7fd37d038640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.636+0000 7fd37d038640 10 monclient: _add_conns ranks=[1,5,4,0,2,7,3,6,8] 2024-08-19T12:49:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.637+0000 7fd37d038640 10 monclient(hunting): picked mon.noname-f con 0x7fd378073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.637+0000 7fd37d038640 10 monclient(hunting): picked mon.noname-h con 0x7fd3780a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.637+0000 7fd37d038640 10 monclient(hunting): picked mon.noname-c con 0x7fd37813f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.637+0000 7fd37d038640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd378131830 con 0x7fd378073470 2024-08-19T12:49:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.637+0000 7fd37d038640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37812ea60 con 0x7fd37813f2e0 2024-08-19T12:49:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.637+0000 7fd37d038640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37813f0b0 con 0x7fd3780a57a0 2024-08-19T12:49:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.637+0000 7fd37d038640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.637+0000 7fd37d038640 10 monclient(hunting): authenticate will time out at 1349.724105s 2024-08-19T12:49:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.637+0000 7fd37759e640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fd37813f2e0 0x7fd3781d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:57562/0 (socket says 172.21.15.26:57562) 2024-08-19T12:49:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.637+0000 7fd37759e640 1 -- 172.21.15.26:0/4065045822 learned_addr learned my addr 172.21.15.26:0/4065045822 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd37759e640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.26:6791/0 conn(0x7fd37813f2e0 legacy=0x7fd3781d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 client.?.objecter ms_handle_connect 0x7fd37813f2e0 2024-08-19T12:49:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 client.?.objecter ms_handle_connect 0x7fd378073470 2024-08-19T12:49:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd376d9d640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.104:6789/0 conn(0x7fd378073470 legacy=0x7fd37812dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 client.?.objecter ms_handle_connect 0x7fd3780a57a0 2024-08-19T12:49:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd37659c640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.104:6791/0 conn(0x7fd3780a57a0 legacy=0x7fd37812e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1947784551 0 0) 0x7fd37812ea60 con 0x7fd37813f2e0 2024-08-19T12:49:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): my global_id is 27121 2024-08-19T12:49:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd364004eb0 con 0x7fd37813f2e0 2024-08-19T12:49:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3370486727 0 0) 0x7fd378131830 con 0x7fd378073470 2024-08-19T12:49:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): my global_id is 27640 2024-08-19T12:49:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd37812ea60 con 0x7fd378073470 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856830823 0 0) 0x7fd37813f0b0 con 0x7fd3780a57a0 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): my global_id is 26861 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd378131830 con 0x7fd3780a57a0 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 245693218 0 0) 0x7fd364004eb0 con 0x7fd37813f2e0 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd37813f0b0 con 0x7fd37813f2e0 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd370003990 con 0x7fd37813f2e0 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): dump: 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 257744985 0 0) 0x7fd37812ea60 con 0x7fd378073470 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd364004eb0 con 0x7fd378073470 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd368004d90 con 0x7fd378073470 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): dump: 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.658 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.658 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3264197252 0 0) 0x7fd378131830 con 0x7fd3780a57a0 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd37812ea60 con 0x7fd3780a57a0 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd36c003610 con 0x7fd3780a57a0 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 10 monclient(hunting): dump: 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.659 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.639+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4106860881 0 0) 0x7fd37813f0b0 con 0x7fd37813f2e0 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.104:6791/0 conn(0x7fd3780a57a0 legacy=0x7fd37812e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.104:6789/0 conn(0x7fd378073470 legacy=0x7fd37812dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 1 monclient: found mon.c 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.660 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd378142b80 con 0x7fd37813f2e0 2024-08-19T12:49:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd37d038640 5 monclient: authenticate success, global_id 27121 2024-08-19T12:49:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd37d038640 10 monclient: _renew_subs 2024-08-19T12:49:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd37d038640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd37d038640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd378133210 con 0x7fd37813f2e0 2024-08-19T12:49:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd37d038640 10 client.27121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd37d038640 10 monclient: _renew_subs 2024-08-19T12:49:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd37d038640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:15.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd37d038640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fd378131550 con 0x7fd37813f2e0 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd37d038640 1 librados: init done 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd370003e90 con 0x7fd37813f2e0 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd3700043c0 con 0x7fd37813f2e0 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd37559a640 4 set_mon_vals no callback set 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd35f7fe640 10 monclient: dump: 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.662 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.640+0000 7fd37d038640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.641+0000 7fd37d038640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd3580065c0 con 0x7fd358002860 2024-08-19T12:49:15.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.641+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd370004560 con 0x7fd37813f2e0 2024-08-19T12:49:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.641+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(409..409 src has 311..409) ==== 5333+0+0 (unknown 2618201168 0 0) 0x7fd37004e400 con 0x7fd37813f2e0 2024-08-19T12:49:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.641+0000 7fd35f7fe640 10 client.27121.objecter ms_dispatch 0x7fd378000b90 osd_map(409..409 src has 311..409) 2024-08-19T12:49:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.641+0000 7fd35f7fe640 3 client.27121.objecter handle_osd_map got epochs [409,409] > 0 2024-08-19T12:49:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.641+0000 7fd35f7fe640 3 client.27121.objecter handle_osd_map decoding full epoch 409 2024-08-19T12:49:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.641+0000 7fd37659c640 10 monclient: get_auth_request con 0x7fd364041770 auth_method 0 2024-08-19T12:49:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.641+0000 7fd35f7fe640 20 client.27121.objecter dump_active .. 0 homeless 2024-08-19T12:49:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 10 client.27121.objecter ms_handle_connect 0x7fd358002860 2024-08-19T12:49:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 10 client.27121.objecter resend_mon_ops 2024-08-19T12:49:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd376d9d640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.104:6789/0 conn(0x7fd358002860 legacy=0x7fd358004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2416839693 0 0) 0x7fd3580065c0 con 0x7fd358002860 2024-08-19T12:49:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 10 monclient(hunting): my global_id is 27649 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd37813f0b0 con 0x7fd358002860 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 10 client.27121.objecter ms_handle_connect 0x7fd364041770 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2049428000 0 0) 0x7fd37813f0b0 con 0x7fd358002860 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3580065c0 con 0x7fd358002860 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd368001580 con 0x7fd358002860 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 10 monclient: dump: 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.665 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.666 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 120716235 0 0) 0x7fd3580065c0 con 0x7fd358002860 2024-08-19T12:49:15.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.643+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd35800a160 con 0x7fd358002860 2024-08-19T12:49:15.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.644+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd368001d30 con 0x7fd358002860 2024-08-19T12:49:15.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.644+0000 7fd35f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:15.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.644+0000 7fd35f7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:15.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.644+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.104:6789/0 conn(0x7fd358002860 legacy=0x7fd358004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.644+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.104:6789/0 conn(0x7fd358002860 legacy=0x7fd358004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.650+0000 7fd37d038640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:15.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.650+0000 7fd37d038640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd358009e70 con 0x7fd358009130 2024-08-19T12:49:15.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.652+0000 7fd376d9d640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.104:6789/0 conn(0x7fd358009130 legacy=0x7fd3580095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.652+0000 7fd35f7fe640 10 client.27121.objecter ms_handle_connect 0x7fd358009130 2024-08-19T12:49:15.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.652+0000 7fd35f7fe640 10 client.27121.objecter resend_mon_ops 2024-08-19T12:49:15.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 902272181 0 0) 0x7fd358009e70 con 0x7fd358009130 2024-08-19T12:49:15.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 10 monclient(hunting): my global_id is 27658 2024-08-19T12:49:15.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3580065c0 con 0x7fd358009130 2024-08-19T12:49:15.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2631245546 0 0) 0x7fd3580065c0 con 0x7fd358009130 2024-08-19T12:49:15.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd358009e70 con 0x7fd358009130 2024-08-19T12:49:15.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd368005d30 con 0x7fd358009130 2024-08-19T12:49:15.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:49:15.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 10 monclient: dump: 2024-08-19T12:49:15.670 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.670 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.670 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.670 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.670 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.670 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.670 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.670 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1367577915 0 0) 0x7fd358009e70 con 0x7fd358009130 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd35800c3f0 con 0x7fd358009130 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 3718019211) 0x7fd368008070 con 0x7fd358009130 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:15.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:15.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.104:6789/0 conn(0x7fd358009130 legacy=0x7fd3580095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.653+0000 7fd35f7fe640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.104:6789/0 conn(0x7fd358009130 legacy=0x7fd3580095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.672 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 668657, 2024-08-19T12:49:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:15.672 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:49:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:15.673 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.674 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.675 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:15.676 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.677 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:15.683 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:15.684 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:15.685 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.686 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.687 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:15.690 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:15.691 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:49:15.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.693 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.694 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:15.694 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 10 librados: watch_flush enter 2024-08-19T12:49:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 10 librados: watch_flush exit 2024-08-19T12:49:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 20 client.27121.objecter shutdown clearing up homeless session... 2024-08-19T12:49:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 10 client.27121.objecter successfully canceled tick 2024-08-19T12:49:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.104:6800/899887506 conn(0x7fd364041770 legacy=0x7fd364043c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 10 monclient: shutdown 2024-08-19T12:49:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 1 -- 172.21.15.26:0/4065045822 >> v1:172.21.15.26:6791/0 conn(0x7fd37813f2e0 legacy=0x7fd3781d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37759e640 1 -- 172.21.15.26:0/4065045822 reap_dead start 2024-08-19T12:49:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37759e640 1 -- 172.21.15.26:0/4065045822 reap_dead start 2024-08-19T12:49:15.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 1 -- 172.21.15.26:0/4065045822 shutdown_connections 2024-08-19T12:49:15.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 1 -- 172.21.15.26:0/4065045822 >> 172.21.15.26:0/4065045822 conn(0x7fd3780abff0 msgr2=0x7fd3780a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:15.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 1 -- 172.21.15.26:0/4065045822 shutdown_connections 2024-08-19T12:49:15.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 1 -- 172.21.15.26:0/4065045822 wait complete. 2024-08-19T12:49:15.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.654+0000 7fd37d038640 1 librados: shutdown 2024-08-19T12:49:15.697 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:15.697 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:49:15.697 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:49:15.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.796+0000 7f5ff1a2f640 10 monclient: get_monmap_and_config 2024-08-19T12:49:15.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.796+0000 7f5ff1a2f640 10 monclient: build_initial_monmap 2024-08-19T12:49:15.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.796+0000 7f5ff1a2f640 10 monclient: monmap: 2024-08-19T12:49:15.798 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:15.798 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.798 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:15.797723+0000 2024-08-19T12:49:15.798 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:15.797723+0000 2024-08-19T12:49:15.798 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:15.798 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.798 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:15.798 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:15.798 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:15.798 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:15.798 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:15.799 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:15.799 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:15.799 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:15.799 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:15.799 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.797+0000 7f5ff1a2f640 1 Processor -- start 2024-08-19T12:49:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.797+0000 7f5ff1a2f640 1 -- start start 2024-08-19T12:49:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.797+0000 7f5ff1a2f640 10 monclient: init 2024-08-19T12:49:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5ff1a2f640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5ff1a2f640 10 monclient: _add_conns ranks=[0,2,7,4,5,3,6,8,1] 2024-08-19T12:49:15.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5ff1a2f640 10 monclient(hunting): picked mon.noname-a con 0x7f5fec06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:15.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5ff1a2f640 10 monclient(hunting): picked mon.noname-b con 0x7f5fec0a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:15.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5ff1a2f640 10 monclient(hunting): picked mon.noname-i con 0x7f5fec0a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:15.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5ff1a2f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec136790 con 0x7f5fec06e7b0 2024-08-19T12:49:15.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5ff1a2f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec09ddf0 con 0x7f5fec0a5920 2024-08-19T12:49:15.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5ff1a2f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec09e0d0 con 0x7f5fec0a5d50 2024-08-19T12:49:15.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5ff1a2f640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5ff1a2f640 10 monclient(hunting): authenticate will time out at 1349.885176s 2024-08-19T12:49:15.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5feaffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5fec06e7b0 0x7f5fec136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39916/0 (socket says 172.21.15.26:39916) 2024-08-19T12:49:15.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5fea7fc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5fec0a5920 0x7f5fec13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59428/0 (socket says 172.21.15.26:59428) 2024-08-19T12:49:15.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.798+0000 7f5feaffd640 1 -- 172.21.15.26:0/478447238 learned_addr learned my addr 172.21.15.26:0/478447238 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:15.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5feaffd640 1 -- 172.21.15.26:0/478447238 >> v1:172.21.15.26:6789/0 conn(0x7f5fec06e7b0 legacy=0x7f5fec136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fea7fc640 1 -- 172.21.15.26:0/478447238 >> v1:172.21.15.26:6790/0 conn(0x7f5fec0a5920 legacy=0x7f5fec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 1 -- 172.21.15.26:0/478447238 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3469014372 0 0) 0x7f5fec136790 con 0x7f5fec06e7b0 2024-08-19T12:49:15.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient(hunting): my global_id is 145134 2024-08-19T12:49:15.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 1 -- 172.21.15.26:0/478447238 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fd8003520 con 0x7f5fec06e7b0 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 1 -- 172.21.15.26:0/478447238 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1999557888 0 0) 0x7f5fd4001450 con 0x7f5fec0a5920 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient(hunting): my global_id is 26795 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 1 -- 172.21.15.26:0/478447238 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fec136790 con 0x7f5fec0a5920 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5feb7fe640 1 -- 172.21.15.26:0/478447238 >> v1:172.21.15.104:6792/0 conn(0x7f5fec0a5d50 legacy=0x7f5fec141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 1 -- 172.21.15.26:0/478447238 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1949414361 0 0) 0x7f5fec136790 con 0x7f5fec0a5920 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 1 -- 172.21.15.26:0/478447238 >> v1:172.21.15.104:6792/0 conn(0x7f5fec0a5d50 legacy=0x7f5fec141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 1 -- 172.21.15.26:0/478447238 >> v1:172.21.15.26:6789/0 conn(0x7f5fec06e7b0 legacy=0x7f5fec136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 1 monclient: found mon.noname-b 2024-08-19T12:49:15.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 1 -- 172.21.15.26:0/478447238 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fec09e3b0 con 0x7f5fec0a5920 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 1 -- 172.21.15.26:0/478447238 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5fd4001c90 con 0x7f5fec0a5920 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5ff1a2f640 5 monclient: authenticate success, global_id 26795 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5ff1a2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5fe9ffb640 10 monclient: dump: 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.807 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.808 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.800+0000 7f5ff1a2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5fe9ffb640 1 -- 172.21.15.26:0/478447238 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5fd4004860 con 0x7f5fec0a5920 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5fe9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5fe9ffb640 1 -- 172.21.15.26:0/478447238 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5fd4004e10 con 0x7f5fec0a5920 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5ff1a2f640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5ff1a2f640 4 set_mon_vals no callback set 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5fe9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5fe9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5fe9ffb640 10 monclient: dump: 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.809 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5ff1a2f640 10 monclient: shutdown 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5ff1a2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/478447238 >> v1:172.21.15.26:6790/0 conn(0x7f5fec0a5920 legacy=0x7f5fec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/478447238 shutdown_connections 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/478447238 >> 172.21.15.26:0/478447238 conn(0x7f5fec0a8740 msgr2=0x7f5fec0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/478447238 shutdown_connections 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.801+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/478447238 wait complete. 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.802+0000 7f5ff1a2f640 10 monclient: build_initial_monmap 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.802+0000 7f5ff1a2f640 10 monclient: monmap: 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:15.811 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.802+0000 7f5ff1a2f640 1 librados: starting msgr at 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.802+0000 7f5ff1a2f640 1 librados: starting objecter 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.802+0000 7f5ff1a2f640 1 Processor -- start 2024-08-19T12:49:15.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.802+0000 7f5ff1a2f640 1 -- start start 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.802+0000 7f5ff1a2f640 1 librados: setting wanted keys 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.802+0000 7f5ff1a2f640 1 librados: calling monclient init 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.802+0000 7f5ff1a2f640 10 monclient: init 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.803+0000 7f5ff1a2f640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.803+0000 7f5ff1a2f640 10 monclient: _add_conns ranks=[5,1,0,6,7,3,8,4,2] 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.803+0000 7f5ff1a2f640 10 monclient(hunting): picked mon.noname-h con 0x7f5fec06e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.803+0000 7f5ff1a2f640 10 monclient(hunting): picked mon.noname-f con 0x7f5fec0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.803+0000 7f5ff1a2f640 10 monclient(hunting): picked mon.noname-a con 0x7f5fec0a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.803+0000 7f5ff1a2f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec123be0 con 0x7f5fec0a5d50 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.803+0000 7f5ff1a2f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec1346c0 con 0x7f5fec0a5920 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.803+0000 7f5ff1a2f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec125680 con 0x7f5fec06e7b0 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.803+0000 7f5ff1a2f640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.803+0000 7f5ff1a2f640 10 monclient(hunting): authenticate will time out at 1349.890095s 2024-08-19T12:49:15.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.803+0000 7f5feb7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5fec0a5d50 0x7f5fec1d6dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39928/0 (socket says 172.21.15.26:39928) 2024-08-19T12:49:15.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.803+0000 7f5feb7fe640 1 -- 172.21.15.26:0/86198303 learned_addr learned my addr 172.21.15.26:0/86198303 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5feb7fe640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.26:6789/0 conn(0x7f5fec0a5d50 legacy=0x7f5fec1d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 client.?.objecter ms_handle_connect 0x7f5fec0a5d50 2024-08-19T12:49:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 657973757 0 0) 0x7f5fec123be0 con 0x7f5fec0a5d50 2024-08-19T12:49:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): my global_id is 145143 2024-08-19T12:49:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fcc004eb0 con 0x7f5fec0a5d50 2024-08-19T12:49:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 client.?.objecter ms_handle_connect 0x7f5fec0a5920 2024-08-19T12:49:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fea7fc640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6789/0 conn(0x7f5fec0a5920 legacy=0x7f5fec1d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 client.?.objecter ms_handle_connect 0x7f5fec06e7b0 2024-08-19T12:49:15.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5feaffd640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6791/0 conn(0x7f5fec06e7b0 legacy=0x7f5fec141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2587208813 0 0) 0x7f5fcc004eb0 con 0x7f5fec0a5d50 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fec123be0 con 0x7f5fec0a5d50 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5fe4003610 con 0x7f5fec0a5d50 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): dump: 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.821 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2370450203 0 0) 0x7f5fec1346c0 con 0x7f5fec0a5920 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): my global_id is 27667 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fcc004eb0 con 0x7f5fec0a5920 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2813246852 0 0) 0x7f5fec123be0 con 0x7f5fec0a5d50 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6791/0 conn(0x7f5fec06e7b0 legacy=0x7f5fec141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6789/0 conn(0x7f5fec0a5920 legacy=0x7f5fec1d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 monclient: found mon.a 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fec125f20 con 0x7f5fec0a5d50 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5ff1a2f640 5 monclient: authenticate success, global_id 145143 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5ff1a2f640 10 monclient: _renew_subs 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5ff1a2f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5fec124720 con 0x7f5fec0a5d50 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5feb7fe640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.26:6789/0 conn(0x7f5fec0a5d50 legacy=0x7f5fec1d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5ff1a2f640 10 client.145143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5feb7fe640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.26:6789/0 conn(0x7f5fec0a5d50 legacy=0x7f5fec1d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5ff1a2f640 10 monclient: _renew_subs 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5feb7fe640 1 --1- 172.21.15.26:0/86198303 >> v1:172.21.15.26:6789/0 conn(0x7f5fec0a5d50 0x7f5fec1d6dc0 :-1 s=OPENED pgs=3644 cs=1 l=1).handle_message read tag failed 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5feb7fe640 1 --1- 172.21.15.26:0/86198303 >> v1:172.21.15.26:6789/0 conn(0x7f5fec0a5d50 0x7f5fec1d6dc0 :-1 s=OPENED pgs=3644 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5ff1a2f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5fec1232a0 con 0x7f5fec0a5d50 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5ff1a2f640 1 librados: init done 2024-08-19T12:49:15.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient: ms_handle_reset current mon v1:172.21.15.26:6789/0 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.26:6789/0 conn(0x7f5fec0a5d50 legacy=0x7f5fec1d6dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient: _add_conns ranks=[1,5,4,7,2,3,8,0,6] 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.805+0000 7f5fe3fff640 10 monclient(hunting): picked mon.f con 0x7f5fec0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.806+0000 7f5fe3fff640 10 monclient(hunting): picked mon.h con 0x7f5fec06e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.806+0000 7f5fe3fff640 10 monclient(hunting): picked mon.c con 0x7f5fcc00df80 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.806+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5fec123be0 con 0x7f5fec0a5920 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.806+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5fec125680 con 0x7f5fcc00df80 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.806+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5fec1346c0 con 0x7f5fec06e7b0 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.806+0000 7f5fe3fff640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.806+0000 7f5ff1a2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.806+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5fec136740 con 0x7f5fec142640 2024-08-19T12:49:15.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5feb7fe640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.26:6791/0 conn(0x7f5fcc00df80 legacy=0x7f5fcc010440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 client.145143.objecter ms_handle_connect 0x7f5fcc00df80 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 client.145143.objecter resend_mon_ops 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fea7fc640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6791/0 conn(0x7f5fec06e7b0 legacy=0x7f5fcc00c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 client.145143.objecter ms_handle_connect 0x7f5fec06e7b0 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 client.145143.objecter resend_mon_ops 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 client.145143.objecter ms_handle_connect 0x7f5fec0a5920 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 client.145143.objecter resend_mon_ops 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5feaffd640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6789/0 conn(0x7f5fec0a5920 legacy=0x7f5fcc006860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2862420248 0 0) 0x7f5fec125680 con 0x7f5fcc00df80 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.26:6791/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f5fcc014150 con 0x7f5fcc00df80 2024-08-19T12:49:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1900081801 0 0) 0x7f5fec123be0 con 0x7f5fec0a5920 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f5fec125680 con 0x7f5fec0a5920 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1205966758 0 0) 0x7f5fec1346c0 con 0x7f5fec06e7b0 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6791/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f5fec123be0 con 0x7f5fec06e7b0 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 1951478684 0 0) 0x7f5fcc014150 con 0x7f5fcc00df80 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6791/0 conn(0x7f5fec06e7b0 legacy=0x7f5fcc00c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6789/0 conn(0x7f5fec0a5920 legacy=0x7f5fcc006860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 monclient: found mon.c 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f5fcc0153a0 con 0x7f5fcc00df80 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5fe40040e0 con 0x7f5fcc00df80 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient: dump: 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.828 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5fe40043b0 con 0x7f5fcc00df80 2024-08-19T12:49:15.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:15.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe97fa640 4 set_mon_vals no callback set 2024-08-19T12:49:15.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 client.145143.objecter ms_handle_connect 0x7f5fec142640 2024-08-19T12:49:15.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.808+0000 7f5fe3fff640 10 client.145143.objecter resend_mon_ops 2024-08-19T12:49:15.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5feaffd640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6790/0 conn(0x7f5fec142640 legacy=0x7f5fec122290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5fe4011a80 con 0x7f5fcc00df80 2024-08-19T12:49:15.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.4 v1:172.21.15.26:6791/0 6 ==== osd_map(409..409 src has 311..409) ==== 5333+0+0 (unknown 2618201168 0 0) 0x7f5fe404e430 con 0x7f5fcc00df80 2024-08-19T12:49:15.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fe3fff640 10 client.145143.objecter ms_dispatch 0x7f5fec000b90 osd_map(409..409 src has 311..409) 2024-08-19T12:49:15.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fe3fff640 3 client.145143.objecter handle_osd_map got epochs [409,409] > 0 2024-08-19T12:49:15.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fe3fff640 3 client.145143.objecter handle_osd_map decoding full epoch 409 2024-08-19T12:49:15.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fe3fff640 20 client.145143.objecter dump_active .. 0 homeless 2024-08-19T12:49:15.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3527654848 0 0) 0x7f5fec125680 con 0x7f5fec142640 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fe3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fe3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fe3fff640 10 monclient(hunting): my global_id is 154587 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fcc014150 con 0x7f5fec142640 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.809+0000 7f5fea7fc640 10 monclient: get_auth_request con 0x7f5fcc04ccf0 auth_method 0 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1812977544 0 0) 0x7f5fcc014150 con 0x7f5fec142640 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fec125680 con 0x7f5fec142640 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5fdc004b60 con 0x7f5fec142640 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 10 monclient: dump: 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.831 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.832 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1722317139 0 0) 0x7f5fec125680 con 0x7f5fec142640 2024-08-19T12:49:15.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5fec134a80 con 0x7f5fec142640 2024-08-19T12:49:15.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5fdc007480 con 0x7f5fec142640 2024-08-19T12:49:15.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:15.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:15.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6790/0 conn(0x7f5fec142640 legacy=0x7f5fec122290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.810+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6790/0 conn(0x7f5fec142640 legacy=0x7f5fec122290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.812+0000 7f5fe3fff640 10 client.145143.objecter ms_handle_connect 0x7f5fcc04ccf0 2024-08-19T12:49:15.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.816+0000 7f5ff1a2f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:15.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.817+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5fec142a20 con 0x7f5fec02e1d0 2024-08-19T12:49:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.819+0000 7f5feaffd640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6790/0 conn(0x7f5fec02e1d0 legacy=0x7f5fec1229a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.819+0000 7f5fe3fff640 10 client.145143.objecter ms_handle_connect 0x7f5fec02e1d0 2024-08-19T12:49:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.819+0000 7f5fe3fff640 10 client.145143.objecter resend_mon_ops 2024-08-19T12:49:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.819+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4125725478 0 0) 0x7f5fec142a20 con 0x7f5fec02e1d0 2024-08-19T12:49:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.819+0000 7f5fe3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.819+0000 7f5fe3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.819+0000 7f5fe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.819+0000 7f5fe3fff640 10 monclient(hunting): my global_id is 154596 2024-08-19T12:49:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.819+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fec125680 con 0x7f5fec02e1d0 2024-08-19T12:49:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 376936953 0 0) 0x7f5fec125680 con 0x7f5fec02e1d0 2024-08-19T12:49:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fec142a20 con 0x7f5fec02e1d0 2024-08-19T12:49:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5fdc0056c0 con 0x7f5fec02e1d0 2024-08-19T12:49:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:49:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 10 monclient: dump: 2024-08-19T12:49:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.836 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4261497921 0 0) 0x7f5fec142a20 con 0x7f5fec02e1d0 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5fec125af0 con 0x7f5fec02e1d0 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 2791657483) 0x7f5fdc007630 con 0x7f5fec02e1d0 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:15.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6790/0 conn(0x7f5fec02e1d0 legacy=0x7f5fec1229a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5fe3fff640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6790/0 conn(0x7f5fec02e1d0 legacy=0x7f5fec1229a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5feb7fe640 1 -- 172.21.15.26:0/86198303 reap_dead start 2024-08-19T12:49:15.838 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 267550, 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:15.839 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.840 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.841 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:15.842 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:15.843 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.844 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.847 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.848 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.849 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.854 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.855 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.857 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:15.858 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:15.858 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:15.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.820+0000 7f5ff1a2f640 10 librados: watch_flush enter 2024-08-19T12:49:15.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.821+0000 7f5ff1a2f640 10 librados: watch_flush exit 2024-08-19T12:49:15.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.821+0000 7f5ff1a2f640 20 client.145143.objecter shutdown clearing up homeless session... 2024-08-19T12:49:15.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.821+0000 7f5ff1a2f640 10 client.145143.objecter successfully canceled tick 2024-08-19T12:49:15.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.821+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.104:6800/899887506 conn(0x7f5fcc04ccf0 legacy=0x7f5fcc04f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.821+0000 7f5ff1a2f640 10 monclient: shutdown 2024-08-19T12:49:15.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.821+0000 7f5ff1a2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:15.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.821+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/86198303 >> v1:172.21.15.26:6791/0 conn(0x7f5fcc00df80 legacy=0x7f5fcc010440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.821+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/86198303 shutdown_connections 2024-08-19T12:49:15.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.821+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/86198303 >> 172.21.15.26:0/86198303 conn(0x7f5fec0a8740 msgr2=0x7f5fec06dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:15.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.821+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/86198303 shutdown_connections 2024-08-19T12:49:15.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.821+0000 7f5ff1a2f640 1 -- 172.21.15.26:0/86198303 wait complete. 2024-08-19T12:49:15.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.821+0000 7f5ff1a2f640 1 librados: shutdown 2024-08-19T12:49:15.861 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:15.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:49:15.861 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:49:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.961+0000 7f79514fd640 10 monclient: get_monmap_and_config 2024-08-19T12:49:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.961+0000 7f79514fd640 10 monclient: build_initial_monmap 2024-08-19T12:49:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.961+0000 7f79514fd640 10 monclient: monmap: 2024-08-19T12:49:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:15.961941+0000 2024-08-19T12:49:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:15.961941+0000 2024-08-19T12:49:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:15.963 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.961+0000 7f79514fd640 1 Processor -- start 2024-08-19T12:49:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.961+0000 7f79514fd640 1 -- start start 2024-08-19T12:49:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.961+0000 7f79514fd640 10 monclient: init 2024-08-19T12:49:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f79514fd640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f79514fd640 10 monclient: _add_conns ranks=[0,7,2,6,3,5,8,1,4] 2024-08-19T12:49:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f79514fd640 10 monclient(hunting): picked mon.noname-a con 0x7f794c073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f79514fd640 10 monclient(hunting): picked mon.noname-i con 0x7f794c0a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f79514fd640 10 monclient(hunting): picked mon.noname-b con 0x7f794c13f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f79514fd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794c09dae0 con 0x7f794c073470 2024-08-19T12:49:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f79514fd640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794c136910 con 0x7f794c13f2e0 2024-08-19T12:49:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f79514fd640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794c09dd50 con 0x7f794c0a57a0 2024-08-19T12:49:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f79514fd640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f79514fd640 10 monclient(hunting): authenticate will time out at 1350.049255s 2024-08-19T12:49:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f794b7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f794c13f2e0 0x7f794c1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:59432/0 (socket says 172.21.15.26:59432) 2024-08-19T12:49:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f794affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f794c073470 0x7f794c136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39936/0 (socket says 172.21.15.26:39936) 2024-08-19T12:49:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.962+0000 7f794b7fe640 1 -- 172.21.15.26:0/1781646155 learned_addr learned my addr 172.21.15.26:0/1781646155 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f794b7fe640 1 -- 172.21.15.26:0/1781646155 >> v1:172.21.15.26:6790/0 conn(0x7f794c13f2e0 legacy=0x7f794c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f794affd640 1 -- 172.21.15.26:0/1781646155 >> v1:172.21.15.26:6789/0 conn(0x7f794c073470 legacy=0x7f794c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 1 -- 172.21.15.26:0/1781646155 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1863627460 0 0) 0x7f794c09dae0 con 0x7f794c073470 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 10 monclient(hunting): my global_id is 145152 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 1 -- 172.21.15.26:0/1781646155 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7934003520 con 0x7f794c073470 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 1 -- 172.21.15.26:0/1781646155 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1062445561 0 0) 0x7f794c136910 con 0x7f794c13f2e0 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 10 monclient(hunting): my global_id is 26804 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 1 -- 172.21.15.26:0/1781646155 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f794c09dae0 con 0x7f794c13f2e0 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f794a7fc640 1 -- 172.21.15.26:0/1781646155 >> v1:172.21.15.104:6792/0 conn(0x7f794c0a57a0 legacy=0x7f794c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.964+0000 7f7949ffb640 1 -- 172.21.15.26:0/1781646155 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3803652432 0 0) 0x7f7934003520 con 0x7f794c073470 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 1 -- 172.21.15.26:0/1781646155 >> v1:172.21.15.104:6792/0 conn(0x7f794c0a57a0 legacy=0x7f794c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 1 -- 172.21.15.26:0/1781646155 >> v1:172.21.15.26:6790/0 conn(0x7f794c13f2e0 legacy=0x7f794c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 1 monclient: found mon.noname-a 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 1 -- 172.21.15.26:0/1781646155 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794c09e030 con 0x7f794c073470 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 1 -- 172.21.15.26:0/1781646155 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f793c001e60 con 0x7f794c073470 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f79514fd640 5 monclient: authenticate success, global_id 145152 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f79514fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 10 monclient: dump: 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.972 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.973 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.973 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.973 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.973 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.973 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f79514fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:15.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 1 -- 172.21.15.26:0/1781646155 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f793c004b60 con 0x7f794c073470 2024-08-19T12:49:15.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:15.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 1 -- 172.21.15.26:0/1781646155 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f793c0050d0 con 0x7f794c073470 2024-08-19T12:49:15.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f79514fd640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:15.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f79514fd640 4 set_mon_vals no callback set 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f7949ffb640 10 monclient: dump: 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.974 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f79514fd640 10 monclient: shutdown 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f79514fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f79514fd640 1 -- 172.21.15.26:0/1781646155 >> v1:172.21.15.26:6789/0 conn(0x7f794c073470 legacy=0x7f794c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f79514fd640 1 -- 172.21.15.26:0/1781646155 shutdown_connections 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f79514fd640 1 -- 172.21.15.26:0/1781646155 >> 172.21.15.26:0/1781646155 conn(0x7f794c0abff0 msgr2=0x7f794c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f79514fd640 1 -- 172.21.15.26:0/1781646155 shutdown_connections 2024-08-19T12:49:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.965+0000 7f79514fd640 1 -- 172.21.15.26:0/1781646155 wait complete. 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.966+0000 7f79514fd640 10 monclient: build_initial_monmap 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.966+0000 7f79514fd640 10 monclient: monmap: 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.966+0000 7f79514fd640 1 librados: starting msgr at 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.966+0000 7f79514fd640 1 librados: starting objecter 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.966+0000 7f79514fd640 1 Processor -- start 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.966+0000 7f79514fd640 1 -- start start 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.966+0000 7f79514fd640 1 librados: setting wanted keys 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.966+0000 7f79514fd640 1 librados: calling monclient init 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.966+0000 7f79514fd640 10 monclient: init 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.967+0000 7f79514fd640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.967+0000 7f79514fd640 10 monclient: _add_conns ranks=[1,0,4,7,2,6,3,8,5] 2024-08-19T12:49:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.967+0000 7f79514fd640 10 monclient(hunting): picked mon.noname-f con 0x7f794c073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.967+0000 7f79514fd640 10 monclient(hunting): picked mon.noname-a con 0x7f794c0a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.967+0000 7f79514fd640 10 monclient(hunting): picked mon.noname-c con 0x7f794c13f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.967+0000 7f79514fd640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794c0a5360 con 0x7f794c0a57a0 2024-08-19T12:49:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.967+0000 7f79514fd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794c13f0b0 con 0x7f794c073470 2024-08-19T12:49:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.967+0000 7f79514fd640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f794c06d8f0 con 0x7f794c13f2e0 2024-08-19T12:49:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.967+0000 7f79514fd640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.967+0000 7f79514fd640 10 monclient(hunting): authenticate will time out at 1350.054410s 2024-08-19T12:49:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.967+0000 7f794a7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f794c0a57a0 0x7f794c1d3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39942/0 (socket says 172.21.15.26:39942) 2024-08-19T12:49:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.967+0000 7f794a7fc640 1 -- 172.21.15.26:0/1496967590 learned_addr learned my addr 172.21.15.26:0/1496967590 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f794a7fc640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.26:6789/0 conn(0x7f794c0a57a0 legacy=0x7f794c1d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f79337fe640 10 client.?.objecter ms_handle_connect 0x7f794c0a57a0 2024-08-19T12:49:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f79337fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f794b7fe640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.26:6791/0 conn(0x7f794c13f2e0 legacy=0x7f794c1d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f79337fe640 10 client.?.objecter ms_handle_connect 0x7f794c13f2e0 2024-08-19T12:49:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f79337fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f794a7fc640 1 --1- 172.21.15.26:0/1496967590 >> v1:172.21.15.26:6789/0 conn(0x7f794c0a57a0 0x7f794c1d3340 :-1 s=OPENED pgs=3646 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f794c0a5360 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 438305109 0 0) 0x7f794c06d8f0 con 0x7f794c13f2e0 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f79337fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f79337fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f79337fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f794affd640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.104:6789/0 conn(0x7f794c073470 legacy=0x7f794c1ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f79337fe640 10 monclient(hunting): my global_id is 27130 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7938004eb0 con 0x7f794c13f2e0 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f79337fe640 10 client.?.objecter ms_handle_connect 0x7f794c073470 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.969+0000 7f79337fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3734806875 0 0) 0x7f7938004eb0 con 0x7f794c13f2e0 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f794c06d8f0 con 0x7f794c13f2e0 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7944001c90 con 0x7f794c13f2e0 2024-08-19T12:49:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient(hunting): dump: 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1667649307 0 0) 0x7f794c13f0b0 con 0x7f794c073470 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient(hunting): my global_id is 27676 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7938004eb0 con 0x7f794c073470 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 537996420 0 0) 0x7f794c06d8f0 con 0x7f794c13f2e0 2024-08-19T12:49:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.104:6789/0 conn(0x7f794c073470 legacy=0x7f794c1ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.26:6789/0 conn(0x7f794c0a57a0 legacy=0x7f794c1d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient: _finish_hunting 0 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 monclient: found mon.c 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794c0a3b60 con 0x7f794c13f2e0 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient: _finish_auth 0 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient: _check_auth_tickets 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79514fd640 5 monclient: authenticate success, global_id 27130 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4232100983 0 0) 0x7f794c0a5360 con 0x7f794c0a57a0 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79514fd640 10 monclient: _renew_subs 2024-08-19T12:49:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79514fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79514fd640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f794c137500 con 0x7f794c13f2e0 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79514fd640 10 client.27130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79514fd640 10 monclient: _renew_subs 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79514fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79514fd640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f794c137e70 con 0x7f794c13f2e0 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79514fd640 1 librados: init done 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7944004a90 con 0x7f794c13f2e0 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7944004f80 con 0x7f794c13f2e0 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79497fa640 4 set_mon_vals no callback set 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:49:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.970+0000 7f79337fe640 10 monclient: dump: 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.989 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.990 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.971+0000 7f79514fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.971+0000 7f79514fd640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f792c0065c0 con 0x7f792c002860 2024-08-19T12:49:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.971+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f79440051e0 con 0x7f794c13f2e0 2024-08-19T12:49:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.971+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(409..409 src has 311..409) ==== 5333+0+0 (unknown 2618201168 0 0) 0x7f794404df60 con 0x7f794c13f2e0 2024-08-19T12:49:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.971+0000 7f79337fe640 10 client.27130.objecter ms_dispatch 0x7f794c000b90 osd_map(409..409 src has 311..409) 2024-08-19T12:49:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.971+0000 7f79337fe640 3 client.27130.objecter handle_osd_map got epochs [409,409] > 0 2024-08-19T12:49:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.971+0000 7f79337fe640 3 client.27130.objecter handle_osd_map decoding full epoch 409 2024-08-19T12:49:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.971+0000 7f79337fe640 20 client.27130.objecter dump_active .. 0 homeless 2024-08-19T12:49:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.971+0000 7f794a7fc640 10 monclient: get_auth_request con 0x7f7938040f60 auth_method 0 2024-08-19T12:49:15.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f794affd640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.104:6791/0 conn(0x7f792c002860 legacy=0x7f792c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 10 client.27130.objecter ms_handle_connect 0x7f792c002860 2024-08-19T12:49:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 10 client.27130.objecter resend_mon_ops 2024-08-19T12:49:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3101537586 0 0) 0x7f792c0065c0 con 0x7f792c002860 2024-08-19T12:49:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 10 monclient(hunting): my global_id is 26879 2024-08-19T12:49:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f794c0a5360 con 0x7f792c002860 2024-08-19T12:49:15.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1469384196 0 0) 0x7f794c0a5360 con 0x7f792c002860 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f792c0065c0 con 0x7f792c002860 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 10 client.27130.objecter ms_handle_connect 0x7f7938040f60 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f793c004ea0 con 0x7f792c002860 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.973+0000 7f79337fe640 10 monclient: dump: 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.992 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.974+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3195624629 0 0) 0x7f792c0065c0 con 0x7f792c002860 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.974+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f792c00a160 con 0x7f792c002860 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.974+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f793c0050c0 con 0x7f792c002860 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.974+0000 7f79337fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:15.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.974+0000 7f79337fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:15.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.974+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.104:6791/0 conn(0x7f792c002860 legacy=0x7f792c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.974+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.104:6791/0 conn(0x7f792c002860 legacy=0x7f792c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.980+0000 7f79514fd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:15.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.980+0000 7f79514fd640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f792c009e70 con 0x7f792c009130 2024-08-19T12:49:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f794affd640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.104:6791/0 conn(0x7f792c009130 legacy=0x7f792c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 10 client.27130.objecter ms_handle_connect 0x7f792c009130 2024-08-19T12:49:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 10 client.27130.objecter resend_mon_ops 2024-08-19T12:49:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 555665523 0 0) 0x7f792c009e70 con 0x7f792c009130 2024-08-19T12:49:15.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 10 monclient(hunting): my global_id is 26888 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f792c0065c0 con 0x7f792c009130 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3629885891 0 0) 0x7f792c0065c0 con 0x7f792c009130 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f792c009e70 con 0x7f792c009130 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f793c004ea0 con 0x7f792c009130 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 10 monclient: dump: 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.996 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:15.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2476338801 0 0) 0x7f792c009e70 con 0x7f792c009130 2024-08-19T12:49:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.983+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f792c00c3f0 con 0x7f792c009130 2024-08-19T12:49:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7878 (unknown 0 0 1496650344) 0x7f793c005750 con 0x7f792c009130 2024-08-19T12:49:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79337fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79337fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.104:6791/0 conn(0x7f792c009130 legacy=0x7f792c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:15.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79337fe640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.104:6791/0 conn(0x7f792c009130 legacy=0x7f792c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:15.998 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:15.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 669006, 2024-08-19T12:49:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:15.999 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:49:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:16.000 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:16.001 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:16.002 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:16.003 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:16.004 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:16.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:16.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:16.010 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:16.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:16.010 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.010 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.010 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:16.011 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:16.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:16.012 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.013 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:16.017 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.018 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.019 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:16.020 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:49:16.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.020 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:16.020 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:16.020 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:16.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.020 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:16.020 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:16.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 10 librados: watch_flush enter 2024-08-19T12:49:16.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 10 librados: watch_flush exit 2024-08-19T12:49:16.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 20 client.27130.objecter shutdown clearing up homeless session... 2024-08-19T12:49:16.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 10 client.27130.objecter successfully canceled tick 2024-08-19T12:49:16.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.104:6800/899887506 conn(0x7f7938040f60 legacy=0x7f7938043420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:16.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 10 monclient: shutdown 2024-08-19T12:49:16.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:16.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 1 -- 172.21.15.26:0/1496967590 >> v1:172.21.15.26:6791/0 conn(0x7f794c13f2e0 legacy=0x7f794c1d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:16.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f794b7fe640 1 -- 172.21.15.26:0/1496967590 reap_dead start 2024-08-19T12:49:16.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f794b7fe640 1 -- 172.21.15.26:0/1496967590 reap_dead start 2024-08-19T12:49:16.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 1 -- 172.21.15.26:0/1496967590 shutdown_connections 2024-08-19T12:49:16.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 1 -- 172.21.15.26:0/1496967590 >> 172.21.15.26:0/1496967590 conn(0x7f794c0abff0 msgr2=0x7f794c134900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:16.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 1 -- 172.21.15.26:0/1496967590 shutdown_connections 2024-08-19T12:49:16.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 1 -- 172.21.15.26:0/1496967590 wait complete. 2024-08-19T12:49:16.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:15.984+0000 7f79514fd640 1 librados: shutdown 2024-08-19T12:49:16.023 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:16.023 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:49:16.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:49:16.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.024+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.024+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839432608}) 2024-08-19T12:49:16.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.024+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=476766790}) 2024-08-19T12:49:16.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839432608}) 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2910565732}) 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839432608}) 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3399188689}) 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839432608}) 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=207720446}) 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839432608}) 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2213307478}) 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839432608}) 2024-08-19T12:49:16.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4044161968}) 2024-08-19T12:49:16.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839432608}) 2024-08-19T12:49:16.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1762578175}) 2024-08-19T12:49:16.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839432608}) 2024-08-19T12:49:16.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.025+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1817629341}) 2024-08-19T12:49:16.041 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.026+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.041 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.026+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623813390}) 2024-08-19T12:49:16.041 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.026+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=919887949}) 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.026+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.026+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623813390}) 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.026+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1828078902}) 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.026+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.026+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623813390}) 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1280606839}) 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623813390}) 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3677143469}) 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623813390}) 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3847396883}) 2024-08-19T12:49:16.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623813390}) 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2372000132}) 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623813390}) 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=676409852}) 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623813390}) 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.027+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1671993045}) 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144656473}) 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=487127909}) 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144656473}) 2024-08-19T12:49:16.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3650666517}) 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144656473}) 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=582546161}) 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144656473}) 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=409022604}) 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144656473}) 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2411090181}) 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144656473}) 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1197034750}) 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.044 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144656473}) 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=180690758}) 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144656473}) 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.028+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=126169672}) 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656853533}) 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=285283099}) 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656853533}) 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1207378210}) 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656853533}) 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2195475140}) 2024-08-19T12:49:16.045 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.046 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656853533}) 2024-08-19T12:49:16.046 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2056820333}) 2024-08-19T12:49:16.046 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.046 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656853533}) 2024-08-19T12:49:16.046 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3540306086}) 2024-08-19T12:49:16.046 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.046 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656853533}) 2024-08-19T12:49:16.046 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3899075195}) 2024-08-19T12:49:16.046 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.046 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656853533}) 2024-08-19T12:49:16.046 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2293944019}) 2024-08-19T12:49:16.046 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656853533}) 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.030+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3781245016}) 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268060711}) 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1962077786}) 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268060711}) 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=875612838}) 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268060711}) 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=522024297}) 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268060711}) 2024-08-19T12:49:16.047 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=860281707}) 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268060711}) 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=778511243}) 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268060711}) 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3914253135}) 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268060711}) 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3275554038}) 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268060711}) 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.031+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=917193703}) 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.048 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932836346}) 2024-08-19T12:49:16.049 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1455517376}) 2024-08-19T12:49:16.049 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.049 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932836346}) 2024-08-19T12:49:16.049 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4134721060}) 2024-08-19T12:49:16.049 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.049 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932836346}) 2024-08-19T12:49:16.049 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=193487763}) 2024-08-19T12:49:16.049 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.049 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932836346}) 2024-08-19T12:49:16.049 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2214271930}) 2024-08-19T12:49:16.049 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932836346}) 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2035622794}) 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932836346}) 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4274023671}) 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932836346}) 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4212339314}) 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932836346}) 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.032+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3243749635}) 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.050 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360234248}) 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1757282548}) 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360234248}) 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2659826133}) 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360234248}) 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=119059652}) 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360234248}) 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3345644718}) 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360234248}) 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1469783057}) 2024-08-19T12:49:16.051 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.052 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360234248}) 2024-08-19T12:49:16.052 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3620876873}) 2024-08-19T12:49:16.052 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.052 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360234248}) 2024-08-19T12:49:16.052 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1459934339}) 2024-08-19T12:49:16.052 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.052 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360234248}) 2024-08-19T12:49:16.052 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.034+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4210588284}) 2024-08-19T12:49:16.052 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.053 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998394642}) 2024-08-19T12:49:16.053 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1881388942}) 2024-08-19T12:49:16.053 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.053 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998394642}) 2024-08-19T12:49:16.053 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375025826}) 2024-08-19T12:49:16.053 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.053 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998394642}) 2024-08-19T12:49:16.053 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2043540965}) 2024-08-19T12:49:16.053 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.053 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998394642}) 2024-08-19T12:49:16.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4102549929}) 2024-08-19T12:49:16.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998394642}) 2024-08-19T12:49:16.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3299131470}) 2024-08-19T12:49:16.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998394642}) 2024-08-19T12:49:16.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2105231454}) 2024-08-19T12:49:16.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998394642}) 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=210004336}) 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998394642}) 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.035+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2677714688}) 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296850857}) 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3258489292}) 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296850857}) 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1377300332}) 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296850857}) 2024-08-19T12:49:16.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1454708618}) 2024-08-19T12:49:16.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296850857}) 2024-08-19T12:49:16.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3916596435}) 2024-08-19T12:49:16.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296850857}) 2024-08-19T12:49:16.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2757450255}) 2024-08-19T12:49:16.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296850857}) 2024-08-19T12:49:16.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4075333519}) 2024-08-19T12:49:16.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296850857}) 2024-08-19T12:49:16.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1432913770}) 2024-08-19T12:49:16.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296850857}) 2024-08-19T12:49:16.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.037+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=868591798}) 2024-08-19T12:49:16.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429758698}) 2024-08-19T12:49:16.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1175696303}) 2024-08-19T12:49:16.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429758698}) 2024-08-19T12:49:16.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3321633925}) 2024-08-19T12:49:16.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429758698}) 2024-08-19T12:49:16.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2991376972}) 2024-08-19T12:49:16.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429758698}) 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=813400202}) 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429758698}) 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=267716152}) 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429758698}) 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2003885999}) 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429758698}) 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=989102732}) 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429758698}) 2024-08-19T12:49:16.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.054+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2814665191}) 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101673698}) 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4068141908}) 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101673698}) 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=699690118}) 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101673698}) 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2498389820}) 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101673698}) 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3342521121}) 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101673698}) 2024-08-19T12:49:16.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=414119962}) 2024-08-19T12:49:16.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101673698}) 2024-08-19T12:49:16.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=900188380}) 2024-08-19T12:49:16.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101673698}) 2024-08-19T12:49:16.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1491111925}) 2024-08-19T12:49:16.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101673698}) 2024-08-19T12:49:16.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.056+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1959163585}) 2024-08-19T12:49:16.082 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.065+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.082 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.065+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2364748553}) 2024-08-19T12:49:16.082 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.065+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1673044725}) 2024-08-19T12:49:16.082 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.065+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.082 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.065+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2364748553}) 2024-08-19T12:49:16.082 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.065+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2703935780}) 2024-08-19T12:49:16.082 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.065+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.065+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2364748553}) 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.065+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1905423216}) 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.065+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.065+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2364748553}) 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.065+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3959115849}) 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.066+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.066+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2364748553}) 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.066+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=136049523}) 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.066+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.066+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2364748553}) 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.066+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1185360791}) 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.066+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.066+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2364748553}) 2024-08-19T12:49:16.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.066+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1053035445}) 2024-08-19T12:49:16.084 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.066+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.084 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.066+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2364748553}) 2024-08-19T12:49:16.084 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.066+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2067086651}) 2024-08-19T12:49:16.084 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.084 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126569301}) 2024-08-19T12:49:16.084 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603124035}) 2024-08-19T12:49:16.084 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.084 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126569301}) 2024-08-19T12:49:16.084 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1882671489}) 2024-08-19T12:49:16.084 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.084 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126569301}) 2024-08-19T12:49:16.085 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1923401303}) 2024-08-19T12:49:16.085 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.085 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126569301}) 2024-08-19T12:49:16.085 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1818961680}) 2024-08-19T12:49:16.085 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.086 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126569301}) 2024-08-19T12:49:16.086 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=29824731}) 2024-08-19T12:49:16.086 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.086 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126569301}) 2024-08-19T12:49:16.086 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=928177215}) 2024-08-19T12:49:16.086 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.087 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126569301}) 2024-08-19T12:49:16.087 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2129838700}) 2024-08-19T12:49:16.087 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.087 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126569301}) 2024-08-19T12:49:16.087 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.067+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3767180225}) 2024-08-19T12:49:16.088 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.068+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.088 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.068+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221744184}) 2024-08-19T12:49:16.088 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.068+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=774346474}) 2024-08-19T12:49:16.088 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.068+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.088 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.068+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221744184}) 2024-08-19T12:49:16.088 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.068+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722979425}) 2024-08-19T12:49:16.088 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.068+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.088 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.068+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221744184}) 2024-08-19T12:49:16.088 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.068+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3944887389}) 2024-08-19T12:49:16.088 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.088 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221744184}) 2024-08-19T12:49:16.088 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=678844210}) 2024-08-19T12:49:16.089 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.089 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221744184}) 2024-08-19T12:49:16.089 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2830177490}) 2024-08-19T12:49:16.089 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.089 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221744184}) 2024-08-19T12:49:16.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2474848652}) 2024-08-19T12:49:16.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221744184}) 2024-08-19T12:49:16.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3138828258}) 2024-08-19T12:49:16.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221744184}) 2024-08-19T12:49:16.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.069+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3644707883}) 2024-08-19T12:49:16.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517178606}) 2024-08-19T12:49:16.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2409969596}) 2024-08-19T12:49:16.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517178606}) 2024-08-19T12:49:16.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=454608260}) 2024-08-19T12:49:16.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517178606}) 2024-08-19T12:49:16.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1482381412}) 2024-08-19T12:49:16.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517178606}) 2024-08-19T12:49:16.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1985883279}) 2024-08-19T12:49:16.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517178606}) 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3774110029}) 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517178606}) 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=98065472}) 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517178606}) 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=409496503}) 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517178606}) 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.070+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2991938671}) 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.071+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.071+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212212808}) 2024-08-19T12:49:16.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.071+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826577913}) 2024-08-19T12:49:16.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.071+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.071+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212212808}) 2024-08-19T12:49:16.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.071+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2209256264}) 2024-08-19T12:49:16.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.071+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.071+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212212808}) 2024-08-19T12:49:16.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.071+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3202805554}) 2024-08-19T12:49:16.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212212808}) 2024-08-19T12:49:16.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2765867306}) 2024-08-19T12:49:16.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212212808}) 2024-08-19T12:49:16.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=358143827}) 2024-08-19T12:49:16.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212212808}) 2024-08-19T12:49:16.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3753547818}) 2024-08-19T12:49:16.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212212808}) 2024-08-19T12:49:16.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2795568361}) 2024-08-19T12:49:16.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212212808}) 2024-08-19T12:49:16.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.072+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1644312122}) 2024-08-19T12:49:16.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597966372}) 2024-08-19T12:49:16.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3810322869}) 2024-08-19T12:49:16.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597966372}) 2024-08-19T12:49:16.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1591595858}) 2024-08-19T12:49:16.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597966372}) 2024-08-19T12:49:16.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2806691318}) 2024-08-19T12:49:16.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597966372}) 2024-08-19T12:49:16.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2037350176}) 2024-08-19T12:49:16.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597966372}) 2024-08-19T12:49:16.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3710559271}) 2024-08-19T12:49:16.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597966372}) 2024-08-19T12:49:16.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3607491866}) 2024-08-19T12:49:16.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597966372}) 2024-08-19T12:49:16.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3001218611}) 2024-08-19T12:49:16.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597966372}) 2024-08-19T12:49:16.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.077+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3241546715}) 2024-08-19T12:49:16.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678664464}) 2024-08-19T12:49:16.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1564369912}) 2024-08-19T12:49:16.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678664464}) 2024-08-19T12:49:16.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2957182401}) 2024-08-19T12:49:16.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678664464}) 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4180220992}) 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678664464}) 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3880935599}) 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678664464}) 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3676236959}) 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678664464}) 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2623540485}) 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678664464}) 2024-08-19T12:49:16.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3633432641}) 2024-08-19T12:49:16.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678664464}) 2024-08-19T12:49:16.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.082+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=400490803}) 2024-08-19T12:49:16.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995223247}) 2024-08-19T12:49:16.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1025930255}) 2024-08-19T12:49:16.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995223247}) 2024-08-19T12:49:16.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2896468650}) 2024-08-19T12:49:16.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.110 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995223247}) 2024-08-19T12:49:16.110 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=898952459}) 2024-08-19T12:49:16.110 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.110 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995223247}) 2024-08-19T12:49:16.110 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1644381234}) 2024-08-19T12:49:16.110 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.110 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995223247}) 2024-08-19T12:49:16.110 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1982242426}) 2024-08-19T12:49:16.110 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995223247}) 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3087823851}) 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995223247}) 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=705705186}) 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995223247}) 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4126877719}) 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313079020}) 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1270272308}) 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.111 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313079020}) 2024-08-19T12:49:16.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3877294592}) 2024-08-19T12:49:16.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313079020}) 2024-08-19T12:49:16.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=42699270}) 2024-08-19T12:49:16.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313079020}) 2024-08-19T12:49:16.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2722283165}) 2024-08-19T12:49:16.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313079020}) 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4220001451}) 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313079020}) 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3410419078}) 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313079020}) 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3275809122}) 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313079020}) 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3814185957}) 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226236871}) 2024-08-19T12:49:16.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2857238066}) 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226236871}) 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1754344831}) 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226236871}) 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4154877193}) 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226236871}) 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=598133967}) 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226236871}) 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=545690017}) 2024-08-19T12:49:16.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226236871}) 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=520244777}) 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226236871}) 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=226121532}) 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226236871}) 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1525247321}) 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388561127}) 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=753120652}) 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388561127}) 2024-08-19T12:49:16.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=989953300}) 2024-08-19T12:49:16.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388561127}) 2024-08-19T12:49:16.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=183856755}) 2024-08-19T12:49:16.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388561127}) 2024-08-19T12:49:16.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2447946781}) 2024-08-19T12:49:16.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388561127}) 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=84250172}) 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388561127}) 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3539694374}) 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388561127}) 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2315512235}) 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388561127}) 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2437944489}) 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296957814}) 2024-08-19T12:49:16.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2336715309}) 2024-08-19T12:49:16.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296957814}) 2024-08-19T12:49:16.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=605643541}) 2024-08-19T12:49:16.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296957814}) 2024-08-19T12:49:16.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3473845024}) 2024-08-19T12:49:16.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296957814}) 2024-08-19T12:49:16.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2631220648}) 2024-08-19T12:49:16.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296957814}) 2024-08-19T12:49:16.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3240897606}) 2024-08-19T12:49:16.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296957814}) 2024-08-19T12:49:16.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=227168626}) 2024-08-19T12:49:16.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296957814}) 2024-08-19T12:49:16.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=359965307}) 2024-08-19T12:49:16.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296957814}) 2024-08-19T12:49:16.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3937188847}) 2024-08-19T12:49:16.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.136+0000 7fa7ee44f640 10 monclient: get_monmap_and_config 2024-08-19T12:49:16.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.136+0000 7fa7ee44f640 10 monclient: build_initial_monmap 2024-08-19T12:49:16.137 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.136+0000 7fa7ee44f640 10 monclient: monmap: 2024-08-19T12:49:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:16.137648+0000 2024-08-19T12:49:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:16.137648+0000 2024-08-19T12:49:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:16.138 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:16.139 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:16.139 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:16.139 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:16.139 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:16.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.137+0000 7fa7ee44f640 1 Processor -- start 2024-08-19T12:49:16.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.137+0000 7fa7ee44f640 1 -- start start 2024-08-19T12:49:16.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.137+0000 7fa7ee44f640 10 monclient: init 2024-08-19T12:49:16.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.138+0000 7fa7ee44f640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:16.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.138+0000 7fa7ee44f640 10 monclient: _add_conns ranks=[1,6,5,8,3,4,7,2,0] 2024-08-19T12:49:16.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.138+0000 7fa7ee44f640 10 monclient(hunting): picked mon.noname-f con 0x7fa7e806e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:16.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.138+0000 7fa7ee44f640 10 monclient(hunting): picked mon.noname-d con 0x7fa7e80a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:16.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.138+0000 7fa7ee44f640 10 monclient(hunting): picked mon.noname-h con 0x7fa7e80a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:16.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.138+0000 7fa7ee44f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e8136790 con 0x7fa7e806e7b0 2024-08-19T12:49:16.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.138+0000 7fa7ee44f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e809ddf0 con 0x7fa7e80a5d50 2024-08-19T12:49:16.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.138+0000 7fa7ee44f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e809e0d0 con 0x7fa7e80a5920 2024-08-19T12:49:16.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.138+0000 7fa7ee44f640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:16.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.138+0000 7fa7ee44f640 10 monclient(hunting): authenticate will time out at 1350.225123s 2024-08-19T12:49:16.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.138+0000 7fa7e77fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fa7e80a5920 0x7fa7e813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51046/0 (socket says 172.21.15.26:51046) 2024-08-19T12:49:16.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.138+0000 7fa7e77fe640 1 -- 172.21.15.26:0/1737920673 learned_addr learned my addr 172.21.15.26:0/1737920673 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:16.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e77fe640 1 -- 172.21.15.26:0/1737920673 >> v1:172.21.15.26:6792/0 conn(0x7fa7e80a5920 legacy=0x7fa7e813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:16.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7ec9ec640 1 -- 172.21.15.26:0/1737920673 >> v1:172.21.15.104:6791/0 conn(0x7fa7e80a5d50 legacy=0x7fa7e8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:16.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e7fff640 1 -- 172.21.15.26:0/1737920673 >> v1:172.21.15.104:6789/0 conn(0x7fa7e806e7b0 legacy=0x7fa7e8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:16.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3403747218 0 0) 0x7fa7e809e0d0 con 0x7fa7e80a5920 2024-08-19T12:49:16.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:16.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient(hunting): my global_id is 164454 2024-08-19T12:49:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7d0003520 con 0x7fa7e80a5920 2024-08-19T12:49:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660231412 0 0) 0x7fa7e0001820 con 0x7fa7e80a5d50 2024-08-19T12:49:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient(hunting): my global_id is 26897 2024-08-19T12:49:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e809e0d0 con 0x7fa7e80a5d50 2024-08-19T12:49:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1674061358 0 0) 0x7fa7e8136790 con 0x7fa7e806e7b0 2024-08-19T12:49:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient(hunting): my global_id is 27685 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e0001820 con 0x7fa7e806e7b0 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3193665174 0 0) 0x7fa7d0003520 con 0x7fa7e80a5920 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 >> v1:172.21.15.104:6791/0 conn(0x7fa7e80a5d50 legacy=0x7fa7e8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 >> v1:172.21.15.104:6789/0 conn(0x7fa7e806e7b0 legacy=0x7fa7e8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 1 monclient: found mon.noname-d 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e809e3b0 con 0x7fa7e80a5920 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient: _finish_auth 0 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:49:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7dc001ae0 con 0x7fa7e80a5920 2024-08-19T12:49:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7ee44f640 5 monclient: authenticate success, global_id 164454 2024-08-19T12:49:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7ee44f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.140+0000 7fa7e6ffd640 10 monclient: dump: 2024-08-19T12:49:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:16.155 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:16.156 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:16.157 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7ee44f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7dc0048a0 con 0x7fa7e80a5920 2024-08-19T12:49:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7e6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7ee44f640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:16.157 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7e6ffd640 1 -- 172.21.15.26:0/1737920673 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7dc004e10 con 0x7fa7e80a5920 2024-08-19T12:49:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7ee44f640 4 set_mon_vals no callback set 2024-08-19T12:49:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7e6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7e6ffd640 10 monclient: dump: 2024-08-19T12:49:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7ee44f640 10 monclient: shutdown 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7ee44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7ee44f640 1 -- 172.21.15.26:0/1737920673 >> v1:172.21.15.26:6792/0 conn(0x7fa7e80a5920 legacy=0x7fa7e813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7ee44f640 1 -- 172.21.15.26:0/1737920673 shutdown_connections 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7ee44f640 1 -- 172.21.15.26:0/1737920673 >> 172.21.15.26:0/1737920673 conn(0x7fa7e80a8740 msgr2=0x7fa7e80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7ee44f640 1 -- 172.21.15.26:0/1737920673 shutdown_connections 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.141+0000 7fa7ee44f640 1 -- 172.21.15.26:0/1737920673 wait complete. 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.142+0000 7fa7ee44f640 10 monclient: build_initial_monmap 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.142+0000 7fa7ee44f640 10 monclient: monmap: 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:16.160 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.142+0000 7fa7ee44f640 1 librados: starting msgr at 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.142+0000 7fa7ee44f640 1 librados: starting objecter 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.142+0000 7fa7ee44f640 1 Processor -- start 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.142+0000 7fa7ee44f640 1 -- start start 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.142+0000 7fa7ee44f640 1 librados: setting wanted keys 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.142+0000 7fa7ee44f640 1 librados: calling monclient init 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.142+0000 7fa7ee44f640 10 monclient: init 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.143+0000 7fa7ee44f640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:16.161 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.143+0000 7fa7ee44f640 10 monclient: _add_conns ranks=[5,1,6,0,8,2,3,7,4] 2024-08-19T12:49:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.143+0000 7fa7ee44f640 10 monclient(hunting): picked mon.noname-h con 0x7fa7e806e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.143+0000 7fa7ee44f640 10 monclient(hunting): picked mon.noname-f con 0x7fa7e80a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.143+0000 7fa7ee44f640 10 monclient(hunting): picked mon.noname-d con 0x7fa7e80a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.143+0000 7fa7ee44f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e813aa00 con 0x7fa7e80a5920 2024-08-19T12:49:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.143+0000 7fa7ee44f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e81346c0 con 0x7fa7e806e7b0 2024-08-19T12:49:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.143+0000 7fa7ee44f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e8139200 con 0x7fa7e80a5d50 2024-08-19T12:49:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.143+0000 7fa7ee44f640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.143+0000 7fa7ee44f640 10 monclient(hunting): authenticate will time out at 1350.230151s 2024-08-19T12:49:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.143+0000 7fa7ec9ec640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fa7e80a5d50 0x7fa7e81d6c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51062/0 (socket says 172.21.15.26:51062) 2024-08-19T12:49:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.143+0000 7fa7ec9ec640 1 -- 172.21.15.26:0/3874488601 learned_addr learned my addr 172.21.15.26:0/3874488601 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7ec9ec640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.26:6792/0 conn(0x7fa7e80a5d50 legacy=0x7fa7e81d6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 client.?.objecter ms_handle_connect 0x7fa7e80a5d50 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 client.?.objecter ms_handle_connect 0x7fa7e806e7b0 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e7fff640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.104:6791/0 conn(0x7fa7e806e7b0 legacy=0x7fa7e81ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2190510502 0 0) 0x7fa7e8139200 con 0x7fa7e80a5d50 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): my global_id is 164463 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7d4004eb0 con 0x7fa7e80a5d50 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 client.?.objecter ms_handle_connect 0x7fa7e80a5920 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e77fe640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.104:6789/0 conn(0x7fa7e80a5920 legacy=0x7fa7e81d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1043208391 0 0) 0x7fa7e81346c0 con 0x7fa7e806e7b0 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): my global_id is 26906 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e8139200 con 0x7fa7e806e7b0 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3047254096 0 0) 0x7fa7e813aa00 con 0x7fa7e80a5920 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): my global_id is 27694 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e81346c0 con 0x7fa7e80a5920 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3701107115 0 0) 0x7fa7d4004eb0 con 0x7fa7e80a5d50 2024-08-19T12:49:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7e813aa00 con 0x7fa7e80a5d50 2024-08-19T12:49:16.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7e0003cd0 con 0x7fa7e80a5d50 2024-08-19T12:49:16.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:16.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:16.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): dump: 2024-08-19T12:49:16.165 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:16.165 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:16.165 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.165 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.165 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:16.165 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:16.165 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:16.166 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:16.166 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:16.166 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:16.166 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:16.166 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:16.166 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:16.166 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:16.166 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:16.166 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:16.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:16.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 827520097 0 0) 0x7fa7e8139200 con 0x7fa7e806e7b0 2024-08-19T12:49:16.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7d4004eb0 con 0x7fa7e806e7b0 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7d8001c40 con 0x7fa7e806e7b0 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.145+0000 7fa7e4ff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 10 monclient(hunting): dump: 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:16.167 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1832959561 0 0) 0x7fa7e813aa00 con 0x7fa7e80a5d50 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.104:6791/0 conn(0x7fa7e806e7b0 legacy=0x7fa7e81ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.104:6789/0 conn(0x7fa7e80a5920 legacy=0x7fa7e81d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 1 monclient: found mon.d 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e813b560 con 0x7fa7e80a5d50 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 10 monclient: _finish_auth 0 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7ee44f640 5 monclient: authenticate success, global_id 164463 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7ee44f640 10 monclient: _renew_subs 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7ee44f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7ee44f640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e813acc0 con 0x7fa7e80a5d50 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7ee44f640 10 client.164463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7ee44f640 10 monclient: _renew_subs 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7ee44f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7ee44f640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa7e813bf90 con 0x7fa7e80a5d50 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7ee44f640 1 librados: init done 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7e0002090 con 0x7fa7e80a5d50 2024-08-19T12:49:16.169 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:16.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7e0004510 con 0x7fa7e80a5d50 2024-08-19T12:49:16.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:16.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:16.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e67fc640 4 set_mon_vals no callback set 2024-08-19T12:49:16.170 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 10 monclient: dump: 2024-08-19T12:49:16.170 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:16.170 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:16.170 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.170 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.170 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:16.170 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:16.170 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:16.171 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:16.171 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:16.171 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:16.171 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:16.171 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:16.171 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:16.171 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:16.171 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:16.171 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:16.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7ee44f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:16.171 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7ee44f640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa7c80065c0 con 0x7fa7c8002860 2024-08-19T12:49:16.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.146+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa7e00046b0 con 0x7fa7e80a5d50 2024-08-19T12:49:16.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.147+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(409..409 src has 311..409) ==== 5333+0+0 (unknown 2618201168 0 0) 0x7fa7e004e5b0 con 0x7fa7e80a5d50 2024-08-19T12:49:16.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.147+0000 7fa7e4ff9640 10 client.164463.objecter ms_dispatch 0x7fa7e8000b90 osd_map(409..409 src has 311..409) 2024-08-19T12:49:16.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.147+0000 7fa7e4ff9640 3 client.164463.objecter handle_osd_map got epochs [409,409] > 0 2024-08-19T12:49:16.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.147+0000 7fa7e4ff9640 3 client.164463.objecter handle_osd_map decoding full epoch 409 2024-08-19T12:49:16.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.147+0000 7fa7e4ff9640 20 client.164463.objecter dump_active .. 0 homeless 2024-08-19T12:49:16.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.147+0000 7fa7e77fe640 10 monclient: get_auth_request con 0x7fa7d4041480 auth_method 0 2024-08-19T12:49:16.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e7fff640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.104:6792/0 conn(0x7fa7c8002860 legacy=0x7fa7c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:16.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 10 client.164463.objecter ms_handle_connect 0x7fa7c8002860 2024-08-19T12:49:16.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 10 client.164463.objecter resend_mon_ops 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 311162599 0 0) 0x7fa7c80065c0 con 0x7fa7c8002860 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 10 monclient(hunting): my global_id is 135754 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e813aa00 con 0x7fa7c8002860 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 574070195 0 0) 0x7fa7e813aa00 con 0x7fa7c8002860 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7c80065c0 con 0x7fa7c8002860 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 10 client.164463.objecter ms_handle_connect 0x7fa7d4041480 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7d8003c90 con 0x7fa7c8002860 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 10 monclient: dump: 2024-08-19T12:49:16.173 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:16.174 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:16.175 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:16.175 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:16.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 977662315 0 0) 0x7fa7c80065c0 con 0x7fa7c8002860 2024-08-19T12:49:16.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.149+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7c800a160 con 0x7fa7c8002860 2024-08-19T12:49:16.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.150+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa7d8003f50 con 0x7fa7c8002860 2024-08-19T12:49:16.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.150+0000 7fa7e4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:16.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.150+0000 7fa7e4ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:16.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.150+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.104:6792/0 conn(0x7fa7c8002860 legacy=0x7fa7c8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:16.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.150+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.104:6792/0 conn(0x7fa7c8002860 legacy=0x7fa7c8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:16.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.156+0000 7fa7ee44f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:16.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.156+0000 7fa7ee44f640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa7c8009e70 con 0x7fa7c8009130 2024-08-19T12:49:16.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e7fff640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.104:6792/0 conn(0x7fa7c8009130 legacy=0x7fa7c80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:16.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 10 client.164463.objecter ms_handle_connect 0x7fa7c8009130 2024-08-19T12:49:16.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 10 client.164463.objecter resend_mon_ops 2024-08-19T12:49:16.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3118638599 0 0) 0x7fa7c8009e70 con 0x7fa7c8009130 2024-08-19T12:49:16.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:49:16.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:16.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:16.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 10 monclient(hunting): my global_id is 135763 2024-08-19T12:49:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7c80065c0 con 0x7fa7c8009130 2024-08-19T12:49:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3545085150 0 0) 0x7fa7c80065c0 con 0x7fa7c8009130 2024-08-19T12:49:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7c8009e70 con 0x7fa7c8009130 2024-08-19T12:49:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa7d8005b00 con 0x7fa7c8009130 2024-08-19T12:49:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 10 monclient: dump: 2024-08-19T12:49:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3884952527 0 0) 0x7fa7c8009e70 con 0x7fa7c8009130 2024-08-19T12:49:16.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7c800c410 con 0x7fa7c8009130 2024-08-19T12:49:16.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1968308202) 0x7fa7d8005b70 con 0x7fa7c8009130 2024-08-19T12:49:16.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:16.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:16.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.104:6792/0 conn(0x7fa7c8009130 legacy=0x7fa7c80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:16.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.159+0000 7fa7e4ff9640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.104:6792/0 conn(0x7fa7c8009130 legacy=0x7fa7c80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:16.180 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:16.180 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 68, 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 526517, 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:49:16.181 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:16.182 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:16.183 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:16.184 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:16.185 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:16.200 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:16.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 10 librados: watch_flush enter 2024-08-19T12:49:16.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 10 librados: watch_flush exit 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 20 client.164463.objecter shutdown clearing up homeless session... 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 10 client.164463.objecter successfully canceled tick 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.104:6800/899887506 conn(0x7fa7d4041480 legacy=0x7fa7d4043940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 10 monclient: shutdown 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 1 -- 172.21.15.26:0/3874488601 >> v1:172.21.15.26:6792/0 conn(0x7fa7e80a5d50 legacy=0x7fa7e81d6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ec9ec640 1 -- 172.21.15.26:0/3874488601 reap_dead start 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ec9ec640 1 -- 172.21.15.26:0/3874488601 reap_dead start 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 1 -- 172.21.15.26:0/3874488601 shutdown_connections 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 1 -- 172.21.15.26:0/3874488601 >> 172.21.15.26:0/3874488601 conn(0x7fa7e80a8740 msgr2=0x7fa7e8135200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 1 -- 172.21.15.26:0/3874488601 shutdown_connections 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 1 -- 172.21.15.26:0/3874488601 wait complete. 2024-08-19T12:49:16.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:16.160+0000 7fa7ee44f640 1 librados: shutdown 2024-08-19T12:49:16.203 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:49:16.203 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['h', 'f', 'e', 'c'] 2024-08-19T12:49:16.203 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['i'] 2024-08-19T12:49:16.203 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h 2024-08-19T12:49:16.203 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-08-19T12:49:16.204 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-08-19T12:49:16.204 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:49:16.222 INFO:tasks.ceph.mon.h:Stopped 2024-08-19T12:49:16.222 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f 2024-08-19T12:49:16.222 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-08-19T12:49:16.222 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-08-19T12:49:16.222 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:49:16.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876912938}) 2024-08-19T12:49:16.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4029001264}) 2024-08-19T12:49:16.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876912938}) 2024-08-19T12:49:16.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3691813297}) 2024-08-19T12:49:16.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876912938}) 2024-08-19T12:49:16.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3923546045}) 2024-08-19T12:49:16.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876912938}) 2024-08-19T12:49:16.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2144083889}) 2024-08-19T12:49:16.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876912938}) 2024-08-19T12:49:16.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2229767113}) 2024-08-19T12:49:16.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876912938}) 2024-08-19T12:49:16.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3146379949}) 2024-08-19T12:49:16.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876912938}) 2024-08-19T12:49:16.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2474219506}) 2024-08-19T12:49:16.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:49:16.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876912938}) 2024-08-19T12:49:16.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:49:16.260+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1049412578}) 2024-08-19T12:49:16.280 INFO:tasks.ceph.mon.f:Stopped 2024-08-19T12:49:16.280 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-08-19T12:49:16.280 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-08-19T12:49:16.281 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-08-19T12:49:16.281 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:49:16.292 INFO:tasks.ceph.mon.e:Stopped 2024-08-19T12:49:16.292 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-08-19T12:49:16.292 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-08-19T12:49:16.292 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-19T12:49:16.293 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:49:16.347 INFO:tasks.ceph.mon.c:Stopped 2024-08-19T12:49:16.347 INFO:tasks.mon_thrash:Sending STOP to mon i 2024-08-19T12:49:16.347 INFO:tasks.ceph.mon.i:Sent signal 19 2024-08-19T12:49:16.347 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:49:31.349 INFO:tasks.mon_thrash:Sending CONT to mon i 2024-08-19T12:49:31.350 INFO:tasks.ceph.mon.i:Sent signal 18 2024-08-19T12:49:31.350 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-08-19T12:49:31.351 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:49:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.460+0000 7f10b70e5640 10 monclient: get_monmap_and_config 2024-08-19T12:49:31.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.460+0000 7f10b70e5640 10 monclient: build_initial_monmap 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.460+0000 7f10b70e5640 10 monclient: monmap: 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:31.462682+0000 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:31.462682+0000 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:31.463 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:31.464 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:31.464 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.461+0000 7f10b70e5640 1 Processor -- start 2024-08-19T12:49:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.461+0000 7f10b70e5640 1 -- start start 2024-08-19T12:49:31.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.461+0000 7f10b70e5640 10 monclient: init 2024-08-19T12:49:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b70e5640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b70e5640 10 monclient: _add_conns ranks=[0,6,5,8,2,3,1,4,7] 2024-08-19T12:49:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b70e5640 10 monclient(hunting): picked mon.noname-a con 0x7f10b006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b70e5640 10 monclient(hunting): picked mon.noname-d con 0x7f10b00a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b70e5640 10 monclient(hunting): picked mon.noname-h con 0x7f10b00a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b70e5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b0136790 con 0x7f10b006e7b0 2024-08-19T12:49:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b70e5640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b009ddf0 con 0x7f10b00a5d50 2024-08-19T12:49:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b70e5640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b009e0d0 con 0x7f10b00a5920 2024-08-19T12:49:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b70e5640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:31.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b70e5640 10 monclient(hunting): authenticate will time out at 1365.550259s 2024-08-19T12:49:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b4e81640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f10b006e7b0 0x7f10b0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:37334/0 (socket says 172.21.15.26:37334) 2024-08-19T12:49:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b4e81640 1 -- 172.21.15.26:0/1294848018 learned_addr learned my addr 172.21.15.26:0/1294848018 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10affff640 1 --1- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6792/0 conn(0x7f10b00a5920 0x7f10b013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51010/0 (socket says 172.21.15.26:51010) 2024-08-19T12:49:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.462+0000 7f10b5682640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6791/0 conn(0x7f10b00a5d50 legacy=0x7f10b0141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:49:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.464+0000 7f10b4e81640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6789/0 conn(0x7f10b006e7b0 legacy=0x7f10b0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:31.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.464+0000 7f10affff640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6792/0 conn(0x7f10b00a5920 legacy=0x7f10b013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:31.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:31.663+0000 7f10b5682640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6791/0 conn(0x7f10b00a5d50 legacy=0x7f10b0141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:49:32.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:32.064+0000 7f10b5682640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6791/0 conn(0x7f10b00a5d50 legacy=0x7f10b0141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:49:32.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:32.865+0000 7f10b5682640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6791/0 conn(0x7f10b00a5d50 legacy=0x7f10b0141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:49:34.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 10 monclient(hunting): tick 2024-08-19T12:49:34.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 1 monclient(hunting): continuing hunt 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6792/0 conn(0x7f10b00a5920 legacy=0x7f10b013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6791/0 conn(0x7f10b00a5d50 legacy=0x7f10b0141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6789/0 conn(0x7f10b006e7b0 legacy=0x7f10b0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 10 monclient: _add_conns ranks=[4,3,7,1,8,2] 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f109c0038c0 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 10 monclient(hunting): picked mon.noname-g con 0x7f109c007550 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 10 monclient(hunting): picked mon.noname-i con 0x7f109c00b2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 1 -- 172.21.15.26:0/1294848018 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b009e3b0 con 0x7f109c007550 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 1 -- 172.21.15.26:0/1294848018 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f109c010040 con 0x7f109c0038c0 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 1 -- 172.21.15.26:0/1294848018 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f109c0112e0 con 0x7f109c00b2e0 2024-08-19T12:49:34.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10ae7fc640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.463+0000 7f10b4e81640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6791/0 conn(0x7f109c0038c0 legacy=0x7f109c005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:49:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.465+0000 7f10b5682640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6792/0 conn(0x7f109c00b2e0 legacy=0x7f109c00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:34.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.465+0000 7f10affff640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6790/0 conn(0x7f109c007550 legacy=0x7f109c009a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:34.665+0000 7f10b4e81640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6791/0 conn(0x7f109c0038c0 legacy=0x7f109c005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:49:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:35.066+0000 7f10b4e81640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6791/0 conn(0x7f109c0038c0 legacy=0x7f109c005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:49:35.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:35.867+0000 7f10b4e81640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6791/0 conn(0x7f109c0038c0 legacy=0x7f109c005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:49:37.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 10 monclient(hunting): tick 2024-08-19T12:49:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:49:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 1 monclient(hunting): continuing hunt 2024-08-19T12:49:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:49:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6792/0 conn(0x7f109c00b2e0 legacy=0x7f109c00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6791/0 conn(0x7f109c0038c0 legacy=0x7f109c005d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:49:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6790/0 conn(0x7f109c007550 legacy=0x7f109c009a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 10 monclient: _add_conns ranks=[8,2,1] 2024-08-19T12:49:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 10 monclient(hunting): picked mon.noname-e con 0x7f109c00f310 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10b5682640 1 -- 172.21.15.26:0/1294848018 reap_dead start 2024-08-19T12:49:37.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f109c0114b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 10 monclient(hunting): picked mon.noname-f con 0x7f109c00a1e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 1 -- 172.21.15.26:0/1294848018 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f109c012640 con 0x7f109c00a1e0 2024-08-19T12:49:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 1 -- 172.21.15.26:0/1294848018 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f109c010040 con 0x7f109c0114b0 2024-08-19T12:49:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10b4e81640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6793/0 conn(0x7f109c00f310 legacy=0x7f109c00f750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:49:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 1 -- 172.21.15.26:0/1294848018 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f109c012280 con 0x7f109c00f310 2024-08-19T12:49:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10ae7fc640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.463+0000 7f10b5682640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6789/0 conn(0x7f109c00a1e0 legacy=0x7f109c009a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:49:37.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.465+0000 7f10affff640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6790/0 conn(0x7f109c0114b0 legacy=0x7f109c011890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:37.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.664+0000 7f10b4e81640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6793/0 conn(0x7f109c00f310 legacy=0x7f109c00f750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:49:37.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.664+0000 7f10af7fe640 1 -- 172.21.15.26:0/1294848018 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:49:37.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:37.664+0000 7f10b5682640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6789/0 conn(0x7f109c00a1e0 legacy=0x7f109c009a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:49:38.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:38.065+0000 7f10b5682640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6789/0 conn(0x7f109c00a1e0 legacy=0x7f109c009a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:49:38.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:38.065+0000 7f10b4e81640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6793/0 conn(0x7f109c00f310 legacy=0x7f109c00f750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:49:38.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:38.866+0000 7f10b4e81640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6793/0 conn(0x7f109c00f310 legacy=0x7f109c00f750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:49:38.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:38.866+0000 7f10b5682640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6789/0 conn(0x7f109c00a1e0 legacy=0x7f109c009a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:49:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.337+0000 7f10af7fe640 1 -- 172.21.15.26:0/1294848018 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3636517749 0 0) 0x7f109c010040 con 0x7f109c0114b0 2024-08-19T12:49:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.337+0000 7f10af7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.337+0000 7f10af7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.337+0000 7f10af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.337+0000 7f10af7fe640 10 monclient(hunting): my global_id is 26813 2024-08-19T12:49:40.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.337+0000 7f10af7fe640 1 -- 172.21.15.26:0/1294848018 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1098003520 con 0x7f109c0114b0 2024-08-19T12:49:40.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 1 -- 172.21.15.26:0/1294848018 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 988398383 0 0) 0x7f1098003520 con 0x7f109c0114b0 2024-08-19T12:49:40.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6793/0 conn(0x7f109c00f310 legacy=0x7f109c00f750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:49:40.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.104:6789/0 conn(0x7f109c00a1e0 legacy=0x7f109c009a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:49:40.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:49:40.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 1 monclient: found mon.noname-b 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 1 -- 172.21.15.26:0/1294848018 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109c005d70 con 0x7f109c0114b0 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 10 monclient: _finish_auth 0 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 1 -- 172.21.15.26:0/1294848018 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f10a40019d0 con 0x7f109c0114b0 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10af7fe640 10 monclient: dump: 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.359 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10b70e5640 5 monclient: authenticate success, global_id 26813 2024-08-19T12:49:40.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.356+0000 7f10b70e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.357+0000 7f10af7fe640 1 -- 172.21.15.26:0/1294848018 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f10a4001b70 con 0x7f109c0114b0 2024-08-19T12:49:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.357+0000 7f10af7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:40.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.357+0000 7f10af7fe640 1 -- 172.21.15.26:0/1294848018 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f10a4004b20 con 0x7f109c0114b0 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.357+0000 7f10b70e5640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.357+0000 7f10b70e5640 4 set_mon_vals no callback set 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.357+0000 7f10af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.357+0000 7f10af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.357+0000 7f10af7fe640 10 monclient: dump: 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.364 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.365 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.358+0000 7f10b70e5640 10 monclient: shutdown 2024-08-19T12:49:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.358+0000 7f10b70e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.358+0000 7f10b70e5640 1 -- 172.21.15.26:0/1294848018 >> v1:172.21.15.26:6790/0 conn(0x7f109c0114b0 legacy=0x7f109c011890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.358+0000 7f10b70e5640 1 -- 172.21.15.26:0/1294848018 shutdown_connections 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.358+0000 7f10b70e5640 1 -- 172.21.15.26:0/1294848018 >> 172.21.15.26:0/1294848018 conn(0x7f10b00a8740 msgr2=0x7f10b00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.358+0000 7f10b70e5640 1 -- 172.21.15.26:0/1294848018 shutdown_connections 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.358+0000 7f10b70e5640 1 -- 172.21.15.26:0/1294848018 wait complete. 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.359+0000 7f10b70e5640 10 monclient: build_initial_monmap 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.359+0000 7f10b70e5640 10 monclient: monmap: 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:40.366 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.359+0000 7f10b70e5640 1 librados: starting msgr at 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.359+0000 7f10b70e5640 1 librados: starting objecter 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.359+0000 7f10b70e5640 1 Processor -- start 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.359+0000 7f10b70e5640 1 -- start start 2024-08-19T12:49:40.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.359+0000 7f10b70e5640 1 librados: setting wanted keys 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.359+0000 7f10b70e5640 1 librados: calling monclient init 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.359+0000 7f10b70e5640 10 monclient: init 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b70e5640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b70e5640 10 monclient: _add_conns ranks=[0,4,8,7,5,1,2,3,6] 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b70e5640 10 monclient(hunting): picked mon.noname-a con 0x7f109c00a1e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b70e5640 10 monclient(hunting): picked mon.noname-c con 0x7f109c00f310 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b70e5640 10 monclient(hunting): picked mon.noname-e con 0x7f109c0114b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b70e5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b0132960 con 0x7f109c00a1e0 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b70e5640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b00a2f00 con 0x7f109c00f310 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b70e5640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b00a4ea0 con 0x7f109c0114b0 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b70e5640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b70e5640 10 monclient(hunting): authenticate will time out at 1374.448543s 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10affff640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f109c00f310 legacy=0x7f10b0132250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:49:40.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b5682640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f109c0114b0 legacy=0x7f10b01d65f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:49:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b4e81640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f109c00a1e0 0x7f10b0131b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50580/0 (socket says 172.21.15.26:50580) 2024-08-19T12:49:40.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.360+0000 7f10b4e81640 1 -- 172.21.15.26:0/195013838 learned_addr learned my addr 172.21.15.26:0/195013838 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.362+0000 7f10b4e81640 1 -- 172.21.15.26:0/195013838 >> v1:172.21.15.26:6789/0 conn(0x7f109c00a1e0 legacy=0x7f10b0131b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.362+0000 7f10ad7fa640 10 client.?.objecter ms_handle_connect 0x7f109c00a1e0 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.362+0000 7f10ad7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.362+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1510684876 0 0) 0x7f10b0132960 con 0x7f109c00a1e0 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient(hunting): my global_id is 145170 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f109c00f030 con 0x7f109c00a1e0 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1768296661 0 0) 0x7f109c00f030 con 0x7f109c00a1e0 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10b0132960 con 0x7f109c00a1e0 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f10a00047e0 con 0x7f109c00a1e0 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:49:40.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient(hunting): dump: 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.373 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1539887008 0 0) 0x7f10b0132960 con 0x7f109c00a1e0 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 >> v1:172.21.15.26:6793/0 conn(0x7f109c0114b0 legacy=0x7f10b01d65f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 >> v1:172.21.15.26:6791/0 conn(0x7f109c00f310 legacy=0x7f10b0132250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient: _finish_hunting 0 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 1 monclient: found mon.a 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b00a48e0 con 0x7f109c00a1e0 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient: _finish_auth 0 2024-08-19T12:49:40.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient: _check_auth_tickets 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10b70e5640 5 monclient: authenticate success, global_id 145170 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10b70e5640 10 monclient: _renew_subs 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10b70e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10b70e5640 1 -- 172.21.15.26:0/195013838 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f10b00a4bc0 con 0x7f109c00a1e0 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10b70e5640 10 client.145170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10b70e5640 10 monclient: _renew_subs 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10b70e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10b70e5640 1 -- 172.21.15.26:0/195013838 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f10b00a3980 con 0x7f109c00a1e0 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10b70e5640 1 librados: init done 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f10a0004d20 con 0x7f109c00a1e0 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f10a0005250 con 0x7f109c00a1e0 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:40.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10aeffd640 4 set_mon_vals no callback set 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.363+0000 7f10ad7fa640 10 monclient: dump: 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.376 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.364+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f10a00053f0 con 0x7f109c00a1e0 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.364+0000 7f10b70e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.364+0000 7f10b70e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.364+0000 7f10b70e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.364+0000 7f10b70e5640 1 -- 172.21.15.26:0/195013838 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1090005080 con 0x7f109c00a1e0 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.365+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(409..409 src has 311..409) ==== 5333+0+0 (unknown 2618201168 0 0) 0x7f10a004e2e0 con 0x7f109c00a1e0 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.365+0000 7f10ad7fa640 10 client.145170.objecter ms_dispatch 0x7f10b0000b90 osd_map(409..409 src has 311..409) 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.365+0000 7f10ad7fa640 3 client.145170.objecter handle_osd_map got epochs [409,409] > 0 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.365+0000 7f10ad7fa640 3 client.145170.objecter handle_osd_map decoding full epoch 409 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.365+0000 7f10ad7fa640 20 client.145170.objecter dump_active .. 0 homeless 2024-08-19T12:49:40.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.365+0000 7f10affff640 10 monclient: get_auth_request con 0x7f109c043020 auth_method 0 2024-08-19T12:49:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.368+0000 7f10ad7fa640 10 client.145170.objecter ms_handle_connect 0x7f109c043020 2024-08-19T12:49:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.373+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f10a0017690 con 0x7f109c00a1e0 2024-08-19T12:49:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.373+0000 7f10ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:40.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.373+0000 7f10ad7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:40.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.630+0000 7f10b70e5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:49:40.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.630+0000 7f10b70e5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:49:40.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.630+0000 7f10b70e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:40.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.630+0000 7f10b70e5640 1 -- 172.21.15.26:0/195013838 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1090005370 con 0x7f109c00a1e0 2024-08-19T12:49:40.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.630+0000 7f10ad7fa640 1 -- 172.21.15.26:0/195013838 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2705 (unknown 254746657 0 3077543202) 0x7f10a0016f90 con 0x7f109c00a1e0 2024-08-19T12:49:40.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.630+0000 7f10ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:49:40.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.630+0000 7f10ad7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:40.633 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":76,"quorum":[0,2,3,6,7],"quorum_names":["a","b","g","d","i"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:49:40.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 10 librados: watch_flush enter 2024-08-19T12:49:40.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 10 librados: watch_flush exit 2024-08-19T12:49:40.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 20 client.145170.objecter shutdown clearing up homeless session... 2024-08-19T12:49:40.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 10 client.145170.objecter successfully canceled tick 2024-08-19T12:49:40.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 1 -- 172.21.15.26:0/195013838 >> v1:172.21.15.104:6800/899887506 conn(0x7f109c043020 legacy=0x7f109c0454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 10 monclient: shutdown 2024-08-19T12:49:40.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:40.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 1 -- 172.21.15.26:0/195013838 >> v1:172.21.15.26:6789/0 conn(0x7f109c00a1e0 legacy=0x7f10b0131b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 1 -- 172.21.15.26:0/195013838 shutdown_connections 2024-08-19T12:49:40.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 1 -- 172.21.15.26:0/195013838 >> 172.21.15.26:0/195013838 conn(0x7f10b00a8740 msgr2=0x7f10b0134190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:40.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 1 -- 172.21.15.26:0/195013838 shutdown_connections 2024-08-19T12:49:40.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 1 -- 172.21.15.26:0/195013838 wait complete. 2024-08-19T12:49:40.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.632+0000 7f10b70e5640 1 librados: shutdown 2024-08-19T12:49:40.645 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-08-19T12:49:40.645 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon a 2024-08-19T12:49:40.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:49:40.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.743+0000 7f9ab787e640 10 monclient: get_monmap_and_config 2024-08-19T12:49:40.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.743+0000 7f9ab787e640 10 monclient: build_initial_monmap 2024-08-19T12:49:40.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.743+0000 7f9ab787e640 10 monclient: monmap: 2024-08-19T12:49:40.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:40.745 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.745 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:40.745432+0000 2024-08-19T12:49:40.745 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:40.745432+0000 2024-08-19T12:49:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:40.746 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:40.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 1 Processor -- start 2024-08-19T12:49:40.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 1 -- start start 2024-08-19T12:49:40.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 10 monclient: init 2024-08-19T12:49:40.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:40.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 10 monclient: _add_conns ranks=[0,2,8,3,4,5,1,7,6] 2024-08-19T12:49:40.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 10 monclient(hunting): picked mon.noname-a con 0x7f9ab006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:40.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 10 monclient(hunting): picked mon.noname-b con 0x7f9ab00a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:40.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 10 monclient(hunting): picked mon.noname-e con 0x7f9ab00a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:40.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab0136790 con 0x7f9ab006e7b0 2024-08-19T12:49:40.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab009ddf0 con 0x7f9ab00a5920 2024-08-19T12:49:40.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab009e0d0 con 0x7f9ab00a5d50 2024-08-19T12:49:40.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:40.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab787e640 10 monclient(hunting): authenticate will time out at 1374.832947s 2024-08-19T12:49:40.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.744+0000 7f9ab5e1b640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f9ab00a5d50 legacy=0x7f9ab0141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:49:40.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.745+0000 7f9ab4e19640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9ab00a5920 0x7f9ab013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45184/0 (socket says 172.21.15.26:45184) 2024-08-19T12:49:40.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.745+0000 7f9ab561a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9ab006e7b0 0x7f9ab0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50596/0 (socket says 172.21.15.26:50596) 2024-08-19T12:49:40.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.745+0000 7f9ab4e19640 1 -- 172.21.15.26:0/2325152517 learned_addr learned my addr 172.21.15.26:0/2325152517 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9ab4e19640 1 -- 172.21.15.26:0/2325152517 >> v1:172.21.15.26:6790/0 conn(0x7f9ab00a5920 legacy=0x7f9ab013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9ab561a640 1 -- 172.21.15.26:0/2325152517 >> v1:172.21.15.26:6789/0 conn(0x7f9ab006e7b0 legacy=0x7f9ab0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 1 -- 172.21.15.26:0/2325152517 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 642088877 0 0) 0x7f9aa4001450 con 0x7f9ab00a5920 2024-08-19T12:49:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient(hunting): my global_id is 26822 2024-08-19T12:49:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 1 -- 172.21.15.26:0/2325152517 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a9c003520 con 0x7f9ab00a5920 2024-08-19T12:49:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 1 -- 172.21.15.26:0/2325152517 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 220859411 0 0) 0x7f9ab0136790 con 0x7f9ab006e7b0 2024-08-19T12:49:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient(hunting): my global_id is 145179 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 1 -- 172.21.15.26:0/2325152517 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9aa4001450 con 0x7f9ab006e7b0 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 1 -- 172.21.15.26:0/2325152517 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2244030945 0 0) 0x7f9a9c003520 con 0x7f9ab00a5920 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 1 -- 172.21.15.26:0/2325152517 >> v1:172.21.15.26:6793/0 conn(0x7f9ab00a5d50 legacy=0x7f9ab0141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 1 -- 172.21.15.26:0/2325152517 >> v1:172.21.15.26:6789/0 conn(0x7f9ab006e7b0 legacy=0x7f9ab0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient: _finish_hunting 0 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 1 monclient: found mon.noname-b 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 1 -- 172.21.15.26:0/2325152517 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab009e3b0 con 0x7f9ab00a5920 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient: _finish_auth 0 2024-08-19T12:49:40.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient: _check_auth_tickets 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 1 -- 172.21.15.26:0/2325152517 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9aa4001cb0 con 0x7f9ab00a5920 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9ab787e640 5 monclient: authenticate success, global_id 26822 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9ab787e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient: dump: 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.756 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.756 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.756 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.756 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.756 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.756 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.756 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.756 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.756 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9ab787e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:40.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 1 -- 172.21.15.26:0/2325152517 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9aa4004990 con 0x7f9ab00a5920 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9ab787e640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 1 -- 172.21.15.26:0/2325152517 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9aa4004ec0 con 0x7f9ab00a5920 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9ab787e640 4 set_mon_vals no callback set 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.747+0000 7f9aa3fff640 10 monclient: dump: 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.757 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.748+0000 7f9ab787e640 10 monclient: shutdown 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.748+0000 7f9ab787e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.748+0000 7f9ab787e640 1 -- 172.21.15.26:0/2325152517 >> v1:172.21.15.26:6790/0 conn(0x7f9ab00a5920 legacy=0x7f9ab013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.748+0000 7f9ab787e640 1 -- 172.21.15.26:0/2325152517 shutdown_connections 2024-08-19T12:49:40.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.748+0000 7f9ab787e640 1 -- 172.21.15.26:0/2325152517 >> 172.21.15.26:0/2325152517 conn(0x7f9ab00a8740 msgr2=0x7f9ab00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.748+0000 7f9ab787e640 1 -- 172.21.15.26:0/2325152517 shutdown_connections 2024-08-19T12:49:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.748+0000 7f9ab787e640 1 -- 172.21.15.26:0/2325152517 wait complete. 2024-08-19T12:49:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.748+0000 7f9ab787e640 10 monclient: build_initial_monmap 2024-08-19T12:49:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.748+0000 7f9ab787e640 10 monclient: monmap: 2024-08-19T12:49:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.759 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 1 librados: starting msgr at 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 1 librados: starting objecter 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 1 Processor -- start 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 1 -- start start 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 1 librados: setting wanted keys 2024-08-19T12:49:40.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 1 librados: calling monclient init 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 10 monclient: init 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 10 monclient: _add_conns ranks=[3,7,1,5,0,4,8,2,6] 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 10 monclient(hunting): picked mon.noname-g con 0x7f9ab006e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 10 monclient(hunting): picked mon.noname-i con 0x7f9ab00a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 10 monclient(hunting): picked mon.noname-f con 0x7f9ab00a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab0134890 con 0x7f9ab00a5d50 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab0142f00 con 0x7f9ab006e7b0 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab0138210 con 0x7f9ab00a5920 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab787e640 10 monclient(hunting): authenticate will time out at 1374.837765s 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.749+0000 7f9ab5e1b640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f9ab00a5d50 legacy=0x7f9ab01d6290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:49:40.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.750+0000 7f9ab561a640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f9ab006e7b0 0x7f9ab01373f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:53864/0 (socket says 172.21.15.26:53864) 2024-08-19T12:49:40.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.750+0000 7f9ab4e19640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f9ab00a5920 0x7f9ab0137b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:35064/0 (socket says 172.21.15.26:35064) 2024-08-19T12:49:40.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.750+0000 7f9ab561a640 1 -- 172.21.15.26:0/963616565 learned_addr learned my addr 172.21.15.26:0/963616565 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9ab4e19640 1 -- 172.21.15.26:0/963616565 >> v1:172.21.15.104:6792/0 conn(0x7f9ab00a5920 legacy=0x7f9ab0137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 client.?.objecter ms_handle_connect 0x7f9ab00a5920 2024-08-19T12:49:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 client.?.objecter ms_handle_connect 0x7f9ab006e7b0 2024-08-19T12:49:40.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9ab561a640 1 -- 172.21.15.26:0/963616565 >> v1:172.21.15.104:6790/0 conn(0x7f9ab006e7b0 legacy=0x7f9ab01373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1265826257 0 0) 0x7f9aac001580 con 0x7f9ab006e7b0 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): my global_id is 154605 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a94004eb0 con 0x7f9ab006e7b0 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4052793026 0 0) 0x7f9ab0138210 con 0x7f9ab00a5920 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): my global_id is 135772 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9aac001580 con 0x7f9ab00a5920 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2447627300 0 0) 0x7f9a94004eb0 con 0x7f9ab006e7b0 2024-08-19T12:49:40.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ab0138210 con 0x7f9ab006e7b0 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9aac004b20 con 0x7f9ab006e7b0 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): dump: 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.770 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 981673507 0 0) 0x7f9aac001580 con 0x7f9ab00a5920 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a94004eb0 con 0x7f9ab00a5920 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9aa4001c70 con 0x7f9ab00a5920 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient(hunting): dump: 2024-08-19T12:49:40.771 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.772 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3771195439 0 0) 0x7f9ab0138210 con 0x7f9ab006e7b0 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 >> v1:172.21.15.104:6792/0 conn(0x7f9ab00a5920 legacy=0x7f9ab0137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 >> v1:172.21.15.104:6789/0 conn(0x7f9ab00a5d50 legacy=0x7f9ab01d6290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 monclient: found mon.g 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab013ae90 con 0x7f9ab006e7b0 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient: _finish_auth 0 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9aa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9ab787e640 5 monclient: authenticate success, global_id 154605 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9ab787e640 10 monclient: _renew_subs 2024-08-19T12:49:40.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9ab787e640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.752+0000 7f9ab787e640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab013b170 con 0x7f9ab006e7b0 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.753+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.3 v1:172.21.15.104:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9aac004f70 con 0x7f9ab006e7b0 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.753+0000 7f9aa1ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.753+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.3 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9aac0054e0 con 0x7f9ab006e7b0 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.753+0000 7f9aa1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.753+0000 7f9aa1ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.753+0000 7f9aa1ffb640 10 monclient: dump: 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.774 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.753+0000 7f9ab787e640 10 client.154605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.753+0000 7f9ab787e640 10 monclient: _renew_subs 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.753+0000 7f9aa37fe640 4 set_mon_vals no callback set 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.753+0000 7f9ab787e640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:49:40.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.753+0000 7f9ab787e640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9ab01428e0 con 0x7f9ab006e7b0 2024-08-19T12:49:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.753+0000 7f9ab787e640 1 librados: init done 2024-08-19T12:49:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.754+0000 7f9ab787e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.754+0000 7f9ab787e640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9ab00008d0 con 0x7f9ab00a08c0 2024-08-19T12:49:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.754+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.3 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f9aac005680 con 0x7f9ab006e7b0 2024-08-19T12:49:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.754+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.3 v1:172.21.15.104:6790/0 8 ==== osd_map(409..409 src has 311..409) ==== 5333+0+0 (unknown 2618201168 0 0) 0x7f9aac04e560 con 0x7f9ab006e7b0 2024-08-19T12:49:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.754+0000 7f9aa1ffb640 10 client.154605.objecter ms_dispatch 0x7f9ab0000b90 osd_map(409..409 src has 311..409) 2024-08-19T12:49:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.754+0000 7f9aa1ffb640 3 client.154605.objecter handle_osd_map got epochs [409,409] > 0 2024-08-19T12:49:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.754+0000 7f9aa1ffb640 3 client.154605.objecter handle_osd_map decoding full epoch 409 2024-08-19T12:49:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.754+0000 7f9aa1ffb640 20 client.154605.objecter dump_active .. 0 homeless 2024-08-19T12:49:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.754+0000 7f9ab5e1b640 10 monclient: get_auth_request con 0x7f9a94041440 auth_method 0 2024-08-19T12:49:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9ab4e19640 1 -- 172.21.15.26:0/963616565 >> v1:172.21.15.26:6789/0 conn(0x7f9ab00a08c0 legacy=0x7f9ab00a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:40.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 10 client.154605.objecter ms_handle_connect 0x7f9ab00a08c0 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 10 client.154605.objecter resend_mon_ops 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3065047639 0 0) 0x7f9ab00008d0 con 0x7f9ab00a08c0 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 10 monclient(hunting): my global_id is 145188 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ab0138210 con 0x7f9ab00a08c0 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3848761595 0 0) 0x7f9ab0138210 con 0x7f9ab00a08c0 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ab00008d0 con 0x7f9ab00a08c0 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9aa4004f00 con 0x7f9ab00a08c0 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:40.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 10 monclient: dump: 2024-08-19T12:49:40.778 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.778 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.778 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.778 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.778 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.778 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.778 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.778 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.778 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.778 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.778 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.778 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 937672107 0 0) 0x7f9ab00008d0 con 0x7f9ab00a08c0 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ab0143110 con 0x7f9ab00a08c0 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.756+0000 7f9aa1ffb640 10 client.154605.objecter ms_handle_connect 0x7f9a94041440 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.757+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9aa4005200 con 0x7f9ab00a08c0 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.757+0000 7f9aa1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.757+0000 7f9aa1ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.757+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 >> v1:172.21.15.26:6789/0 conn(0x7f9ab00a08c0 legacy=0x7f9ab00a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.757+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 >> v1:172.21.15.26:6789/0 conn(0x7f9ab00a08c0 legacy=0x7f9ab00a0ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:40.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.763+0000 7f9ab787e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.763+0000 7f9ab787e640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9ab00a1790 con 0x7f9ab00a13b0 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9ab4e19640 1 -- 172.21.15.26:0/963616565 >> v1:172.21.15.26:6789/0 conn(0x7f9ab00a13b0 legacy=0x7f9ab01db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 10 client.154605.objecter ms_handle_connect 0x7f9ab00a13b0 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 10 client.154605.objecter resend_mon_ops 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3237819479 0 0) 0x7f9ab00a1790 con 0x7f9ab00a13b0 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 10 monclient(hunting): my global_id is 145197 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ab00008d0 con 0x7f9ab00a13b0 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 335762889 0 0) 0x7f9ab00008d0 con 0x7f9ab00a13b0 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ab00a1790 con 0x7f9ab00a13b0 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9aa40058e0 con 0x7f9ab00a13b0 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.765+0000 7f9aa1ffb640 10 monclient: dump: 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.781 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.782 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2102153324 0 0) 0x7f9ab00a1790 con 0x7f9ab00a13b0 2024-08-19T12:49:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9ab01df280 con 0x7f9ab00a13b0 2024-08-19T12:49:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7502 (unknown 0 0 1531115865) 0x7f9aa40086b0 con 0x7f9ab00a13b0 2024-08-19T12:49:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9aa1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9aa1ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 >> v1:172.21.15.26:6789/0 conn(0x7f9ab00a13b0 legacy=0x7f9ab01db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9aa1ffb640 1 -- 172.21.15.26:0/963616565 >> v1:172.21.15.26:6789/0 conn(0x7f9ab00a13b0 legacy=0x7f9ab01db9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:40.782 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:40.782 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 76, 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 432684, 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:40.783 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:40.784 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:40.785 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:40.786 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.786 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:40.786 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:40.786 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:40.786 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:40.786 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:40.786 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:40.786 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:40.786 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:40.786 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:40.786 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:40.787 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:40.788 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.789 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:40.792 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.794 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:40.794 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:40.794 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.794 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.794 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.794 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:40.794 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.794 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.794 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.795 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:40.798 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.799 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:40.800 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:40.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.801 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:40.801 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:40.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.801 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:40.801 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:40.801 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:49:40.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.801 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.801 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:40.801 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:40.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 10 librados: watch_flush enter 2024-08-19T12:49:40.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 10 librados: watch_flush exit 2024-08-19T12:49:40.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 20 client.154605.objecter shutdown clearing up homeless session... 2024-08-19T12:49:40.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 10 client.154605.objecter successfully canceled tick 2024-08-19T12:49:40.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 1 -- 172.21.15.26:0/963616565 >> v1:172.21.15.104:6800/899887506 conn(0x7f9a94041440 legacy=0x7f9a94043900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 10 monclient: shutdown 2024-08-19T12:49:40.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:40.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 1 -- 172.21.15.26:0/963616565 >> v1:172.21.15.104:6790/0 conn(0x7f9ab006e7b0 legacy=0x7f9ab01373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab5e1b640 1 -- 172.21.15.26:0/963616565 reap_dead start 2024-08-19T12:49:40.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab5e1b640 1 -- 172.21.15.26:0/963616565 reap_dead start 2024-08-19T12:49:40.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 1 -- 172.21.15.26:0/963616565 shutdown_connections 2024-08-19T12:49:40.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 1 -- 172.21.15.26:0/963616565 >> 172.21.15.26:0/963616565 conn(0x7f9ab00a8740 msgr2=0x7f9ab00a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:40.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 1 -- 172.21.15.26:0/963616565 shutdown_connections 2024-08-19T12:49:40.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 1 -- 172.21.15.26:0/963616565 wait complete. 2024-08-19T12:49:40.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.766+0000 7f9ab787e640 1 librados: shutdown 2024-08-19T12:49:40.804 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T12:49:40.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon b 2024-08-19T12:49:40.805 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:49:40.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.902+0000 7fd05318a640 10 monclient: get_monmap_and_config 2024-08-19T12:49:40.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.902+0000 7fd05318a640 10 monclient: build_initial_monmap 2024-08-19T12:49:40.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.902+0000 7fd05318a640 10 monclient: monmap: 2024-08-19T12:49:40.904 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:40.904630+0000 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:40.904630+0000 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:40.905 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:40.906 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.903+0000 7fd05318a640 1 Processor -- start 2024-08-19T12:49:40.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.903+0000 7fd05318a640 1 -- start start 2024-08-19T12:49:40.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.903+0000 7fd05318a640 10 monclient: init 2024-08-19T12:49:40.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd05318a640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:40.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd05318a640 10 monclient: _add_conns ranks=[3,1,5,6,0,2,4,8,7] 2024-08-19T12:49:40.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd05318a640 10 monclient(hunting): picked mon.noname-g con 0x7fd04c06e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:40.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd05318a640 10 monclient(hunting): picked mon.noname-f con 0x7fd04c0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:40.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd05318a640 10 monclient(hunting): picked mon.noname-h con 0x7fd04c0a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:40.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd05318a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd04c136790 con 0x7fd04c0a5920 2024-08-19T12:49:40.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd05318a640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd04c09ddf0 con 0x7fd04c06e7b0 2024-08-19T12:49:40.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd05318a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd04c09e0d0 con 0x7fd04c0a5d50 2024-08-19T12:49:40.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd05318a640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:40.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd05318a640 10 monclient(hunting): authenticate will time out at 1374.992201s 2024-08-19T12:49:40.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd04bfff640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7fd04c0a5920 legacy=0x7fd04c13e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:49:40.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd051727640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7fd04c0a5d50 legacy=0x7fd04c141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:49:40.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd050f26640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fd04c06e7b0 0x7fd04c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:53868/0 (socket says 172.21.15.26:53868) 2024-08-19T12:49:40.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.904+0000 7fd050f26640 1 -- 172.21.15.26:0/345906497 learned_addr learned my addr 172.21.15.26:0/345906497 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:40.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.906+0000 7fd050f26640 1 -- 172.21.15.26:0/345906497 >> v1:172.21.15.104:6790/0 conn(0x7fd04c06e7b0 legacy=0x7fd04c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:40.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.906+0000 7fd04b7fe640 1 -- 172.21.15.26:0/345906497 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4015106507 0 0) 0x7fd03c0017d0 con 0x7fd04c06e7b0 2024-08-19T12:49:40.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.906+0000 7fd04b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.906+0000 7fd04b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.906+0000 7fd04b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.906+0000 7fd04b7fe640 10 monclient(hunting): my global_id is 154614 2024-08-19T12:49:40.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.906+0000 7fd04b7fe640 1 -- 172.21.15.26:0/345906497 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd038003520 con 0x7fd04c06e7b0 2024-08-19T12:49:40.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 1 -- 172.21.15.26:0/345906497 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3419125000 0 0) 0x7fd038003520 con 0x7fd04c06e7b0 2024-08-19T12:49:40.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 1 -- 172.21.15.26:0/345906497 >> v1:172.21.15.104:6791/0 conn(0x7fd04c0a5d50 legacy=0x7fd04c141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:49:40.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 1 -- 172.21.15.26:0/345906497 >> v1:172.21.15.104:6789/0 conn(0x7fd04c0a5920 legacy=0x7fd04c13e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:49:40.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:49:40.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 1 monclient: found mon.noname-g 2024-08-19T12:49:40.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:40.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:49:40.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 1 -- 172.21.15.26:0/345906497 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04c09e3b0 con 0x7fd04c06e7b0 2024-08-19T12:49:40.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 10 monclient: _finish_auth 0 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd05318a640 5 monclient: authenticate success, global_id 154614 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 1 -- 172.21.15.26:0/345906497 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd03c002030 con 0x7fd04c06e7b0 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd05318a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 10 monclient: dump: 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.913 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.914 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.914 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.914 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.914 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.914 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.914 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.914 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.914 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.914 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd05318a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:40.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 1 -- 172.21.15.26:0/345906497 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd03c004d10 con 0x7fd04c06e7b0 2024-08-19T12:49:40.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd05318a640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 1 -- 172.21.15.26:0/345906497 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd03c005240 con 0x7fd04c06e7b0 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd05318a640 4 set_mon_vals no callback set 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd04b7fe640 10 monclient: dump: 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.915 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd05318a640 10 monclient: shutdown 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd05318a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd05318a640 1 -- 172.21.15.26:0/345906497 >> v1:172.21.15.104:6790/0 conn(0x7fd04c06e7b0 legacy=0x7fd04c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd05318a640 1 -- 172.21.15.26:0/345906497 shutdown_connections 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd05318a640 1 -- 172.21.15.26:0/345906497 >> 172.21.15.26:0/345906497 conn(0x7fd04c0a8740 msgr2=0x7fd04c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd05318a640 1 -- 172.21.15.26:0/345906497 shutdown_connections 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.907+0000 7fd05318a640 1 -- 172.21.15.26:0/345906497 wait complete. 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.908+0000 7fd05318a640 10 monclient: build_initial_monmap 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.908+0000 7fd05318a640 10 monclient: monmap: 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:40.917 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.908+0000 7fd05318a640 1 librados: starting msgr at 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.908+0000 7fd05318a640 1 librados: starting objecter 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.908+0000 7fd05318a640 1 Processor -- start 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.908+0000 7fd05318a640 1 -- start start 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.908+0000 7fd05318a640 1 librados: setting wanted keys 2024-08-19T12:49:40.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.908+0000 7fd05318a640 1 librados: calling monclient init 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.908+0000 7fd05318a640 10 monclient: init 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd05318a640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd05318a640 10 monclient: _add_conns ranks=[5,6,7,8,3,2,0,4,1] 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd05318a640 10 monclient(hunting): picked mon.noname-h con 0x7fd04c06e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd05318a640 10 monclient(hunting): picked mon.noname-d con 0x7fd04c0a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd05318a640 10 monclient(hunting): picked mon.noname-i con 0x7fd04c0a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd05318a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd04c134680 con 0x7fd04c06e7b0 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd05318a640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd04c13b1a0 con 0x7fd04c0a5920 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd05318a640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd04c13b480 con 0x7fd04c0a5d50 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd05318a640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd05318a640 10 monclient(hunting): authenticate will time out at 1374.997430s 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd050f26640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7fd04c06e7b0 legacy=0x7fd04c137b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd04bfff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fd04c0a5920 0x7fd04c138230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:46774/0 (socket says 172.21.15.26:46774) 2024-08-19T12:49:40.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.909+0000 7fd04bfff640 1 -- 172.21.15.26:0/2919364378 learned_addr learned my addr 172.21.15.26:0/2919364378 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:40.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.911+0000 7fd04bfff640 1 -- 172.21.15.26:0/2919364378 >> v1:172.21.15.26:6792/0 conn(0x7fd04c0a5920 legacy=0x7fd04c138230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.911+0000 7fd0497fa640 10 client.?.objecter ms_handle_connect 0x7fd04c0a5920 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.911+0000 7fd0497fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.911+0000 7fd0497fa640 10 client.?.objecter ms_handle_connect 0x7fd04c0a5d50 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.911+0000 7fd051727640 1 -- 172.21.15.26:0/2919364378 >> v1:172.21.15.104:6792/0 conn(0x7fd04c0a5d50 legacy=0x7fd04c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.911+0000 7fd0497fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3667903539 0 0) 0x7fd04c13b1a0 con 0x7fd04c0a5920 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): my global_id is 164472 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd02c004eb0 con 0x7fd04c0a5920 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3453035394 0 0) 0x7fd04c13b480 con 0x7fd04c0a5d50 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): my global_id is 135781 2024-08-19T12:49:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd04c13b1a0 con 0x7fd04c0a5d50 2024-08-19T12:49:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2202336757 0 0) 0x7fd02c004eb0 con 0x7fd04c0a5920 2024-08-19T12:49:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd04c13b480 con 0x7fd04c0a5920 2024-08-19T12:49:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd034004730 con 0x7fd04c0a5920 2024-08-19T12:49:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): dump: 2024-08-19T12:49:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.926 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3460404115 0 0) 0x7fd04c13b1a0 con 0x7fd04c0a5d50 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd02c004eb0 con 0x7fd04c0a5d50 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd0440044c0 con 0x7fd04c0a5d50 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient(hunting): dump: 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.928 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2164464294 0 0) 0x7fd04c13b480 con 0x7fd04c0a5920 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 >> v1:172.21.15.104:6792/0 conn(0x7fd04c0a5d50 legacy=0x7fd04c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 >> v1:172.21.15.104:6791/0 conn(0x7fd04c06e7b0 legacy=0x7fd04c137b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient: _finish_hunting 0 2024-08-19T12:49:40.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 monclient: found mon.d 2024-08-19T12:49:40.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:40.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:40.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04c139c80 con 0x7fd04c0a5920 2024-08-19T12:49:40.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient: _finish_auth 0 2024-08-19T12:49:40.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient: _check_auth_tickets 2024-08-19T12:49:40.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:40.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd05318a640 5 monclient: authenticate success, global_id 164472 2024-08-19T12:49:40.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd05318a640 10 monclient: _renew_subs 2024-08-19T12:49:40.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd05318a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:40.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd05318a640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd04c139120 con 0x7fd04c0a5920 2024-08-19T12:49:40.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd05318a640 10 client.164472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:40.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd05318a640 10 monclient: _renew_subs 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd05318a640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd05318a640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fd04c142300 con 0x7fd04c0a5920 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd05318a640 1 librados: init done 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd034004950 con 0x7fd04c0a5920 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd034004e80 con 0x7fd04c0a5920 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd04affd640 4 set_mon_vals no callback set 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.912+0000 7fd0497fa640 10 monclient: dump: 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.931 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.913+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd034005150 con 0x7fd04c0a5920 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.913+0000 7fd05318a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:40.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.913+0000 7fd05318a640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd0300065c0 con 0x7fd030002860 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.913+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(409..409 src has 311..409) ==== 5333+0+0 (unknown 2618201168 0 0) 0x7fd03404e320 con 0x7fd04c0a5920 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.913+0000 7fd0497fa640 10 client.164472.objecter ms_dispatch 0x7fd04c000b90 osd_map(409..409 src has 311..409) 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.913+0000 7fd0497fa640 3 client.164472.objecter handle_osd_map got epochs [409,409] > 0 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.913+0000 7fd0497fa640 3 client.164472.objecter handle_osd_map decoding full epoch 409 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.913+0000 7fd0497fa640 20 client.164472.objecter dump_active .. 0 homeless 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.913+0000 7fd051727640 10 monclient: get_auth_request con 0x7fd02c0413a0 auth_method 0 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.915+0000 7fd050f26640 1 -- 172.21.15.26:0/2919364378 >> v1:172.21.15.26:6790/0 conn(0x7fd030002860 legacy=0x7fd030004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.915+0000 7fd0497fa640 10 client.164472.objecter ms_handle_connect 0x7fd030002860 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.915+0000 7fd0497fa640 10 client.164472.objecter resend_mon_ops 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.915+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1626630669 0 0) 0x7fd0300065c0 con 0x7fd030002860 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 10 monclient(hunting): my global_id is 26831 2024-08-19T12:49:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd04c13b480 con 0x7fd030002860 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4116620677 0 0) 0x7fd04c13b480 con 0x7fd030002860 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0300065c0 con 0x7fd030002860 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd03c001e50 con 0x7fd030002860 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 10 monclient: dump: 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.934 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 10 client.164472.objecter ms_handle_connect 0x7fd02c0413a0 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3258357364 0 0) 0x7fd0300065c0 con 0x7fd030002860 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd03000a160 con 0x7fd030002860 2024-08-19T12:49:40.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd03c001e50 con 0x7fd030002860 2024-08-19T12:49:40.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:40.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:40.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 >> v1:172.21.15.26:6790/0 conn(0x7fd030002860 legacy=0x7fd030004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.916+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 >> v1:172.21.15.26:6790/0 conn(0x7fd030002860 legacy=0x7fd030004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:40.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.922+0000 7fd05318a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:40.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.922+0000 7fd05318a640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd030009e50 con 0x7fd030009110 2024-08-19T12:49:40.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd050f26640 1 -- 172.21.15.26:0/2919364378 >> v1:172.21.15.26:6790/0 conn(0x7fd030009110 legacy=0x7fd0300095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:40.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 10 client.164472.objecter ms_handle_connect 0x7fd030009110 2024-08-19T12:49:40.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 10 client.164472.objecter resend_mon_ops 2024-08-19T12:49:40.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4272224494 0 0) 0x7fd030009e50 con 0x7fd030009110 2024-08-19T12:49:40.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 10 monclient(hunting): _negotiate 2024-08-19T12:49:40.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:40.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:40.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 10 monclient(hunting): my global_id is 26840 2024-08-19T12:49:40.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0300065c0 con 0x7fd030009110 2024-08-19T12:49:40.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1970681503 0 0) 0x7fd0300065c0 con 0x7fd030009110 2024-08-19T12:49:40.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd030009e50 con 0x7fd030009110 2024-08-19T12:49:40.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd03c005580 con 0x7fd030009110 2024-08-19T12:49:40.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 10 monclient: dump: 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:40.939 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2160648181 0 0) 0x7fd030009e50 con 0x7fd030009110 2024-08-19T12:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd03000c3f0 con 0x7fd030009110 2024-08-19T12:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7667 (unknown 0 0 206821014) 0x7fd03c0056b0 con 0x7fd030009110 2024-08-19T12:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 >> v1:172.21.15.26:6790/0 conn(0x7fd030009110 legacy=0x7fd0300095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.925+0000 7fd0497fa640 1 -- 172.21.15.26:0/2919364378 >> v1:172.21.15.26:6790/0 conn(0x7fd030009110 legacy=0x7fd0300095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:40.941 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:40.941 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:40.941 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:40.941 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:40.941 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 76, 2024-08-19T12:49:40.941 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 693980, 2024-08-19T12:49:40.941 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:40.941 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:40.941 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:40.941 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:40.941 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:40.941 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:40.942 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:40.943 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:40.944 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:40.945 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:40.947 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:40.949 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:40.950 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:40.953 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.955 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:40.956 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:40.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:40.958 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:40.959 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:40.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 10 librados: watch_flush enter 2024-08-19T12:49:40.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 10 librados: watch_flush exit 2024-08-19T12:49:40.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 20 client.164472.objecter shutdown clearing up homeless session... 2024-08-19T12:49:40.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 10 client.164472.objecter successfully canceled tick 2024-08-19T12:49:40.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 1 -- 172.21.15.26:0/2919364378 >> v1:172.21.15.104:6800/899887506 conn(0x7fd02c0413a0 legacy=0x7fd02c043860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 10 monclient: shutdown 2024-08-19T12:49:40.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:40.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 1 -- 172.21.15.26:0/2919364378 >> v1:172.21.15.26:6792/0 conn(0x7fd04c0a5920 legacy=0x7fd04c138230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:40.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd051727640 1 -- 172.21.15.26:0/2919364378 reap_dead start 2024-08-19T12:49:40.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd051727640 1 -- 172.21.15.26:0/2919364378 reap_dead start 2024-08-19T12:49:40.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 1 -- 172.21.15.26:0/2919364378 shutdown_connections 2024-08-19T12:49:40.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 1 -- 172.21.15.26:0/2919364378 >> 172.21.15.26:0/2919364378 conn(0x7fd04c0a8740 msgr2=0x7fd04c0aa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:40.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 1 -- 172.21.15.26:0/2919364378 shutdown_connections 2024-08-19T12:49:40.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 1 -- 172.21.15.26:0/2919364378 wait complete. 2024-08-19T12:49:40.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:40.926+0000 7fd05318a640 1 librados: shutdown 2024-08-19T12:49:40.962 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T12:49:40.962 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon d 2024-08-19T12:49:40.963 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:49:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.060+0000 7f2b80308640 10 monclient: get_monmap_and_config 2024-08-19T12:49:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.060+0000 7f2b80308640 10 monclient: build_initial_monmap 2024-08-19T12:49:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.060+0000 7f2b80308640 10 monclient: monmap: 2024-08-19T12:49:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:41.062 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:41.062472+0000 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:41.062472+0000 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:41.064 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:41.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 1 Processor -- start 2024-08-19T12:49:41.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 1 -- start start 2024-08-19T12:49:41.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 10 monclient: init 2024-08-19T12:49:41.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:41.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 10 monclient: _add_conns ranks=[6,7,8,1,4,5,3,2,0] 2024-08-19T12:49:41.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 10 monclient(hunting): picked mon.noname-d con 0x7f2b7806e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:41.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 10 monclient(hunting): picked mon.noname-i con 0x7f2b780a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:41.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 10 monclient(hunting): picked mon.noname-e con 0x7f2b780a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:41.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b78136790 con 0x7f2b7806e7b0 2024-08-19T12:49:41.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b7809ddf0 con 0x7f2b780a5920 2024-08-19T12:49:41.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b7809e0d0 con 0x7f2b780a5d50 2024-08-19T12:49:41.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:41.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b80308640 10 monclient(hunting): authenticate will time out at 1375.149891s 2024-08-19T12:49:41.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b7e8a5640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2b780a5d50 legacy=0x7f2b78141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:49:41.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.061+0000 7f2b7e0a4640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f2b7806e7b0 0x7f2b78136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:46790/0 (socket says 172.21.15.26:46790) 2024-08-19T12:49:41.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.062+0000 7f2b7e0a4640 1 -- 172.21.15.26:0/1553316062 learned_addr learned my addr 172.21.15.26:0/1553316062 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:41.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7e0a4640 1 -- 172.21.15.26:0/1553316062 >> v1:172.21.15.26:6792/0 conn(0x7f2b7806e7b0 legacy=0x7f2b78136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:41.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 1 -- 172.21.15.26:0/1553316062 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1831721366 0 0) 0x7f2b78136790 con 0x7f2b7806e7b0 2024-08-19T12:49:41.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient(hunting): _negotiate 2024-08-19T12:49:41.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:41.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:41.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient(hunting): my global_id is 164481 2024-08-19T12:49:41.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 1 -- 172.21.15.26:0/1553316062 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b64003520 con 0x7f2b7806e7b0 2024-08-19T12:49:41.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d8a3640 1 -- 172.21.15.26:0/1553316062 >> v1:172.21.15.104:6792/0 conn(0x7f2b780a5920 legacy=0x7f2b7813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:41.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 1 -- 172.21.15.26:0/1553316062 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3703064735 0 0) 0x7f2b64003520 con 0x7f2b7806e7b0 2024-08-19T12:49:41.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 1 -- 172.21.15.26:0/1553316062 >> v1:172.21.15.26:6793/0 conn(0x7f2b780a5d50 legacy=0x7f2b78141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:49:41.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 1 -- 172.21.15.26:0/1553316062 >> v1:172.21.15.104:6792/0 conn(0x7f2b780a5920 legacy=0x7f2b7813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:41.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient: _finish_hunting 0 2024-08-19T12:49:41.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 1 monclient: found mon.noname-d 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 1 -- 172.21.15.26:0/1553316062 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7809e3b0 con 0x7f2b7806e7b0 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient: _finish_auth 0 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient: _check_auth_tickets 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 1 -- 172.21.15.26:0/1553316062 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2b74001ae0 con 0x7f2b7806e7b0 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b80308640 5 monclient: authenticate success, global_id 164481 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b80308640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient: dump: 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:41.071 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:41.072 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b80308640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 1 -- 172.21.15.26:0/1553316062 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b740048a0 con 0x7f2b7806e7b0 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 1 -- 172.21.15.26:0/1553316062 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2b74004e10 con 0x7f2b7806e7b0 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b80308640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b80308640 4 set_mon_vals no callback set 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.064+0000 7f2b7d0a2640 10 monclient: dump: 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:41.073 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.065+0000 7f2b80308640 10 monclient: shutdown 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.065+0000 7f2b80308640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:41.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.065+0000 7f2b80308640 1 -- 172.21.15.26:0/1553316062 >> v1:172.21.15.26:6792/0 conn(0x7f2b7806e7b0 legacy=0x7f2b78136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.065+0000 7f2b80308640 1 -- 172.21.15.26:0/1553316062 shutdown_connections 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.065+0000 7f2b80308640 1 -- 172.21.15.26:0/1553316062 >> 172.21.15.26:0/1553316062 conn(0x7f2b780a8740 msgr2=0x7f2b780a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.065+0000 7f2b80308640 1 -- 172.21.15.26:0/1553316062 shutdown_connections 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.065+0000 7f2b80308640 1 -- 172.21.15.26:0/1553316062 wait complete. 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.065+0000 7f2b80308640 10 monclient: build_initial_monmap 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.065+0000 7f2b80308640 10 monclient: monmap: 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:41.075 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.065+0000 7f2b80308640 1 librados: starting msgr at 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.065+0000 7f2b80308640 1 librados: starting objecter 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 1 Processor -- start 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 1 -- start start 2024-08-19T12:49:41.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 1 librados: setting wanted keys 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 1 librados: calling monclient init 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 10 monclient: init 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 10 monclient: _add_conns ranks=[8,4,1,0,6,2,5,7,3] 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 10 monclient(hunting): picked mon.noname-e con 0x7f2b7806e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 10 monclient(hunting): picked mon.noname-c con 0x7f2b780a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 10 monclient(hunting): picked mon.noname-f con 0x7f2b780a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b7813b410 con 0x7f2b780a5d50 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b78142ec0 con 0x7f2b780a5920 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b7e0a4640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2b7806e7b0 legacy=0x7f2b781ce4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b781390b0 con 0x7f2b7806e7b0 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:41.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b80308640 10 monclient(hunting): authenticate will time out at 1375.154563s 2024-08-19T12:49:41.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b7d8a3640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2b780a5920 legacy=0x7f2b781d3480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:49:41.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.066+0000 7f2b7e8a5640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2b780a5d50 legacy=0x7f2b781d6bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:49:41.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.267+0000 7f2b7e0a4640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2b7806e7b0 legacy=0x7f2b781ce4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:49:41.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.267+0000 7f2b7d8a3640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2b780a5920 legacy=0x7f2b781d3480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:49:41.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.268+0000 7f2b7e8a5640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2b780a5d50 legacy=0x7f2b781d6bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:49:41.504 INFO:tasks.workunit.client.0.smithi104.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (19583 ms) 2024-08-19T12:49:41.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-08-19T12:49:41.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (44070 ms) 2024-08-19T12:49:41.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-08-19T12:49:41.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3602 ms) 2024-08-19T12:49:41.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-08-19T12:49:41.504 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3041 ms) 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (2355 ms) 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3720 ms) 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5083 ms) 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39668 ms) 2024-08-19T12:49:41.505 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-08-19T12:49:41.506 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.506 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16556 ms) 2024-08-19T12:49:41.506 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-08-19T12:49:41.506 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.506 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (6036 ms) 2024-08-19T12:49:41.506 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-08-19T12:49:41.506 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.506 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3020 ms) 2024-08-19T12:49:41.506 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-08-19T12:49:41.506 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.506 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (45011 ms) 2024-08-19T12:49:41.506 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15148 ms) 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (4783 ms) 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4123 ms) 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: waiting for scrubs... 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: done waiting 2024-08-19T12:49:41.507 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25538 ms) 2024-08-19T12:49:41.508 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-08-19T12:49:41.508 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.508 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (11053 ms) 2024-08-19T12:49:41.508 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-08-19T12:49:41.508 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.508 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (16265 ms) 2024-08-19T12:49:41.508 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-08-19T12:49:41.508 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:49:41.508 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9086 ms) 2024-08-19T12:49:41.508 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-08-19T12:49:41.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.668+0000 7f2b7e0a4640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2b7806e7b0 legacy=0x7f2b781ce4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:49:41.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.668+0000 7f2b7d8a3640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2b780a5920 legacy=0x7f2b781d3480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:49:41.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:41.669+0000 7f2b7e8a5640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2b780a5d50 legacy=0x7f2b781d6bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:49:42.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:42.470+0000 7f2b7e0a4640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2b7806e7b0 legacy=0x7f2b781ce4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:49:42.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:42.470+0000 7f2b7d8a3640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2b780a5920 legacy=0x7f2b781d3480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:49:42.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:42.470+0000 7f2b7e8a5640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2b780a5d50 legacy=0x7f2b781d6bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:49:44.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 10 monclient(hunting): tick 2024-08-19T12:49:44.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:49:44.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 1 monclient(hunting): continuing hunt 2024-08-19T12:49:44.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:49:44.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2b7806e7b0 legacy=0x7f2b781ce4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:49:44.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2b780a5920 legacy=0x7f2b781d3480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:49:44.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2b780a5d50 legacy=0x7f2b781d6bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:49:44.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 10 monclient: _add_conns ranks=[6,2,7,3,5,0] 2024-08-19T12:49:44.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 10 monclient(hunting): picked mon.noname-d con 0x7f2b60003780 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:44.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 10 monclient(hunting): picked mon.noname-b con 0x7f2b60007410 addr v1:172.21.15.26:6790/0 2024-08-19T12:49:44.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 10 monclient(hunting): picked mon.noname-i con 0x7f2b6000b1a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:44.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b7813ab70 con 0x7f2b60007410 2024-08-19T12:49:44.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b7813b410 con 0x7f2b60003780 2024-08-19T12:49:44.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b78142ec0 con 0x7f2b6000b1a0 2024-08-19T12:49:44.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b6dffb640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:44.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b7e0a4640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f2b60003780 0x7f2b60005bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:46806/0 (socket says 172.21.15.26:46806) 2024-08-19T12:49:44.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.067+0000 7f2b7e0a4640 1 -- 172.21.15.26:0/496958313 learned_addr learned my addr 172.21.15.26:0/496958313 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:44.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b7e0a4640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.26:6792/0 conn(0x7f2b60003780 legacy=0x7f2b60005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 client.?.objecter ms_handle_connect 0x7f2b60003780 2024-08-19T12:49:44.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b7d8a3640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.26:6790/0 conn(0x7f2b60007410 legacy=0x7f2b600098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:44.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 client.?.objecter ms_handle_connect 0x7f2b60007410 2024-08-19T12:49:44.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 client.?.objecter ms_handle_connect 0x7f2b6000b1a0 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b7e8a5640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.104:6792/0 conn(0x7f2b6000b1a0 legacy=0x7f2b6000d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1120021232 0 0) 0x7f2b7813b410 con 0x7f2b60003780 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): my global_id is 164490 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b5c004eb0 con 0x7f2b60003780 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 134772587 0 0) 0x7f2b7813ab70 con 0x7f2b60007410 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): my global_id is 26849 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b7813b410 con 0x7f2b60007410 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 918595249 0 0) 0x7f2b78142ec0 con 0x7f2b6000b1a0 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): my global_id is 135799 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b7813ab70 con 0x7f2b6000b1a0 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2640075996 0 0) 0x7f2b5c004eb0 con 0x7f2b60003780 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b78142ec0 con 0x7f2b60003780 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2b74004d90 con 0x7f2b60003780 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): dump: 2024-08-19T12:49:44.076 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.077 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.077 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.077 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.077 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.077 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.077 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.077 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.077 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.077 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.077 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.077 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.077 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3837614877 0 0) 0x7f2b7813b410 con 0x7f2b60007410 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b5c004eb0 con 0x7f2b60007410 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1211622864 0 0) 0x7f2b7813ab70 con 0x7f2b6000b1a0 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b7813b410 con 0x7f2b6000b1a0 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2b68001cd0 con 0x7f2b60007410 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): dump: 2024-08-19T12:49:44.078 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.079 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2b700049c0 con 0x7f2b6000b1a0 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 10 monclient(hunting): dump: 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.080 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.070+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1383938451 0 0) 0x7f2b78142ec0 con 0x7f2b60003780 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.104:6792/0 conn(0x7f2b6000b1a0 legacy=0x7f2b6000d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.26:6790/0 conn(0x7f2b60007410 legacy=0x7f2b600098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 10 monclient: _finish_hunting 0 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 1 monclient: found mon.d 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b7e8a5640 1 -- 172.21.15.26:0/496958313 reap_dead start 2024-08-19T12:49:44.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:44.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:44.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b781390b0 con 0x7f2b60003780 2024-08-19T12:49:44.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 10 monclient: _finish_auth 0 2024-08-19T12:49:44.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 10 monclient: _check_auth_tickets 2024-08-19T12:49:44.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:44.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b80308640 5 monclient: authenticate success, global_id 164490 2024-08-19T12:49:44.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b80308640 10 monclient: _renew_subs 2024-08-19T12:49:44.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b80308640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:44.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b80308640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2b7813ae50 con 0x7f2b60003780 2024-08-19T12:49:44.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b80308640 10 client.164490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:44.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b80308640 10 monclient: _renew_subs 2024-08-19T12:49:44.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b80308640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b80308640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f2b7813a7b0 con 0x7f2b60003780 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b80308640 1 librados: init done 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b74001320 con 0x7f2b60003780 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2b74001a80 con 0x7f2b60003780 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b7c8a1640 4 set_mon_vals no callback set 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.071+0000 7f2b6effd640 10 monclient: dump: 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.072+0000 7f2b80308640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:44.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.072+0000 7f2b80308640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2b540065c0 con 0x7f2b54002860 2024-08-19T12:49:44.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.072+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2b74011a00 con 0x7f2b60003780 2024-08-19T12:49:44.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.072+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(412..412 src has 311..412) ==== 5745+0+0 (unknown 570832459 0 0) 0x7f2b7404e460 con 0x7f2b60003780 2024-08-19T12:49:44.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.072+0000 7f2b6effd640 10 client.164490.objecter ms_dispatch 0x7f2b78000b90 osd_map(412..412 src has 311..412) 2024-08-19T12:49:44.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.072+0000 7f2b6effd640 3 client.164490.objecter handle_osd_map got epochs [412,412] > 0 2024-08-19T12:49:44.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.072+0000 7f2b6effd640 3 client.164490.objecter handle_osd_map decoding full epoch 412 2024-08-19T12:49:44.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.073+0000 7f2b7e8a5640 10 monclient: get_auth_request con 0x7f2b5c041650 auth_method 0 2024-08-19T12:49:44.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.073+0000 7f2b6effd640 20 client.164490.objecter dump_active .. 0 homeless 2024-08-19T12:49:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.074+0000 7f2b7d8a3640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.26:6792/0 conn(0x7f2b54002860 legacy=0x7f2b54004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.074+0000 7f2b6effd640 10 client.164490.objecter ms_handle_connect 0x7f2b54002860 2024-08-19T12:49:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.074+0000 7f2b6effd640 10 client.164490.objecter resend_mon_ops 2024-08-19T12:49:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2749930630 0 0) 0x7f2b540065c0 con 0x7f2b54002860 2024-08-19T12:49:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 10 monclient(hunting): my global_id is 164499 2024-08-19T12:49:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b78142ec0 con 0x7f2b54002860 2024-08-19T12:49:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2369960404 0 0) 0x7f2b78142ec0 con 0x7f2b54002860 2024-08-19T12:49:44.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b540065c0 con 0x7f2b54002860 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2b68004010 con 0x7f2b54002860 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 10 monclient: dump: 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.089 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 10 client.164490.objecter ms_handle_connect 0x7f2b5c041650 2024-08-19T12:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2812072353 0 0) 0x7f2b540065c0 con 0x7f2b54002860 2024-08-19T12:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b5400a160 con 0x7f2b54002860 2024-08-19T12:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2b680040a0 con 0x7f2b54002860 2024-08-19T12:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:44.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.26:6792/0 conn(0x7f2b54002860 legacy=0x7f2b54004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.075+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.26:6792/0 conn(0x7f2b54002860 legacy=0x7f2b54004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.082+0000 7f2b80308640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.082+0000 7f2b80308640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2b54009e70 con 0x7f2b54009130 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.085+0000 7f2b7d8a3640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.26:6792/0 conn(0x7f2b54009130 legacy=0x7f2b540095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.085+0000 7f2b6effd640 10 client.164490.objecter ms_handle_connect 0x7f2b54009130 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.085+0000 7f2b6effd640 10 client.164490.objecter resend_mon_ops 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.085+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296960092 0 0) 0x7f2b54009e70 con 0x7f2b54009130 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.085+0000 7f2b6effd640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.085+0000 7f2b6effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.085+0000 7f2b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.085+0000 7f2b6effd640 10 monclient(hunting): my global_id is 164508 2024-08-19T12:49:44.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.085+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b540065c0 con 0x7f2b54009130 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1500201968 0 0) 0x7f2b540065c0 con 0x7f2b54009130 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b54009e70 con 0x7f2b54009130 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2b68005cf0 con 0x7f2b54009130 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 10 monclient: dump: 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.092 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2308501768 0 0) 0x7f2b54009e70 con 0x7f2b54009130 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2b5400c410 con 0x7f2b54009130 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7667 (unknown 0 0 2203543664) 0x7f2b680071e0 con 0x7f2b54009130 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:44.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.26:6792/0 conn(0x7f2b54009130 legacy=0x7f2b540095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.086+0000 7f2b6effd640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.26:6792/0 conn(0x7f2b54009130 legacy=0x7f2b540095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:44.094 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:44.094 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:44.094 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:44.094 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:44.094 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 76, 2024-08-19T12:49:44.094 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 167641, 2024-08-19T12:49:44.094 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:44.094 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:44.095 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:44.095 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:44.095 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:44.095 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:49:44.095 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.095 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:49:44.095 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:44.095 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:44.095 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:44.095 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:44.095 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:44.095 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:44.096 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:44.097 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:44.098 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:44.099 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:44.100 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:44.101 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.102 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:44.103 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.106 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:44.106 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.106 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:44.106 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:44.106 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.106 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.106 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.108 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.109 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:44.110 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.110 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:44.110 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:44.110 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.110 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:44.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.114 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:44.114 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:44.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 10 librados: watch_flush enter 2024-08-19T12:49:44.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 10 librados: watch_flush exit 2024-08-19T12:49:44.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 20 client.164490.objecter shutdown clearing up homeless session... 2024-08-19T12:49:44.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 10 client.164490.objecter successfully canceled tick 2024-08-19T12:49:44.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.104:6800/899887506 conn(0x7f2b5c041650 legacy=0x7f2b5c043b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 10 monclient: shutdown 2024-08-19T12:49:44.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:44.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 1 -- 172.21.15.26:0/496958313 >> v1:172.21.15.26:6792/0 conn(0x7f2b60003780 legacy=0x7f2b60005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 1 -- 172.21.15.26:0/496958313 shutdown_connections 2024-08-19T12:49:44.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 1 -- 172.21.15.26:0/496958313 >> 172.21.15.26:0/496958313 conn(0x7f2b780a8740 msgr2=0x7f2b780aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:44.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 1 -- 172.21.15.26:0/496958313 shutdown_connections 2024-08-19T12:49:44.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 1 -- 172.21.15.26:0/496958313 wait complete. 2024-08-19T12:49:44.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.087+0000 7f2b80308640 1 librados: shutdown 2024-08-19T12:49:44.116 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T12:49:44.117 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon g 2024-08-19T12:49:44.117 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:49:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.224+0000 7f46bd2b3640 10 monclient: get_monmap_and_config 2024-08-19T12:49:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.224+0000 7f46bd2b3640 10 monclient: build_initial_monmap 2024-08-19T12:49:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.224+0000 7f46bd2b3640 10 monclient: monmap: 2024-08-19T12:49:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:44.225410+0000 2024-08-19T12:49:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:44.225410+0000 2024-08-19T12:49:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:44.225 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:44.226 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 1 Processor -- start 2024-08-19T12:49:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 1 -- start start 2024-08-19T12:49:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 10 monclient: init 2024-08-19T12:49:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 10 monclient: _add_conns ranks=[5,3,7,1,2,4,0,6,8] 2024-08-19T12:49:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 10 monclient(hunting): picked mon.noname-h con 0x7f46b806e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 10 monclient(hunting): picked mon.noname-g con 0x7f46b80a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 10 monclient(hunting): picked mon.noname-i con 0x7f46b80a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:49:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b8136790 con 0x7f46b80a5920 2024-08-19T12:49:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b809ddf0 con 0x7f46b806e7b0 2024-08-19T12:49:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b809e0d0 con 0x7f46b80a5d50 2024-08-19T12:49:44.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46bd2b3640 10 monclient(hunting): authenticate will time out at 1378.312860s 2024-08-19T12:49:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.225+0000 7f46b6ffd640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7f46b806e7b0 legacy=0x7f46b8136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:49:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.226+0000 7f46ae7fc640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f46b80a5920 0x7f46b813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:53880/0 (socket says 172.21.15.26:53880) 2024-08-19T12:49:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.226+0000 7f46ae7fc640 1 -- 172.21.15.26:0/905566570 learned_addr learned my addr 172.21.15.26:0/905566570 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:44.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.226+0000 7f46b77fe640 1 --1- 172.21.15.26:0/905566570 >> v1:172.21.15.104:6792/0 conn(0x7f46b80a5d50 0x7f46b8141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:35108/0 (socket says 172.21.15.26:35108) 2024-08-19T12:49:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46ae7fc640 1 -- 172.21.15.26:0/905566570 >> v1:172.21.15.104:6790/0 conn(0x7f46b80a5920 legacy=0x7f46b813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b77fe640 1 -- 172.21.15.26:0/905566570 >> v1:172.21.15.104:6792/0 conn(0x7f46b80a5d50 legacy=0x7f46b8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 1 -- 172.21.15.26:0/905566570 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3883922342 0 0) 0x7f46b809e0d0 con 0x7f46b80a5d50 2024-08-19T12:49:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient(hunting): my global_id is 135808 2024-08-19T12:49:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 1 -- 172.21.15.26:0/905566570 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46a4003520 con 0x7f46b80a5d50 2024-08-19T12:49:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 1 -- 172.21.15.26:0/905566570 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3364579760 0 0) 0x7f46b8136790 con 0x7f46b80a5920 2024-08-19T12:49:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient(hunting): my global_id is 154623 2024-08-19T12:49:44.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 1 -- 172.21.15.26:0/905566570 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46b809e0d0 con 0x7f46b80a5920 2024-08-19T12:49:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 1 -- 172.21.15.26:0/905566570 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 514273698 0 0) 0x7f46a4003520 con 0x7f46b80a5d50 2024-08-19T12:49:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 1 -- 172.21.15.26:0/905566570 >> v1:172.21.15.104:6791/0 conn(0x7f46b806e7b0 legacy=0x7f46b8136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:49:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 1 -- 172.21.15.26:0/905566570 >> v1:172.21.15.104:6790/0 conn(0x7f46b80a5920 legacy=0x7f46b813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient: _finish_hunting 0 2024-08-19T12:49:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 1 monclient: found mon.noname-i 2024-08-19T12:49:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:49:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 1 -- 172.21.15.26:0/905566570 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b809e3b0 con 0x7f46b80a5d50 2024-08-19T12:49:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient: _finish_auth 0 2024-08-19T12:49:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient: _check_auth_tickets 2024-08-19T12:49:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:44.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 1 -- 172.21.15.26:0/905566570 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f46b0001d90 con 0x7f46b80a5d50 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46bd2b3640 5 monclient: authenticate success, global_id 135808 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46bd2b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46b67fc640 10 monclient: dump: 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.235 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.228+0000 7f46bd2b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46b67fc640 1 -- 172.21.15.26:0/905566570 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f46b0002060 con 0x7f46b80a5d50 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46b67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46bd2b3640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46bd2b3640 4 set_mon_vals no callback set 2024-08-19T12:49:44.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46b67fc640 1 -- 172.21.15.26:0/905566570 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f46b0004ee0 con 0x7f46b80a5d50 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46b67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46b67fc640 10 monclient: dump: 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.237 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46bd2b3640 10 monclient: shutdown 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46bd2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46bd2b3640 1 -- 172.21.15.26:0/905566570 >> v1:172.21.15.104:6792/0 conn(0x7f46b80a5d50 legacy=0x7f46b8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46bd2b3640 1 -- 172.21.15.26:0/905566570 shutdown_connections 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46bd2b3640 1 -- 172.21.15.26:0/905566570 >> 172.21.15.26:0/905566570 conn(0x7f46b80a8740 msgr2=0x7f46b80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46bd2b3640 1 -- 172.21.15.26:0/905566570 shutdown_connections 2024-08-19T12:49:44.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.229+0000 7f46bd2b3640 1 -- 172.21.15.26:0/905566570 wait complete. 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.230+0000 7f46bd2b3640 10 monclient: build_initial_monmap 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.230+0000 7f46bd2b3640 10 monclient: monmap: 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:44.239 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:44.240 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.230+0000 7f46bd2b3640 1 librados: starting msgr at 2024-08-19T12:49:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.230+0000 7f46bd2b3640 1 librados: starting objecter 2024-08-19T12:49:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.230+0000 7f46bd2b3640 1 Processor -- start 2024-08-19T12:49:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.230+0000 7f46bd2b3640 1 -- start start 2024-08-19T12:49:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.230+0000 7f46bd2b3640 1 librados: setting wanted keys 2024-08-19T12:49:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.230+0000 7f46bd2b3640 1 librados: calling monclient init 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.230+0000 7f46bd2b3640 10 monclient: init 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46bd2b3640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46bd2b3640 10 monclient: _add_conns ranks=[5,1,0,2,4,3,8,6,7] 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46bd2b3640 10 monclient(hunting): picked mon.noname-h con 0x7f46b806e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46bd2b3640 10 monclient(hunting): picked mon.noname-f con 0x7f46b80a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46bd2b3640 10 monclient(hunting): picked mon.noname-a con 0x7f46b80a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46bd2b3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b81cf060 con 0x7f46b80a5d50 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46bd2b3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b81342e0 con 0x7f46b80a5920 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46bd2b3640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b80ab890 con 0x7f46b806e7b0 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46bd2b3640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46bd2b3640 10 monclient(hunting): authenticate will time out at 1378.318300s 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46b6ffd640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7f46b806e7b0 legacy=0x7f46b81ce240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:49:44.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46ae7fc640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f46b80a5920 legacy=0x7f46b81ce950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:49:44.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46b77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f46b80a5d50 0x7f46b81d6510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:50654/0 (socket says 172.21.15.26:50654) 2024-08-19T12:49:44.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.231+0000 7f46b77fe640 1 -- 172.21.15.26:0/1202207054 learned_addr learned my addr 172.21.15.26:0/1202207054 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:44.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46b77fe640 1 -- 172.21.15.26:0/1202207054 >> v1:172.21.15.26:6789/0 conn(0x7f46b80a5d50 legacy=0x7f46b81d6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 10 client.?.objecter ms_handle_connect 0x7f46b80a5d50 2024-08-19T12:49:44.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:44.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2572592765 0 0) 0x7f46b81cf060 con 0x7f46b80a5d50 2024-08-19T12:49:44.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 10 monclient(hunting): my global_id is 145206 2024-08-19T12:49:44.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4698004ee0 con 0x7f46b80a5d50 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 483907860 0 0) 0x7f4698004ee0 con 0x7f46b80a5d50 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46b81cf060 con 0x7f46b80a5d50 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f46b0001df0 con 0x7f46b80a5d50 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 10 monclient(hunting): dump: 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.247 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.233+0000 7f46affff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3105903083 0 0) 0x7f46b81cf060 con 0x7f46b80a5d50 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 >> v1:172.21.15.104:6791/0 conn(0x7f46b806e7b0 legacy=0x7f46b81ce240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 >> v1:172.21.15.104:6789/0 conn(0x7f46b80a5920 legacy=0x7f46b81ce950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 10 monclient: _finish_hunting 0 2024-08-19T12:49:44.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 1 monclient: found mon.a 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b80a0660 con 0x7f46b80a5d50 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 10 monclient: _finish_auth 0 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 10 monclient: _check_auth_tickets 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46bd2b3640 5 monclient: authenticate success, global_id 145206 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46bd2b3640 10 monclient: _renew_subs 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46bd2b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46bd2b3640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f46b80a0940 con 0x7f46b80a5d50 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46bd2b3640 10 client.145206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46bd2b3640 10 monclient: _renew_subs 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46bd2b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:49:44.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46bd2b3640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f46b809f9a0 con 0x7f46b80a5d50 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46bd2b3640 1 librados: init done 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f46b00051d0 con 0x7f46b80a5d50 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f46b0001830 con 0x7f46b80a5d50 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46b5ffb640 4 set_mon_vals no callback set 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46affff640 10 monclient: dump: 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.250 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46bd2b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.234+0000 7f46bd2b3640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f469c0065c0 con 0x7f469c002860 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.235+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f46b0005740 con 0x7f46b80a5d50 2024-08-19T12:49:44.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.235+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(412..412 src has 311..412) ==== 5745+0+0 (unknown 570832459 0 0) 0x7f46b004e5e0 con 0x7f46b80a5d50 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.235+0000 7f46affff640 10 client.145206.objecter ms_dispatch 0x7f46b8000b90 osd_map(412..412 src has 311..412) 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.235+0000 7f46affff640 3 client.145206.objecter handle_osd_map got epochs [412,412] > 0 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.235+0000 7f46affff640 3 client.145206.objecter handle_osd_map decoding full epoch 412 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.235+0000 7f46affff640 20 client.145206.objecter dump_active .. 0 homeless 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.235+0000 7f46ae7fc640 10 monclient: get_auth_request con 0x7f4698040d80 auth_method 0 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46b6ffd640 1 -- 172.21.15.26:0/1202207054 >> v1:172.21.15.104:6790/0 conn(0x7f469c002860 legacy=0x7f469c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 10 client.145206.objecter ms_handle_connect 0x7f469c002860 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 10 client.145206.objecter resend_mon_ops 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3027589596 0 0) 0x7f469c0065c0 con 0x7f469c002860 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 10 monclient(hunting): my global_id is 154632 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46b81cf060 con 0x7f469c002860 2024-08-19T12:49:44.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 10 client.145206.objecter ms_handle_connect 0x7f4698040d80 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2681081350 0 0) 0x7f46b81cf060 con 0x7f469c002860 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f469c0065c0 con 0x7f469c002860 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f46a8004490 con 0x7f469c002860 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.237+0000 7f46affff640 10 monclient: dump: 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.253 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.238+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 261221337 0 0) 0x7f469c0065c0 con 0x7f469c002860 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.238+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f469c00a160 con 0x7f469c002860 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.238+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f46a8006840 con 0x7f469c002860 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.238+0000 7f46affff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.238+0000 7f46affff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.238+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 >> v1:172.21.15.104:6790/0 conn(0x7f469c002860 legacy=0x7f469c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.238+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 >> v1:172.21.15.104:6790/0 conn(0x7f469c002860 legacy=0x7f469c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:44.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.245+0000 7f46bd2b3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:44.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.245+0000 7f46bd2b3640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f469c009e70 con 0x7f469c009130 2024-08-19T12:49:44.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46b6ffd640 1 -- 172.21.15.26:0/1202207054 >> v1:172.21.15.104:6790/0 conn(0x7f469c009130 legacy=0x7f469c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 10 client.145206.objecter ms_handle_connect 0x7f469c009130 2024-08-19T12:49:44.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 10 client.145206.objecter resend_mon_ops 2024-08-19T12:49:44.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3846965973 0 0) 0x7f469c009e70 con 0x7f469c009130 2024-08-19T12:49:44.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 10 monclient(hunting): my global_id is 154641 2024-08-19T12:49:44.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f469c0065c0 con 0x7f469c009130 2024-08-19T12:49:44.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3428763546 0 0) 0x7f469c0065c0 con 0x7f469c009130 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f469c009e70 con 0x7f469c009130 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f46a8004f80 con 0x7f469c009130 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 10 monclient: dump: 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 585019483 0 0) 0x7f469c009e70 con 0x7f469c009130 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f469c00c410 con 0x7f469c009130 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7500 (unknown 0 0 2106133445) 0x7f46a8006320 con 0x7f469c009130 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 >> v1:172.21.15.104:6790/0 conn(0x7f469c009130 legacy=0x7f469c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.248+0000 7f46affff640 1 -- 172.21.15.26:0/1202207054 >> v1:172.21.15.104:6790/0 conn(0x7f469c009130 legacy=0x7f469c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:44.259 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:44.259 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:44.259 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:44.259 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:44.259 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 76, 2024-08-19T12:49:44.259 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 295977, 2024-08-19T12:49:44.259 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:44.259 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:44.259 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:44.259 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:44.259 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:44.259 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:44.260 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:44.261 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:44.262 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:44.263 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.264 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.266 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.267 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:44.268 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.269 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:44.272 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.274 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:44.275 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:44.276 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:44.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.276 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.276 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:44.276 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 10 librados: watch_flush enter 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 10 librados: watch_flush exit 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 20 client.145206.objecter shutdown clearing up homeless session... 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 10 client.145206.objecter successfully canceled tick 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 1 -- 172.21.15.26:0/1202207054 >> v1:172.21.15.104:6800/899887506 conn(0x7f4698040d80 legacy=0x7f4698043240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 10 monclient: shutdown 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 1 -- 172.21.15.26:0/1202207054 >> v1:172.21.15.26:6789/0 conn(0x7f46b80a5d50 legacy=0x7f46b81d6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46b77fe640 1 -- 172.21.15.26:0/1202207054 reap_dead start 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46b77fe640 1 -- 172.21.15.26:0/1202207054 reap_dead start 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 1 -- 172.21.15.26:0/1202207054 shutdown_connections 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 1 -- 172.21.15.26:0/1202207054 >> 172.21.15.26:0/1202207054 conn(0x7f46b80a8740 msgr2=0x7f46b806d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:44.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 1 -- 172.21.15.26:0/1202207054 shutdown_connections 2024-08-19T12:49:44.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 1 -- 172.21.15.26:0/1202207054 wait complete. 2024-08-19T12:49:44.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.249+0000 7f46bd2b3640 1 librados: shutdown 2024-08-19T12:49:44.279 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T12:49:44.279 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon i 2024-08-19T12:49:44.279 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:49:44.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.378+0000 7fbf1bfba640 10 monclient: get_monmap_and_config 2024-08-19T12:49:44.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.378+0000 7fbf1bfba640 10 monclient: build_initial_monmap 2024-08-19T12:49:44.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.378+0000 7fbf1bfba640 10 monclient: monmap: 2024-08-19T12:49:44.379 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:44.379 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:49:44.379761+0000 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:49:44.379761+0000 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:44.380 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.379+0000 7fbf1bfba640 1 Processor -- start 2024-08-19T12:49:44.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.379+0000 7fbf1bfba640 1 -- start start 2024-08-19T12:49:44.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.379+0000 7fbf1bfba640 10 monclient: init 2024-08-19T12:49:44.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf1bfba640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:44.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf1bfba640 10 monclient: _add_conns ranks=[8,5,6,1,2,7,4,0,3] 2024-08-19T12:49:44.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf1bfba640 10 monclient(hunting): picked mon.noname-e con 0x7fbf14073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:49:44.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf1bfba640 10 monclient(hunting): picked mon.noname-h con 0x7fbf140a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:49:44.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf1bfba640 10 monclient(hunting): picked mon.noname-d con 0x7fbf1413f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:44.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf1bfba640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf1409dae0 con 0x7fbf140a57a0 2024-08-19T12:49:44.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf1bfba640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf14136910 con 0x7fbf1413f2e0 2024-08-19T12:49:44.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf1bfba640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf1409dd50 con 0x7fbf14073470 2024-08-19T12:49:44.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf1bfba640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:44.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf1bfba640 10 monclient(hunting): authenticate will time out at 1378.467186s 2024-08-19T12:49:44.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf19d56640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7fbf14073470 legacy=0x7fbf14136200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:49:44.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf1a557640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fbf1413f2e0 0x7fbf141416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:46826/0 (socket says 172.21.15.26:46826) 2024-08-19T12:49:44.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf1a557640 1 -- 172.21.15.26:0/3363406097 learned_addr learned my addr 172.21.15.26:0/3363406097 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:44.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.380+0000 7fbf19555640 1 -- 172.21.15.26:0/3363406097 >> v1:172.21.15.104:6791/0 conn(0x7fbf140a57a0 legacy=0x7fbf140a5b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:49:44.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf1a557640 1 -- 172.21.15.26:0/3363406097 >> v1:172.21.15.26:6792/0 conn(0x7fbf1413f2e0 legacy=0x7fbf141416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 1 -- 172.21.15.26:0/3363406097 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2337267231 0 0) 0x7fbf14136910 con 0x7fbf1413f2e0 2024-08-19T12:49:44.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient(hunting): my global_id is 164517 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 1 -- 172.21.15.26:0/3363406097 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbefc003520 con 0x7fbf1413f2e0 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 1 -- 172.21.15.26:0/3363406097 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 582264632 0 0) 0x7fbefc003520 con 0x7fbf1413f2e0 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 1 -- 172.21.15.26:0/3363406097 >> v1:172.21.15.26:6793/0 conn(0x7fbf14073470 legacy=0x7fbf14136200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 1 -- 172.21.15.26:0/3363406097 >> v1:172.21.15.104:6791/0 conn(0x7fbf140a57a0 legacy=0x7fbf140a5b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient: _finish_hunting 0 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 1 monclient: found mon.noname-d 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 1 -- 172.21.15.26:0/3363406097 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1409e030 con 0x7fbf1413f2e0 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient: _finish_auth 0 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient: _check_auth_tickets 2024-08-19T12:49:44.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 1 -- 172.21.15.26:0/3363406097 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbf04001e60 con 0x7fbf1413f2e0 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient: dump: 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf1bfba640 5 monclient: authenticate success, global_id 164517 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf1bfba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 1 -- 172.21.15.26:0/3363406097 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf04004c20 con 0x7fbf1413f2e0 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 1 -- 172.21.15.26:0/3363406097 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbf04005190 con 0x7fbf1413f2e0 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf1bfba640 10 monclient: get_monmap_and_config success 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf1bfba640 4 set_mon_vals no callback set 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.382+0000 7fbf18d54640 10 monclient: dump: 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.390 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.383+0000 7fbf1bfba640 10 monclient: shutdown 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.383+0000 7fbf1bfba640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.383+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3363406097 >> v1:172.21.15.26:6792/0 conn(0x7fbf1413f2e0 legacy=0x7fbf141416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.383+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3363406097 shutdown_connections 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.383+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3363406097 >> 172.21.15.26:0/3363406097 conn(0x7fbf140abff0 msgr2=0x7fbf140ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.383+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3363406097 shutdown_connections 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.383+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3363406097 wait complete. 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 10 monclient: build_initial_monmap 2024-08-19T12:49:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 10 monclient: monmap: 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:49:44.392 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 1 librados: starting msgr at 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 1 librados: starting objecter 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 1 Processor -- start 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 1 -- start start 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 1 librados: setting wanted keys 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 1 librados: calling monclient init 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 10 monclient: init 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 10 monclient: _reopen_session rank -1 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 10 monclient: _add_conns ranks=[6,3,1,8,5,0,7,2,4] 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 10 monclient(hunting): picked mon.noname-d con 0x7fbf14073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 10 monclient(hunting): picked mon.noname-g con 0x7fbf140a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 10 monclient(hunting): picked mon.noname-f con 0x7fbf1413f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:49:44.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf14142b80 con 0x7fbf1413f2e0 2024-08-19T12:49:44.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf141308e0 con 0x7fbf140a57a0 2024-08-19T12:49:44.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf141323a0 con 0x7fbf14073470 2024-08-19T12:49:44.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 10 monclient(hunting): _renew_subs 2024-08-19T12:49:44.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1bfba640 10 monclient(hunting): authenticate will time out at 1378.471878s 2024-08-19T12:49:44.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf19d56640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fbf14073470 0x7fbf1412fac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:46842/0 (socket says 172.21.15.26:46842) 2024-08-19T12:49:44.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf19d56640 1 -- 172.21.15.26:0/3722656134 learned_addr learned my addr 172.21.15.26:0/3722656134 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:49:44.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.384+0000 7fbf1a557640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6789/0 conn(0x7fbf1413f2e0 legacy=0x7fbf141d61a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:49:44.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf19d56640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.26:6792/0 conn(0x7fbf14073470 legacy=0x7fbf1412fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 client.?.objecter ms_handle_connect 0x7fbf14073470 2024-08-19T12:49:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 958252987 0 0) 0x7fbf141323a0 con 0x7fbf14073470 2024-08-19T12:49:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient(hunting): my global_id is 164526 2024-08-19T12:49:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf00004eb0 con 0x7fbf14073470 2024-08-19T12:49:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 client.?.objecter ms_handle_connect 0x7fbf140a57a0 2024-08-19T12:49:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:49:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf19555640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6790/0 conn(0x7fbf140a57a0 legacy=0x7fbf141301d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2397987265 0 0) 0x7fbf00004eb0 con 0x7fbf14073470 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf141323a0 con 0x7fbf14073470 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbf0c004620 con 0x7fbf14073470 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient(hunting): dump: 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.400 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3683264197 0 0) 0x7fbf141308e0 con 0x7fbf140a57a0 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient(hunting): my global_id is 154650 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf00004eb0 con 0x7fbf140a57a0 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 134031701 0 0) 0x7fbf141323a0 con 0x7fbf14073470 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6790/0 conn(0x7fbf140a57a0 legacy=0x7fbf141301d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6789/0 conn(0x7fbf1413f2e0 legacy=0x7fbf141d61a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient: _finish_hunting 0 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 monclient: found mon.d 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf140a5280 con 0x7fbf14073470 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient: _finish_auth 0 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient: _check_auth_tickets 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:49:44.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf1bfba640 5 monclient: authenticate success, global_id 164526 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf1bfba640 10 monclient: _renew_subs 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf1bfba640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf140a5510 con 0x7fbf14073470 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf1bfba640 10 client.164526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf1bfba640 10 monclient: _renew_subs 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf1bfba640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf141d7aa0 con 0x7fbf14073470 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf1bfba640 1 librados: init done 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf0c004930 con 0x7fbf14073470 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbf0c004ea0 con 0x7fbf14073470 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:49:44.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf13fff640 4 set_mon_vals no callback set 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.387+0000 7fbf127fc640 10 monclient: dump: 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.404 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.388+0000 7fbf1bfba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.388+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbef40065c0 con 0x7fbef4002860 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.388+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fbf0c005040 con 0x7fbf14073470 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.388+0000 7fbf19555640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6792/0 conn(0x7fbef4002860 legacy=0x7fbef4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.388+0000 7fbf19555640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6792/0 conn(0x7fbef4002860 legacy=0x7fbef4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.388+0000 7fbf19555640 1 --1- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6792/0 conn(0x7fbef4002860 0x7fbef4004d00 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.388+0000 7fbf19555640 1 --1- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6792/0 conn(0x7fbef4002860 0x7fbef4004d00 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.389+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6792/0 conn(0x7fbef4002860 legacy=0x7fbef4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.389+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6792/0 conn(0x7fbef4002860 legacy=0x7fbef4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.389+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbf141323a0 con 0x7fbf00044bf0 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.389+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(412..412 src has 311..412) ==== 5745+0+0 (unknown 570832459 0 0) 0x7fbf0c04e380 con 0x7fbf14073470 2024-08-19T12:49:44.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.389+0000 7fbf127fc640 10 client.164526.objecter ms_dispatch 0x7fbf14000b90 osd_map(412..412 src has 311..412) 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.389+0000 7fbf127fc640 3 client.164526.objecter handle_osd_map got epochs [412,412] > 0 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.389+0000 7fbf127fc640 3 client.164526.objecter handle_osd_map decoding full epoch 412 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.389+0000 7fbf127fc640 20 client.164526.objecter dump_active .. 0 homeless 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.389+0000 7fbf19555640 10 monclient: get_auth_request con 0x7fbf00040f20 auth_method 0 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf1a557640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6792/0 conn(0x7fbf00044bf0 legacy=0x7fbf00046fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 10 client.164526.objecter ms_handle_connect 0x7fbf00044bf0 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 10 client.164526.objecter resend_mon_ops 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 10 client.164526.objecter ms_handle_connect 0x7fbf00040f20 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2421038366 0 0) 0x7fbf141323a0 con 0x7fbf00044bf0 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 10 monclient(hunting): my global_id is 135817 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf14142b80 con 0x7fbf00044bf0 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 956930333 0 0) 0x7fbf14142b80 con 0x7fbf00044bf0 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf141323a0 con 0x7fbf00044bf0 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbf04004ea0 con 0x7fbf00044bf0 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 10 monclient: dump: 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.407 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 554563502 0 0) 0x7fbf141323a0 con 0x7fbf00044bf0 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.391+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf000497d0 con 0x7fbf00044bf0 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.392+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbf04001d10 con 0x7fbf00044bf0 2024-08-19T12:49:44.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.392+0000 7fbf127fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:49:44.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.392+0000 7fbf127fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:49:44.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.392+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6792/0 conn(0x7fbf00044bf0 legacy=0x7fbf00046fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.392+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6792/0 conn(0x7fbf00044bf0 legacy=0x7fbf00046fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:44.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.398+0000 7fbf1bfba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:49:44.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.398+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbef4009e10 con 0x7fbef4009160 2024-08-19T12:49:44.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf1a557640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6792/0 conn(0x7fbef4009160 legacy=0x7fbef40095f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:49:44.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 10 client.164526.objecter ms_handle_connect 0x7fbef4009160 2024-08-19T12:49:44.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 10 client.164526.objecter resend_mon_ops 2024-08-19T12:49:44.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1088610032 0 0) 0x7fbef4009e10 con 0x7fbef4009160 2024-08-19T12:49:44.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 10 monclient(hunting): _negotiate 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 10 monclient(hunting): my global_id is 135826 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf141323a0 con 0x7fbef4009160 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1175695048 0 0) 0x7fbf141323a0 con 0x7fbef4009160 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbef4009e10 con 0x7fbef4009160 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbf04005110 con 0x7fbef4009160 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 10 monclient: dump: 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.411 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3114785079 0 0) 0x7fbef4009e10 con 0x7fbef4009160 2024-08-19T12:49:44.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.401+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbef400c350 con 0x7fbef4009160 2024-08-19T12:49:44.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7834 (unknown 0 0 2593088691) 0x7fbf04005770 con 0x7fbef4009160 2024-08-19T12:49:44.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf127fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:49:44.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf127fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:49:44.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6792/0 conn(0x7fbef4009160 legacy=0x7fbef40095f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf127fc640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6792/0 conn(0x7fbef4009160 legacy=0x7fbef40095f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:49:44.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1a557640 1 -- 172.21.15.26:0/3722656134 reap_dead start 2024-08-19T12:49:44.413 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:49:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:49:44.413 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 76, 2024-08-19T12:49:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 554759, 2024-08-19T12:49:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:49:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:49:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:49:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:49:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:49:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-08-19T12:49:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:49:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:49:44.414 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:49:44.415 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:49:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:44.416 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:49:44.417 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:49:44.418 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:49:44.420 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.421 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.426 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.427 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.428 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:49:44.429 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:44.430 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:44.431 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:49:44.432 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:49:44.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 10 librados: watch_flush enter 2024-08-19T12:49:44.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 10 librados: watch_flush exit 2024-08-19T12:49:44.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 20 client.164526.objecter shutdown clearing up homeless session... 2024-08-19T12:49:44.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 10 client.164526.objecter successfully canceled tick 2024-08-19T12:49:44.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.104:6800/899887506 conn(0x7fbf00040f20 legacy=0x7fbf000433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 10 monclient: shutdown 2024-08-19T12:49:44.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:49:44.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3722656134 >> v1:172.21.15.26:6792/0 conn(0x7fbf14073470 legacy=0x7fbf1412fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:49:44.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3722656134 shutdown_connections 2024-08-19T12:49:44.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3722656134 >> 172.21.15.26:0/3722656134 conn(0x7fbf140abff0 msgr2=0x7fbf140a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:49:44.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3722656134 shutdown_connections 2024-08-19T12:49:44.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 1 -- 172.21.15.26:0/3722656134 wait complete. 2024-08-19T12:49:44.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:49:44.402+0000 7fbf1bfba640 1 librados: shutdown 2024-08-19T12:49:44.436 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-08-19T12:49:44.436 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T12:50:39.147 INFO:tasks.ceph.osd.0.smithi026.stderr:2024-08-19T12:50:39.144+0000 7f8afd841640 -1 snap_mapper.add_oid found existing snaps mapped on 104:0aa5c1c5:test-rados-api-smithi104-24748-97::foo:21, removing 2024-08-19T12:50:39.147 INFO:tasks.ceph.osd.4.smithi104.stderr:2024-08-19T12:50:39.145+0000 7f926986d640 -1 snap_mapper.add_oid found existing snaps mapped on 104:0aa5c1c5:test-rados-api-smithi104-24748-97::foo:21, removing 2024-08-19T12:51:14.436 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.h 2024-08-19T12:51:14.437 INFO:tasks.ceph.mon.h:Restarting daemon 2024-08-19T12:51:14.437 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-08-19T12:51:14.441 INFO:tasks.ceph.mon.h:Started 2024-08-19T12:51:14.441 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.f 2024-08-19T12:51:14.441 INFO:tasks.ceph.mon.f:Restarting daemon 2024-08-19T12:51:14.441 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-08-19T12:51:14.443 INFO:tasks.ceph.mon.f:Started 2024-08-19T12:51:14.443 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-08-19T12:51:14.443 INFO:tasks.ceph.mon.e:Restarting daemon 2024-08-19T12:51:14.443 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-08-19T12:51:14.446 INFO:tasks.ceph.mon.e:Started 2024-08-19T12:51:14.446 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-08-19T12:51:14.446 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-19T12:51:14.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-08-19T12:51:14.448 INFO:tasks.ceph.mon.c:Started 2024-08-19T12:51:14.448 INFO:tasks.mon_thrash:Sending STOP to mon i 2024-08-19T12:51:14.448 INFO:tasks.ceph.mon.i:Sent signal 19 2024-08-19T12:51:14.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:51:29.450 INFO:tasks.mon_thrash:Sending CONT to mon i 2024-08-19T12:51:29.451 INFO:tasks.ceph.mon.i:Sent signal 18 2024-08-19T12:51:29.451 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:51:29.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:51:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.570+0000 7f8bf6df4640 10 monclient: get_monmap_and_config 2024-08-19T12:51:29.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.570+0000 7f8bf6df4640 10 monclient: build_initial_monmap 2024-08-19T12:51:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.570+0000 7f8bf6df4640 10 monclient: monmap: 2024-08-19T12:51:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:29.571233+0000 2024-08-19T12:51:29.574 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:29.571233+0000 2024-08-19T12:51:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:29.575 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:29.576 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:29.576 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 1 Processor -- start 2024-08-19T12:51:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 1 -- start start 2024-08-19T12:51:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 10 monclient: init 2024-08-19T12:51:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 10 monclient: _add_conns ranks=[8,6,3,1,5,0,4,2,7] 2024-08-19T12:51:29.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 10 monclient(hunting): picked mon.noname-e con 0x7f8bf0073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:51:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 10 monclient(hunting): picked mon.noname-d con 0x7f8bf00a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 10 monclient(hunting): picked mon.noname-g con 0x7f8bf013f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bf009dae0 con 0x7f8bf013f2e0 2024-08-19T12:51:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bf0136910 con 0x7f8bf00a57a0 2024-08-19T12:51:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bf009dd50 con 0x7f8bf0073470 2024-08-19T12:51:29.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf6df4640 10 monclient(hunting): authenticate will time out at 1483.658931s 2024-08-19T12:51:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf4b90640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f8bf0073470 0x7f8bf0136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:45492/0 (socket says 172.21.15.26:45492) 2024-08-19T12:51:29.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.571+0000 7f8bf4b90640 1 -- 172.21.15.26:0/3590114321 learned_addr learned my addr 172.21.15.26:0/3590114321 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.574+0000 7f8bf4b90640 1 -- 172.21.15.26:0/3590114321 >> v1:172.21.15.26:6793/0 conn(0x7f8bf0073470 legacy=0x7f8bf0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.574+0000 7f8beffff640 1 -- 172.21.15.26:0/3590114321 >> v1:172.21.15.26:6792/0 conn(0x7f8bf00a57a0 legacy=0x7f8bf00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:29.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.574+0000 7f8bf5391640 1 -- 172.21.15.26:0/3590114321 >> v1:172.21.15.104:6790/0 conn(0x7f8bf013f2e0 legacy=0x7f8bf01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:29.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 1 -- 172.21.15.26:0/3590114321 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2565701911 0 0) 0x7f8bf0136910 con 0x7f8bf00a57a0 2024-08-19T12:51:29.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:29.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:29.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:29.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 10 monclient(hunting): my global_id is 164535 2024-08-19T12:51:29.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 1 -- 172.21.15.26:0/3590114321 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8be0003520 con 0x7f8bf00a57a0 2024-08-19T12:51:29.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 1 -- 172.21.15.26:0/3590114321 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1650686196 0 0) 0x7f8be0003520 con 0x7f8bf00a57a0 2024-08-19T12:51:29.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 1 -- 172.21.15.26:0/3590114321 >> v1:172.21.15.26:6793/0 conn(0x7f8bf0073470 legacy=0x7f8bf0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 1 -- 172.21.15.26:0/3590114321 >> v1:172.21.15.104:6790/0 conn(0x7f8bf013f2e0 legacy=0x7f8bf01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 1 monclient: found mon.noname-d 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 1 -- 172.21.15.26:0/3590114321 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf009e030 con 0x7f8bf00a57a0 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 10 monclient: _finish_auth 0 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 1 -- 172.21.15.26:0/3590114321 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8bdc001b30 con 0x7f8bf00a57a0 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bf6df4640 5 monclient: authenticate success, global_id 164535 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bf6df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:29.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.801+0000 7f8bef7fe640 10 monclient: dump: 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:29.805 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.802+0000 7f8bf6df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.802+0000 7f8bef7fe640 1 -- 172.21.15.26:0/3590114321 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8bdc003610 con 0x7f8bf00a57a0 2024-08-19T12:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.802+0000 7f8bef7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.802+0000 7f8bef7fe640 1 -- 172.21.15.26:0/3590114321 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8bdc003b40 con 0x7f8bf00a57a0 2024-08-19T12:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.802+0000 7f8bf6df4640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.802+0000 7f8bf6df4640 4 set_mon_vals no callback set 2024-08-19T12:51:29.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.802+0000 7f8bef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.802+0000 7f8bef7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.802+0000 7f8bef7fe640 10 monclient: dump: 2024-08-19T12:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:29.807 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:29.808 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.803+0000 7f8bf6df4640 10 monclient: shutdown 2024-08-19T12:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.803+0000 7f8bf6df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.803+0000 7f8bf6df4640 1 -- 172.21.15.26:0/3590114321 >> v1:172.21.15.26:6792/0 conn(0x7f8bf00a57a0 legacy=0x7f8bf00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.803+0000 7f8bf6df4640 1 -- 172.21.15.26:0/3590114321 shutdown_connections 2024-08-19T12:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.803+0000 7f8bf6df4640 1 -- 172.21.15.26:0/3590114321 >> 172.21.15.26:0/3590114321 conn(0x7f8bf00abff0 msgr2=0x7f8bf00ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.803+0000 7f8bf6df4640 1 -- 172.21.15.26:0/3590114321 shutdown_connections 2024-08-19T12:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.803+0000 7f8bf6df4640 1 -- 172.21.15.26:0/3590114321 wait complete. 2024-08-19T12:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.804+0000 7f8bf6df4640 10 monclient: build_initial_monmap 2024-08-19T12:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.804+0000 7f8bf6df4640 10 monclient: monmap: 2024-08-19T12:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:29.815 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:29.816 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.804+0000 7f8bf6df4640 1 librados: starting msgr at 2024-08-19T12:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.804+0000 7f8bf6df4640 1 librados: starting objecter 2024-08-19T12:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.805+0000 7f8bf6df4640 1 Processor -- start 2024-08-19T12:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.805+0000 7f8bf6df4640 1 -- start start 2024-08-19T12:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.805+0000 7f8bf6df4640 1 librados: setting wanted keys 2024-08-19T12:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.805+0000 7f8bf6df4640 1 librados: calling monclient init 2024-08-19T12:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.805+0000 7f8bf6df4640 10 monclient: init 2024-08-19T12:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.805+0000 7f8bf6df4640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.805+0000 7f8bf6df4640 10 monclient: _add_conns ranks=[6,7,2,1,0,4,3,8,5] 2024-08-19T12:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.805+0000 7f8bf6df4640 10 monclient(hunting): picked mon.noname-d con 0x7f8bf0073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:29.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.806+0000 7f8bf6df4640 10 monclient(hunting): picked mon.noname-i con 0x7f8bf00a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:29.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.806+0000 7f8bf6df4640 10 monclient(hunting): picked mon.noname-b con 0x7f8bf013f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.806+0000 7f8bf6df4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bf00a3cb0 con 0x7f8bf013f2e0 2024-08-19T12:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.806+0000 7f8bf6df4640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bf01380c0 con 0x7f8bf0073470 2024-08-19T12:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.806+0000 7f8bf6df4640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bf00a4810 con 0x7f8bf00a57a0 2024-08-19T12:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.806+0000 7f8bf6df4640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.806+0000 7f8bf6df4640 10 monclient(hunting): authenticate will time out at 1483.893203s 2024-08-19T12:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.806+0000 7f8bf4b90640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f8bf0073470 0x7f8bf01ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51592/0 (socket says 172.21.15.26:51592) 2024-08-19T12:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.806+0000 7f8bf4b90640 1 -- 172.21.15.26:0/314655306 learned_addr learned my addr 172.21.15.26:0/314655306 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.808+0000 7f8bf4b90640 1 -- 172.21.15.26:0/314655306 >> v1:172.21.15.26:6792/0 conn(0x7f8bf0073470 legacy=0x7f8bf01ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:29.819 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.808+0000 7f8bed7fa640 10 client.?.objecter ms_handle_connect 0x7f8bf0073470 2024-08-19T12:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.808+0000 7f8bed7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.808+0000 7f8bed7fa640 10 client.?.objecter ms_handle_connect 0x7f8bf013f2e0 2024-08-19T12:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.808+0000 7f8bf5391640 1 -- 172.21.15.26:0/314655306 >> v1:172.21.15.26:6790/0 conn(0x7f8bf013f2e0 legacy=0x7f8bf01d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.808+0000 7f8bed7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.808+0000 7f8beffff640 1 -- 172.21.15.26:0/314655306 >> v1:172.21.15.104:6792/0 conn(0x7f8bf00a57a0 legacy=0x7f8bf01d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:29.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.808+0000 7f8bed7fa640 10 client.?.objecter ms_handle_connect 0x7f8bf00a57a0 2024-08-19T12:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.808+0000 7f8bed7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3888666675 0 0) 0x7f8bf009dd50 con 0x7f8bf0073470 2024-08-19T12:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): my global_id is 164544 2024-08-19T12:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bd4004eb0 con 0x7f8bf0073470 2024-08-19T12:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2263421795 0 0) 0x7f8bf00a3cb0 con 0x7f8bf013f2e0 2024-08-19T12:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:29.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): my global_id is 26867 2024-08-19T12:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bf009dd50 con 0x7f8bf013f2e0 2024-08-19T12:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3999704960 0 0) 0x7f8bf00a4810 con 0x7f8bf00a57a0 2024-08-19T12:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): my global_id is 135835 2024-08-19T12:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bf00a3cb0 con 0x7f8bf00a57a0 2024-08-19T12:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 956592475 0 0) 0x7f8bd4004eb0 con 0x7f8bf0073470 2024-08-19T12:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bf00a4810 con 0x7f8bf0073470 2024-08-19T12:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8be40047e0 con 0x7f8bf0073470 2024-08-19T12:51:29.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): dump: 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:29.823 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 541742912 0 0) 0x7f8bf009dd50 con 0x7f8bf013f2e0 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bd4004eb0 con 0x7f8bf013f2e0 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 968710732 0 0) 0x7f8bf00a3cb0 con 0x7f8bf00a57a0 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bf009dd50 con 0x7f8bf00a57a0 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8be8002070 con 0x7f8bf013f2e0 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:29.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): dump: 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8bdc003b00 con 0x7f8bf00a57a0 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient(hunting): dump: 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2365274351 0 0) 0x7f8bf00a4810 con 0x7f8bf0073470 2024-08-19T12:51:29.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 >> v1:172.21.15.104:6792/0 conn(0x7f8bf00a57a0 legacy=0x7f8bf01d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 >> v1:172.21.15.26:6790/0 conn(0x7f8bf013f2e0 legacy=0x7f8bf01d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient: _finish_hunting 0 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 monclient: found mon.d 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf0136e40 con 0x7f8bf0073470 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient: _finish_auth 0 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient: _check_auth_tickets 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bf6df4640 5 monclient: authenticate success, global_id 164544 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bf6df4640 10 monclient: _renew_subs 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bf6df4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:29.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bf6df4640 1 -- 172.21.15.26:0/314655306 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf0137120 con 0x7f8bf0073470 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bf6df4640 10 client.164544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bf6df4640 10 monclient: _renew_subs 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bf6df4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bf6df4640 1 -- 172.21.15.26:0/314655306 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f8bf00a32c0 con 0x7f8bf0073470 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bf6df4640 1 librados: init done 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8be4004cb0 con 0x7f8bf0073470 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8be40051a0 con 0x7f8bf0073470 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8beeffd640 4 set_mon_vals no callback set 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.809+0000 7f8bed7fa640 10 monclient: dump: 2024-08-19T12:51:29.829 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:29.830 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:29.830 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:29.830 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:29.830 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:29.830 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:29.830 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:29.831 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:29.831 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:29.831 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:29.831 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:29.831 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:29.831 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:29.831 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:29.831 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:29.831 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:29.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.810+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f8be40053f0 con 0x7f8bf0073470 2024-08-19T12:51:29.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.811+0000 7f8bf6df4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:29.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.811+0000 7f8bf6df4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:29.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.811+0000 7f8bf6df4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:29.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.811+0000 7f8bf6df4640 1 -- 172.21.15.26:0/314655306 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd0005080 con 0x7f8bf0073470 2024-08-19T12:51:29.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.811+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(481..481 src has 350..481) ==== 6206+0+0 (unknown 3267959302 0 0) 0x7f8be404e630 con 0x7f8bf0073470 2024-08-19T12:51:29.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.811+0000 7f8bed7fa640 10 client.164544.objecter ms_dispatch 0x7f8bf0000b90 osd_map(481..481 src has 350..481) 2024-08-19T12:51:29.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.811+0000 7f8bed7fa640 3 client.164544.objecter handle_osd_map got epochs [481,481] > 0 2024-08-19T12:51:29.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.811+0000 7f8bed7fa640 3 client.164544.objecter handle_osd_map decoding full epoch 481 2024-08-19T12:51:29.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.811+0000 7f8beffff640 10 monclient: get_auth_request con 0x7f8bd40419e0 auth_method 0 2024-08-19T12:51:29.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.811+0000 7f8bed7fa640 20 client.164544.objecter dump_active .. 0 homeless 2024-08-19T12:51:29.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.813+0000 7f8bed7fa640 10 client.164544.objecter ms_handle_connect 0x7f8bd40419e0 2024-08-19T12:51:29.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.816+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f8bf01380c0 con 0x7f8bf0073470 2024-08-19T12:51:29.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.816+0000 7f8bed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:29.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:29.816+0000 7f8bed7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:30.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.071+0000 7f8bf6df4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:51:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.071+0000 7f8bf6df4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:51:30.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.071+0000 7f8bf6df4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.071+0000 7f8bf6df4640 1 -- 172.21.15.26:0/314655306 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8bd0005370 con 0x7f8bf0073470 2024-08-19T12:51:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.072+0000 7f8bed7fa640 1 -- 172.21.15.26:0/314655306 <== mon.6 v1:172.21.15.26:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 2631029786) 0x7f8be4017680 con 0x7f8bf0073470 2024-08-19T12:51:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.072+0000 7f8bed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:51:30.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.072+0000 7f8bed7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:30.074 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":92,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:51:30.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 10 librados: watch_flush enter 2024-08-19T12:51:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 10 librados: watch_flush exit 2024-08-19T12:51:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 20 client.164544.objecter shutdown clearing up homeless session... 2024-08-19T12:51:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 10 client.164544.objecter successfully canceled tick 2024-08-19T12:51:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 1 -- 172.21.15.26:0/314655306 >> v1:172.21.15.104:6800/899887506 conn(0x7f8bd40419e0 legacy=0x7f8bd4043ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 10 monclient: shutdown 2024-08-19T12:51:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 1 -- 172.21.15.26:0/314655306 >> v1:172.21.15.26:6792/0 conn(0x7f8bf0073470 legacy=0x7f8bf01ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 1 -- 172.21.15.26:0/314655306 shutdown_connections 2024-08-19T12:51:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 1 -- 172.21.15.26:0/314655306 >> 172.21.15.26:0/314655306 conn(0x7f8bf00abff0 msgr2=0x7f8bf0134b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 1 -- 172.21.15.26:0/314655306 shutdown_connections 2024-08-19T12:51:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 1 -- 172.21.15.26:0/314655306 wait complete. 2024-08-19T12:51:30.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.074+0000 7f8bf6df4640 1 librados: shutdown 2024-08-19T12:51:30.085 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:51:30.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:51:30.086 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:51:30.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.185+0000 7ff88c638640 10 monclient: get_monmap_and_config 2024-08-19T12:51:30.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.185+0000 7ff88c638640 10 monclient: build_initial_monmap 2024-08-19T12:51:30.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.185+0000 7ff88c638640 10 monclient: monmap: 2024-08-19T12:51:30.186 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:30.186 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:30.186448+0000 2024-08-19T12:51:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:30.186448+0000 2024-08-19T12:51:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:30.187 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:30.188 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:30.188 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 1 Processor -- start 2024-08-19T12:51:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 1 -- start start 2024-08-19T12:51:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 10 monclient: init 2024-08-19T12:51:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 10 monclient: _add_conns ranks=[2,3,6,8,7,4,5,0,1] 2024-08-19T12:51:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 10 monclient(hunting): picked mon.noname-b con 0x7ff88406e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 10 monclient(hunting): picked mon.noname-g con 0x7ff8840a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:30.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 10 monclient(hunting): picked mon.noname-d con 0x7ff8840a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff884136790 con 0x7ff88406e7b0 2024-08-19T12:51:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff88409ddf0 con 0x7ff8840a5920 2024-08-19T12:51:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff88409e0d0 con 0x7ff8840a5d50 2024-08-19T12:51:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88c638640 10 monclient(hunting): authenticate will time out at 1484.273884s 2024-08-19T12:51:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88a3d4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff88406e7b0 0x7ff884136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54516/0 (socket says 172.21.15.26:54516) 2024-08-19T12:51:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88a3d4640 1 -- 172.21.15.26:0/893734826 learned_addr learned my addr 172.21.15.26:0/893734826 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.186+0000 7ff88abd5640 1 --1- 172.21.15.26:0/893734826 >> v1:172.21.15.26:6792/0 conn(0x7ff8840a5d50 0x7ff884141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51602/0 (socket says 172.21.15.26:51602) 2024-08-19T12:51:30.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff88abd5640 1 -- 172.21.15.26:0/893734826 >> v1:172.21.15.26:6792/0 conn(0x7ff8840a5d50 legacy=0x7ff884141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff88a3d4640 1 -- 172.21.15.26:0/893734826 >> v1:172.21.15.26:6790/0 conn(0x7ff88406e7b0 legacy=0x7ff884136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 1 -- 172.21.15.26:0/893734826 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1325242470 0 0) 0x7ff88409e0d0 con 0x7ff8840a5d50 2024-08-19T12:51:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient(hunting): my global_id is 164553 2024-08-19T12:51:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 1 -- 172.21.15.26:0/893734826 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff870003520 con 0x7ff8840a5d50 2024-08-19T12:51:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 1 -- 172.21.15.26:0/893734826 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1315922994 0 0) 0x7ff884136790 con 0x7ff88406e7b0 2024-08-19T12:51:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient(hunting): my global_id is 26876 2024-08-19T12:51:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 1 -- 172.21.15.26:0/893734826 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff88409e0d0 con 0x7ff88406e7b0 2024-08-19T12:51:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff889bd3640 1 -- 172.21.15.26:0/893734826 >> v1:172.21.15.104:6790/0 conn(0x7ff8840a5920 legacy=0x7ff88413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 1 -- 172.21.15.26:0/893734826 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3703813062 0 0) 0x7ff88409e0d0 con 0x7ff88406e7b0 2024-08-19T12:51:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 1 -- 172.21.15.26:0/893734826 >> v1:172.21.15.26:6792/0 conn(0x7ff8840a5d50 legacy=0x7ff884141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 1 -- 172.21.15.26:0/893734826 >> v1:172.21.15.104:6790/0 conn(0x7ff8840a5920 legacy=0x7ff88413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient: _finish_hunting 0 2024-08-19T12:51:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 1 monclient: found mon.noname-b 2024-08-19T12:51:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:51:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 1 -- 172.21.15.26:0/893734826 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88409e3b0 con 0x7ff88406e7b0 2024-08-19T12:51:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient: _finish_auth 0 2024-08-19T12:51:30.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient: _check_auth_tickets 2024-08-19T12:51:30.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:30.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 1 -- 172.21.15.26:0/893734826 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff880001ac0 con 0x7ff88406e7b0 2024-08-19T12:51:30.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:51:30.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient: dump: 2024-08-19T12:51:30.202 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.202 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.202 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.202 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.203 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.203 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.203 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.203 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.203 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.203 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.203 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.203 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.203 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.203 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.203 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.203 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff88c638640 5 monclient: authenticate success, global_id 26876 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff88c638640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 1 -- 172.21.15.26:0/893734826 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff8800048a0 con 0x7ff88406e7b0 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 1 -- 172.21.15.26:0/893734826 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff880004dd0 con 0x7ff88406e7b0 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff8893d2640 10 monclient: dump: 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.204 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff88c638640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:30.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.189+0000 7ff88c638640 4 set_mon_vals no callback set 2024-08-19T12:51:30.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.190+0000 7ff88c638640 10 monclient: shutdown 2024-08-19T12:51:30.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.190+0000 7ff88c638640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:30.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.190+0000 7ff88c638640 1 -- 172.21.15.26:0/893734826 >> v1:172.21.15.26:6790/0 conn(0x7ff88406e7b0 legacy=0x7ff884136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.190+0000 7ff88c638640 1 -- 172.21.15.26:0/893734826 shutdown_connections 2024-08-19T12:51:30.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.190+0000 7ff88c638640 1 -- 172.21.15.26:0/893734826 >> 172.21.15.26:0/893734826 conn(0x7ff8840a8740 msgr2=0x7ff8840a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:30.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.190+0000 7ff88c638640 1 -- 172.21.15.26:0/893734826 shutdown_connections 2024-08-19T12:51:30.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.190+0000 7ff88c638640 1 -- 172.21.15.26:0/893734826 wait complete. 2024-08-19T12:51:30.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.190+0000 7ff88c638640 10 monclient: build_initial_monmap 2024-08-19T12:51:30.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.190+0000 7ff88c638640 10 monclient: monmap: 2024-08-19T12:51:30.206 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:30.207 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:30.208 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 1 librados: starting msgr at 2024-08-19T12:51:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 1 librados: starting objecter 2024-08-19T12:51:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 1 Processor -- start 2024-08-19T12:51:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 1 -- start start 2024-08-19T12:51:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 1 librados: setting wanted keys 2024-08-19T12:51:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 1 librados: calling monclient init 2024-08-19T12:51:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 10 monclient: init 2024-08-19T12:51:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:30.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 10 monclient: _add_conns ranks=[1,3,5,6,4,7,0,2,8] 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 10 monclient(hunting): picked mon.noname-f con 0x7ff88406e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 10 monclient(hunting): picked mon.noname-g con 0x7ff8840a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 10 monclient(hunting): picked mon.noname-h con 0x7ff8840a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff88413b3f0 con 0x7ff88406e7b0 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff884139090 con 0x7ff8840a5920 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff88413ab50 con 0x7ff8840a5d50 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88c638640 10 monclient(hunting): authenticate will time out at 1484.278820s 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88a3d4640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff88406e7b0 0x7ff8841ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:43242/0 (socket says 172.21.15.26:43242) 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.191+0000 7ff88a3d4640 1 -- 172.21.15.26:0/3533558328 learned_addr learned my addr 172.21.15.26:0/3533558328 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff88a3d4640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.104:6789/0 conn(0x7ff88406e7b0 legacy=0x7ff8841ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff88abd5640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.104:6791/0 conn(0x7ff8840a5d50 legacy=0x7ff8841d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 client.?.objecter ms_handle_connect 0x7ff88406e7b0 2024-08-19T12:51:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 client.?.objecter ms_handle_connect 0x7ff8840a5d50 2024-08-19T12:51:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 client.?.objecter ms_handle_connect 0x7ff8840a5920 2024-08-19T12:51:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff889bd3640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.104:6790/0 conn(0x7ff8840a5920 legacy=0x7ff8841d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4212684735 0 0) 0x7ff88413b3f0 con 0x7ff88406e7b0 2024-08-19T12:51:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): my global_id is 184105 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff868004eb0 con 0x7ff88406e7b0 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1392370877 0 0) 0x7ff884139090 con 0x7ff8840a5920 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): my global_id is 154677 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff88413b3f0 con 0x7ff8840a5920 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3795099510 0 0) 0x7ff88413ab50 con 0x7ff8840a5d50 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): my global_id is 184109 2024-08-19T12:51:30.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff884139090 con 0x7ff8840a5d50 2024-08-19T12:51:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1693592445 0 0) 0x7ff884139090 con 0x7ff8840a5d50 2024-08-19T12:51:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff88413ab50 con 0x7ff8840a5d50 2024-08-19T12:51:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff87c0016a0 con 0x7ff8840a5d50 2024-08-19T12:51:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:51:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): dump: 2024-08-19T12:51:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.212 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 743637722 0 0) 0x7ff868004eb0 con 0x7ff88406e7b0 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff884139090 con 0x7ff88406e7b0 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2882693503 0 0) 0x7ff88413b3f0 con 0x7ff8840a5920 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff868004eb0 con 0x7ff8840a5920 2024-08-19T12:51:30.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff880004d90 con 0x7ff88406e7b0 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): dump: 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff874003610 con 0x7ff8840a5920 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient(hunting): dump: 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.215 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2373830412 0 0) 0x7ff88413ab50 con 0x7ff8840a5d50 2024-08-19T12:51:30.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.104:6790/0 conn(0x7ff8840a5920 legacy=0x7ff8841d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.104:6789/0 conn(0x7ff88406e7b0 legacy=0x7ff8841ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient: _finish_hunting 0 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 monclient: found mon.h 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.194+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88413ae30 con 0x7ff8840a5d50 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff87affd640 10 monclient: _finish_auth 0 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff87affd640 10 monclient: _check_auth_tickets 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff87affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff88c638640 5 monclient: authenticate success, global_id 184109 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff88c638640 10 monclient: _renew_subs 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff88c638640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff88c638640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff884139bf0 con 0x7ff8840a5d50 2024-08-19T12:51:30.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff88c638640 10 client.184109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff88c638640 10 monclient: _renew_subs 2024-08-19T12:51:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff88c638640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:51:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff88c638640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7ff8841386a0 con 0x7ff8840a5d50 2024-08-19T12:51:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff88c638640 1 librados: init done 2024-08-19T12:51:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff87c004ea0 con 0x7ff8840a5d50 2024-08-19T12:51:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff87affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff87c005390 con 0x7ff8840a5d50 2024-08-19T12:51:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff87affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff888bd1640 4 set_mon_vals no callback set 2024-08-19T12:51:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff87affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:51:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff87affd640 10 monclient: dump: 2024-08-19T12:51:30.218 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.219 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff88c638640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.195+0000 7ff88c638640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff86c0065c0 con 0x7ff86c002860 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.196+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff87c0117f0 con 0x7ff8840a5d50 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.196+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(481..481 src has 350..481) ==== 6206+0+0 (unknown 3267959302 0 0) 0x7ff87c04e6c0 con 0x7ff8840a5d50 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.196+0000 7ff87affd640 10 client.184109.objecter ms_dispatch 0x7ff884000b90 osd_map(481..481 src has 350..481) 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.196+0000 7ff87affd640 3 client.184109.objecter handle_osd_map got epochs [481,481] > 0 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.196+0000 7ff87affd640 3 client.184109.objecter handle_osd_map decoding full epoch 481 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.196+0000 7ff87affd640 20 client.184109.objecter dump_active .. 0 homeless 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.196+0000 7ff889bd3640 10 monclient: get_auth_request con 0x7ff868041b10 auth_method 0 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 10 client.184109.objecter ms_handle_connect 0x7ff86c002860 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 10 client.184109.objecter resend_mon_ops 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff88a3d4640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.26:6789/0 conn(0x7ff86c002860 legacy=0x7ff86c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1965233841 0 0) 0x7ff86c0065c0 con 0x7ff86c002860 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 10 monclient(hunting): my global_id is 145251 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff88413ab50 con 0x7ff86c002860 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 68989877 0 0) 0x7ff88413ab50 con 0x7ff86c002860 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff86c0065c0 con 0x7ff86c002860 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff8800048a0 con 0x7ff86c002860 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 10 monclient: dump: 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.221 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.222 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.223 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2593686088 0 0) 0x7ff86c0065c0 con 0x7ff86c002860 2024-08-19T12:51:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff86c00a160 con 0x7ff86c002860 2024-08-19T12:51:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 10 client.184109.objecter ms_handle_connect 0x7ff868041b10 2024-08-19T12:51:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff880005450 con 0x7ff86c002860 2024-08-19T12:51:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.26:6789/0 conn(0x7ff86c002860 legacy=0x7ff86c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.198+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.26:6789/0 conn(0x7ff86c002860 legacy=0x7ff86c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.205+0000 7ff88c638640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.205+0000 7ff88c638640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff86c009e70 con 0x7ff86c009130 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.207+0000 7ff87affd640 10 client.184109.objecter ms_handle_connect 0x7ff86c009130 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.207+0000 7ff88a3d4640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.26:6789/0 conn(0x7ff86c009130 legacy=0x7ff86c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.207+0000 7ff87affd640 10 client.184109.objecter resend_mon_ops 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.207+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3365384103 0 0) 0x7ff86c009e70 con 0x7ff86c009130 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.207+0000 7ff87affd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.207+0000 7ff87affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.207+0000 7ff87affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.207+0000 7ff87affd640 10 monclient(hunting): my global_id is 145260 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.207+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff86c0065c0 con 0x7ff86c009130 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1128118998 0 0) 0x7ff86c0065c0 con 0x7ff86c009130 2024-08-19T12:51:30.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff86c009e70 con 0x7ff86c009130 2024-08-19T12:51:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff880005c30 con 0x7ff86c009130 2024-08-19T12:51:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 10 monclient: dump: 2024-08-19T12:51:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.226 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 676823695 0 0) 0x7ff86c009e70 con 0x7ff86c009130 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff86c00c410 con 0x7ff86c009130 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7880 (unknown 0 0 1395959941) 0x7ff880006110 con 0x7ff86c009130 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:30.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.26:6789/0 conn(0x7ff86c009130 legacy=0x7ff86c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff87affd640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.26:6789/0 conn(0x7ff86c009130 legacy=0x7ff86c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:30.228 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:30.228 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:30.228 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:30.228 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:51:30.228 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:30.228 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 542126, 2024-08-19T12:51:30.228 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:30.229 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:30.230 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:30.230 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:30.230 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:30.230 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:30.230 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:30.230 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:30.230 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:30.230 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:30.230 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:30.230 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:30.230 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:30.230 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:30.231 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.231 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:30.231 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:30.231 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:30.231 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:30.231 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:30.231 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:30.231 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:30.231 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:30.231 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:30.231 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:30.231 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:30.232 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:30.233 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:30.233 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:30.233 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:30.233 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:30.233 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:30.233 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:30.233 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:30.233 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:30.233 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.234 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:30.235 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.239 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:30.239 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:30.239 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:30.239 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.239 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:30.242 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.242 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:30.242 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:30.242 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.242 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.242 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:30.243 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.244 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.244 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.244 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:30.244 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:30.244 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:30.244 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:30.245 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:30.245 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.245 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.245 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.245 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:30.245 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.246 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:51:30.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.248 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:51:30.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.249 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:30.249 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:30.249 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:30.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.249 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:30.249 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff88c638640 10 librados: watch_flush enter 2024-08-19T12:51:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff88c638640 10 librados: watch_flush exit 2024-08-19T12:51:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff88c638640 20 client.184109.objecter shutdown clearing up homeless session... 2024-08-19T12:51:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.208+0000 7ff88c638640 10 client.184109.objecter successfully canceled tick 2024-08-19T12:51:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.209+0000 7ff88c638640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.104:6800/899887506 conn(0x7ff868041b10 legacy=0x7ff868043fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.209+0000 7ff88c638640 10 monclient: shutdown 2024-08-19T12:51:30.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.209+0000 7ff88c638640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:30.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.209+0000 7ff88c638640 1 -- 172.21.15.26:0/3533558328 >> v1:172.21.15.104:6791/0 conn(0x7ff8840a5d50 legacy=0x7ff8841d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.209+0000 7ff88abd5640 1 -- 172.21.15.26:0/3533558328 reap_dead start 2024-08-19T12:51:30.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.209+0000 7ff88abd5640 1 -- 172.21.15.26:0/3533558328 reap_dead start 2024-08-19T12:51:30.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.209+0000 7ff88c638640 1 -- 172.21.15.26:0/3533558328 shutdown_connections 2024-08-19T12:51:30.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.209+0000 7ff88c638640 1 -- 172.21.15.26:0/3533558328 >> 172.21.15.26:0/3533558328 conn(0x7ff8840a8740 msgr2=0x7ff8840aa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:30.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.209+0000 7ff88c638640 1 -- 172.21.15.26:0/3533558328 shutdown_connections 2024-08-19T12:51:30.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.209+0000 7ff88c638640 1 -- 172.21.15.26:0/3533558328 wait complete. 2024-08-19T12:51:30.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.209+0000 7ff88c638640 1 librados: shutdown 2024-08-19T12:51:30.252 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:30.253 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:51:30.253 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:51:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.352+0000 7ff0b2e15640 10 monclient: get_monmap_and_config 2024-08-19T12:51:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.352+0000 7ff0b2e15640 10 monclient: build_initial_monmap 2024-08-19T12:51:30.353 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.352+0000 7ff0b2e15640 10 monclient: monmap: 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:30.353804+0000 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:30.353804+0000 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:30.354 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:30.355 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:30.355 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.353+0000 7ff0b2e15640 1 Processor -- start 2024-08-19T12:51:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.353+0000 7ff0b2e15640 1 -- start start 2024-08-19T12:51:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.353+0000 7ff0b2e15640 10 monclient: init 2024-08-19T12:51:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.354+0000 7ff0b2e15640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.354+0000 7ff0b2e15640 10 monclient: _add_conns ranks=[4,6,8,7,3,2,5,1,0] 2024-08-19T12:51:30.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.354+0000 7ff0b2e15640 10 monclient(hunting): picked mon.noname-c con 0x7ff0ac06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.354+0000 7ff0b2e15640 10 monclient(hunting): picked mon.noname-d con 0x7ff0ac0a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.354+0000 7ff0b2e15640 10 monclient(hunting): picked mon.noname-e con 0x7ff0ac0a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:51:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.354+0000 7ff0b2e15640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac136790 con 0x7ff0ac06e7b0 2024-08-19T12:51:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.354+0000 7ff0b2e15640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac09ddf0 con 0x7ff0ac0a5920 2024-08-19T12:51:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.354+0000 7ff0b2e15640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac09e0d0 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.354+0000 7ff0b2e15640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.354+0000 7ff0b2e15640 10 monclient(hunting): authenticate will time out at 1484.441360s 2024-08-19T12:51:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.354+0000 7ff0b0bb1640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7ff0ac06e7b0 0x7ff0ac136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41554/0 (socket says 172.21.15.26:41554) 2024-08-19T12:51:30.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.354+0000 7ff0b0bb1640 1 -- 172.21.15.26:0/1175967675 learned_addr learned my addr 172.21.15.26:0/1175967675 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:30.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0abfff640 1 -- 172.21.15.26:0/1175967675 >> v1:172.21.15.26:6792/0 conn(0x7ff0ac0a5920 legacy=0x7ff0ac13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0b13b2640 1 -- 172.21.15.26:0/1175967675 >> v1:172.21.15.26:6793/0 conn(0x7ff0ac0a5d50 legacy=0x7ff0ac141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0b0bb1640 1 -- 172.21.15.26:0/1175967675 >> v1:172.21.15.26:6791/0 conn(0x7ff0ac06e7b0 legacy=0x7ff0ac136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2107325602 0 0) 0x7ff0940014a0 con 0x7ff0ac0a5920 2024-08-19T12:51:30.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 10 monclient(hunting): my global_id is 164562 2024-08-19T12:51:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff098003520 con 0x7ff0ac0a5920 2024-08-19T12:51:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2825834660 0 0) 0x7ff0ac136790 con 0x7ff0ac06e7b0 2024-08-19T12:51:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 10 monclient(hunting): my global_id is 184108 2024-08-19T12:51:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0940014a0 con 0x7ff0ac06e7b0 2024-08-19T12:51:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434687710 0 0) 0x7ff0ac09e0d0 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 10 monclient(hunting): my global_id is 184112 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.356+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0ac136790 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3686904633 0 0) 0x7ff0940014a0 con 0x7ff0ac06e7b0 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0abfff640 1 --1- 172.21.15.26:0/1175967675 >> v1:172.21.15.26:6792/0 conn(0x7ff0ac0a5920 0x7ff0ac13e2f0 :-1 s=OPENED pgs=3480 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7ff094001d00 mon_map magic: 0 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 >> v1:172.21.15.26:6793/0 conn(0x7ff0ac0a5d50 legacy=0x7ff0ac141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 >> v1:172.21.15.26:6792/0 conn(0x7ff0ac0a5920 legacy=0x7ff0ac13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 1 monclient: found mon.noname-c 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:51:30.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ac09e3b0 con 0x7ff0ac06e7b0 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 10 monclient: _finish_auth 0 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff09c001e80 con 0x7ff0ac06e7b0 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0b2e15640 5 monclient: authenticate success, global_id 184108 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0b2e15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 10 monclient: dump: 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.364 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0b2e15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:30.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff094001d00 con 0x7ff0ac0a5920 2024-08-19T12:51:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:51:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff09c0020a0 con 0x7ff0ac06e7b0 2024-08-19T12:51:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 1 -- 172.21.15.26:0/1175967675 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff09c004f60 con 0x7ff0ac06e7b0 2024-08-19T12:51:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0ab7fe640 10 monclient: dump: 2024-08-19T12:51:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.366 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0b2e15640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:30.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0b2e15640 4 set_mon_vals no callback set 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0b2e15640 10 monclient: shutdown 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0b2e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1175967675 >> v1:172.21.15.26:6791/0 conn(0x7ff0ac06e7b0 legacy=0x7ff0ac136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1175967675 shutdown_connections 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1175967675 >> 172.21.15.26:0/1175967675 conn(0x7ff0ac0a8740 msgr2=0x7ff0ac0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1175967675 shutdown_connections 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.357+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1175967675 wait complete. 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.358+0000 7ff0b2e15640 10 monclient: build_initial_monmap 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.358+0000 7ff0b2e15640 10 monclient: monmap: 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:30.368 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.358+0000 7ff0b2e15640 1 librados: starting msgr at 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.358+0000 7ff0b2e15640 1 librados: starting objecter 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.358+0000 7ff0b2e15640 1 Processor -- start 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.358+0000 7ff0b2e15640 1 -- start start 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.358+0000 7ff0b2e15640 1 librados: setting wanted keys 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.358+0000 7ff0b2e15640 1 librados: calling monclient init 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.358+0000 7ff0b2e15640 10 monclient: init 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0b2e15640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0b2e15640 10 monclient: _add_conns ranks=[3,4,6,1,2,5,0,8,7] 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0b2e15640 10 monclient(hunting): picked mon.noname-g con 0x7ff0ac06e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0b2e15640 10 monclient(hunting): picked mon.noname-c con 0x7ff0ac0a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0b2e15640 10 monclient(hunting): picked mon.noname-d con 0x7ff0ac0a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0b2e15640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac138cd0 con 0x7ff0ac06e7b0 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0b2e15640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac13b030 con 0x7ff0ac0a5920 2024-08-19T12:51:30.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0b2e15640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac13a790 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0b2e15640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:30.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0b2e15640 10 monclient(hunting): authenticate will time out at 1484.446391s 2024-08-19T12:51:30.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0b13b2640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7ff0ac0a5d50 0x7ff0ac138250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51626/0 (socket says 172.21.15.26:51626) 2024-08-19T12:51:30.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0b13b2640 1 -- 172.21.15.26:0/1487139214 learned_addr learned my addr 172.21.15.26:0/1487139214 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:30.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.359+0000 7ff0abfff640 1 --1- 172.21.15.26:0/1487139214 >> v1:172.21.15.26:6791/0 conn(0x7ff0ac0a5920 0x7ff0ac137b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41570/0 (socket says 172.21.15.26:41570) 2024-08-19T12:51:30.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0b13b2640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.26:6792/0 conn(0x7ff0ac0a5d50 legacy=0x7ff0ac138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 client.?.objecter ms_handle_connect 0x7ff0ac0a5d50 2024-08-19T12:51:30.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:30.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 client.?.objecter ms_handle_connect 0x7ff0ac0a5920 2024-08-19T12:51:30.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:30.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0abfff640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.26:6791/0 conn(0x7ff0ac0a5920 legacy=0x7ff0ac137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2486780413 0 0) 0x7ff0ac13a790 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 monclient(hunting): my global_id is 164571 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff08c004eb0 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737496316 0 0) 0x7ff0ac13b030 con 0x7ff0ac0a5920 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 monclient(hunting): my global_id is 184117 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0ac13a790 con 0x7ff0ac0a5920 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 742873439 0 0) 0x7ff08c004eb0 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0ac13b030 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0b0bb1640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.104:6790/0 conn(0x7ff0ac06e7b0 legacy=0x7ff0ac137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 client.?.objecter ms_handle_connect 0x7ff0ac06e7b0 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:30.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff0a4003950 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.361+0000 7ff0a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:30.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:51:30.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient(hunting): dump: 2024-08-19T12:51:30.379 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.379 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.379 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.379 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.379 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.379 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.379 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.379 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.379 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1870353822 0 0) 0x7ff0ac13a790 con 0x7ff0ac0a5920 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff08c004eb0 con 0x7ff0ac0a5920 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff0940014a0 con 0x7ff0ac0a5920 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:30.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient(hunting): dump: 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.381 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2379488900 0 0) 0x7ff0ac13b030 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.26:6791/0 conn(0x7ff0ac0a5920 legacy=0x7ff0ac137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.104:6790/0 conn(0x7ff0ac06e7b0 legacy=0x7ff0ac137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient: _finish_hunting 0 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 1 monclient: found mon.d 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ac139830 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient: _finish_auth 0 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient: _check_auth_tickets 2024-08-19T12:51:30.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0b2e15640 5 monclient: authenticate success, global_id 164571 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0b2e15640 10 monclient: _renew_subs 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0b2e15640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff0ac139ac0 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0b2e15640 10 client.164571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0b2e15640 10 monclient: _renew_subs 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0b2e15640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7ff0ac1d7480 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0b2e15640 1 librados: init done 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0a4001c10 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff0a4003eb0 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0aaffd640 4 set_mon_vals no callback set 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.362+0000 7ff0a97fa640 10 monclient: dump: 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.384 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.385 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.385 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.385 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.385 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.363+0000 7ff0b2e15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:30.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.363+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff0900065c0 con 0x7ff090002860 2024-08-19T12:51:30.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.363+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff0a4004100 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.363+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(481..481 src has 350..481) ==== 6206+0+0 (unknown 3267959302 0 0) 0x7ff0a404e310 con 0x7ff0ac0a5d50 2024-08-19T12:51:30.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.363+0000 7ff0a97fa640 10 client.164571.objecter ms_dispatch 0x7ff0ac000b90 osd_map(481..481 src has 350..481) 2024-08-19T12:51:30.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.363+0000 7ff0a97fa640 3 client.164571.objecter handle_osd_map got epochs [481,481] > 0 2024-08-19T12:51:30.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.363+0000 7ff0a97fa640 3 client.164571.objecter handle_osd_map decoding full epoch 481 2024-08-19T12:51:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.363+0000 7ff0a97fa640 20 client.164571.objecter dump_active .. 0 homeless 2024-08-19T12:51:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.363+0000 7ff0abfff640 10 monclient: get_auth_request con 0x7ff08c041540 auth_method 0 2024-08-19T12:51:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0b0bb1640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.26:6790/0 conn(0x7ff090002860 legacy=0x7ff090004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 10 client.164571.objecter ms_handle_connect 0x7ff090002860 2024-08-19T12:51:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 10 client.164571.objecter resend_mon_ops 2024-08-19T12:51:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1810212809 0 0) 0x7ff0900065c0 con 0x7ff090002860 2024-08-19T12:51:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 10 monclient(hunting): my global_id is 26885 2024-08-19T12:51:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0ac13b030 con 0x7ff090002860 2024-08-19T12:51:30.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3782775043 0 0) 0x7ff0ac13b030 con 0x7ff090002860 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0900065c0 con 0x7ff090002860 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 10 client.164571.objecter ms_handle_connect 0x7ff08c041540 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff09c0018c0 con 0x7ff090002860 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.365+0000 7ff0a97fa640 10 monclient: dump: 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.387 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.388 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.388 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.388 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.388 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.388 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.388 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.388 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.388 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.388 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.366+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1915952278 0 0) 0x7ff0900065c0 con 0x7ff090002860 2024-08-19T12:51:30.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.366+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff09000a160 con 0x7ff090002860 2024-08-19T12:51:30.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.366+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff09c001b60 con 0x7ff090002860 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.366+0000 7ff0a97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.366+0000 7ff0a97fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.366+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.26:6790/0 conn(0x7ff090002860 legacy=0x7ff090004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.366+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.26:6790/0 conn(0x7ff090002860 legacy=0x7ff090004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.372+0000 7ff0b2e15640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.372+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff090009e50 con 0x7ff090009110 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.374+0000 7ff0b0bb1640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.26:6790/0 conn(0x7ff090009110 legacy=0x7ff0900095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.374+0000 7ff0a97fa640 10 client.164571.objecter ms_handle_connect 0x7ff090009110 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.374+0000 7ff0a97fa640 10 client.164571.objecter resend_mon_ops 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.374+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 576024541 0 0) 0x7ff090009e50 con 0x7ff090009110 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.374+0000 7ff0a97fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.374+0000 7ff0a97fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.374+0000 7ff0a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.374+0000 7ff0a97fa640 10 monclient(hunting): my global_id is 26894 2024-08-19T12:51:30.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.374+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0900065c0 con 0x7ff090009110 2024-08-19T12:51:30.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4186260775 0 0) 0x7ff0900065c0 con 0x7ff090009110 2024-08-19T12:51:30.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff090009e50 con 0x7ff090009110 2024-08-19T12:51:30.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff09c005610 con 0x7ff090009110 2024-08-19T12:51:30.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:30.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 10 monclient: dump: 2024-08-19T12:51:30.390 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.390 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.390 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2391334944 0 0) 0x7ff090009e50 con 0x7ff090009110 2024-08-19T12:51:30.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff09000c3f0 con 0x7ff090009110 2024-08-19T12:51:30.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 995068436) 0x7ff09c008780 con 0x7ff090009110 2024-08-19T12:51:30.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:30.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:30.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.26:6790/0 conn(0x7ff090009110 legacy=0x7ff0900095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0a97fa640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.26:6790/0 conn(0x7ff090009110 legacy=0x7ff0900095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:30.392 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:30.392 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 803430, 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:30.393 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:51:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:30.394 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:30.395 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:30.396 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:30.397 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:30.398 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.399 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.400 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.403 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.405 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:30.406 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:30.407 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.408 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:30.409 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.410 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.411 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:51:30.412 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.413 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:30.413 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0b2e15640 10 librados: watch_flush enter 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0b2e15640 10 librados: watch_flush exit 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0b2e15640 20 client.164571.objecter shutdown clearing up homeless session... 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0b2e15640 10 client.164571.objecter successfully canceled tick 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.104:6800/899887506 conn(0x7ff08c041540 legacy=0x7ff08c043a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0b2e15640 10 monclient: shutdown 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0b2e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1487139214 >> v1:172.21.15.26:6792/0 conn(0x7ff0ac0a5d50 legacy=0x7ff0ac138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0b13b2640 1 -- 172.21.15.26:0/1487139214 reap_dead start 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0b13b2640 1 -- 172.21.15.26:0/1487139214 reap_dead start 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.375+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1487139214 shutdown_connections 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.376+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1487139214 >> 172.21.15.26:0/1487139214 conn(0x7ff0ac0a8740 msgr2=0x7ff0ac0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.376+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1487139214 shutdown_connections 2024-08-19T12:51:30.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.376+0000 7ff0b2e15640 1 -- 172.21.15.26:0/1487139214 wait complete. 2024-08-19T12:51:30.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.376+0000 7ff0b2e15640 1 librados: shutdown 2024-08-19T12:51:30.416 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:30.416 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:51:30.416 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:51:30.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.516+0000 7f6b7d54c640 10 monclient: get_monmap_and_config 2024-08-19T12:51:30.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.516+0000 7f6b7d54c640 10 monclient: build_initial_monmap 2024-08-19T12:51:30.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.516+0000 7f6b7d54c640 10 monclient: monmap: 2024-08-19T12:51:30.517 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:30.517 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.517 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:30.516918+0000 2024-08-19T12:51:30.517 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:30.516918+0000 2024-08-19T12:51:30.517 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:30.517 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.517 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:30.517 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:30.518 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:30.518 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:30.518 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:30.518 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:30.518 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:30.518 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:30.518 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:30.518 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.516+0000 7f6b7d54c640 1 Processor -- start 2024-08-19T12:51:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.516+0000 7f6b7d54c640 1 -- start start 2024-08-19T12:51:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.516+0000 7f6b7d54c640 10 monclient: init 2024-08-19T12:51:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.517+0000 7f6b7d54c640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.517+0000 7f6b7d54c640 10 monclient: _add_conns ranks=[0,6,7,3,8,5,1,2,4] 2024-08-19T12:51:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.517+0000 7f6b7d54c640 10 monclient(hunting): picked mon.noname-a con 0x7f6b78073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:30.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.517+0000 7f6b7d54c640 10 monclient(hunting): picked mon.noname-d con 0x7f6b780a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.517+0000 7f6b7d54c640 10 monclient(hunting): picked mon.noname-i con 0x7f6b7813f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.517+0000 7f6b7d54c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b7809dae0 con 0x7f6b78073470 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.517+0000 7f6b7d54c640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b78136910 con 0x7f6b780a57a0 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.517+0000 7f6b7d54c640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b7809dd50 con 0x7f6b7813f2e0 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.517+0000 7f6b7d54c640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.517+0000 7f6b7d54c640 10 monclient(hunting): authenticate will time out at 1484.604242s 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.517+0000 7f6b76ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6b78073470 0x7f6b78136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60120/0 (socket says 172.21.15.26:60120) 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.517+0000 7f6b76ffd640 1 -- 172.21.15.26:0/722433726 learned_addr learned my addr 172.21.15.26:0/722433726 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b767fc640 1 -- 172.21.15.26:0/722433726 >> v1:172.21.15.26:6792/0 conn(0x7f6b780a57a0 legacy=0x7f6b780a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b76ffd640 1 -- 172.21.15.26:0/722433726 >> v1:172.21.15.26:6789/0 conn(0x7f6b78073470 legacy=0x7f6b78136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 1 -- 172.21.15.26:0/722433726 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2053213389 0 0) 0x7f6b78136910 con 0x7f6b780a57a0 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient(hunting): my global_id is 164580 2024-08-19T12:51:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 1 -- 172.21.15.26:0/722433726 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b64003520 con 0x7f6b780a57a0 2024-08-19T12:51:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 1 -- 172.21.15.26:0/722433726 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2807777702 0 0) 0x7f6b7809dae0 con 0x7f6b78073470 2024-08-19T12:51:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient(hunting): my global_id is 145269 2024-08-19T12:51:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 1 -- 172.21.15.26:0/722433726 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b78136910 con 0x7f6b78073470 2024-08-19T12:51:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b777fe640 1 -- 172.21.15.26:0/722433726 >> v1:172.21.15.104:6792/0 conn(0x7f6b7813f2e0 legacy=0x7f6b781416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 1 -- 172.21.15.26:0/722433726 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4085626483 0 0) 0x7f6b64003520 con 0x7f6b780a57a0 2024-08-19T12:51:30.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 1 -- 172.21.15.26:0/722433726 >> v1:172.21.15.104:6792/0 conn(0x7f6b7813f2e0 legacy=0x7f6b781416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 1 -- 172.21.15.26:0/722433726 >> v1:172.21.15.26:6789/0 conn(0x7f6b78073470 legacy=0x7f6b78136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 1 monclient: found mon.noname-d 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 1 -- 172.21.15.26:0/722433726 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7809e030 con 0x7f6b780a57a0 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient: _finish_auth 0 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b7d54c640 5 monclient: authenticate success, global_id 164580 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 1 -- 172.21.15.26:0/722433726 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6b60001b30 con 0x7f6b780a57a0 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b75ffb640 10 monclient: dump: 2024-08-19T12:51:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.528 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.519+0000 7f6b7d54c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b75ffb640 1 -- 172.21.15.26:0/722433726 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6b60003610 con 0x7f6b780a57a0 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b75ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b75ffb640 1 -- 172.21.15.26:0/722433726 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6b60003b80 con 0x7f6b780a57a0 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b7d54c640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b7d54c640 4 set_mon_vals no callback set 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b75ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b75ffb640 10 monclient: dump: 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.529 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b7d54c640 10 monclient: shutdown 2024-08-19T12:51:30.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b7d54c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b7d54c640 1 -- 172.21.15.26:0/722433726 >> v1:172.21.15.26:6792/0 conn(0x7f6b780a57a0 legacy=0x7f6b780a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b7d54c640 1 -- 172.21.15.26:0/722433726 shutdown_connections 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b7d54c640 1 -- 172.21.15.26:0/722433726 >> 172.21.15.26:0/722433726 conn(0x7f6b780abff0 msgr2=0x7f6b780ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b7d54c640 1 -- 172.21.15.26:0/722433726 shutdown_connections 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.520+0000 7f6b7d54c640 1 -- 172.21.15.26:0/722433726 wait complete. 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.521+0000 7f6b7d54c640 10 monclient: build_initial_monmap 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.521+0000 7f6b7d54c640 10 monclient: monmap: 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.531 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.521+0000 7f6b7d54c640 1 librados: starting msgr at 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.521+0000 7f6b7d54c640 1 librados: starting objecter 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.521+0000 7f6b7d54c640 1 Processor -- start 2024-08-19T12:51:30.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.521+0000 7f6b7d54c640 1 -- start start 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.521+0000 7f6b7d54c640 1 librados: setting wanted keys 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.521+0000 7f6b7d54c640 1 librados: calling monclient init 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.521+0000 7f6b7d54c640 10 monclient: init 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.521+0000 7f6b7d54c640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.522+0000 7f6b7d54c640 10 monclient: _add_conns ranks=[5,1,4,7,2,8,3,6,0] 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.522+0000 7f6b7d54c640 10 monclient(hunting): picked mon.noname-h con 0x7f6b78073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.522+0000 7f6b7d54c640 10 monclient(hunting): picked mon.noname-f con 0x7f6b780a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.522+0000 7f6b7d54c640 10 monclient(hunting): picked mon.noname-c con 0x7f6b7813f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.522+0000 7f6b7d54c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b7812ecb0 con 0x7f6b780a57a0 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.522+0000 7f6b7d54c640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b781312b0 con 0x7f6b7813f2e0 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.522+0000 7f6b7d54c640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b78131590 con 0x7f6b78073470 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.522+0000 7f6b7d54c640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:30.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.522+0000 7f6b7d54c640 10 monclient(hunting): authenticate will time out at 1484.609080s 2024-08-19T12:51:30.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.522+0000 7f6b777fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f6b7813f2e0 0x7f6b781d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41572/0 (socket says 172.21.15.26:41572) 2024-08-19T12:51:30.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.522+0000 7f6b777fe640 1 -- 172.21.15.26:0/20510832 learned_addr learned my addr 172.21.15.26:0/20510832 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:30.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b777fe640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.26:6791/0 conn(0x7f6b7813f2e0 legacy=0x7f6b781d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 client.?.objecter ms_handle_connect 0x7f6b7813f2e0 2024-08-19T12:51:30.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:30.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 458714529 0 0) 0x7f6b781312b0 con 0x7f6b7813f2e0 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): my global_id is 184126 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b58004eb0 con 0x7f6b7813f2e0 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 client.?.objecter ms_handle_connect 0x7f6b780a57a0 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b767fc640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.104:6789/0 conn(0x7f6b780a57a0 legacy=0x7f6b7812e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 client.?.objecter ms_handle_connect 0x7f6b78073470 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b76ffd640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.104:6791/0 conn(0x7f6b78073470 legacy=0x7f6b7812de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3611563046 0 0) 0x7f6b58004eb0 con 0x7f6b7813f2e0 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b781312b0 con 0x7f6b7813f2e0 2024-08-19T12:51:30.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6b68003990 con 0x7f6b7813f2e0 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): dump: 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.541 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.542 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.542 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.542 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.542 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.542 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:30.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3187184386 0 0) 0x7f6b78131590 con 0x7f6b78073470 2024-08-19T12:51:30.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): my global_id is 184118 2024-08-19T12:51:30.542 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b58004eb0 con 0x7f6b78073470 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 443947446 0 0) 0x7f6b7812ecb0 con 0x7f6b780a57a0 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient(hunting): my global_id is 184114 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b78131590 con 0x7f6b780a57a0 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2971298591 0 0) 0x7f6b781312b0 con 0x7f6b7813f2e0 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.104:6791/0 conn(0x7f6b78073470 legacy=0x7f6b7812de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.104:6789/0 conn(0x7f6b780a57a0 legacy=0x7f6b7812e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 monclient: found mon.c 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:30.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b78131870 con 0x7f6b7813f2e0 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b7d54c640 5 monclient: authenticate success, global_id 184126 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b7d54c640 10 monclient: _renew_subs 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b7d54c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b7d54c640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6b781332c0 con 0x7f6b7813f2e0 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b7d54c640 10 client.184126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b7d54c640 10 monclient: _renew_subs 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b7d54c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b7d54c640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f6b78131f70 con 0x7f6b7813f2e0 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.524+0000 7f6b7d54c640 1 librados: init done 2024-08-19T12:51:30.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.525+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6b68003ed0 con 0x7f6b7813f2e0 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.525+0000 7f6b6f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.525+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6b680043c0 con 0x7f6b7813f2e0 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.525+0000 7f6b6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.525+0000 7f6b6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.525+0000 7f6b757fa640 4 set_mon_vals no callback set 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.525+0000 7f6b6f7fe640 10 monclient: dump: 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.545 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.525+0000 7f6b7d54c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.525+0000 7f6b7d54c640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6b5c0065c0 con 0x7f6b5c002860 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.525+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6b68004560 con 0x7f6b7813f2e0 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.526+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(481..481 src has 350..481) ==== 6206+0+0 (unknown 3267959302 0 0) 0x7f6b6804e820 con 0x7f6b7813f2e0 2024-08-19T12:51:30.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.526+0000 7f6b6f7fe640 10 client.184126.objecter ms_dispatch 0x7f6b78000b90 osd_map(481..481 src has 350..481) 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.526+0000 7f6b6f7fe640 3 client.184126.objecter handle_osd_map got epochs [481,481] > 0 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.526+0000 7f6b6f7fe640 3 client.184126.objecter handle_osd_map decoding full epoch 481 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.526+0000 7f6b6f7fe640 20 client.184126.objecter dump_active .. 0 homeless 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.526+0000 7f6b767fc640 10 monclient: get_auth_request con 0x7f6b58041120 auth_method 0 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.527+0000 7f6b76ffd640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.26:6791/0 conn(0x7f6b5c002860 legacy=0x7f6b5c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.527+0000 7f6b6f7fe640 10 client.184126.objecter ms_handle_connect 0x7f6b5c002860 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.527+0000 7f6b6f7fe640 10 client.184126.objecter resend_mon_ops 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292384076 0 0) 0x7f6b5c0065c0 con 0x7f6b5c002860 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 10 monclient(hunting): my global_id is 184135 2024-08-19T12:51:30.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b781312b0 con 0x7f6b5c002860 2024-08-19T12:51:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1601540827 0 0) 0x7f6b781312b0 con 0x7f6b5c002860 2024-08-19T12:51:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b5c0065c0 con 0x7f6b5c002860 2024-08-19T12:51:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6b700047e0 con 0x7f6b5c002860 2024-08-19T12:51:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 10 monclient: dump: 2024-08-19T12:51:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.548 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1454041547 0 0) 0x7f6b5c0065c0 con 0x7f6b5c002860 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6b5c00a160 con 0x7f6b5c002860 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 10 client.184126.objecter ms_handle_connect 0x7f6b58041120 2024-08-19T12:51:30.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6b70006a20 con 0x7f6b5c002860 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.26:6791/0 conn(0x7f6b5c002860 legacy=0x7f6b5c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.528+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.26:6791/0 conn(0x7f6b5c002860 legacy=0x7f6b5c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.534+0000 7f6b7d54c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.534+0000 7f6b7d54c640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6b5c009e50 con 0x7f6b5c009110 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 10 client.184126.objecter ms_handle_connect 0x7f6b5c009110 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 10 client.184126.objecter resend_mon_ops 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b76ffd640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.26:6791/0 conn(0x7f6b5c009110 legacy=0x7f6b5c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3383440697 0 0) 0x7f6b5c009e50 con 0x7f6b5c009110 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 10 monclient(hunting): my global_id is 184144 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b5c0065c0 con 0x7f6b5c009110 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1631316490 0 0) 0x7f6b5c0065c0 con 0x7f6b5c009110 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b5c009e50 con 0x7f6b5c009110 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6b70006250 con 0x7f6b5c009110 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 10 monclient: dump: 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.551 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3376733012 0 0) 0x7f6b5c009e50 con 0x7f6b5c009110 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6b5c00c3f0 con 0x7f6b5c009110 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 802509467) 0x7f6b700064f0 con 0x7f6b5c009110 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.26:6791/0 conn(0x7f6b5c009110 legacy=0x7f6b5c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.537+0000 7f6b6f7fe640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.26:6791/0 conn(0x7f6b5c009110 legacy=0x7f6b5c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:30.553 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:30.553 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:30.553 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:30.553 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:30.553 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:30.553 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 16002, 2024-08-19T12:51:30.553 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:30.553 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:30.553 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:30.553 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:30.553 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:30.553 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:30.554 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:30.554 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:30.554 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:30.554 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:30.554 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.554 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:51:30.554 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:30.554 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:30.554 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:30.554 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:30.554 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:30.554 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:30.555 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:30.556 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:30.557 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:30.557 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:30.557 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:30.557 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:30.557 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:30.557 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:30.557 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:30.557 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:30.557 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:30.557 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:30.557 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:30.557 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:30.558 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:30.558 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:30.558 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:30.558 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:30.558 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:30.558 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:30.558 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:30.558 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:30.558 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:30.558 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:30.558 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:30.558 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:30.559 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:30.559 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.559 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:30.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.559 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:30.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.559 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:30.559 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:30.559 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.559 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.559 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:30.561 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.562 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:30.564 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.565 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:30.567 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:30.567 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.567 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.567 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:30.570 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:30.572 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.573 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:30.574 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 10 librados: watch_flush enter 2024-08-19T12:51:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 10 librados: watch_flush exit 2024-08-19T12:51:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 20 client.184126.objecter shutdown clearing up homeless session... 2024-08-19T12:51:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 10 client.184126.objecter successfully canceled tick 2024-08-19T12:51:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.104:6800/899887506 conn(0x7f6b58041120 legacy=0x7f6b580435e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 10 monclient: shutdown 2024-08-19T12:51:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 1 -- 172.21.15.26:0/20510832 >> v1:172.21.15.26:6791/0 conn(0x7f6b7813f2e0 legacy=0x7f6b781d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b777fe640 1 -- 172.21.15.26:0/20510832 reap_dead start 2024-08-19T12:51:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b777fe640 1 -- 172.21.15.26:0/20510832 reap_dead start 2024-08-19T12:51:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 1 -- 172.21.15.26:0/20510832 shutdown_connections 2024-08-19T12:51:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 1 -- 172.21.15.26:0/20510832 >> 172.21.15.26:0/20510832 conn(0x7f6b780abff0 msgr2=0x7f6b780a9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 1 -- 172.21.15.26:0/20510832 shutdown_connections 2024-08-19T12:51:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 1 -- 172.21.15.26:0/20510832 wait complete. 2024-08-19T12:51:30.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.538+0000 7f6b7d54c640 1 librados: shutdown 2024-08-19T12:51:30.577 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:30.577 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:51:30.578 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:51:30.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.676+0000 7f7b3766e640 10 monclient: get_monmap_and_config 2024-08-19T12:51:30.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.676+0000 7f7b3766e640 10 monclient: build_initial_monmap 2024-08-19T12:51:30.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.676+0000 7f7b3766e640 10 monclient: monmap: 2024-08-19T12:51:30.677 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:30.677 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.677 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:30.677569+0000 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:30.677569+0000 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:30.678 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.677+0000 7f7b3766e640 1 Processor -- start 2024-08-19T12:51:30.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.677+0000 7f7b3766e640 1 -- start start 2024-08-19T12:51:30.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.677+0000 7f7b3766e640 10 monclient: init 2024-08-19T12:51:30.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.677+0000 7f7b3766e640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:30.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.677+0000 7f7b3766e640 10 monclient: _add_conns ranks=[0,7,8,5,2,3,1,6,4] 2024-08-19T12:51:30.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.677+0000 7f7b3766e640 10 monclient(hunting): picked mon.noname-a con 0x7f7b3006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:30.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.677+0000 7f7b3766e640 10 monclient(hunting): picked mon.noname-i con 0x7f7b300a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:30.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.677+0000 7f7b3766e640 10 monclient(hunting): picked mon.noname-e con 0x7f7b300a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:51:30.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.678+0000 7f7b3766e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b30136790 con 0x7f7b3006e7b0 2024-08-19T12:51:30.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.678+0000 7f7b3766e640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b3009ddf0 con 0x7f7b300a5920 2024-08-19T12:51:30.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.678+0000 7f7b3766e640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b3009e0d0 con 0x7f7b300a5d50 2024-08-19T12:51:30.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.678+0000 7f7b3766e640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:30.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.678+0000 7f7b3766e640 10 monclient(hunting): authenticate will time out at 1484.765033s 2024-08-19T12:51:30.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.678+0000 7f7b35c0b640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f7b300a5d50 0x7f7b30141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:45512/0 (socket says 172.21.15.26:45512) 2024-08-19T12:51:30.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.678+0000 7f7b3540a640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7b3006e7b0 0x7f7b30136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60136/0 (socket says 172.21.15.26:60136) 2024-08-19T12:51:30.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.678+0000 7f7b35c0b640 1 -- 172.21.15.26:0/1125152626 learned_addr learned my addr 172.21.15.26:0/1125152626 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.680+0000 7f7b35c0b640 1 -- 172.21.15.26:0/1125152626 >> v1:172.21.15.26:6793/0 conn(0x7f7b300a5d50 legacy=0x7f7b30141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.680+0000 7f7b3540a640 1 -- 172.21.15.26:0/1125152626 >> v1:172.21.15.26:6789/0 conn(0x7f7b3006e7b0 legacy=0x7f7b30136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.680+0000 7f7b34c09640 1 -- 172.21.15.26:0/1125152626 >> v1:172.21.15.104:6792/0 conn(0x7f7b300a5920 legacy=0x7f7b3013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.680+0000 7f7b27fff640 1 -- 172.21.15.26:0/1125152626 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2325273640 0 0) 0x7f7b30136790 con 0x7f7b3006e7b0 2024-08-19T12:51:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.680+0000 7f7b27fff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.680+0000 7f7b27fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.680+0000 7f7b27fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.680+0000 7f7b27fff640 10 monclient(hunting): my global_id is 145278 2024-08-19T12:51:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.680+0000 7f7b27fff640 1 -- 172.21.15.26:0/1125152626 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b1c003520 con 0x7f7b3006e7b0 2024-08-19T12:51:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.680+0000 7f7b27fff640 1 -- 172.21.15.26:0/1125152626 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 343896210 0 0) 0x7f7b1c003520 con 0x7f7b3006e7b0 2024-08-19T12:51:30.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.680+0000 7f7b27fff640 1 -- 172.21.15.26:0/1125152626 >> v1:172.21.15.26:6793/0 conn(0x7f7b300a5d50 legacy=0x7f7b30141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.680+0000 7f7b27fff640 1 -- 172.21.15.26:0/1125152626 >> v1:172.21.15.104:6792/0 conn(0x7f7b300a5920 legacy=0x7f7b3013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 10 monclient: _finish_hunting 0 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 1 monclient: found mon.noname-a 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 1 -- 172.21.15.26:0/1125152626 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3009e3b0 con 0x7f7b3006e7b0 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 10 monclient: _finish_auth 0 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 10 monclient: _check_auth_tickets 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 1 -- 172.21.15.26:0/1125152626 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7b2c001a10 con 0x7f7b3006e7b0 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b3766e640 5 monclient: authenticate success, global_id 145278 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b3766e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:30.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 10 monclient: dump: 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.687 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b3766e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 1 -- 172.21.15.26:0/1125152626 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7b2c001d20 con 0x7f7b3006e7b0 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 1 -- 172.21.15.26:0/1125152626 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7b2c004b20 con 0x7f7b3006e7b0 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b3766e640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b3766e640 4 set_mon_vals no callback set 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b27fff640 10 monclient: dump: 2024-08-19T12:51:30.688 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.689 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b3766e640 10 monclient: shutdown 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b3766e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b3766e640 1 -- 172.21.15.26:0/1125152626 >> v1:172.21.15.26:6789/0 conn(0x7f7b3006e7b0 legacy=0x7f7b30136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b3766e640 1 -- 172.21.15.26:0/1125152626 shutdown_connections 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b3766e640 1 -- 172.21.15.26:0/1125152626 >> 172.21.15.26:0/1125152626 conn(0x7f7b300a8740 msgr2=0x7f7b300a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b3766e640 1 -- 172.21.15.26:0/1125152626 shutdown_connections 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.681+0000 7f7b3766e640 1 -- 172.21.15.26:0/1125152626 wait complete. 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.682+0000 7f7b3766e640 10 monclient: build_initial_monmap 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.682+0000 7f7b3766e640 10 monclient: monmap: 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:30.690 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.682+0000 7f7b3766e640 1 librados: starting msgr at 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.682+0000 7f7b3766e640 1 librados: starting objecter 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.682+0000 7f7b3766e640 1 Processor -- start 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.682+0000 7f7b3766e640 1 -- start start 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.682+0000 7f7b3766e640 1 librados: setting wanted keys 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.682+0000 7f7b3766e640 1 librados: calling monclient init 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.682+0000 7f7b3766e640 10 monclient: init 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.683+0000 7f7b3766e640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.683+0000 7f7b3766e640 10 monclient: _add_conns ranks=[6,3,0,7,1,8,2,4,5] 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.683+0000 7f7b3766e640 10 monclient(hunting): picked mon.noname-d con 0x7f7b3006e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.683+0000 7f7b3766e640 10 monclient(hunting): picked mon.noname-g con 0x7f7b300a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.683+0000 7f7b3766e640 10 monclient(hunting): picked mon.noname-a con 0x7f7b300a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.683+0000 7f7b3766e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b301349d0 con 0x7f7b300a5d50 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.683+0000 7f7b3766e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b300a2e00 con 0x7f7b300a5920 2024-08-19T12:51:30.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.683+0000 7f7b3766e640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b300a1bc0 con 0x7f7b3006e7b0 2024-08-19T12:51:30.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.683+0000 7f7b3766e640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:30.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.683+0000 7f7b3766e640 10 monclient(hunting): authenticate will time out at 1484.770288s 2024-08-19T12:51:30.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.683+0000 7f7b3540a640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f7b3006e7b0 0x7f7b300a48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51658/0 (socket says 172.21.15.26:51658) 2024-08-19T12:51:30.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.683+0000 7f7b3540a640 1 -- 172.21.15.26:0/1637083310 learned_addr learned my addr 172.21.15.26:0/1637083310 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:30.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b35c0b640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.26:6789/0 conn(0x7f7b300a5d50 legacy=0x7f7b301d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b3540a640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.26:6792/0 conn(0x7f7b3006e7b0 legacy=0x7f7b300a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b25ffb640 10 client.?.objecter ms_handle_connect 0x7f7b300a5d50 2024-08-19T12:51:30.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b25ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:30.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b25ffb640 10 client.?.objecter ms_handle_connect 0x7f7b3006e7b0 2024-08-19T12:51:30.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b25ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b34c09640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.104:6790/0 conn(0x7f7b300a5920 legacy=0x7f7b301d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b25ffb640 10 client.?.objecter ms_handle_connect 0x7f7b300a5920 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b25ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3952346127 0 0) 0x7f7b301349d0 con 0x7f7b300a5d50 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b25ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b25ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b25ffb640 10 monclient(hunting): my global_id is 145287 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b14004eb0 con 0x7f7b300a5d50 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.685+0000 7f7b35c0b640 1 --1- 172.21.15.26:0/1637083310 >> v1:172.21.15.26:6789/0 conn(0x7f7b300a5d50 0x7f7b301d69a0 :-1 s=OPENED pgs=3836 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f7b14004eb0 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.686+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 677744266 0 0) 0x7f7b14004eb0 con 0x7f7b300a5d50 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.686+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b301349d0 con 0x7f7b300a5d50 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.686+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7b28001fe0 con 0x7f7b300a5d50 2024-08-19T12:51:30.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.686+0000 7f7b25ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.686+0000 7f7b25ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.686+0000 7f7b25ffb640 10 monclient(hunting): dump: 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.696 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.697 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.697 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.697 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.686+0000 7f7b25ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.874+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1247705575 0 0) 0x7f7b300a1bc0 con 0x7f7b3006e7b0 2024-08-19T12:51:30.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.874+0000 7f7b25ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.874+0000 7f7b25ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.874+0000 7f7b25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.874+0000 7f7b25ffb640 10 monclient(hunting): my global_id is 164589 2024-08-19T12:51:30.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.874+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b14004eb0 con 0x7f7b3006e7b0 2024-08-19T12:51:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 659754219 0 0) 0x7f7b14004eb0 con 0x7f7b3006e7b0 2024-08-19T12:51:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b300a1bc0 con 0x7f7b3006e7b0 2024-08-19T12:51:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7b2c001990 con 0x7f7b3006e7b0 2024-08-19T12:51:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:30.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 10 monclient(hunting): dump: 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.880 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 924181473 0 0) 0x7f7b300a1bc0 con 0x7f7b3006e7b0 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.104:6790/0 conn(0x7f7b300a5920 legacy=0x7f7b301d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.26:6789/0 conn(0x7f7b300a5d50 legacy=0x7f7b301d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 1 monclient: found mon.d 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b300a2b20 con 0x7f7b3006e7b0 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 10 monclient: _finish_auth 0 2024-08-19T12:51:30.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:51:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.875+0000 7f7b25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.876+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7b2c004b40 con 0x7f7b3006e7b0 2024-08-19T12:51:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.876+0000 7f7b25ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.876+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7b2c005410 con 0x7f7b3006e7b0 2024-08-19T12:51:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.876+0000 7f7b25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.876+0000 7f7b25ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:30.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.876+0000 7f7b25ffb640 10 monclient: dump: 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.883 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.876+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1129706171 0 0) 0x7f7b301349d0 con 0x7f7b300a5d50 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.876+0000 7f7b25ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.876+0000 7f7b277fe640 4 set_mon_vals no callback set 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.877+0000 7f7b3766e640 5 monclient: authenticate success, global_id 164589 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.877+0000 7f7b3766e640 10 monclient: _renew_subs 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.877+0000 7f7b3766e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.877+0000 7f7b3766e640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7b300a2810 con 0x7f7b3006e7b0 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.877+0000 7f7b3766e640 10 client.164589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.877+0000 7f7b3766e640 10 monclient: _renew_subs 2024-08-19T12:51:30.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.877+0000 7f7b3766e640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.877+0000 7f7b3766e640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f7b300a51e0 con 0x7f7b3006e7b0 2024-08-19T12:51:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.877+0000 7f7b3766e640 1 librados: init done 2024-08-19T12:51:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.878+0000 7f7b3766e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:30.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.878+0000 7f7b3766e640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7b30136740 con 0x7f7b30132a20 2024-08-19T12:51:30.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.879+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f7b2c011870 con 0x7f7b3006e7b0 2024-08-19T12:51:30.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.880+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(482..482 src has 350..482) ==== 6214+0+0 (unknown 1180833201 0 0) 0x7f7b2c04cb00 con 0x7f7b3006e7b0 2024-08-19T12:51:30.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.880+0000 7f7b25ffb640 10 client.164589.objecter ms_dispatch 0x7f7b30000b90 osd_map(482..482 src has 350..482) 2024-08-19T12:51:30.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.880+0000 7f7b25ffb640 3 client.164589.objecter handle_osd_map got epochs [482,482] > 0 2024-08-19T12:51:30.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.880+0000 7f7b25ffb640 3 client.164589.objecter handle_osd_map decoding full epoch 482 2024-08-19T12:51:30.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.880+0000 7f7b25ffb640 20 client.164589.objecter dump_active .. 0 homeless 2024-08-19T12:51:30.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.881+0000 7f7b35c0b640 10 monclient: get_auth_request con 0x7f7b140414f0 auth_method 0 2024-08-19T12:51:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.882+0000 7f7b34c09640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.26:6792/0 conn(0x7f7b30132a20 legacy=0x7f7b301253f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 10 client.164589.objecter ms_handle_connect 0x7f7b30132a20 2024-08-19T12:51:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 10 client.164589.objecter resend_mon_ops 2024-08-19T12:51:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129712953 0 0) 0x7f7b300a2e00 con 0x7f7b30132a20 2024-08-19T12:51:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 10 monclient(hunting): my global_id is 164598 2024-08-19T12:51:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b301349d0 con 0x7f7b30132a20 2024-08-19T12:51:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 10 client.164589.objecter ms_handle_connect 0x7f7b140414f0 2024-08-19T12:51:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2492898640 0 0) 0x7f7b301349d0 con 0x7f7b30132a20 2024-08-19T12:51:30.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b300a2e00 con 0x7f7b30132a20 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7b20001be0 con 0x7f7b30132a20 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.883+0000 7f7b25ffb640 10 monclient: dump: 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.889 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.884+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 886505331 0 0) 0x7f7b300a2e00 con 0x7f7b30132a20 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.884+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b30077810 con 0x7f7b30132a20 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.884+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7b200057c0 con 0x7f7b30132a20 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.884+0000 7f7b25ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.884+0000 7f7b25ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.884+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.26:6792/0 conn(0x7f7b30132a20 legacy=0x7f7b301253f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.884+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.26:6792/0 conn(0x7f7b30132a20 legacy=0x7f7b301253f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:30.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.891+0000 7f7b3766e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:30.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.891+0000 7f7b3766e640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7b301345a0 con 0x7f7b3002e1d0 2024-08-19T12:51:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b34c09640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.26:6792/0 conn(0x7f7b3002e1d0 legacy=0x7f7b30125b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 10 client.164589.objecter ms_handle_connect 0x7f7b3002e1d0 2024-08-19T12:51:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 10 client.164589.objecter resend_mon_ops 2024-08-19T12:51:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4263837163 0 0) 0x7f7b301345a0 con 0x7f7b3002e1d0 2024-08-19T12:51:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 10 monclient(hunting): my global_id is 164607 2024-08-19T12:51:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b300a2e00 con 0x7f7b3002e1d0 2024-08-19T12:51:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2994747956 0 0) 0x7f7b300a2e00 con 0x7f7b3002e1d0 2024-08-19T12:51:30.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b301345a0 con 0x7f7b3002e1d0 2024-08-19T12:51:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7b20003e10 con 0x7f7b3002e1d0 2024-08-19T12:51:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.893+0000 7f7b25ffb640 10 monclient: dump: 2024-08-19T12:51:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:30.896 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1592906919 0 0) 0x7f7b301345a0 con 0x7f7b3002e1d0 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7b30132e00 con 0x7f7b3002e1d0 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 51752756) 0x7f7b20004110 con 0x7f7b3002e1d0 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b25ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b25ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:30.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.26:6792/0 conn(0x7f7b3002e1d0 legacy=0x7f7b30125b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b25ffb640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.26:6792/0 conn(0x7f7b3002e1d0 legacy=0x7f7b30125b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:30.898 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 274448, 2024-08-19T12:51:30.898 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:30.899 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:30.900 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:30.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:30.902 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:30.903 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.905 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.907 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:30.910 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:30.911 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.912 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:30.915 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:30.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:30.917 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:51:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:30.918 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:30.918 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b3766e640 10 librados: watch_flush enter 2024-08-19T12:51:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b3766e640 10 librados: watch_flush exit 2024-08-19T12:51:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b3766e640 20 client.164589.objecter shutdown clearing up homeless session... 2024-08-19T12:51:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b3766e640 10 client.164589.objecter successfully canceled tick 2024-08-19T12:51:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b3766e640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.104:6800/899887506 conn(0x7f7b140414f0 legacy=0x7f7b140439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b3766e640 10 monclient: shutdown 2024-08-19T12:51:30.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b3766e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b3766e640 1 -- 172.21.15.26:0/1637083310 >> v1:172.21.15.26:6792/0 conn(0x7f7b3006e7b0 legacy=0x7f7b300a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b35c0b640 1 -- 172.21.15.26:0/1637083310 reap_dead start 2024-08-19T12:51:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b35c0b640 1 -- 172.21.15.26:0/1637083310 reap_dead start 2024-08-19T12:51:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b3766e640 1 -- 172.21.15.26:0/1637083310 shutdown_connections 2024-08-19T12:51:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.894+0000 7f7b3766e640 1 -- 172.21.15.26:0/1637083310 >> 172.21.15.26:0/1637083310 conn(0x7f7b300a8740 msgr2=0x7f7b300aa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.895+0000 7f7b3766e640 1 -- 172.21.15.26:0/1637083310 shutdown_connections 2024-08-19T12:51:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.895+0000 7f7b3766e640 1 -- 172.21.15.26:0/1637083310 wait complete. 2024-08-19T12:51:30.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:30.895+0000 7f7b3766e640 1 librados: shutdown 2024-08-19T12:51:30.921 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:30.921 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:51:30.922 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:51:31.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.020+0000 7f2156f86640 10 monclient: get_monmap_and_config 2024-08-19T12:51:31.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.020+0000 7f2156f86640 10 monclient: build_initial_monmap 2024-08-19T12:51:31.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.020+0000 7f2156f86640 10 monclient: monmap: 2024-08-19T12:51:31.021 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:31.021 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:31.021631+0000 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:31.021631+0000 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:31.022 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:31.023 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.021+0000 7f2156f86640 1 Processor -- start 2024-08-19T12:51:31.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.021+0000 7f2156f86640 1 -- start start 2024-08-19T12:51:31.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.021+0000 7f2156f86640 10 monclient: init 2024-08-19T12:51:31.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.021+0000 7f2156f86640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:31.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.021+0000 7f2156f86640 10 monclient: _add_conns ranks=[0,2,4,7,6,3,1,5,8] 2024-08-19T12:51:31.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.021+0000 7f2156f86640 10 monclient(hunting): picked mon.noname-a con 0x7f215006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:31.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.021+0000 7f2156f86640 10 monclient(hunting): picked mon.noname-b con 0x7f21500a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:31.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.022+0000 7f2156f86640 10 monclient(hunting): picked mon.noname-c con 0x7f21500a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:31.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.022+0000 7f2156f86640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2150136790 con 0x7f215006e7b0 2024-08-19T12:51:31.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.022+0000 7f2156f86640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f215009ddf0 con 0x7f21500a5920 2024-08-19T12:51:31.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.022+0000 7f2156f86640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f215009e0d0 con 0x7f21500a5d50 2024-08-19T12:51:31.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.022+0000 7f2156f86640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:31.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.022+0000 7f2156f86640 10 monclient(hunting): authenticate will time out at 1485.109057s 2024-08-19T12:51:31.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.022+0000 7f2155523640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f21500a5d50 0x7f2150141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41604/0 (socket says 172.21.15.26:41604) 2024-08-19T12:51:31.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.022+0000 7f2154d22640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f215006e7b0 0x7f2150136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60150/0 (socket says 172.21.15.26:60150) 2024-08-19T12:51:31.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.022+0000 7f2155523640 1 -- 172.21.15.26:0/1712213001 learned_addr learned my addr 172.21.15.26:0/1712213001 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:31.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.022+0000 7f214ffff640 1 --1- 172.21.15.26:0/1712213001 >> v1:172.21.15.26:6790/0 conn(0x7f21500a5920 0x7f215013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54542/0 (socket says 172.21.15.26:54542) 2024-08-19T12:51:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f2154d22640 1 -- 172.21.15.26:0/1712213001 >> v1:172.21.15.26:6789/0 conn(0x7f215006e7b0 legacy=0x7f2150136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f2155523640 1 -- 172.21.15.26:0/1712213001 >> v1:172.21.15.26:6791/0 conn(0x7f21500a5d50 legacy=0x7f2150141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214ffff640 1 -- 172.21.15.26:0/1712213001 >> v1:172.21.15.26:6790/0 conn(0x7f21500a5920 legacy=0x7f215013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1336235393 0 0) 0x7f2150136790 con 0x7f215006e7b0 2024-08-19T12:51:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient(hunting): my global_id is 145296 2024-08-19T12:51:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2140003520 con 0x7f215006e7b0 2024-08-19T12:51:31.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1574308615 0 0) 0x7f213c001450 con 0x7f21500a5920 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient(hunting): my global_id is 26903 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2150136790 con 0x7f21500a5920 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3526022625 0 0) 0x7f215009e0d0 con 0x7f21500a5d50 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient(hunting): my global_id is 184153 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f213c001450 con 0x7f21500a5d50 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2112483693 0 0) 0x7f2150136790 con 0x7f21500a5920 2024-08-19T12:51:31.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 >> v1:172.21.15.26:6791/0 conn(0x7f21500a5d50 legacy=0x7f2150141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 >> v1:172.21.15.26:6789/0 conn(0x7f215006e7b0 legacy=0x7f2150136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 1 monclient: found mon.noname-b 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215009e3b0 con 0x7f21500a5920 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f213c001c10 con 0x7f21500a5920 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f2156f86640 5 monclient: authenticate success, global_id 26903 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f2156f86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f214f7fe640 10 monclient: dump: 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.032 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.024+0000 7f2156f86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f213c0047e0 con 0x7f21500a5920 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f214f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f214f7fe640 1 -- 172.21.15.26:0/1712213001 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f213c004d50 con 0x7f21500a5920 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f2156f86640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f2156f86640 4 set_mon_vals no callback set 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f214f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f214f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f214f7fe640 10 monclient: dump: 2024-08-19T12:51:31.033 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.034 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f2156f86640 10 monclient: shutdown 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f2156f86640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f2156f86640 1 -- 172.21.15.26:0/1712213001 >> v1:172.21.15.26:6790/0 conn(0x7f21500a5920 legacy=0x7f215013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f2156f86640 1 -- 172.21.15.26:0/1712213001 shutdown_connections 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f2156f86640 1 -- 172.21.15.26:0/1712213001 >> 172.21.15.26:0/1712213001 conn(0x7f21500a8740 msgr2=0x7f21500a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f2156f86640 1 -- 172.21.15.26:0/1712213001 shutdown_connections 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.025+0000 7f2156f86640 1 -- 172.21.15.26:0/1712213001 wait complete. 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.026+0000 7f2156f86640 10 monclient: build_initial_monmap 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.026+0000 7f2156f86640 10 monclient: monmap: 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:31.035 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:31.036 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.026+0000 7f2156f86640 1 librados: starting msgr at 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.026+0000 7f2156f86640 1 librados: starting objecter 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.026+0000 7f2156f86640 1 Processor -- start 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.026+0000 7f2156f86640 1 -- start start 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.026+0000 7f2156f86640 1 librados: setting wanted keys 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.026+0000 7f2156f86640 1 librados: calling monclient init 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.026+0000 7f2156f86640 10 monclient: init 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f2156f86640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f2156f86640 10 monclient: _add_conns ranks=[4,0,6,5,3,1,2,8,7] 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f2156f86640 10 monclient(hunting): picked mon.noname-c con 0x7f215006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f2156f86640 10 monclient(hunting): picked mon.noname-a con 0x7f21500a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f2156f86640 10 monclient(hunting): picked mon.noname-d con 0x7f21500a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f2156f86640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2150134530 con 0x7f21500a5920 2024-08-19T12:51:31.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f2156f86640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f215013b5e0 con 0x7f215006e7b0 2024-08-19T12:51:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f2156f86640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f215013a0a0 con 0x7f21500a5d50 2024-08-19T12:51:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f2156f86640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f2156f86640 10 monclient(hunting): authenticate will time out at 1485.114156s 2024-08-19T12:51:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f2154d22640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f215006e7b0 0x7f21501ce2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41606/0 (socket says 172.21.15.26:41606) 2024-08-19T12:51:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f2154d22640 1 -- 172.21.15.26:0/4242292123 learned_addr learned my addr 172.21.15.26:0/4242292123 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:31.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.027+0000 7f214ffff640 1 --1- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6789/0 conn(0x7f21500a5920 0x7f2150141e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60156/0 (socket says 172.21.15.26:60156) 2024-08-19T12:51:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f2154d22640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6791/0 conn(0x7f215006e7b0 legacy=0x7f21501ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 client.?.objecter ms_handle_connect 0x7f215006e7b0 2024-08-19T12:51:31.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214ffff640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6789/0 conn(0x7f21500a5920 legacy=0x7f2150141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 client.?.objecter ms_handle_connect 0x7f21500a5920 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 client.?.objecter ms_handle_connect 0x7f21500a5d50 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f2155523640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6792/0 conn(0x7f21500a5d50 legacy=0x7f21501d6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1524028154 0 0) 0x7f215013b5e0 con 0x7f215006e7b0 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): my global_id is 184162 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2134004eb0 con 0x7f215006e7b0 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659949244 0 0) 0x7f215009e3b0 con 0x7f21500a5920 2024-08-19T12:51:31.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): my global_id is 145305 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f215013b5e0 con 0x7f21500a5920 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2231460697 0 0) 0x7f215013a0a0 con 0x7f21500a5d50 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): my global_id is 164616 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f215009e3b0 con 0x7f21500a5d50 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2159650708 0 0) 0x7f215013b5e0 con 0x7f21500a5920 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f215013a0a0 con 0x7f21500a5920 2024-08-19T12:51:31.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f213c004cd0 con 0x7f21500a5920 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): dump: 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.047 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4004964000 0 0) 0x7f2134004eb0 con 0x7f215006e7b0 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f215013b5e0 con 0x7f215006e7b0 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2904581667 0 0) 0x7f215009e3b0 con 0x7f21500a5d50 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2134004eb0 con 0x7f21500a5d50 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f21440047e0 con 0x7f215006e7b0 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:31.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): dump: 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.049 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2148001af0 con 0x7f21500a5d50 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.029+0000 7f214d7fa640 10 monclient(hunting): dump: 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.050 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1199341975 0 0) 0x7f215013a0a0 con 0x7f21500a5920 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6792/0 conn(0x7f21500a5d50 legacy=0x7f21501d6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6791/0 conn(0x7f215006e7b0 legacy=0x7f21501ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 10 monclient: _finish_hunting 0 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 1 monclient: found mon.a 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2150134860 con 0x7f21500a5920 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 10 monclient: _finish_auth 0 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 10 monclient: _check_auth_tickets 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f2156f86640 5 monclient: authenticate success, global_id 145305 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f2156f86640 10 monclient: _renew_subs 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f2156f86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f2156f86640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2150139540 con 0x7f21500a5920 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f2156f86640 10 client.145305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:31.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f2156f86640 10 monclient: _renew_subs 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f2156f86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f2156f86640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2150142710 con 0x7f21500a5920 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f2156f86640 1 librados: init done 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f213c0047e0 con 0x7f21500a5920 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f213c001c30 con 0x7f21500a5920 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214effd640 4 set_mon_vals no callback set 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f214d7fa640 10 monclient: dump: 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.053 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f2156f86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.030+0000 7f2156f86640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f21300065c0 con 0x7f2130002860 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.031+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f213c011940 con 0x7f21500a5920 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.031+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(482..482 src has 350..482) ==== 6214+0+0 (unknown 1180833201 0 0) 0x7f213c04e400 con 0x7f21500a5920 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.031+0000 7f214d7fa640 10 client.145305.objecter ms_dispatch 0x7f2150000b90 osd_map(482..482 src has 350..482) 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.031+0000 7f214d7fa640 3 client.145305.objecter handle_osd_map got epochs [482,482] > 0 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.031+0000 7f214d7fa640 3 client.145305.objecter handle_osd_map decoding full epoch 482 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.031+0000 7f214d7fa640 20 client.145305.objecter dump_active .. 0 homeless 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.031+0000 7f2155523640 10 monclient: get_auth_request con 0x7f2134041a50 auth_method 0 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 10 client.145305.objecter ms_handle_connect 0x7f2130002860 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 10 client.145305.objecter resend_mon_ops 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f2154d22640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6793/0 conn(0x7f2130002860 legacy=0x7f2130004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2781403073 0 0) 0x7f21300065c0 con 0x7f2130002860 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 10 monclient(hunting): my global_id is 184121 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f215013a0a0 con 0x7f2130002860 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1228300338 0 0) 0x7f215013a0a0 con 0x7f2130002860 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21300065c0 con 0x7f2130002860 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f21440053a0 con 0x7f2130002860 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 10 monclient: dump: 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1274490457 0 0) 0x7f21300065c0 con 0x7f2130002860 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.033+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f213000a160 con 0x7f2130002860 2024-08-19T12:51:31.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.034+0000 7f214d7fa640 10 client.145305.objecter ms_handle_connect 0x7f2134041a50 2024-08-19T12:51:31.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.034+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f21440075e0 con 0x7f2130002860 2024-08-19T12:51:31.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.034+0000 7f214d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.034+0000 7f214d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:31.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.034+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6793/0 conn(0x7f2130002860 legacy=0x7f2130004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.034+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6793/0 conn(0x7f2130002860 legacy=0x7f2130004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:31.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.040+0000 7f2156f86640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:31.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.040+0000 7f2156f86640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2130009e50 con 0x7f2130009110 2024-08-19T12:51:31.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.042+0000 7f2154d22640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6793/0 conn(0x7f2130009110 legacy=0x7f21300095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.042+0000 7f214d7fa640 10 client.145305.objecter ms_handle_connect 0x7f2130009110 2024-08-19T12:51:31.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.042+0000 7f214d7fa640 10 client.145305.objecter resend_mon_ops 2024-08-19T12:51:31.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.042+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3477086123 0 0) 0x7f2130009e50 con 0x7f2130009110 2024-08-19T12:51:31.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.042+0000 7f214d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.042+0000 7f214d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.042+0000 7f214d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.042+0000 7f214d7fa640 10 monclient(hunting): my global_id is 184130 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.042+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21300065c0 con 0x7f2130009110 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1132245218 0 0) 0x7f21300065c0 con 0x7f2130009110 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2130009e50 con 0x7f2130009110 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2144006e10 con 0x7f2130009110 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 10 monclient: dump: 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.059 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 864057018 0 0) 0x7f2130009e50 con 0x7f2130009110 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f213000c3a0 con 0x7f2130009110 2024-08-19T12:51:31.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7710 (unknown 0 0 3612019741) 0x7f21440070f0 con 0x7f2130009110 2024-08-19T12:51:31.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:31.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:31.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6793/0 conn(0x7f2130009110 legacy=0x7f21300095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f214d7fa640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6793/0 conn(0x7f2130009110 legacy=0x7f21300095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:31.061 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:31.061 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:31.061 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:31.061 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:31.061 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:31.061 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 16509, 2024-08-19T12:51:31.062 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:31.062 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:31.062 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:31.062 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:31.062 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:31.062 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:31.062 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:31.062 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:31.062 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:31.062 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:31.062 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.062 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.063 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.064 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:31.065 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.066 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.067 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:31.068 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.069 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:31.073 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.077 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.078 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:31.079 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:31.080 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:31.081 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:31.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.081 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:31.081 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:31.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f2156f86640 10 librados: watch_flush enter 2024-08-19T12:51:31.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f2156f86640 10 librados: watch_flush exit 2024-08-19T12:51:31.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f2156f86640 20 client.145305.objecter shutdown clearing up homeless session... 2024-08-19T12:51:31.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f2156f86640 10 client.145305.objecter successfully canceled tick 2024-08-19T12:51:31.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f2156f86640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.104:6800/899887506 conn(0x7f2134041a50 legacy=0x7f2134043f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f2156f86640 10 monclient: shutdown 2024-08-19T12:51:31.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f2156f86640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:31.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f2156f86640 1 -- 172.21.15.26:0/4242292123 >> v1:172.21.15.26:6789/0 conn(0x7f21500a5920 legacy=0x7f2150141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f2155523640 1 -- 172.21.15.26:0/4242292123 reap_dead start 2024-08-19T12:51:31.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f2155523640 1 -- 172.21.15.26:0/4242292123 reap_dead start 2024-08-19T12:51:31.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f2156f86640 1 -- 172.21.15.26:0/4242292123 shutdown_connections 2024-08-19T12:51:31.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.043+0000 7f2156f86640 1 -- 172.21.15.26:0/4242292123 >> 172.21.15.26:0/4242292123 conn(0x7f21500a8740 msgr2=0x7f21500a51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:31.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.044+0000 7f2156f86640 1 -- 172.21.15.26:0/4242292123 shutdown_connections 2024-08-19T12:51:31.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.044+0000 7f2156f86640 1 -- 172.21.15.26:0/4242292123 wait complete. 2024-08-19T12:51:31.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.044+0000 7f2156f86640 1 librados: shutdown 2024-08-19T12:51:31.084 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:31.084 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:51:31.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:51:31.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.183+0000 7f2533656640 10 monclient: get_monmap_and_config 2024-08-19T12:51:31.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.183+0000 7f2533656640 10 monclient: build_initial_monmap 2024-08-19T12:51:31.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.183+0000 7f2533656640 10 monclient: monmap: 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:31.184606+0000 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:31.184606+0000 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:31.185 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:31.186 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:31.186 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.184+0000 7f2533656640 1 Processor -- start 2024-08-19T12:51:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.184+0000 7f2533656640 1 -- start start 2024-08-19T12:51:31.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.184+0000 7f2533656640 10 monclient: init 2024-08-19T12:51:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.185+0000 7f2533656640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.185+0000 7f2533656640 10 monclient: _add_conns ranks=[5,7,4,1,2,6,8,3,0] 2024-08-19T12:51:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.185+0000 7f2533656640 10 monclient(hunting): picked mon.noname-h con 0x7f252c06e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.185+0000 7f2533656640 10 monclient(hunting): picked mon.noname-i con 0x7f252c0a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.185+0000 7f2533656640 10 monclient(hunting): picked mon.noname-c con 0x7f252c0a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.185+0000 7f2533656640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c136790 con 0x7f252c0a5d50 2024-08-19T12:51:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.185+0000 7f2533656640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c09ddf0 con 0x7f252c06e7b0 2024-08-19T12:51:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.185+0000 7f2533656640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c09e0d0 con 0x7f252c0a5920 2024-08-19T12:51:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.185+0000 7f2533656640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.185+0000 7f2533656640 10 monclient(hunting): authenticate will time out at 1485.272118s 2024-08-19T12:51:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.185+0000 7f2531bf3640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f252c0a5d50 0x7f252c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41618/0 (socket says 172.21.15.26:41618) 2024-08-19T12:51:31.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.185+0000 7f2531bf3640 1 -- 172.21.15.26:0/3203180635 learned_addr learned my addr 172.21.15.26:0/3203180635 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f2531bf3640 1 -- 172.21.15.26:0/3203180635 >> v1:172.21.15.26:6791/0 conn(0x7f252c0a5d50 legacy=0x7f252c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f25313f2640 1 -- 172.21.15.26:0/3203180635 >> v1:172.21.15.104:6791/0 conn(0x7f252c06e7b0 legacy=0x7f252c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f2530bf1640 1 -- 172.21.15.26:0/3203180635 >> v1:172.21.15.104:6792/0 conn(0x7f252c0a5920 legacy=0x7f252c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 49785261 0 0) 0x7f252c136790 con 0x7f252c0a5d50 2024-08-19T12:51:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient(hunting): my global_id is 184171 2024-08-19T12:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2514003520 con 0x7f252c0a5d50 2024-08-19T12:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1682946741 0 0) 0x7f252c09e0d0 con 0x7f252c0a5920 2024-08-19T12:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient(hunting): my global_id is 135853 2024-08-19T12:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f252c136790 con 0x7f252c0a5920 2024-08-19T12:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3095533545 0 0) 0x7f25240014a0 con 0x7f252c06e7b0 2024-08-19T12:51:31.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient(hunting): my global_id is 184127 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f252c09e0d0 con 0x7f252c06e7b0 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2336996809 0 0) 0x7f2514003520 con 0x7f252c0a5d50 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 >> v1:172.21.15.104:6792/0 conn(0x7f252c0a5920 legacy=0x7f252c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 >> v1:172.21.15.104:6791/0 conn(0x7f252c06e7b0 legacy=0x7f252c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 1 monclient: found mon.noname-c 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.187+0000 7f252bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252c09e3b0 con 0x7f252c0a5d50 2024-08-19T12:51:31.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 10 monclient: _finish_auth 0 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f251c001e60 con 0x7f252c0a5d50 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f2533656640 5 monclient: authenticate success, global_id 184171 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f2533656640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 10 monclient: dump: 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.195 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f2533656640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f251c004c20 con 0x7f252c0a5d50 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:31.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 1 -- 172.21.15.26:0/3203180635 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f251c005190 con 0x7f252c0a5d50 2024-08-19T12:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f2533656640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f2533656640 4 set_mon_vals no callback set 2024-08-19T12:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f252bfff640 10 monclient: dump: 2024-08-19T12:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.197 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f2533656640 10 monclient: shutdown 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f2533656640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f2533656640 1 -- 172.21.15.26:0/3203180635 >> v1:172.21.15.26:6791/0 conn(0x7f252c0a5d50 legacy=0x7f252c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f2533656640 1 -- 172.21.15.26:0/3203180635 shutdown_connections 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f2533656640 1 -- 172.21.15.26:0/3203180635 >> 172.21.15.26:0/3203180635 conn(0x7f252c0a8740 msgr2=0x7f252c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f2533656640 1 -- 172.21.15.26:0/3203180635 shutdown_connections 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.188+0000 7f2533656640 1 -- 172.21.15.26:0/3203180635 wait complete. 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.189+0000 7f2533656640 10 monclient: build_initial_monmap 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.189+0000 7f2533656640 10 monclient: monmap: 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:31.199 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.189+0000 7f2533656640 1 librados: starting msgr at 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.189+0000 7f2533656640 1 librados: starting objecter 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.189+0000 7f2533656640 1 Processor -- start 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.189+0000 7f2533656640 1 -- start start 2024-08-19T12:51:31.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.189+0000 7f2533656640 1 librados: setting wanted keys 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.189+0000 7f2533656640 1 librados: calling monclient init 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.189+0000 7f2533656640 10 monclient: init 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f2533656640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f2533656640 10 monclient: _add_conns ranks=[2,8,6,1,3,5,7,4,0] 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f2533656640 10 monclient(hunting): picked mon.noname-b con 0x7f252c06e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f2533656640 10 monclient(hunting): picked mon.noname-e con 0x7f252c0a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f2533656640 10 monclient(hunting): picked mon.noname-d con 0x7f252c0a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f2533656640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c0a0fa0 con 0x7f252c06e7b0 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f2533656640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c0a2a60 con 0x7f252c0a5d50 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f2533656640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c0a3300 con 0x7f252c0a5920 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f2533656640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f2533656640 10 monclient(hunting): authenticate will time out at 1485.277175s 2024-08-19T12:51:31.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f25313f2640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f252c06e7b0 0x7f252c1ce510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54552/0 (socket says 172.21.15.26:54552) 2024-08-19T12:51:31.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f25313f2640 1 -- 172.21.15.26:0/910503638 learned_addr learned my addr 172.21.15.26:0/910503638 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:31.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.190+0000 7f2530bf1640 1 --1- 172.21.15.26:0/910503638 >> v1:172.21.15.26:6793/0 conn(0x7f252c0a5920 0x7f252c1d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:45534/0 (socket says 172.21.15.26:45534) 2024-08-19T12:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2531bf3640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.26:6792/0 conn(0x7f252c0a5d50 legacy=0x7f252c1d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 client.?.objecter ms_handle_connect 0x7f252c0a5d50 2024-08-19T12:51:31.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 client.?.objecter ms_handle_connect 0x7f252c0a5920 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2530bf1640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.26:6793/0 conn(0x7f252c0a5920 legacy=0x7f252c1d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f25313f2640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.26:6790/0 conn(0x7f252c06e7b0 legacy=0x7f252c1ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 client.?.objecter ms_handle_connect 0x7f252c06e7b0 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 599932319 0 0) 0x7f252c0a2a60 con 0x7f252c0a5d50 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): my global_id is 164625 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2518004eb0 con 0x7f252c0a5d50 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2474327559 0 0) 0x7f252c0a3300 con 0x7f252c0a5920 2024-08-19T12:51:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): my global_id is 184139 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f252c0a2a60 con 0x7f252c0a5920 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2319238702 0 0) 0x7f252c0a0fa0 con 0x7f252c06e7b0 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): my global_id is 26912 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f252c0a3300 con 0x7f252c06e7b0 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2613095083 0 0) 0x7f2518004eb0 con 0x7f252c0a5d50 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f252c0a0fa0 con 0x7f252c0a5d50 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f251c0020b0 con 0x7f252c0a5d50 2024-08-19T12:51:31.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): dump: 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.209 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2858156935 0 0) 0x7f252c0a2a60 con 0x7f252c0a5920 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2518004eb0 con 0x7f252c0a5920 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.192+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1694523272 0 0) 0x7f252c0a3300 con 0x7f252c06e7b0 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f252c0a2a60 con 0x7f252c06e7b0 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2520003610 con 0x7f252c0a5920 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient(hunting): dump: 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.211 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2524003950 con 0x7f252c06e7b0 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient(hunting): dump: 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.212 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2648488786 0 0) 0x7f252c0a0fa0 con 0x7f252c0a5d50 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.26:6793/0 conn(0x7f252c0a5920 legacy=0x7f252c1d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.26:6790/0 conn(0x7f252c06e7b0 legacy=0x7f252c1ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 1 monclient: found mon.d 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252c0a1b00 con 0x7f252c0a5d50 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient: _finish_auth 0 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2533656640 5 monclient: authenticate success, global_id 164625 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2533656640 10 monclient: _renew_subs 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2533656640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2533656640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f252c0a3ce0 con 0x7f252c0a5d50 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f251c001e40 con 0x7f252c0a5d50 2024-08-19T12:51:31.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f251c005700 con 0x7f252c0a5d50 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2529ffb640 10 monclient: dump: 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.215 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2533656640 10 client.164625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2533656640 10 monclient: _renew_subs 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2533656640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f252b7fe640 4 set_mon_vals no callback set 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2533656640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f252c0a0af0 con 0x7f252c0a5d50 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.193+0000 7f2533656640 1 librados: init done 2024-08-19T12:51:31.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.194+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f251c011b60 con 0x7f252c0a5d50 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.194+0000 7f2533656640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.194+0000 7f2533656640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f252c1cf000 con 0x7f252c1cec20 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.194+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(482..482 src has 350..482) ==== 6214+0+0 (unknown 1180833201 0 0) 0x7f251c04e580 con 0x7f252c0a5d50 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.194+0000 7f2529ffb640 10 client.164625.objecter ms_dispatch 0x7f252c000b90 osd_map(482..482 src has 350..482) 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.194+0000 7f2529ffb640 3 client.164625.objecter handle_osd_map got epochs [482,482] > 0 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.194+0000 7f2529ffb640 3 client.164625.objecter handle_osd_map decoding full epoch 482 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.194+0000 7f2529ffb640 20 client.164625.objecter dump_active .. 0 homeless 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.194+0000 7f2530bf1640 10 monclient: get_auth_request con 0x7f2518041b10 auth_method 0 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.196+0000 7f2529ffb640 10 client.164625.objecter ms_handle_connect 0x7f252c1cec20 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.196+0000 7f25313f2640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.104:6789/0 conn(0x7f252c1cec20 legacy=0x7f252c1252b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.196+0000 7f2529ffb640 10 client.164625.objecter resend_mon_ops 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3101808810 0 0) 0x7f252c1cf000 con 0x7f252c1cec20 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 10 monclient(hunting): my global_id is 184123 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f252c0a0fa0 con 0x7f252c1cec20 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 10 client.164625.objecter ms_handle_connect 0x7f2518041b10 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3684230583 0 0) 0x7f252c0a0fa0 con 0x7f252c1cec20 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f252c1cf000 con 0x7f252c1cec20 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2524003e30 con 0x7f252c1cec20 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 10 monclient: dump: 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.218 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 253345260 0 0) 0x7f252c1cf000 con 0x7f252c1cec20 2024-08-19T12:51:31.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.197+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f252c134e00 con 0x7f252c1cec20 2024-08-19T12:51:31.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.198+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2524004090 con 0x7f252c1cec20 2024-08-19T12:51:31.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.198+0000 7f2529ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.198+0000 7f2529ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:31.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.198+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.104:6789/0 conn(0x7f252c1cec20 legacy=0x7f252c1252b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.198+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.104:6789/0 conn(0x7f252c1cec20 legacy=0x7f252c1252b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:31.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.205+0000 7f2533656640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:31.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.205+0000 7f2533656640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f252c068df0 con 0x7f252c02e1d0 2024-08-19T12:51:31.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.207+0000 7f25313f2640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.104:6789/0 conn(0x7f252c02e1d0 legacy=0x7f252c1259c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.207+0000 7f2529ffb640 10 client.164625.objecter ms_handle_connect 0x7f252c02e1d0 2024-08-19T12:51:31.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.207+0000 7f2529ffb640 10 client.164625.objecter resend_mon_ops 2024-08-19T12:51:31.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3997049674 0 0) 0x7f252c068df0 con 0x7f252c02e1d0 2024-08-19T12:51:31.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 10 monclient(hunting): my global_id is 184132 2024-08-19T12:51:31.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f252c1cf000 con 0x7f252c02e1d0 2024-08-19T12:51:31.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 87588705 0 0) 0x7f252c1cf000 con 0x7f252c02e1d0 2024-08-19T12:51:31.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f252c068df0 con 0x7f252c02e1d0 2024-08-19T12:51:31.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2524005b00 con 0x7f252c02e1d0 2024-08-19T12:51:31.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:51:31.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 10 monclient: dump: 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.223 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.224 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 261606457 0 0) 0x7f252c068df0 con 0x7f252c02e1d0 2024-08-19T12:51:31.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.208+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f252c02e5b0 con 0x7f252c02e1d0 2024-08-19T12:51:31.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 2852553048) 0x7f2524006f50 con 0x7f252c02e1d0 2024-08-19T12:51:31.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2529ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:31.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2529ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:31.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.104:6789/0 conn(0x7f252c02e1d0 legacy=0x7f252c1259c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2529ffb640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.104:6789/0 conn(0x7f252c02e1d0 legacy=0x7f252c1259c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:31.224 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 16676, 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:31.225 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.226 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.227 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:31.228 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.229 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.230 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.231 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.233 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.234 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.235 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.236 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:31.237 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:31.238 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.241 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:31.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.243 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:31.244 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:31.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 10 librados: watch_flush enter 2024-08-19T12:51:31.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 10 librados: watch_flush exit 2024-08-19T12:51:31.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 20 client.164625.objecter shutdown clearing up homeless session... 2024-08-19T12:51:31.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 10 client.164625.objecter successfully canceled tick 2024-08-19T12:51:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.104:6800/899887506 conn(0x7f2518041b10 legacy=0x7f2518043fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 10 monclient: shutdown 2024-08-19T12:51:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 1 -- 172.21.15.26:0/910503638 >> v1:172.21.15.26:6792/0 conn(0x7f252c0a5d50 legacy=0x7f252c1d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2531bf3640 1 -- 172.21.15.26:0/910503638 reap_dead start 2024-08-19T12:51:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2531bf3640 1 -- 172.21.15.26:0/910503638 reap_dead start 2024-08-19T12:51:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 1 -- 172.21.15.26:0/910503638 shutdown_connections 2024-08-19T12:51:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 1 -- 172.21.15.26:0/910503638 >> 172.21.15.26:0/910503638 conn(0x7f252c0a8740 msgr2=0x7f252c0aa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 1 -- 172.21.15.26:0/910503638 shutdown_connections 2024-08-19T12:51:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 1 -- 172.21.15.26:0/910503638 wait complete. 2024-08-19T12:51:31.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.209+0000 7f2533656640 1 librados: shutdown 2024-08-19T12:51:31.247 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:31.247 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:51:31.247 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:51:31.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.347+0000 7f5c43e70640 10 monclient: get_monmap_and_config 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.347+0000 7f5c43e70640 10 monclient: build_initial_monmap 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.347+0000 7f5c43e70640 10 monclient: monmap: 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:31.347968+0000 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:31.347968+0000 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:31.348 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:31.349 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:31.349 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:31.349 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:31.349 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:31.349 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.347+0000 7f5c43e70640 1 Processor -- start 2024-08-19T12:51:31.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.347+0000 7f5c43e70640 1 -- start start 2024-08-19T12:51:31.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.347+0000 7f5c43e70640 10 monclient: init 2024-08-19T12:51:31.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.348+0000 7f5c43e70640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:31.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.348+0000 7f5c43e70640 10 monclient: _add_conns ranks=[4,3,5,8,7,1,2,0,6] 2024-08-19T12:51:31.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.348+0000 7f5c43e70640 10 monclient(hunting): picked mon.noname-c con 0x7f5c3c073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:31.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.348+0000 7f5c43e70640 10 monclient(hunting): picked mon.noname-g con 0x7f5c3c0a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:31.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.348+0000 7f5c43e70640 10 monclient(hunting): picked mon.noname-h con 0x7f5c3c13f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:31.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.348+0000 7f5c43e70640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c09dae0 con 0x7f5c3c0a57a0 2024-08-19T12:51:31.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.348+0000 7f5c43e70640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c136910 con 0x7f5c3c073470 2024-08-19T12:51:31.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.348+0000 7f5c43e70640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c09dd50 con 0x7f5c3c13f2e0 2024-08-19T12:51:31.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.348+0000 7f5c43e70640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:31.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.348+0000 7f5c43e70640 10 monclient(hunting): authenticate will time out at 1485.435444s 2024-08-19T12:51:31.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.348+0000 7f5c41c0c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f5c3c073470 0x7f5c3c136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41624/0 (socket says 172.21.15.26:41624) 2024-08-19T12:51:31.351 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.348+0000 7f5c41c0c640 1 -- 172.21.15.26:0/3876633292 learned_addr learned my addr 172.21.15.26:0/3876633292 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:31.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.350+0000 7f5c41c0c640 1 -- 172.21.15.26:0/3876633292 >> v1:172.21.15.26:6791/0 conn(0x7f5c3c073470 legacy=0x7f5c3c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.350+0000 7f5c4140b640 1 -- 172.21.15.26:0/3876633292 >> v1:172.21.15.104:6790/0 conn(0x7f5c3c0a57a0 legacy=0x7f5c3c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.350+0000 7f5c4240d640 1 -- 172.21.15.26:0/3876633292 >> v1:172.21.15.104:6791/0 conn(0x7f5c3c13f2e0 legacy=0x7f5c3c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.350+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 484649925 0 0) 0x7f5c3c136910 con 0x7f5c3c073470 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.350+0000 7f5c40c0a640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.350+0000 7f5c40c0a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.350+0000 7f5c40c0a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.350+0000 7f5c40c0a640 10 monclient(hunting): my global_id is 184180 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.350+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c28003520 con 0x7f5c3c073470 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1774975696 0 0) 0x7f5c3c09dae0 con 0x7f5c3c0a57a0 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient(hunting): my global_id is 154695 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c3c136910 con 0x7f5c3c0a57a0 2024-08-19T12:51:31.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1447969357 0 0) 0x7f5c3c09dd50 con 0x7f5c3c13f2e0 2024-08-19T12:51:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient(hunting): my global_id is 184136 2024-08-19T12:51:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c3c09dae0 con 0x7f5c3c13f2e0 2024-08-19T12:51:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2414143582 0 0) 0x7f5c28003520 con 0x7f5c3c073470 2024-08-19T12:51:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 >> v1:172.21.15.104:6791/0 conn(0x7f5c3c13f2e0 legacy=0x7f5c3c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 >> v1:172.21.15.104:6790/0 conn(0x7f5c3c0a57a0 legacy=0x7f5c3c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient: _finish_hunting 0 2024-08-19T12:51:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 1 monclient: found mon.noname-c 2024-08-19T12:51:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3c09e030 con 0x7f5c3c073470 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient: _finish_auth 0 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient: _check_auth_tickets 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c38001ae0 con 0x7f5c3c073470 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c43e70640 5 monclient: authenticate success, global_id 184180 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c43e70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient: dump: 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.358 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c43e70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:31.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c380048a0 con 0x7f5c3c073470 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 1 -- 172.21.15.26:0/3876633292 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c38004e10 con 0x7f5c3c073470 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c43e70640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c43e70640 4 set_mon_vals no callback set 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c40c0a640 10 monclient: dump: 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.360 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c43e70640 10 monclient: shutdown 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c43e70640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c43e70640 1 -- 172.21.15.26:0/3876633292 >> v1:172.21.15.26:6791/0 conn(0x7f5c3c073470 legacy=0x7f5c3c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c43e70640 1 -- 172.21.15.26:0/3876633292 shutdown_connections 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c43e70640 1 -- 172.21.15.26:0/3876633292 >> 172.21.15.26:0/3876633292 conn(0x7f5c3c0abff0 msgr2=0x7f5c3c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.351+0000 7f5c43e70640 1 -- 172.21.15.26:0/3876633292 shutdown_connections 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.352+0000 7f5c43e70640 1 -- 172.21.15.26:0/3876633292 wait complete. 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.352+0000 7f5c43e70640 10 monclient: build_initial_monmap 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.352+0000 7f5c43e70640 10 monclient: monmap: 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:31.362 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.352+0000 7f5c43e70640 1 librados: starting msgr at 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.352+0000 7f5c43e70640 1 librados: starting objecter 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.352+0000 7f5c43e70640 1 Processor -- start 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.352+0000 7f5c43e70640 1 -- start start 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.352+0000 7f5c43e70640 1 librados: setting wanted keys 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.352+0000 7f5c43e70640 1 librados: calling monclient init 2024-08-19T12:51:31.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.352+0000 7f5c43e70640 10 monclient: init 2024-08-19T12:51:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.353+0000 7f5c43e70640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.353+0000 7f5c43e70640 10 monclient: _add_conns ranks=[5,1,8,0,3,6,4,2,7] 2024-08-19T12:51:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.353+0000 7f5c43e70640 10 monclient(hunting): picked mon.noname-h con 0x7f5c3c073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.353+0000 7f5c43e70640 10 monclient(hunting): picked mon.noname-f con 0x7f5c3c0a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:51:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.353+0000 7f5c43e70640 10 monclient(hunting): picked mon.noname-e con 0x7f5c3c13f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:51:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.353+0000 7f5c43e70640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c13e9e0 con 0x7f5c3c0a57a0 2024-08-19T12:51:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.353+0000 7f5c43e70640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c0a2e10 con 0x7f5c3c073470 2024-08-19T12:51:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.353+0000 7f5c43e70640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c137160 con 0x7f5c3c13f2e0 2024-08-19T12:51:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.353+0000 7f5c43e70640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.353+0000 7f5c43e70640 10 monclient(hunting): authenticate will time out at 1485.440504s 2024-08-19T12:51:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.353+0000 7f5c4240d640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f5c3c13f2e0 0x7f5c3c1d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:45546/0 (socket says 172.21.15.26:45546) 2024-08-19T12:51:31.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.353+0000 7f5c4240d640 1 -- 172.21.15.26:0/3363793608 learned_addr learned my addr 172.21.15.26:0/3363793608 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:31.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c4240d640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.26:6793/0 conn(0x7f5c3c13f2e0 legacy=0x7f5c3c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c327fc640 10 client.?.objecter ms_handle_connect 0x7f5c3c13f2e0 2024-08-19T12:51:31.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c327fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c41c0c640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.104:6791/0 conn(0x7f5c3c073470 legacy=0x7f5c3c0a1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c327fc640 10 client.?.objecter ms_handle_connect 0x7f5c3c073470 2024-08-19T12:51:31.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c327fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c327fc640 10 client.?.objecter ms_handle_connect 0x7f5c3c0a57a0 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c4140b640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.104:6789/0 conn(0x7f5c3c0a57a0 legacy=0x7f5c3c0a2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c327fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c4240d640 1 --1- 172.21.15.26:0/3363793608 >> v1:172.21.15.26:6793/0 conn(0x7f5c3c13f2e0 0x7f5c3c1d6150 :-1 s=OPENED pgs=3450 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f5c3c137160 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3359852789 0 0) 0x7f5c3c0a2e10 con 0x7f5c3c073470 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c327fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c327fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c327fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c327fc640 10 monclient(hunting): my global_id is 184145 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.355+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c20004ee0 con 0x7f5c3c073470 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3073546012 0 0) 0x7f5c3c13e9e0 con 0x7f5c3c0a57a0 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient(hunting): my global_id is 184141 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c3c0a2e10 con 0x7f5c3c0a57a0 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 926308731 0 0) 0x7f5c20004ee0 con 0x7f5c3c073470 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c3c13e9e0 con 0x7f5c3c073470 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c38004d90 con 0x7f5c3c073470 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient(hunting): dump: 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.371 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2264315036 0 0) 0x7f5c3c0a2e10 con 0x7f5c3c0a57a0 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c20004ee0 con 0x7f5c3c0a57a0 2024-08-19T12:51:31.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c2c003610 con 0x7f5c3c0a57a0 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient(hunting): dump: 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.373 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 480954557 0 0) 0x7f5c3c13e9e0 con 0x7f5c3c073470 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.26:6793/0 conn(0x7f5c3c13f2e0 legacy=0x7f5c3c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.104:6789/0 conn(0x7f5c3c0a57a0 legacy=0x7f5c3c0a2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient: _finish_hunting 0 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 monclient: found mon.h 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3c0a5500 con 0x7f5c3c073470 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient: _finish_auth 0 2024-08-19T12:51:31.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient: _check_auth_tickets 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3880976013 0 0) 0x7f5c3c137160 con 0x7f5c3c13f2e0 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c43e70640 5 monclient: authenticate success, global_id 184145 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c43e70640 10 monclient: _renew_subs 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c43e70640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c43e70640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5c3c0a45a0 con 0x7f5c3c073470 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c43e70640 10 client.184145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c43e70640 10 monclient: _renew_subs 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c43e70640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c43e70640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f5c3c1d7870 con 0x7f5c3c073470 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c43e70640 1 librados: init done 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c38001d30 con 0x7f5c3c073470 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:31.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c380048a0 con 0x7f5c3c073470 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c33fff640 4 set_mon_vals no callback set 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.356+0000 7f5c327fc640 10 monclient: dump: 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.376 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.357+0000 7f5c43e70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.357+0000 7f5c43e70640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5c240065c0 con 0x7f5c24002860 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.357+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5c38011a00 con 0x7f5c3c073470 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.357+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(482..482 src has 350..482) ==== 6214+0+0 (unknown 1180833201 0 0) 0x7f5c3804e540 con 0x7f5c3c073470 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.357+0000 7f5c327fc640 10 client.184145.objecter ms_dispatch 0x7f5c3c000b90 osd_map(482..482 src has 350..482) 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.357+0000 7f5c327fc640 3 client.184145.objecter handle_osd_map got epochs [482,482] > 0 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.357+0000 7f5c327fc640 3 client.184145.objecter handle_osd_map decoding full epoch 482 2024-08-19T12:51:31.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.357+0000 7f5c327fc640 20 client.184145.objecter dump_active .. 0 homeless 2024-08-19T12:51:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.357+0000 7f5c4240d640 10 monclient: get_auth_request con 0x7f5c200416a0 auth_method 0 2024-08-19T12:51:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.359+0000 7f5c327fc640 10 client.184145.objecter ms_handle_connect 0x7f5c24002860 2024-08-19T12:51:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.359+0000 7f5c327fc640 10 client.184145.objecter resend_mon_ops 2024-08-19T12:51:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.359+0000 7f5c4140b640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.104:6790/0 conn(0x7f5c24002860 legacy=0x7f5c24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.359+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1473299560 0 0) 0x7f5c240065c0 con 0x7f5c24002860 2024-08-19T12:51:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.359+0000 7f5c327fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.359+0000 7f5c327fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.359+0000 7f5c327fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.359+0000 7f5c327fc640 10 monclient(hunting): my global_id is 154704 2024-08-19T12:51:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.359+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c3c137160 con 0x7f5c24002860 2024-08-19T12:51:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1831485737 0 0) 0x7f5c3c137160 con 0x7f5c24002860 2024-08-19T12:51:31.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c240065c0 con 0x7f5c24002860 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c2c003ea0 con 0x7f5c24002860 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 10 monclient: dump: 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.379 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 10 client.184145.objecter ms_handle_connect 0x7f5c200416a0 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 471510661 0 0) 0x7f5c240065c0 con 0x7f5c24002860 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c2400a160 con 0x7f5c24002860 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5c2c0041b0 con 0x7f5c24002860 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.104:6790/0 conn(0x7f5c24002860 legacy=0x7f5c24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.360+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.104:6790/0 conn(0x7f5c24002860 legacy=0x7f5c24004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:31.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.366+0000 7f5c43e70640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:31.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.366+0000 7f5c43e70640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5c2400c4f0 con 0x7f5c24009110 2024-08-19T12:51:31.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c4140b640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.104:6790/0 conn(0x7f5c24009110 legacy=0x7f5c24009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 10 client.184145.objecter ms_handle_connect 0x7f5c24009110 2024-08-19T12:51:31.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 10 client.184145.objecter resend_mon_ops 2024-08-19T12:51:31.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2022072072 0 0) 0x7f5c2400c4f0 con 0x7f5c24009110 2024-08-19T12:51:31.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 10 monclient(hunting): my global_id is 154713 2024-08-19T12:51:31.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c240065c0 con 0x7f5c24009110 2024-08-19T12:51:31.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2156855721 0 0) 0x7f5c240065c0 con 0x7f5c24009110 2024-08-19T12:51:31.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c2400c4f0 con 0x7f5c24009110 2024-08-19T12:51:31.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5c2c005cf0 con 0x7f5c24009110 2024-08-19T12:51:31.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:51:31.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 10 monclient: dump: 2024-08-19T12:51:31.383 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.383 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.386 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.386 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.386 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.386 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.386 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.386 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.386 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.386 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.386 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.386 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.387 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.387 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.387 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.387 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3846569486 0 0) 0x7f5c2400c4f0 con 0x7f5c24009110 2024-08-19T12:51:31.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.369+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c24009ce0 con 0x7f5c24009110 2024-08-19T12:51:31.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.370+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 2909494973) 0x7f5c2c007d80 con 0x7f5c24009110 2024-08-19T12:51:31.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.370+0000 7f5c327fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:31.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.370+0000 7f5c327fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:31.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.370+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.104:6790/0 conn(0x7f5c24009110 legacy=0x7f5c24009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.370+0000 7f5c327fc640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.104:6790/0 conn(0x7f5c24009110 legacy=0x7f5c24009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:31.388 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:31.388 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:31.388 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:31.388 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:31.388 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:31.388 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 403098, 2024-08-19T12:51:31.388 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:31.388 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:31.388 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:31.389 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.390 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.390 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.390 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.390 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.390 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.390 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.390 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.390 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.390 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.390 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.390 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.390 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:31.392 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.393 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.394 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:31.395 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:31.397 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:31.398 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:31.399 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.400 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.401 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:31.402 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.403 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.404 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.405 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.406 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:31.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.407 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:31.407 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:31.407 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:31.407 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.407 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.407 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:31.407 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.370+0000 7f5c43e70640 10 librados: watch_flush enter 2024-08-19T12:51:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.370+0000 7f5c43e70640 10 librados: watch_flush exit 2024-08-19T12:51:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.370+0000 7f5c43e70640 20 client.184145.objecter shutdown clearing up homeless session... 2024-08-19T12:51:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.370+0000 7f5c43e70640 10 client.184145.objecter successfully canceled tick 2024-08-19T12:51:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.371+0000 7f5c43e70640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.104:6800/899887506 conn(0x7f5c200416a0 legacy=0x7f5c20043b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.371+0000 7f5c43e70640 10 monclient: shutdown 2024-08-19T12:51:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.371+0000 7f5c43e70640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.371+0000 7f5c43e70640 1 -- 172.21.15.26:0/3363793608 >> v1:172.21.15.104:6791/0 conn(0x7f5c3c073470 legacy=0x7f5c3c0a1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.371+0000 7f5c4240d640 1 -- 172.21.15.26:0/3363793608 reap_dead start 2024-08-19T12:51:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.371+0000 7f5c4240d640 1 -- 172.21.15.26:0/3363793608 reap_dead start 2024-08-19T12:51:31.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.371+0000 7f5c43e70640 1 -- 172.21.15.26:0/3363793608 shutdown_connections 2024-08-19T12:51:31.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.371+0000 7f5c43e70640 1 -- 172.21.15.26:0/3363793608 >> 172.21.15.26:0/3363793608 conn(0x7f5c3c0abff0 msgr2=0x7f5c3c133960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:31.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.371+0000 7f5c43e70640 1 -- 172.21.15.26:0/3363793608 shutdown_connections 2024-08-19T12:51:31.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.371+0000 7f5c43e70640 1 -- 172.21.15.26:0/3363793608 wait complete. 2024-08-19T12:51:31.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.371+0000 7f5c43e70640 1 librados: shutdown 2024-08-19T12:51:31.410 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:31.410 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:51:31.410 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:51:31.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.510+0000 7f2deeda1640 10 monclient: get_monmap_and_config 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.510+0000 7f2deeda1640 10 monclient: build_initial_monmap 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.510+0000 7f2deeda1640 10 monclient: monmap: 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:31.511799+0000 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:31.511799+0000 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:31.512 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:31.513 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:31.513 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:31.513 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:31.513 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:31.513 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.511+0000 7f2deeda1640 1 Processor -- start 2024-08-19T12:51:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.511+0000 7f2deeda1640 1 -- start start 2024-08-19T12:51:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.511+0000 7f2deeda1640 10 monclient: init 2024-08-19T12:51:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.512+0000 7f2deeda1640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.512+0000 7f2deeda1640 10 monclient: _add_conns ranks=[7,1,0,8,4,3,5,6,2] 2024-08-19T12:51:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.512+0000 7f2deeda1640 10 monclient(hunting): picked mon.noname-i con 0x7f2de8073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.512+0000 7f2deeda1640 10 monclient(hunting): picked mon.noname-f con 0x7f2de80a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:51:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.512+0000 7f2deeda1640 10 monclient(hunting): picked mon.noname-a con 0x7f2de813f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.512+0000 7f2deeda1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de809dae0 con 0x7f2de813f2e0 2024-08-19T12:51:31.519 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.512+0000 7f2deeda1640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de8136910 con 0x7f2de80a57a0 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.512+0000 7f2deeda1640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de809dd50 con 0x7f2de8073470 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.512+0000 7f2deeda1640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.512+0000 7f2deeda1640 10 monclient(hunting): authenticate will time out at 1485.599374s 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.512+0000 7f2ded33e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2de813f2e0 0x7f2de81416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60164/0 (socket says 172.21.15.26:60164) 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.512+0000 7f2ded33e640 1 -- 172.21.15.26:0/517838462 learned_addr learned my addr 172.21.15.26:0/517838462 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2ded33e640 1 -- 172.21.15.26:0/517838462 >> v1:172.21.15.26:6789/0 conn(0x7f2de813f2e0 legacy=0x7f2de81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de7fff640 1 -- 172.21.15.26:0/517838462 >> v1:172.21.15.104:6789/0 conn(0x7f2de80a57a0 legacy=0x7f2de80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2decb3d640 1 -- 172.21.15.26:0/517838462 >> v1:172.21.15.104:6792/0 conn(0x7f2de8073470 legacy=0x7f2de8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2294032651 0 0) 0x7f2de809dae0 con 0x7f2de813f2e0 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de77fe640 10 monclient(hunting): my global_id is 145314 2024-08-19T12:51:31.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dd0003520 con 0x7f2de813f2e0 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2291902616 0 0) 0x7f2de8136910 con 0x7f2de80a57a0 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de77fe640 10 monclient(hunting): my global_id is 184150 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.514+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2de809dae0 con 0x7f2de80a57a0 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3968903331 0 0) 0x7f2de809dd50 con 0x7f2de8073470 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient(hunting): my global_id is 135862 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2de8136910 con 0x7f2de8073470 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 313116983 0 0) 0x7f2dd0003520 con 0x7f2de813f2e0 2024-08-19T12:51:31.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 >> v1:172.21.15.104:6792/0 conn(0x7f2de8073470 legacy=0x7f2de8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 >> v1:172.21.15.104:6789/0 conn(0x7f2de80a57a0 legacy=0x7f2de80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient: _finish_hunting 0 2024-08-19T12:51:31.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 1 monclient: found mon.noname-a 2024-08-19T12:51:31.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:31.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de809e030 con 0x7f2de813f2e0 2024-08-19T12:51:31.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient: _finish_auth 0 2024-08-19T12:51:31.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient: _check_auth_tickets 2024-08-19T12:51:31.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:31.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2de0001e60 con 0x7f2de813f2e0 2024-08-19T12:51:31.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2deeda1640 5 monclient: authenticate success, global_id 145314 2024-08-19T12:51:31.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2deeda1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:31.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:31.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient: dump: 2024-08-19T12:51:31.523 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.523 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.523 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.523 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2deeda1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:31.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2de0004c20 con 0x7f2de813f2e0 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 1 -- 172.21.15.26:0/517838462 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2de0005190 con 0x7f2de813f2e0 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2deeda1640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2deeda1640 4 set_mon_vals no callback set 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2de77fe640 10 monclient: dump: 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.525 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2deeda1640 10 monclient: shutdown 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2deeda1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2deeda1640 1 -- 172.21.15.26:0/517838462 >> v1:172.21.15.26:6789/0 conn(0x7f2de813f2e0 legacy=0x7f2de81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2deeda1640 1 -- 172.21.15.26:0/517838462 shutdown_connections 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2deeda1640 1 -- 172.21.15.26:0/517838462 >> 172.21.15.26:0/517838462 conn(0x7f2de80abff0 msgr2=0x7f2de80ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2deeda1640 1 -- 172.21.15.26:0/517838462 shutdown_connections 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.515+0000 7f2deeda1640 1 -- 172.21.15.26:0/517838462 wait complete. 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.516+0000 7f2deeda1640 10 monclient: build_initial_monmap 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.516+0000 7f2deeda1640 10 monclient: monmap: 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:31.527 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.516+0000 7f2deeda1640 1 librados: starting msgr at 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.516+0000 7f2deeda1640 1 librados: starting objecter 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.516+0000 7f2deeda1640 1 Processor -- start 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.516+0000 7f2deeda1640 1 -- start start 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.516+0000 7f2deeda1640 1 librados: setting wanted keys 2024-08-19T12:51:31.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.516+0000 7f2deeda1640 1 librados: calling monclient init 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.516+0000 7f2deeda1640 10 monclient: init 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.517+0000 7f2deeda1640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.517+0000 7f2deeda1640 10 monclient: _add_conns ranks=[1,3,0,8,2,6,5,4,7] 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.517+0000 7f2deeda1640 10 monclient(hunting): picked mon.noname-f con 0x7f2de8073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.517+0000 7f2deeda1640 10 monclient(hunting): picked mon.noname-g con 0x7f2de80a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.517+0000 7f2deeda1640 10 monclient(hunting): picked mon.noname-a con 0x7f2de813f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.517+0000 7f2deeda1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de80a4790 con 0x7f2de813f2e0 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.517+0000 7f2deeda1640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de81cef20 con 0x7f2de8073470 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.517+0000 7f2deeda1640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de813f0b0 con 0x7f2de80a57a0 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.517+0000 7f2deeda1640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.517+0000 7f2deeda1640 10 monclient(hunting): authenticate will time out at 1485.604469s 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.517+0000 7f2ded33e640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2de813f2e0 0x7f2de81d6360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60180/0 (socket says 172.21.15.26:60180) 2024-08-19T12:51:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.517+0000 7f2ded33e640 1 -- 172.21.15.26:0/2938696620 learned_addr learned my addr 172.21.15.26:0/2938696620 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:31.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2ded33e640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.26:6789/0 conn(0x7f2de813f2e0 legacy=0x7f2de81d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 client.?.objecter ms_handle_connect 0x7f2de813f2e0 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2decb3d640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.104:6789/0 conn(0x7f2de8073470 legacy=0x7f2de81ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 client.?.objecter ms_handle_connect 0x7f2de8073470 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 client.?.objecter ms_handle_connect 0x7f2de80a57a0 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de7fff640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.104:6790/0 conn(0x7f2de80a57a0 legacy=0x7f2de81ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3039891271 0 0) 0x7f2de80a4790 con 0x7f2de813f2e0 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 monclient(hunting): my global_id is 145323 2024-08-19T12:51:31.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dd4004ee0 con 0x7f2de813f2e0 2024-08-19T12:51:31.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1536535129 0 0) 0x7f2de81cef20 con 0x7f2de8073470 2024-08-19T12:51:31.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 monclient(hunting): my global_id is 184159 2024-08-19T12:51:31.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2de80a4790 con 0x7f2de8073470 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 336959898 0 0) 0x7f2de813f0b0 con 0x7f2de80a57a0 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 10 monclient(hunting): my global_id is 154722 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.519+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2de81cef20 con 0x7f2de80a57a0 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3427987010 0 0) 0x7f2dd4004ee0 con 0x7f2de813f2e0 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2de813f0b0 con 0x7f2de813f2e0 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2de00020b0 con 0x7f2de813f2e0 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient(hunting): dump: 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.539 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:31.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 741146453 0 0) 0x7f2de80a4790 con 0x7f2de8073470 2024-08-19T12:51:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2dd4004ee0 con 0x7f2de8073470 2024-08-19T12:51:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2dd8001c80 con 0x7f2de8073470 2024-08-19T12:51:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:51:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient(hunting): dump: 2024-08-19T12:51:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.543 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 738504647 0 0) 0x7f2de81cef20 con 0x7f2de80a57a0 2024-08-19T12:51:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2de80a4790 con 0x7f2de80a57a0 2024-08-19T12:51:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2ddc003610 con 0x7f2de80a57a0 2024-08-19T12:51:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:51:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient(hunting): dump: 2024-08-19T12:51:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.543 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.544 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1665312046 0 0) 0x7f2de813f0b0 con 0x7f2de813f2e0 2024-08-19T12:51:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.104:6790/0 conn(0x7f2de80a57a0 legacy=0x7f2de81ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.104:6789/0 conn(0x7f2de8073470 legacy=0x7f2de81ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient: _finish_hunting 0 2024-08-19T12:51:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 monclient: found mon.a 2024-08-19T12:51:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de8142b60 con 0x7f2de813f2e0 2024-08-19T12:51:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient: _finish_auth 0 2024-08-19T12:51:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient: _check_auth_tickets 2024-08-19T12:51:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:31.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2deeda1640 5 monclient: authenticate success, global_id 145323 2024-08-19T12:51:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2deeda1640 10 monclient: _renew_subs 2024-08-19T12:51:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2deeda1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2deeda1640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2de80a3960 con 0x7f2de813f2e0 2024-08-19T12:51:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2deeda1640 10 client.145323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2deeda1640 10 monclient: _renew_subs 2024-08-19T12:51:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2deeda1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2deeda1640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2de80a44c0 con 0x7f2de813f2e0 2024-08-19T12:51:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2deeda1640 1 librados: init done 2024-08-19T12:51:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2de00053c0 con 0x7f2de813f2e0 2024-08-19T12:51:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2de0004c20 con 0x7f2de813f2e0 2024-08-19T12:51:31.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de6ffd640 4 set_mon_vals no callback set 2024-08-19T12:51:31.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:31.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.520+0000 7f2de57fa640 10 monclient: dump: 2024-08-19T12:51:31.547 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.547 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.547 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.547 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.547 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.548 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.521+0000 7f2deeda1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.521+0000 7f2deeda1640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2dc80065c0 con 0x7f2dc8002860 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.521+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2de0011b70 con 0x7f2de813f2e0 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.521+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(482..482 src has 350..482) ==== 6214+0+0 (unknown 1180833201 0 0) 0x7f2de004e7e0 con 0x7f2de813f2e0 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.521+0000 7f2de57fa640 10 client.145323.objecter ms_dispatch 0x7f2de8000b90 osd_map(482..482 src has 350..482) 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.521+0000 7f2de57fa640 3 client.145323.objecter handle_osd_map got epochs [482,482] > 0 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.521+0000 7f2de57fa640 3 client.145323.objecter handle_osd_map decoding full epoch 482 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.521+0000 7f2de57fa640 20 client.145323.objecter dump_active .. 0 homeless 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.521+0000 7f2de7fff640 10 monclient: get_auth_request con 0x7f2dd4041ba0 auth_method 0 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.523+0000 7f2decb3d640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.104:6791/0 conn(0x7f2dc8002860 legacy=0x7f2dc8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.523+0000 7f2de57fa640 10 client.145323.objecter ms_handle_connect 0x7f2dc8002860 2024-08-19T12:51:31.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.523+0000 7f2de57fa640 10 client.145323.objecter resend_mon_ops 2024-08-19T12:51:31.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.523+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2627656091 0 0) 0x7f2dc80065c0 con 0x7f2dc8002860 2024-08-19T12:51:31.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 10 monclient(hunting): my global_id is 184154 2024-08-19T12:51:31.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2de813f0b0 con 0x7f2dc8002860 2024-08-19T12:51:31.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 10 client.145323.objecter ms_handle_connect 0x7f2dd4041ba0 2024-08-19T12:51:31.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2307323527 0 0) 0x7f2de813f0b0 con 0x7f2dc8002860 2024-08-19T12:51:31.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2dc80065c0 con 0x7f2dc8002860 2024-08-19T12:51:31.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2dd8003f30 con 0x7f2dc8002860 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 10 monclient: dump: 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.551 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1401272611 0 0) 0x7f2dc80065c0 con 0x7f2dc8002860 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2dc800a160 con 0x7f2dc8002860 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2dd80042b0 con 0x7f2dc8002860 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.524+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.104:6791/0 conn(0x7f2dc8002860 legacy=0x7f2dc8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.525+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.104:6791/0 conn(0x7f2dc8002860 legacy=0x7f2dc8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:31.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.531+0000 7f2deeda1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.531+0000 7f2deeda1640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2dc8009e50 con 0x7f2dc8009110 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2decb3d640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.104:6791/0 conn(0x7f2dc8009110 legacy=0x7f2dc80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 10 client.145323.objecter ms_handle_connect 0x7f2dc8009110 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 10 client.145323.objecter resend_mon_ops 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3280898151 0 0) 0x7f2dc8009e50 con 0x7f2dc8009110 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 10 monclient(hunting): my global_id is 184163 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dc80065c0 con 0x7f2dc8009110 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1656547272 0 0) 0x7f2dc80065c0 con 0x7f2dc8009110 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2dc8009e50 con 0x7f2dc8009110 2024-08-19T12:51:31.553 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2dd8005d30 con 0x7f2dc8009110 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.533+0000 7f2de57fa640 10 monclient: dump: 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.554 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.555 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.555 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.555 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.555 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.555 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.534+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2255995074 0 0) 0x7f2dc8009e50 con 0x7f2dc8009110 2024-08-19T12:51:31.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.534+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2dc800c3a0 con 0x7f2dc8009110 2024-08-19T12:51:31.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.534+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 1011887109) 0x7f2dd8008090 con 0x7f2dc8009110 2024-08-19T12:51:31.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.534+0000 7f2de57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:31.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.534+0000 7f2de57fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:31.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.534+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.104:6791/0 conn(0x7f2dc8009110 legacy=0x7f2dc80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.534+0000 7f2de57fa640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.104:6791/0 conn(0x7f2dc8009110 legacy=0x7f2dc80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:31.556 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:31.556 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:31.556 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:31.556 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:31.556 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:31.556 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 17003, 2024-08-19T12:51:31.556 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:31.556 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:31.556 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:31.556 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:31.556 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.557 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.558 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.559 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.559 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.559 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.559 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.559 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.559 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.559 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.559 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.559 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:31.559 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:31.559 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:31.560 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.562 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.562 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:31.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.562 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:31.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.562 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:31.562 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:31.562 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:31.563 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:31.565 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.568 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.569 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.570 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:31.570 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:31.570 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.570 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:31.570 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.570 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.570 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:31.572 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.573 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:31.573 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:31.573 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:31.573 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:31.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.575 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:31.575 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:31.575 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:31.575 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.575 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.575 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:31.575 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:31.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.534+0000 7f2deeda1640 10 librados: watch_flush enter 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.534+0000 7f2deeda1640 10 librados: watch_flush exit 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.534+0000 7f2deeda1640 20 client.145323.objecter shutdown clearing up homeless session... 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.534+0000 7f2deeda1640 10 client.145323.objecter successfully canceled tick 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.535+0000 7f2deeda1640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.104:6800/899887506 conn(0x7f2dd4041ba0 legacy=0x7f2dd4044060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.535+0000 7f2deeda1640 10 monclient: shutdown 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.535+0000 7f2deeda1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.535+0000 7f2deeda1640 1 -- 172.21.15.26:0/2938696620 >> v1:172.21.15.26:6789/0 conn(0x7f2de813f2e0 legacy=0x7f2de81d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.535+0000 7f2ded33e640 1 -- 172.21.15.26:0/2938696620 reap_dead start 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.535+0000 7f2ded33e640 1 -- 172.21.15.26:0/2938696620 reap_dead start 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.535+0000 7f2deeda1640 1 -- 172.21.15.26:0/2938696620 shutdown_connections 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.535+0000 7f2deeda1640 1 -- 172.21.15.26:0/2938696620 >> 172.21.15.26:0/2938696620 conn(0x7f2de80abff0 msgr2=0x7f2de81349f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.535+0000 7f2deeda1640 1 -- 172.21.15.26:0/2938696620 shutdown_connections 2024-08-19T12:51:31.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.535+0000 7f2deeda1640 1 -- 172.21.15.26:0/2938696620 wait complete. 2024-08-19T12:51:31.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.535+0000 7f2deeda1640 1 librados: shutdown 2024-08-19T12:51:31.578 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:31.578 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:51:31.578 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:51:31.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.677+0000 7f674e798640 10 monclient: get_monmap_and_config 2024-08-19T12:51:31.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.677+0000 7f674e798640 10 monclient: build_initial_monmap 2024-08-19T12:51:31.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.677+0000 7f674e798640 10 monclient: monmap: 2024-08-19T12:51:31.678 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:31.678 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.679 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:31.678230+0000 2024-08-19T12:51:31.679 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:31.678230+0000 2024-08-19T12:51:31.679 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:31.679 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.679 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:31.679 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:31.679 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:31.679 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:31.680 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:31.680 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:31.680 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:31.680 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:31.680 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:31.680 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 1 Processor -- start 2024-08-19T12:51:31.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 1 -- start start 2024-08-19T12:51:31.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 10 monclient: init 2024-08-19T12:51:31.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:31.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 10 monclient: _add_conns ranks=[4,7,0,5,8,3,2,6,1] 2024-08-19T12:51:31.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 10 monclient(hunting): picked mon.noname-c con 0x7f674806e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:31.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 10 monclient(hunting): picked mon.noname-i con 0x7f67480a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:31.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 10 monclient(hunting): picked mon.noname-a con 0x7f67480a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:31.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6748136790 con 0x7f67480a5d50 2024-08-19T12:51:31.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f674809ddf0 con 0x7f674806e7b0 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f674809e0d0 con 0x7f67480a5920 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f674e798640 10 monclient(hunting): authenticate will time out at 1485.765665s 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f6747fff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f674806e7b0 0x7f6748136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41626/0 (socket says 172.21.15.26:41626) 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.678+0000 7f6747fff640 1 -- 172.21.15.26:0/494828411 learned_addr learned my addr 172.21.15.26:0/494828411 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.680+0000 7f674cd35640 1 -- 172.21.15.26:0/494828411 >> v1:172.21.15.26:6789/0 conn(0x7f67480a5d50 legacy=0x7f6748141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.680+0000 7f6747fff640 1 -- 172.21.15.26:0/494828411 >> v1:172.21.15.26:6791/0 conn(0x7f674806e7b0 legacy=0x7f6748136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 1 -- 172.21.15.26:0/494828411 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1248686017 0 0) 0x7f6748136790 con 0x7f67480a5d50 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient(hunting): my global_id is 145332 2024-08-19T12:51:31.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 1 -- 172.21.15.26:0/494828411 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6730003520 con 0x7f67480a5d50 2024-08-19T12:51:31.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 1 -- 172.21.15.26:0/494828411 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 171827628 0 0) 0x7f6738001450 con 0x7f674806e7b0 2024-08-19T12:51:31.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient(hunting): my global_id is 184189 2024-08-19T12:51:31.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 1 -- 172.21.15.26:0/494828411 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6748136790 con 0x7f674806e7b0 2024-08-19T12:51:31.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f67477fe640 1 -- 172.21.15.26:0/494828411 >> v1:172.21.15.104:6792/0 conn(0x7f67480a5920 legacy=0x7f674813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 1 -- 172.21.15.26:0/494828411 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2020933310 0 0) 0x7f6730003520 con 0x7f67480a5d50 2024-08-19T12:51:31.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 1 -- 172.21.15.26:0/494828411 >> v1:172.21.15.104:6792/0 conn(0x7f67480a5920 legacy=0x7f674813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 1 -- 172.21.15.26:0/494828411 >> v1:172.21.15.26:6791/0 conn(0x7f674806e7b0 legacy=0x7f6748136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 1 monclient: found mon.noname-a 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 1 -- 172.21.15.26:0/494828411 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674809e3b0 con 0x7f67480a5d50 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient: _finish_auth 0 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 1 -- 172.21.15.26:0/494828411 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6740001eb0 con 0x7f67480a5d50 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f674e798640 5 monclient: authenticate success, global_id 145332 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f674e798640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:31.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient: dump: 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.697 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.698 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f674e798640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f674cd35640 1 --1- 172.21.15.26:0/494828411 >> v1:172.21.15.26:6789/0 conn(0x7f67480a5d50 0x7f6748141a20 :-1 s=OPENED pgs=3841 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f6740003f00 mon_map magic: 0 2024-08-19T12:51:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 1 -- 172.21.15.26:0/494828411 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6740003990 con 0x7f67480a5d50 2024-08-19T12:51:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f6746ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f674e798640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.681+0000 7f674e798640 4 set_mon_vals no callback set 2024-08-19T12:51:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f674e798640 10 monclient: shutdown 2024-08-19T12:51:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f674e798640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:31.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f674e798640 1 -- 172.21.15.26:0/494828411 >> v1:172.21.15.26:6789/0 conn(0x7f67480a5d50 legacy=0x7f6748141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f6746ffd640 1 -- 172.21.15.26:0/494828411 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6740003f00 con 0x7f67480a5d50 2024-08-19T12:51:31.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f6746ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:51:31.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f674e798640 1 -- 172.21.15.26:0/494828411 shutdown_connections 2024-08-19T12:51:31.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f674e798640 1 -- 172.21.15.26:0/494828411 >> 172.21.15.26:0/494828411 conn(0x7f67480a8740 msgr2=0x7f67480a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:31.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f674e798640 1 -- 172.21.15.26:0/494828411 shutdown_connections 2024-08-19T12:51:31.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f674e798640 1 -- 172.21.15.26:0/494828411 wait complete. 2024-08-19T12:51:31.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f674e798640 10 monclient: build_initial_monmap 2024-08-19T12:51:31.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f674e798640 10 monclient: monmap: 2024-08-19T12:51:31.699 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:31.700 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f674e798640 1 librados: starting msgr at 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.682+0000 7f674e798640 1 librados: starting objecter 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 1 Processor -- start 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 1 -- start start 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 1 librados: setting wanted keys 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 1 librados: calling monclient init 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 10 monclient: init 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 10 monclient: _add_conns ranks=[0,5,1,4,3,8,6,7,2] 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 10 monclient(hunting): picked mon.noname-a con 0x7f674806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 10 monclient(hunting): picked mon.noname-h con 0x7f67480a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 10 monclient(hunting): picked mon.noname-f con 0x7f67480a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6748123a40 con 0x7f674806e7b0 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6748125500 con 0x7f67480a5d50 2024-08-19T12:51:31.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67481257e0 con 0x7f67480a5920 2024-08-19T12:51:31.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:31.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f674e798640 10 monclient(hunting): authenticate will time out at 1485.770753s 2024-08-19T12:51:31.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f6747fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f674806e7b0 0x7f6748141f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60194/0 (socket says 172.21.15.26:60194) 2024-08-19T12:51:31.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.683+0000 7f6747fff640 1 -- 172.21.15.26:0/267283182 learned_addr learned my addr 172.21.15.26:0/267283182 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:31.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.685+0000 7f6747fff640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.26:6789/0 conn(0x7f674806e7b0 legacy=0x7f6748141f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.685+0000 7f6744ff9640 10 client.?.objecter ms_handle_connect 0x7f674806e7b0 2024-08-19T12:51:31.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.685+0000 7f6744ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 843520370 0 0) 0x7f6748123a40 con 0x7f674806e7b0 2024-08-19T12:51:31.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): my global_id is 145341 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6734004eb0 con 0x7f674806e7b0 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 client.?.objecter ms_handle_connect 0x7f67480a5920 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 client.?.objecter ms_handle_connect 0x7f67480a5d50 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f67477fe640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.104:6791/0 conn(0x7f67480a5920 legacy=0x7f67481d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f674cd35640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.104:6789/0 conn(0x7f67480a5d50 legacy=0x7f67481d6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3293533899 0 0) 0x7f6734004eb0 con 0x7f674806e7b0 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6748123a40 con 0x7f674806e7b0 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6738001400 con 0x7f674806e7b0 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:31.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): dump: 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.704 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2519215132 0 0) 0x7f6748125500 con 0x7f67480a5d50 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): my global_id is 184168 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6734004eb0 con 0x7f67480a5d50 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3187647107 0 0) 0x7f67481257e0 con 0x7f67480a5920 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient(hunting): my global_id is 184172 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6748125500 con 0x7f67480a5920 2024-08-19T12:51:31.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1708241216 0 0) 0x7f6748123a40 con 0x7f674806e7b0 2024-08-19T12:51:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.104:6791/0 conn(0x7f67480a5920 legacy=0x7f67481d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.104:6789/0 conn(0x7f67480a5d50 legacy=0x7f67481d6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:51:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 monclient: found mon.a 2024-08-19T12:51:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6748125ac0 con 0x7f674806e7b0 2024-08-19T12:51:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient: _finish_auth 0 2024-08-19T12:51:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:51:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f674e798640 5 monclient: authenticate success, global_id 145341 2024-08-19T12:51:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f674e798640 10 monclient: _renew_subs 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f674e798640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f674e798640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f67481251f0 con 0x7f674806e7b0 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f674e798640 10 client.145341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f674e798640 10 monclient: _renew_subs 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f674e798640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f674e798640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6748126780 con 0x7f674806e7b0 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f674e798640 1 librados: init done 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6738004c70 con 0x7f674806e7b0 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6738005160 con 0x7f674806e7b0 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f67467fc640 4 set_mon_vals no callback set 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.686+0000 7f6744ff9640 10 monclient: dump: 2024-08-19T12:51:31.707 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.708 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.687+0000 7f674e798640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.687+0000 7f674e798640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f67280065c0 con 0x7f6728002860 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.687+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6738005300 con 0x7f674806e7b0 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.687+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(482..482 src has 350..482) ==== 6214+0+0 (unknown 1180833201 0 0) 0x7f673804e620 con 0x7f674806e7b0 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.687+0000 7f6744ff9640 10 client.145341.objecter ms_dispatch 0x7f6748000b90 osd_map(482..482 src has 350..482) 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.687+0000 7f6744ff9640 3 client.145341.objecter handle_osd_map got epochs [482,482] > 0 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.687+0000 7f6744ff9640 3 client.145341.objecter handle_osd_map decoding full epoch 482 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.688+0000 7f6744ff9640 20 client.145341.objecter dump_active .. 0 homeless 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.688+0000 7f674cd35640 10 monclient: get_auth_request con 0x7f67340411b0 auth_method 0 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.689+0000 7f67477fe640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.104:6792/0 conn(0x7f6728002860 legacy=0x7f6728004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.689+0000 7f6744ff9640 10 client.145341.objecter ms_handle_connect 0x7f6728002860 2024-08-19T12:51:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.689+0000 7f6744ff9640 10 client.145341.objecter resend_mon_ops 2024-08-19T12:51:31.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4107145862 0 0) 0x7f67280065c0 con 0x7f6728002860 2024-08-19T12:51:31.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 10 monclient(hunting): my global_id is 135880 2024-08-19T12:51:31.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6748123a40 con 0x7f6728002860 2024-08-19T12:51:31.710 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1290167078 0 0) 0x7f6748123a40 con 0x7f6728002860 2024-08-19T12:51:31.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67280065c0 con 0x7f6728002860 2024-08-19T12:51:31.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f673c001cf0 con 0x7f6728002860 2024-08-19T12:51:31.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:51:31.711 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 10 monclient: dump: 2024-08-19T12:51:31.712 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.712 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.712 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.712 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.712 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.712 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.712 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.713 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.713 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.713 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.713 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.713 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.713 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.713 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.713 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.713 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 10 client.145341.objecter ms_handle_connect 0x7f67340411b0 2024-08-19T12:51:31.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3466847216 0 0) 0x7f67280065c0 con 0x7f6728002860 2024-08-19T12:51:31.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f672800a160 con 0x7f6728002860 2024-08-19T12:51:31.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f673c0057c0 con 0x7f6728002860 2024-08-19T12:51:31.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:31.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.104:6792/0 conn(0x7f6728002860 legacy=0x7f6728004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.690+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.104:6792/0 conn(0x7f6728002860 legacy=0x7f6728004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.697+0000 7f674e798640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.697+0000 7f674e798640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6728009e50 con 0x7f6728009110 2024-08-19T12:51:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.699+0000 7f67477fe640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.104:6792/0 conn(0x7f6728009110 legacy=0x7f67280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.699+0000 7f6744ff9640 10 client.145341.objecter ms_handle_connect 0x7f6728009110 2024-08-19T12:51:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.699+0000 7f6744ff9640 10 client.145341.objecter resend_mon_ops 2024-08-19T12:51:31.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.699+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 442657470 0 0) 0x7f6728009e50 con 0x7f6728009110 2024-08-19T12:51:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.699+0000 7f6744ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.699+0000 7f6744ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.699+0000 7f6744ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.699+0000 7f6744ff9640 10 monclient(hunting): my global_id is 135889 2024-08-19T12:51:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.699+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67280065c0 con 0x7f6728009110 2024-08-19T12:51:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2680544146 0 0) 0x7f67280065c0 con 0x7f6728009110 2024-08-19T12:51:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6728009e50 con 0x7f6728009110 2024-08-19T12:51:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f673c003e00 con 0x7f6728009110 2024-08-19T12:51:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:51:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 10 monclient: dump: 2024-08-19T12:51:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.718 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4263985980 0 0) 0x7f6728009e50 con 0x7f6728009110 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f672800c3f0 con 0x7f6728009110 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 1502362055) 0x7f673c0040a0 con 0x7f6728009110 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.104:6792/0 conn(0x7f6728009110 legacy=0x7f67280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.700+0000 7f6744ff9640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.104:6792/0 conn(0x7f6728009110 legacy=0x7f67280095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:31.720 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:31.720 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:31.720 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:31.720 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:31.720 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:31.720 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 662057, 2024-08-19T12:51:31.720 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:31.720 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:31.720 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:31.720 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.721 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.722 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.723 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.723 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.723 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.723 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.723 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.723 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.723 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.723 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.723 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.723 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:31.723 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:31.724 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:31.725 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:31.727 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.728 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:31.733 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.734 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:31.736 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:31.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:31.738 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:31.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:51:31.740 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:31.741 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 10 librados: watch_flush enter 2024-08-19T12:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 10 librados: watch_flush exit 2024-08-19T12:51:31.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 20 client.145341.objecter shutdown clearing up homeless session... 2024-08-19T12:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 10 client.145341.objecter successfully canceled tick 2024-08-19T12:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.104:6800/899887506 conn(0x7f67340411b0 legacy=0x7f6734043670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 10 monclient: shutdown 2024-08-19T12:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 1 -- 172.21.15.26:0/267283182 >> v1:172.21.15.26:6789/0 conn(0x7f674806e7b0 legacy=0x7f6748141f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674cd35640 1 -- 172.21.15.26:0/267283182 reap_dead start 2024-08-19T12:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674cd35640 1 -- 172.21.15.26:0/267283182 reap_dead start 2024-08-19T12:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 1 -- 172.21.15.26:0/267283182 shutdown_connections 2024-08-19T12:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 1 -- 172.21.15.26:0/267283182 >> 172.21.15.26:0/267283182 conn(0x7f67480a8740 msgr2=0x7f67480aa680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 1 -- 172.21.15.26:0/267283182 shutdown_connections 2024-08-19T12:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 1 -- 172.21.15.26:0/267283182 wait complete. 2024-08-19T12:51:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.701+0000 7f674e798640 1 librados: shutdown 2024-08-19T12:51:31.746 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:31.746 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T12:51:31.746 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T12:51:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.845+0000 7fa607456640 10 monclient: get_monmap_and_config 2024-08-19T12:51:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.845+0000 7fa607456640 10 monclient: build_initial_monmap 2024-08-19T12:51:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.845+0000 7fa607456640 10 monclient: monmap: 2024-08-19T12:51:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:31.846 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:31.845917+0000 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:31.845917+0000 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.845+0000 7fa607456640 1 Processor -- start 2024-08-19T12:51:31.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.845+0000 7fa607456640 1 -- start start 2024-08-19T12:51:31.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.845+0000 7fa607456640 10 monclient: init 2024-08-19T12:51:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.846+0000 7fa607456640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.846+0000 7fa607456640 10 monclient: _add_conns ranks=[0,3,1,2,5,7,6,4,8] 2024-08-19T12:51:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.846+0000 7fa607456640 10 monclient(hunting): picked mon.noname-a con 0x7fa600073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.846+0000 7fa607456640 10 monclient(hunting): picked mon.noname-g con 0x7fa6000a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.846+0000 7fa607456640 10 monclient(hunting): picked mon.noname-f con 0x7fa60013f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:51:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.846+0000 7fa607456640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa60009dae0 con 0x7fa600073470 2024-08-19T12:51:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.846+0000 7fa607456640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa600136910 con 0x7fa60013f2e0 2024-08-19T12:51:31.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.846+0000 7fa607456640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa60009dd50 con 0x7fa6000a57a0 2024-08-19T12:51:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.846+0000 7fa607456640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.846+0000 7fa607456640 10 monclient(hunting): authenticate will time out at 1485.933518s 2024-08-19T12:51:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.846+0000 7fa6051f2640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa600073470 0x7fa600136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60204/0 (socket says 172.21.15.26:60204) 2024-08-19T12:51:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.846+0000 7fa6051f2640 1 -- 172.21.15.26:0/2129805685 learned_addr learned my addr 172.21.15.26:0/2129805685 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.848+0000 7fa6051f2640 1 -- 172.21.15.26:0/2129805685 >> v1:172.21.15.26:6789/0 conn(0x7fa600073470 legacy=0x7fa600136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.848+0000 7fa5fffff640 1 -- 172.21.15.26:0/2129805685 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 94886464 0 0) 0x7fa60009dae0 con 0x7fa600073470 2024-08-19T12:51:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.848+0000 7fa5fffff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.848+0000 7fa5fffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.848+0000 7fa5fffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.848+0000 7fa5fffff640 10 monclient(hunting): my global_id is 145350 2024-08-19T12:51:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.848+0000 7fa5fffff640 1 -- 172.21.15.26:0/2129805685 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5e8003520 con 0x7fa600073470 2024-08-19T12:51:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.848+0000 7fa6049f1640 1 -- 172.21.15.26:0/2129805685 >> v1:172.21.15.104:6790/0 conn(0x7fa6000a57a0 legacy=0x7fa6000a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.848+0000 7fa6059f3640 1 -- 172.21.15.26:0/2129805685 >> v1:172.21.15.104:6789/0 conn(0x7fa60013f2e0 legacy=0x7fa6001416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 1 -- 172.21.15.26:0/2129805685 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4202984129 0 0) 0x7fa60009dd50 con 0x7fa6000a57a0 2024-08-19T12:51:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient(hunting): my global_id is 154731 2024-08-19T12:51:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 1 -- 172.21.15.26:0/2129805685 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa60009dae0 con 0x7fa6000a57a0 2024-08-19T12:51:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 1 -- 172.21.15.26:0/2129805685 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 126205569 0 0) 0x7fa5e8003520 con 0x7fa600073470 2024-08-19T12:51:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 1 -- 172.21.15.26:0/2129805685 >> v1:172.21.15.104:6790/0 conn(0x7fa6000a57a0 legacy=0x7fa6000a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 1 -- 172.21.15.26:0/2129805685 >> v1:172.21.15.104:6789/0 conn(0x7fa60013f2e0 legacy=0x7fa6001416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient: _finish_hunting 0 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 1 monclient: found mon.noname-a 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 1 -- 172.21.15.26:0/2129805685 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60009e030 con 0x7fa600073470 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient: _finish_auth 0 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient: _check_auth_tickets 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa607456640 5 monclient: authenticate success, global_id 145350 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa607456640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 1 -- 172.21.15.26:0/2129805685 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa5f8001ae0 con 0x7fa600073470 2024-08-19T12:51:31.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:31.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient: dump: 2024-08-19T12:51:31.857 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.857 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.857 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.857 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.857 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.857 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.857 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.857 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.857 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.858 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.858 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.858 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.858 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.858 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.858 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.858 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa607456640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:31.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 1 -- 172.21.15.26:0/2129805685 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa5f80048a0 con 0x7fa600073470 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 1 -- 172.21.15.26:0/2129805685 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa5f8004e10 con 0x7fa600073470 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa607456640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa607456640 4 set_mon_vals no callback set 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa5fffff640 10 monclient: dump: 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.859 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa607456640 10 monclient: shutdown 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa607456640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa607456640 1 -- 172.21.15.26:0/2129805685 >> v1:172.21.15.26:6789/0 conn(0x7fa600073470 legacy=0x7fa600136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa607456640 1 -- 172.21.15.26:0/2129805685 shutdown_connections 2024-08-19T12:51:31.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa607456640 1 -- 172.21.15.26:0/2129805685 >> 172.21.15.26:0/2129805685 conn(0x7fa6000abff0 msgr2=0x7fa6000ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa607456640 1 -- 172.21.15.26:0/2129805685 shutdown_connections 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.849+0000 7fa607456640 1 -- 172.21.15.26:0/2129805685 wait complete. 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.850+0000 7fa607456640 10 monclient: build_initial_monmap 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.850+0000 7fa607456640 10 monclient: monmap: 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:31.861 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:31.862 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.850+0000 7fa607456640 1 librados: starting msgr at 2024-08-19T12:51:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.850+0000 7fa607456640 1 librados: starting objecter 2024-08-19T12:51:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.850+0000 7fa607456640 1 Processor -- start 2024-08-19T12:51:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.850+0000 7fa607456640 1 -- start start 2024-08-19T12:51:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.850+0000 7fa607456640 1 librados: setting wanted keys 2024-08-19T12:51:31.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.850+0000 7fa607456640 1 librados: calling monclient init 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.850+0000 7fa607456640 10 monclient: init 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa607456640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa607456640 10 monclient: _add_conns ranks=[6,0,5,8,1,3,4,7,2] 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa607456640 10 monclient(hunting): picked mon.noname-d con 0x7fa600073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa607456640 10 monclient(hunting): picked mon.noname-a con 0x7fa6000a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa607456640 10 monclient(hunting): picked mon.noname-h con 0x7fa60013f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa607456640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa600142b80 con 0x7fa6000a57a0 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa607456640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6001370e0 con 0x7fa60013f2e0 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa607456640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa60013f0b0 con 0x7fa600073470 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa607456640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa607456640 10 monclient(hunting): authenticate will time out at 1485.938458s 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa6051f2640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fa600073470 0x7fa6000a2280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51702/0 (socket says 172.21.15.26:51702) 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa6051f2640 1 -- 172.21.15.26:0/347340632 learned_addr learned my addr 172.21.15.26:0/347340632 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:31.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.851+0000 7fa6049f1640 1 --1- 172.21.15.26:0/347340632 >> v1:172.21.15.26:6789/0 conn(0x7fa6000a57a0 0x7fa6000a2990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60220/0 (socket says 172.21.15.26:60220) 2024-08-19T12:51:31.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa6049f1640 1 -- 172.21.15.26:0/347340632 >> v1:172.21.15.26:6789/0 conn(0x7fa6000a57a0 legacy=0x7fa6000a2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 client.?.objecter ms_handle_connect 0x7fa6000a57a0 2024-08-19T12:51:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa6051f2640 1 -- 172.21.15.26:0/347340632 >> v1:172.21.15.26:6792/0 conn(0x7fa600073470 legacy=0x7fa6000a2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 client.?.objecter ms_handle_connect 0x7fa600073470 2024-08-19T12:51:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 513634101 0 0) 0x7fa60009dae0 con 0x7fa6000a57a0 2024-08-19T12:51:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 monclient(hunting): my global_id is 145359 2024-08-19T12:51:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa6059f3640 1 -- 172.21.15.26:0/347340632 >> v1:172.21.15.104:6791/0 conn(0x7fa60013f2e0 legacy=0x7fa6001d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:31.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5ec004eb0 con 0x7fa6000a57a0 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 client.?.objecter ms_handle_connect 0x7fa60013f2e0 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 399318287 0 0) 0x7fa60013f0b0 con 0x7fa600073470 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 monclient(hunting): my global_id is 164634 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa60009dae0 con 0x7fa600073470 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2904643475 0 0) 0x7fa6001370e0 con 0x7fa60013f2e0 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:31.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 10 monclient(hunting): my global_id is 184181 2024-08-19T12:51:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.853+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa60013f0b0 con 0x7fa60013f2e0 2024-08-19T12:51:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 890269791 0 0) 0x7fa5ec004eb0 con 0x7fa6000a57a0 2024-08-19T12:51:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6001370e0 con 0x7fa6000a57a0 2024-08-19T12:51:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa5f4003610 con 0x7fa6000a57a0 2024-08-19T12:51:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient(hunting): dump: 2024-08-19T12:51:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.874 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.874 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.874 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.874 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.874 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4115664316 0 0) 0x7fa60009dae0 con 0x7fa600073470 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5ec004eb0 con 0x7fa600073470 2024-08-19T12:51:31.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa5f8004d90 con 0x7fa600073470 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient(hunting): dump: 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.876 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3365526782 0 0) 0x7fa60013f0b0 con 0x7fa60013f2e0 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa60009dae0 con 0x7fa60013f2e0 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa5f0001ff0 con 0x7fa60013f2e0 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient(hunting): dump: 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.877 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.878 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.879 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 834191723 0 0) 0x7fa6001370e0 con 0x7fa6000a57a0 2024-08-19T12:51:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 >> v1:172.21.15.26:6792/0 conn(0x7fa600073470 legacy=0x7fa6000a2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 >> v1:172.21.15.104:6791/0 conn(0x7fa60013f2e0 legacy=0x7fa6001d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient: _finish_hunting 0 2024-08-19T12:51:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 monclient: found mon.a 2024-08-19T12:51:31.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6000a4460 con 0x7fa6000a57a0 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient: _finish_auth 0 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient: _check_auth_tickets 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa607456640 5 monclient: authenticate success, global_id 145359 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa607456640 10 monclient: _renew_subs 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa607456640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa607456640 1 -- 172.21.15.26:0/347340632 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa600136e00 con 0x7fa6000a57a0 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa5f4003b90 con 0x7fa6000a57a0 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa5f40040c0 con 0x7fa6000a57a0 2024-08-19T12:51:31.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5fdffb640 10 monclient: dump: 2024-08-19T12:51:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:31.881 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa607456640 10 client.145359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:31.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa607456640 10 monclient: _renew_subs 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa607456640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa607456640 1 -- 172.21.15.26:0/347340632 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa600135250 con 0x7fa6000a57a0 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa5ff7fe640 4 set_mon_vals no callback set 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.854+0000 7fa607456640 1 librados: init done 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.855+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa5f4004260 con 0x7fa6000a57a0 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.855+0000 7fa607456640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.855+0000 7fa607456640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.855+0000 7fa607456640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.855+0000 7fa607456640 1 -- 172.21.15.26:0/347340632 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6000008d0 con 0x7fa6000a57a0 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.855+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(482..482 src has 350..482) ==== 6214+0+0 (unknown 1180833201 0 0) 0x7fa5f404e470 con 0x7fa6000a57a0 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.855+0000 7fa5fdffb640 10 client.145359.objecter ms_dispatch 0x7fa600000b90 osd_map(482..482 src has 350..482) 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.855+0000 7fa5fdffb640 3 client.145359.objecter handle_osd_map got epochs [482,482] > 0 2024-08-19T12:51:31.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.855+0000 7fa5fdffb640 3 client.145359.objecter handle_osd_map decoding full epoch 482 2024-08-19T12:51:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.855+0000 7fa5fdffb640 20 client.145359.objecter dump_active .. 0 homeless 2024-08-19T12:51:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.855+0000 7fa6051f2640 10 monclient: get_auth_request con 0x7fa5ec0418d0 auth_method 0 2024-08-19T12:51:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.858+0000 7fa5fdffb640 10 client.145359.objecter ms_handle_connect 0x7fa5ec0418d0 2024-08-19T12:51:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.860+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fa600142b80 con 0x7fa6000a57a0 2024-08-19T12:51:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.860+0000 7fa5fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:31.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:31.860+0000 7fa5fdffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:32.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.118+0000 7fa607456640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T12:51:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.118+0000 7fa607456640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T12:51:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.118+0000 7fa607456640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.118+0000 7fa607456640 1 -- 172.21.15.26:0/347340632 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa600142df0 con 0x7fa6000a57a0 2024-08-19T12:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.119+0000 7fa5fdffb640 1 -- 172.21.15.26:0/347340632 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fa5f4017510 con 0x7fa6000a57a0 2024-08-19T12:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.119+0000 7fa5fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T12:51:32.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.119+0000 7fa5fdffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:32.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.122+0000 7fa607456640 10 librados: watch_flush enter 2024-08-19T12:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.123+0000 7fa607456640 10 librados: watch_flush exit 2024-08-19T12:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.123+0000 7fa607456640 20 client.145359.objecter shutdown clearing up homeless session... 2024-08-19T12:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.123+0000 7fa607456640 10 client.145359.objecter successfully canceled tick 2024-08-19T12:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.123+0000 7fa607456640 1 -- 172.21.15.26:0/347340632 >> v1:172.21.15.104:6800/899887506 conn(0x7fa5ec0418d0 legacy=0x7fa5ec043d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.123+0000 7fa607456640 10 monclient: shutdown 2024-08-19T12:51:32.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.123+0000 7fa607456640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.123+0000 7fa607456640 1 -- 172.21.15.26:0/347340632 >> v1:172.21.15.26:6789/0 conn(0x7fa6000a57a0 legacy=0x7fa6000a2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.124+0000 7fa607456640 1 -- 172.21.15.26:0/347340632 shutdown_connections 2024-08-19T12:51:32.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.124+0000 7fa607456640 1 -- 172.21.15.26:0/347340632 >> 172.21.15.26:0/347340632 conn(0x7fa6000abff0 msgr2=0x7fa6001338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.125+0000 7fa607456640 1 -- 172.21.15.26:0/347340632 shutdown_connections 2024-08-19T12:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.125+0000 7fa607456640 1 -- 172.21.15.26:0/347340632 wait complete. 2024-08-19T12:51:32.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:32.125+0000 7fa607456640 1 librados: shutdown 2024-08-19T12:51:32.129 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.127+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.129 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.127+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=10,kv=11,logm=52} crc {auth=3645453739,config=1228700967,health=2038360310,kv=3904374251,logm=951262095}) 2024-08-19T12:51:32.129 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=12,logm=48} crc {auth=2002869679,config=1097430493,health=2488849171,kv=2934104134,logm=3266634750}) 2024-08-19T12:51:32.129 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.129 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=10,kv=11,logm=52} crc {auth=3645453739,config=1228700967,health=2038360310,kv=3904374251,logm=951262095}) 2024-08-19T12:51:32.129 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=13,logm=45} crc {auth=2957918155,config=1097430493,health=2408690321,kv=1344821656,logm=1688872499}) 2024-08-19T12:51:32.129 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.129 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=10,kv=11,logm=52} crc {auth=3645453739,config=1228700967,health=2038360310,kv=3904374251,logm=951262095}) 2024-08-19T12:51:32.129 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=28,config=2,health=11,kv=13,logm=46} crc {auth=2957918155,config=1097430494,health=1832372627,kv=1344821656,logm=244099920}) 2024-08-19T12:51:32.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=10,kv=11,logm=52} crc {auth=3645453739,config=1228700967,health=2038360310,kv=3904374251,logm=951262095}) 2024-08-19T12:51:32.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=2957918155,config=1097430493,health=2408690321,kv=858770998,logm=3363787449}) 2024-08-19T12:51:32.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=10,kv=11,logm=52} crc {auth=3645453739,config=1228700967,health=2038360310,kv=3904374251,logm=951262095}) 2024-08-19T12:51:32.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=26,config=2,health=10,kv=13,logm=49} crc {auth=2942439378,config=1097430493,health=3427717332,kv=4140988685,logm=484703584}) 2024-08-19T12:51:32.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=10,kv=11,logm=52} crc {auth=3645453739,config=1228700967,health=2038360310,kv=3904374251,logm=951262095}) 2024-08-19T12:51:32.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=27,config=2,health=11,kv=11,logm=49} crc {auth=184783646,config=1097430493,health=2268952674,kv=1293530160,logm=351293391}) 2024-08-19T12:51:32.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=10,kv=11,logm=52} crc {auth=3645453739,config=1228700967,health=2038360310,kv=3904374251,logm=951262095}) 2024-08-19T12:51:32.130 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=27,config=2,health=12,kv=13,logm=46} crc {auth=2406731470,config=1097430493,health=2408690321,kv=1344821656,logm=1423479410}) 2024-08-19T12:51:32.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=10,kv=11,logm=52} crc {auth=3645453739,config=1228700967,health=2038360310,kv=3904374251,logm=951262095}) 2024-08-19T12:51:32.131 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.128+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=3341539930,config=1097430493,health=2408690322,kv=1466468802,logm=2512064716}) 2024-08-19T12:51:32.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278931453}) 2024-08-19T12:51:32.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=734029928}) 2024-08-19T12:51:32.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278931453}) 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1843409517}) 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278931453}) 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2266160709}) 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278931453}) 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=74477706}) 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278931453}) 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.135+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3960409136}) 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.136+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.136+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278931453}) 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.136+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4142440842}) 2024-08-19T12:51:32.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.136+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.136+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278931453}) 2024-08-19T12:51:32.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.136+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147803592}) 2024-08-19T12:51:32.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.136+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.136+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1278931453}) 2024-08-19T12:51:32.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.136+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1874628525}) 2024-08-19T12:51:32.147 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T12:51:32.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.560+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.560+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169606324}) 2024-08-19T12:51:32.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.560+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=687215}) 2024-08-19T12:51:32.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.560+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.560+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169606324}) 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.560+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3945115183}) 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.560+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.560+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169606324}) 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.560+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4083771756}) 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.560+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.560+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169606324}) 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1436588171}) 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169606324}) 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=305105204}) 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169606324}) 2024-08-19T12:51:32.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4166894595}) 2024-08-19T12:51:32.565 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.565 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169606324}) 2024-08-19T12:51:32.565 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2699029017}) 2024-08-19T12:51:32.565 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:32.565 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169606324}) 2024-08-19T12:51:32.565 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:32.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=553936398}) 2024-08-19T12:51:33.123 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.123 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919044457}) 2024-08-19T12:51:33.123 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=559693001}) 2024-08-19T12:51:33.123 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.123 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919044457}) 2024-08-19T12:51:33.123 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=128599611}) 2024-08-19T12:51:33.123 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.123 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919044457}) 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1974169622}) 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919044457}) 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3355104724}) 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919044457}) 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=46022941}) 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919044457}) 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4220268060}) 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919044457}) 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2701787735}) 2024-08-19T12:51:33.124 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.125 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919044457}) 2024-08-19T12:51:33.125 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.121+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=619531115}) 2024-08-19T12:51:33.133 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.133 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597273069}) 2024-08-19T12:51:33.133 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1984497289}) 2024-08-19T12:51:33.133 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597273069}) 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3500270579}) 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597273069}) 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=166608207}) 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597273069}) 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3805407773}) 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597273069}) 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=519377811}) 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597273069}) 2024-08-19T12:51:33.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3309761284}) 2024-08-19T12:51:33.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597273069}) 2024-08-19T12:51:33.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=264749558}) 2024-08-19T12:51:33.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3597273069}) 2024-08-19T12:51:33.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2884791209}) 2024-08-19T12:51:33.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50865570}) 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3438863421}) 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50865570}) 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323709989}) 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50865570}) 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4025893038}) 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50865570}) 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=796720836}) 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50865570}) 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2529503841}) 2024-08-19T12:51:33.140 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.141 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50865570}) 2024-08-19T12:51:33.141 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4277001306}) 2024-08-19T12:51:33.141 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.138+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.141 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.139+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50865570}) 2024-08-19T12:51:33.141 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.139+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3424176677}) 2024-08-19T12:51:33.141 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.139+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.141 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.139+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50865570}) 2024-08-19T12:51:33.141 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.139+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4111782186}) 2024-08-19T12:51:33.148 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:51:33.148 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:51:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.251+0000 7eff6f238640 10 monclient: get_monmap_and_config 2024-08-19T12:51:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.251+0000 7eff6f238640 10 monclient: build_initial_monmap 2024-08-19T12:51:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.251+0000 7eff6f238640 10 monclient: monmap: 2024-08-19T12:51:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:33.252360+0000 2024-08-19T12:51:33.252 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:33.252360+0000 2024-08-19T12:51:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:33.253 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:33.253 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 1 Processor -- start 2024-08-19T12:51:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 1 -- start start 2024-08-19T12:51:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 10 monclient: init 2024-08-19T12:51:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 10 monclient: _add_conns ranks=[0,7,6,8,1,2,3,4,5] 2024-08-19T12:51:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 10 monclient(hunting): picked mon.noname-a con 0x7eff6806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 10 monclient(hunting): picked mon.noname-i con 0x7eff680a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 10 monclient(hunting): picked mon.noname-d con 0x7eff680a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:33.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff68136790 con 0x7eff6806e7b0 2024-08-19T12:51:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff6809ddf0 con 0x7eff680a5d50 2024-08-19T12:51:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff6809e0d0 con 0x7eff680a5920 2024-08-19T12:51:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6f238640 10 monclient(hunting): authenticate will time out at 1487.339834s 2024-08-19T12:51:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6cfd4640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7eff6806e7b0 0x7eff68136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60246/0 (socket says 172.21.15.26:60246) 2024-08-19T12:51:33.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.252+0000 7eff6cfd4640 1 -- 172.21.15.26:0/7558549 learned_addr learned my addr 172.21.15.26:0/7558549 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff6cfd4640 1 -- 172.21.15.26:0/7558549 >> v1:172.21.15.26:6789/0 conn(0x7eff6806e7b0 legacy=0x7eff68136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff6d7d5640 1 -- 172.21.15.26:0/7558549 >> v1:172.21.15.26:6792/0 conn(0x7eff680a5d50 legacy=0x7eff68141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 1 -- 172.21.15.26:0/7558549 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 329190009 0 0) 0x7eff68136790 con 0x7eff6806e7b0 2024-08-19T12:51:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient(hunting): my global_id is 145368 2024-08-19T12:51:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 1 -- 172.21.15.26:0/7558549 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff58003520 con 0x7eff6806e7b0 2024-08-19T12:51:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 1 -- 172.21.15.26:0/7558549 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3590520009 0 0) 0x7eff60001820 con 0x7eff680a5d50 2024-08-19T12:51:33.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient(hunting): my global_id is 164643 2024-08-19T12:51:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 1 -- 172.21.15.26:0/7558549 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff68136790 con 0x7eff680a5d50 2024-08-19T12:51:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff67fff640 1 -- 172.21.15.26:0/7558549 >> v1:172.21.15.104:6792/0 conn(0x7eff680a5920 legacy=0x7eff6813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 1 -- 172.21.15.26:0/7558549 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1620793442 0 0) 0x7eff58003520 con 0x7eff6806e7b0 2024-08-19T12:51:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 1 -- 172.21.15.26:0/7558549 >> v1:172.21.15.104:6792/0 conn(0x7eff680a5920 legacy=0x7eff6813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 1 -- 172.21.15.26:0/7558549 >> v1:172.21.15.26:6792/0 conn(0x7eff680a5d50 legacy=0x7eff68141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient: _finish_hunting 0 2024-08-19T12:51:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 1 monclient: found mon.noname-a 2024-08-19T12:51:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:33.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 1 -- 172.21.15.26:0/7558549 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6809e3b0 con 0x7eff6806e7b0 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient: _finish_auth 0 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient: _check_auth_tickets 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 1 -- 172.21.15.26:0/7558549 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7eff5c001ae0 con 0x7eff6806e7b0 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient: dump: 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.267 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff6f238640 5 monclient: authenticate success, global_id 145368 2024-08-19T12:51:33.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff6f238640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff6cfd4640 1 --1- 172.21.15.26:0/7558549 >> v1:172.21.15.26:6789/0 conn(0x7eff6806e7b0 0x7eff68136080 :-1 s=OPENED pgs=3849 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7eff5c004e10 mon_map magic: 0 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 1 -- 172.21.15.26:0/7558549 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff5c0048a0 con 0x7eff6806e7b0 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff677fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff6f238640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.255+0000 7eff6f238640 4 set_mon_vals no callback set 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.256+0000 7eff6f238640 10 monclient: shutdown 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.256+0000 7eff6f238640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.256+0000 7eff6f238640 1 -- 172.21.15.26:0/7558549 >> v1:172.21.15.26:6789/0 conn(0x7eff6806e7b0 legacy=0x7eff68136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.256+0000 7eff677fe640 1 -- 172.21.15.26:0/7558549 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7eff5c004e10 con 0x7eff6806e7b0 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.256+0000 7eff677fe640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.256+0000 7eff6f238640 1 -- 172.21.15.26:0/7558549 shutdown_connections 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.256+0000 7eff6f238640 1 -- 172.21.15.26:0/7558549 >> 172.21.15.26:0/7558549 conn(0x7eff680a8740 msgr2=0x7eff680a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:33.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.256+0000 7eff6f238640 1 -- 172.21.15.26:0/7558549 shutdown_connections 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.256+0000 7eff6f238640 1 -- 172.21.15.26:0/7558549 wait complete. 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.256+0000 7eff6f238640 10 monclient: build_initial_monmap 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.256+0000 7eff6f238640 10 monclient: monmap: 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:33.270 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 1 librados: starting msgr at 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 1 librados: starting objecter 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 1 Processor -- start 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 1 -- start start 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 1 librados: setting wanted keys 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 1 librados: calling monclient init 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 10 monclient: init 2024-08-19T12:51:33.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 10 monclient: _add_conns ranks=[6,3,2,8,0,1,4,5,7] 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 10 monclient(hunting): picked mon.noname-d con 0x7eff6806e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 10 monclient(hunting): picked mon.noname-g con 0x7eff680a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 10 monclient(hunting): picked mon.noname-b con 0x7eff680a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff680a2ca0 con 0x7eff680a5d50 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff680a2f80 con 0x7eff680a5920 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff680a3260 con 0x7eff6806e7b0 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6f238640 10 monclient(hunting): authenticate will time out at 1487.344911s 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6cfd4640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7eff6806e7b0 0x7eff6809fb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51740/0 (socket says 172.21.15.26:51740) 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.257+0000 7eff6cfd4640 1 -- 172.21.15.26:0/732853068 learned_addr learned my addr 172.21.15.26:0/732853068 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff6cfd4640 1 -- 172.21.15.26:0/732853068 >> v1:172.21.15.26:6792/0 conn(0x7eff6806e7b0 legacy=0x7eff6809fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 client.?.objecter ms_handle_connect 0x7eff6806e7b0 2024-08-19T12:51:33.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 client.?.objecter ms_handle_connect 0x7eff680a5d50 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff6d7d5640 1 -- 172.21.15.26:0/732853068 >> v1:172.21.15.26:6790/0 conn(0x7eff680a5d50 legacy=0x7eff681d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 828957557 0 0) 0x7eff680a3260 con 0x7eff6806e7b0 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): my global_id is 164652 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff4c004eb0 con 0x7eff6806e7b0 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046338782 0 0) 0x7eff680a2ca0 con 0x7eff680a5d50 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): my global_id is 26921 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff680a3260 con 0x7eff680a5d50 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 client.?.objecter ms_handle_connect 0x7eff680a5920 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff67fff640 1 -- 172.21.15.26:0/732853068 >> v1:172.21.15.104:6790/0 conn(0x7eff680a5920 legacy=0x7eff680a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 359909496 0 0) 0x7eff4c004eb0 con 0x7eff6806e7b0 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff680a2ca0 con 0x7eff6806e7b0 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7eff5c001a60 con 0x7eff6806e7b0 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): dump: 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.274 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.275 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.276 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:33.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1981363532 0 0) 0x7eff680a2f80 con 0x7eff680a5920 2024-08-19T12:51:33.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): my global_id is 154740 2024-08-19T12:51:33.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff4c004eb0 con 0x7eff680a5920 2024-08-19T12:51:33.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4107323498 0 0) 0x7eff680a3260 con 0x7eff680a5d50 2024-08-19T12:51:33.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff680a2f80 con 0x7eff680a5d50 2024-08-19T12:51:33.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7eff60001840 con 0x7eff680a5d50 2024-08-19T12:51:33.276 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient(hunting): dump: 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.277 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 844362146 0 0) 0x7eff680a2ca0 con 0x7eff6806e7b0 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 >> v1:172.21.15.104:6790/0 conn(0x7eff680a5920 legacy=0x7eff680a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 >> v1:172.21.15.26:6790/0 conn(0x7eff680a5d50 legacy=0x7eff681d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient: _finish_hunting 0 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 monclient: found mon.d 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff680a1a60 con 0x7eff6806e7b0 2024-08-19T12:51:33.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient: _finish_auth 0 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient: _check_auth_tickets 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff6f238640 5 monclient: authenticate success, global_id 164652 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff6f238640 10 monclient: _renew_subs 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff6f238640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff6f238640 1 -- 172.21.15.26:0/732853068 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7eff680a2650 con 0x7eff6806e7b0 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff6f238640 10 client.164652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff6f238640 10 monclient: _renew_subs 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff6f238640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff6f238640 1 -- 172.21.15.26:0/732853068 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7eff680a0990 con 0x7eff6806e7b0 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff6f238640 1 librados: init done 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff5c0048a0 con 0x7eff6806e7b0 2024-08-19T12:51:33.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7eff5c004db0 con 0x7eff6806e7b0 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff66ffd640 4 set_mon_vals no callback set 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.260+0000 7eff657fa640 10 monclient: dump: 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.280 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.261+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7eff5c0119c0 con 0x7eff6806e7b0 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.261+0000 7eff6f238640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.261+0000 7eff6f238640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.261+0000 7eff6f238640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.261+0000 7eff6f238640 1 -- 172.21.15.26:0/732853068 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff48005080 con 0x7eff6806e7b0 2024-08-19T12:51:33.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.261+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(484..484 src has 350..484) ==== 6214+0+0 (unknown 3296332199 0 0) 0x7eff5c04e5c0 con 0x7eff6806e7b0 2024-08-19T12:51:33.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.262+0000 7eff657fa640 10 client.164652.objecter ms_dispatch 0x7eff68000b90 osd_map(484..484 src has 350..484) 2024-08-19T12:51:33.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.262+0000 7eff657fa640 3 client.164652.objecter handle_osd_map got epochs [484,484] > 0 2024-08-19T12:51:33.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.262+0000 7eff657fa640 3 client.164652.objecter handle_osd_map decoding full epoch 484 2024-08-19T12:51:33.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.262+0000 7eff657fa640 20 client.164652.objecter dump_active .. 0 homeless 2024-08-19T12:51:33.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.262+0000 7eff67fff640 10 monclient: get_auth_request con 0x7eff4c0417c0 auth_method 0 2024-08-19T12:51:33.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.264+0000 7eff657fa640 10 client.164652.objecter ms_handle_connect 0x7eff4c0417c0 2024-08-19T12:51:33.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.266+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7eff5c017680 con 0x7eff6806e7b0 2024-08-19T12:51:33.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.266+0000 7eff657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:33.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.266+0000 7eff657fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:33.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834659853}) 2024-08-19T12:51:33.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4289481430}) 2024-08-19T12:51:33.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834659853}) 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3696060718}) 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834659853}) 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3279317556}) 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834659853}) 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4212932570}) 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834659853}) 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1579082101}) 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834659853}) 2024-08-19T12:51:33.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=565079393}) 2024-08-19T12:51:33.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834659853}) 2024-08-19T12:51:33.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3041527802}) 2024-08-19T12:51:33.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834659853}) 2024-08-19T12:51:33.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.268+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1227573774}) 2024-08-19T12:51:33.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364204111}) 2024-08-19T12:51:33.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1780552985}) 2024-08-19T12:51:33.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364204111}) 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3687708354}) 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364204111}) 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=329673500}) 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364204111}) 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2577493348}) 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364204111}) 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=619234901}) 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364204111}) 2024-08-19T12:51:33.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1290243193}) 2024-08-19T12:51:33.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364204111}) 2024-08-19T12:51:33.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3068865647}) 2024-08-19T12:51:33.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364204111}) 2024-08-19T12:51:33.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.274+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4032328596}) 2024-08-19T12:51:33.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.285+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863392301}) 2024-08-19T12:51:33.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3994037101}) 2024-08-19T12:51:33.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863392301}) 2024-08-19T12:51:33.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1766601207}) 2024-08-19T12:51:33.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863392301}) 2024-08-19T12:51:33.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3831495795}) 2024-08-19T12:51:33.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863392301}) 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4074436875}) 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863392301}) 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=317334401}) 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863392301}) 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4061352031}) 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863392301}) 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=463914528}) 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863392301}) 2024-08-19T12:51:33.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.286+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=7579834}) 2024-08-19T12:51:33.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3269689483}) 2024-08-19T12:51:33.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3373663118}) 2024-08-19T12:51:33.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3269689483}) 2024-08-19T12:51:33.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830241552}) 2024-08-19T12:51:33.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3269689483}) 2024-08-19T12:51:33.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4104136682}) 2024-08-19T12:51:33.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3269689483}) 2024-08-19T12:51:33.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2426561049}) 2024-08-19T12:51:33.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3269689483}) 2024-08-19T12:51:33.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2558973221}) 2024-08-19T12:51:33.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3269689483}) 2024-08-19T12:51:33.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=765087615}) 2024-08-19T12:51:33.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3269689483}) 2024-08-19T12:51:33.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2620213831}) 2024-08-19T12:51:33.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3269689483}) 2024-08-19T12:51:33.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.304+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3734624954}) 2024-08-19T12:51:33.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384036913}) 2024-08-19T12:51:33.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=121165726}) 2024-08-19T12:51:33.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384036913}) 2024-08-19T12:51:33.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150414233}) 2024-08-19T12:51:33.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384036913}) 2024-08-19T12:51:33.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=527379351}) 2024-08-19T12:51:33.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384036913}) 2024-08-19T12:51:33.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3141036718}) 2024-08-19T12:51:33.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384036913}) 2024-08-19T12:51:33.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3983265286}) 2024-08-19T12:51:33.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384036913}) 2024-08-19T12:51:33.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2141165926}) 2024-08-19T12:51:33.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384036913}) 2024-08-19T12:51:33.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2919493274}) 2024-08-19T12:51:33.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384036913}) 2024-08-19T12:51:33.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.306+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4278890651}) 2024-08-19T12:51:33.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141360482}) 2024-08-19T12:51:33.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=879248201}) 2024-08-19T12:51:33.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141360482}) 2024-08-19T12:51:33.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2364024435}) 2024-08-19T12:51:33.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141360482}) 2024-08-19T12:51:33.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3903818060}) 2024-08-19T12:51:33.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141360482}) 2024-08-19T12:51:33.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3255400338}) 2024-08-19T12:51:33.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141360482}) 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=485828127}) 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141360482}) 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4262307706}) 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141360482}) 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2723932097}) 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141360482}) 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.309+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2646082080}) 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918884316}) 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2513533951}) 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918884316}) 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=893224556}) 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918884316}) 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1692895096}) 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918884316}) 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2249594873}) 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918884316}) 2024-08-19T12:51:33.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2142316657}) 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918884316}) 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=163153917}) 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918884316}) 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=388124684}) 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918884316}) 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.311+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2371414978}) 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257518320}) 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2977120528}) 2024-08-19T12:51:33.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257518320}) 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2145411221}) 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257518320}) 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1598417890}) 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257518320}) 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3125756405}) 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257518320}) 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4164985271}) 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257518320}) 2024-08-19T12:51:33.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4056243472}) 2024-08-19T12:51:33.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257518320}) 2024-08-19T12:51:33.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1224070600}) 2024-08-19T12:51:33.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.312+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.313+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257518320}) 2024-08-19T12:51:33.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.313+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2923376017}) 2024-08-19T12:51:33.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249132878}) 2024-08-19T12:51:33.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4073647952}) 2024-08-19T12:51:33.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249132878}) 2024-08-19T12:51:33.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2883950900}) 2024-08-19T12:51:33.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249132878}) 2024-08-19T12:51:33.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=916684300}) 2024-08-19T12:51:33.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249132878}) 2024-08-19T12:51:33.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2620250373}) 2024-08-19T12:51:33.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249132878}) 2024-08-19T12:51:33.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=136825723}) 2024-08-19T12:51:33.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249132878}) 2024-08-19T12:51:33.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=571182597}) 2024-08-19T12:51:33.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249132878}) 2024-08-19T12:51:33.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=534108722}) 2024-08-19T12:51:33.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249132878}) 2024-08-19T12:51:33.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.314+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1863780679}) 2024-08-19T12:51:33.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600048465}) 2024-08-19T12:51:33.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=14431132}) 2024-08-19T12:51:33.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600048465}) 2024-08-19T12:51:33.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1523766091}) 2024-08-19T12:51:33.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600048465}) 2024-08-19T12:51:33.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=904252069}) 2024-08-19T12:51:33.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600048465}) 2024-08-19T12:51:33.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1904954994}) 2024-08-19T12:51:33.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600048465}) 2024-08-19T12:51:33.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1123505041}) 2024-08-19T12:51:33.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600048465}) 2024-08-19T12:51:33.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3931430462}) 2024-08-19T12:51:33.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600048465}) 2024-08-19T12:51:33.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3170683034}) 2024-08-19T12:51:33.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600048465}) 2024-08-19T12:51:33.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.315+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1453266552}) 2024-08-19T12:51:33.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596631841}) 2024-08-19T12:51:33.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2811726735}) 2024-08-19T12:51:33.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596631841}) 2024-08-19T12:51:33.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1762689206}) 2024-08-19T12:51:33.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596631841}) 2024-08-19T12:51:33.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1924196102}) 2024-08-19T12:51:33.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596631841}) 2024-08-19T12:51:33.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1440744961}) 2024-08-19T12:51:33.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596631841}) 2024-08-19T12:51:33.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2697919843}) 2024-08-19T12:51:33.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596631841}) 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2949993756}) 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596631841}) 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3024906613}) 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596631841}) 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.317+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=55214414}) 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909041613}) 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2035868330}) 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909041613}) 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1727041141}) 2024-08-19T12:51:33.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909041613}) 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1310765356}) 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909041613}) 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3144604353}) 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909041613}) 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3475077535}) 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909041613}) 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4011634241}) 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909041613}) 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=452696310}) 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909041613}) 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.318+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1522999672}) 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094863640}) 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2819887223}) 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094863640}) 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2389453215}) 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094863640}) 2024-08-19T12:51:33.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2684212176}) 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094863640}) 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2762447415}) 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094863640}) 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1831883083}) 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094863640}) 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4061820503}) 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094863640}) 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=662833744}) 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094863640}) 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.320+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1053275133}) 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3784018524}) 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2248833777}) 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3784018524}) 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=961270022}) 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3784018524}) 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2421108075}) 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3784018524}) 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1541637111}) 2024-08-19T12:51:33.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3784018524}) 2024-08-19T12:51:33.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2724334624}) 2024-08-19T12:51:33.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3784018524}) 2024-08-19T12:51:33.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4104433699}) 2024-08-19T12:51:33.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3784018524}) 2024-08-19T12:51:33.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1839982699}) 2024-08-19T12:51:33.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3784018524}) 2024-08-19T12:51:33.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.321+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1584694501}) 2024-08-19T12:51:33.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3925935232}) 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596655631}) 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3925935232}) 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1644625079}) 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3925935232}) 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1825991950}) 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3925935232}) 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=734000706}) 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3925935232}) 2024-08-19T12:51:33.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3442906658}) 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3925935232}) 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=806814537}) 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3925935232}) 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=798317306}) 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3925935232}) 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.323+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=564181719}) 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376740664}) 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3301835319}) 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376740664}) 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=196539197}) 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376740664}) 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2164120090}) 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376740664}) 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1694763024}) 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376740664}) 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1749899513}) 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376740664}) 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=322739497}) 2024-08-19T12:51:33.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376740664}) 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1858110977}) 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376740664}) 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.324+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1321966672}) 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.325+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011547365}) 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2982866855}) 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011547365}) 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1978850653}) 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011547365}) 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1254705486}) 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011547365}) 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3430633741}) 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011547365}) 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4149177270}) 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011547365}) 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=86347473}) 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011547365}) 2024-08-19T12:51:33.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1431487845}) 2024-08-19T12:51:33.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2011547365}) 2024-08-19T12:51:33.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.326+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3416289225}) 2024-08-19T12:51:33.365 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.365 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=920249353}) 2024-08-19T12:51:33.365 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1555743431}) 2024-08-19T12:51:33.365 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.365 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=920249353}) 2024-08-19T12:51:33.365 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=505926472}) 2024-08-19T12:51:33.366 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.366 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=920249353}) 2024-08-19T12:51:33.366 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4018475837}) 2024-08-19T12:51:33.366 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.366 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=920249353}) 2024-08-19T12:51:33.366 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=243270718}) 2024-08-19T12:51:33.366 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.366 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=920249353}) 2024-08-19T12:51:33.366 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3302572190}) 2024-08-19T12:51:33.366 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.366 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=920249353}) 2024-08-19T12:51:33.366 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3107843323}) 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=920249353}) 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3466722987}) 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=920249353}) 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.335+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1390392221}) 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350026734}) 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1091845377}) 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350026734}) 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3713082142}) 2024-08-19T12:51:33.367 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350026734}) 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4159237162}) 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350026734}) 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3135173663}) 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350026734}) 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2228904079}) 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350026734}) 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=975701615}) 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350026734}) 2024-08-19T12:51:33.368 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3700686821}) 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350026734}) 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.337+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4187699310}) 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.338+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.338+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840038880}) 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.338+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2204509332}) 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.338+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.338+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840038880}) 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.338+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2407550443}) 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.338+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.338+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840038880}) 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.338+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4152371564}) 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.338+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.369 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.338+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840038880}) 2024-08-19T12:51:33.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2414191176}) 2024-08-19T12:51:33.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840038880}) 2024-08-19T12:51:33.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3798220376}) 2024-08-19T12:51:33.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840038880}) 2024-08-19T12:51:33.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2192351430}) 2024-08-19T12:51:33.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840038880}) 2024-08-19T12:51:33.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3747634847}) 2024-08-19T12:51:33.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.370 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840038880}) 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.339+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=929324249}) 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145403183}) 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1158208958}) 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145403183}) 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=648707356}) 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145403183}) 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=350349521}) 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145403183}) 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2840314494}) 2024-08-19T12:51:33.371 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145403183}) 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1564648756}) 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145403183}) 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1182974041}) 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145403183}) 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2888327636}) 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145403183}) 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.340+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2255250013}) 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318873594}) 2024-08-19T12:51:33.372 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2906846414}) 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318873594}) 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4068009463}) 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318873594}) 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=25077661}) 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318873594}) 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=761127901}) 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318873594}) 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3414367974}) 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.373 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318873594}) 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2793889354}) 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318873594}) 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3747293649}) 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318873594}) 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.341+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1957587443}) 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741050890}) 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937348375}) 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741050890}) 2024-08-19T12:51:33.374 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4070202453}) 2024-08-19T12:51:33.375 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.375 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741050890}) 2024-08-19T12:51:33.375 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3157909527}) 2024-08-19T12:51:33.375 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.375 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741050890}) 2024-08-19T12:51:33.375 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1203771756}) 2024-08-19T12:51:33.375 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.375 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741050890}) 2024-08-19T12:51:33.375 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3768802886}) 2024-08-19T12:51:33.375 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.375 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741050890}) 2024-08-19T12:51:33.375 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1888066006}) 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741050890}) 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3364683109}) 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741050890}) 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.342+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2651653904}) 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.343+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.343+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984885846}) 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.343+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1958754728}) 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984885846}) 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435165581}) 2024-08-19T12:51:33.376 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984885846}) 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2950561963}) 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984885846}) 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3825963318}) 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984885846}) 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=693975730}) 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984885846}) 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4089623200}) 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.377 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984885846}) 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1382290632}) 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984885846}) 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.344+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1407254429}) 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783423237}) 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1976603347}) 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783423237}) 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973510845}) 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783423237}) 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2330401749}) 2024-08-19T12:51:33.378 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783423237}) 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=145053040}) 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783423237}) 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3718466907}) 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783423237}) 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=166325704}) 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783423237}) 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=359424841}) 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2783423237}) 2024-08-19T12:51:33.379 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.345+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2069526208}) 2024-08-19T12:51:33.516 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.516 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556843924}) 2024-08-19T12:51:33.516 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1804955919}) 2024-08-19T12:51:33.516 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.516 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556843924}) 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1911073798}) 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556843924}) 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1026971232}) 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556843924}) 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3313669984}) 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556843924}) 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1924560440}) 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556843924}) 2024-08-19T12:51:33.517 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3028297873}) 2024-08-19T12:51:33.518 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.518 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556843924}) 2024-08-19T12:51:33.518 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=95312229}) 2024-08-19T12:51:33.518 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.513+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:33.518 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.514+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556843924}) 2024-08-19T12:51:33.518 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:33.514+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2120422844}) 2024-08-19T12:51:33.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.553+0000 7eff6f238640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:51:33.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.553+0000 7eff6f238640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:51:33.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.553+0000 7eff6f238640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:33.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.553+0000 7eff6f238640 1 -- 172.21.15.26:0/732853068 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7eff48005370 con 0x7eff6806e7b0 2024-08-19T12:51:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.554+0000 7eff657fa640 1 -- 172.21.15.26:0/732853068 <== mon.6 v1:172.21.15.26:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2729 (unknown 254746657 0 2366592125) 0x7eff5c01b520 con 0x7eff6806e7b0 2024-08-19T12:51:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.554+0000 7eff657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:51:33.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.554+0000 7eff657fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:33.555 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":92,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:51:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.555+0000 7eff6f238640 10 librados: watch_flush enter 2024-08-19T12:51:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.555+0000 7eff6f238640 10 librados: watch_flush exit 2024-08-19T12:51:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.556+0000 7eff6f238640 20 client.164652.objecter shutdown clearing up homeless session... 2024-08-19T12:51:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.556+0000 7eff6f238640 10 client.164652.objecter successfully canceled tick 2024-08-19T12:51:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.556+0000 7eff6f238640 1 -- 172.21.15.26:0/732853068 >> v1:172.21.15.104:6800/899887506 conn(0x7eff4c0417c0 legacy=0x7eff4c043c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.556+0000 7eff6f238640 10 monclient: shutdown 2024-08-19T12:51:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.556+0000 7eff6f238640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.556+0000 7eff6f238640 1 -- 172.21.15.26:0/732853068 >> v1:172.21.15.26:6792/0 conn(0x7eff6806e7b0 legacy=0x7eff6809fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.556+0000 7eff6f238640 1 -- 172.21.15.26:0/732853068 shutdown_connections 2024-08-19T12:51:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.556+0000 7eff6f238640 1 -- 172.21.15.26:0/732853068 >> 172.21.15.26:0/732853068 conn(0x7eff680a8740 msgr2=0x7eff680aa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:33.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.556+0000 7eff6f238640 1 -- 172.21.15.26:0/732853068 shutdown_connections 2024-08-19T12:51:33.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.556+0000 7eff6f238640 1 -- 172.21.15.26:0/732853068 wait complete. 2024-08-19T12:51:33.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.556+0000 7eff6f238640 1 librados: shutdown 2024-08-19T12:51:33.567 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:51:33.567 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T12:51:33.567 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:51:33.568 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:51:33.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.666+0000 7fb426b72640 10 monclient: get_monmap_and_config 2024-08-19T12:51:33.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.666+0000 7fb426b72640 10 monclient: build_initial_monmap 2024-08-19T12:51:33.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.666+0000 7fb426b72640 10 monclient: monmap: 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:33.667641+0000 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:33.667641+0000 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:33.668 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:33.669 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:33.669 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.667+0000 7fb426b72640 1 Processor -- start 2024-08-19T12:51:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.667+0000 7fb426b72640 1 -- start start 2024-08-19T12:51:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.667+0000 7fb426b72640 10 monclient: init 2024-08-19T12:51:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.668+0000 7fb426b72640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.668+0000 7fb426b72640 10 monclient: _add_conns ranks=[3,6,7,5,2,0,8,4,1] 2024-08-19T12:51:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.668+0000 7fb426b72640 10 monclient(hunting): picked mon.noname-g con 0x7fb42006e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.668+0000 7fb426b72640 10 monclient(hunting): picked mon.noname-d con 0x7fb4200a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.668+0000 7fb426b72640 10 monclient(hunting): picked mon.noname-i con 0x7fb4200a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:33.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.668+0000 7fb426b72640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb420136790 con 0x7fb42006e7b0 2024-08-19T12:51:33.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.668+0000 7fb426b72640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb42009ddf0 con 0x7fb4200a5920 2024-08-19T12:51:33.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.668+0000 7fb426b72640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb42009e0d0 con 0x7fb4200a5d50 2024-08-19T12:51:33.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.668+0000 7fb426b72640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:33.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.668+0000 7fb426b72640 10 monclient(hunting): authenticate will time out at 1487.755146s 2024-08-19T12:51:33.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.668+0000 7fb41ffff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fb4200a5920 0x7fb42013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51766/0 (socket says 172.21.15.26:51766) 2024-08-19T12:51:33.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.668+0000 7fb41ffff640 1 -- 172.21.15.26:0/748214743 learned_addr learned my addr 172.21.15.26:0/748214743 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41ffff640 1 -- 172.21.15.26:0/748214743 >> v1:172.21.15.26:6792/0 conn(0x7fb4200a5920 legacy=0x7fb42013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb42510f640 1 -- 172.21.15.26:0/748214743 >> v1:172.21.15.104:6792/0 conn(0x7fb4200a5d50 legacy=0x7fb420141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb42490e640 1 -- 172.21.15.26:0/748214743 >> v1:172.21.15.104:6790/0 conn(0x7fb42006e7b0 legacy=0x7fb420136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3244980129 0 0) 0x7fb4100017d0 con 0x7fb4200a5920 2024-08-19T12:51:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient(hunting): my global_id is 164661 2024-08-19T12:51:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb40c003520 con 0x7fb4200a5920 2024-08-19T12:51:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4022914029 0 0) 0x7fb42009e0d0 con 0x7fb4200a5d50 2024-08-19T12:51:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient(hunting): my global_id is 135907 2024-08-19T12:51:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4100017d0 con 0x7fb4200a5d50 2024-08-19T12:51:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2297456317 0 0) 0x7fb420136790 con 0x7fb42006e7b0 2024-08-19T12:51:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient(hunting): my global_id is 154749 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb42009e0d0 con 0x7fb42006e7b0 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2107460209 0 0) 0x7fb4100017d0 con 0x7fb4200a5d50 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 >> v1:172.21.15.26:6792/0 conn(0x7fb4200a5920 legacy=0x7fb42013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 >> v1:172.21.15.104:6790/0 conn(0x7fb42006e7b0 legacy=0x7fb420136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 1 monclient: found mon.noname-i 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42009e3b0 con 0x7fb4200a5d50 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:33.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb418001bd0 con 0x7fb4200a5d50 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb41f7fe640 10 monclient: dump: 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.678 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb426b72640 5 monclient: authenticate success, global_id 135907 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.670+0000 7fb426b72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb418003610 con 0x7fb4200a5d50 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb41f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb41f7fe640 1 -- 172.21.15.26:0/748214743 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb418003b80 con 0x7fb4200a5d50 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb41f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb41f7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:51:33.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb41f7fe640 10 monclient: dump: 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.680 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb426b72640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb426b72640 4 set_mon_vals no callback set 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb426b72640 10 monclient: shutdown 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb426b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb426b72640 1 -- 172.21.15.26:0/748214743 >> v1:172.21.15.104:6792/0 conn(0x7fb4200a5d50 legacy=0x7fb420141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb426b72640 1 -- 172.21.15.26:0/748214743 shutdown_connections 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb426b72640 1 -- 172.21.15.26:0/748214743 >> 172.21.15.26:0/748214743 conn(0x7fb4200a8740 msgr2=0x7fb4200a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb426b72640 1 -- 172.21.15.26:0/748214743 shutdown_connections 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.671+0000 7fb426b72640 1 -- 172.21.15.26:0/748214743 wait complete. 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.672+0000 7fb426b72640 10 monclient: build_initial_monmap 2024-08-19T12:51:33.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.672+0000 7fb426b72640 10 monclient: monmap: 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:33.682 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.672+0000 7fb426b72640 1 librados: starting msgr at 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.672+0000 7fb426b72640 1 librados: starting objecter 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.672+0000 7fb426b72640 1 Processor -- start 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.672+0000 7fb426b72640 1 -- start start 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.672+0000 7fb426b72640 1 librados: setting wanted keys 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.672+0000 7fb426b72640 1 librados: calling monclient init 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.672+0000 7fb426b72640 10 monclient: init 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.673+0000 7fb426b72640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.673+0000 7fb426b72640 10 monclient: _add_conns ranks=[4,1,6,3,5,7,2,8,0] 2024-08-19T12:51:33.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.673+0000 7fb426b72640 10 monclient(hunting): picked mon.noname-c con 0x7fb42006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.673+0000 7fb426b72640 10 monclient(hunting): picked mon.noname-f con 0x7fb4200a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:51:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.673+0000 7fb426b72640 10 monclient(hunting): picked mon.noname-d con 0x7fb4200a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.673+0000 7fb426b72640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4200a30f0 con 0x7fb4200a5920 2024-08-19T12:51:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.673+0000 7fb426b72640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4200a55e0 con 0x7fb42006e7b0 2024-08-19T12:51:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.673+0000 7fb426b72640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4200a0d90 con 0x7fb4200a5d50 2024-08-19T12:51:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.673+0000 7fb426b72640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.673+0000 7fb426b72640 10 monclient(hunting): authenticate will time out at 1487.760300s 2024-08-19T12:51:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.673+0000 7fb42510f640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fb4200a5d50 0x7fb4201d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51772/0 (socket says 172.21.15.26:51772) 2024-08-19T12:51:33.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.673+0000 7fb42510f640 1 -- 172.21.15.26:0/333162938 learned_addr learned my addr 172.21.15.26:0/333162938 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:33.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb42510f640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.26:6792/0 conn(0x7fb4200a5d50 legacy=0x7fb4201d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 client.?.objecter ms_handle_connect 0x7fb4200a5d50 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 client.?.objecter ms_handle_connect 0x7fb42006e7b0 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb42490e640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.26:6791/0 conn(0x7fb42006e7b0 legacy=0x7fb42009f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 client.?.objecter ms_handle_connect 0x7fb4200a5920 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41ffff640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.104:6789/0 conn(0x7fb4200a5920 legacy=0x7fb42009fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 837654901 0 0) 0x7fb4200a0d90 con 0x7fb4200a5d50 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): my global_id is 164670 2024-08-19T12:51:33.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb400004eb0 con 0x7fb4200a5d50 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4230831763 0 0) 0x7fb4200a30f0 con 0x7fb4200a5920 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): my global_id is 184186 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4200a0d90 con 0x7fb4200a5920 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319066027 0 0) 0x7fb4200a55e0 con 0x7fb42006e7b0 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): my global_id is 184198 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4200a30f0 con 0x7fb42006e7b0 2024-08-19T12:51:33.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1169189185 0 0) 0x7fb400004eb0 con 0x7fb4200a5d50 2024-08-19T12:51:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4200a55e0 con 0x7fb4200a5d50 2024-08-19T12:51:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2324614150 0 0) 0x7fb4200a0d90 con 0x7fb4200a5920 2024-08-19T12:51:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb400004eb0 con 0x7fb4200a5920 2024-08-19T12:51:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb418001b90 con 0x7fb4200a5d50 2024-08-19T12:51:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:51:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): dump: 2024-08-19T12:51:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.693 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.675+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1791345910 0 0) 0x7fb4200a30f0 con 0x7fb42006e7b0 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4200a0d90 con 0x7fb42006e7b0 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb410001e50 con 0x7fb4200a5920 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient(hunting): dump: 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.695 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb408003610 con 0x7fb42006e7b0 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient(hunting): dump: 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.697 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.698 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 992183219 0 0) 0x7fb4200a55e0 con 0x7fb4200a5d50 2024-08-19T12:51:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.26:6791/0 conn(0x7fb42006e7b0 legacy=0x7fb42009f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.104:6789/0 conn(0x7fb4200a5920 legacy=0x7fb42009fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient: _finish_hunting 0 2024-08-19T12:51:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 1 monclient: found mon.d 2024-08-19T12:51:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4200a2850 con 0x7fb4200a5d50 2024-08-19T12:51:33.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient: _finish_auth 0 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient: _check_auth_tickets 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb426b72640 5 monclient: authenticate success, global_id 164670 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb426b72640 10 monclient: _renew_subs 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb426b72640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb426b72640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb4200a18f0 con 0x7fb4200a5d50 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb426b72640 10 client.164670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb426b72640 10 monclient: _renew_subs 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb426b72640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb426b72640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fb4200a03a0 con 0x7fb4200a5d50 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb426b72640 1 librados: init done 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb418003610 con 0x7fb4200a5d50 2024-08-19T12:51:33.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb4180043c0 con 0x7fb4200a5d50 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41effd640 4 set_mon_vals no callback set 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb41d7fa640 10 monclient: dump: 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.700 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.676+0000 7fb426b72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.677+0000 7fb426b72640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb4040065c0 con 0x7fb404002860 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.677+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fb418011a80 con 0x7fb4200a5d50 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.677+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(485..485 src has 350..485) ==== 6214+0+0 (unknown 131736049 0 0) 0x7fb41804e420 con 0x7fb4200a5d50 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.677+0000 7fb41d7fa640 10 client.164670.objecter ms_dispatch 0x7fb420000b90 osd_map(485..485 src has 350..485) 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.677+0000 7fb41d7fa640 3 client.164670.objecter handle_osd_map got epochs [485,485] > 0 2024-08-19T12:51:33.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.677+0000 7fb41d7fa640 3 client.164670.objecter handle_osd_map decoding full epoch 485 2024-08-19T12:51:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.677+0000 7fb41d7fa640 20 client.164670.objecter dump_active .. 0 homeless 2024-08-19T12:51:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.677+0000 7fb41ffff640 10 monclient: get_auth_request con 0x7fb4000418c0 auth_method 0 2024-08-19T12:51:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb42490e640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.26:6789/0 conn(0x7fb404002860 legacy=0x7fb404004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 10 client.164670.objecter ms_handle_connect 0x7fb404002860 2024-08-19T12:51:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 10 client.164670.objecter resend_mon_ops 2024-08-19T12:51:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 559321995 0 0) 0x7fb4040065c0 con 0x7fb404002860 2024-08-19T12:51:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 10 monclient(hunting): my global_id is 145377 2024-08-19T12:51:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4200a55e0 con 0x7fb404002860 2024-08-19T12:51:33.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1976319510 0 0) 0x7fb4200a55e0 con 0x7fb404002860 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4040065c0 con 0x7fb404002860 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb408003e10 con 0x7fb404002860 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 10 monclient: dump: 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.703 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1004252323 0 0) 0x7fb4040065c0 con 0x7fb404002860 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb40400a160 con 0x7fb404002860 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.679+0000 7fb41d7fa640 10 client.164670.objecter ms_handle_connect 0x7fb4000418c0 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.680+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb408004050 con 0x7fb404002860 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.680+0000 7fb41d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:33.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.680+0000 7fb41d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.680+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.26:6789/0 conn(0x7fb404002860 legacy=0x7fb404004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.680+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.26:6789/0 conn(0x7fb404002860 legacy=0x7fb404004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.686+0000 7fb426b72640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.686+0000 7fb426b72640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb404009e50 con 0x7fb404009110 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb42490e640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.26:6789/0 conn(0x7fb404009110 legacy=0x7fb4040095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 10 client.164670.objecter ms_handle_connect 0x7fb404009110 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 10 client.164670.objecter resend_mon_ops 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3912004050 0 0) 0x7fb404009e50 con 0x7fb404009110 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 10 monclient(hunting): my global_id is 145386 2024-08-19T12:51:33.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4040065c0 con 0x7fb404009110 2024-08-19T12:51:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1777237356 0 0) 0x7fb4040065c0 con 0x7fb404009110 2024-08-19T12:51:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb404009e50 con 0x7fb404009110 2024-08-19T12:51:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb408005b00 con 0x7fb404009110 2024-08-19T12:51:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.688+0000 7fb41d7fa640 10 monclient: dump: 2024-08-19T12:51:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.706 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2212527508 0 0) 0x7fb404009e50 con 0x7fb404009110 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb40400c3f0 con 0x7fb404009110 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7880 (unknown 0 0 3236218923) 0x7fb408007ed0 con 0x7fb404009110 2024-08-19T12:51:33.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb41d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb41d7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.26:6789/0 conn(0x7fb404009110 legacy=0x7fb4040095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb41d7fa640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.26:6789/0 conn(0x7fb404009110 legacy=0x7fb4040095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:33.708 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:51:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 545606, 2024-08-19T12:51:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:33.714 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:33.718 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.719 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.722 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:33.723 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:33.725 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:33.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.727 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:33.728 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:51:33.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.728 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:33.728 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:33.728 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:33.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.728 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.728 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:33.728 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 10 librados: watch_flush enter 2024-08-19T12:51:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 10 librados: watch_flush exit 2024-08-19T12:51:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 20 client.164670.objecter shutdown clearing up homeless session... 2024-08-19T12:51:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 10 client.164670.objecter successfully canceled tick 2024-08-19T12:51:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.104:6800/899887506 conn(0x7fb4000418c0 legacy=0x7fb400043d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 10 monclient: shutdown 2024-08-19T12:51:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 1 -- 172.21.15.26:0/333162938 >> v1:172.21.15.26:6792/0 conn(0x7fb4200a5d50 legacy=0x7fb4201d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb42510f640 1 -- 172.21.15.26:0/333162938 reap_dead start 2024-08-19T12:51:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb42510f640 1 -- 172.21.15.26:0/333162938 reap_dead start 2024-08-19T12:51:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 1 -- 172.21.15.26:0/333162938 shutdown_connections 2024-08-19T12:51:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 1 -- 172.21.15.26:0/333162938 >> 172.21.15.26:0/333162938 conn(0x7fb4200a8740 msgr2=0x7fb4200aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 1 -- 172.21.15.26:0/333162938 shutdown_connections 2024-08-19T12:51:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 1 -- 172.21.15.26:0/333162938 wait complete. 2024-08-19T12:51:33.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.689+0000 7fb426b72640 1 librados: shutdown 2024-08-19T12:51:33.731 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:33.732 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:51:33.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:51:33.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.830+0000 7f3a4bfff640 10 monclient: get_monmap_and_config 2024-08-19T12:51:33.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.830+0000 7f3a4bfff640 10 monclient: build_initial_monmap 2024-08-19T12:51:33.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.830+0000 7f3a4bfff640 10 monclient: monmap: 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:33.831758+0000 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:33.831758+0000 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:33.832 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:33.833 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.831+0000 7f3a4bfff640 1 Processor -- start 2024-08-19T12:51:33.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.831+0000 7f3a4bfff640 1 -- start start 2024-08-19T12:51:33.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.831+0000 7f3a4bfff640 10 monclient: init 2024-08-19T12:51:33.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a4bfff640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:33.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a4bfff640 10 monclient: _add_conns ranks=[0,4,5,2,3,8,1,6,7] 2024-08-19T12:51:33.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a4bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f3a4c073e70 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:33.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a4bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f3a4c135dc0 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:33.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a4bfff640 10 monclient(hunting): picked mon.noname-h con 0x7f3a4c13f8d0 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:33.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a4bfff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a4c0a5f70 con 0x7f3a4c073e70 2024-08-19T12:51:33.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a4bfff640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a4c0a7cc0 con 0x7f3a4c135dc0 2024-08-19T12:51:33.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a4bfff640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a4c09da70 con 0x7f3a4c13f8d0 2024-08-19T12:51:33.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a4bfff640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:33.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a4bfff640 10 monclient(hunting): authenticate will time out at 1487.919171s 2024-08-19T12:51:33.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a4affd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3a4c073e70 0x7f3a4c0a5860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60288/0 (socket says 172.21.15.26:60288) 2024-08-19T12:51:33.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a4affd640 1 -- 172.21.15.26:0/2473187639 learned_addr learned my addr 172.21.15.26:0/2473187639 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:33.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.832+0000 7f3a43fff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f3a4c135dc0 0x7f3a4c13e1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41654/0 (socket says 172.21.15.26:41654) 2024-08-19T12:51:33.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a43fff640 1 -- 172.21.15.26:0/2473187639 >> v1:172.21.15.26:6791/0 conn(0x7f3a4c135dc0 legacy=0x7f3a4c13e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4affd640 1 -- 172.21.15.26:0/2473187639 >> v1:172.21.15.26:6789/0 conn(0x7f3a4c073e70 legacy=0x7f3a4c0a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 1 -- 172.21.15.26:0/2473187639 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3017156899 0 0) 0x7f3a4c0a5f70 con 0x7f3a4c073e70 2024-08-19T12:51:33.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient(hunting): my global_id is 145395 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 1 -- 172.21.15.26:0/2473187639 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a38003520 con 0x7f3a4c073e70 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 1 -- 172.21.15.26:0/2473187639 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1177157230 0 0) 0x7f3a4c0a7cc0 con 0x7f3a4c135dc0 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient(hunting): my global_id is 184207 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 1 -- 172.21.15.26:0/2473187639 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a4c0a5f70 con 0x7f3a4c135dc0 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4b7fe640 1 -- 172.21.15.26:0/2473187639 >> v1:172.21.15.104:6791/0 conn(0x7f3a4c13f8d0 legacy=0x7f3a4c141cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 1 -- 172.21.15.26:0/2473187639 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 805470587 0 0) 0x7f3a38003520 con 0x7f3a4c073e70 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 1 -- 172.21.15.26:0/2473187639 >> v1:172.21.15.104:6791/0 conn(0x7f3a4c13f8d0 legacy=0x7f3a4c141cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 1 -- 172.21.15.26:0/2473187639 >> v1:172.21.15.26:6791/0 conn(0x7f3a4c135dc0 legacy=0x7f3a4c13e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 1 monclient: found mon.noname-a 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 1 -- 172.21.15.26:0/2473187639 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c09dd30 con 0x7f3a4c073e70 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 1 -- 172.21.15.26:0/2473187639 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3a3c001ae0 con 0x7f3a4c073e70 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4a7fc640 10 monclient: dump: 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.850 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.851 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4bfff640 5 monclient: authenticate success, global_id 145395 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.834+0000 7f3a4bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4a7fc640 1 -- 172.21.15.26:0/2473187639 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3a3c0048a0 con 0x7f3a4c073e70 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4a7fc640 1 -- 172.21.15.26:0/2473187639 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3a3c004e10 con 0x7f3a4c073e70 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4bfff640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4bfff640 4 set_mon_vals no callback set 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4a7fc640 10 monclient: dump: 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.852 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.853 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.853 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.853 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.853 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.853 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.853 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.853 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.853 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.853 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.853 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.853 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.853 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.854 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4bfff640 10 monclient: shutdown 2024-08-19T12:51:33.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:33.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4bfff640 1 -- 172.21.15.26:0/2473187639 >> v1:172.21.15.26:6789/0 conn(0x7f3a4c073e70 legacy=0x7f3a4c0a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4bfff640 1 -- 172.21.15.26:0/2473187639 shutdown_connections 2024-08-19T12:51:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4bfff640 1 -- 172.21.15.26:0/2473187639 >> 172.21.15.26:0/2473187639 conn(0x7f3a4c0abfb0 msgr2=0x7f3a4c0a8480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4bfff640 1 -- 172.21.15.26:0/2473187639 shutdown_connections 2024-08-19T12:51:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.835+0000 7f3a4bfff640 1 -- 172.21.15.26:0/2473187639 wait complete. 2024-08-19T12:51:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.836+0000 7f3a4bfff640 10 monclient: build_initial_monmap 2024-08-19T12:51:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.836+0000 7f3a4bfff640 10 monclient: monmap: 2024-08-19T12:51:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:33.855 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.836+0000 7f3a4bfff640 1 librados: starting msgr at 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.836+0000 7f3a4bfff640 1 librados: starting objecter 2024-08-19T12:51:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.836+0000 7f3a4bfff640 1 Processor -- start 2024-08-19T12:51:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.836+0000 7f3a4bfff640 1 -- start start 2024-08-19T12:51:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.836+0000 7f3a4bfff640 1 librados: setting wanted keys 2024-08-19T12:51:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.836+0000 7f3a4bfff640 1 librados: calling monclient init 2024-08-19T12:51:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.836+0000 7f3a4bfff640 10 monclient: init 2024-08-19T12:51:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.837+0000 7f3a4bfff640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.837+0000 7f3a4bfff640 10 monclient: _add_conns ranks=[8,2,7,3,1,0,4,6,5] 2024-08-19T12:51:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.837+0000 7f3a4bfff640 10 monclient(hunting): picked mon.noname-e con 0x7f3a4c073e70 addr v1:172.21.15.26:6793/0 2024-08-19T12:51:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.837+0000 7f3a4bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f3a4c135dc0 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.837+0000 7f3a4bfff640 10 monclient(hunting): picked mon.noname-i con 0x7f3a4c13f8d0 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.837+0000 7f3a4bfff640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a4c0a5610 con 0x7f3a4c135dc0 2024-08-19T12:51:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.837+0000 7f3a4bfff640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a4c073b90 con 0x7f3a4c13f8d0 2024-08-19T12:51:33.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.837+0000 7f3a4bfff640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a4c0a5040 con 0x7f3a4c073e70 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.837+0000 7f3a4bfff640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.837+0000 7f3a4bfff640 10 monclient(hunting): authenticate will time out at 1487.924271s 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.837+0000 7f3a4affd640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f3a4c073e70 0x7f3a4c0a2130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:45582/0 (socket says 172.21.15.26:45582) 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.837+0000 7f3a4affd640 1 -- 172.21.15.26:0/605024726 learned_addr learned my addr 172.21.15.26:0/605024726 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a4affd640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.26:6793/0 conn(0x7f3a4c073e70 legacy=0x7f3a4c0a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 client.?.objecter ms_handle_connect 0x7f3a4c073e70 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 client.?.objecter ms_handle_connect 0x7f3a4c135dc0 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a43fff640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.26:6790/0 conn(0x7f3a4c135dc0 legacy=0x7f3a4c0a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 client.?.objecter ms_handle_connect 0x7f3a4c13f8d0 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a4b7fe640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.104:6792/0 conn(0x7f3a4c13f8d0 legacy=0x7f3a4c1d62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3907173772 0 0) 0x7f3a4c0a5040 con 0x7f3a4c073e70 2024-08-19T12:51:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): my global_id is 184157 2024-08-19T12:51:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a2c004eb0 con 0x7f3a4c073e70 2024-08-19T12:51:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145470541 0 0) 0x7f3a4c0a5610 con 0x7f3a4c135dc0 2024-08-19T12:51:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): my global_id is 26930 2024-08-19T12:51:33.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a4c0a5040 con 0x7f3a4c135dc0 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 137551097 0 0) 0x7f3a4c073b90 con 0x7f3a4c13f8d0 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): my global_id is 135916 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a4c0a5610 con 0x7f3a4c13f8d0 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 978022289 0 0) 0x7f3a4c0a5040 con 0x7f3a4c135dc0 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a4c073b90 con 0x7f3a4c135dc0 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3a34001320 con 0x7f3a4c135dc0 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): dump: 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.860 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:33.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.839+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1936456073 0 0) 0x7f3a2c004eb0 con 0x7f3a4c073e70 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a4c0a5040 con 0x7f3a4c073e70 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3a3c004d90 con 0x7f3a4c073e70 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 10 monclient(hunting): dump: 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.862 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 284238885 0 0) 0x7f3a4c073b90 con 0x7f3a4c135dc0 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.26:6793/0 conn(0x7f3a4c073e70 legacy=0x7f3a4c0a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.104:6792/0 conn(0x7f3a4c13f8d0 legacy=0x7f3a4c1d62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 10 monclient: _finish_hunting 0 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 1 monclient: found mon.b 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:51:33.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c0a3b50 con 0x7f3a4c135dc0 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 10 monclient: _finish_auth 0 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 10 monclient: _check_auth_tickets 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a4bfff640 5 monclient: authenticate success, global_id 26930 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a4bfff640 10 monclient: _renew_subs 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a4bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a4bfff640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4c137060 con 0x7f3a4c135dc0 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a4bfff640 10 client.26930.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a4bfff640 10 monclient: _renew_subs 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a4bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a4bfff640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3a4c0a3130 con 0x7f3a4c135dc0 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a4bfff640 1 librados: init done 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3a34004b10 con 0x7f3a4c135dc0 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:33.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3a34005000 con 0x7f3a4c135dc0 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a49ffb640 4 set_mon_vals no callback set 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a437fe640 10 monclient: dump: 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.865 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a4bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.840+0000 7f3a4bfff640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3a300065c0 con 0x7f3a30002860 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.841+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3a340051a0 con 0x7f3a4c135dc0 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.841+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(485..485 src has 350..485) ==== 6214+0+0 (unknown 131736049 0 0) 0x7f3a3404e470 con 0x7f3a4c135dc0 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.841+0000 7f3a437fe640 10 client.26930.objecter ms_dispatch 0x7f3a4c000b90 osd_map(485..485 src has 350..485) 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.841+0000 7f3a437fe640 3 client.26930.objecter handle_osd_map got epochs [485,485] > 0 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.841+0000 7f3a437fe640 3 client.26930.objecter handle_osd_map decoding full epoch 485 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.841+0000 7f3a437fe640 20 client.26930.objecter dump_active .. 0 homeless 2024-08-19T12:51:33.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.841+0000 7f3a4b7fe640 10 monclient: get_auth_request con 0x7f3a2c0416b0 auth_method 0 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 10 client.26930.objecter ms_handle_connect 0x7f3a30002860 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 10 client.26930.objecter resend_mon_ops 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a4affd640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.26:6790/0 conn(0x7f3a30002860 legacy=0x7f3a30004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1606983669 0 0) 0x7f3a300065c0 con 0x7f3a30002860 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 10 monclient(hunting): my global_id is 26939 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a4c073b90 con 0x7f3a30002860 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3071688620 0 0) 0x7f3a4c073b90 con 0x7f3a30002860 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a300065c0 con 0x7f3a30002860 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3a3c001d30 con 0x7f3a30002860 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:33.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 10 monclient: dump: 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.869 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4096216068 0 0) 0x7f3a300065c0 con 0x7f3a30002860 2024-08-19T12:51:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a3000a160 con 0x7f3a30002860 2024-08-19T12:51:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 10 client.26930.objecter ms_handle_connect 0x7f3a2c0416b0 2024-08-19T12:51:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3a3c005520 con 0x7f3a30002860 2024-08-19T12:51:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.26:6790/0 conn(0x7f3a30002860 legacy=0x7f3a30004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.843+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.26:6790/0 conn(0x7f3a30002860 legacy=0x7f3a30004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.850+0000 7f3a4bfff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.850+0000 7f3a4bfff640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3a30009e70 con 0x7f3a30009130 2024-08-19T12:51:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a4affd640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.26:6790/0 conn(0x7f3a30009130 legacy=0x7f3a300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 10 client.26930.objecter ms_handle_connect 0x7f3a30009130 2024-08-19T12:51:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 10 client.26930.objecter resend_mon_ops 2024-08-19T12:51:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1497955321 0 0) 0x7f3a30009e70 con 0x7f3a30009130 2024-08-19T12:51:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:33.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 10 monclient(hunting): my global_id is 26948 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a300065c0 con 0x7f3a30009130 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1003720530 0 0) 0x7f3a300065c0 con 0x7f3a30009130 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a30009e70 con 0x7f3a30009130 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3a3c005d40 con 0x7f3a30009130 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 10 monclient: dump: 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.872 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2888116877 0 0) 0x7f3a30009e70 con 0x7f3a30009130 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.852+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3a3000c3f0 con 0x7f3a30009130 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1345314634) 0x7f3a3c006150 con 0x7f3a30009130 2024-08-19T12:51:33.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a437fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:33.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a437fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:33.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.26:6790/0 conn(0x7f3a30009130 legacy=0x7f3a300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a437fe640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.26:6790/0 conn(0x7f3a30009130 legacy=0x7f3a300095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:33.874 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 806907, 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:51:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:33.893 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:33.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 10 librados: watch_flush enter 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 10 librados: watch_flush exit 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 20 client.26930.objecter shutdown clearing up homeless session... 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 10 client.26930.objecter successfully canceled tick 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.104:6800/899887506 conn(0x7f3a2c0416b0 legacy=0x7f3a2c043b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 10 monclient: shutdown 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 1 -- 172.21.15.26:0/605024726 >> v1:172.21.15.26:6790/0 conn(0x7f3a4c135dc0 legacy=0x7f3a4c0a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4b7fe640 1 -- 172.21.15.26:0/605024726 reap_dead start 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4b7fe640 1 -- 172.21.15.26:0/605024726 reap_dead start 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 1 -- 172.21.15.26:0/605024726 shutdown_connections 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 1 -- 172.21.15.26:0/605024726 >> 172.21.15.26:0/605024726 conn(0x7f3a4c0abfb0 msgr2=0x7f3a4c134fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 1 -- 172.21.15.26:0/605024726 shutdown_connections 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 1 -- 172.21.15.26:0/605024726 wait complete. 2024-08-19T12:51:33.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.853+0000 7f3a4bfff640 1 librados: shutdown 2024-08-19T12:51:33.897 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:33.897 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:51:33.897 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:51:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.996+0000 7f0e3cf11640 10 monclient: get_monmap_and_config 2024-08-19T12:51:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.996+0000 7f0e3cf11640 10 monclient: build_initial_monmap 2024-08-19T12:51:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.996+0000 7f0e3cf11640 10 monclient: monmap: 2024-08-19T12:51:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:33.996987+0000 2024-08-19T12:51:33.997 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:33.996987+0000 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.996+0000 7f0e3cf11640 1 Processor -- start 2024-08-19T12:51:33.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.996+0000 7f0e3cf11640 1 -- start start 2024-08-19T12:51:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.996+0000 7f0e3cf11640 10 monclient: init 2024-08-19T12:51:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e3cf11640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e3cf11640 10 monclient: _add_conns ranks=[4,3,2,1,7,5,8,0,6] 2024-08-19T12:51:34.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e3cf11640 10 monclient(hunting): picked mon.noname-c con 0x7f0e3806e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e3cf11640 10 monclient(hunting): picked mon.noname-g con 0x7f0e380a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e3cf11640 10 monclient(hunting): picked mon.noname-b con 0x7f0e380a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e3cf11640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e38136790 con 0x7f0e380a5d50 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e3cf11640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e3809ddf0 con 0x7f0e380a5920 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e3cf11640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e3809e0d0 con 0x7f0e3806e7b0 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e3cf11640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e3cf11640 10 monclient(hunting): authenticate will time out at 1488.084327s 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e3659c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f0e3806e7b0 0x7f0e38136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41658/0 (socket says 172.21.15.26:41658) 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e3659c640 1 -- 172.21.15.26:0/540605627 learned_addr learned my addr 172.21.15.26:0/540605627 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.997+0000 7f0e36d9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0e380a5d50 0x7f0e38141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54608/0 (socket says 172.21.15.26:54608) 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e36d9d640 1 -- 172.21.15.26:0/540605627 >> v1:172.21.15.26:6790/0 conn(0x7f0e380a5d50 legacy=0x7f0e38141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e3659c640 1 -- 172.21.15.26:0/540605627 >> v1:172.21.15.26:6791/0 conn(0x7f0e3806e7b0 legacy=0x7f0e38136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e2dd9b640 1 -- 172.21.15.26:0/540605627 >> v1:172.21.15.104:6790/0 conn(0x7f0e380a5920 legacy=0x7f0e3813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1371922660 0 0) 0x7f0e38136790 con 0x7f0e380a5d50 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 10 monclient(hunting): my global_id is 26957 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e24003520 con 0x7f0e380a5d50 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2178795960 0 0) 0x7f0e3809e0d0 con 0x7f0e3806e7b0 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 10 monclient(hunting): my global_id is 184216 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e38136790 con 0x7f0e3806e7b0 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1586876414 0 0) 0x7f0e200014a0 con 0x7f0e380a5920 2024-08-19T12:51:34.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 10 monclient(hunting): my global_id is 154758 2024-08-19T12:51:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:33.999+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e3809e0d0 con 0x7f0e380a5920 2024-08-19T12:51:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1599962633 0 0) 0x7f0e24003520 con 0x7f0e380a5d50 2024-08-19T12:51:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 >> v1:172.21.15.26:6791/0 conn(0x7f0e3806e7b0 legacy=0x7f0e38136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 >> v1:172.21.15.104:6790/0 conn(0x7f0e380a5920 legacy=0x7f0e3813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:51:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 1 monclient: found mon.noname-b 2024-08-19T12:51:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:51:34.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3809e3b0 con 0x7f0e380a5d50 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 10 monclient: _finish_auth 0 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e30001e60 con 0x7f0e380a5d50 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e3cf11640 5 monclient: authenticate success, global_id 26957 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e3cf11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 10 monclient: dump: 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.007 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e3cf11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0e30004b60 con 0x7f0e380a5d50 2024-08-19T12:51:34.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e3cf11640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 1 -- 172.21.15.26:0/540605627 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e300050d0 con 0x7f0e380a5d50 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e3cf11640 4 set_mon_vals no callback set 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e35d9b640 10 monclient: dump: 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.009 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e3cf11640 10 monclient: shutdown 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e3cf11640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e3cf11640 1 -- 172.21.15.26:0/540605627 >> v1:172.21.15.26:6790/0 conn(0x7f0e380a5d50 legacy=0x7f0e38141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e3cf11640 1 -- 172.21.15.26:0/540605627 shutdown_connections 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e3cf11640 1 -- 172.21.15.26:0/540605627 >> 172.21.15.26:0/540605627 conn(0x7f0e380a8740 msgr2=0x7f0e380a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e3cf11640 1 -- 172.21.15.26:0/540605627 shutdown_connections 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.000+0000 7f0e3cf11640 1 -- 172.21.15.26:0/540605627 wait complete. 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.001+0000 7f0e3cf11640 10 monclient: build_initial_monmap 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.001+0000 7f0e3cf11640 10 monclient: monmap: 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.001+0000 7f0e3cf11640 1 librados: starting msgr at 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.001+0000 7f0e3cf11640 1 librados: starting objecter 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.001+0000 7f0e3cf11640 1 Processor -- start 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 1 -- start start 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 1 librados: setting wanted keys 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 1 librados: calling monclient init 2024-08-19T12:51:34.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 10 monclient: init 2024-08-19T12:51:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 10 monclient: _add_conns ranks=[0,6,1,2,4,3,7,5,8] 2024-08-19T12:51:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 10 monclient(hunting): picked mon.noname-a con 0x7f0e3806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 10 monclient(hunting): picked mon.noname-d con 0x7f0e380a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 10 monclient(hunting): picked mon.noname-f con 0x7f0e380a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:51:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e3813b360 con 0x7f0e3806e7b0 2024-08-19T12:51:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e38139000 con 0x7f0e380a5d50 2024-08-19T12:51:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e3813aac0 con 0x7f0e380a5920 2024-08-19T12:51:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3cf11640 10 monclient(hunting): authenticate will time out at 1488.089492s 2024-08-19T12:51:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0e3806e7b0 0x7f0e381ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60294/0 (socket says 172.21.15.26:60294) 2024-08-19T12:51:34.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.002+0000 7f0e3659c640 1 -- 172.21.15.26:0/2987812136 learned_addr learned my addr 172.21.15.26:0/2987812136 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e3659c640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.26:6789/0 conn(0x7f0e3806e7b0 legacy=0x7f0e381ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 client.?.objecter ms_handle_connect 0x7f0e3806e7b0 2024-08-19T12:51:34.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2dd9b640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.26:6792/0 conn(0x7f0e380a5920 legacy=0x7f0e381d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 client.?.objecter ms_handle_connect 0x7f0e380a5920 2024-08-19T12:51:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 client.?.objecter ms_handle_connect 0x7f0e380a5d50 2024-08-19T12:51:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e36d9d640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.104:6789/0 conn(0x7f0e380a5d50 legacy=0x7f0e381d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3658465219 0 0) 0x7f0e3813b360 con 0x7f0e3806e7b0 2024-08-19T12:51:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 monclient(hunting): my global_id is 145404 2024-08-19T12:51:34.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e18004eb0 con 0x7f0e3806e7b0 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2240486220 0 0) 0x7f0e3813aac0 con 0x7f0e380a5920 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 10 monclient(hunting): my global_id is 164679 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.004+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e3813b360 con 0x7f0e380a5920 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1440769567 0 0) 0x7f0e38139000 con 0x7f0e380a5d50 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): my global_id is 184195 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e3813aac0 con 0x7f0e380a5d50 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3317941190 0 0) 0x7f0e18004eb0 con 0x7f0e3806e7b0 2024-08-19T12:51:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e38139000 con 0x7f0e3806e7b0 2024-08-19T12:51:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e280047e0 con 0x7f0e3806e7b0 2024-08-19T12:51:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): dump: 2024-08-19T12:51:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3947304185 0 0) 0x7f0e3813b360 con 0x7f0e380a5920 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e18004eb0 con 0x7f0e380a5920 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1397562878 0 0) 0x7f0e3813aac0 con 0x7f0e380a5d50 2024-08-19T12:51:34.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e3813b360 con 0x7f0e380a5d50 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e200014c0 con 0x7f0e380a5920 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): dump: 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.024 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.025 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e300020b0 con 0x7f0e380a5d50 2024-08-19T12:51:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:51:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient(hunting): dump: 2024-08-19T12:51:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.025 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.026 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 117868466 0 0) 0x7f0e38139000 con 0x7f0e3806e7b0 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.26:6792/0 conn(0x7f0e380a5920 legacy=0x7f0e381d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.104:6789/0 conn(0x7f0e380a5d50 legacy=0x7f0e381d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 monclient: found mon.a 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3813ada0 con 0x7f0e3806e7b0 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e3cf11640 5 monclient: authenticate success, global_id 145404 2024-08-19T12:51:34.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e3cf11640 10 monclient: _renew_subs 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e3cf11640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e3cf11640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0e3813a790 con 0x7f0e3806e7b0 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e3cf11640 10 client.145404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e3cf11640 10 monclient: _renew_subs 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e3cf11640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e3cf11640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0e381385f0 con 0x7f0e3806e7b0 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e3cf11640 1 librados: init done 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0e28004c30 con 0x7f0e3806e7b0 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e28005160 con 0x7f0e3806e7b0 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:34.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e3559a640 4 set_mon_vals no callback set 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.005+0000 7f0e2f7fe640 10 monclient: dump: 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.029 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.030 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.006+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f0e28005300 con 0x7f0e3806e7b0 2024-08-19T12:51:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.006+0000 7f0e3cf11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.006+0000 7f0e3cf11640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0e1c0065c0 con 0x7f0e1c002860 2024-08-19T12:51:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.006+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(485..485 src has 350..485) ==== 6214+0+0 (unknown 131736049 0 0) 0x7f0e2804e580 con 0x7f0e3806e7b0 2024-08-19T12:51:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.006+0000 7f0e2f7fe640 10 client.145404.objecter ms_dispatch 0x7f0e38000b90 osd_map(485..485 src has 350..485) 2024-08-19T12:51:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.006+0000 7f0e2f7fe640 3 client.145404.objecter handle_osd_map got epochs [485,485] > 0 2024-08-19T12:51:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.006+0000 7f0e2f7fe640 3 client.145404.objecter handle_osd_map decoding full epoch 485 2024-08-19T12:51:34.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.006+0000 7f0e36d9d640 10 monclient: get_auth_request con 0x7f0e18041af0 auth_method 0 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.006+0000 7f0e2f7fe640 20 client.145404.objecter dump_active .. 0 homeless 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.008+0000 7f0e2dd9b640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.26:6791/0 conn(0x7f0e1c002860 legacy=0x7f0e1c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.008+0000 7f0e2f7fe640 10 client.145404.objecter ms_handle_connect 0x7f0e1c002860 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.008+0000 7f0e2f7fe640 10 client.145404.objecter resend_mon_ops 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.008+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213842180 0 0) 0x7f0e1c0065c0 con 0x7f0e1c002860 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.008+0000 7f0e2f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.008+0000 7f0e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.008+0000 7f0e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.008+0000 7f0e2f7fe640 10 monclient(hunting): my global_id is 184225 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.008+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e38139000 con 0x7f0e1c002860 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2797497312 0 0) 0x7f0e38139000 con 0x7f0e1c002860 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e1c0065c0 con 0x7f0e1c002860 2024-08-19T12:51:34.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 10 client.145404.objecter ms_handle_connect 0x7f0e18041af0 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e20003cf0 con 0x7f0e1c002860 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 10 monclient: dump: 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.032 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 60916258 0 0) 0x7f0e1c0065c0 con 0x7f0e1c002860 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e1c00a160 con 0x7f0e1c002860 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0e20004010 con 0x7f0e1c002860 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.26:6791/0 conn(0x7f0e1c002860 legacy=0x7f0e1c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.009+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.26:6791/0 conn(0x7f0e1c002860 legacy=0x7f0e1c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:34.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.015+0000 7f0e3cf11640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.015+0000 7f0e3cf11640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0e1c009e20 con 0x7f0e1c0090e0 2024-08-19T12:51:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2dd9b640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.26:6791/0 conn(0x7f0e1c0090e0 legacy=0x7f0e1c009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 10 client.145404.objecter ms_handle_connect 0x7f0e1c0090e0 2024-08-19T12:51:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 10 client.145404.objecter resend_mon_ops 2024-08-19T12:51:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2682633128 0 0) 0x7f0e1c009e20 con 0x7f0e1c0090e0 2024-08-19T12:51:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 10 monclient(hunting): my global_id is 184234 2024-08-19T12:51:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e1c0065c0 con 0x7f0e1c0090e0 2024-08-19T12:51:34.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1234109702 0 0) 0x7f0e1c0065c0 con 0x7f0e1c0090e0 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e1c009e20 con 0x7f0e1c0090e0 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0e20005b00 con 0x7f0e1c0090e0 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 10 monclient: dump: 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.035 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2100432438 0 0) 0x7f0e1c009e20 con 0x7f0e1c0090e0 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.018+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0e1c00c3f0 con 0x7f0e1c0090e0 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7710 (unknown 0 0 3524428182) 0x7f0e20006fd0 con 0x7f0e1c0090e0 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e2f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e2f7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:34.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.26:6791/0 conn(0x7f0e1c0090e0 legacy=0x7f0e1c009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e2f7fe640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.26:6791/0 conn(0x7f0e1c0090e0 legacy=0x7f0e1c009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:34.037 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:34.037 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:34.037 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:34.037 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:34.037 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:34.037 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 19483, 2024-08-19T12:51:34.037 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:34.037 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:34.038 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:34.039 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.040 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:34.041 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.042 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.046 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:34.046 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:34.046 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.046 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.046 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.046 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:34.046 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.046 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.046 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.047 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:34.049 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.050 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:34.051 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:34.053 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:34.054 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:34.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.056 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:34.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.057 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.057 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.057 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:34.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.057 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.057 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:34.057 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 10 librados: watch_flush enter 2024-08-19T12:51:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 10 librados: watch_flush exit 2024-08-19T12:51:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 20 client.145404.objecter shutdown clearing up homeless session... 2024-08-19T12:51:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 10 client.145404.objecter successfully canceled tick 2024-08-19T12:51:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.104:6800/899887506 conn(0x7f0e18041af0 legacy=0x7f0e18043fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 10 monclient: shutdown 2024-08-19T12:51:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 1 -- 172.21.15.26:0/2987812136 >> v1:172.21.15.26:6789/0 conn(0x7f0e3806e7b0 legacy=0x7f0e381ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e36d9d640 1 -- 172.21.15.26:0/2987812136 reap_dead start 2024-08-19T12:51:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e36d9d640 1 -- 172.21.15.26:0/2987812136 reap_dead start 2024-08-19T12:51:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 1 -- 172.21.15.26:0/2987812136 shutdown_connections 2024-08-19T12:51:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 1 -- 172.21.15.26:0/2987812136 >> 172.21.15.26:0/2987812136 conn(0x7f0e380a8740 msgr2=0x7f0e380aa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 1 -- 172.21.15.26:0/2987812136 shutdown_connections 2024-08-19T12:51:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 1 -- 172.21.15.26:0/2987812136 wait complete. 2024-08-19T12:51:34.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.019+0000 7f0e3cf11640 1 librados: shutdown 2024-08-19T12:51:34.060 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:34.060 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:51:34.060 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:51:34.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307599501}) 2024-08-19T12:51:34.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4057050633}) 2024-08-19T12:51:34.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307599501}) 2024-08-19T12:51:34.090 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1857073184}) 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307599501}) 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1672022990}) 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307599501}) 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3139677653}) 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.083+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307599501}) 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=979833193}) 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307599501}) 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=802007644}) 2024-08-19T12:51:34.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307599501}) 2024-08-19T12:51:34.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3217267178}) 2024-08-19T12:51:34.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307599501}) 2024-08-19T12:51:34.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.084+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3340693998}) 2024-08-19T12:51:34.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297360614}) 2024-08-19T12:51:34.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4013419841}) 2024-08-19T12:51:34.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297360614}) 2024-08-19T12:51:34.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3089430013}) 2024-08-19T12:51:34.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297360614}) 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=261763519}) 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297360614}) 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3692833662}) 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297360614}) 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=378504583}) 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297360614}) 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=706619762}) 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297360614}) 2024-08-19T12:51:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=729727807}) 2024-08-19T12:51:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297360614}) 2024-08-19T12:51:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.085+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1855148335}) 2024-08-19T12:51:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471395738}) 2024-08-19T12:51:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2047670548}) 2024-08-19T12:51:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471395738}) 2024-08-19T12:51:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3036153495}) 2024-08-19T12:51:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.094 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.086+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471395738}) 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3050165061}) 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471395738}) 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=570665824}) 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471395738}) 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=621367952}) 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471395738}) 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3126787363}) 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471395738}) 2024-08-19T12:51:34.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2677845668}) 2024-08-19T12:51:34.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471395738}) 2024-08-19T12:51:34.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.087+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4085613848}) 2024-08-19T12:51:34.133 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.133 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606652209}) 2024-08-19T12:51:34.133 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3068956177}) 2024-08-19T12:51:34.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606652209}) 2024-08-19T12:51:34.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=557911007}) 2024-08-19T12:51:34.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606652209}) 2024-08-19T12:51:34.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1708826954}) 2024-08-19T12:51:34.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606652209}) 2024-08-19T12:51:34.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=59008919}) 2024-08-19T12:51:34.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606652209}) 2024-08-19T12:51:34.134 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1331414253}) 2024-08-19T12:51:34.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606652209}) 2024-08-19T12:51:34.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=474225761}) 2024-08-19T12:51:34.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606652209}) 2024-08-19T12:51:34.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=672977369}) 2024-08-19T12:51:34.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606652209}) 2024-08-19T12:51:34.135 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.131+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1787916257}) 2024-08-19T12:51:34.169 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.169 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959636811}) 2024-08-19T12:51:34.169 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=296697961}) 2024-08-19T12:51:34.169 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.170 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959636811}) 2024-08-19T12:51:34.170 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785799333}) 2024-08-19T12:51:34.170 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.170 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959636811}) 2024-08-19T12:51:34.170 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3160080173}) 2024-08-19T12:51:34.170 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.170 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959636811}) 2024-08-19T12:51:34.170 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2130087655}) 2024-08-19T12:51:34.170 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.170 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959636811}) 2024-08-19T12:51:34.170 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3424512243}) 2024-08-19T12:51:34.170 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.171 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959636811}) 2024-08-19T12:51:34.171 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1133908943}) 2024-08-19T12:51:34.171 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.171 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959636811}) 2024-08-19T12:51:34.171 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2235447791}) 2024-08-19T12:51:34.171 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.171 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959636811}) 2024-08-19T12:51:34.171 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.167+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2035737925}) 2024-08-19T12:51:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.172+0000 7f76b7d14640 10 monclient: get_monmap_and_config 2024-08-19T12:51:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.172+0000 7f76b7d14640 10 monclient: build_initial_monmap 2024-08-19T12:51:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.172+0000 7f76b7d14640 10 monclient: monmap: 2024-08-19T12:51:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:34.173427+0000 2024-08-19T12:51:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:34.173427+0000 2024-08-19T12:51:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.173 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:34.174 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:34.174 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:34.174 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:34.174 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:34.174 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:34.174 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:34.174 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:34.174 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:34.174 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.173+0000 7f76b7d14640 1 Processor -- start 2024-08-19T12:51:34.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.173+0000 7f76b7d14640 1 -- start start 2024-08-19T12:51:34.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.173+0000 7f76b7d14640 10 monclient: init 2024-08-19T12:51:34.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.173+0000 7f76b7d14640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.173+0000 7f76b7d14640 10 monclient: _add_conns ranks=[7,6,4,3,5,8,1,0,2] 2024-08-19T12:51:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.173+0000 7f76b7d14640 10 monclient(hunting): picked mon.noname-i con 0x7f76b006e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.173+0000 7f76b7d14640 10 monclient(hunting): picked mon.noname-d con 0x7f76b00a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.174+0000 7f76b7d14640 10 monclient(hunting): picked mon.noname-c con 0x7f76b00a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.174+0000 7f76b7d14640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76b0136790 con 0x7f76b00a5d50 2024-08-19T12:51:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.174+0000 7f76b7d14640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76b009ddf0 con 0x7f76b00a5920 2024-08-19T12:51:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.174+0000 7f76b7d14640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76b009e0d0 con 0x7f76b006e7b0 2024-08-19T12:51:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.174+0000 7f76b7d14640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.174+0000 7f76b7d14640 10 monclient(hunting): authenticate will time out at 1488.261063s 2024-08-19T12:51:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.174+0000 7f76b5ab0640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f76b006e7b0 0x7f76b0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:44200/0 (socket says 172.21.15.26:44200) 2024-08-19T12:51:34.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.174+0000 7f76b5ab0640 1 -- 172.21.15.26:0/6041357 learned_addr learned my addr 172.21.15.26:0/6041357 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:34.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b62b1640 1 -- 172.21.15.26:0/6041357 >> v1:172.21.15.26:6791/0 conn(0x7f76b00a5d50 legacy=0x7f76b0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b52af640 1 -- 172.21.15.26:0/6041357 >> v1:172.21.15.26:6792/0 conn(0x7f76b00a5920 legacy=0x7f76b013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b5ab0640 1 -- 172.21.15.26:0/6041357 >> v1:172.21.15.104:6792/0 conn(0x7f76b006e7b0 legacy=0x7f76b0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 254992211 0 0) 0x7f76b0136790 con 0x7f76b00a5d50 2024-08-19T12:51:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient(hunting): my global_id is 184243 2024-08-19T12:51:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f769c003520 con 0x7f76b00a5d50 2024-08-19T12:51:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2470549098 0 0) 0x7f76b009e0d0 con 0x7f76b006e7b0 2024-08-19T12:51:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient(hunting): my global_id is 135925 2024-08-19T12:51:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76b0136790 con 0x7f76b006e7b0 2024-08-19T12:51:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 761595610 0 0) 0x7f76a00013d0 con 0x7f76b00a5920 2024-08-19T12:51:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient(hunting): my global_id is 164688 2024-08-19T12:51:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76b009e0d0 con 0x7f76b00a5920 2024-08-19T12:51:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1749690620 0 0) 0x7f769c003520 con 0x7f76b00a5d50 2024-08-19T12:51:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b5ab0640 1 --1- 172.21.15.26:0/6041357 >> v1:172.21.15.104:6792/0 conn(0x7f76b006e7b0 0x7f76b0136080 :-1 s=OPENED pgs=629 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f76ac001a40 mon_map magic: 0 2024-08-19T12:51:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 >> v1:172.21.15.104:6792/0 conn(0x7f76b006e7b0 legacy=0x7f76b0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 >> v1:172.21.15.26:6792/0 conn(0x7f76b00a5920 legacy=0x7f76b013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient: _finish_hunting 0 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 1 monclient: found mon.noname-c 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b009e3b0 con 0x7f76b00a5d50 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient: _finish_auth 0 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient: _check_auth_tickets 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f76a8001de0 con 0x7f76b00a5d50 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b4aae640 10 monclient: dump: 2024-08-19T12:51:34.183 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.184 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b7d14640 5 monclient: authenticate success, global_id 184243 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.176+0000 7f76b7d14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f76a80020b0 con 0x7f76b00a5d50 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b4aae640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b4aae640 1 -- 172.21.15.26:0/6041357 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f76a8003cd0 con 0x7f76b00a5d50 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b7d14640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b7d14640 4 set_mon_vals no callback set 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b4aae640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b4aae640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b4aae640 10 monclient: dump: 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.185 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.186 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b7d14640 10 monclient: shutdown 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b7d14640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b7d14640 1 -- 172.21.15.26:0/6041357 >> v1:172.21.15.26:6791/0 conn(0x7f76b00a5d50 legacy=0x7f76b0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b7d14640 1 -- 172.21.15.26:0/6041357 shutdown_connections 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b7d14640 1 -- 172.21.15.26:0/6041357 >> 172.21.15.26:0/6041357 conn(0x7f76b00a8740 msgr2=0x7f76b00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b7d14640 1 -- 172.21.15.26:0/6041357 shutdown_connections 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.177+0000 7f76b7d14640 1 -- 172.21.15.26:0/6041357 wait complete. 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.178+0000 7f76b7d14640 10 monclient: build_initial_monmap 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.178+0000 7f76b7d14640 10 monclient: monmap: 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:34.187 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:34.188 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.178+0000 7f76b7d14640 1 librados: starting msgr at 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.178+0000 7f76b7d14640 1 librados: starting objecter 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.178+0000 7f76b7d14640 1 Processor -- start 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.178+0000 7f76b7d14640 1 -- start start 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.178+0000 7f76b7d14640 1 librados: setting wanted keys 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.178+0000 7f76b7d14640 1 librados: calling monclient init 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.178+0000 7f76b7d14640 10 monclient: init 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b7d14640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b7d14640 10 monclient: _add_conns ranks=[2,7,4,3,6,5,8,0,1] 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b7d14640 10 monclient(hunting): picked mon.noname-b con 0x7f76b006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b7d14640 10 monclient(hunting): picked mon.noname-i con 0x7f76b00a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b7d14640 10 monclient(hunting): picked mon.noname-c con 0x7f76b00a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b7d14640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76b013b360 con 0x7f76b006e7b0 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b7d14640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76b0139000 con 0x7f76b00a5d50 2024-08-19T12:51:34.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b7d14640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76b013aac0 con 0x7f76b00a5920 2024-08-19T12:51:34.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b7d14640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:34.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b7d14640 10 monclient(hunting): authenticate will time out at 1488.266342s 2024-08-19T12:51:34.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b5ab0640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f76b006e7b0 0x7f76b01ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54610/0 (socket says 172.21.15.26:54610) 2024-08-19T12:51:34.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b62b1640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f76b00a5d50 0x7f76b01d6bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41690/0 (socket says 172.21.15.26:41690) 2024-08-19T12:51:34.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.179+0000 7f76b5ab0640 1 -- 172.21.15.26:0/2176638320 learned_addr learned my addr 172.21.15.26:0/2176638320 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:34.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.181+0000 7f76b62b1640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.26:6791/0 conn(0x7f76b00a5d50 legacy=0x7f76b01d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.181+0000 7f76a67fc640 10 client.?.objecter ms_handle_connect 0x7f76b00a5d50 2024-08-19T12:51:34.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.181+0000 7f76b5ab0640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.26:6790/0 conn(0x7f76b006e7b0 legacy=0x7f76b01ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.181+0000 7f76a67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.181+0000 7f76a67fc640 10 client.?.objecter ms_handle_connect 0x7f76b006e7b0 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.181+0000 7f76a67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.181+0000 7f76a67fc640 10 client.?.objecter ms_handle_connect 0x7f76b00a5920 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.181+0000 7f76a67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.181+0000 7f76b52af640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.104:6792/0 conn(0x7f76b00a5920 legacy=0x7f76b01d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3644424646 0 0) 0x7f76b013aac0 con 0x7f76b00a5920 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): my global_id is 135934 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7694004eb0 con 0x7f76b00a5920 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1090949732 0 0) 0x7f76b0139000 con 0x7f76b00a5d50 2024-08-19T12:51:34.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): my global_id is 184252 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76b013aac0 con 0x7f76b00a5d50 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3454035577 0 0) 0x7f76b013b360 con 0x7f76b006e7b0 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): my global_id is 26966 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76b0139000 con 0x7f76b006e7b0 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 128088983 0 0) 0x7f7694004eb0 con 0x7f76b00a5920 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f76b013b360 con 0x7f76b00a5920 2024-08-19T12:51:34.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f76a00013d0 con 0x7f76b00a5920 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): dump: 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.199 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3401733974 0 0) 0x7f76b013aac0 con 0x7f76b00a5d50 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7694004eb0 con 0x7f76b00a5d50 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f76a8003cd0 con 0x7f76b00a5d50 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:34.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): dump: 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1676773028 0 0) 0x7f76b0139000 con 0x7f76b006e7b0 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f76b013aac0 con 0x7f76b006e7b0 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f76ac0047e0 con 0x7f76b006e7b0 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient(hunting): dump: 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.202 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1115131025 0 0) 0x7f76b013b360 con 0x7f76b00a5920 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.26:6791/0 conn(0x7f76b00a5d50 legacy=0x7f76b01d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.26:6790/0 conn(0x7f76b006e7b0 legacy=0x7f76b01ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient: _finish_hunting 0 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 monclient: found mon.i 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b013ada0 con 0x7f76b00a5920 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient: _finish_auth 0 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient: _check_auth_tickets 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76b7d14640 5 monclient: authenticate success, global_id 135934 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76b7d14640 10 monclient: _renew_subs 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76b7d14640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76b7d14640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f76b013a790 con 0x7f76b00a5920 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76b7d14640 10 client.135934.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76b7d14640 10 monclient: _renew_subs 2024-08-19T12:51:34.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76b7d14640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76b7d14640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f76b01385f0 con 0x7f76b00a5920 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76b7d14640 1 librados: init done 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f76a00039a0 con 0x7f76b00a5920 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f76a0003e90 con 0x7f76b00a5920 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a7fff640 4 set_mon_vals no callback set 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.182+0000 7f76a67fc640 10 monclient: dump: 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.205 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.206 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.206 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.206 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.206 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.206 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.206 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.206 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.206 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.206 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.206 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.206 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.206 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.183+0000 7f76b7d14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.183+0000 7f76b7d14640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f76980065c0 con 0x7f7698002860 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.183+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f76a0004030 con 0x7f76b00a5920 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.183+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(485..485 src has 350..485) ==== 6214+0+0 (unknown 131736049 0 0) 0x7f76a004e380 con 0x7f76b00a5920 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.183+0000 7f76a67fc640 10 client.135934.objecter ms_dispatch 0x7f76b0000b90 osd_map(485..485 src has 350..485) 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.183+0000 7f76a67fc640 3 client.135934.objecter handle_osd_map got epochs [485,485] > 0 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.183+0000 7f76a67fc640 3 client.135934.objecter handle_osd_map decoding full epoch 485 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.184+0000 7f76a67fc640 20 client.135934.objecter dump_active .. 0 homeless 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.184+0000 7f76b62b1640 10 monclient: get_auth_request con 0x7f76940417f0 auth_method 0 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.185+0000 7f76b5ab0640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.26:6792/0 conn(0x7f7698002860 legacy=0x7f7698004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.185+0000 7f76a67fc640 10 client.135934.objecter ms_handle_connect 0x7f7698002860 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.185+0000 7f76a67fc640 10 client.135934.objecter resend_mon_ops 2024-08-19T12:51:34.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.185+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422432723 0 0) 0x7f76980065c0 con 0x7f7698002860 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.185+0000 7f76a67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.185+0000 7f76a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.185+0000 7f76a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.185+0000 7f76a67fc640 10 monclient(hunting): my global_id is 164697 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.185+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76b013b360 con 0x7f7698002860 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 941381876 0 0) 0x7f76b013b360 con 0x7f7698002860 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f76980065c0 con 0x7f7698002860 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f76ac005160 con 0x7f7698002860 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 10 monclient: dump: 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.208 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2305860560 0 0) 0x7f76980065c0 con 0x7f7698002860 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f769800a160 con 0x7f7698002860 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 10 client.135934.objecter ms_handle_connect 0x7f76940417f0 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f76ac0074f0 con 0x7f7698002860 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.26:6792/0 conn(0x7f7698002860 legacy=0x7f7698004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.186+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.26:6792/0 conn(0x7f7698002860 legacy=0x7f7698004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.192+0000 7f76b7d14640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.192+0000 7f76b7d14640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7698009e50 con 0x7f7698009110 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76b5ab0640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.26:6792/0 conn(0x7f7698009110 legacy=0x7f76980095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 10 client.135934.objecter ms_handle_connect 0x7f7698009110 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 10 client.135934.objecter resend_mon_ops 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559297576 0 0) 0x7f7698009e50 con 0x7f7698009110 2024-08-19T12:51:34.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 10 monclient(hunting): my global_id is 164706 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76980065c0 con 0x7f7698009110 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3831509708 0 0) 0x7f76980065c0 con 0x7f7698009110 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7698009e50 con 0x7f7698009110 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f76ac005bd0 con 0x7f7698009110 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 10 monclient: dump: 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.211 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2120371360 0 0) 0x7f7698009e50 con 0x7f7698009110 2024-08-19T12:51:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f769800c3a0 con 0x7f7698009110 2024-08-19T12:51:34.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7711 (unknown 0 0 2812229288) 0x7f76ac0076a0 con 0x7f7698009110 2024-08-19T12:51:34.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:34.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:34.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.26:6792/0 conn(0x7f7698009110 legacy=0x7f76980095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.195+0000 7f76a67fc640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.26:6792/0 conn(0x7f7698009110 legacy=0x7f76980095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:34.213 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:34.213 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:34.213 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:34.213 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:34.213 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:34.213 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 277749, 2024-08-19T12:51:34.214 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:34.214 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:34.214 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:34.214 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:34.214 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:34.214 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:34.214 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:34.214 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:34.214 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:34.214 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:34.214 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.214 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.215 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.216 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:34.217 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.218 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:34.220 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:34.221 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.221 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:34.221 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:34.221 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:34.221 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.222 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.222 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.222 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:34.222 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:34.222 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:34.222 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:34.223 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:34.225 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:34.228 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:34.230 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.231 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.233 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:34.233 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:34.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 10 librados: watch_flush enter 2024-08-19T12:51:34.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 10 librados: watch_flush exit 2024-08-19T12:51:34.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 20 client.135934.objecter shutdown clearing up homeless session... 2024-08-19T12:51:34.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 10 client.135934.objecter successfully canceled tick 2024-08-19T12:51:34.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.104:6800/899887506 conn(0x7f76940417f0 legacy=0x7f7694043cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 10 monclient: shutdown 2024-08-19T12:51:34.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:34.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 1 -- 172.21.15.26:0/2176638320 >> v1:172.21.15.104:6792/0 conn(0x7f76b00a5920 legacy=0x7f76b01d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b62b1640 1 -- 172.21.15.26:0/2176638320 reap_dead start 2024-08-19T12:51:34.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b62b1640 1 -- 172.21.15.26:0/2176638320 reap_dead start 2024-08-19T12:51:34.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 1 -- 172.21.15.26:0/2176638320 shutdown_connections 2024-08-19T12:51:34.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 1 -- 172.21.15.26:0/2176638320 >> 172.21.15.26:0/2176638320 conn(0x7f76b00a8740 msgr2=0x7f76b00aa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:34.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 1 -- 172.21.15.26:0/2176638320 shutdown_connections 2024-08-19T12:51:34.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 1 -- 172.21.15.26:0/2176638320 wait complete. 2024-08-19T12:51:34.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.196+0000 7f76b7d14640 1 librados: shutdown 2024-08-19T12:51:34.236 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:34.236 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:51:34.236 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:51:34.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178178096}) 2024-08-19T12:51:34.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3620054437}) 2024-08-19T12:51:34.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178178096}) 2024-08-19T12:51:34.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3576959993}) 2024-08-19T12:51:34.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178178096}) 2024-08-19T12:51:34.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2833283062}) 2024-08-19T12:51:34.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178178096}) 2024-08-19T12:51:34.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1563521119}) 2024-08-19T12:51:34.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178178096}) 2024-08-19T12:51:34.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1487718347}) 2024-08-19T12:51:34.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178178096}) 2024-08-19T12:51:34.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3605101690}) 2024-08-19T12:51:34.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178178096}) 2024-08-19T12:51:34.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1706041545}) 2024-08-19T12:51:34.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178178096}) 2024-08-19T12:51:34.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.271+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=505795587}) 2024-08-19T12:51:34.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358916566}) 2024-08-19T12:51:34.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2930718844}) 2024-08-19T12:51:34.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358916566}) 2024-08-19T12:51:34.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=462820042}) 2024-08-19T12:51:34.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358916566}) 2024-08-19T12:51:34.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=189276457}) 2024-08-19T12:51:34.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.280 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358916566}) 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2281924061}) 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358916566}) 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.277+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1226342878}) 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.278+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.278+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358916566}) 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.278+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2582741895}) 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.278+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.278+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358916566}) 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.278+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=345811613}) 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.278+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.278+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358916566}) 2024-08-19T12:51:34.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.278+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1339798449}) 2024-08-19T12:51:34.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575507169}) 2024-08-19T12:51:34.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4093841319}) 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575507169}) 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673129678}) 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575507169}) 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3821658722}) 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575507169}) 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=627307789}) 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575507169}) 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1986874314}) 2024-08-19T12:51:34.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575507169}) 2024-08-19T12:51:34.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1967364447}) 2024-08-19T12:51:34.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575507169}) 2024-08-19T12:51:34.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=156711371}) 2024-08-19T12:51:34.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575507169}) 2024-08-19T12:51:34.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.284+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2932296042}) 2024-08-19T12:51:34.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667839137}) 2024-08-19T12:51:34.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1778932660}) 2024-08-19T12:51:34.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667839137}) 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2848523399}) 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667839137}) 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3404535117}) 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667839137}) 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3656179423}) 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667839137}) 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2804592637}) 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667839137}) 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=716309459}) 2024-08-19T12:51:34.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667839137}) 2024-08-19T12:51:34.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=706056051}) 2024-08-19T12:51:34.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667839137}) 2024-08-19T12:51:34.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.288+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3659389657}) 2024-08-19T12:51:34.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.376+0000 7fe7549e9640 10 monclient: get_monmap_and_config 2024-08-19T12:51:34.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.376+0000 7fe7549e9640 10 monclient: build_initial_monmap 2024-08-19T12:51:34.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.376+0000 7fe7549e9640 10 monclient: monmap: 2024-08-19T12:51:34.377 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:34.377 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.377 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:34.377066+0000 2024-08-19T12:51:34.377 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:34.377066+0000 2024-08-19T12:51:34.377 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:34.377 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.377 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:34.377 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:34.378 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:34.378 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:34.378 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:34.378 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:34.378 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:34.378 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:34.378 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:34.378 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.376+0000 7fe7549e9640 1 Processor -- start 2024-08-19T12:51:34.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.376+0000 7fe7549e9640 1 -- start start 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.376+0000 7fe7549e9640 10 monclient: init 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe7549e9640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe7549e9640 10 monclient: _add_conns ranks=[3,0,2,7,6,4,8,5,1] 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe7549e9640 10 monclient(hunting): picked mon.noname-g con 0x7fe75006e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe7549e9640 10 monclient(hunting): picked mon.noname-a con 0x7fe7500a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe7549e9640 10 monclient(hunting): picked mon.noname-b con 0x7fe7500a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe7549e9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe750136790 con 0x7fe7500a5920 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe7549e9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe75009ddf0 con 0x7fe7500a5d50 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe7549e9640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe75009e0d0 con 0x7fe75006e7b0 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe7549e9640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe7549e9640 10 monclient(hunting): authenticate will time out at 1488.464486s 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe74ed9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe7500a5d50 0x7fe750141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54612/0 (socket says 172.21.15.26:54612) 2024-08-19T12:51:34.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe74ed9d640 1 -- 172.21.15.26:0/1002682459 learned_addr learned my addr 172.21.15.26:0/1002682459 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:34.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.377+0000 7fe747fff640 1 --1- 172.21.15.26:0/1002682459 >> v1:172.21.15.26:6789/0 conn(0x7fe7500a5920 0x7fe75013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60304/0 (socket says 172.21.15.26:60304) 2024-08-19T12:51:34.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.379+0000 7fe747fff640 1 -- 172.21.15.26:0/1002682459 >> v1:172.21.15.26:6789/0 conn(0x7fe7500a5920 legacy=0x7fe75013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.379+0000 7fe74ed9d640 1 -- 172.21.15.26:0/1002682459 >> v1:172.21.15.26:6790/0 conn(0x7fe7500a5d50 legacy=0x7fe750141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74e59c640 1 -- 172.21.15.26:0/1002682459 >> v1:172.21.15.104:6790/0 conn(0x7fe75006e7b0 legacy=0x7fe750136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2080416923 0 0) 0x7fe750136790 con 0x7fe7500a5920 2024-08-19T12:51:34.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient(hunting): my global_id is 145413 2024-08-19T12:51:34.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe73c003520 con 0x7fe7500a5920 2024-08-19T12:51:34.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2937469101 0 0) 0x7fe7480017d0 con 0x7fe7500a5d50 2024-08-19T12:51:34.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient(hunting): my global_id is 26975 2024-08-19T12:51:34.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe750136790 con 0x7fe7500a5d50 2024-08-19T12:51:34.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2154430057 0 0) 0x7fe75009e0d0 con 0x7fe75006e7b0 2024-08-19T12:51:34.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient(hunting): my global_id is 154767 2024-08-19T12:51:34.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7480017d0 con 0x7fe75006e7b0 2024-08-19T12:51:34.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1240466558 0 0) 0x7fe73c003520 con 0x7fe7500a5920 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 >> v1:172.21.15.104:6790/0 conn(0x7fe75006e7b0 legacy=0x7fe750136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 >> v1:172.21.15.26:6790/0 conn(0x7fe7500a5d50 legacy=0x7fe750141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient: _finish_hunting 0 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 monclient: found mon.noname-a 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75009e3b0 con 0x7fe7500a5920 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient: _finish_auth 0 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient: _check_auth_tickets 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe738001ae0 con 0x7fe7500a5920 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe7549e9640 5 monclient: authenticate success, global_id 145413 2024-08-19T12:51:34.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe7549e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient: dump: 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.388 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe7549e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe7380047e0 con 0x7fe7500a5920 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 1 -- 172.21.15.26:0/1002682459 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe738004d50 con 0x7fe7500a5920 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe7549e9640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe7549e9640 4 set_mon_vals no callback set 2024-08-19T12:51:34.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.380+0000 7fe74dd9b640 10 monclient: dump: 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.390 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.381+0000 7fe7549e9640 10 monclient: shutdown 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.381+0000 7fe7549e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.381+0000 7fe7549e9640 1 -- 172.21.15.26:0/1002682459 >> v1:172.21.15.26:6789/0 conn(0x7fe7500a5920 legacy=0x7fe75013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.381+0000 7fe7549e9640 1 -- 172.21.15.26:0/1002682459 shutdown_connections 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.381+0000 7fe7549e9640 1 -- 172.21.15.26:0/1002682459 >> 172.21.15.26:0/1002682459 conn(0x7fe7500a8740 msgr2=0x7fe7500a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.381+0000 7fe7549e9640 1 -- 172.21.15.26:0/1002682459 shutdown_connections 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.381+0000 7fe7549e9640 1 -- 172.21.15.26:0/1002682459 wait complete. 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.381+0000 7fe7549e9640 10 monclient: build_initial_monmap 2024-08-19T12:51:34.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.381+0000 7fe7549e9640 10 monclient: monmap: 2024-08-19T12:51:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 1 librados: starting msgr at 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 1 librados: starting objecter 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 1 Processor -- start 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 1 -- start start 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 1 librados: setting wanted keys 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 1 librados: calling monclient init 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 10 monclient: init 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 10 monclient: _add_conns ranks=[6,4,5,2,3,7,1,8,0] 2024-08-19T12:51:34.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 10 monclient(hunting): picked mon.noname-d con 0x7fe75006e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:34.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 10 monclient(hunting): picked mon.noname-c con 0x7fe7500a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:34.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 10 monclient(hunting): picked mon.noname-h con 0x7fe7500a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:34.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe750138e10 con 0x7fe7500a5920 2024-08-19T12:51:34.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7501382c0 con 0x7fe7500a5d50 2024-08-19T12:51:34.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe75013b170 con 0x7fe75006e7b0 2024-08-19T12:51:34.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:34.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe7549e9640 10 monclient(hunting): authenticate will time out at 1488.469715s 2024-08-19T12:51:34.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe747fff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fe7500a5920 0x7fe750137bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41702/0 (socket says 172.21.15.26:41702) 2024-08-19T12:51:34.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.382+0000 7fe747fff640 1 -- 172.21.15.26:0/435436546 learned_addr learned my addr 172.21.15.26:0/435436546 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:34.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.384+0000 7fe74e59c640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.26:6792/0 conn(0x7fe75006e7b0 legacy=0x7fe7501374a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.384+0000 7fe746ffd640 10 client.?.objecter ms_handle_connect 0x7fe75006e7b0 2024-08-19T12:51:34.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.384+0000 7fe746ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.384+0000 7fe746ffd640 10 client.?.objecter ms_handle_connect 0x7fe7500a5920 2024-08-19T12:51:34.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.384+0000 7fe746ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.384+0000 7fe747fff640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.26:6791/0 conn(0x7fe7500a5920 legacy=0x7fe750137bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1239396282 0 0) 0x7fe75013b170 con 0x7fe75006e7b0 2024-08-19T12:51:34.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): my global_id is 164715 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe74ed9d640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.104:6791/0 conn(0x7fe7500a5d50 legacy=0x7fe7501d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe730004eb0 con 0x7fe75006e7b0 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 client.?.objecter ms_handle_connect 0x7fe7500a5d50 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3524275695 0 0) 0x7fe750138e10 con 0x7fe7500a5920 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): my global_id is 184261 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe75013b170 con 0x7fe7500a5920 2024-08-19T12:51:34.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 474589306 0 0) 0x7fe730004eb0 con 0x7fe75006e7b0 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe750138e10 con 0x7fe75006e7b0 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe740003610 con 0x7fe75006e7b0 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): dump: 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.402 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3728429783 0 0) 0x7fe75013b170 con 0x7fe7500a5920 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe730004eb0 con 0x7fe7500a5920 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 865998256 0 0) 0x7fe7501382c0 con 0x7fe7500a5d50 2024-08-19T12:51:34.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): my global_id is 184199 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe75013b170 con 0x7fe7500a5d50 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe738004cd0 con 0x7fe7500a5920 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient(hunting): dump: 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.404 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1468035960 0 0) 0x7fe750138e10 con 0x7fe75006e7b0 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.104:6791/0 conn(0x7fe7500a5d50 legacy=0x7fe7501d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.26:6791/0 conn(0x7fe7500a5920 legacy=0x7fe750137bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:51:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 monclient: found mon.d 2024-08-19T12:51:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75013a8d0 con 0x7fe75006e7b0 2024-08-19T12:51:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient: _finish_auth 0 2024-08-19T12:51:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:51:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe7549e9640 5 monclient: authenticate success, global_id 164715 2024-08-19T12:51:34.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe7549e9640 10 monclient: _renew_subs 2024-08-19T12:51:34.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe7549e9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:34.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe7549e9640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe75013b590 con 0x7fe75006e7b0 2024-08-19T12:51:34.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe7549e9640 10 client.164715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:34.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe7549e9640 10 monclient: _renew_subs 2024-08-19T12:51:34.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe7549e9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:51:34.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe7549e9640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe7501428c0 con 0x7fe75006e7b0 2024-08-19T12:51:34.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe7549e9640 1 librados: init done 2024-08-19T12:51:34.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe740003b60 con 0x7fe75006e7b0 2024-08-19T12:51:34.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:34.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe740004050 con 0x7fe75006e7b0 2024-08-19T12:51:34.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:34.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe74d59a640 4 set_mon_vals no callback set 2024-08-19T12:51:34.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.385+0000 7fe746ffd640 10 monclient: dump: 2024-08-19T12:51:34.408 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.409 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.409 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.409 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.409 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.409 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.409 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.409 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.409 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.409 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.410 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.410 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.410 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.410 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.410 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.410 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.386+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe7400041f0 con 0x7fe75006e7b0 2024-08-19T12:51:34.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.386+0000 7fe7549e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:34.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.386+0000 7fe7549e9640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe7340065c0 con 0x7fe734002860 2024-08-19T12:51:34.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.386+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(486..486 src has 350..486) ==== 6206+0+0 (unknown 2867174868 0 0) 0x7fe74004e620 con 0x7fe75006e7b0 2024-08-19T12:51:34.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.386+0000 7fe746ffd640 10 client.164715.objecter ms_dispatch 0x7fe750000b90 osd_map(486..486 src has 350..486) 2024-08-19T12:51:34.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.386+0000 7fe746ffd640 3 client.164715.objecter handle_osd_map got epochs [486,486] > 0 2024-08-19T12:51:34.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.386+0000 7fe746ffd640 3 client.164715.objecter handle_osd_map decoding full epoch 486 2024-08-19T12:51:34.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.386+0000 7fe746ffd640 20 client.164715.objecter dump_active .. 0 homeless 2024-08-19T12:51:34.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.386+0000 7fe74ed9d640 10 monclient: get_auth_request con 0x7fe730041750 auth_method 0 2024-08-19T12:51:34.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 10 client.164715.objecter ms_handle_connect 0x7fe734002860 2024-08-19T12:51:34.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe747fff640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.26:6793/0 conn(0x7fe734002860 legacy=0x7fe734004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 10 client.164715.objecter resend_mon_ops 2024-08-19T12:51:34.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 17390831 0 0) 0x7fe7340065c0 con 0x7fe734002860 2024-08-19T12:51:34.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 10 monclient(hunting): my global_id is 184166 2024-08-19T12:51:34.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe750138e10 con 0x7fe734002860 2024-08-19T12:51:34.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 10 client.164715.objecter ms_handle_connect 0x7fe730041750 2024-08-19T12:51:34.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3072517650 0 0) 0x7fe750138e10 con 0x7fe734002860 2024-08-19T12:51:34.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7340065c0 con 0x7fe734002860 2024-08-19T12:51:34.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe7380047e0 con 0x7fe734002860 2024-08-19T12:51:34.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:51:34.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 10 monclient: dump: 2024-08-19T12:51:34.414 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.414 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.414 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.414 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.414 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.414 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.414 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.415 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.415 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.415 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.415 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.415 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.415 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.415 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.415 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.416 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 293468323 0 0) 0x7fe7340065c0 con 0x7fe734002860 2024-08-19T12:51:34.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe73400a160 con 0x7fe734002860 2024-08-19T12:51:34.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe7380053d0 con 0x7fe734002860 2024-08-19T12:51:34.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:34.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:34.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.26:6793/0 conn(0x7fe734002860 legacy=0x7fe734004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.389+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.26:6793/0 conn(0x7fe734002860 legacy=0x7fe734004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.396+0000 7fe7549e9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.396+0000 7fe7549e9640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe734009e70 con 0x7fe734009130 2024-08-19T12:51:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe747fff640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.26:6793/0 conn(0x7fe734009130 legacy=0x7fe7340095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 10 client.164715.objecter ms_handle_connect 0x7fe734009130 2024-08-19T12:51:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 10 client.164715.objecter resend_mon_ops 2024-08-19T12:51:34.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 131330 0 0) 0x7fe734009e70 con 0x7fe734009130 2024-08-19T12:51:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 10 monclient(hunting): my global_id is 184175 2024-08-19T12:51:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7340065c0 con 0x7fe734009130 2024-08-19T12:51:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 983218297 0 0) 0x7fe7340065c0 con 0x7fe734009130 2024-08-19T12:51:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe734009e70 con 0x7fe734009130 2024-08-19T12:51:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe738005b30 con 0x7fe734009130 2024-08-19T12:51:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:51:34.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 10 monclient: dump: 2024-08-19T12:51:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.419 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.420 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1597057796 0 0) 0x7fe734009e70 con 0x7fe734009130 2024-08-19T12:51:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.398+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe73400c3f0 con 0x7fe734009130 2024-08-19T12:51:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7710 (unknown 0 0 2103765561) 0x7fe7380080d0 con 0x7fe734009130 2024-08-19T12:51:34.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe746ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:34.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe746ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:34.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.26:6793/0 conn(0x7fe734009130 legacy=0x7fe7340095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe746ffd640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.26:6793/0 conn(0x7fe734009130 legacy=0x7fe7340095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:34.421 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:34.421 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:34.421 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:34.421 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:34.422 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:34.422 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 19864, 2024-08-19T12:51:34.422 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:34.422 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:34.422 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:34.422 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:34.422 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:34.422 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:34.422 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:34.422 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:34.422 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:34.422 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.423 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.424 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.424 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.424 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.424 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.424 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:34.424 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:34.424 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.424 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.424 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.424 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.424 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.424 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:34.425 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.426 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.427 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.428 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:34.429 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.430 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.430 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:34.430 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:34.430 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.430 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.430 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.431 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:34.431 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:34.431 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.431 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.431 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.432 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:34.432 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.432 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.432 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:34.432 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:34.432 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.433 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.433 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.433 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:34.433 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:34.433 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.433 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.434 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.434 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:34.434 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.434 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.434 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.434 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:34.435 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:34.435 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.435 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.435 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.435 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:34.436 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:34.436 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.436 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.436 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.436 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:34.436 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.436 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.437 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.437 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:34.437 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:34.437 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.437 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.437 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.438 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:34.438 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:34.438 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.438 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.438 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:34.439 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.439 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.439 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:34.439 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:34.440 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.440 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.440 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.440 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:34.440 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:34.440 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.441 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:34.441 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.441 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:34.441 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:34.441 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.441 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.442 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.442 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:34.442 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:34.442 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.443 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.444 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.445 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.446 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.447 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:34.447 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:34.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 10 librados: watch_flush enter 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 10 librados: watch_flush exit 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 20 client.164715.objecter shutdown clearing up homeless session... 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 10 client.164715.objecter successfully canceled tick 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.104:6800/899887506 conn(0x7fe730041750 legacy=0x7fe730043c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 10 monclient: shutdown 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 1 -- 172.21.15.26:0/435436546 >> v1:172.21.15.26:6792/0 conn(0x7fe75006e7b0 legacy=0x7fe7501374a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe74ed9d640 1 -- 172.21.15.26:0/435436546 reap_dead start 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe74ed9d640 1 -- 172.21.15.26:0/435436546 reap_dead start 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 1 -- 172.21.15.26:0/435436546 shutdown_connections 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 1 -- 172.21.15.26:0/435436546 >> 172.21.15.26:0/435436546 conn(0x7fe7500a8740 msgr2=0x7fe7500aa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 1 -- 172.21.15.26:0/435436546 shutdown_connections 2024-08-19T12:51:34.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 1 -- 172.21.15.26:0/435436546 wait complete. 2024-08-19T12:51:34.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.399+0000 7fe7549e9640 1 librados: shutdown 2024-08-19T12:51:34.450 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:34.450 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:51:34.450 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:51:34.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99259612}) 2024-08-19T12:51:34.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2974423294}) 2024-08-19T12:51:34.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99259612}) 2024-08-19T12:51:34.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1496712167}) 2024-08-19T12:51:34.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99259612}) 2024-08-19T12:51:34.563 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=813684239}) 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99259612}) 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2028541848}) 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99259612}) 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1215836090}) 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99259612}) 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=850370207}) 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99259612}) 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2325486412}) 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99259612}) 2024-08-19T12:51:34.564 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.561+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1307643922}) 2024-08-19T12:51:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.590+0000 7f20f4029640 10 monclient: get_monmap_and_config 2024-08-19T12:51:34.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.590+0000 7f20f4029640 10 monclient: build_initial_monmap 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.590+0000 7f20f4029640 10 monclient: monmap: 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:34.590895+0000 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:34.590895+0000 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:34.591 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:34.592 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:34.592 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:34.592 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.590+0000 7f20f4029640 1 Processor -- start 2024-08-19T12:51:34.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.590+0000 7f20f4029640 1 -- start start 2024-08-19T12:51:34.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.590+0000 7f20f4029640 10 monclient: init 2024-08-19T12:51:34.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.591+0000 7f20f4029640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:34.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.591+0000 7f20f4029640 10 monclient: _add_conns ranks=[0,7,3,2,4,6,1,5,8] 2024-08-19T12:51:34.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.591+0000 7f20f4029640 10 monclient(hunting): picked mon.noname-a con 0x7f20ec073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:34.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.591+0000 7f20f4029640 10 monclient(hunting): picked mon.noname-i con 0x7f20ec0a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:34.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.591+0000 7f20f4029640 10 monclient(hunting): picked mon.noname-g con 0x7f20ec13f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:34.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.591+0000 7f20f4029640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20ec09dae0 con 0x7f20ec073470 2024-08-19T12:51:34.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.591+0000 7f20f4029640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20ec136910 con 0x7f20ec13f2e0 2024-08-19T12:51:34.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.591+0000 7f20f4029640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20ec09dd50 con 0x7f20ec0a57a0 2024-08-19T12:51:34.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.591+0000 7f20f4029640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:34.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.591+0000 7f20f4029640 10 monclient(hunting): authenticate will time out at 1488.678430s 2024-08-19T12:51:34.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.591+0000 7f20f1dc5640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f20ec073470 0x7f20ec136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60312/0 (socket says 172.21.15.26:60312) 2024-08-19T12:51:34.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.591+0000 7f20f1dc5640 1 -- 172.21.15.26:0/902056020 learned_addr learned my addr 172.21.15.26:0/902056020 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.593+0000 7f20f1dc5640 1 -- 172.21.15.26:0/902056020 >> v1:172.21.15.26:6789/0 conn(0x7f20ec073470 legacy=0x7f20ec136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.593+0000 7f20f25c6640 1 -- 172.21.15.26:0/902056020 >> v1:172.21.15.104:6790/0 conn(0x7f20ec13f2e0 legacy=0x7f20ec1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.593+0000 7f20f15c4640 1 -- 172.21.15.26:0/902056020 >> v1:172.21.15.104:6792/0 conn(0x7f20ec0a57a0 legacy=0x7f20ec0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 1 -- 172.21.15.26:0/902056020 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 234613129 0 0) 0x7f20ec09dd50 con 0x7f20ec0a57a0 2024-08-19T12:51:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient(hunting): my global_id is 135943 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 1 -- 172.21.15.26:0/902056020 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d8003520 con 0x7f20ec0a57a0 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 1 -- 172.21.15.26:0/902056020 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 592524516 0 0) 0x7f20ec136910 con 0x7f20ec13f2e0 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient(hunting): my global_id is 154776 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 1 -- 172.21.15.26:0/902056020 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20ec09dd50 con 0x7f20ec13f2e0 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 1 -- 172.21.15.26:0/902056020 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 424121200 0 0) 0x7f20d8003520 con 0x7f20ec0a57a0 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 1 -- 172.21.15.26:0/902056020 >> v1:172.21.15.104:6790/0 conn(0x7f20ec13f2e0 legacy=0x7f20ec1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 1 -- 172.21.15.26:0/902056020 >> v1:172.21.15.26:6789/0 conn(0x7f20ec073470 legacy=0x7f20ec136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient: _finish_hunting 0 2024-08-19T12:51:34.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 1 monclient: found mon.noname-i 2024-08-19T12:51:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:51:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 1 -- 172.21.15.26:0/902056020 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20ec09e030 con 0x7f20ec0a57a0 2024-08-19T12:51:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient: _finish_auth 0 2024-08-19T12:51:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient: _check_auth_tickets 2024-08-19T12:51:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 1 -- 172.21.15.26:0/902056020 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20dc001bf0 con 0x7f20ec0a57a0 2024-08-19T12:51:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:51:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient: dump: 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.601 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.602 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f4029640 5 monclient: authenticate success, global_id 135943 2024-08-19T12:51:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f4029640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 1 -- 172.21.15.26:0/902056020 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20dc003610 con 0x7f20ec0a57a0 2024-08-19T12:51:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 1 -- 172.21.15.26:0/902056020 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20dc003b80 con 0x7f20ec0a57a0 2024-08-19T12:51:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f4029640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f4029640 4 set_mon_vals no callback set 2024-08-19T12:51:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.594+0000 7f20f0dc3640 10 monclient: dump: 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.595+0000 7f20f4029640 10 monclient: shutdown 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.595+0000 7f20f4029640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.595+0000 7f20f4029640 1 -- 172.21.15.26:0/902056020 >> v1:172.21.15.104:6792/0 conn(0x7f20ec0a57a0 legacy=0x7f20ec0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.595+0000 7f20f4029640 1 -- 172.21.15.26:0/902056020 shutdown_connections 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.595+0000 7f20f4029640 1 -- 172.21.15.26:0/902056020 >> 172.21.15.26:0/902056020 conn(0x7f20ec0abff0 msgr2=0x7f20ec0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.595+0000 7f20f4029640 1 -- 172.21.15.26:0/902056020 shutdown_connections 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.595+0000 7f20f4029640 1 -- 172.21.15.26:0/902056020 wait complete. 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.595+0000 7f20f4029640 10 monclient: build_initial_monmap 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.595+0000 7f20f4029640 10 monclient: monmap: 2024-08-19T12:51:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:34.605 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 1 librados: starting msgr at 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 1 librados: starting objecter 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 1 Processor -- start 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 1 -- start start 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 1 librados: setting wanted keys 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 1 librados: calling monclient init 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 10 monclient: init 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 10 monclient: _add_conns ranks=[3,2,5,4,1,0,8,7,6] 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 10 monclient(hunting): picked mon.noname-g con 0x7f20ec073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 10 monclient(hunting): picked mon.noname-b con 0x7f20ec0a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 10 monclient(hunting): picked mon.noname-h con 0x7f20ec13f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20ec131d10 con 0x7f20ec0a57a0 2024-08-19T12:51:34.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20ec1cef20 con 0x7f20ec073470 2024-08-19T12:51:34.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20ec13f0b0 con 0x7f20ec13f2e0 2024-08-19T12:51:34.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:34.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f4029640 10 monclient(hunting): authenticate will time out at 1488.683791s 2024-08-19T12:51:34.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f15c4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f20ec0a57a0 0x7f20ec1ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54624/0 (socket says 172.21.15.26:54624) 2024-08-19T12:51:34.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.596+0000 7f20f15c4640 1 -- 172.21.15.26:0/2831265627 learned_addr learned my addr 172.21.15.26:0/2831265627 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:34.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20f15c4640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.26:6790/0 conn(0x7f20ec0a57a0 legacy=0x7f20ec1ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 client.?.objecter ms_handle_connect 0x7f20ec0a57a0 2024-08-19T12:51:34.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 client.?.objecter ms_handle_connect 0x7f20ec13f2e0 2024-08-19T12:51:34.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20f25c6640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.104:6791/0 conn(0x7f20ec13f2e0 legacy=0x7f20ec1d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 client.?.objecter ms_handle_connect 0x7f20ec073470 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20f1dc5640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.104:6790/0 conn(0x7f20ec073470 legacy=0x7f20ec1ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1908520845 0 0) 0x7f20ec13f0b0 con 0x7f20ec13f2e0 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): my global_id is 184208 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d0004eb0 con 0x7f20ec13f2e0 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 950967666 0 0) 0x7f20ec1cef20 con 0x7f20ec073470 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): my global_id is 154785 2024-08-19T12:51:34.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20ec13f0b0 con 0x7f20ec073470 2024-08-19T12:51:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2626626175 0 0) 0x7f20ec131d10 con 0x7f20ec0a57a0 2024-08-19T12:51:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): my global_id is 26984 2024-08-19T12:51:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20ec1cef20 con 0x7f20ec0a57a0 2024-08-19T12:51:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2275874825 0 0) 0x7f20d0004eb0 con 0x7f20ec13f2e0 2024-08-19T12:51:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20ec131d10 con 0x7f20ec13f2e0 2024-08-19T12:51:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20e4002030 con 0x7f20ec13f2e0 2024-08-19T12:51:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:51:34.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): dump: 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3932423758 0 0) 0x7f20ec13f0b0 con 0x7f20ec073470 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20d0004eb0 con 0x7f20ec073470 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3386855628 0 0) 0x7f20ec1cef20 con 0x7f20ec0a57a0 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20ec13f0b0 con 0x7f20ec0a57a0 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20e8001bc0 con 0x7f20ec073470 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): dump: 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20dc001bb0 con 0x7f20ec0a57a0 2024-08-19T12:51:34.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient(hunting): dump: 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.618 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 120981256 0 0) 0x7f20ec131d10 con 0x7f20ec13f2e0 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.104:6790/0 conn(0x7f20ec073470 legacy=0x7f20ec1ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.26:6790/0 conn(0x7f20ec0a57a0 legacy=0x7f20ec1ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient: _finish_hunting 0 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 monclient: found mon.h 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20ec12f9b0 con 0x7f20ec13f2e0 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient: _finish_auth 0 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 10 monclient: _check_auth_tickets 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:34.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.599+0000 7f20f4029640 5 monclient: authenticate success, global_id 184208 2024-08-19T12:51:34.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20f4029640 10 monclient: _renew_subs 2024-08-19T12:51:34.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20f4029640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:51:34.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20f4029640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f20ec131a30 con 0x7f20ec13f2e0 2024-08-19T12:51:34.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20f4029640 10 client.184208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:34.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20f4029640 10 monclient: _renew_subs 2024-08-19T12:51:34.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20f4029640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:51:34.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20f4029640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f20ec1324a0 con 0x7f20ec13f2e0 2024-08-19T12:51:34.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20f4029640 1 librados: init done 2024-08-19T12:51:34.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f20e4003e60 con 0x7f20ec13f2e0 2024-08-19T12:51:34.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20e27fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:34.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20e4003a10 con 0x7f20ec13f2e0 2024-08-19T12:51:34.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20e27fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:51:34.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20e3fff640 4 set_mon_vals no callback set 2024-08-19T12:51:34.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20e27fc640 10 monclient: dump: 2024-08-19T12:51:34.621 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.621 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.621 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.621 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.621 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.621 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.621 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.621 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.622 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.622 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.622 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.622 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.622 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.622 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.622 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.622 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20f4029640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:34.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.600+0000 7f20f4029640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f20d40065c0 con 0x7f20d4002860 2024-08-19T12:51:34.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.601+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f20e4004440 con 0x7f20ec13f2e0 2024-08-19T12:51:34.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.601+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(486..486 src has 350..486) ==== 6206+0+0 (unknown 2867174868 0 0) 0x7f20e404e720 con 0x7f20ec13f2e0 2024-08-19T12:51:34.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.601+0000 7f20e27fc640 10 client.184208.objecter ms_dispatch 0x7f20ec000b90 osd_map(486..486 src has 350..486) 2024-08-19T12:51:34.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.601+0000 7f20e27fc640 3 client.184208.objecter handle_osd_map got epochs [486,486] > 0 2024-08-19T12:51:34.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.601+0000 7f20e27fc640 3 client.184208.objecter handle_osd_map decoding full epoch 486 2024-08-19T12:51:34.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.601+0000 7f20e27fc640 20 client.184208.objecter dump_active .. 0 homeless 2024-08-19T12:51:34.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.601+0000 7f20f15c4640 10 monclient: get_auth_request con 0x7f20d00418a0 auth_method 0 2024-08-19T12:51:34.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 10 client.184208.objecter ms_handle_connect 0x7f20d4002860 2024-08-19T12:51:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 10 client.184208.objecter resend_mon_ops 2024-08-19T12:51:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20f1dc5640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.104:6789/0 conn(0x7f20d4002860 legacy=0x7f20d4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 493990491 0 0) 0x7f20d40065c0 con 0x7f20d4002860 2024-08-19T12:51:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 10 monclient(hunting): my global_id is 184204 2024-08-19T12:51:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20ec131d10 con 0x7f20d4002860 2024-08-19T12:51:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2255275137 0 0) 0x7f20ec131d10 con 0x7f20d4002860 2024-08-19T12:51:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20d40065c0 con 0x7f20d4002860 2024-08-19T12:51:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20e8005040 con 0x7f20d4002860 2024-08-19T12:51:34.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:51:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 10 monclient: dump: 2024-08-19T12:51:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.626 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.628 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 10 client.184208.objecter ms_handle_connect 0x7f20d00418a0 2024-08-19T12:51:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 635301703 0 0) 0x7f20d40065c0 con 0x7f20d4002860 2024-08-19T12:51:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.603+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20d400a160 con 0x7f20d4002860 2024-08-19T12:51:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.604+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f20e80052e0 con 0x7f20d4002860 2024-08-19T12:51:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.604+0000 7f20e27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.604+0000 7f20e27fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.604+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.104:6789/0 conn(0x7f20d4002860 legacy=0x7f20d4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.604+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.104:6789/0 conn(0x7f20d4002860 legacy=0x7f20d4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.610+0000 7f20f4029640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:34.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.610+0000 7f20f4029640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f20d4009e70 con 0x7f20d4009130 2024-08-19T12:51:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.612+0000 7f20f1dc5640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.104:6789/0 conn(0x7f20d4009130 legacy=0x7f20d40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.612+0000 7f20e27fc640 10 client.184208.objecter ms_handle_connect 0x7f20d4009130 2024-08-19T12:51:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.612+0000 7f20e27fc640 10 client.184208.objecter resend_mon_ops 2024-08-19T12:51:34.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.612+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 123699819 0 0) 0x7f20d4009e70 con 0x7f20d4009130 2024-08-19T12:51:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.612+0000 7f20e27fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.612+0000 7f20e27fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.612+0000 7f20e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.612+0000 7f20e27fc640 10 monclient(hunting): my global_id is 184213 2024-08-19T12:51:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.612+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d40065c0 con 0x7f20d4009130 2024-08-19T12:51:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 225992178 0 0) 0x7f20d40065c0 con 0x7f20d4009130 2024-08-19T12:51:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20d4009e70 con 0x7f20d4009130 2024-08-19T12:51:34.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f20e8005c10 con 0x7f20d4009130 2024-08-19T12:51:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:51:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 10 monclient: dump: 2024-08-19T12:51:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.634 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.635 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.636 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4199707235 0 0) 0x7f20d4009e70 con 0x7f20d4009130 2024-08-19T12:51:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f20d400c410 con 0x7f20d4009130 2024-08-19T12:51:34.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 3157817549) 0x7f20e8005f20 con 0x7f20d4009130 2024-08-19T12:51:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.104:6789/0 conn(0x7f20d4009130 legacy=0x7f20d40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20e27fc640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.104:6789/0 conn(0x7f20d4009130 legacy=0x7f20d40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:34.637 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:34.638 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:34.638 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:34.638 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:34.638 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:34.638 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 20081, 2024-08-19T12:51:34.638 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:34.638 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:34.639 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:34.639 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:34.639 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:34.639 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:34.639 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:34.639 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:34.639 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:34.639 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:34.640 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.640 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:51:34.640 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:34.640 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:34.640 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:34.640 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.640 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.641 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.641 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.641 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.641 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.641 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.641 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.641 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.641 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.641 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.641 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.642 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:34.642 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:34.642 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.642 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.642 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.642 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.642 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.642 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.642 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.642 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.642 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.642 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.643 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.643 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.643 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:34.643 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:34.643 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:34.643 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:34.643 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.644 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:34.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.646 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:34.646 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:34.646 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:34.646 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:34.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.654 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:34.654 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:34.654 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:34.654 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.654 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.654 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:34.655 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:34.655 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.655 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.655 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.659 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:34.659 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:34.659 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:34.661 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.662 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.662 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.662 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.662 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:34.662 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:34.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.662 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.662 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.662 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:34.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.663 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.663 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:34.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.663 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.663 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.663 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:34.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.663 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:34.663 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:34.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20f4029640 10 librados: watch_flush enter 2024-08-19T12:51:34.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20f4029640 10 librados: watch_flush exit 2024-08-19T12:51:34.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20f4029640 20 client.184208.objecter shutdown clearing up homeless session... 2024-08-19T12:51:34.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.613+0000 7f20f4029640 10 client.184208.objecter successfully canceled tick 2024-08-19T12:51:34.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.614+0000 7f20f4029640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.104:6800/899887506 conn(0x7f20d00418a0 legacy=0x7f20d0043d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.614+0000 7f20f4029640 10 monclient: shutdown 2024-08-19T12:51:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.614+0000 7f20f4029640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.614+0000 7f20f4029640 1 -- 172.21.15.26:0/2831265627 >> v1:172.21.15.104:6791/0 conn(0x7f20ec13f2e0 legacy=0x7f20ec1d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.614+0000 7f20f25c6640 1 -- 172.21.15.26:0/2831265627 reap_dead start 2024-08-19T12:51:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.614+0000 7f20f25c6640 1 -- 172.21.15.26:0/2831265627 reap_dead start 2024-08-19T12:51:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.614+0000 7f20f4029640 1 -- 172.21.15.26:0/2831265627 shutdown_connections 2024-08-19T12:51:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.614+0000 7f20f4029640 1 -- 172.21.15.26:0/2831265627 >> 172.21.15.26:0/2831265627 conn(0x7f20ec0abff0 msgr2=0x7f20ec076880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.614+0000 7f20f4029640 1 -- 172.21.15.26:0/2831265627 shutdown_connections 2024-08-19T12:51:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.614+0000 7f20f4029640 1 -- 172.21.15.26:0/2831265627 wait complete. 2024-08-19T12:51:34.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.614+0000 7f20f4029640 1 librados: shutdown 2024-08-19T12:51:34.666 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:34.667 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:51:34.667 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:51:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.766+0000 7f9819b72640 10 monclient: get_monmap_and_config 2024-08-19T12:51:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.766+0000 7f9819b72640 10 monclient: build_initial_monmap 2024-08-19T12:51:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.766+0000 7f9819b72640 10 monclient: monmap: 2024-08-19T12:51:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:34.767247+0000 2024-08-19T12:51:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:34.767247+0000 2024-08-19T12:51:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:34.768 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:34.768 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 1 Processor -- start 2024-08-19T12:51:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 1 -- start start 2024-08-19T12:51:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 10 monclient: init 2024-08-19T12:51:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 10 monclient: _add_conns ranks=[6,7,1,5,2,3,4,0,8] 2024-08-19T12:51:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 10 monclient(hunting): picked mon.noname-d con 0x7f981406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 10 monclient(hunting): picked mon.noname-i con 0x7f98140a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 10 monclient(hunting): picked mon.noname-f con 0x7f98140a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:51:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9814136790 con 0x7f98140a5d50 2024-08-19T12:51:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f981409ddf0 con 0x7f981406e7b0 2024-08-19T12:51:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f981409e0d0 con 0x7f98140a5920 2024-08-19T12:51:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:34.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f9819b72640 10 monclient(hunting): authenticate will time out at 1488.854785s 2024-08-19T12:51:34.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f98137fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f981406e7b0 0x7f9814136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51836/0 (socket says 172.21.15.26:51836) 2024-08-19T12:51:34.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.767+0000 7f98137fe640 1 -- 172.21.15.26:0/789668704 learned_addr learned my addr 172.21.15.26:0/789668704 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:34.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.769+0000 7f98137fe640 1 -- 172.21.15.26:0/789668704 >> v1:172.21.15.26:6792/0 conn(0x7f981406e7b0 legacy=0x7f9814136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 1 -- 172.21.15.26:0/789668704 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1652431574 0 0) 0x7f9804001450 con 0x7f981406e7b0 2024-08-19T12:51:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient(hunting): my global_id is 164724 2024-08-19T12:51:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 1 -- 172.21.15.26:0/789668704 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97fc003520 con 0x7f981406e7b0 2024-08-19T12:51:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f9812ffd640 1 -- 172.21.15.26:0/789668704 >> v1:172.21.15.104:6792/0 conn(0x7f98140a5920 legacy=0x7f981413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f9813fff640 1 -- 172.21.15.26:0/789668704 >> v1:172.21.15.104:6789/0 conn(0x7f98140a5d50 legacy=0x7f9814141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 1 -- 172.21.15.26:0/789668704 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3084977547 0 0) 0x7f97fc003520 con 0x7f981406e7b0 2024-08-19T12:51:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 1 -- 172.21.15.26:0/789668704 >> v1:172.21.15.104:6792/0 conn(0x7f98140a5920 legacy=0x7f981413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 1 -- 172.21.15.26:0/789668704 >> v1:172.21.15.104:6789/0 conn(0x7f98140a5d50 legacy=0x7f9814141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient: _finish_hunting 0 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 1 monclient: found mon.noname-d 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 1 -- 172.21.15.26:0/789668704 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f981409e3b0 con 0x7f981406e7b0 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient: _finish_auth 0 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient: _check_auth_tickets 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 1 -- 172.21.15.26:0/789668704 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9804001cb0 con 0x7f981406e7b0 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient: dump: 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.776 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.777 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f9819b72640 5 monclient: authenticate success, global_id 164724 2024-08-19T12:51:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f9819b72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 1 -- 172.21.15.26:0/789668704 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9804004990 con 0x7f981406e7b0 2024-08-19T12:51:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 1 -- 172.21.15.26:0/789668704 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9804004f00 con 0x7f981406e7b0 2024-08-19T12:51:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f98127fc640 10 monclient: dump: 2024-08-19T12:51:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.778 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f9819b72640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.770+0000 7f9819b72640 4 set_mon_vals no callback set 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.771+0000 7f9819b72640 10 monclient: shutdown 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.771+0000 7f9819b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.771+0000 7f9819b72640 1 -- 172.21.15.26:0/789668704 >> v1:172.21.15.26:6792/0 conn(0x7f981406e7b0 legacy=0x7f9814136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.771+0000 7f9819b72640 1 -- 172.21.15.26:0/789668704 shutdown_connections 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.771+0000 7f9819b72640 1 -- 172.21.15.26:0/789668704 >> 172.21.15.26:0/789668704 conn(0x7f98140a8740 msgr2=0x7f98140a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.771+0000 7f9819b72640 1 -- 172.21.15.26:0/789668704 shutdown_connections 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.771+0000 7f9819b72640 1 -- 172.21.15.26:0/789668704 wait complete. 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.771+0000 7f9819b72640 10 monclient: build_initial_monmap 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.771+0000 7f9819b72640 10 monclient: monmap: 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:34.780 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.771+0000 7f9819b72640 1 librados: starting msgr at 2024-08-19T12:51:34.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.771+0000 7f9819b72640 1 librados: starting objecter 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 1 Processor -- start 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 1 -- start start 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 1 librados: setting wanted keys 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 1 librados: calling monclient init 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 10 monclient: init 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 10 monclient: _add_conns ranks=[5,0,8,1,7,3,6,2,4] 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 10 monclient(hunting): picked mon.noname-h con 0x7f981406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 10 monclient(hunting): picked mon.noname-a con 0x7f98140a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 10 monclient(hunting): picked mon.noname-e con 0x7f98140a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f981413a8f0 con 0x7f98140a5920 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9814142e30 con 0x7f981406e7b0 2024-08-19T12:51:34.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9814138e30 con 0x7f98140a5d50 2024-08-19T12:51:34.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:34.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9819b72640 10 monclient(hunting): authenticate will time out at 1488.859767s 2024-08-19T12:51:34.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9813fff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f98140a5d50 0x7f98141d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:45618/0 (socket says 172.21.15.26:45618) 2024-08-19T12:51:34.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9813fff640 1 -- 172.21.15.26:0/1578889501 learned_addr learned my addr 172.21.15.26:0/1578889501 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:34.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.772+0000 7f9812ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f98140a5920 0x7f9814137b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:60328/0 (socket says 172.21.15.26:60328) 2024-08-19T12:51:34.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.774+0000 7f9813fff640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.26:6793/0 conn(0x7f98140a5d50 legacy=0x7f98141d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.774+0000 7f9812ffd640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.26:6789/0 conn(0x7f98140a5920 legacy=0x7f9814137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.774+0000 7f97fbfff640 10 client.?.objecter ms_handle_connect 0x7f98140a5d50 2024-08-19T12:51:34.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 client.?.objecter ms_handle_connect 0x7f98140a5920 2024-08-19T12:51:34.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 client.?.objecter ms_handle_connect 0x7f981406e7b0 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f98137fe640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.104:6791/0 conn(0x7f981406e7b0 legacy=0x7f9814137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3509073555 0 0) 0x7f9808001630 con 0x7f98140a5920 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): my global_id is 145422 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9800004eb0 con 0x7f98140a5920 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030489947 0 0) 0x7f9814138e30 con 0x7f98140a5d50 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): my global_id is 184184 2024-08-19T12:51:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9808001630 con 0x7f98140a5d50 2024-08-19T12:51:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3933111885 0 0) 0x7f9814142e30 con 0x7f981406e7b0 2024-08-19T12:51:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): my global_id is 184217 2024-08-19T12:51:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9814138e30 con 0x7f981406e7b0 2024-08-19T12:51:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2258222824 0 0) 0x7f9800004eb0 con 0x7f98140a5920 2024-08-19T12:51:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9814142e30 con 0x7f98140a5920 2024-08-19T12:51:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9808003950 con 0x7f98140a5920 2024-08-19T12:51:34.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): dump: 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.791 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3013903367 0 0) 0x7f9808001630 con 0x7f98140a5d50 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9800004eb0 con 0x7f98140a5d50 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f980c001f90 con 0x7f98140a5d50 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): dump: 2024-08-19T12:51:34.792 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.793 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 46537591 0 0) 0x7f9814138e30 con 0x7f981406e7b0 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9808001630 con 0x7f981406e7b0 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9804001c30 con 0x7f981406e7b0 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient(hunting): dump: 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.794 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.795 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.795 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.795 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.795 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.795 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.795 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.795 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.795 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.795 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3571893249 0 0) 0x7f9814142e30 con 0x7f98140a5920 2024-08-19T12:51:34.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.26:6793/0 conn(0x7f98140a5d50 legacy=0x7f98141d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.104:6791/0 conn(0x7f981406e7b0 legacy=0x7f9814137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient: _finish_hunting 0 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 monclient: found mon.a 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9814138250 con 0x7f98140a5920 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient: _finish_auth 0 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient: _check_auth_tickets 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f9819b72640 5 monclient: authenticate success, global_id 145422 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f9819b72640 10 monclient: _renew_subs 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f9819b72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f9819b72640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f981413b190 con 0x7f98140a5920 2024-08-19T12:51:34.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f9819b72640 10 client.145422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f9819b72640 10 monclient: _renew_subs 2024-08-19T12:51:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f9819b72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f9819b72640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9814142870 con 0x7f98140a5920 2024-08-19T12:51:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f9819b72640 1 librados: init done 2024-08-19T12:51:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9808001d20 con 0x7f98140a5920 2024-08-19T12:51:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9808004160 con 0x7f98140a5920 2024-08-19T12:51:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f9811ffb640 4 set_mon_vals no callback set 2024-08-19T12:51:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.775+0000 7f97fbfff640 10 monclient: dump: 2024-08-19T12:51:34.797 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.798 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.776+0000 7f9819b72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.776+0000 7f9819b72640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f97f40065c0 con 0x7f97f4002860 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.776+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f9808004300 con 0x7f98140a5920 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.777+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(486..486 src has 350..486) ==== 6206+0+0 (unknown 2867174868 0 0) 0x7f980804e6b0 con 0x7f98140a5920 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.777+0000 7f97fbfff640 10 client.145422.objecter ms_dispatch 0x7f9814000b90 osd_map(486..486 src has 350..486) 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.777+0000 7f97fbfff640 3 client.145422.objecter handle_osd_map got epochs [486,486] > 0 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.777+0000 7f97fbfff640 3 client.145422.objecter handle_osd_map decoding full epoch 486 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.777+0000 7f9813fff640 10 monclient: get_auth_request con 0x7f98000418a0 auth_method 0 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.777+0000 7f97fbfff640 20 client.145422.objecter dump_active .. 0 homeless 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f98137fe640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.104:6790/0 conn(0x7f97f4002860 legacy=0x7f97f4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 10 client.145422.objecter ms_handle_connect 0x7f97f4002860 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 10 client.145422.objecter resend_mon_ops 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2641397222 0 0) 0x7f97f40065c0 con 0x7f97f4002860 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 10 monclient(hunting): my global_id is 154794 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9814142e30 con 0x7f97f4002860 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2343003897 0 0) 0x7f9814142e30 con 0x7f97f4002860 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97f40065c0 con 0x7f97f4002860 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 10 client.145422.objecter ms_handle_connect 0x7f98000418a0 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9804004ec0 con 0x7f97f4002860 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:51:34.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 10 monclient: dump: 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.801 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.802 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3532996989 0 0) 0x7f97f40065c0 con 0x7f97f4002860 2024-08-19T12:51:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97f400a160 con 0x7f97f4002860 2024-08-19T12:51:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f98040072e0 con 0x7f97f4002860 2024-08-19T12:51:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.104:6790/0 conn(0x7f97f4002860 legacy=0x7f97f4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.779+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.104:6790/0 conn(0x7f97f4002860 legacy=0x7f97f4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.786+0000 7f9819b72640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:34.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.786+0000 7f9819b72640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f97f4009e40 con 0x7f97f4009100 2024-08-19T12:51:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f98137fe640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.104:6790/0 conn(0x7f97f4009100 legacy=0x7f97f4009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 10 client.145422.objecter ms_handle_connect 0x7f97f4009100 2024-08-19T12:51:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 10 client.145422.objecter resend_mon_ops 2024-08-19T12:51:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3777480407 0 0) 0x7f97f4009e40 con 0x7f97f4009100 2024-08-19T12:51:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 10 monclient(hunting): my global_id is 154803 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97f40065c0 con 0x7f97f4009100 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 706504598 0 0) 0x7f97f40065c0 con 0x7f97f4009100 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97f4009e40 con 0x7f97f4009100 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f98040058c0 con 0x7f97f4009100 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.788+0000 7f97fbfff640 10 monclient: dump: 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.805 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.806 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.806 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2215088399 0 0) 0x7f97f4009e40 con 0x7f97f4009100 2024-08-19T12:51:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f97f400c3f0 con 0x7f97f4009100 2024-08-19T12:51:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1024596765) 0x7f9804006c20 con 0x7f97f4009100 2024-08-19T12:51:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f97fbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f97fbfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.104:6790/0 conn(0x7f97f4009100 legacy=0x7f97f4009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f97fbfff640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.104:6790/0 conn(0x7f97f4009100 legacy=0x7f97f4009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:34.807 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:34.807 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 406518, 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:34.808 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.809 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.810 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:34.811 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:34.812 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.813 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:34.814 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.815 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:34.819 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:34.821 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.822 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:34.826 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:34.827 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:34.827 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 10 librados: watch_flush enter 2024-08-19T12:51:34.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 10 librados: watch_flush exit 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 20 client.145422.objecter shutdown clearing up homeless session... 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 10 client.145422.objecter successfully canceled tick 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.104:6800/899887506 conn(0x7f98000418a0 legacy=0x7f9800043d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 10 monclient: shutdown 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 1 -- 172.21.15.26:0/1578889501 >> v1:172.21.15.26:6789/0 conn(0x7f98140a5920 legacy=0x7f9814137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9813fff640 1 -- 172.21.15.26:0/1578889501 reap_dead start 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9813fff640 1 -- 172.21.15.26:0/1578889501 reap_dead start 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 1 -- 172.21.15.26:0/1578889501 shutdown_connections 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 1 -- 172.21.15.26:0/1578889501 >> 172.21.15.26:0/1578889501 conn(0x7f98140a8740 msgr2=0x7f98140aa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 1 -- 172.21.15.26:0/1578889501 shutdown_connections 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 1 -- 172.21.15.26:0/1578889501 wait complete. 2024-08-19T12:51:34.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.789+0000 7f9819b72640 1 librados: shutdown 2024-08-19T12:51:34.830 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:34.830 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:51:34.830 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:51:34.865 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.865 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285278767}) 2024-08-19T12:51:34.865 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=140631361}) 2024-08-19T12:51:34.865 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.865 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285278767}) 2024-08-19T12:51:34.865 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=242797217}) 2024-08-19T12:51:34.865 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.865 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285278767}) 2024-08-19T12:51:34.865 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2653262570}) 2024-08-19T12:51:34.865 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.865 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285278767}) 2024-08-19T12:51:34.866 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2836972114}) 2024-08-19T12:51:34.866 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.866 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285278767}) 2024-08-19T12:51:34.866 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3359208561}) 2024-08-19T12:51:34.866 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.866 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285278767}) 2024-08-19T12:51:34.866 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3899515504}) 2024-08-19T12:51:34.866 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.866 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285278767}) 2024-08-19T12:51:34.866 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867899864}) 2024-08-19T12:51:34.866 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.866 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285278767}) 2024-08-19T12:51:34.867 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.863+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3811486563}) 2024-08-19T12:51:34.876 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.876 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622478208}) 2024-08-19T12:51:34.876 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3547218402}) 2024-08-19T12:51:34.876 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.876 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622478208}) 2024-08-19T12:51:34.876 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2039099510}) 2024-08-19T12:51:34.876 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622478208}) 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=818580542}) 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622478208}) 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2583029292}) 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622478208}) 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2295811889}) 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622478208}) 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2600292684}) 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.877 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622478208}) 2024-08-19T12:51:34.878 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1187646726}) 2024-08-19T12:51:34.878 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.878 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622478208}) 2024-08-19T12:51:34.878 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.865+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2771539448}) 2024-08-19T12:51:34.878 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.878 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891983797}) 2024-08-19T12:51:34.878 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2858258743}) 2024-08-19T12:51:34.878 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.878 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891983797}) 2024-08-19T12:51:34.878 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179065145}) 2024-08-19T12:51:34.878 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.878 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891983797}) 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2154328794}) 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891983797}) 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3649060835}) 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891983797}) 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1829807430}) 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891983797}) 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1068510627}) 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891983797}) 2024-08-19T12:51:34.879 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=744931058}) 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891983797}) 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.866+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1616507025}) 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314353516}) 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2375900708}) 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314353516}) 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1951505255}) 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314353516}) 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1158891707}) 2024-08-19T12:51:34.880 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314353516}) 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1208431209}) 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314353516}) 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3672415910}) 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314353516}) 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=283748835}) 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314353516}) 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=16250398}) 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314353516}) 2024-08-19T12:51:34.881 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.868+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2202240488}) 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276197235}) 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3796651466}) 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276197235}) 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=694670544}) 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276197235}) 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3098334829}) 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276197235}) 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3666236541}) 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.882 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276197235}) 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3612802091}) 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276197235}) 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=618647362}) 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276197235}) 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1151167714}) 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276197235}) 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.869+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=773576902}) 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476146631}) 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2644491338}) 2024-08-19T12:51:34.883 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.884 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476146631}) 2024-08-19T12:51:34.884 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4103458164}) 2024-08-19T12:51:34.884 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.884 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476146631}) 2024-08-19T12:51:34.884 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3914750452}) 2024-08-19T12:51:34.884 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.884 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476146631}) 2024-08-19T12:51:34.884 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=703361816}) 2024-08-19T12:51:34.884 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.884 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476146631}) 2024-08-19T12:51:34.884 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=418989197}) 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476146631}) 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1491945193}) 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476146631}) 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3955870741}) 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476146631}) 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.870+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2188681959}) 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.871+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.871+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301512513}) 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.871+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2272518558}) 2024-08-19T12:51:34.885 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.871+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.886 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.871+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301512513}) 2024-08-19T12:51:34.886 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.871+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1187966334}) 2024-08-19T12:51:34.886 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.871+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.886 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301512513}) 2024-08-19T12:51:34.886 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=961058692}) 2024-08-19T12:51:34.886 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.886 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301512513}) 2024-08-19T12:51:34.886 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3500213113}) 2024-08-19T12:51:34.886 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.886 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301512513}) 2024-08-19T12:51:34.886 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4058881336}) 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301512513}) 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2677874123}) 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301512513}) 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3247601819}) 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301512513}) 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.872+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1277070566}) 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146696632}) 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4197370511}) 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.887 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146696632}) 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=385535131}) 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146696632}) 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1325889482}) 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146696632}) 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2192866021}) 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146696632}) 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2045496775}) 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146696632}) 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2890802681}) 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.888 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146696632}) 2024-08-19T12:51:34.889 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1748204042}) 2024-08-19T12:51:34.889 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.889 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146696632}) 2024-08-19T12:51:34.889 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.873+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2906387839}) 2024-08-19T12:51:34.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870040199}) 2024-08-19T12:51:34.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3135264059}) 2024-08-19T12:51:34.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870040199}) 2024-08-19T12:51:34.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=347316206}) 2024-08-19T12:51:34.894 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870040199}) 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3598964741}) 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870040199}) 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=706114426}) 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870040199}) 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=332464755}) 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870040199}) 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1513486931}) 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870040199}) 2024-08-19T12:51:34.895 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=660827920}) 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870040199}) 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.882+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=656202895}) 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.883+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.883+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902011648}) 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.883+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3254671458}) 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.883+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.883+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902011648}) 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.883+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=684696904}) 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.883+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.883+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902011648}) 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1238916276}) 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.896 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902011648}) 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1718309943}) 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902011648}) 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2566645490}) 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902011648}) 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4021756329}) 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902011648}) 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1074684289}) 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902011648}) 2024-08-19T12:51:34.897 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.884+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2501149778}) 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678805454}) 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3737733636}) 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678805454}) 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3564104359}) 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678805454}) 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=471774776}) 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678805454}) 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=193159307}) 2024-08-19T12:51:34.898 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678805454}) 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3211865396}) 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678805454}) 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4069794043}) 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678805454}) 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1092083532}) 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678805454}) 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.885+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=953642436}) 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.899 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447657963}) 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585260265}) 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447657963}) 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3281876155}) 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447657963}) 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3598972157}) 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447657963}) 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=409120368}) 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447657963}) 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1206952697}) 2024-08-19T12:51:34.900 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.901 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447657963}) 2024-08-19T12:51:34.901 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2683066001}) 2024-08-19T12:51:34.901 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.886+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.901 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.887+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447657963}) 2024-08-19T12:51:34.901 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.887+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2104756046}) 2024-08-19T12:51:34.901 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.887+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.901 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.887+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3447657963}) 2024-08-19T12:51:34.901 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.887+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1927685736}) 2024-08-19T12:51:34.925 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.925 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457759872}) 2024-08-19T12:51:34.925 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=693588486}) 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457759872}) 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969980125}) 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457759872}) 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2830339048}) 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457759872}) 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1347654879}) 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457759872}) 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2801081036}) 2024-08-19T12:51:34.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457759872}) 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3210070140}) 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457759872}) 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2090027739}) 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457759872}) 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.899+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4012665465}) 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208063542}) 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273612731}) 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208063542}) 2024-08-19T12:51:34.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=899487302}) 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208063542}) 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1001239298}) 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208063542}) 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1921926313}) 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208063542}) 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2691129528}) 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208063542}) 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1257670115}) 2024-08-19T12:51:34.928 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.900+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.901+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208063542}) 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.901+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1006773703}) 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.901+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.901+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208063542}) 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.901+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3061021289}) 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933515277}) 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1601745047}) 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933515277}) 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1802568839}) 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933515277}) 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2787062944}) 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933515277}) 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3651586279}) 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933515277}) 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2360360963}) 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933515277}) 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1917146450}) 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933515277}) 2024-08-19T12:51:34.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4168688121}) 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933515277}) 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.902+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=177461711}) 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798743029}) 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3911256205}) 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798743029}) 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3875187792}) 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798743029}) 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3309866571}) 2024-08-19T12:51:34.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798743029}) 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=192125240}) 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798743029}) 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2901311306}) 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798743029}) 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=6107949}) 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798743029}) 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2967612165}) 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798743029}) 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.903+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=666592640}) 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521196761}) 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4192500799}) 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521196761}) 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=210970182}) 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521196761}) 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2103881265}) 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521196761}) 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4189461298}) 2024-08-19T12:51:34.933 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.904+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521196761}) 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.905+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1176763493}) 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.905+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.905+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521196761}) 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.905+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=204763078}) 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.905+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.905+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521196761}) 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.905+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1732351095}) 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.905+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.905+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521196761}) 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.905+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=21560845}) 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.934 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892148321}) 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2913999636}) 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892148321}) 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2638294518}) 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892148321}) 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4240654691}) 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892148321}) 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=256415643}) 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892148321}) 2024-08-19T12:51:34.935 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=825005904}) 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892148321}) 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2807689544}) 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892148321}) 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2260242940}) 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892148321}) 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.906+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=161573866}) 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074766231}) 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=674381636}) 2024-08-19T12:51:34.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074766231}) 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1559227166}) 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074766231}) 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=601790527}) 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074766231}) 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1767084228}) 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074766231}) 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=953120077}) 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074766231}) 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4177146669}) 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074766231}) 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3452654867}) 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1074766231}) 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.907+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=53993258}) 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.908+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.908+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850722505}) 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.908+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=488093407}) 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.908+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.908+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850722505}) 2024-08-19T12:51:34.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.908+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1357519287}) 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850722505}) 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3045970950}) 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850722505}) 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=652420374}) 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850722505}) 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3921695293}) 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850722505}) 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4216592337}) 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.939 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850722505}) 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4279215275}) 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850722505}) 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.909+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2008254935}) 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368249746}) 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1802814932}) 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368249746}) 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2640041941}) 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368249746}) 2024-08-19T12:51:34.940 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3063423236}) 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368249746}) 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3809017686}) 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368249746}) 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3532419577}) 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368249746}) 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1357984905}) 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368249746}) 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1529091707}) 2024-08-19T12:51:34.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368249746}) 2024-08-19T12:51:34.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.910+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=456866196}) 2024-08-19T12:51:34.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.911+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.911+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074820795}) 2024-08-19T12:51:34.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.911+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266089142}) 2024-08-19T12:51:34.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.911+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.911+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074820795}) 2024-08-19T12:51:34.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.911+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3604196584}) 2024-08-19T12:51:34.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.911+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074820795}) 2024-08-19T12:51:34.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2132751484}) 2024-08-19T12:51:34.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074820795}) 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1496024044}) 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074820795}) 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3808834019}) 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074820795}) 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=84512580}) 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074820795}) 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=88030059}) 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074820795}) 2024-08-19T12:51:34.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.912+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=340889060}) 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052759597}) 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=978432567}) 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052759597}) 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3823763013}) 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052759597}) 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=296102768}) 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052759597}) 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2919892285}) 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052759597}) 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=38290246}) 2024-08-19T12:51:34.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052759597}) 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3039547274}) 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052759597}) 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=640839382}) 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052759597}) 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.913+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1097084940}) 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.914+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.914+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237921412}) 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.914+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2938281055}) 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.914+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.914+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237921412}) 2024-08-19T12:51:34.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.914+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=452936944}) 2024-08-19T12:51:34.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.914+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.914+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237921412}) 2024-08-19T12:51:34.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.914+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3271420954}) 2024-08-19T12:51:34.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237921412}) 2024-08-19T12:51:34.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4031676879}) 2024-08-19T12:51:34.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237921412}) 2024-08-19T12:51:34.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3511234981}) 2024-08-19T12:51:34.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237921412}) 2024-08-19T12:51:34.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4267813220}) 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237921412}) 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=102734274}) 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237921412}) 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.915+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4057585356}) 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473592026}) 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3523697415}) 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473592026}) 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=61191673}) 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473592026}) 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=173282833}) 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473592026}) 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3906697796}) 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473592026}) 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=606304620}) 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473592026}) 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3494633377}) 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473592026}) 2024-08-19T12:51:34.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3322288688}) 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473592026}) 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.916+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3160470844}) 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502810162}) 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1111984383}) 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502810162}) 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3285323543}) 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502810162}) 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1979070899}) 2024-08-19T12:51:34.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502810162}) 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2061059230}) 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502810162}) 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3230670881}) 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502810162}) 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=800055422}) 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502810162}) 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4248882529}) 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.917+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.918+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502810162}) 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.918+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=541944814}) 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185925002}) 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2331320917}) 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185925002}) 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805774120}) 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185925002}) 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=698660531}) 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185925002}) 2024-08-19T12:51:34.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3243441649}) 2024-08-19T12:51:34.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185925002}) 2024-08-19T12:51:34.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=875577829}) 2024-08-19T12:51:34.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185925002}) 2024-08-19T12:51:34.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4163345808}) 2024-08-19T12:51:34.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185925002}) 2024-08-19T12:51:34.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=611873626}) 2024-08-19T12:51:34.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185925002}) 2024-08-19T12:51:34.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.919+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4234054897}) 2024-08-19T12:51:34.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663972389}) 2024-08-19T12:51:34.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=203158644}) 2024-08-19T12:51:34.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663972389}) 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3655880883}) 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663972389}) 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1204944370}) 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663972389}) 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2028682915}) 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663972389}) 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.928+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=805146391}) 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.961 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663972389}) 2024-08-19T12:51:34.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1171465089}) 2024-08-19T12:51:34.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663972389}) 2024-08-19T12:51:34.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2663888545}) 2024-08-19T12:51:34.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663972389}) 2024-08-19T12:51:34.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.929+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3815456961}) 2024-08-19T12:51:34.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.962 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785243949}) 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=475359551}) 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785243949}) 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=327759880}) 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785243949}) 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=123061869}) 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785243949}) 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2488187162}) 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785243949}) 2024-08-19T12:51:34.963 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3487074010}) 2024-08-19T12:51:34.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785243949}) 2024-08-19T12:51:34.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=871425190}) 2024-08-19T12:51:34.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785243949}) 2024-08-19T12:51:34.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1510811534}) 2024-08-19T12:51:34.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785243949}) 2024-08-19T12:51:34.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.930+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1434522800}) 2024-08-19T12:51:34.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703255871}) 2024-08-19T12:51:34.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1550237918}) 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703255871}) 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972205034}) 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703255871}) 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3239551906}) 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703255871}) 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2367756422}) 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703255871}) 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2389683115}) 2024-08-19T12:51:34.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703255871}) 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4262111836}) 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703255871}) 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=860929685}) 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703255871}) 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.932+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2098093480}) 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824278887}) 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3558420457}) 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824278887}) 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=988177522}) 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824278887}) 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4253376301}) 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824278887}) 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1401753335}) 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824278887}) 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2088462994}) 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824278887}) 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2803420010}) 2024-08-19T12:51:34.967 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824278887}) 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2887662183}) 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824278887}) 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.933+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1161762929}) 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270789044}) 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=842521060}) 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270789044}) 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3474156068}) 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.968 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270789044}) 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=870524098}) 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270789044}) 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=464666031}) 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270789044}) 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3670947831}) 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270789044}) 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4090746095}) 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270789044}) 2024-08-19T12:51:34.969 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4237079736}) 2024-08-19T12:51:34.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270789044}) 2024-08-19T12:51:34.970 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.946+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3122603829}) 2024-08-19T12:51:34.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.946+0000 7f5f543f4640 10 monclient: get_monmap_and_config 2024-08-19T12:51:34.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.946+0000 7f5f543f4640 10 monclient: build_initial_monmap 2024-08-19T12:51:34.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.946+0000 7f5f543f4640 10 monclient: monmap: 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:34.947747+0000 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:34.947747+0000 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:34.971 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:34.972 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:34.972 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3834245381}) 2024-08-19T12:51:34.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2841687092}) 2024-08-19T12:51:34.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3834245381}) 2024-08-19T12:51:34.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3829956548}) 2024-08-19T12:51:34.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3834245381}) 2024-08-19T12:51:34.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3196187909}) 2024-08-19T12:51:34.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.973 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3834245381}) 2024-08-19T12:51:34.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.947+0000 7f5f543f4640 1 Processor -- start 2024-08-19T12:51:34.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2307145995}) 2024-08-19T12:51:34.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3834245381}) 2024-08-19T12:51:34.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3412301020}) 2024-08-19T12:51:34.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.975 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3834245381}) 2024-08-19T12:51:34.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.947+0000 7f5f543f4640 1 -- start start 2024-08-19T12:51:34.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.947+0000 7f5f543f4640 10 monclient: init 2024-08-19T12:51:34.976 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3415480772}) 2024-08-19T12:51:34.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3834245381}) 2024-08-19T12:51:34.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=26172324}) 2024-08-19T12:51:34.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.947+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:34.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3834245381}) 2024-08-19T12:51:34.977 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.948+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1794713537}) 2024-08-19T12:51:34.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.948+0000 7f5f543f4640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.948+0000 7f5f543f4640 10 monclient: _add_conns ranks=[5,8,4,0,7,1,3,6,2] 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.948+0000 7f5f543f4640 10 monclient(hunting): picked mon.noname-h con 0x7f5f4c073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.948+0000 7f5f543f4640 10 monclient(hunting): picked mon.noname-e con 0x7f5f4c0a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.948+0000 7f5f543f4640 10 monclient(hunting): picked mon.noname-c con 0x7f5f4c13f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.948+0000 7f5f543f4640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4c09dae0 con 0x7f5f4c13f2e0 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.948+0000 7f5f543f4640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4c136910 con 0x7f5f4c073470 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.948+0000 7f5f543f4640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4c09dd50 con 0x7f5f4c0a57a0 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.948+0000 7f5f543f4640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.948+0000 7f5f543f4640 10 monclient(hunting): authenticate will time out at 1489.035986s 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.949+0000 7f5f52991640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f5f4c13f2e0 0x7f5f4c1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41704/0 (socket says 172.21.15.26:41704) 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.949+0000 7f5f52991640 1 -- 172.21.15.26:0/157312092 learned_addr learned my addr 172.21.15.26:0/157312092 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5198f640 1 -- 172.21.15.26:0/157312092 >> v1:172.21.15.26:6793/0 conn(0x7f5f4c0a57a0 legacy=0x7f5f4c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f52991640 1 -- 172.21.15.26:0/157312092 >> v1:172.21.15.26:6791/0 conn(0x7f5f4c13f2e0 legacy=0x7f5f4c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 1 -- 172.21.15.26:0/157312092 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2215277343 0 0) 0x7f5f4c09dd50 con 0x7f5f4c0a57a0 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient(hunting): my global_id is 184193 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 1 -- 172.21.15.26:0/157312092 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f38003520 con 0x7f5f4c0a57a0 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 1 -- 172.21.15.26:0/157312092 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2446504058 0 0) 0x7f5f4c09dae0 con 0x7f5f4c13f2e0 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient(hunting): my global_id is 184270 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 1 -- 172.21.15.26:0/157312092 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f4c09dd50 con 0x7f5f4c13f2e0 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 1 -- 172.21.15.26:0/157312092 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1464848587 0 0) 0x7f5f38003520 con 0x7f5f4c0a57a0 2024-08-19T12:51:34.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 1 -- 172.21.15.26:0/157312092 >> v1:172.21.15.104:6791/0 conn(0x7f5f4c073470 legacy=0x7f5f4c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 1 -- 172.21.15.26:0/157312092 >> v1:172.21.15.26:6791/0 conn(0x7f5f4c13f2e0 legacy=0x7f5f4c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient: _finish_hunting 0 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 1 monclient: found mon.noname-e 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 1 -- 172.21.15.26:0/157312092 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4c09e030 con 0x7f5f4c0a57a0 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient: _finish_auth 0 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient: _check_auth_tickets 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 1 -- 172.21.15.26:0/157312092 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5f3c001b30 con 0x7f5f4c0a57a0 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f543f4640 5 monclient: authenticate success, global_id 184193 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f543f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:34.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:51:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient: dump: 2024-08-19T12:51:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.987 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f543f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 1 -- 172.21.15.26:0/157312092 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f3c003610 con 0x7f5f4c0a57a0 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 1 -- 172.21.15.26:0/157312092 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5f3c003b80 con 0x7f5f4c0a57a0 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f543f4640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f543f4640 4 set_mon_vals no callback set 2024-08-19T12:51:34.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:34.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:51:34.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.951+0000 7f5f5118e640 10 monclient: dump: 2024-08-19T12:51:34.989 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.989 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.989 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.989 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.989 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.989 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.989 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.989 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.989 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.989 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.990 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.990 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.990 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.990 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.990 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.990 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.952+0000 7f5f543f4640 10 monclient: shutdown 2024-08-19T12:51:34.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.952+0000 7f5f543f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:34.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.952+0000 7f5f543f4640 1 -- 172.21.15.26:0/157312092 >> v1:172.21.15.26:6793/0 conn(0x7f5f4c0a57a0 legacy=0x7f5f4c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.952+0000 7f5f543f4640 1 -- 172.21.15.26:0/157312092 shutdown_connections 2024-08-19T12:51:34.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.952+0000 7f5f543f4640 1 -- 172.21.15.26:0/157312092 >> 172.21.15.26:0/157312092 conn(0x7f5f4c0abff0 msgr2=0x7f5f4c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:34.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.952+0000 7f5f543f4640 1 -- 172.21.15.26:0/157312092 shutdown_connections 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.952+0000 7f5f543f4640 1 -- 172.21.15.26:0/157312092 wait complete. 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 10 monclient: build_initial_monmap 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 10 monclient: monmap: 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 1 librados: starting msgr at 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 1 librados: starting objecter 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 1 Processor -- start 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 1 -- start start 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 1 librados: setting wanted keys 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 1 librados: calling monclient init 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 10 monclient: init 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:34.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 10 monclient: _add_conns ranks=[2,5,0,8,3,1,6,7,4] 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 10 monclient(hunting): picked mon.noname-b con 0x7f5f4c073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 10 monclient(hunting): picked mon.noname-h con 0x7f5f4c0a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 10 monclient(hunting): picked mon.noname-a con 0x7f5f4c13f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.953+0000 7f5f543f4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4c070bd0 con 0x7f5f4c13f2e0 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.954+0000 7f5f543f4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4c13f0b0 con 0x7f5f4c073470 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.954+0000 7f5f52190640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f5f4c073470 0x7f5f4c06ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54640/0 (socket says 172.21.15.26:54640) 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.954+0000 7f5f52190640 1 -- 172.21.15.26:0/5439198 learned_addr learned my addr 172.21.15.26:0/5439198 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.954+0000 7f5f543f4640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4c071fb0 con 0x7f5f4c0a57a0 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.954+0000 7f5f543f4640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.954+0000 7f5f543f4640 10 monclient(hunting): authenticate will time out at 1489.041074s 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f52190640 1 -- 172.21.15.26:0/5439198 >> v1:172.21.15.26:6790/0 conn(0x7f5f4c073470 legacy=0x7f5f4c06ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 client.?.objecter ms_handle_connect 0x7f5f4c073470 2024-08-19T12:51:34.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f52991640 1 -- 172.21.15.26:0/5439198 >> v1:172.21.15.26:6789/0 conn(0x7f5f4c13f2e0 legacy=0x7f5f4c1da920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 client.?.objecter ms_handle_connect 0x7f5f4c13f2e0 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3901066452 0 0) 0x7f5f480014e0 con 0x7f5f4c073470 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): my global_id is 26993 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f30004eb0 con 0x7f5f4c073470 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1918120838 0 0) 0x7f5f4c070bd0 con 0x7f5f4c13f2e0 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:34.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): my global_id is 145431 2024-08-19T12:51:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f480014e0 con 0x7f5f4c13f2e0 2024-08-19T12:51:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f52190640 1 --1- 172.21.15.26:0/5439198 >> v1:172.21.15.26:6790/0 conn(0x7f5f4c073470 0x7f5f4c06ea50 :-1 s=OPENED pgs=3618 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f5f30004eb0 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:51:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3065309780 0 0) 0x7f5f30004eb0 con 0x7f5f4c073470 2024-08-19T12:51:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f4c070bd0 con 0x7f5f4c073470 2024-08-19T12:51:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 765220299 0 0) 0x7f5f480014e0 con 0x7f5f4c13f2e0 2024-08-19T12:51:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f30004eb0 con 0x7f5f4c13f2e0 2024-08-19T12:51:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5f44002090 con 0x7f5f4c13f2e0 2024-08-19T12:51:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:51:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): dump: 2024-08-19T12:51:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.996 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5f48003930 con 0x7f5f4c073470 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient(hunting): dump: 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:34.997 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:34.998 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3833281831 0 0) 0x7f5f30004eb0 con 0x7f5f4c13f2e0 2024-08-19T12:51:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 >> v1:172.21.15.104:6791/0 conn(0x7f5f4c0a57a0 legacy=0x7f5f4c06f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 >> v1:172.21.15.26:6790/0 conn(0x7f5f4c073470 legacy=0x7f5f4c06ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:34.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 monclient: found mon.a 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4c072290 con 0x7f5f4c13f2e0 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient: _finish_auth 0 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f543f4640 5 monclient: authenticate success, global_id 145431 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3308552248 0 0) 0x7f5f4c070bd0 con 0x7f5f4c073470 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f42ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f543f4640 10 monclient: _renew_subs 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f543f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:34.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.956+0000 7f5f543f4640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4c072850 con 0x7f5f4c13f2e0 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.957+0000 7f5f543f4640 10 client.145431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.957+0000 7f5f543f4640 10 monclient: _renew_subs 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.957+0000 7f5f543f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.957+0000 7f5f543f4640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5f4c06f870 con 0x7f5f4c13f2e0 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.957+0000 7f5f543f4640 1 librados: init done 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.957+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f44004e10 con 0x7f5f4c13f2e0 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.957+0000 7f5f42ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.957+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5f44005300 con 0x7f5f4c13f2e0 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.957+0000 7f5f42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.957+0000 7f5f5098d640 4 set_mon_vals no callback set 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.957+0000 7f5f42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.957+0000 7f5f42ffd640 10 monclient: dump: 2024-08-19T12:51:35.000 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:35.001 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:35.002 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729629342}) 2024-08-19T12:51:35.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074344715}) 2024-08-19T12:51:35.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729629342}) 2024-08-19T12:51:35.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=529194563}) 2024-08-19T12:51:35.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729629342}) 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3518010755}) 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729629342}) 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2690151766}) 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729629342}) 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2415228574}) 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729629342}) 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1356083976}) 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729629342}) 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1288582703}) 2024-08-19T12:51:35.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.957+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.006 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729629342}) 2024-08-19T12:51:35.006 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.958+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=12075030}) 2024-08-19T12:51:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.958+0000 7f5f543f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.959+0000 7f5f543f4640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5f340065c0 con 0x7f5f34002860 2024-08-19T12:51:35.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.959+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5f440054a0 con 0x7f5f4c13f2e0 2024-08-19T12:51:35.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057587699}) 2024-08-19T12:51:35.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=546085480}) 2024-08-19T12:51:35.008 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057587699}) 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1805432929}) 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057587699}) 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2680119175}) 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057587699}) 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3667507593}) 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057587699}) 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=382684609}) 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.009 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057587699}) 2024-08-19T12:51:35.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1274769691}) 2024-08-19T12:51:35.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057587699}) 2024-08-19T12:51:35.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=436738458}) 2024-08-19T12:51:35.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057587699}) 2024-08-19T12:51:35.010 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.959+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2358240016}) 2024-08-19T12:51:35.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.959+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(486..486 src has 350..486) ==== 6206+0+0 (unknown 2867174868 0 0) 0x7f5f4404e670 con 0x7f5f4c13f2e0 2024-08-19T12:51:35.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.959+0000 7f5f42ffd640 10 client.145431.objecter ms_dispatch 0x7f5f4c000b90 osd_map(486..486 src has 350..486) 2024-08-19T12:51:35.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.959+0000 7f5f42ffd640 3 client.145431.objecter handle_osd_map got epochs [486,486] > 0 2024-08-19T12:51:35.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.959+0000 7f5f42ffd640 3 client.145431.objecter handle_osd_map decoding full epoch 486 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.959+0000 7f5f42ffd640 20 client.145431.objecter dump_active .. 0 homeless 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.959+0000 7f5f5198f640 10 monclient: get_auth_request con 0x7f5f30041450 auth_method 0 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.960+0000 7f5f42ffd640 10 client.145431.objecter ms_handle_connect 0x7f5f34002860 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.960+0000 7f5f42ffd640 10 client.145431.objecter resend_mon_ops 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.960+0000 7f5f52190640 1 -- 172.21.15.26:0/5439198 >> v1:172.21.15.104:6791/0 conn(0x7f5f34002860 legacy=0x7f5f34004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2836361305 0 0) 0x7f5f340065c0 con 0x7f5f34002860 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 10 monclient(hunting): my global_id is 184226 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f4c070bd0 con 0x7f5f34002860 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 714065342 0 0) 0x7f5f4c070bd0 con 0x7f5f34002860 2024-08-19T12:51:35.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f340065c0 con 0x7f5f34002860 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5f48004460 con 0x7f5f34002860 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 10 monclient: dump: 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:35.013 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:35.014 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:35.014 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:35.014 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:35.014 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:35.014 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:35.014 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659372202}) 2024-08-19T12:51:35.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=48947817}) 2024-08-19T12:51:35.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659372202}) 2024-08-19T12:51:35.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3777827606 0 0) 0x7f5f340065c0 con 0x7f5f34002860 2024-08-19T12:51:35.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f3400a160 con 0x7f5f34002860 2024-08-19T12:51:35.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=335046927}) 2024-08-19T12:51:35.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659372202}) 2024-08-19T12:51:35.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2501533815}) 2024-08-19T12:51:35.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659372202}) 2024-08-19T12:51:35.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1433065174}) 2024-08-19T12:51:35.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659372202}) 2024-08-19T12:51:35.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3691295226}) 2024-08-19T12:51:35.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659372202}) 2024-08-19T12:51:35.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3542399263}) 2024-08-19T12:51:35.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659372202}) 2024-08-19T12:51:35.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4244596345}) 2024-08-19T12:51:35.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659372202}) 2024-08-19T12:51:35.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:34.961+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2472613804}) 2024-08-19T12:51:35.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5f480077c0 con 0x7f5f34002860 2024-08-19T12:51:35.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:35.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:35.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 >> v1:172.21.15.104:6791/0 conn(0x7f5f34002860 legacy=0x7f5f34004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.961+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 >> v1:172.21.15.104:6791/0 conn(0x7f5f34002860 legacy=0x7f5f34004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.962+0000 7f5f42ffd640 10 client.145431.objecter ms_handle_connect 0x7f5f30041450 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.968+0000 7f5f543f4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.968+0000 7f5f543f4640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5f34009e50 con 0x7f5f34009110 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f52190640 1 -- 172.21.15.26:0/5439198 >> v1:172.21.15.104:6791/0 conn(0x7f5f34009110 legacy=0x7f5f340095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 10 client.145431.objecter ms_handle_connect 0x7f5f34009110 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 10 client.145431.objecter resend_mon_ops 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1474416056 0 0) 0x7f5f34009e50 con 0x7f5f34009110 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 10 monclient(hunting): my global_id is 184235 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f340065c0 con 0x7f5f34009110 2024-08-19T12:51:35.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3638736905 0 0) 0x7f5f340065c0 con 0x7f5f34009110 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f34009e50 con 0x7f5f34009110 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5f48006f60 con 0x7f5f34009110 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 10 monclient: dump: 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:35.022 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1393140159 0 0) 0x7f5f34009e50 con 0x7f5f34009110 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5f3400c3f0 con 0x7f5f34009110 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7543 (unknown 0 0 1461748995) 0x7f5f48007370 con 0x7f5f34009110 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:35.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 >> v1:172.21.15.104:6791/0 conn(0x7f5f34009110 legacy=0x7f5f340095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.971+0000 7f5f42ffd640 1 -- 172.21.15.26:0/5439198 >> v1:172.21.15.104:6791/0 conn(0x7f5f34009110 legacy=0x7f5f340095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:35.024 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 20440, 2024-08-19T12:51:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:35.024 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:35.025 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:35.026 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:35.028 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:35.029 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:35.030 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:35.030 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:35.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.030 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:35.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.030 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:35.030 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:35.030 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:35.031 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.032 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.033 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.035 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.036 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.037 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.039 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:35.040 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.041 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.042 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:35.043 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 10 librados: watch_flush enter 2024-08-19T12:51:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 10 librados: watch_flush exit 2024-08-19T12:51:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 20 client.145431.objecter shutdown clearing up homeless session... 2024-08-19T12:51:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 10 client.145431.objecter successfully canceled tick 2024-08-19T12:51:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 1 -- 172.21.15.26:0/5439198 >> v1:172.21.15.104:6800/899887506 conn(0x7f5f30041450 legacy=0x7f5f30043910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 10 monclient: shutdown 2024-08-19T12:51:35.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:35.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 1 -- 172.21.15.26:0/5439198 >> v1:172.21.15.26:6789/0 conn(0x7f5f4c13f2e0 legacy=0x7f5f4c1da920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f52991640 1 -- 172.21.15.26:0/5439198 reap_dead start 2024-08-19T12:51:35.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f52991640 1 -- 172.21.15.26:0/5439198 reap_dead start 2024-08-19T12:51:35.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 1 -- 172.21.15.26:0/5439198 shutdown_connections 2024-08-19T12:51:35.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 1 -- 172.21.15.26:0/5439198 >> 172.21.15.26:0/5439198 conn(0x7f5f4c0abff0 msgr2=0x7f5f4c0a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:35.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 1 -- 172.21.15.26:0/5439198 shutdown_connections 2024-08-19T12:51:35.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 1 -- 172.21.15.26:0/5439198 wait complete. 2024-08-19T12:51:35.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:34.972+0000 7f5f543f4640 1 librados: shutdown 2024-08-19T12:51:35.046 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:35.046 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:51:35.047 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:51:35.154 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.154 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920732695}) 2024-08-19T12:51:35.154 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3619958149}) 2024-08-19T12:51:35.154 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.154 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920732695}) 2024-08-19T12:51:35.154 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=664656604}) 2024-08-19T12:51:35.154 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.154 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920732695}) 2024-08-19T12:51:35.154 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3613013256}) 2024-08-19T12:51:35.154 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.154 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920732695}) 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1110206072}) 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920732695}) 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1974341616}) 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920732695}) 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1748309957}) 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920732695}) 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1073060896}) 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920732695}) 2024-08-19T12:51:35.155 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.151+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3914963608}) 2024-08-19T12:51:35.186 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.186 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988890615}) 2024-08-19T12:51:35.186 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1425587880}) 2024-08-19T12:51:35.186 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.186 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988890615}) 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=18025038}) 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988890615}) 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1877237964}) 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988890615}) 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3943305545}) 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988890615}) 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2473651031}) 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988890615}) 2024-08-19T12:51:35.187 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3666027048}) 2024-08-19T12:51:35.188 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.188 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988890615}) 2024-08-19T12:51:35.188 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3021257618}) 2024-08-19T12:51:35.188 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.188 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988890615}) 2024-08-19T12:51:35.188 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.184+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1646504754}) 2024-08-19T12:51:35.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.198+0000 7f40b738a640 10 monclient: get_monmap_and_config 2024-08-19T12:51:35.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.198+0000 7f40b738a640 10 monclient: build_initial_monmap 2024-08-19T12:51:35.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.198+0000 7f40b738a640 10 monclient: monmap: 2024-08-19T12:51:35.198 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:35.198871+0000 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:35.198871+0000 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:35.199 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:35.200 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.198+0000 7f40b738a640 1 Processor -- start 2024-08-19T12:51:35.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.198+0000 7f40b738a640 1 -- start start 2024-08-19T12:51:35.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.198+0000 7f40b738a640 10 monclient: init 2024-08-19T12:51:35.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b738a640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:35.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b738a640 10 monclient: _add_conns ranks=[3,4,8,2,7,0,6,1,5] 2024-08-19T12:51:35.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b738a640 10 monclient(hunting): picked mon.noname-g con 0x7f40b0073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:35.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b738a640 10 monclient(hunting): picked mon.noname-c con 0x7f40b00a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:35.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b738a640 10 monclient(hunting): picked mon.noname-e con 0x7f40b013f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:51:35.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b738a640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b009dae0 con 0x7f40b0073470 2024-08-19T12:51:35.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b738a640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b0136910 con 0x7f40b00a57a0 2024-08-19T12:51:35.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b738a640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b009dd50 con 0x7f40b013f2e0 2024-08-19T12:51:35.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b738a640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:35.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b738a640 10 monclient(hunting): authenticate will time out at 1489.286327s 2024-08-19T12:51:35.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b5927640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f40b013f2e0 0x7f40b01416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:45632/0 (socket says 172.21.15.26:45632) 2024-08-19T12:51:35.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b4925640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f40b00a57a0 0x7f40b00a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:41712/0 (socket says 172.21.15.26:41712) 2024-08-19T12:51:35.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.199+0000 7f40b5927640 1 -- 172.21.15.26:0/3416254847 learned_addr learned my addr 172.21.15.26:0/3416254847 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:35.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40b4925640 1 -- 172.21.15.26:0/3416254847 >> v1:172.21.15.26:6791/0 conn(0x7f40b00a57a0 legacy=0x7f40b00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:35.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40b5927640 1 -- 172.21.15.26:0/3416254847 >> v1:172.21.15.26:6793/0 conn(0x7f40b013f2e0 legacy=0x7f40b01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:35.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40b5126640 1 -- 172.21.15.26:0/3416254847 >> v1:172.21.15.104:6790/0 conn(0x7f40b0073470 legacy=0x7f40b0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:35.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2812731711 0 0) 0x7f40b0136910 con 0x7f40b00a57a0 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 10 monclient(hunting): my global_id is 184279 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4098003520 con 0x7f40b00a57a0 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 389911672 0 0) 0x7f40b009dd50 con 0x7f40b013f2e0 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 10 monclient(hunting): my global_id is 184202 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40b0136910 con 0x7f40b013f2e0 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2186515018 0 0) 0x7f40b009dae0 con 0x7f40b0073470 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 10 monclient(hunting): _negotiate 2024-08-19T12:51:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 10 monclient(hunting): my global_id is 154812 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.201+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40b009dd50 con 0x7f40b0073470 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3325578019 0 0) 0x7f40b0136910 con 0x7f40b013f2e0 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 >> v1:172.21.15.26:6791/0 conn(0x7f40b00a57a0 legacy=0x7f40b00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 >> v1:172.21.15.104:6790/0 conn(0x7f40b0073470 legacy=0x7f40b0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 10 monclient: _finish_hunting 0 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 1 monclient: found mon.noname-e 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b009e030 con 0x7f40b013f2e0 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 10 monclient: _finish_auth 0 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 10 monclient: _check_auth_tickets 2024-08-19T12:51:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40b738a640 5 monclient: authenticate success, global_id 184202 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f40a0001dc0 con 0x7f40b013f2e0 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 10 monclient: dump: 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:35.209 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:35.210 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:35.210 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:35.210 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:35.210 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:35.210 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:35.210 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:35.210 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40b738a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:51:35.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40a0002090 con 0x7f40b013f2e0 2024-08-19T12:51:35.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:35.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 1 -- 172.21.15.26:0/3416254847 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f40a0004f60 con 0x7f40b013f2e0 2024-08-19T12:51:35.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40b738a640 10 monclient: get_monmap_and_config success 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40b738a640 4 set_mon_vals no callback set 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40affff640 10 monclient: dump: 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:35.211 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40b738a640 10 monclient: shutdown 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40b738a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40b738a640 1 -- 172.21.15.26:0/3416254847 >> v1:172.21.15.26:6793/0 conn(0x7f40b013f2e0 legacy=0x7f40b01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40b738a640 1 -- 172.21.15.26:0/3416254847 shutdown_connections 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40b738a640 1 -- 172.21.15.26:0/3416254847 >> 172.21.15.26:0/3416254847 conn(0x7f40b00abff0 msgr2=0x7f40b00ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40b738a640 1 -- 172.21.15.26:0/3416254847 shutdown_connections 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.202+0000 7f40b738a640 1 -- 172.21.15.26:0/3416254847 wait complete. 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.203+0000 7f40b738a640 10 monclient: build_initial_monmap 2024-08-19T12:51:35.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.203+0000 7f40b738a640 10 monclient: monmap: 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:35.213 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.203+0000 7f40b738a640 1 librados: starting msgr at 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.203+0000 7f40b738a640 1 librados: starting objecter 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.203+0000 7f40b738a640 1 Processor -- start 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.203+0000 7f40b738a640 1 -- start start 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.203+0000 7f40b738a640 1 librados: setting wanted keys 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.203+0000 7f40b738a640 1 librados: calling monclient init 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.203+0000 7f40b738a640 10 monclient: init 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b738a640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b738a640 10 monclient: _add_conns ranks=[2,6,8,0,4,5,3,7,1] 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b738a640 10 monclient(hunting): picked mon.noname-b con 0x7f40b0073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b738a640 10 monclient(hunting): picked mon.noname-d con 0x7f40b00a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:35.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b738a640 10 monclient(hunting): picked mon.noname-e con 0x7f40b013f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:51:35.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b738a640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b0137010 con 0x7f40b0073470 2024-08-19T12:51:35.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b738a640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b00a3240 con 0x7f40b00a57a0 2024-08-19T12:51:35.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b738a640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b00a3860 con 0x7f40b013f2e0 2024-08-19T12:51:35.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b738a640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:35.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b738a640 10 monclient(hunting): authenticate will time out at 1489.291196s 2024-08-19T12:51:35.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b4925640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f40b00a57a0 0x7f40b00a2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:51850/0 (socket says 172.21.15.26:51850) 2024-08-19T12:51:35.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b5126640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f40b0073470 0x7f40b00a1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:54644/0 (socket says 172.21.15.26:54644) 2024-08-19T12:51:35.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b4925640 1 -- 172.21.15.26:0/1695018364 learned_addr learned my addr 172.21.15.26:0/1695018364 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:35.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.204+0000 7f40b5927640 1 --1- 172.21.15.26:0/1695018364 >> v1:172.21.15.26:6793/0 conn(0x7f40b013f2e0 0x7f40b01d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:45636/0 (socket says 172.21.15.26:45636) 2024-08-19T12:51:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.206+0000 7f40b5927640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.26:6793/0 conn(0x7f40b013f2e0 legacy=0x7f40b01d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.206+0000 7f40adffb640 10 client.?.objecter ms_handle_connect 0x7f40b013f2e0 2024-08-19T12:51:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.206+0000 7f40adffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.206+0000 7f40adffb640 10 client.?.objecter ms_handle_connect 0x7f40b0073470 2024-08-19T12:51:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.206+0000 7f40adffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.206+0000 7f40b4925640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.26:6792/0 conn(0x7f40b00a57a0 legacy=0x7f40b00a2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.206+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:51:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.206+0000 7f40b5126640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.26:6790/0 conn(0x7f40b0073470 legacy=0x7f40b00a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 client.?.objecter ms_handle_connect 0x7f40b00a57a0 2024-08-19T12:51:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:51:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2891609644 0 0) 0x7f40b00a3860 con 0x7f40b013f2e0 2024-08-19T12:51:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): my global_id is 184211 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f409c004eb0 con 0x7f40b013f2e0 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3837494067 0 0) 0x7f40b0137010 con 0x7f40b0073470 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): my global_id is 27002 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40b00a3860 con 0x7f40b0073470 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 89904064 0 0) 0x7f40b00a3240 con 0x7f40b00a57a0 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): my global_id is 164733 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40b0137010 con 0x7f40b00a57a0 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1323495554 0 0) 0x7f40b00a3860 con 0x7f40b0073470 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40b00a3240 con 0x7f40b0073470 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f40a8003610 con 0x7f40b0073470 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): dump: 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1492994000 0 0) 0x7f40b0137010 con 0x7f40b00a57a0 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40b00a3860 con 0x7f40b00a57a0 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3416223701 0 0) 0x7f409c004eb0 con 0x7f40b013f2e0 2024-08-19T12:51:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40b0137010 con 0x7f40b013f2e0 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f40a4001a10 con 0x7f40b00a57a0 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): dump: 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f40a0001da0 con 0x7f40b013f2e0 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.208+0000 7f40adffb640 10 monclient(hunting): dump: 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 244855032 0 0) 0x7f40b00a3240 con 0x7f40b0073470 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.26:6793/0 conn(0x7f40b013f2e0 legacy=0x7f40b01d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.26:6792/0 conn(0x7f40b00a57a0 legacy=0x7f40b00a2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 10 monclient: _finish_hunting 0 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 1 monclient: found mon.b 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b00a5320 con 0x7f40b0073470 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 10 monclient: _finish_auth 0 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 10 monclient: _check_auth_tickets 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40b738a640 5 monclient: authenticate success, global_id 27002 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40b738a640 10 monclient: _renew_subs 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40b738a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40b738a640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f40b00a43c0 con 0x7f40b0073470 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40a8003b10 con 0x7f40b0073470 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f40a8004000 con 0x7f40b0073470 2024-08-19T12:51:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40adffb640 10 monclient: dump: 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40b738a640 10 client.27002.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40b738a640 10 monclient: _renew_subs 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40b738a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40af7fe640 4 set_mon_vals no callback set 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40b738a640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f40b00a2b90 con 0x7f40b0073470 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.209+0000 7f40b738a640 1 librados: init done 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.210+0000 7f40b738a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.210+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f40a80041a0 con 0x7f40b0073470 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.210+0000 7f40b738a640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f40b013edb0 con 0x7f40b0142df0 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.210+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(486..486 src has 350..486) ==== 6206+0+0 (unknown 2867174868 0 0) 0x7f40a804e420 con 0x7f40b0073470 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.210+0000 7f40adffb640 10 client.27002.objecter ms_dispatch 0x7f40b0000b90 osd_map(486..486 src has 350..486) 2024-08-19T12:51:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.210+0000 7f40adffb640 3 client.27002.objecter handle_osd_map got epochs [486,486] > 0 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.210+0000 7f40adffb640 3 client.27002.objecter handle_osd_map decoding full epoch 486 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.210+0000 7f40adffb640 20 client.27002.objecter dump_active .. 0 homeless 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.210+0000 7f40b5927640 10 monclient: get_auth_request con 0x7f409c041a60 auth_method 0 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.212+0000 7f40b4925640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.104:6792/0 conn(0x7f40b0142df0 legacy=0x7f40b009e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.212+0000 7f40adffb640 10 client.27002.objecter ms_handle_connect 0x7f40b0142df0 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.212+0000 7f40adffb640 10 client.27002.objecter resend_mon_ops 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.212+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3144836646 0 0) 0x7f40b013edb0 con 0x7f40b0142df0 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.212+0000 7f40adffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.212+0000 7f40adffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.212+0000 7f40adffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.212+0000 7f40adffb640 10 monclient(hunting): my global_id is 135961 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.212+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40b00a3240 con 0x7f40b0142df0 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.212+0000 7f40adffb640 10 client.27002.objecter ms_handle_connect 0x7f409c041a60 2024-08-19T12:51:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3241743138 0 0) 0x7f40b00a3240 con 0x7f40b0142df0 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40b013edb0 con 0x7f40b0142df0 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f40a4005050 con 0x7f40b0142df0 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 10 monclient: dump: 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 698778179 0 0) 0x7f40b013edb0 con 0x7f40b0142df0 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40b00a2f20 con 0x7f40b0142df0 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f40a4008480 con 0x7f40b0142df0 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.104:6792/0 conn(0x7f40b0142df0 legacy=0x7f40b009e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.213+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.104:6792/0 conn(0x7f40b0142df0 legacy=0x7f40b009e5f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.219+0000 7f40b738a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:51:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.219+0000 7f40b738a640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f40b002e5b0 con 0x7f40b002e1d0 2024-08-19T12:51:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40b4925640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.104:6792/0 conn(0x7f40b002e1d0 legacy=0x7f40b009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 10 client.27002.objecter ms_handle_connect 0x7f40b002e1d0 2024-08-19T12:51:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 10 client.27002.objecter resend_mon_ops 2024-08-19T12:51:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4038183343 0 0) 0x7f40b002e5b0 con 0x7f40b002e1d0 2024-08-19T12:51:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 10 monclient(hunting): _negotiate 2024-08-19T12:51:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:51:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:51:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 10 monclient(hunting): my global_id is 135970 2024-08-19T12:51:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40b013edb0 con 0x7f40b002e1d0 2024-08-19T12:51:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2301290915 0 0) 0x7f40b013edb0 con 0x7f40b002e1d0 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40b002e5b0 con 0x7f40b002e1d0 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f40a40058b0 con 0x7f40b002e1d0 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 10 monclient: dump: 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:51:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2377090872 0 0) 0x7f40b002e5b0 con 0x7f40b002e1d0 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.222+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f40b006c840 con 0x7f40b002e1d0 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 1293988955) 0x7f40a4007c80 con 0x7f40b002e1d0 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40adffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40adffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.104:6792/0 conn(0x7f40b002e1d0 legacy=0x7f40b009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40adffb640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.104:6792/0 conn(0x7f40b002e1d0 legacy=0x7f40b009ed00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:51:35.237 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:51:35.237 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:35.237 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:35.237 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:51:35.237 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 92, 2024-08-19T12:51:35.237 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 665579, 2024-08-19T12:51:35.237 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:51:35.237 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:51:35.237 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:51:35.237 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:51:35.237 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:35.238 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:35.239 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:51:35.240 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:51:35.241 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:35.242 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:51:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:51:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:51:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:51:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:51:35.256 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:51:35.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 10 librados: watch_flush enter 2024-08-19T12:51:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 10 librados: watch_flush exit 2024-08-19T12:51:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 20 client.27002.objecter shutdown clearing up homeless session... 2024-08-19T12:51:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 10 client.27002.objecter successfully canceled tick 2024-08-19T12:51:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.104:6800/899887506 conn(0x7f409c041a60 legacy=0x7f409c043f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 10 monclient: shutdown 2024-08-19T12:51:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:51:35.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 1 -- 172.21.15.26:0/1695018364 >> v1:172.21.15.26:6790/0 conn(0x7f40b0073470 legacy=0x7f40b00a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b5927640 1 -- 172.21.15.26:0/1695018364 reap_dead start 2024-08-19T12:51:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b5927640 1 -- 172.21.15.26:0/1695018364 reap_dead start 2024-08-19T12:51:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 1 -- 172.21.15.26:0/1695018364 shutdown_connections 2024-08-19T12:51:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 1 -- 172.21.15.26:0/1695018364 >> 172.21.15.26:0/1695018364 conn(0x7f40b00abff0 msgr2=0x7f40b01338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:51:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 1 -- 172.21.15.26:0/1695018364 shutdown_connections 2024-08-19T12:51:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 1 -- 172.21.15.26:0/1695018364 wait complete. 2024-08-19T12:51:35.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:35.223+0000 7f40b738a640 1 librados: shutdown 2024-08-19T12:51:35.259 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:51:35.259 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['d', 'g', 'a'] 2024-08-19T12:51:35.260 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['i'] 2024-08-19T12:51:35.260 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-08-19T12:51:35.260 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-08-19T12:51:35.260 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-08-19T12:51:35.260 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:51:35.274 INFO:tasks.ceph.mon.d:Stopped 2024-08-19T12:51:35.275 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-08-19T12:51:35.275 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-08-19T12:51:35.275 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-08-19T12:51:35.275 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268193972}) 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4061591573}) 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268193972}) 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3661656571}) 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268193972}) 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3314600881}) 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268193972}) 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=674051326}) 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268193972}) 2024-08-19T12:51:35.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2061523315}) 2024-08-19T12:51:35.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268193972}) 2024-08-19T12:51:35.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3288759696}) 2024-08-19T12:51:35.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268193972}) 2024-08-19T12:51:35.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4220713894}) 2024-08-19T12:51:35.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:51:35.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4268193972}) 2024-08-19T12:51:35.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:51:35.273+0000 7fc85d06e640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=977311112}) 2024-08-19T12:51:35.287 INFO:tasks.ceph.mon.g:Stopped 2024-08-19T12:51:35.288 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-08-19T12:51:35.288 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-08-19T12:51:35.288 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-19T12:51:35.288 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:51:35.329 INFO:tasks.ceph.mon.a:Stopped 2024-08-19T12:51:35.330 INFO:tasks.mon_thrash:Sending STOP to mon i 2024-08-19T12:51:35.330 INFO:tasks.ceph.mon.i:Sent signal 19 2024-08-19T12:51:35.330 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:51:50.331 INFO:tasks.mon_thrash:Sending CONT to mon i 2024-08-19T12:51:50.332 INFO:tasks.ceph.mon.i:Sent signal 18 2024-08-19T12:51:50.332 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 6 2024-08-19T12:51:50.332 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:51:50.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.441+0000 7f2531108640 10 monclient: get_monmap_and_config 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.441+0000 7f2531108640 10 monclient: build_initial_monmap 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.441+0000 7f2531108640 10 monclient: monmap: 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:51:50.443479+0000 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:51:50.443479+0000 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:51:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:51:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:51:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:51:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:51:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:51:50.445 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:51:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 1 Processor -- start 2024-08-19T12:51:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 1 -- start start 2024-08-19T12:51:50.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 10 monclient: init 2024-08-19T12:51:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 10 monclient: _reopen_session rank -1 2024-08-19T12:51:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 10 monclient: _add_conns ranks=[3,2,1,4,8,6,0,5,7] 2024-08-19T12:51:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 10 monclient(hunting): picked mon.noname-g con 0x7f252c073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 10 monclient(hunting): picked mon.noname-b con 0x7f252c0a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 10 monclient(hunting): picked mon.noname-f con 0x7f252c13f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:51:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c09dae0 con 0x7f252c13f2e0 2024-08-19T12:51:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c136910 con 0x7f252c0a57a0 2024-08-19T12:51:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c09dd50 con 0x7f252c073470 2024-08-19T12:51:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:50.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.442+0000 7f2531108640 10 monclient(hunting): authenticate will time out at 1504.530938s 2024-08-19T12:51:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.443+0000 7f252ad9d640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f252c073470 legacy=0x7f252c136200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:51:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.443+0000 7f252a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f252c0a57a0 0x7f252c0a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:45916/0 (socket says 172.21.15.26:45916) 2024-08-19T12:51:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.443+0000 7f252a59c640 1 -- 172.21.15.26:0/618041003 learned_addr learned my addr 172.21.15.26:0/618041003 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:51:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.445+0000 7f252a59c640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6790/0 conn(0x7f252c0a57a0 legacy=0x7f252c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:50.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.445+0000 7f252b59e640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6789/0 conn(0x7f252c13f2e0 legacy=0x7f252c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:50.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:50.644+0000 7f252ad9d640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6790/0 conn(0x7f252c073470 legacy=0x7f252c136200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:51:51.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:51.045+0000 7f252ad9d640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6790/0 conn(0x7f252c073470 legacy=0x7f252c136200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:51:51.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:51.846+0000 7f252ad9d640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6790/0 conn(0x7f252c073470 legacy=0x7f252c136200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:51:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.442+0000 7f2528d99640 10 monclient(hunting): tick 2024-08-19T12:51:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.442+0000 7f2528d99640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:51:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.442+0000 7f2528d99640 1 monclient(hunting): continuing hunt 2024-08-19T12:51:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.442+0000 7f2528d99640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:51:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.442+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6790/0 conn(0x7f252c073470 legacy=0x7f252c136200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:51:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.442+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6790/0 conn(0x7f252c0a57a0 legacy=0x7f252c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:53.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.442+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6789/0 conn(0x7f252c13f2e0 legacy=0x7f252c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.443+0000 7f2528d99640 10 monclient: _add_conns ranks=[7,5,6,0,8,4] 2024-08-19T12:51:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.443+0000 7f2528d99640 10 monclient(hunting): picked mon.noname-i con 0x7f250c0038c0 addr v1:172.21.15.104:6792/0 2024-08-19T12:51:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.443+0000 7f2528d99640 10 monclient(hunting): picked mon.noname-h con 0x7f250c007550 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.443+0000 7f2528d99640 10 monclient(hunting): picked mon.noname-d con 0x7f250c00b2a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:51:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.443+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c09e030 con 0x7f250c007550 2024-08-19T12:51:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.443+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c09dd50 con 0x7f250c00b2a0 2024-08-19T12:51:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.443+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f250c011110 con 0x7f250c0038c0 2024-08-19T12:51:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.443+0000 7f2528d99640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:53.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.443+0000 7f252b59e640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6792/0 conn(0x7f250c00b2a0 legacy=0x7f250c00d760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:51:53.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.445+0000 7f252a59c640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6791/0 conn(0x7f250c007550 legacy=0x7f250c0099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:53.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.445+0000 7f252ad9d640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6792/0 conn(0x7f250c0038c0 legacy=0x7f250c005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:53.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:53.644+0000 7f252b59e640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6792/0 conn(0x7f250c00b2a0 legacy=0x7f250c00d760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:51:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:54.045+0000 7f252b59e640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6792/0 conn(0x7f250c00b2a0 legacy=0x7f250c00d760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:51:54.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:54.846+0000 7f252b59e640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6792/0 conn(0x7f250c00b2a0 legacy=0x7f250c00d760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:51:56.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 10 monclient(hunting): tick 2024-08-19T12:51:56.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:51:56.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 1 monclient(hunting): continuing hunt 2024-08-19T12:51:56.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:51:56.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6792/0 conn(0x7f250c0038c0 legacy=0x7f250c005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6792/0 conn(0x7f250c00b2a0 legacy=0x7f250c00d760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:51:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6791/0 conn(0x7f250c007550 legacy=0x7f250c0099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 10 monclient: _add_conns ranks=[8,0,4] 2024-08-19T12:51:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f252b59e640 1 -- 172.21.15.26:0/618041003 reap_dead start 2024-08-19T12:51:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 10 monclient(hunting): picked mon.noname-e con 0x7f250c011540 addr v1:172.21.15.26:6793/0 2024-08-19T12:51:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f252b59e640 1 -- 172.21.15.26:0/618041003 reap_dead start 2024-08-19T12:51:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 10 monclient(hunting): picked mon.noname-a con 0x7f250c0064e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:51:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 10 monclient(hunting): picked mon.noname-c con 0x7f250c0068c0 addr v1:172.21.15.26:6791/0 2024-08-19T12:51:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f250c012410 con 0x7f250c0064e0 2024-08-19T12:51:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c09dd50 con 0x7f250c0068c0 2024-08-19T12:51:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f250c006e60 con 0x7f250c011540 2024-08-19T12:51:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f2528d99640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:56.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.444+0000 7f252a59c640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6789/0 conn(0x7f250c0064e0 legacy=0x7f250c00d760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:51:56.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.446+0000 7f252ad9d640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6793/0 conn(0x7f250c011540 legacy=0x7f250c011980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:56.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.446+0000 7f252b59e640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6791/0 conn(0x7f250c0068c0 legacy=0x7f250c005cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:56.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:56.645+0000 7f252a59c640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6789/0 conn(0x7f250c0064e0 legacy=0x7f250c00d760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:51:57.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:57.046+0000 7f252a59c640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6789/0 conn(0x7f250c0064e0 legacy=0x7f250c00d760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:51:57.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:57.847+0000 7f252a59c640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6789/0 conn(0x7f250c0064e0 legacy=0x7f250c00d760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:51:59.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.444+0000 7f2528d99640 10 monclient(hunting): tick 2024-08-19T12:51:59.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.444+0000 7f2528d99640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:51:59.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.444+0000 7f2528d99640 1 monclient(hunting): continuing hunt 2024-08-19T12:51:59.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.444+0000 7f2528d99640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:51:59.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.444+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6793/0 conn(0x7f250c011540 legacy=0x7f250c011980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:59.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.444+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6791/0 conn(0x7f250c0068c0 legacy=0x7f250c005cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:51:59.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.444+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6789/0 conn(0x7f250c0064e0 legacy=0x7f250c00d760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:51:59.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.444+0000 7f2528d99640 10 monclient: _add_conns ranks=[5,3,2,8,4,1,0,7,6] 2024-08-19T12:51:59.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.444+0000 7f2528d99640 10 monclient(hunting): picked mon.noname-h con 0x7f250c00f010 addr v1:172.21.15.104:6791/0 2024-08-19T12:51:59.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.444+0000 7f2528d99640 10 monclient(hunting): picked mon.noname-g con 0x7f250c00f3f0 addr v1:172.21.15.104:6790/0 2024-08-19T12:51:59.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.444+0000 7f2528d99640 10 monclient(hunting): picked mon.noname-b con 0x7f250c018780 addr v1:172.21.15.26:6790/0 2024-08-19T12:51:59.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.445+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f250c0070d0 con 0x7f250c018780 2024-08-19T12:51:59.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.445+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f250c012410 con 0x7f250c00f3f0 2024-08-19T12:51:59.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.445+0000 7f2528d99640 1 -- 172.21.15.26:0/618041003 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f250c01e4b0 con 0x7f250c00f010 2024-08-19T12:51:59.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.445+0000 7f2528d99640 10 monclient(hunting): _renew_subs 2024-08-19T12:51:59.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.445+0000 7f252a59c640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6790/0 conn(0x7f250c00f3f0 legacy=0x7f250c00f820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:51:59.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.447+0000 7f252ad9d640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6791/0 conn(0x7f250c00f010 legacy=0x7f250c009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:59.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.447+0000 7f252b59e640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6790/0 conn(0x7f250c018780 legacy=0x7f250c01ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:51:59.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:51:59.646+0000 7f252a59c640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6790/0 conn(0x7f250c00f3f0 legacy=0x7f250c00f820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:52:00.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.047+0000 7f252a59c640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6790/0 conn(0x7f250c00f3f0 legacy=0x7f250c00f820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:52:00.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.344+0000 7f2529d9b640 1 -- 172.21.15.26:0/618041003 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1984638014 0 0) 0x7f250c01e4b0 con 0x7f250c00f010 2024-08-19T12:52:00.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.344+0000 7f2529d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:52:00.345 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.344+0000 7f2529d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.344+0000 7f2529d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.344+0000 7f2529d9b640 10 monclient(hunting): my global_id is 184244 2024-08-19T12:52:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.344+0000 7f2529d9b640 1 -- 172.21.15.26:0/618041003 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2518003520 con 0x7f250c00f010 2024-08-19T12:52:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.345+0000 7f2529d9b640 1 -- 172.21.15.26:0/618041003 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3485498058 0 0) 0x7f250c0070d0 con 0x7f250c018780 2024-08-19T12:52:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.345+0000 7f2529d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:52:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.345+0000 7f2529d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.345+0000 7f2529d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.345+0000 7f2529d9b640 10 monclient(hunting): my global_id is 27011 2024-08-19T12:52:00.346 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.345+0000 7f2529d9b640 1 -- 172.21.15.26:0/618041003 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f250c01e4b0 con 0x7f250c018780 2024-08-19T12:52:00.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 1 -- 172.21.15.26:0/618041003 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2645798434 0 0) 0x7f250c01e4b0 con 0x7f250c018780 2024-08-19T12:52:00.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6791/0 conn(0x7f250c00f010 legacy=0x7f250c009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:00.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.104:6790/0 conn(0x7f250c00f3f0 legacy=0x7f250c00f820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:52:00.347 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 1 monclient: found mon.noname-b 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 1 -- 172.21.15.26:0/618041003 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f250c01f750 con 0x7f250c018780 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 10 monclient: _finish_auth 0 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 1 -- 172.21.15.26:0/618041003 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2524001e50 con 0x7f250c018780 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2531108640 5 monclient: authenticate success, global_id 27011 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2531108640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f252b59e640 1 -- 172.21.15.26:0/618041003 reap_dead start 2024-08-19T12:52:00.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2529d9b640 10 monclient: dump: 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:00.349 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.346+0000 7f2531108640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.347+0000 7f2529d9b640 1 -- 172.21.15.26:0/618041003 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2524001ff0 con 0x7f250c018780 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.347+0000 7f2529d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.347+0000 7f2529d9b640 1 -- 172.21.15.26:0/618041003 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2524003cd0 con 0x7f250c018780 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.347+0000 7f2529d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.347+0000 7f2529d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.347+0000 7f2529d9b640 10 monclient: dump: 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:00.350 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:00.351 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:00.352 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:00.352 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:00.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.347+0000 7f2531108640 10 monclient: get_monmap_and_config success 2024-08-19T12:52:00.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.347+0000 7f2531108640 4 set_mon_vals no callback set 2024-08-19T12:52:00.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.348+0000 7f2531108640 10 monclient: shutdown 2024-08-19T12:52:00.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.348+0000 7f2531108640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:00.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.348+0000 7f2531108640 1 -- 172.21.15.26:0/618041003 >> v1:172.21.15.26:6790/0 conn(0x7f250c018780 legacy=0x7f250c01ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.348+0000 7f2531108640 1 -- 172.21.15.26:0/618041003 shutdown_connections 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.348+0000 7f2531108640 1 -- 172.21.15.26:0/618041003 >> 172.21.15.26:0/618041003 conn(0x7f252c0abff0 msgr2=0x7f252c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.348+0000 7f2531108640 1 -- 172.21.15.26:0/618041003 shutdown_connections 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.348+0000 7f2531108640 1 -- 172.21.15.26:0/618041003 wait complete. 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.350+0000 7f2531108640 10 monclient: build_initial_monmap 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.350+0000 7f2531108640 10 monclient: monmap: 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:00.355 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.350+0000 7f2531108640 1 librados: starting msgr at 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.350+0000 7f2531108640 1 librados: starting objecter 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.350+0000 7f2531108640 1 Processor -- start 2024-08-19T12:52:00.356 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 1 -- start start 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 1 librados: setting wanted keys 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 1 librados: calling monclient init 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 10 monclient: init 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 10 monclient: _add_conns ranks=[7,3,1,8,2,5,6,4,0] 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 10 monclient(hunting): picked mon.noname-i con 0x7f250c018780 addr v1:172.21.15.104:6792/0 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 10 monclient(hunting): picked mon.noname-g con 0x7f252c13e500 addr v1:172.21.15.104:6790/0 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 10 monclient(hunting): picked mon.noname-f con 0x7f252c1d44b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c13ef40 con 0x7f252c1d44b0 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c13aac0 con 0x7f252c13e500 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c13ada0 con 0x7f250c018780 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f2531108640 10 monclient(hunting): authenticate will time out at 1514.438886s 2024-08-19T12:52:00.357 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.351+0000 7f252a59c640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f252c13e500 legacy=0x7f252c137a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:52:00.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.353+0000 7f252ad9d640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f250c018780 0x7f252c137350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:58030/0 (socket says 172.21.15.26:58030) 2024-08-19T12:52:00.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.353+0000 7f252ad9d640 1 -- 172.21.15.26:0/1338058380 learned_addr learned my addr 172.21.15.26:0/1338058380 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:00.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.355+0000 7f252ad9d640 1 -- 172.21.15.26:0/1338058380 >> v1:172.21.15.104:6792/0 conn(0x7f250c018780 legacy=0x7f252c137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:00.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.355+0000 7f25237fe640 10 client.?.objecter ms_handle_connect 0x7f250c018780 2024-08-19T12:52:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.355+0000 7f25237fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.355+0000 7f25237fe640 10 client.?.objecter ms_handle_connect 0x7f252c1d44b0 2024-08-19T12:52:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.355+0000 7f25237fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.355+0000 7f252b59e640 1 -- 172.21.15.26:0/1338058380 >> v1:172.21.15.104:6789/0 conn(0x7f252c1d44b0 legacy=0x7f252c1d68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.356+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3320510399 0 0) 0x7f250c01f750 con 0x7f252c1d44b0 2024-08-19T12:52:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.356+0000 7f25237fe640 10 monclient(hunting): _negotiate 2024-08-19T12:52:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.356+0000 7f25237fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.356+0000 7f25237fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.356+0000 7f25237fe640 10 monclient(hunting): my global_id is 184231 2024-08-19T12:52:00.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.356+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f250c010ee0 con 0x7f252c1d44b0 2024-08-19T12:52:00.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 795723657 0 0) 0x7f252c09dd50 con 0x7f252c1d44b0 2024-08-19T12:52:00.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f250c01f750 con 0x7f252c1d44b0 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2524003cd0 con 0x7f252c1d44b0 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient(hunting): dump: 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:00.363 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3311779162 0 0) 0x7f252c13ada0 con 0x7f250c018780 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient(hunting): _negotiate 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:00.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient(hunting): my global_id is 135979 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f252c09dd50 con 0x7f250c018780 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2834957591 0 0) 0x7f250c01f750 con 0x7f252c1d44b0 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 >> v1:172.21.15.104:6792/0 conn(0x7f250c018780 legacy=0x7f252c137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 >> v1:172.21.15.104:6790/0 conn(0x7f252c13e500 legacy=0x7f252c137a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient: _finish_hunting 0 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 1 monclient: found mon.f 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252c138170 con 0x7f252c1d44b0 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient: _finish_auth 0 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient: _check_auth_tickets 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:00.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f2531108640 5 monclient: authenticate success, global_id 184231 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f2531108640 10 monclient: _renew_subs 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f2531108640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f2531108640 1 -- 172.21.15.26:0/1338058380 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f252c139880 con 0x7f252c1d44b0 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f2531108640 10 client.184231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f2531108640 10 monclient: _renew_subs 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f2531108640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f2531108640 1 -- 172.21.15.26:0/1338058380 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f252c1d7fc0 con 0x7f252c1d44b0 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f2531108640 1 librados: init done 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2524001ef0 con 0x7f252c1d44b0 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f252959a640 4 set_mon_vals no callback set 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2524004520 con 0x7f252c1d44b0 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:00.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:52:00.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.360+0000 7f25237fe640 10 monclient: dump: 2024-08-19T12:52:00.367 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:00.367 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:00.367 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:00.367 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:00.367 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:00.367 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:00.367 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:00.367 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:00.367 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:00.367 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:00.368 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:00.368 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:00.368 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:00.368 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:00.368 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:00.368 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:00.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.361+0000 7f2531108640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:00.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.361+0000 7f2531108640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:00.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.361+0000 7f2531108640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:52:00.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.361+0000 7f2531108640 1 -- 172.21.15.26:0/1338058380 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2510005080 con 0x7f252c1d44b0 2024-08-19T12:52:00.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.363+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2524011e00 con 0x7f252c1d44b0 2024-08-19T12:52:00.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.363+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(487..487 src has 350..487) ==== 5814+0+0 (unknown 268007112 0 0) 0x7f252404e430 con 0x7f252c1d44b0 2024-08-19T12:52:00.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.363+0000 7f25237fe640 10 client.184231.objecter ms_dispatch 0x7f252c000b90 osd_map(487..487 src has 350..487) 2024-08-19T12:52:00.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.363+0000 7f25237fe640 3 client.184231.objecter handle_osd_map got epochs [487,487] > 0 2024-08-19T12:52:00.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.363+0000 7f25237fe640 3 client.184231.objecter handle_osd_map decoding full epoch 487 2024-08-19T12:52:00.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.363+0000 7f25237fe640 20 client.184231.objecter dump_active .. 0 homeless 2024-08-19T12:52:00.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.363+0000 7f252ad9d640 10 monclient: get_auth_request con 0x7f250c0433e0 auth_method 0 2024-08-19T12:52:00.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.366+0000 7f25237fe640 10 client.184231.objecter ms_handle_connect 0x7f250c0433e0 2024-08-19T12:52:00.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.371+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 <== mon.1 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f250c010ee0 con 0x7f252c1d44b0 2024-08-19T12:52:00.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.371+0000 7f25237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:00.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.371+0000 7f25237fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:52:00.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.645+0000 7f2531108640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:52:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.645+0000 7f2531108640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:52:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.645+0000 7f2531108640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:52:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.645+0000 7f2531108640 1 -- 172.21.15.26:0/1338058380 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2510005370 con 0x7f252c1d44b0 2024-08-19T12:52:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.645+0000 7f25237fe640 1 -- 172.21.15.26:0/1338058380 <== mon.1 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2711 (unknown 254746657 0 1650719980) 0x7f252c13ef40 con 0x7f252c1d44b0 2024-08-19T12:52:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.645+0000 7f25237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:52:00.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.645+0000 7f25237fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:52:00.680 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":98,"quorum":[1,2,4,5,7,8],"quorum_names":["f","b","c","h","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:52:00.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 10 librados: watch_flush enter 2024-08-19T12:52:00.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 10 librados: watch_flush exit 2024-08-19T12:52:00.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 20 client.184231.objecter shutdown clearing up homeless session... 2024-08-19T12:52:00.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 10 client.184231.objecter successfully canceled tick 2024-08-19T12:52:00.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 1 -- 172.21.15.26:0/1338058380 >> v1:172.21.15.104:6800/899887506 conn(0x7f250c0433e0 legacy=0x7f250c0458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:00.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 10 monclient: shutdown 2024-08-19T12:52:00.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:00.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 1 -- 172.21.15.26:0/1338058380 >> v1:172.21.15.104:6789/0 conn(0x7f252c1d44b0 legacy=0x7f252c1d68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:00.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 1 -- 172.21.15.26:0/1338058380 shutdown_connections 2024-08-19T12:52:00.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 1 -- 172.21.15.26:0/1338058380 >> 172.21.15.26:0/1338058380 conn(0x7f252c0abff0 msgr2=0x7f252c0aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:00.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 1 -- 172.21.15.26:0/1338058380 shutdown_connections 2024-08-19T12:52:00.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 1 -- 172.21.15.26:0/1338058380 wait complete. 2024-08-19T12:52:00.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.648+0000 7f2531108640 1 librados: shutdown 2024-08-19T12:52:00.683 INFO:tasks.mon_thrash.ceph_manager:quorum is size 6 2024-08-19T12:52:00.683 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon b 2024-08-19T12:52:00.683 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:52:00.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.782+0000 7f5713eec640 10 monclient: get_monmap_and_config 2024-08-19T12:52:00.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.782+0000 7f5713eec640 10 monclient: build_initial_monmap 2024-08-19T12:52:00.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.782+0000 7f5713eec640 10 monclient: monmap: 2024-08-19T12:52:00.783 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:00.783 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:00.783 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:52:00.783524+0000 2024-08-19T12:52:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:52:00.783524+0000 2024-08-19T12:52:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:00.785 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 1 Processor -- start 2024-08-19T12:52:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 1 -- start start 2024-08-19T12:52:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 10 monclient: init 2024-08-19T12:52:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 10 monclient: _add_conns ranks=[6,0,8,4,7,1,3,2,5] 2024-08-19T12:52:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 10 monclient(hunting): picked mon.noname-d con 0x7f570c06e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:52:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 10 monclient(hunting): picked mon.noname-a con 0x7f570c0a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:52:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 10 monclient(hunting): picked mon.noname-e con 0x7f570c0a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:52:00.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c136790 con 0x7f570c0a5920 2024-08-19T12:52:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c09ddf0 con 0x7f570c06e7b0 2024-08-19T12:52:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c09e0d0 con 0x7f570c0a5d50 2024-08-19T12:52:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5713eec640 10 monclient(hunting): authenticate will time out at 1514.870982s 2024-08-19T12:52:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5711487640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f570c0a5920 legacy=0x7f570c13e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:52:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.783+0000 7f5711c88640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f570c06e7b0 legacy=0x7f570c136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:52:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.784+0000 7f5712489640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f570c0a5d50 0x7f570c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:44420/0 (socket says 172.21.15.26:44420) 2024-08-19T12:52:00.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.784+0000 7f5712489640 1 -- 172.21.15.26:0/4182368084 learned_addr learned my addr 172.21.15.26:0/4182368084 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:00.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5712489640 1 -- 172.21.15.26:0/4182368084 >> v1:172.21.15.26:6793/0 conn(0x7f570c0a5d50 legacy=0x7f570c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 1 -- 172.21.15.26:0/4182368084 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1167170583 0 0) 0x7f570c09e0d0 con 0x7f570c0a5d50 2024-08-19T12:52:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 10 monclient(hunting): _negotiate 2024-08-19T12:52:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 10 monclient(hunting): my global_id is 184220 2024-08-19T12:52:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 1 -- 172.21.15.26:0/4182368084 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56f8003520 con 0x7f570c0a5d50 2024-08-19T12:52:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 1 -- 172.21.15.26:0/4182368084 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 943349774 0 0) 0x7f56f8003520 con 0x7f570c0a5d50 2024-08-19T12:52:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 1 -- 172.21.15.26:0/4182368084 >> v1:172.21.15.26:6792/0 conn(0x7f570c06e7b0 legacy=0x7f570c136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:52:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 1 -- 172.21.15.26:0/4182368084 >> v1:172.21.15.26:6789/0 conn(0x7f570c0a5920 legacy=0x7f570c13e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:52:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 10 monclient: _finish_hunting 0 2024-08-19T12:52:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 1 monclient: found mon.noname-e 2024-08-19T12:52:00.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:52:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 1 -- 172.21.15.26:0/4182368084 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570c09e3b0 con 0x7f570c0a5d50 2024-08-19T12:52:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 10 monclient: _finish_auth 0 2024-08-19T12:52:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 10 monclient: _check_auth_tickets 2024-08-19T12:52:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 1 -- 172.21.15.26:0/4182368084 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5704001e60 con 0x7f570c0a5d50 2024-08-19T12:52:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5713eec640 5 monclient: authenticate success, global_id 184220 2024-08-19T12:52:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5713eec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:52:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5710c86640 10 monclient: dump: 2024-08-19T12:52:00.792 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:00.793 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.786+0000 7f5713eec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5710c86640 1 -- 172.21.15.26:0/4182368084 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5704004c20 con 0x7f570c0a5d50 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5710c86640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5710c86640 1 -- 172.21.15.26:0/4182368084 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5704005190 con 0x7f570c0a5d50 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5713eec640 10 monclient: get_monmap_and_config success 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5713eec640 4 set_mon_vals no callback set 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5710c86640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5710c86640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5710c86640 10 monclient: dump: 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:00.794 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:00.795 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5713eec640 10 monclient: shutdown 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5713eec640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5713eec640 1 -- 172.21.15.26:0/4182368084 >> v1:172.21.15.26:6793/0 conn(0x7f570c0a5d50 legacy=0x7f570c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5713eec640 1 -- 172.21.15.26:0/4182368084 shutdown_connections 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5713eec640 1 -- 172.21.15.26:0/4182368084 >> 172.21.15.26:0/4182368084 conn(0x7f570c0a8740 msgr2=0x7f570c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5713eec640 1 -- 172.21.15.26:0/4182368084 shutdown_connections 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.787+0000 7f5713eec640 1 -- 172.21.15.26:0/4182368084 wait complete. 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.788+0000 7f5713eec640 10 monclient: build_initial_monmap 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.788+0000 7f5713eec640 10 monclient: monmap: 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:52:00.796 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.788+0000 7f5713eec640 1 librados: starting msgr at 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.788+0000 7f5713eec640 1 librados: starting objecter 2024-08-19T12:52:00.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.788+0000 7f5713eec640 1 Processor -- start 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.788+0000 7f5713eec640 1 -- start start 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.788+0000 7f5713eec640 1 librados: setting wanted keys 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.788+0000 7f5713eec640 1 librados: calling monclient init 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.788+0000 7f5713eec640 10 monclient: init 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.788+0000 7f5713eec640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.788+0000 7f5713eec640 10 monclient: _add_conns ranks=[3,6,0,5,1,8,4,7,2] 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.789+0000 7f5713eec640 10 monclient(hunting): picked mon.noname-g con 0x7f570c06e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.789+0000 7f5713eec640 10 monclient(hunting): picked mon.noname-d con 0x7f570c0a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.789+0000 7f5713eec640 10 monclient(hunting): picked mon.noname-a con 0x7f570c0a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.789+0000 7f5713eec640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c0a0a20 con 0x7f570c0a5d50 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.789+0000 7f5711487640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f570c0a5920 legacy=0x7f570c1d2bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.789+0000 7f5713eec640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c1342f0 con 0x7f570c06e7b0 2024-08-19T12:52:00.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.789+0000 7f5713eec640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c09eda0 con 0x7f570c0a5920 2024-08-19T12:52:00.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.789+0000 7f5712489640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f570c0a5d50 legacy=0x7f570c1d62f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:52:00.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.789+0000 7f5713eec640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:00.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.789+0000 7f5713eec640 10 monclient(hunting): authenticate will time out at 1514.876118s 2024-08-19T12:52:00.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.789+0000 7f5711c88640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f570c06e7b0 legacy=0x7f570c0a4f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:52:00.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.990+0000 7f5712489640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f570c0a5d50 legacy=0x7f570c1d62f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:52:00.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.990+0000 7f5711487640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f570c0a5920 legacy=0x7f570c1d2bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:52:00.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:00.990+0000 7f5711c88640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f570c06e7b0 legacy=0x7f570c0a4f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:52:01.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:01.391+0000 7f5711487640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f570c0a5920 legacy=0x7f570c1d2bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:52:01.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:01.391+0000 7f5712489640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f570c0a5d50 legacy=0x7f570c1d62f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:52:01.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:01.391+0000 7f5711c88640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f570c06e7b0 legacy=0x7f570c0a4f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:52:02.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:02.192+0000 7f5712489640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f570c0a5d50 legacy=0x7f570c1d62f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:52:02.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:02.192+0000 7f5711487640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f570c0a5920 legacy=0x7f570c1d2bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:52:02.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:02.192+0000 7f5711c88640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f570c06e7b0 legacy=0x7f570c0a4f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:52:03.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 10 monclient(hunting): tick 2024-08-19T12:52:03.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:52:03.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 1 monclient(hunting): continuing hunt 2024-08-19T12:52:03.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:52:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f570c0a5920 legacy=0x7f570c1d2bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:52:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f570c06e7b0 legacy=0x7f570c0a4f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:52:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f570c0a5d50 legacy=0x7f570c1d62f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:52:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 10 monclient: _add_conns ranks=[7,4,5,1,2,8] 2024-08-19T12:52:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 10 monclient(hunting): picked mon.noname-i con 0x7f56f4003780 addr v1:172.21.15.104:6792/0 2024-08-19T12:52:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f56f4007410 addr v1:172.21.15.26:6791/0 2024-08-19T12:52:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 10 monclient(hunting): picked mon.noname-h con 0x7f56f400b1a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:52:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c0a0740 con 0x7f56f4007410 2024-08-19T12:52:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c0a0a20 con 0x7f56f400b1a0 2024-08-19T12:52:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c1342f0 con 0x7f56f4003780 2024-08-19T12:52:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f56fd7fa640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:03.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f5711487640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f56f4007410 0x7f56f40098d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:55254/0 (socket says 172.21.15.26:55254) 2024-08-19T12:52:03.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.789+0000 7f5711487640 1 -- 172.21.15.26:0/820828016 learned_addr learned my addr 172.21.15.26:0/820828016 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:03.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.791+0000 7f5711487640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.26:6791/0 conn(0x7f56f4007410 legacy=0x7f56f40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:03.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.791+0000 7f56fe7fc640 10 client.?.objecter ms_handle_connect 0x7f56f4007410 2024-08-19T12:52:03.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.791+0000 7f56fe7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:03.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.791+0000 7f5711c88640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.104:6792/0 conn(0x7f56f4003780 legacy=0x7f56f4005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:03.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.791+0000 7f5712489640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.104:6791/0 conn(0x7f56f400b1a0 legacy=0x7f56f400d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:03.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.791+0000 7f56fe7fc640 10 client.?.objecter ms_handle_connect 0x7f56f4003780 2024-08-19T12:52:03.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.791+0000 7f56fe7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:03.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.791+0000 7f56fe7fc640 10 client.?.objecter ms_handle_connect 0x7f56f400b1a0 2024-08-19T12:52:03.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:03.791+0000 7f56fe7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:04.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 838921417 0 0) 0x7f570c0a0740 con 0x7f56f4007410 2024-08-19T12:52:04.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): my global_id is 184297 2024-08-19T12:52:04.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56f0004eb0 con 0x7f56f4007410 2024-08-19T12:52:04.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3400862559 0 0) 0x7f570c0a0a20 con 0x7f56f400b1a0 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): my global_id is 184253 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f570c0a0740 con 0x7f56f400b1a0 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 145072791 0 0) 0x7f570c1342f0 con 0x7f56f4003780 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): my global_id is 135988 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f570c0a0a20 con 0x7f56f4003780 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 231444832 0 0) 0x7f56f0004eb0 con 0x7f56f4007410 2024-08-19T12:52:04.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f570c1342f0 con 0x7f56f4007410 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f57000046c0 con 0x7f56f4007410 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): dump: 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.101 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f5711487640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.26:6791/0 conn(0x7f56f4007410 legacy=0x7f56f40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f5711487640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.26:6791/0 conn(0x7f56f4007410 legacy=0x7f56f40098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f5711487640 1 --1- 172.21.15.26:0/820828016 >> v1:172.21.15.26:6791/0 conn(0x7f56f4007410 0x7f56f40098d0 :-1 s=OPENED pgs=3549 cs=1 l=1).handle_message read tag failed 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f5711487640 1 --1- 172.21.15.26:0/820828016 >> v1:172.21.15.26:6791/0 conn(0x7f56f4007410 0x7f56f40098d0 :-1 s=OPENED pgs=3549 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:52:04.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 128732481 0 0) 0x7f570c0a0740 con 0x7f56f400b1a0 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56f0004eb0 con 0x7f56f400b1a0 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6791/0 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5704001de0 con 0x7f56f400b1a0 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): dump: 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.103 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 370238697 0 0) 0x7f570c0a0a20 con 0x7f56f4003780 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f570c0a0740 con 0x7f56f4003780 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f57080045f0 con 0x7f56f4003780 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:52:04.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient(hunting): dump: 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.105 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 627317723 0 0) 0x7f56f0004eb0 con 0x7f56f400b1a0 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.104:6792/0 conn(0x7f56f4003780 legacy=0x7f56f4005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.26:6791/0 conn(0x7f56f4007410 legacy=0x7f56f40098d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 monclient: found mon.h 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f5712489640 1 -- 172.21.15.26:0/820828016 reap_dead start 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570c09eda0 con 0x7f56f400b1a0 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient: _finish_auth 0 2024-08-19T12:52:04.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f56fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f5713eec640 5 monclient: authenticate success, global_id 184253 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f5713eec640 10 monclient: _renew_subs 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f5713eec640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.093+0000 7f5713eec640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f570c09f440 con 0x7f56f400b1a0 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f5713eec640 10 client.184253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f5713eec640 10 monclient: _renew_subs 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5704005130 con 0x7f56f400b1a0 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f5713eec640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f5713eec640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f570c1ca4c0 con 0x7f56f400b1a0 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f5713eec640 1 librados: init done 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f56fe7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:04.107 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5704005a20 con 0x7f56f400b1a0 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f56fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f56fe7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f56fe7fc640 10 monclient: dump: 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f56fffff640 4 set_mon_vals no callback set 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.094+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5704011e80 con 0x7f56f400b1a0 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.095+0000 7f5713eec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.095+0000 7f5713eec640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f570c0a0180 con 0x7f570c0a4f60 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.095+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(490..490 src has 350..490) ==== 5868+0+0 (unknown 1431477109 0 0) 0x7f570404e520 con 0x7f56f400b1a0 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.095+0000 7f56fe7fc640 10 client.184253.objecter ms_dispatch 0x7f570c000b90 osd_map(490..490 src has 350..490) 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.095+0000 7f56fe7fc640 3 client.184253.objecter handle_osd_map got epochs [490,490] > 0 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.095+0000 7f56fe7fc640 3 client.184253.objecter handle_osd_map decoding full epoch 490 2024-08-19T12:52:04.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.095+0000 7f56fe7fc640 20 client.184253.objecter dump_active .. 0 homeless 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.095+0000 7f5711c88640 10 monclient: get_auth_request con 0x7f56f0041840 auth_method 0 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.097+0000 7f56fe7fc640 10 client.184253.objecter ms_handle_connect 0x7f570c0a4f60 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.097+0000 7f5711487640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.26:6790/0 conn(0x7f570c0a4f60 legacy=0x7f570c1cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.097+0000 7f56fe7fc640 10 client.184253.objecter resend_mon_ops 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.097+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3247636514 0 0) 0x7f570c0a0180 con 0x7f570c0a4f60 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.097+0000 7f56fe7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.097+0000 7f56fe7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.097+0000 7f56fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.097+0000 7f56fe7fc640 10 monclient(hunting): my global_id is 27020 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.097+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56f0004eb0 con 0x7f570c0a4f60 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.097+0000 7f56fe7fc640 10 client.184253.objecter ms_handle_connect 0x7f56f0041840 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.097+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3584828646 0 0) 0x7f56f0004eb0 con 0x7f570c0a4f60 2024-08-19T12:52:04.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.098+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f570c0a0180 con 0x7f570c0a4f60 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.098+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f57000050b0 con 0x7f570c0a4f60 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.098+0000 7f56fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.098+0000 7f56fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.098+0000 7f56fe7fc640 10 monclient: dump: 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.111 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.112 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.112 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.112 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.112 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.112 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.098+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2849173075 0 0) 0x7f570c0a0180 con 0x7f570c0a4f60 2024-08-19T12:52:04.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.098+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f570c134840 con 0x7f570c0a4f60 2024-08-19T12:52:04.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.098+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f57000073f0 con 0x7f570c0a4f60 2024-08-19T12:52:04.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.098+0000 7f56fe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:04.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.098+0000 7f56fe7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:52:04.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.098+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.26:6790/0 conn(0x7f570c0a4f60 legacy=0x7f570c1cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.112 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.098+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.26:6790/0 conn(0x7f570c0a4f60 legacy=0x7f570c1cd840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.105+0000 7f5713eec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:52:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.105+0000 7f5713eec640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f570c1ce660 con 0x7f570c0a5340 2024-08-19T12:52:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 10 client.184253.objecter ms_handle_connect 0x7f570c0a5340 2024-08-19T12:52:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f5711487640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.26:6790/0 conn(0x7f570c0a5340 legacy=0x7f570c1cdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 10 client.184253.objecter resend_mon_ops 2024-08-19T12:52:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 917895351 0 0) 0x7f570c1ce660 con 0x7f570c0a5340 2024-08-19T12:52:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 10 monclient(hunting): my global_id is 27029 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f570c0a0180 con 0x7f570c0a5340 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3502728794 0 0) 0x7f570c0a0180 con 0x7f570c0a5340 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f570c1ce660 con 0x7f570c0a5340 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5700005af0 con 0x7f570c0a5340 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.107+0000 7f56fe7fc640 10 monclient: dump: 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.115 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2484248906 0 0) 0x7f570c1ce660 con 0x7f570c0a5340 2024-08-19T12:52:04.116 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f570c0a5720 con 0x7f570c0a5340 2024-08-19T12:52:04.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7678 (unknown 0 0 4057196722) 0x7f5700006ed0 con 0x7f570c0a5340 2024-08-19T12:52:04.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f56fe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:52:04.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f56fe7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:52:04.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.26:6790/0 conn(0x7f570c0a5340 legacy=0x7f570c1cdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f56fe7fc640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.26:6790/0 conn(0x7f570c0a5340 legacy=0x7f570c1cdf50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.117 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:52:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:52:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:52:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:52:04.117 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 98, 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 837163, 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:52:04.118 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.119 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:52:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:52:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:52:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:52:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:52:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:52:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:52:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:52:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:52:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:52:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:52:04.121 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.122 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.124 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.125 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:52:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:52:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:52:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:52:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:52:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:52:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:52:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:52:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:52:04.132 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:52:04.133 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:52:04.134 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.135 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:52:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:52:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.136 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.137 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:52:04.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:52:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:52:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.139 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:52:04.139 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:52:04.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f5713eec640 10 librados: watch_flush enter 2024-08-19T12:52:04.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f5713eec640 10 librados: watch_flush exit 2024-08-19T12:52:04.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f5713eec640 20 client.184253.objecter shutdown clearing up homeless session... 2024-08-19T12:52:04.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f5713eec640 10 client.184253.objecter successfully canceled tick 2024-08-19T12:52:04.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f5713eec640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.104:6800/899887506 conn(0x7f56f0041840 legacy=0x7f56f0043d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f5713eec640 10 monclient: shutdown 2024-08-19T12:52:04.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f5713eec640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:04.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f5713eec640 1 -- 172.21.15.26:0/820828016 >> v1:172.21.15.104:6791/0 conn(0x7f56f400b1a0 legacy=0x7f56f400d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f5713eec640 1 -- 172.21.15.26:0/820828016 shutdown_connections 2024-08-19T12:52:04.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.108+0000 7f5713eec640 1 -- 172.21.15.26:0/820828016 >> 172.21.15.26:0/820828016 conn(0x7f570c0a8740 msgr2=0x7f570c06d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:04.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.109+0000 7f5713eec640 1 -- 172.21.15.26:0/820828016 shutdown_connections 2024-08-19T12:52:04.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.109+0000 7f5713eec640 1 -- 172.21.15.26:0/820828016 wait complete. 2024-08-19T12:52:04.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.109+0000 7f5713eec640 1 librados: shutdown 2024-08-19T12:52:04.142 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T12:52:04.142 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon c 2024-08-19T12:52:04.143 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:52:04.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.248+0000 7fbf5b6de640 10 monclient: get_monmap_and_config 2024-08-19T12:52:04.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.248+0000 7fbf5b6de640 10 monclient: build_initial_monmap 2024-08-19T12:52:04.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.248+0000 7fbf5b6de640 10 monclient: monmap: 2024-08-19T12:52:04.249 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:04.249 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.249 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:52:04.249361+0000 2024-08-19T12:52:04.249 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:52:04.249361+0000 2024-08-19T12:52:04.250 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:04.250 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.250 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:04.250 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:04.250 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:04.250 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:04.250 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:04.250 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:04.250 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:04.250 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:04.251 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:04.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 1 Processor -- start 2024-08-19T12:52:04.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 1 -- start start 2024-08-19T12:52:04.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 10 monclient: init 2024-08-19T12:52:04.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:04.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 10 monclient: _add_conns ranks=[4,3,2,8,6,5,1,7,0] 2024-08-19T12:52:04.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 10 monclient(hunting): picked mon.noname-c con 0x7fbf5406e790 addr v1:172.21.15.26:6791/0 2024-08-19T12:52:04.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 10 monclient(hunting): picked mon.noname-g con 0x7fbf54139010 addr v1:172.21.15.104:6790/0 2024-08-19T12:52:04.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5413f9d0 addr v1:172.21.15.26:6790/0 2024-08-19T12:52:04.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf5409ddc0 con 0x7fbf5413f9d0 2024-08-19T12:52:04.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf54139450 con 0x7fbf54139010 2024-08-19T12:52:04.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf541396d0 con 0x7fbf5406e790 2024-08-19T12:52:04.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:04.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5b6de640 10 monclient(hunting): authenticate will time out at 1518.336879s 2024-08-19T12:52:04.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5947a640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fbf5406e790 0x7fbf540a58a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:55256/0 (socket says 172.21.15.26:55256) 2024-08-19T12:52:04.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf5947a640 1 -- 172.21.15.26:0/1670297338 learned_addr learned my addr 172.21.15.26:0/1670297338 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:04.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.249+0000 7fbf59c7b640 1 --1- 172.21.15.26:0/1670297338 >> v1:172.21.15.26:6790/0 conn(0x7fbf5413f9d0 0x7fbf54141dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33222/0 (socket says 172.21.15.26:33222) 2024-08-19T12:52:04.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.250+0000 7fbf58c79640 1 -- 172.21.15.26:0/1670297338 >> v1:172.21.15.104:6790/0 conn(0x7fbf54139010 legacy=0x7fbf5413e2b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:52:04.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf5947a640 1 -- 172.21.15.26:0/1670297338 >> v1:172.21.15.26:6791/0 conn(0x7fbf5406e790 legacy=0x7fbf540a58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf59c7b640 1 -- 172.21.15.26:0/1670297338 >> v1:172.21.15.26:6790/0 conn(0x7fbf5413f9d0 legacy=0x7fbf54141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 1 -- 172.21.15.26:0/1670297338 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2777098693 0 0) 0x7fbf5409ddc0 con 0x7fbf5413f9d0 2024-08-19T12:52:04.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient(hunting): my global_id is 27038 2024-08-19T12:52:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 1 -- 172.21.15.26:0/1670297338 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf3c003520 con 0x7fbf5413f9d0 2024-08-19T12:52:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 1 -- 172.21.15.26:0/1670297338 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1464761741 0 0) 0x7fbf541396d0 con 0x7fbf5406e790 2024-08-19T12:52:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient(hunting): my global_id is 184306 2024-08-19T12:52:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 1 -- 172.21.15.26:0/1670297338 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf5409ddc0 con 0x7fbf5406e790 2024-08-19T12:52:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 1 -- 172.21.15.26:0/1670297338 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3757255851 0 0) 0x7fbf3c003520 con 0x7fbf5413f9d0 2024-08-19T12:52:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 1 -- 172.21.15.26:0/1670297338 >> v1:172.21.15.26:6791/0 conn(0x7fbf5406e790 legacy=0x7fbf540a58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 1 -- 172.21.15.26:0/1670297338 >> v1:172.21.15.104:6790/0 conn(0x7fbf54139010 legacy=0x7fbf5413e2b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:52:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient: _finish_hunting 0 2024-08-19T12:52:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 1 monclient: found mon.noname-b 2024-08-19T12:52:04.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:04.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:52:04.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 1 -- 172.21.15.26:0/1670297338 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5409e030 con 0x7fbf5413f9d0 2024-08-19T12:52:04.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient: _finish_auth 0 2024-08-19T12:52:04.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient: _check_auth_tickets 2024-08-19T12:52:04.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:04.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf5b6de640 5 monclient: authenticate success, global_id 27038 2024-08-19T12:52:04.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 1 -- 172.21.15.26:0/1670297338 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbf44001a10 con 0x7fbf5413f9d0 2024-08-19T12:52:04.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf5b6de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:04.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:52:04.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient: dump: 2024-08-19T12:52:04.261 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.261 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.261 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.261 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.261 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.261 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.261 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.261 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.261 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.261 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.261 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.262 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.262 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.262 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.262 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.262 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf5b6de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:04.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 1 -- 172.21.15.26:0/1670297338 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf44001ce0 con 0x7fbf5413f9d0 2024-08-19T12:52:04.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:04.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 1 -- 172.21.15.26:0/1670297338 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbf44004b20 con 0x7fbf5413f9d0 2024-08-19T12:52:04.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf5b6de640 10 monclient: get_monmap_and_config success 2024-08-19T12:52:04.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf5b6de640 4 set_mon_vals no callback set 2024-08-19T12:52:04.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:52:04.263 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.252+0000 7fbf53fff640 10 monclient: dump: 2024-08-19T12:52:04.263 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.263 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.263 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.263 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.263 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.263 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.263 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.263 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.263 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.264 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.264 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.264 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.264 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.264 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.264 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.264 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.253+0000 7fbf5b6de640 10 monclient: shutdown 2024-08-19T12:52:04.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.253+0000 7fbf5b6de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:04.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.253+0000 7fbf5b6de640 1 -- 172.21.15.26:0/1670297338 >> v1:172.21.15.26:6790/0 conn(0x7fbf5413f9d0 legacy=0x7fbf54141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.253+0000 7fbf5b6de640 1 -- 172.21.15.26:0/1670297338 shutdown_connections 2024-08-19T12:52:04.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.253+0000 7fbf5b6de640 1 -- 172.21.15.26:0/1670297338 >> 172.21.15.26:0/1670297338 conn(0x7fbf540a8740 msgr2=0x7fbf540a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:04.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.253+0000 7fbf5b6de640 1 -- 172.21.15.26:0/1670297338 shutdown_connections 2024-08-19T12:52:04.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.253+0000 7fbf5b6de640 1 -- 172.21.15.26:0/1670297338 wait complete. 2024-08-19T12:52:04.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.253+0000 7fbf5b6de640 10 monclient: build_initial_monmap 2024-08-19T12:52:04.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.253+0000 7fbf5b6de640 10 monclient: monmap: 2024-08-19T12:52:04.265 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:04.265 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:52:04.265 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:52:04.266 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:52:04.266 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:04.266 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.266 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:04.266 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:04.266 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:04.266 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:04.266 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:04.266 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:04.266 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:04.266 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:04.267 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:04.267 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 1 librados: starting msgr at 2024-08-19T12:52:04.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 1 librados: starting objecter 2024-08-19T12:52:04.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 1 Processor -- start 2024-08-19T12:52:04.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 1 -- start start 2024-08-19T12:52:04.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 1 librados: setting wanted keys 2024-08-19T12:52:04.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 1 librados: calling monclient init 2024-08-19T12:52:04.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 10 monclient: init 2024-08-19T12:52:04.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:04.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 10 monclient: _add_conns ranks=[6,5,0,3,4,1,7,2,8] 2024-08-19T12:52:04.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 10 monclient(hunting): picked mon.noname-d con 0x7fbf5406e790 addr v1:172.21.15.26:6792/0 2024-08-19T12:52:04.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 10 monclient(hunting): picked mon.noname-h con 0x7fbf54139010 addr v1:172.21.15.104:6791/0 2024-08-19T12:52:04.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5413f9d0 addr v1:172.21.15.26:6789/0 2024-08-19T12:52:04.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf540a0530 con 0x7fbf5413f9d0 2024-08-19T12:52:04.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf5406e2e0 con 0x7fbf54139010 2024-08-19T12:52:04.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5947a640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fbf5406e790 legacy=0x7fbf5409e7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:52:04.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf540a0db0 con 0x7fbf5406e790 2024-08-19T12:52:04.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:04.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf59c7b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fbf5413f9d0 legacy=0x7fbf541d5fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:52:04.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.254+0000 7fbf5b6de640 10 monclient(hunting): authenticate will time out at 1518.341872s 2024-08-19T12:52:04.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.255+0000 7fbf58c79640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7fbf54139010 0x7fbf5409eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:43916/0 (socket says 172.21.15.26:43916) 2024-08-19T12:52:04.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.255+0000 7fbf58c79640 1 -- 172.21.15.26:0/3666537096 learned_addr learned my addr 172.21.15.26:0/3666537096 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf58c79640 1 -- 172.21.15.26:0/3666537096 >> v1:172.21.15.104:6791/0 conn(0x7fbf54139010 legacy=0x7fbf5409eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 client.?.objecter ms_handle_connect 0x7fbf54139010 2024-08-19T12:52:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2311994493 0 0) 0x7fbf48004190 con 0x7fbf54139010 2024-08-19T12:52:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 monclient(hunting): my global_id is 184262 2024-08-19T12:52:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf40004eb0 con 0x7fbf54139010 2024-08-19T12:52:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3489687671 0 0) 0x7fbf40004eb0 con 0x7fbf54139010 2024-08-19T12:52:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf48004190 con 0x7fbf54139010 2024-08-19T12:52:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbf48004c80 con 0x7fbf54139010 2024-08-19T12:52:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:52:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:52:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 monclient(hunting): dump: 2024-08-19T12:52:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.275 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.276 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.276 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.276 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.276 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.276 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.276 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.276 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.276 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.276 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.276 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.276 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.277 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:52:04.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3038646094 0 0) 0x7fbf48004190 con 0x7fbf54139010 2024-08-19T12:52:04.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 >> v1:172.21.15.26:6792/0 conn(0x7fbf5406e790 legacy=0x7fbf5409e7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:52:04.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 >> v1:172.21.15.26:6789/0 conn(0x7fbf5413f9d0 legacy=0x7fbf541d5fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:52:04.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:52:04.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 1 monclient: found mon.h 2024-08-19T12:52:04.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:04.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:52:04.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5409f5f0 con 0x7fbf54139010 2024-08-19T12:52:04.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 monclient: _finish_auth 0 2024-08-19T12:52:04.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:52:04.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:04.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf5b6de640 5 monclient: authenticate success, global_id 184262 2024-08-19T12:52:04.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf5b6de640 10 monclient: _renew_subs 2024-08-19T12:52:04.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf5b6de640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:52:04.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.257+0000 7fbf5b6de640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf540a1090 con 0x7fbf54139010 2024-08-19T12:52:04.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf48004ea0 con 0x7fbf54139010 2024-08-19T12:52:04.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf51ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:04.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbf48005450 con 0x7fbf54139010 2024-08-19T12:52:04.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.278 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf51ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:52:04.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf51ffb640 10 monclient: dump: 2024-08-19T12:52:04.279 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.279 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.279 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.279 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.279 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.279 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.279 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.279 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.279 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.279 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.280 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.280 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.280 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.280 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.280 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.280 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf5b6de640 10 client.184262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:52:04.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf537fe640 4 set_mon_vals no callback set 2024-08-19T12:52:04.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf5b6de640 10 monclient: _renew_subs 2024-08-19T12:52:04.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf5b6de640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:52:04.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf5b6de640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf5413de80 con 0x7fbf54139010 2024-08-19T12:52:04.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf5b6de640 1 librados: init done 2024-08-19T12:52:04.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf5b6de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:04.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf5b6de640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbf540008d0 con 0x7fbf5413a030 2024-08-19T12:52:04.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.258+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fbf480056f0 con 0x7fbf54139010 2024-08-19T12:52:04.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.259+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(490..490 src has 350..490) ==== 5868+0+0 (unknown 1431477109 0 0) 0x7fbf4804ead0 con 0x7fbf54139010 2024-08-19T12:52:04.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.259+0000 7fbf51ffb640 10 client.184262.objecter ms_dispatch 0x7fbf54000b90 osd_map(490..490 src has 350..490) 2024-08-19T12:52:04.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.259+0000 7fbf51ffb640 3 client.184262.objecter handle_osd_map got epochs [490,490] > 0 2024-08-19T12:52:04.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.259+0000 7fbf51ffb640 3 client.184262.objecter handle_osd_map decoding full epoch 490 2024-08-19T12:52:04.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.259+0000 7fbf51ffb640 20 client.184262.objecter dump_active .. 0 homeless 2024-08-19T12:52:04.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.259+0000 7fbf59c7b640 10 monclient: get_auth_request con 0x7fbf40040d70 auth_method 0 2024-08-19T12:52:04.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 10 client.184262.objecter ms_handle_connect 0x7fbf5413a030 2024-08-19T12:52:04.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 10 client.184262.objecter resend_mon_ops 2024-08-19T12:52:04.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf5947a640 1 -- 172.21.15.26:0/3666537096 >> v1:172.21.15.26:6791/0 conn(0x7fbf5413a030 legacy=0x7fbf5413a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 479259913 0 0) 0x7fbf540008d0 con 0x7fbf5413a030 2024-08-19T12:52:04.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 10 monclient(hunting): my global_id is 184315 2024-08-19T12:52:04.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf48004190 con 0x7fbf5413a030 2024-08-19T12:52:04.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2527784876 0 0) 0x7fbf48004190 con 0x7fbf5413a030 2024-08-19T12:52:04.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf540008d0 con 0x7fbf5413a030 2024-08-19T12:52:04.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbf4c0019b0 con 0x7fbf5413a030 2024-08-19T12:52:04.282 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:52:04.283 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 10 monclient: dump: 2024-08-19T12:52:04.283 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.283 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.283 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.283 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.283 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.283 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.283 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.283 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.283 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.284 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.284 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.284 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.284 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.284 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.284 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.284 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4115254545 0 0) 0x7fbf540008d0 con 0x7fbf5413a030 2024-08-19T12:52:04.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf5406df20 con 0x7fbf5413a030 2024-08-19T12:52:04.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 10 client.184262.objecter ms_handle_connect 0x7fbf40040d70 2024-08-19T12:52:04.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbf4c001970 con 0x7fbf5413a030 2024-08-19T12:52:04.284 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:52:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 >> v1:172.21.15.26:6791/0 conn(0x7fbf5413a030 legacy=0x7fbf5413a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.261+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 >> v1:172.21.15.26:6791/0 conn(0x7fbf5413a030 legacy=0x7fbf5413a410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.268+0000 7fbf5b6de640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:52:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.268+0000 7fbf5b6de640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbf5413af00 con 0x7fbf5413ab20 2024-08-19T12:52:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 10 client.184262.objecter ms_handle_connect 0x7fbf5413ab20 2024-08-19T12:52:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 10 client.184262.objecter resend_mon_ops 2024-08-19T12:52:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf5947a640 1 -- 172.21.15.26:0/3666537096 >> v1:172.21.15.26:6791/0 conn(0x7fbf5413ab20 legacy=0x7fbf541db700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3712550354 0 0) 0x7fbf5413af00 con 0x7fbf5413ab20 2024-08-19T12:52:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.285 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 10 monclient(hunting): my global_id is 184324 2024-08-19T12:52:04.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf540008d0 con 0x7fbf5413ab20 2024-08-19T12:52:04.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 189831597 0 0) 0x7fbf540008d0 con 0x7fbf5413ab20 2024-08-19T12:52:04.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf5413af00 con 0x7fbf5413ab20 2024-08-19T12:52:04.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbf4c005050 con 0x7fbf5413ab20 2024-08-19T12:52:04.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:52:04.286 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 10 monclient: dump: 2024-08-19T12:52:04.286 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.286 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.286 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.288 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4232260477 0 0) 0x7fbf5413af00 con 0x7fbf5413ab20 2024-08-19T12:52:04.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.270+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbf541351c0 con 0x7fbf5413ab20 2024-08-19T12:52:04.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7844 (unknown 0 0 2215847092) 0x7fbf4c0073b0 con 0x7fbf5413ab20 2024-08-19T12:52:04.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf51ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:52:04.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf51ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:52:04.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 >> v1:172.21.15.26:6791/0 conn(0x7fbf5413ab20 legacy=0x7fbf541db700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.288 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf51ffb640 1 -- 172.21.15.26:0/3666537096 >> v1:172.21.15.26:6791/0 conn(0x7fbf5413ab20 legacy=0x7fbf541db700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.288 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:52:04.289 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:52:04.289 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:52:04.289 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:52:04.289 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 98, 2024-08-19T12:52:04.289 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 49736, 2024-08-19T12:52:04.289 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:52:04.289 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:52:04.289 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:52:04.289 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:52:04.289 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:52:04.289 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:52:04.290 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:52:04.290 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.290 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:52:04.290 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:52:04.290 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:52:04.290 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:52:04.290 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.290 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.290 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.290 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.290 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.291 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.291 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.291 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.291 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.291 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.291 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.291 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.291 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:52:04.291 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:52:04.291 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.291 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.291 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.292 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.292 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.292 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.292 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.292 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.292 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.292 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.292 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.292 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.292 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:52:04.293 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:52:04.293 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:52:04.293 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:52:04.293 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:52:04.293 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:52:04.293 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:52:04.293 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:52:04.293 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:52:04.293 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:52:04.293 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:52:04.293 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:52:04.293 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:52:04.294 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:52:04.294 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:52:04.294 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:52:04.294 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:52:04.294 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.294 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.294 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.294 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.294 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.294 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.294 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.295 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.295 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.295 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.295 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.295 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.295 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:52:04.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.295 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:52:04.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.295 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:52:04.295 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:52:04.296 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.296 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.296 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.296 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:52:04.296 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.296 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.296 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:52:04.296 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:52:04.296 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.297 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.297 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.297 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:52:04.297 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:52:04.297 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.297 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.297 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.297 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:52:04.298 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.298 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.298 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.298 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:52:04.298 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:52:04.298 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.298 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.298 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.299 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:52:04.299 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:52:04.299 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.299 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.299 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.299 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:52:04.299 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.299 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.299 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:52:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:52:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:52:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:52:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:52:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:52:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:52:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.302 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:52:04.302 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:52:04.302 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.302 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.302 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.302 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:52:04.302 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.302 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.303 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:52:04.303 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:52:04.303 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.303 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.303 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.303 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:52:04.303 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:52:04.303 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.303 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.304 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.304 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:52:04.304 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.304 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.304 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.304 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:52:04.304 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:52:04.304 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.304 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.304 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.305 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:52:04.305 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:52:04.305 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.305 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.305 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.305 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:52:04.305 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:52:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:52:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:52:04.306 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:52:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:52:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:52:04.307 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:52:04.308 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.308 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.308 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.308 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:52:04.308 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:52:04.308 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.308 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.308 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.309 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:52:04.309 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.309 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.309 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.309 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:52:04.309 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:52:04.309 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.309 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.309 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.309 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.310 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.310 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:52:04.310 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:52:04.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.310 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.310 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.310 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:52:04.310 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.310 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.310 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:52:04.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.311 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.311 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.311 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:52:04.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.311 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.311 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:52:04.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.311 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.311 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.311 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:52:04.312 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.312 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.312 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:52:04.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.312 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.312 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.312 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:52:04.312 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.312 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.312 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:52:04.312 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:52:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 10 librados: watch_flush enter 2024-08-19T12:52:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 10 librados: watch_flush exit 2024-08-19T12:52:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 20 client.184262.objecter shutdown clearing up homeless session... 2024-08-19T12:52:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 10 client.184262.objecter successfully canceled tick 2024-08-19T12:52:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 1 -- 172.21.15.26:0/3666537096 >> v1:172.21.15.104:6800/899887506 conn(0x7fbf40040d70 legacy=0x7fbf40043230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 10 monclient: shutdown 2024-08-19T12:52:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 1 -- 172.21.15.26:0/3666537096 >> v1:172.21.15.104:6791/0 conn(0x7fbf54139010 legacy=0x7fbf5409eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf59c7b640 1 -- 172.21.15.26:0/3666537096 reap_dead start 2024-08-19T12:52:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf59c7b640 1 -- 172.21.15.26:0/3666537096 reap_dead start 2024-08-19T12:52:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 1 -- 172.21.15.26:0/3666537096 shutdown_connections 2024-08-19T12:52:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 1 -- 172.21.15.26:0/3666537096 >> 172.21.15.26:0/3666537096 conn(0x7fbf540a8740 msgr2=0x7fbf54135910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 1 -- 172.21.15.26:0/3666537096 shutdown_connections 2024-08-19T12:52:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 1 -- 172.21.15.26:0/3666537096 wait complete. 2024-08-19T12:52:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.271+0000 7fbf5b6de640 1 librados: shutdown 2024-08-19T12:52:04.316 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T12:52:04.316 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon e 2024-08-19T12:52:04.316 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:52:04.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.416+0000 7fee1cb98640 10 monclient: get_monmap_and_config 2024-08-19T12:52:04.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.416+0000 7fee1cb98640 10 monclient: build_initial_monmap 2024-08-19T12:52:04.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.416+0000 7fee1cb98640 10 monclient: monmap: 2024-08-19T12:52:04.417 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:52:04.417109+0000 2024-08-19T12:52:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:52:04.417109+0000 2024-08-19T12:52:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:04.418 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:04.419 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:04.419 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:04.419 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.416+0000 7fee1cb98640 1 Processor -- start 2024-08-19T12:52:04.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1cb98640 1 -- start start 2024-08-19T12:52:04.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1cb98640 10 monclient: init 2024-08-19T12:52:04.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1cb98640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:04.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1cb98640 10 monclient: _add_conns ranks=[6,0,5,2,7,3,1,8,4] 2024-08-19T12:52:04.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1cb98640 10 monclient(hunting): picked mon.noname-d con 0x7fee18073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:52:04.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1cb98640 10 monclient(hunting): picked mon.noname-a con 0x7fee180a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:52:04.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1cb98640 10 monclient(hunting): picked mon.noname-h con 0x7fee1813f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:52:04.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1cb98640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee1809dae0 con 0x7fee180a57a0 2024-08-19T12:52:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1cb98640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee18136910 con 0x7fee1813f2e0 2024-08-19T12:52:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1cb98640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee1809dd50 con 0x7fee18073470 2024-08-19T12:52:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1cb98640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1cb98640 10 monclient(hunting): authenticate will time out at 1518.504559s 2024-08-19T12:52:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee0ffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fee180a57a0 legacy=0x7fee180a5b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:52:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee1659c640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fee18073470 legacy=0x7fee18136200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:52:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee16d9d640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7fee1813f2e0 0x7fee181416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:43928/0 (socket says 172.21.15.26:43928) 2024-08-19T12:52:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.417+0000 7fee16d9d640 1 -- 172.21.15.26:0/2788853330 learned_addr learned my addr 172.21.15.26:0/2788853330 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee16d9d640 1 -- 172.21.15.26:0/2788853330 >> v1:172.21.15.104:6791/0 conn(0x7fee1813f2e0 legacy=0x7fee181416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 1 -- 172.21.15.26:0/2788853330 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 36984053 0 0) 0x7fee18136910 con 0x7fee1813f2e0 2024-08-19T12:52:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.426 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient(hunting): my global_id is 184271 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 1 -- 172.21.15.26:0/2788853330 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee04003520 con 0x7fee1813f2e0 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 1 -- 172.21.15.26:0/2788853330 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 712288660 0 0) 0x7fee04003520 con 0x7fee1813f2e0 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 1 -- 172.21.15.26:0/2788853330 >> v1:172.21.15.26:6792/0 conn(0x7fee18073470 legacy=0x7fee18136200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 1 -- 172.21.15.26:0/2788853330 >> v1:172.21.15.26:6789/0 conn(0x7fee180a57a0 legacy=0x7fee180a5b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 1 monclient: found mon.noname-h 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 1 -- 172.21.15.26:0/2788853330 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee1809e030 con 0x7fee1813f2e0 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient: _finish_auth 0 2024-08-19T12:52:04.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 1 -- 172.21.15.26:0/2788853330 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee10001f20 con 0x7fee1813f2e0 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee1cb98640 5 monclient: authenticate success, global_id 184271 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee1cb98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient: dump: 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.428 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee1cb98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 1 -- 172.21.15.26:0/2788853330 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee10004c20 con 0x7fee1813f2e0 2024-08-19T12:52:04.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee1cb98640 10 monclient: get_monmap_and_config success 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee1cb98640 4 set_mon_vals no callback set 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 1 -- 172.21.15.26:0/2788853330 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee10005110 con 0x7fee1813f2e0 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.420+0000 7fee15d9b640 10 monclient: dump: 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.430 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.421+0000 7fee1cb98640 10 monclient: shutdown 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.421+0000 7fee1cb98640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.421+0000 7fee1cb98640 1 -- 172.21.15.26:0/2788853330 >> v1:172.21.15.104:6791/0 conn(0x7fee1813f2e0 legacy=0x7fee181416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.421+0000 7fee1cb98640 1 -- 172.21.15.26:0/2788853330 shutdown_connections 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.421+0000 7fee1cb98640 1 -- 172.21.15.26:0/2788853330 >> 172.21.15.26:0/2788853330 conn(0x7fee180abff0 msgr2=0x7fee180ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.421+0000 7fee1cb98640 1 -- 172.21.15.26:0/2788853330 shutdown_connections 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.421+0000 7fee1cb98640 1 -- 172.21.15.26:0/2788853330 wait complete. 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 10 monclient: build_initial_monmap 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 10 monclient: monmap: 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:04.432 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 1 librados: starting msgr at 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 1 librados: starting objecter 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 1 Processor -- start 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 1 -- start start 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 1 librados: setting wanted keys 2024-08-19T12:52:04.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 1 librados: calling monclient init 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 10 monclient: init 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 10 monclient: _add_conns ranks=[7,0,8,5,4,3,2,1,6] 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 10 monclient(hunting): picked mon.noname-i con 0x7fee18073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 10 monclient(hunting): picked mon.noname-a con 0x7fee180a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 10 monclient(hunting): picked mon.noname-e con 0x7fee1813f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee18142b80 con 0x7fee180a57a0 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee181308e0 con 0x7fee18073470 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee0ffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fee180a57a0 legacy=0x7fee181301d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee181323a0 con 0x7fee1813f2e0 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.422+0000 7fee1cb98640 10 monclient(hunting): authenticate will time out at 1518.509969s 2024-08-19T12:52:04.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.423+0000 7fee16d9d640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fee1813f2e0 0x7fee181d61a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:44426/0 (socket says 172.21.15.26:44426) 2024-08-19T12:52:04.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.423+0000 7fee16d9d640 1 -- 172.21.15.26:0/2986863743 learned_addr learned my addr 172.21.15.26:0/2986863743 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee16d9d640 1 -- 172.21.15.26:0/2986863743 >> v1:172.21.15.26:6793/0 conn(0x7fee1813f2e0 legacy=0x7fee181d61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 client.?.objecter ms_handle_connect 0x7fee1813f2e0 2024-08-19T12:52:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 223350440 0 0) 0x7fee181323a0 con 0x7fee1813f2e0 2024-08-19T12:52:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient(hunting): my global_id is 184229 2024-08-19T12:52:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedf8004eb0 con 0x7fee1813f2e0 2024-08-19T12:52:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 client.?.objecter ms_handle_connect 0x7fee18073470 2024-08-19T12:52:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee1659c640 1 -- 172.21.15.26:0/2986863743 >> v1:172.21.15.104:6792/0 conn(0x7fee18073470 legacy=0x7fee1812fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 54140556 0 0) 0x7fedf8004eb0 con 0x7fee1813f2e0 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee181323a0 con 0x7fee1813f2e0 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 295036173 0 0) 0x7fee181308e0 con 0x7fee18073470 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient(hunting): my global_id is 135997 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedf8004eb0 con 0x7fee18073470 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee10001ee0 con 0x7fee1813f2e0 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient(hunting): dump: 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.442 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.443 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1062052848 0 0) 0x7fee181323a0 con 0x7fee1813f2e0 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 >> v1:172.21.15.104:6792/0 conn(0x7fee18073470 legacy=0x7fee1812fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 >> v1:172.21.15.26:6789/0 conn(0x7fee180a57a0 legacy=0x7fee181301d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient: _finish_hunting 0 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 monclient: found mon.e 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee180a5280 con 0x7fee1813f2e0 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient: _finish_auth 0 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient: _check_auth_tickets 2024-08-19T12:52:04.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee1cb98640 5 monclient: authenticate success, global_id 184229 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee1cb98640 10 monclient: _renew_subs 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee1cb98640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee1cb98640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fee180a5510 con 0x7fee1813f2e0 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee1cb98640 10 client.184229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee1cb98640 10 monclient: _renew_subs 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee1cb98640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee1cb98640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fee181d7aa0 con 0x7fee1813f2e0 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee1cb98640 1 librados: init done 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee100016a0 con 0x7fee1813f2e0 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.425+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee10005950 con 0x7fee1813f2e0 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.426+0000 7fee0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.426+0000 7fee1559a640 4 set_mon_vals no callback set 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.426+0000 7fee0effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.426+0000 7fee0effd640 10 monclient: dump: 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.446 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.426+0000 7fee1cb98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.426+0000 7fee1cb98640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fedfc0065c0 con 0x7fedfc002860 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.426+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fee10011db0 con 0x7fee1813f2e0 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.426+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(490..490 src has 350..490) ==== 5868+0+0 (unknown 1431477109 0 0) 0x7fee1004e5d0 con 0x7fee1813f2e0 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.426+0000 7fee0effd640 10 client.184229.objecter ms_dispatch 0x7fee18000b90 osd_map(490..490 src has 350..490) 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.426+0000 7fee0effd640 3 client.184229.objecter handle_osd_map got epochs [490,490] > 0 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.426+0000 7fee0effd640 3 client.184229.objecter handle_osd_map decoding full epoch 490 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.427+0000 7fee0effd640 20 client.184229.objecter dump_active .. 0 homeless 2024-08-19T12:52:04.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.427+0000 7fee0ffff640 10 monclient: get_auth_request con 0x7fedf8040f90 auth_method 0 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.428+0000 7fee0effd640 10 client.184229.objecter ms_handle_connect 0x7fedfc002860 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.428+0000 7fee0effd640 10 client.184229.objecter resend_mon_ops 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.428+0000 7fee1659c640 1 -- 172.21.15.26:0/2986863743 >> v1:172.21.15.26:6793/0 conn(0x7fedfc002860 legacy=0x7fedfc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 417861084 0 0) 0x7fedfc0065c0 con 0x7fedfc002860 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 10 monclient(hunting): my global_id is 184238 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee181323a0 con 0x7fedfc002860 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 416703102 0 0) 0x7fee181323a0 con 0x7fedfc002860 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedfc0065c0 con 0x7fedfc002860 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee080044f0 con 0x7fedfc002860 2024-08-19T12:52:04.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 10 monclient: dump: 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.449 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 530577359 0 0) 0x7fedfc0065c0 con 0x7fedfc002860 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fedfc00a160 con 0x7fedfc002860 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 10 client.184229.objecter ms_handle_connect 0x7fedf8040f90 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fee080059e0 con 0x7fedfc002860 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 >> v1:172.21.15.26:6793/0 conn(0x7fedfc002860 legacy=0x7fedfc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.429+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 >> v1:172.21.15.26:6793/0 conn(0x7fedfc002860 legacy=0x7fedfc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.435+0000 7fee1cb98640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.435+0000 7fee1cb98640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fedfc009e70 con 0x7fedfc009130 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee1659c640 1 -- 172.21.15.26:0/2986863743 >> v1:172.21.15.26:6793/0 conn(0x7fedfc009130 legacy=0x7fedfc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 10 client.184229.objecter ms_handle_connect 0x7fedfc009130 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 10 client.184229.objecter resend_mon_ops 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2965053076 0 0) 0x7fedfc009e70 con 0x7fedfc009130 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 10 monclient(hunting): my global_id is 184247 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedfc0065c0 con 0x7fedfc009130 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3325068404 0 0) 0x7fedfc0065c0 con 0x7fedfc009130 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedfc009e70 con 0x7fedfc009130 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee08004d40 con 0x7fedfc009130 2024-08-19T12:52:04.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 10 monclient: dump: 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.452 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.453 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2765775870 0 0) 0x7fedfc009e70 con 0x7fedfc009130 2024-08-19T12:52:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fedfc00c410 con 0x7fedfc009130 2024-08-19T12:52:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7677 (unknown 0 0 2436198989) 0x7fee08005020 con 0x7fedfc009130 2024-08-19T12:52:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:52:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:52:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 >> v1:172.21.15.26:6793/0 conn(0x7fedfc009130 legacy=0x7fedfc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.438+0000 7fee0effd640 1 -- 172.21.15.26:0/2986863743 >> v1:172.21.15.26:6793/0 conn(0x7fedfc009130 legacy=0x7fedfc0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.454 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:52:04.454 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:52:04.454 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:52:04.454 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:52:04.454 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 98, 2024-08-19T12:52:04.454 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 49905, 2024-08-19T12:52:04.454 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:52:04.454 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:52:04.454 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:52:04.454 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.455 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.456 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:52:04.457 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:52:04.458 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:52:04.459 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.459 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.459 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.459 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.459 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.459 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.459 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.459 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.459 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.459 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.459 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.459 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.460 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:52:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:52:04.467 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:52:04.468 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.469 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.472 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.473 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:52:04.474 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.474 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.474 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:52:04.474 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:52:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 10 librados: watch_flush enter 2024-08-19T12:52:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 10 librados: watch_flush exit 2024-08-19T12:52:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 20 client.184229.objecter shutdown clearing up homeless session... 2024-08-19T12:52:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 10 client.184229.objecter successfully canceled tick 2024-08-19T12:52:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 1 -- 172.21.15.26:0/2986863743 >> v1:172.21.15.104:6800/899887506 conn(0x7fedf8040f90 legacy=0x7fedf8043450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 10 monclient: shutdown 2024-08-19T12:52:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 1 -- 172.21.15.26:0/2986863743 >> v1:172.21.15.26:6793/0 conn(0x7fee1813f2e0 legacy=0x7fee181d61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee16d9d640 1 -- 172.21.15.26:0/2986863743 reap_dead start 2024-08-19T12:52:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee16d9d640 1 -- 172.21.15.26:0/2986863743 reap_dead start 2024-08-19T12:52:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 1 -- 172.21.15.26:0/2986863743 shutdown_connections 2024-08-19T12:52:04.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 1 -- 172.21.15.26:0/2986863743 >> 172.21.15.26:0/2986863743 conn(0x7fee180abff0 msgr2=0x7fee180a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:04.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 1 -- 172.21.15.26:0/2986863743 shutdown_connections 2024-08-19T12:52:04.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 1 -- 172.21.15.26:0/2986863743 wait complete. 2024-08-19T12:52:04.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.439+0000 7fee1cb98640 1 librados: shutdown 2024-08-19T12:52:04.477 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T12:52:04.477 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon f 2024-08-19T12:52:04.477 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:52:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.591+0000 7efca1084640 10 monclient: get_monmap_and_config 2024-08-19T12:52:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.591+0000 7efca1084640 10 monclient: build_initial_monmap 2024-08-19T12:52:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.591+0000 7efca1084640 10 monclient: monmap: 2024-08-19T12:52:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:52:04.592076+0000 2024-08-19T12:52:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:52:04.592076+0000 2024-08-19T12:52:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.592 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:04.593 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:04.593 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.591+0000 7efca1084640 1 Processor -- start 2024-08-19T12:52:04.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.591+0000 7efca1084640 1 -- start start 2024-08-19T12:52:04.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.591+0000 7efca1084640 10 monclient: init 2024-08-19T12:52:04.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.592+0000 7efca1084640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.592+0000 7efca1084640 10 monclient: _add_conns ranks=[7,2,5,4,8,0,6,3,1] 2024-08-19T12:52:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.592+0000 7efca1084640 10 monclient(hunting): picked mon.noname-i con 0x7efc9c073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:52:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.592+0000 7efca1084640 10 monclient(hunting): picked mon.noname-b con 0x7efc9c0a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:52:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.592+0000 7efca1084640 10 monclient(hunting): picked mon.noname-h con 0x7efc9c13f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:52:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.592+0000 7efca1084640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c09dae0 con 0x7efc9c0a57a0 2024-08-19T12:52:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.592+0000 7efca1084640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c136910 con 0x7efc9c13f2e0 2024-08-19T12:52:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.592+0000 7efca1084640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c09dd50 con 0x7efc9c073470 2024-08-19T12:52:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.592+0000 7efca1084640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.592+0000 7efca1084640 10 monclient(hunting): authenticate will time out at 1518.679380s 2024-08-19T12:52:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.592+0000 7efc9a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7efc9c0a57a0 0x7efc9c0a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33238/0 (socket says 172.21.15.26:33238) 2024-08-19T12:52:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.592+0000 7efc9a59c640 1 -- 172.21.15.26:0/2143653600 learned_addr learned my addr 172.21.15.26:0/2143653600 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:04.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc9a59c640 1 -- 172.21.15.26:0/2143653600 >> v1:172.21.15.26:6790/0 conn(0x7efc9c0a57a0 legacy=0x7efc9c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc9ad9d640 1 -- 172.21.15.26:0/2143653600 >> v1:172.21.15.104:6792/0 conn(0x7efc9c073470 legacy=0x7efc9c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc9b59e640 1 -- 172.21.15.26:0/2143653600 >> v1:172.21.15.104:6791/0 conn(0x7efc9c13f2e0 legacy=0x7efc9c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2717050002 0 0) 0x7efc9c09dae0 con 0x7efc9c0a57a0 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 10 monclient(hunting): my global_id is 27047 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc84003520 con 0x7efc9c0a57a0 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1464560666 0 0) 0x7efc9c09dd50 con 0x7efc9c073470 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 10 monclient(hunting): my global_id is 136006 2024-08-19T12:52:04.600 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc9c09dae0 con 0x7efc9c073470 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2298163486 0 0) 0x7efc9c136910 con 0x7efc9c13f2e0 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 10 monclient(hunting): my global_id is 184280 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.594+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc9c09dd50 con 0x7efc9c13f2e0 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2058504756 0 0) 0x7efc84003520 con 0x7efc9c0a57a0 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 >> v1:172.21.15.104:6792/0 conn(0x7efc9c073470 legacy=0x7efc9c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 >> v1:172.21.15.104:6791/0 conn(0x7efc9c13f2e0 legacy=0x7efc9c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 1 monclient: found mon.noname-b 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:04.601 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc9c09e030 con 0x7efc9c0a57a0 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 10 monclient: _finish_auth 0 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efca1084640 5 monclient: authenticate success, global_id 27047 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efca1084640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efc90001ae0 con 0x7efc9c0a57a0 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 10 monclient: dump: 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efca1084640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efc900048a0 con 0x7efc9c0a57a0 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efca1084640 10 monclient: get_monmap_and_config success 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 1 -- 172.21.15.26:0/2143653600 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efc90004e50 con 0x7efc9c0a57a0 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efca1084640 4 set_mon_vals no callback set 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efc99d9b640 10 monclient: dump: 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.604 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efca1084640 10 monclient: shutdown 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efca1084640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:04.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efca1084640 1 -- 172.21.15.26:0/2143653600 >> v1:172.21.15.26:6790/0 conn(0x7efc9c0a57a0 legacy=0x7efc9c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efca1084640 1 -- 172.21.15.26:0/2143653600 shutdown_connections 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efca1084640 1 -- 172.21.15.26:0/2143653600 >> 172.21.15.26:0/2143653600 conn(0x7efc9c0abff0 msgr2=0x7efc9c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efca1084640 1 -- 172.21.15.26:0/2143653600 shutdown_connections 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.595+0000 7efca1084640 1 -- 172.21.15.26:0/2143653600 wait complete. 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.596+0000 7efca1084640 10 monclient: build_initial_monmap 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.596+0000 7efca1084640 10 monclient: monmap: 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.606 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.596+0000 7efca1084640 1 librados: starting msgr at 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.596+0000 7efca1084640 1 librados: starting objecter 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.596+0000 7efca1084640 1 Processor -- start 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.596+0000 7efca1084640 1 -- start start 2024-08-19T12:52:04.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.596+0000 7efca1084640 1 librados: setting wanted keys 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.596+0000 7efca1084640 1 librados: calling monclient init 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.596+0000 7efca1084640 10 monclient: init 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.597+0000 7efca1084640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.597+0000 7efca1084640 10 monclient: _add_conns ranks=[5,1,2,7,4,8,6,3,0] 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.597+0000 7efca1084640 10 monclient(hunting): picked mon.noname-h con 0x7efc9c073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.597+0000 7efca1084640 10 monclient(hunting): picked mon.noname-f con 0x7efc9c0a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.597+0000 7efca1084640 10 monclient(hunting): picked mon.noname-b con 0x7efc9c13f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.597+0000 7efca1084640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c136b20 con 0x7efc9c0a57a0 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.597+0000 7efca1084640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c0a3c10 con 0x7efc9c13f2e0 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.597+0000 7efca1084640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9c1373a0 con 0x7efc9c073470 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.597+0000 7efca1084640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.597+0000 7efca1084640 10 monclient(hunting): authenticate will time out at 1518.684402s 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.597+0000 7efc9b59e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7efc9c13f2e0 0x7efc9c1d6a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:33246/0 (socket says 172.21.15.26:33246) 2024-08-19T12:52:04.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.597+0000 7efc9b59e640 1 -- 172.21.15.26:0/3417359966 learned_addr learned my addr 172.21.15.26:0/3417359966 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:04.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc9b59e640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.26:6790/0 conn(0x7efc9c13f2e0 legacy=0x7efc9c1d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 client.?.objecter ms_handle_connect 0x7efc9c13f2e0 2024-08-19T12:52:04.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:04.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 client.?.objecter ms_handle_connect 0x7efc9c073470 2024-08-19T12:52:04.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:04.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc9ad9d640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.104:6791/0 conn(0x7efc9c073470 legacy=0x7efc9c1ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 client.?.objecter ms_handle_connect 0x7efc9c0a57a0 2024-08-19T12:52:04.615 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc9a59c640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.104:6789/0 conn(0x7efc9c0a57a0 legacy=0x7efc9c1d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3727574487 0 0) 0x7efc9c0a3c10 con 0x7efc9c13f2e0 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 monclient(hunting): my global_id is 27056 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc88004eb0 con 0x7efc9c13f2e0 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2739556022 0 0) 0x7efc9c1373a0 con 0x7efc9c073470 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 monclient(hunting): my global_id is 184289 2024-08-19T12:52:04.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc9c0a3c10 con 0x7efc9c073470 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3544107014 0 0) 0x7efc9c136b20 con 0x7efc9c0a57a0 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 10 monclient(hunting): my global_id is 184258 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.599+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc9c1373a0 con 0x7efc9c0a57a0 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2209571408 0 0) 0x7efc88004eb0 con 0x7efc9c13f2e0 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc9c136b20 con 0x7efc9c13f2e0 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efc94003610 con 0x7efc9c13f2e0 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient(hunting): dump: 2024-08-19T12:52:04.617 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.618 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1821212135 0 0) 0x7efc9c0a3c10 con 0x7efc9c073470 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc88004eb0 con 0x7efc9c073470 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efc8c003990 con 0x7efc9c073470 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient(hunting): dump: 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.619 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.620 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.620 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.620 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.620 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.620 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.620 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.620 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.620 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.620 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2944705635 0 0) 0x7efc9c1373a0 con 0x7efc9c0a57a0 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc9c0a3c10 con 0x7efc9c0a57a0 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2803204487 0 0) 0x7efc9c136b20 con 0x7efc9c13f2e0 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.104:6791/0 conn(0x7efc9c073470 legacy=0x7efc9c1ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.104:6789/0 conn(0x7efc9c0a57a0 legacy=0x7efc9c1d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient: _finish_hunting 0 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 monclient: found mon.b 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:52:04.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc9c0a4770 con 0x7efc9c13f2e0 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient: _finish_auth 0 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient: _check_auth_tickets 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efca1084640 5 monclient: authenticate success, global_id 27056 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efca1084640 10 monclient: _renew_subs 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efca1084640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efca1084640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efc9c0a5310 con 0x7efc9c13f2e0 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efca1084640 10 client.27056.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efca1084640 10 monclient: _renew_subs 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efca1084640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efca1084640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efc9c0a3340 con 0x7efc9c13f2e0 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efca1084640 1 librados: init done 2024-08-19T12:52:04.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efc94003b10 con 0x7efc9c13f2e0 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efc94004080 con 0x7efc9c13f2e0 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc9959a640 4 set_mon_vals no callback set 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.600+0000 7efc837fe640 10 monclient: dump: 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.623 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.601+0000 7efca1084640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.601+0000 7efca1084640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efc7c0065c0 con 0x7efc7c002860 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.601+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7efc94004220 con 0x7efc9c13f2e0 2024-08-19T12:52:04.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.601+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(491..491 src has 350..491) ==== 6287+0+0 (unknown 850093190 0 0) 0x7efc9404e3f0 con 0x7efc9c13f2e0 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.601+0000 7efc837fe640 10 client.27056.objecter ms_dispatch 0x7efc9c000b90 osd_map(491..491 src has 350..491) 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.601+0000 7efc837fe640 3 client.27056.objecter handle_osd_map got epochs [491,491] > 0 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.601+0000 7efc837fe640 3 client.27056.objecter handle_osd_map decoding full epoch 491 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.601+0000 7efc837fe640 20 client.27056.objecter dump_active .. 0 homeless 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.601+0000 7efc9a59c640 10 monclient: get_auth_request con 0x7efc88041780 auth_method 0 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.603+0000 7efc9ad9d640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.104:6789/0 conn(0x7efc7c002860 legacy=0x7efc7c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.603+0000 7efc837fe640 10 client.27056.objecter ms_handle_connect 0x7efc7c002860 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.603+0000 7efc837fe640 10 client.27056.objecter resend_mon_ops 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.603+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1347993288 0 0) 0x7efc7c0065c0 con 0x7efc7c002860 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.603+0000 7efc837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.603+0000 7efc837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.603+0000 7efc837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.603+0000 7efc837fe640 10 monclient(hunting): my global_id is 184267 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.603+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc9c136b20 con 0x7efc7c002860 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1068537543 0 0) 0x7efc9c136b20 con 0x7efc7c002860 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc7c0065c0 con 0x7efc7c002860 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efc8c004220 con 0x7efc7c002860 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 10 monclient: dump: 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 10 client.27056.objecter ms_handle_connect 0x7efc88041780 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3258845396 0 0) 0x7efc7c0065c0 con 0x7efc7c002860 2024-08-19T12:52:04.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc7c00a160 con 0x7efc7c002860 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7efc8c004530 con 0x7efc7c002860 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.104:6789/0 conn(0x7efc7c002860 legacy=0x7efc7c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.604+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.104:6789/0 conn(0x7efc7c002860 legacy=0x7efc7c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.610+0000 7efca1084640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.610+0000 7efca1084640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efc7c009e30 con 0x7efc7c009110 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc9ad9d640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.104:6789/0 conn(0x7efc7c009110 legacy=0x7efc7c009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 10 client.27056.objecter ms_handle_connect 0x7efc7c009110 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 10 client.27056.objecter resend_mon_ops 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4160352220 0 0) 0x7efc7c009e30 con 0x7efc7c009110 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 10 monclient(hunting): my global_id is 184276 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc7c0065c0 con 0x7efc7c009110 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2891537273 0 0) 0x7efc7c0065c0 con 0x7efc7c009110 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc7c009e30 con 0x7efc7c009110 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efc8c006070 con 0x7efc7c009110 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 10 monclient: dump: 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.629 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2897502582 0 0) 0x7efc7c009e30 con 0x7efc7c009110 2024-08-19T12:52:04.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.613+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efc7c00c3f0 con 0x7efc7c009110 2024-08-19T12:52:04.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7512 (unknown 0 0 3148945768) 0x7efc8c008270 con 0x7efc7c009110 2024-08-19T12:52:04.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efc837fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:52:04.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efc837fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:52:04.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.104:6789/0 conn(0x7efc7c009110 legacy=0x7efc7c009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efc837fe640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.104:6789/0 conn(0x7efc7c009110 legacy=0x7efc7c009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.631 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:52:04.631 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:52:04.631 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 98, 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 50082, 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:52:04.632 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.633 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.634 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:52:04.635 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.636 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:52:04.637 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.638 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.640 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:52:04.640 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:52:04.640 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.640 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.640 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.640 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:52:04.640 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:52:04.640 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.640 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.641 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:52:04.641 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.641 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.641 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:52:04.641 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:52:04.641 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.641 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.641 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:52:04.642 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:52:04.643 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.646 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:52:04.648 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.649 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.650 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:52:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:52:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.651 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:52:04.651 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:52:04.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efca1084640 10 librados: watch_flush enter 2024-08-19T12:52:04.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efca1084640 10 librados: watch_flush exit 2024-08-19T12:52:04.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efca1084640 20 client.27056.objecter shutdown clearing up homeless session... 2024-08-19T12:52:04.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efca1084640 10 client.27056.objecter successfully canceled tick 2024-08-19T12:52:04.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efca1084640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.104:6800/899887506 conn(0x7efc88041780 legacy=0x7efc88043c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efca1084640 10 monclient: shutdown 2024-08-19T12:52:04.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efca1084640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:04.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efca1084640 1 -- 172.21.15.26:0/3417359966 >> v1:172.21.15.26:6790/0 conn(0x7efc9c13f2e0 legacy=0x7efc9c1d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efc9b59e640 1 -- 172.21.15.26:0/3417359966 reap_dead start 2024-08-19T12:52:04.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efc9b59e640 1 -- 172.21.15.26:0/3417359966 reap_dead start 2024-08-19T12:52:04.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efca1084640 1 -- 172.21.15.26:0/3417359966 shutdown_connections 2024-08-19T12:52:04.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efca1084640 1 -- 172.21.15.26:0/3417359966 >> 172.21.15.26:0/3417359966 conn(0x7efc9c0abff0 msgr2=0x7efc9c1349c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:04.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efca1084640 1 -- 172.21.15.26:0/3417359966 shutdown_connections 2024-08-19T12:52:04.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.614+0000 7efca1084640 1 -- 172.21.15.26:0/3417359966 wait complete. 2024-08-19T12:52:04.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.615+0000 7efca1084640 1 librados: shutdown 2024-08-19T12:52:04.654 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T12:52:04.654 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon h 2024-08-19T12:52:04.655 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:52:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.753+0000 7f2167179640 10 monclient: get_monmap_and_config 2024-08-19T12:52:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.753+0000 7f2167179640 10 monclient: build_initial_monmap 2024-08-19T12:52:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.753+0000 7f2167179640 10 monclient: monmap: 2024-08-19T12:52:04.754 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:52:04.754627+0000 2024-08-19T12:52:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:52:04.754627+0000 2024-08-19T12:52:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:04.755 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:04.756 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:04.756 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.754+0000 7f2167179640 1 Processor -- start 2024-08-19T12:52:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.754+0000 7f2167179640 1 -- start start 2024-08-19T12:52:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.754+0000 7f2167179640 10 monclient: init 2024-08-19T12:52:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.754+0000 7f2167179640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.755+0000 7f2167179640 10 monclient: _add_conns ranks=[6,4,1,7,2,0,5,3,8] 2024-08-19T12:52:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.755+0000 7f2167179640 10 monclient(hunting): picked mon.noname-d con 0x7f216006e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:52:04.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.755+0000 7f2167179640 10 monclient(hunting): picked mon.noname-c con 0x7f21600a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:52:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.755+0000 7f2167179640 10 monclient(hunting): picked mon.noname-f con 0x7f21600a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:52:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.755+0000 7f2167179640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2160136790 con 0x7f21600a5d50 2024-08-19T12:52:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.755+0000 7f2167179640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f216009ddf0 con 0x7f21600a5920 2024-08-19T12:52:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.755+0000 7f2167179640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f216009e0d0 con 0x7f216006e7b0 2024-08-19T12:52:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.755+0000 7f2167179640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.755+0000 7f2167179640 10 monclient(hunting): authenticate will time out at 1518.842098s 2024-08-19T12:52:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.755+0000 7f2164f15640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f216006e7b0 legacy=0x7f2160136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:52:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.755+0000 7f215ffff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f21600a5920 0x7f216013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:55280/0 (socket says 172.21.15.26:55280) 2024-08-19T12:52:04.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.755+0000 7f215ffff640 1 -- 172.21.15.26:0/1196518488 learned_addr learned my addr 172.21.15.26:0/1196518488 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215ffff640 1 -- 172.21.15.26:0/1196518488 >> v1:172.21.15.26:6791/0 conn(0x7f21600a5920 legacy=0x7f216013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 1 -- 172.21.15.26:0/1196518488 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 503814516 0 0) 0x7f2148001450 con 0x7f21600a5920 2024-08-19T12:52:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f2165716640 1 -- 172.21.15.26:0/1196518488 >> v1:172.21.15.104:6789/0 conn(0x7f21600a5d50 legacy=0x7f2160141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient(hunting): my global_id is 184333 2024-08-19T12:52:04.762 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 1 -- 172.21.15.26:0/1196518488 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f214c003520 con 0x7f21600a5920 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 1 -- 172.21.15.26:0/1196518488 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1046686693 0 0) 0x7f214c003520 con 0x7f21600a5920 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 1 -- 172.21.15.26:0/1196518488 >> v1:172.21.15.26:6792/0 conn(0x7f216006e7b0 legacy=0x7f2160136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 1 -- 172.21.15.26:0/1196518488 >> v1:172.21.15.104:6789/0 conn(0x7f21600a5d50 legacy=0x7f2160141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 1 monclient: found mon.noname-c 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 1 -- 172.21.15.26:0/1196518488 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f216009e3b0 con 0x7f21600a5920 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 1 -- 172.21.15.26:0/1196518488 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2148001cb0 con 0x7f21600a5920 2024-08-19T12:52:04.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f2167179640 5 monclient: authenticate success, global_id 184333 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f2167179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient: dump: 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.764 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f2167179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 1 -- 172.21.15.26:0/1196518488 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2148004990 con 0x7f21600a5920 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 1 -- 172.21.15.26:0/1196518488 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2148004f00 con 0x7f21600a5920 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f2167179640 10 monclient: get_monmap_and_config success 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f2167179640 4 set_mon_vals no callback set 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.757+0000 7f215f7fe640 10 monclient: dump: 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.758+0000 7f2167179640 10 monclient: shutdown 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.758+0000 7f2167179640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.758+0000 7f2167179640 1 -- 172.21.15.26:0/1196518488 >> v1:172.21.15.26:6791/0 conn(0x7f21600a5920 legacy=0x7f216013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.758+0000 7f2167179640 1 -- 172.21.15.26:0/1196518488 shutdown_connections 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.758+0000 7f2167179640 1 -- 172.21.15.26:0/1196518488 >> 172.21.15.26:0/1196518488 conn(0x7f21600a8740 msgr2=0x7f21600a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.758+0000 7f2167179640 1 -- 172.21.15.26:0/1196518488 shutdown_connections 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.758+0000 7f2167179640 1 -- 172.21.15.26:0/1196518488 wait complete. 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.758+0000 7f2167179640 10 monclient: build_initial_monmap 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 10 monclient: monmap: 2024-08-19T12:52:04.767 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:04.768 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 1 librados: starting msgr at 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 1 librados: starting objecter 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 1 Processor -- start 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 1 -- start start 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 1 librados: setting wanted keys 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 1 librados: calling monclient init 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 10 monclient: init 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 10 monclient: _add_conns ranks=[1,0,5,3,8,2,7,4,6] 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 10 monclient(hunting): picked mon.noname-f con 0x7f216006e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 10 monclient(hunting): picked mon.noname-a con 0x7f21600a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 10 monclient(hunting): picked mon.noname-h con 0x7f21600a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2160125ad0 con 0x7f21600a5920 2024-08-19T12:52:04.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2160128910 con 0x7f216006e7b0 2024-08-19T12:52:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2160125230 con 0x7f21600a5d50 2024-08-19T12:52:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.759+0000 7f2167179640 10 monclient(hunting): authenticate will time out at 1518.846986s 2024-08-19T12:52:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.760+0000 7f215ffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f21600a5920 legacy=0x7f2160122c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:52:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.760+0000 7f2164f15640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f216006e7b0 0x7f2160122500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:55436/0 (socket says 172.21.15.26:55436) 2024-08-19T12:52:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.760+0000 7f2165716640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f21600a5d50 0x7f21601d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:43970/0 (socket says 172.21.15.26:43970) 2024-08-19T12:52:04.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.760+0000 7f2164f15640 1 -- 172.21.15.26:0/1785801260 learned_addr learned my addr 172.21.15.26:0/1785801260 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:04.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f2165716640 1 -- 172.21.15.26:0/1785801260 >> v1:172.21.15.104:6791/0 conn(0x7f21600a5d50 legacy=0x7f21601d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 client.?.objecter ms_handle_connect 0x7f21600a5d50 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 client.?.objecter ms_handle_connect 0x7f216006e7b0 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f2164f15640 1 -- 172.21.15.26:0/1785801260 >> v1:172.21.15.104:6789/0 conn(0x7f216006e7b0 legacy=0x7f2160122500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1537473725 0 0) 0x7f2160128910 con 0x7f216006e7b0 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): my global_id is 184294 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2140004eb0 con 0x7f216006e7b0 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3874936 0 0) 0x7f2160125230 con 0x7f21600a5d50 2024-08-19T12:52:04.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): my global_id is 184298 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2160128910 con 0x7f21600a5d50 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2858783744 0 0) 0x7f2160128910 con 0x7f21600a5d50 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2160125230 con 0x7f21600a5d50 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2158001cf0 con 0x7f21600a5d50 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): dump: 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.776 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3346378208 0 0) 0x7f2140004eb0 con 0x7f216006e7b0 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2160128910 con 0x7f216006e7b0 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2150004910 con 0x7f216006e7b0 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.762+0000 7f215d7fa640 10 monclient(hunting): dump: 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.778 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 497121839 0 0) 0x7f2160125230 con 0x7f21600a5d50 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 >> v1:172.21.15.104:6789/0 conn(0x7f216006e7b0 legacy=0x7f2160122500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 >> v1:172.21.15.26:6789/0 conn(0x7f21600a5920 legacy=0x7f2160122c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 10 monclient: _finish_hunting 0 2024-08-19T12:52:04.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 1 monclient: found mon.h 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2160142d40 con 0x7f21600a5d50 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 10 monclient: _finish_auth 0 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 10 monclient: _check_auth_tickets 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f2167179640 5 monclient: authenticate success, global_id 184298 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f2167179640 10 monclient: _renew_subs 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f2167179640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f2167179640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2160124ec0 con 0x7f21600a5d50 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f2167179640 10 client.184298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f2167179640 10 monclient: _renew_subs 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f2167179640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:52:04.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f2167179640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f2160123500 con 0x7f21600a5d50 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f2167179640 1 librados: init done 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f21580038c0 con 0x7f21600a5d50 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2158003e70 con 0x7f21600a5d50 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215e7fc640 4 set_mon_vals no callback set 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f215d7fa640 10 monclient: dump: 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.781 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f2167179640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.763+0000 7f2167179640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f21440065c0 con 0x7f2144002860 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.764+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2158004090 con 0x7f21600a5d50 2024-08-19T12:52:04.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.764+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(491..491 src has 350..491) ==== 6287+0+0 (unknown 850093190 0 0) 0x7f215804e420 con 0x7f21600a5d50 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.764+0000 7f215d7fa640 10 client.184298.objecter ms_dispatch 0x7f2160000b90 osd_map(491..491 src has 350..491) 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.764+0000 7f215d7fa640 3 client.184298.objecter handle_osd_map got epochs [491,491] > 0 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.764+0000 7f215d7fa640 3 client.184298.objecter handle_osd_map decoding full epoch 491 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.764+0000 7f215d7fa640 20 client.184298.objecter dump_active .. 0 homeless 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.764+0000 7f215ffff640 10 monclient: get_auth_request con 0x7f2140041430 auth_method 0 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 10 client.184298.objecter ms_handle_connect 0x7f2144002860 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f2164f15640 1 -- 172.21.15.26:0/1785801260 >> v1:172.21.15.104:6791/0 conn(0x7f2144002860 legacy=0x7f2144004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 10 client.184298.objecter resend_mon_ops 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2937059302 0 0) 0x7f21440065c0 con 0x7f2144002860 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 10 monclient(hunting): my global_id is 184307 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2160125230 con 0x7f2144002860 2024-08-19T12:52:04.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3279862028 0 0) 0x7f2160125230 con 0x7f2144002860 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21440065c0 con 0x7f2144002860 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f21500051f0 con 0x7f2144002860 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 10 monclient: dump: 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.784 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2099757172 0 0) 0x7f21440065c0 con 0x7f2144002860 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f214400a160 con 0x7f2144002860 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 10 client.184298.objecter ms_handle_connect 0x7f2140041430 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2150007510 con 0x7f2144002860 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.766+0000 7f215d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:52:04.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.767+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 >> v1:172.21.15.104:6791/0 conn(0x7f2144002860 legacy=0x7f2144004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.767+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 >> v1:172.21.15.104:6791/0 conn(0x7f2144002860 legacy=0x7f2144004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.773+0000 7f2167179640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:52:04.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.773+0000 7f2167179640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2144009e70 con 0x7f2144009130 2024-08-19T12:52:04.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.775+0000 7f2164f15640 1 -- 172.21.15.26:0/1785801260 >> v1:172.21.15.104:6791/0 conn(0x7f2144009130 legacy=0x7f21440095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.775+0000 7f215d7fa640 10 client.184298.objecter ms_handle_connect 0x7f2144009130 2024-08-19T12:52:04.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.775+0000 7f215d7fa640 10 client.184298.objecter resend_mon_ops 2024-08-19T12:52:04.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.775+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 625463844 0 0) 0x7f2144009e70 con 0x7f2144009130 2024-08-19T12:52:04.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.775+0000 7f215d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.775+0000 7f215d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.775+0000 7f215d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.775+0000 7f215d7fa640 10 monclient(hunting): my global_id is 184316 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.775+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21440065c0 con 0x7f2144009130 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1981749877 0 0) 0x7f21440065c0 con 0x7f2144009130 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2144009e70 con 0x7f2144009130 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2150005b00 con 0x7f2144009130 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 10 monclient: dump: 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.788 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1788208408 0 0) 0x7f2144009e70 con 0x7f2144009130 2024-08-19T12:52:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f214400c410 con 0x7f2144009130 2024-08-19T12:52:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7677 (unknown 0 0 2658107183) 0x7f2150007070 con 0x7f2144009130 2024-08-19T12:52:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:52:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:52:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 >> v1:172.21.15.104:6791/0 conn(0x7f2144009130 legacy=0x7f21440095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f215d7fa640 1 -- 172.21.15.26:0/1785801260 >> v1:172.21.15.104:6791/0 conn(0x7f2144009130 legacy=0x7f21440095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.790 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:52:04.790 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:52:04.790 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:52:04.790 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 98, 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 50246, 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:52:04.791 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.792 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.793 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:52:04.794 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.795 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.798 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:52:04.798 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:52:04.798 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.798 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:52:04.798 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.798 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.798 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.799 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:52:04.799 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:52:04.799 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.799 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.799 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.799 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:52:04.799 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:52:04.799 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.800 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.800 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.800 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:52:04.800 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.800 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.800 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.800 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:52:04.800 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:52:04.800 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.800 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.801 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.801 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:52:04.801 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:52:04.801 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:52:04.802 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.803 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:52:04.803 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.803 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:52:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:52:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:52:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:52:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:52:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:52:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:52:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.806 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:52:04.806 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:52:04.806 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.806 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.806 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:52:04.806 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.806 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.806 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:52:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:52:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:52:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:52:04.807 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.808 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:52:04.808 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.808 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:52:04.808 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:52:04.809 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.809 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.809 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.809 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:52:04.809 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:52:04.809 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.809 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.810 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:52:04.810 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.810 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.810 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:52:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:52:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:52:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:52:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.811 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.812 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:52:04.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.813 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.813 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:52:04.813 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:52:04.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2167179640 10 librados: watch_flush enter 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2167179640 10 librados: watch_flush exit 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2167179640 20 client.184298.objecter shutdown clearing up homeless session... 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2167179640 10 client.184298.objecter successfully canceled tick 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2167179640 1 -- 172.21.15.26:0/1785801260 >> v1:172.21.15.104:6800/899887506 conn(0x7f2140041430 legacy=0x7f21400438f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2167179640 10 monclient: shutdown 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2167179640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2167179640 1 -- 172.21.15.26:0/1785801260 >> v1:172.21.15.104:6791/0 conn(0x7f21600a5d50 legacy=0x7f21601d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2165716640 1 -- 172.21.15.26:0/1785801260 reap_dead start 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2165716640 1 -- 172.21.15.26:0/1785801260 reap_dead start 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2167179640 1 -- 172.21.15.26:0/1785801260 shutdown_connections 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2167179640 1 -- 172.21.15.26:0/1785801260 >> 172.21.15.26:0/1785801260 conn(0x7f21600a8740 msgr2=0x7f21600aa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.776+0000 7f2167179640 1 -- 172.21.15.26:0/1785801260 shutdown_connections 2024-08-19T12:52:04.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.777+0000 7f2167179640 1 -- 172.21.15.26:0/1785801260 wait complete. 2024-08-19T12:52:04.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.777+0000 7f2167179640 1 librados: shutdown 2024-08-19T12:52:04.816 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T12:52:04.816 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon i 2024-08-19T12:52:04.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:52:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.915+0000 7ff1c1b04640 10 monclient: get_monmap_and_config 2024-08-19T12:52:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.915+0000 7ff1c1b04640 10 monclient: build_initial_monmap 2024-08-19T12:52:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.915+0000 7ff1c1b04640 10 monclient: monmap: 2024-08-19T12:52:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:52:04.916033+0000 2024-08-19T12:52:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:52:04.916033+0000 2024-08-19T12:52:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:04.918 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.915+0000 7ff1c1b04640 1 Processor -- start 2024-08-19T12:52:04.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.915+0000 7ff1c1b04640 1 -- start start 2024-08-19T12:52:04.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.915+0000 7ff1c1b04640 10 monclient: init 2024-08-19T12:52:04.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.916+0000 7ff1c1b04640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:04.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.916+0000 7ff1c1b04640 10 monclient: _add_conns ranks=[1,7,4,6,2,0,5,3,8] 2024-08-19T12:52:04.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.916+0000 7ff1c1b04640 10 monclient(hunting): picked mon.noname-f con 0x7ff1bc073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:52:04.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.916+0000 7ff1c1b04640 10 monclient(hunting): picked mon.noname-i con 0x7ff1bc0a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:52:04.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.916+0000 7ff1c1b04640 10 monclient(hunting): picked mon.noname-c con 0x7ff1bc13f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:52:04.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.916+0000 7ff1c1b04640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc09dae0 con 0x7ff1bc073470 2024-08-19T12:52:04.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.916+0000 7ff1c1b04640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc136910 con 0x7ff1bc13f2e0 2024-08-19T12:52:04.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.916+0000 7ff1c1b04640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc09dd50 con 0x7ff1bc0a57a0 2024-08-19T12:52:04.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.916+0000 7ff1c1b04640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:04.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.916+0000 7ff1c1b04640 10 monclient(hunting): authenticate will time out at 1519.003574s 2024-08-19T12:52:04.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.916+0000 7ff1bbfff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7ff1bc13f2e0 0x7ff1bc1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:55290/0 (socket says 172.21.15.26:55290) 2024-08-19T12:52:04.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.916+0000 7ff1bbfff640 1 -- 172.21.15.26:0/673094404 learned_addr learned my addr 172.21.15.26:0/673094404 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:04.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.918+0000 7ff1bbfff640 1 -- 172.21.15.26:0/673094404 >> v1:172.21.15.26:6791/0 conn(0x7ff1bc13f2e0 legacy=0x7ff1bc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.918+0000 7ff1ba7fc640 1 -- 172.21.15.26:0/673094404 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2029283817 0 0) 0x7ff1bc136910 con 0x7ff1bc13f2e0 2024-08-19T12:52:04.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient(hunting): my global_id is 184342 2024-08-19T12:52:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 1 -- 172.21.15.26:0/673094404 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1a4003520 con 0x7ff1bc13f2e0 2024-08-19T12:52:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1bb7fe640 1 -- 172.21.15.26:0/673094404 >> v1:172.21.15.104:6789/0 conn(0x7ff1bc073470 legacy=0x7ff1bc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1baffd640 1 -- 172.21.15.26:0/673094404 >> v1:172.21.15.104:6792/0 conn(0x7ff1bc0a57a0 legacy=0x7ff1bc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 1 -- 172.21.15.26:0/673094404 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2365371881 0 0) 0x7ff1bc09dae0 con 0x7ff1bc073470 2024-08-19T12:52:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient(hunting): my global_id is 184303 2024-08-19T12:52:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 1 -- 172.21.15.26:0/673094404 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1bc136910 con 0x7ff1bc073470 2024-08-19T12:52:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 1 -- 172.21.15.26:0/673094404 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 930187462 0 0) 0x7ff1a4003520 con 0x7ff1bc13f2e0 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 1 -- 172.21.15.26:0/673094404 >> v1:172.21.15.104:6792/0 conn(0x7ff1bc0a57a0 legacy=0x7ff1bc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 1 -- 172.21.15.26:0/673094404 >> v1:172.21.15.104:6789/0 conn(0x7ff1bc073470 legacy=0x7ff1bc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 1 monclient: found mon.noname-c 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 1 -- 172.21.15.26:0/673094404 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1bc09e030 con 0x7ff1bc13f2e0 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient: _finish_auth 0 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 1 -- 172.21.15.26:0/673094404 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1b4001ae0 con 0x7ff1bc13f2e0 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1c1b04640 5 monclient: authenticate success, global_id 184342 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1c1b04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:52:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient: dump: 2024-08-19T12:52:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1c1b04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 1 -- 172.21.15.26:0/673094404 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1b4001d30 con 0x7ff1bc13f2e0 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 1 -- 172.21.15.26:0/673094404 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1b4004be0 con 0x7ff1bc13f2e0 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1c1b04640 10 monclient: get_monmap_and_config success 2024-08-19T12:52:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1c1b04640 4 set_mon_vals no callback set 2024-08-19T12:52:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:52:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1ba7fc640 10 monclient: dump: 2024-08-19T12:52:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.930 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1c1b04640 10 monclient: shutdown 2024-08-19T12:52:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1c1b04640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1c1b04640 1 -- 172.21.15.26:0/673094404 >> v1:172.21.15.26:6791/0 conn(0x7ff1bc13f2e0 legacy=0x7ff1bc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1c1b04640 1 -- 172.21.15.26:0/673094404 shutdown_connections 2024-08-19T12:52:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1c1b04640 1 -- 172.21.15.26:0/673094404 >> 172.21.15.26:0/673094404 conn(0x7ff1bc0abff0 msgr2=0x7ff1bc0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1c1b04640 1 -- 172.21.15.26:0/673094404 shutdown_connections 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.919+0000 7ff1c1b04640 1 -- 172.21.15.26:0/673094404 wait complete. 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.920+0000 7ff1c1b04640 10 monclient: build_initial_monmap 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.920+0000 7ff1c1b04640 10 monclient: monmap: 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:52:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.920+0000 7ff1c1b04640 1 librados: starting msgr at 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.920+0000 7ff1c1b04640 1 librados: starting objecter 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.920+0000 7ff1c1b04640 1 Processor -- start 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 1 -- start start 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 1 librados: setting wanted keys 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 1 librados: calling monclient init 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 10 monclient: init 2024-08-19T12:52:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 10 monclient: _reopen_session rank -1 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 10 monclient: _add_conns ranks=[4,6,0,2,3,1,8,5,7] 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 10 monclient(hunting): picked mon.noname-c con 0x7ff1bc073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 10 monclient(hunting): picked mon.noname-d con 0x7ff1bc0a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 10 monclient(hunting): picked mon.noname-a con 0x7ff1bc13f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc135270 con 0x7ff1bc13f2e0 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc13f070 con 0x7ff1bc073470 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc1318c0 con 0x7ff1bc0a57a0 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 10 monclient(hunting): _renew_subs 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1baffd640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7ff1bc0a57a0 legacy=0x7ff1bc12e5e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1c1b04640 10 monclient(hunting): authenticate will time out at 1519.008491s 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1bbfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7ff1bc13f2e0 legacy=0x7ff1bc1d6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1bb7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7ff1bc073470 0x7ff1bc12ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:55298/0 (socket says 172.21.15.26:55298) 2024-08-19T12:52:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.921+0000 7ff1bb7fe640 1 -- 172.21.15.26:0/3884793998 learned_addr learned my addr 172.21.15.26:0/3884793998 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:52:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.923+0000 7ff1bb7fe640 1 -- 172.21.15.26:0/3884793998 >> v1:172.21.15.26:6791/0 conn(0x7ff1bc073470 legacy=0x7ff1bc12ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.923+0000 7ff1a3fff640 10 client.?.objecter ms_handle_connect 0x7ff1bc073470 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.923+0000 7ff1a3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.923+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1578934771 0 0) 0x7ff1bc136910 con 0x7ff1bc073470 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.923+0000 7ff1a3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.923+0000 7ff1a3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.923+0000 7ff1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.923+0000 7ff1a3fff640 10 monclient(hunting): my global_id is 184351 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.923+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1a8004eb0 con 0x7ff1bc073470 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.923+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 194986817 0 0) 0x7ff1a8004eb0 con 0x7ff1bc073470 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1bc136910 con 0x7ff1bc073470 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1ac003990 con 0x7ff1bc073470 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:52:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 10 monclient(hunting): dump: 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 117236185 0 0) 0x7ff1bc136910 con 0x7ff1bc073470 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 >> v1:172.21.15.26:6792/0 conn(0x7ff1bc0a57a0 legacy=0x7ff1bc12e5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 >> v1:172.21.15.26:6789/0 conn(0x7ff1bc13f2e0 legacy=0x7ff1bc1d6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 10 monclient: _finish_hunting 0 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 1 monclient: found mon.c 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1bc131ba0 con 0x7ff1bc073470 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 10 monclient: _finish_auth 0 2024-08-19T12:52:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 10 monclient: _check_auth_tickets 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1c1b04640 5 monclient: authenticate success, global_id 184351 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1c1b04640 10 monclient: _renew_subs 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1c1b04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1c1b04640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff1bc130520 con 0x7ff1bc073470 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1c1b04640 10 client.184351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1c1b04640 10 monclient: _renew_subs 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1c1b04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1c1b04640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1bc1315e0 con 0x7ff1bc073470 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1c1b04640 1 librados: init done 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1ac003fc0 con 0x7ff1bc073470 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:52:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1ac0044b0 con 0x7ff1bc073470 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1b9ffb640 4 set_mon_vals no callback set 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.924+0000 7ff1a3fff640 10 monclient: dump: 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.925+0000 7ff1c1b04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.925+0000 7ff1c1b04640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff19c0065c0 con 0x7ff19c002860 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.925+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff1ac011e00 con 0x7ff1bc073470 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.925+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(491..491 src has 350..491) ==== 6287+0+0 (unknown 850093190 0 0) 0x7ff1ac04ea20 con 0x7ff1bc073470 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.925+0000 7ff1a3fff640 10 client.184351.objecter ms_dispatch 0x7ff1bc000b90 osd_map(491..491 src has 350..491) 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.925+0000 7ff1a3fff640 3 client.184351.objecter handle_osd_map got epochs [491,491] > 0 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.925+0000 7ff1a3fff640 3 client.184351.objecter handle_osd_map decoding full epoch 491 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.925+0000 7ff1a3fff640 20 client.184351.objecter dump_active .. 0 homeless 2024-08-19T12:52:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.925+0000 7ff1bbfff640 10 monclient: get_auth_request con 0x7ff1a8040da0 auth_method 0 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1baffd640 1 -- 172.21.15.26:0/3884793998 >> v1:172.21.15.104:6792/0 conn(0x7ff19c002860 legacy=0x7ff19c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 10 client.184351.objecter ms_handle_connect 0x7ff19c002860 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 10 client.184351.objecter resend_mon_ops 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4005120686 0 0) 0x7ff19c0065c0 con 0x7ff19c002860 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 10 monclient(hunting): my global_id is 136024 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1bc136910 con 0x7ff19c002860 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 869119805 0 0) 0x7ff1bc136910 con 0x7ff19c002860 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff19c0065c0 con 0x7ff19c002860 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1b0001c10 con 0x7ff19c002860 2024-08-19T12:52:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 10 monclient: dump: 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.927+0000 7ff1a3fff640 10 client.184351.objecter ms_handle_connect 0x7ff1a8040da0 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.928+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2611628971 0 0) 0x7ff19c0065c0 con 0x7ff19c002860 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.928+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff19c00a160 con 0x7ff19c002860 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.928+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff1b0006830 con 0x7ff19c002860 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.928+0000 7ff1a3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.928+0000 7ff1a3fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.928+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 >> v1:172.21.15.104:6792/0 conn(0x7ff19c002860 legacy=0x7ff19c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.928+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 >> v1:172.21.15.104:6792/0 conn(0x7ff19c002860 legacy=0x7ff19c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.934+0000 7ff1c1b04640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.934+0000 7ff1c1b04640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff19c009e70 con 0x7ff19c009130 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1baffd640 1 -- 172.21.15.26:0/3884793998 >> v1:172.21.15.104:6792/0 conn(0x7ff19c009130 legacy=0x7ff19c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 10 client.184351.objecter ms_handle_connect 0x7ff19c009130 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 10 client.184351.objecter resend_mon_ops 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330657628 0 0) 0x7ff19c009e70 con 0x7ff19c009130 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 10 monclient(hunting): my global_id is 136033 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff19c0065c0 con 0x7ff19c009130 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2866632387 0 0) 0x7ff19c0065c0 con 0x7ff19c009130 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff19c009e70 con 0x7ff19c009130 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff1b0003e10 con 0x7ff19c009130 2024-08-19T12:52:04.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 10 monclient: dump: 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:52:04.949 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:52:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:52:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:52:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:52:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:52:04.950 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:52:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1021890974 0 0) 0x7ff19c009e70 con 0x7ff19c009130 2024-08-19T12:52:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.937+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff19c00c410 con 0x7ff19c009130 2024-08-19T12:52:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7511 (unknown 0 0 2473022788) 0x7ff1b0004120 con 0x7ff19c009130 2024-08-19T12:52:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1a3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:52:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1a3fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:52:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 >> v1:172.21.15.104:6792/0 conn(0x7ff19c009130 legacy=0x7ff19c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1a3fff640 1 -- 172.21.15.26:0/3884793998 >> v1:172.21.15.104:6792/0 conn(0x7ff19c009130 legacy=0x7ff19c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:52:04.951 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:52:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:52:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:52:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:52:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 98, 2024-08-19T12:52:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 695295, 2024-08-19T12:52:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:52:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:52:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:52:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:52:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:52:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:52:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.966 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.967 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.968 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:52:04.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:52:04.970 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:52:04.970 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:52:04.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 10 librados: watch_flush enter 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 10 librados: watch_flush exit 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 20 client.184351.objecter shutdown clearing up homeless session... 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 10 client.184351.objecter successfully canceled tick 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 1 -- 172.21.15.26:0/3884793998 >> v1:172.21.15.104:6800/899887506 conn(0x7ff1a8040da0 legacy=0x7ff1a8043260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 10 monclient: shutdown 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 1 -- 172.21.15.26:0/3884793998 >> v1:172.21.15.26:6791/0 conn(0x7ff1bc073470 legacy=0x7ff1bc12ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1bbfff640 1 -- 172.21.15.26:0/3884793998 reap_dead start 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1bbfff640 1 -- 172.21.15.26:0/3884793998 reap_dead start 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 1 -- 172.21.15.26:0/3884793998 shutdown_connections 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 1 -- 172.21.15.26:0/3884793998 >> 172.21.15.26:0/3884793998 conn(0x7ff1bc0abff0 msgr2=0x7ff1bc12c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 1 -- 172.21.15.26:0/3884793998 shutdown_connections 2024-08-19T12:52:04.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 1 -- 172.21.15.26:0/3884793998 wait complete. 2024-08-19T12:52:04.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:52:04.938+0000 7ff1c1b04640 1 librados: shutdown 2024-08-19T12:52:04.973 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T12:52:04.973 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T12:53:34.974 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-08-19T12:53:34.974 INFO:tasks.ceph.mon.d:Restarting daemon 2024-08-19T12:53:34.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-08-19T12:53:34.979 INFO:tasks.ceph.mon.d:Started 2024-08-19T12:53:34.979 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-08-19T12:53:34.979 INFO:tasks.ceph.mon.g:Restarting daemon 2024-08-19T12:53:34.979 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-08-19T12:53:34.982 INFO:tasks.ceph.mon.g:Started 2024-08-19T12:53:34.983 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-08-19T12:53:34.983 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-19T12:53:34.983 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-19T12:53:34.985 INFO:tasks.ceph.mon.a:Started 2024-08-19T12:53:34.985 INFO:tasks.mon_thrash:Sending STOP to mon i 2024-08-19T12:53:34.985 INFO:tasks.ceph.mon.i:Sent signal 19 2024-08-19T12:53:34.986 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:53:49.986 INFO:tasks.mon_thrash:Sending CONT to mon i 2024-08-19T12:53:49.987 INFO:tasks.ceph.mon.i:Sent signal 18 2024-08-19T12:53:49.987 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:53:49.987 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:53:50.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.113+0000 7f3df9ad5640 10 monclient: get_monmap_and_config 2024-08-19T12:53:50.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.113+0000 7f3df9ad5640 10 monclient: build_initial_monmap 2024-08-19T12:53:50.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.113+0000 7f3df9ad5640 10 monclient: monmap: 2024-08-19T12:53:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:50.114350+0000 2024-08-19T12:53:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:50.114350+0000 2024-08-19T12:53:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:50.115 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:50.116 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:50.116 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:50.116 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:50.116 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:50.116 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.114+0000 7f3df9ad5640 1 Processor -- start 2024-08-19T12:53:50.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.114+0000 7f3df9ad5640 1 -- start start 2024-08-19T12:53:50.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.114+0000 7f3df9ad5640 10 monclient: init 2024-08-19T12:53:50.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.115+0000 7f3df9ad5640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:50.117 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.115+0000 7f3df9ad5640 10 monclient: _add_conns ranks=[0,7,8,6,5,2,4,1,3] 2024-08-19T12:53:50.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.115+0000 7f3df9ad5640 10 monclient(hunting): picked mon.noname-a con 0x7f3df4073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:50.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.115+0000 7f3df9ad5640 10 monclient(hunting): picked mon.noname-i con 0x7f3df40a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:50.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.115+0000 7f3df9ad5640 10 monclient(hunting): picked mon.noname-e con 0x7f3df413f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:50.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.115+0000 7f3df9ad5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3df409dae0 con 0x7f3df4073470 2024-08-19T12:53:50.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.115+0000 7f3df9ad5640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3df4136910 con 0x7f3df40a57a0 2024-08-19T12:53:50.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.115+0000 7f3df3fff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f3df413f2e0 0x7f3df41416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40604/0 (socket says 172.21.15.26:40604) 2024-08-19T12:53:50.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.116+0000 7f3df3fff640 1 -- 172.21.15.26:0/607064441 learned_addr learned my addr 172.21.15.26:0/607064441 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:50.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.116+0000 7f3df9ad5640 1 -- 172.21.15.26:0/607064441 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3df409dd50 con 0x7f3df413f2e0 2024-08-19T12:53:50.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.116+0000 7f3df9ad5640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:50.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.116+0000 7f3df9ad5640 10 monclient(hunting): authenticate will time out at 1624.203061s 2024-08-19T12:53:50.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df3fff640 1 -- 172.21.15.26:0/607064441 >> v1:172.21.15.26:6793/0 conn(0x7f3df413f2e0 legacy=0x7f3df41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df37fe640 1 -- 172.21.15.26:0/607064441 >> v1:172.21.15.26:6789/0 conn(0x7f3df4073470 legacy=0x7f3df4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3debfff640 1 -- 172.21.15.26:0/607064441 >> v1:172.21.15.104:6792/0 conn(0x7f3df40a57a0 legacy=0x7f3df40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1104580034 0 0) 0x7f3df409dd50 con 0x7f3df413f2e0 2024-08-19T12:53:50.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient(hunting): my global_id is 184256 2024-08-19T12:53:50.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3de0003520 con 0x7f3df413f2e0 2024-08-19T12:53:50.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 12510870 0 0) 0x7f3df409dae0 con 0x7f3df4073470 2024-08-19T12:53:50.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient(hunting): my global_id is 194103 2024-08-19T12:53:50.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3df409dd50 con 0x7f3df4073470 2024-08-19T12:53:50.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3301902606 0 0) 0x7f3df4136910 con 0x7f3df40a57a0 2024-08-19T12:53:50.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient(hunting): my global_id is 136060 2024-08-19T12:53:50.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3df409dae0 con 0x7f3df40a57a0 2024-08-19T12:53:50.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3064950379 0 0) 0x7f3de0003520 con 0x7f3df413f2e0 2024-08-19T12:53:50.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 >> v1:172.21.15.104:6792/0 conn(0x7f3df40a57a0 legacy=0x7f3df40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 >> v1:172.21.15.26:6789/0 conn(0x7f3df4073470 legacy=0x7f3df4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:53:50.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 monclient: found mon.noname-e 2024-08-19T12:53:50.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:50.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:53:50.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df409e030 con 0x7f3df413f2e0 2024-08-19T12:53:50.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient: _finish_auth 0 2024-08-19T12:53:50.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:53:50.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:50.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dec001f20 con 0x7f3df413f2e0 2024-08-19T12:53:50.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df9ad5640 5 monclient: authenticate success, global_id 184256 2024-08-19T12:53:50.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df9ad5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:50.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:53:50.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient: dump: 2024-08-19T12:53:50.127 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.127 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.128 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.128 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.128 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.128 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.128 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.128 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.128 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.128 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.128 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.128 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.128 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.129 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.129 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.129 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df9ad5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:50.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3dec004c20 con 0x7f3df413f2e0 2024-08-19T12:53:50.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:50.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 1 -- 172.21.15.26:0/607064441 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dec0020c0 con 0x7f3df413f2e0 2024-08-19T12:53:50.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df9ad5640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:50.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df9ad5640 4 set_mon_vals no callback set 2024-08-19T12:53:50.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:50.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.118+0000 7f3df2ffd640 10 monclient: dump: 2024-08-19T12:53:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.130 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.131 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.131 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.131 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.131 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.131 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.131 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.119+0000 7f3df9ad5640 10 monclient: shutdown 2024-08-19T12:53:50.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.119+0000 7f3df9ad5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:50.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.119+0000 7f3df9ad5640 1 -- 172.21.15.26:0/607064441 >> v1:172.21.15.26:6793/0 conn(0x7f3df413f2e0 legacy=0x7f3df41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.119+0000 7f3df9ad5640 1 -- 172.21.15.26:0/607064441 shutdown_connections 2024-08-19T12:53:50.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.119+0000 7f3df9ad5640 1 -- 172.21.15.26:0/607064441 >> 172.21.15.26:0/607064441 conn(0x7f3df40abff0 msgr2=0x7f3df40ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:50.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.119+0000 7f3df9ad5640 1 -- 172.21.15.26:0/607064441 shutdown_connections 2024-08-19T12:53:50.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.119+0000 7f3df9ad5640 1 -- 172.21.15.26:0/607064441 wait complete. 2024-08-19T12:53:50.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.120+0000 7f3df9ad5640 10 monclient: build_initial_monmap 2024-08-19T12:53:50.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.120+0000 7f3df9ad5640 10 monclient: monmap: 2024-08-19T12:53:50.132 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:50.132 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:50.132 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:50.132 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:50.133 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:50.133 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.133 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:50.133 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:50.133 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:50.133 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:50.133 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:50.133 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:50.133 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:50.133 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:50.133 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:50.134 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.120+0000 7f3df9ad5640 1 librados: starting msgr at 2024-08-19T12:53:50.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.120+0000 7f3df9ad5640 1 librados: starting objecter 2024-08-19T12:53:50.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.120+0000 7f3df9ad5640 1 Processor -- start 2024-08-19T12:53:50.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.120+0000 7f3df9ad5640 1 -- start start 2024-08-19T12:53:50.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.120+0000 7f3df9ad5640 1 librados: setting wanted keys 2024-08-19T12:53:50.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.120+0000 7f3df9ad5640 1 librados: calling monclient init 2024-08-19T12:53:50.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.120+0000 7f3df9ad5640 10 monclient: init 2024-08-19T12:53:50.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.121+0000 7f3df9ad5640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:50.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.121+0000 7f3df9ad5640 10 monclient: _add_conns ranks=[1,7,0,5,3,6,8,2,4] 2024-08-19T12:53:50.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.121+0000 7f3df9ad5640 10 monclient(hunting): picked mon.noname-f con 0x7f3df4073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:50.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.121+0000 7f3df9ad5640 10 monclient(hunting): picked mon.noname-i con 0x7f3df40a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:50.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.121+0000 7f3df9ad5640 10 monclient(hunting): picked mon.noname-a con 0x7f3df413f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:50.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.121+0000 7f3df9ad5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3df413f0b0 con 0x7f3df413f2e0 2024-08-19T12:53:50.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.121+0000 7f3df9ad5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3df4142bc0 con 0x7f3df4073470 2024-08-19T12:53:50.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.121+0000 7f3df9ad5640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3df412f260 con 0x7f3df40a57a0 2024-08-19T12:53:50.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.121+0000 7f3df9ad5640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:50.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.121+0000 7f3df9ad5640 10 monclient(hunting): authenticate will time out at 1624.208308s 2024-08-19T12:53:50.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.121+0000 7f3df3fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3df413f2e0 0x7f3df41d60b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43320/0 (socket says 172.21.15.26:43320) 2024-08-19T12:53:50.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.121+0000 7f3df3fff640 1 -- 172.21.15.26:0/2054021368 learned_addr learned my addr 172.21.15.26:0/2054021368 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:50.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df3fff640 1 -- 172.21.15.26:0/2054021368 >> v1:172.21.15.26:6789/0 conn(0x7f3df413f2e0 legacy=0x7f3df41d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 client.?.objecter ms_handle_connect 0x7f3df413f2e0 2024-08-19T12:53:50.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:50.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 client.?.objecter ms_handle_connect 0x7f3df40a57a0 2024-08-19T12:53:50.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:50.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 client.?.objecter ms_handle_connect 0x7f3df4073470 2024-08-19T12:53:50.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:50.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3debfff640 1 -- 172.21.15.26:0/2054021368 >> v1:172.21.15.104:6792/0 conn(0x7f3df40a57a0 legacy=0x7f3df41d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df37fe640 1 -- 172.21.15.26:0/2054021368 >> v1:172.21.15.104:6789/0 conn(0x7f3df4073470 legacy=0x7f3df4132af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2423896317 0 0) 0x7f3df413f0b0 con 0x7f3df413f2e0 2024-08-19T12:53:50.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.141 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): my global_id is 194112 2024-08-19T12:53:50.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dd4004eb0 con 0x7f3df413f2e0 2024-08-19T12:53:50.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2239833887 0 0) 0x7f3df412f260 con 0x7f3df40a57a0 2024-08-19T12:53:50.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): my global_id is 136069 2024-08-19T12:53:50.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3df413f0b0 con 0x7f3df40a57a0 2024-08-19T12:53:50.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1794064492 0 0) 0x7f3df4142bc0 con 0x7f3df4073470 2024-08-19T12:53:50.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): my global_id is 184312 2024-08-19T12:53:50.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3df412f260 con 0x7f3df4073470 2024-08-19T12:53:50.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 421500084 0 0) 0x7f3dd4004eb0 con 0x7f3df413f2e0 2024-08-19T12:53:50.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3df4142bc0 con 0x7f3df413f2e0 2024-08-19T12:53:50.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dec001ee0 con 0x7f3df413f2e0 2024-08-19T12:53:50.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:50.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.123+0000 7f3df0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:53:50.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient(hunting): dump: 2024-08-19T12:53:50.144 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.144 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.144 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.144 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.144 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.145 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.145 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.145 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.145 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.145 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.145 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.145 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.145 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.145 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.145 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.145 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:50.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1302538862 0 0) 0x7f3df413f0b0 con 0x7f3df40a57a0 2024-08-19T12:53:50.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3dd4004eb0 con 0x7f3df40a57a0 2024-08-19T12:53:50.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3ddc001cf0 con 0x7f3df40a57a0 2024-08-19T12:53:50.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:50.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:53:50.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient(hunting): dump: 2024-08-19T12:53:50.146 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.146 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.146 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.147 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.147 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.147 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.147 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.147 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.147 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.147 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.147 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.148 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.148 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.148 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.148 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.148 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1300939640 0 0) 0x7f3df412f260 con 0x7f3df4073470 2024-08-19T12:53:50.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3df413f0b0 con 0x7f3df4073470 2024-08-19T12:53:50.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3de4003610 con 0x7f3df4073470 2024-08-19T12:53:50.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:50.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:50.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient(hunting): dump: 2024-08-19T12:53:50.148 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.149 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.149 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.149 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.149 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.149 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.149 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.149 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.149 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.150 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.150 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.150 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.150 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1913968347 0 0) 0x7f3df4142bc0 con 0x7f3df413f2e0 2024-08-19T12:53:50.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 >> v1:172.21.15.104:6792/0 conn(0x7f3df40a57a0 legacy=0x7f3df41d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 >> v1:172.21.15.104:6789/0 conn(0x7f3df4073470 legacy=0x7f3df4132af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:53:50.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 monclient: found mon.a 2024-08-19T12:53:50.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:50.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:50.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df41315c0 con 0x7f3df413f2e0 2024-08-19T12:53:50.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient: _finish_auth 0 2024-08-19T12:53:50.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:53:50.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:50.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df9ad5640 5 monclient: authenticate success, global_id 194112 2024-08-19T12:53:50.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df9ad5640 10 monclient: _renew_subs 2024-08-19T12:53:50.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df9ad5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:50.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df9ad5640 1 -- 172.21.15.26:0/2054021368 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3df4130d20 con 0x7f3df413f2e0 2024-08-19T12:53:50.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df9ad5640 10 client.194112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:50.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df9ad5640 10 monclient: _renew_subs 2024-08-19T12:53:50.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df9ad5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:50.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df9ad5640 1 -- 172.21.15.26:0/2054021368 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3df4133200 con 0x7f3df413f2e0 2024-08-19T12:53:50.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3dec002080 con 0x7f3df413f2e0 2024-08-19T12:53:50.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df9ad5640 1 librados: init done 2024-08-19T12:53:50.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:50.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dec005830 con 0x7f3df413f2e0 2024-08-19T12:53:50.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:50.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df0ff9640 10 monclient: dump: 2024-08-19T12:53:50.153 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.153 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.153 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.153 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.153 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.153 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.153 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.153 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.154 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.154 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.154 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.154 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.154 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.154 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.124+0000 7f3df27fc640 4 set_mon_vals no callback set 2024-08-19T12:53:50.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.125+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3dec011c90 con 0x7f3df413f2e0 2024-08-19T12:53:50.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.125+0000 7f3df9ad5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:50.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.125+0000 7f3df9ad5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:50.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.125+0000 7f3df9ad5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:50.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.125+0000 7f3df9ad5640 1 -- 172.21.15.26:0/2054021368 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3df40008d0 con 0x7f3df413f2e0 2024-08-19T12:53:50.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.125+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(567..567 src has 350..567) ==== 5849+0+0 (unknown 4291685512 0 0) 0x7f3dec04e5c0 con 0x7f3df413f2e0 2024-08-19T12:53:50.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.125+0000 7f3df0ff9640 10 client.194112.objecter ms_dispatch 0x7f3df4000b90 osd_map(567..567 src has 350..567) 2024-08-19T12:53:50.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.125+0000 7f3df0ff9640 3 client.194112.objecter handle_osd_map got epochs [567,567] > 0 2024-08-19T12:53:50.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.125+0000 7f3df0ff9640 3 client.194112.objecter handle_osd_map decoding full epoch 567 2024-08-19T12:53:50.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.126+0000 7f3df0ff9640 20 client.194112.objecter dump_active .. 0 homeless 2024-08-19T12:53:50.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.126+0000 7f3df37fe640 10 monclient: get_auth_request con 0x7f3dd4041990 auth_method 0 2024-08-19T12:53:50.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.128+0000 7f3df0ff9640 10 client.194112.objecter ms_handle_connect 0x7f3dd4041990 2024-08-19T12:53:50.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.130+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.0 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f3dec01b610 con 0x7f3df413f2e0 2024-08-19T12:53:50.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.130+0000 7f3df0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:50.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.130+0000 7f3df0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:50.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.386+0000 7f3df9ad5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:53:50.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.386+0000 7f3df9ad5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:53:50.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.386+0000 7f3df9ad5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:50.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.386+0000 7f3df9ad5640 1 -- 172.21.15.26:0/2054021368 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3df4142df0 con 0x7f3df413f2e0 2024-08-19T12:53:50.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.386+0000 7f3df0ff9640 1 -- 172.21.15.26:0/2054021368 <== mon.0 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2730 (unknown 254746657 0 1216025970) 0x7f3dec0200c0 con 0x7f3df413f2e0 2024-08-19T12:53:50.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.386+0000 7f3df0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:53:50.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.386+0000 7f3df0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:50.387 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":110,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:53:50.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.388+0000 7f3df9ad5640 10 librados: watch_flush enter 2024-08-19T12:53:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.388+0000 7f3df9ad5640 10 librados: watch_flush exit 2024-08-19T12:53:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.388+0000 7f3df9ad5640 20 client.194112.objecter shutdown clearing up homeless session... 2024-08-19T12:53:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.388+0000 7f3df9ad5640 10 client.194112.objecter successfully canceled tick 2024-08-19T12:53:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.388+0000 7f3df9ad5640 1 -- 172.21.15.26:0/2054021368 >> v1:172.21.15.104:6800/899887506 conn(0x7f3dd4041990 legacy=0x7f3dd4043e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.388+0000 7f3df9ad5640 10 monclient: shutdown 2024-08-19T12:53:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.388+0000 7f3df9ad5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.388+0000 7f3df9ad5640 1 -- 172.21.15.26:0/2054021368 >> v1:172.21.15.26:6789/0 conn(0x7f3df413f2e0 legacy=0x7f3df41d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.388+0000 7f3df9ad5640 1 -- 172.21.15.26:0/2054021368 shutdown_connections 2024-08-19T12:53:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.388+0000 7f3df9ad5640 1 -- 172.21.15.26:0/2054021368 >> 172.21.15.26:0/2054021368 conn(0x7f3df40abff0 msgr2=0x7f3df40a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.389+0000 7f3df9ad5640 1 -- 172.21.15.26:0/2054021368 shutdown_connections 2024-08-19T12:53:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.389+0000 7f3df9ad5640 1 -- 172.21.15.26:0/2054021368 wait complete. 2024-08-19T12:53:50.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.389+0000 7f3df9ad5640 1 librados: shutdown 2024-08-19T12:53:50.400 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:53:50.400 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:53:50.400 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:53:50.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.500+0000 7f3110ac5640 10 monclient: get_monmap_and_config 2024-08-19T12:53:50.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.500+0000 7f3110ac5640 10 monclient: build_initial_monmap 2024-08-19T12:53:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.500+0000 7f3110ac5640 10 monclient: monmap: 2024-08-19T12:53:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:50.500905+0000 2024-08-19T12:53:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:50.500905+0000 2024-08-19T12:53:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:50.502 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:50.503 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:50.503 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:50.503 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:50.503 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:50.503 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:50.503 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.500+0000 7f3110ac5640 1 Processor -- start 2024-08-19T12:53:50.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.500+0000 7f3110ac5640 1 -- start start 2024-08-19T12:53:50.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.500+0000 7f3110ac5640 10 monclient: init 2024-08-19T12:53:50.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.501+0000 7f3110ac5640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:50.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.501+0000 7f3110ac5640 10 monclient: _add_conns ranks=[5,7,4,6,8,2,0,1,3] 2024-08-19T12:53:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.501+0000 7f3110ac5640 10 monclient(hunting): picked mon.noname-h con 0x7f310c073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:53:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.501+0000 7f3110ac5640 10 monclient(hunting): picked mon.noname-i con 0x7f310c0a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.501+0000 7f3110ac5640 10 monclient(hunting): picked mon.noname-c con 0x7f310c13f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.501+0000 7f3110ac5640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f310c09dae0 con 0x7f310c13f2e0 2024-08-19T12:53:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.501+0000 7f3110ac5640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f310c136910 con 0x7f310c073470 2024-08-19T12:53:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.501+0000 7f3110ac5640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f310c09dd50 con 0x7f310c0a57a0 2024-08-19T12:53:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.501+0000 7f3110ac5640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.501+0000 7f3110ac5640 10 monclient(hunting): authenticate will time out at 1624.588336s 2024-08-19T12:53:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.501+0000 7f310ad9d640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f310c13f2e0 0x7f310c1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59136/0 (socket says 172.21.15.26:59136) 2024-08-19T12:53:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.501+0000 7f310ad9d640 1 -- 172.21.15.26:0/2427786811 learned_addr learned my addr 172.21.15.26:0/2427786811 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:50.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.503+0000 7f310ad9d640 1 -- 172.21.15.26:0/2427786811 >> v1:172.21.15.26:6791/0 conn(0x7f310c13f2e0 legacy=0x7f310c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.503+0000 7f3109d9b640 1 -- 172.21.15.26:0/2427786811 >> v1:172.21.15.104:6792/0 conn(0x7f310c0a57a0 legacy=0x7f310c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.503+0000 7f310a59c640 1 -- 172.21.15.26:0/2427786811 >> v1:172.21.15.104:6791/0 conn(0x7f310c073470 legacy=0x7f310c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.503+0000 7f310959a640 1 -- 172.21.15.26:0/2427786811 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3327598952 0 0) 0x7f310c09dae0 con 0x7f310c13f2e0 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.503+0000 7f310959a640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.503+0000 7f310959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.503+0000 7f310959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.503+0000 7f310959a640 10 monclient(hunting): my global_id is 184360 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.503+0000 7f310959a640 1 -- 172.21.15.26:0/2427786811 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30f8003520 con 0x7f310c13f2e0 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.503+0000 7f310959a640 1 -- 172.21.15.26:0/2427786811 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3293981816 0 0) 0x7f310c09dd50 con 0x7f310c0a57a0 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient(hunting): my global_id is 136078 2024-08-19T12:53:50.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 1 -- 172.21.15.26:0/2427786811 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f310c09dae0 con 0x7f310c0a57a0 2024-08-19T12:53:50.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 1 -- 172.21.15.26:0/2427786811 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 319037163 0 0) 0x7f30f8003520 con 0x7f310c13f2e0 2024-08-19T12:53:50.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 1 -- 172.21.15.26:0/2427786811 >> v1:172.21.15.104:6792/0 conn(0x7f310c0a57a0 legacy=0x7f310c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 1 -- 172.21.15.26:0/2427786811 >> v1:172.21.15.104:6791/0 conn(0x7f310c073470 legacy=0x7f310c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient: _finish_hunting 0 2024-08-19T12:53:50.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 1 monclient: found mon.noname-c 2024-08-19T12:53:50.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:50.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:53:50.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 1 -- 172.21.15.26:0/2427786811 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310c09e030 con 0x7f310c13f2e0 2024-08-19T12:53:50.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient: _finish_auth 0 2024-08-19T12:53:50.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient: _check_auth_tickets 2024-08-19T12:53:50.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 1 -- 172.21.15.26:0/2427786811 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3104001ae0 con 0x7f310c13f2e0 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient: dump: 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.511 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.512 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.512 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.512 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.512 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.512 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.512 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f3110ac5640 5 monclient: authenticate success, global_id 184360 2024-08-19T12:53:50.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f3110ac5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:50.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 1 -- 172.21.15.26:0/2427786811 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3104001d30 con 0x7f310c13f2e0 2024-08-19T12:53:50.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:50.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 1 -- 172.21.15.26:0/2427786811 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3104004be0 con 0x7f310c13f2e0 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f310959a640 10 monclient: dump: 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.513 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.514 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.514 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.514 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.514 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f3110ac5640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:50.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f3110ac5640 4 set_mon_vals no callback set 2024-08-19T12:53:50.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f3110ac5640 10 monclient: shutdown 2024-08-19T12:53:50.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f3110ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:50.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f3110ac5640 1 -- 172.21.15.26:0/2427786811 >> v1:172.21.15.26:6791/0 conn(0x7f310c13f2e0 legacy=0x7f310c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f3110ac5640 1 -- 172.21.15.26:0/2427786811 shutdown_connections 2024-08-19T12:53:50.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f3110ac5640 1 -- 172.21.15.26:0/2427786811 >> 172.21.15.26:0/2427786811 conn(0x7f310c0abff0 msgr2=0x7f310c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:50.514 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f3110ac5640 1 -- 172.21.15.26:0/2427786811 shutdown_connections 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.504+0000 7f3110ac5640 1 -- 172.21.15.26:0/2427786811 wait complete. 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.505+0000 7f3110ac5640 10 monclient: build_initial_monmap 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.505+0000 7f3110ac5640 10 monclient: monmap: 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.505+0000 7f3110ac5640 1 librados: starting msgr at 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.505+0000 7f3110ac5640 1 librados: starting objecter 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.505+0000 7f3110ac5640 1 Processor -- start 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.505+0000 7f3110ac5640 1 -- start start 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.505+0000 7f3110ac5640 1 librados: setting wanted keys 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.505+0000 7f3110ac5640 1 librados: calling monclient init 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.505+0000 7f3110ac5640 10 monclient: init 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.506+0000 7f3110ac5640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.506+0000 7f3110ac5640 10 monclient: _add_conns ranks=[2,3,1,5,8,4,0,7,6] 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.506+0000 7f3110ac5640 10 monclient(hunting): picked mon.noname-b con 0x7f310c073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.506+0000 7f3110ac5640 10 monclient(hunting): picked mon.noname-g con 0x7f310c0a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:53:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.506+0000 7f3110ac5640 10 monclient(hunting): picked mon.noname-f con 0x7f310c13f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.506+0000 7f3110ac5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f310c137490 con 0x7f310c13f2e0 2024-08-19T12:53:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.506+0000 7f3110ac5640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f310c137070 con 0x7f310c073470 2024-08-19T12:53:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.506+0000 7f3110ac5640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f310c13f070 con 0x7f310c0a57a0 2024-08-19T12:53:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.506+0000 7f3110ac5640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.506+0000 7f3110ac5640 10 monclient(hunting): authenticate will time out at 1624.593408s 2024-08-19T12:53:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.506+0000 7f310a59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f310c073470 0x7f310c0a1f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47764/0 (socket says 172.21.15.26:47764) 2024-08-19T12:53:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.506+0000 7f310a59c640 1 -- 172.21.15.26:0/2699966737 learned_addr learned my addr 172.21.15.26:0/2699966737 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:50.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f310a59c640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.26:6790/0 conn(0x7f310c073470 legacy=0x7f310c0a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3102ffd640 10 client.?.objecter ms_handle_connect 0x7f310c073470 2024-08-19T12:53:50.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3102ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:50.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3102ffd640 10 client.?.objecter ms_handle_connect 0x7f310c0a57a0 2024-08-19T12:53:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3109d9b640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.104:6790/0 conn(0x7f310c0a57a0 legacy=0x7f310c0a26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3102ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3102ffd640 10 client.?.objecter ms_handle_connect 0x7f310c13f2e0 2024-08-19T12:53:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3102ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f310ad9d640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.104:6789/0 conn(0x7f310c13f2e0 legacy=0x7f310c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3819127810 0 0) 0x7f310c137070 con 0x7f310c073470 2024-08-19T12:53:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3102ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3102ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3102ffd640 10 monclient(hunting): my global_id is 27065 2024-08-19T12:53:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.508+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30ec004eb0 con 0x7f310c073470 2024-08-19T12:53:50.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2836239653 0 0) 0x7f310c09e030 con 0x7f310c13f2e0 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient(hunting): my global_id is 184321 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f310c137070 con 0x7f310c13f2e0 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4012278090 0 0) 0x7f30ec004eb0 con 0x7f310c073470 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f310c09e030 con 0x7f310c073470 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3142011529 0 0) 0x7f310c13f070 con 0x7f310c0a57a0 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient(hunting): my global_id is 194106 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30ec004eb0 con 0x7f310c0a57a0 2024-08-19T12:53:50.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f30fc003990 con 0x7f310c073470 2024-08-19T12:53:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:53:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient(hunting): dump: 2024-08-19T12:53:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.526 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.527 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1131290347 0 0) 0x7f310c09e030 con 0x7f310c073470 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.104:6790/0 conn(0x7f310c0a57a0 legacy=0x7f310c0a26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.104:6789/0 conn(0x7f310c13f2e0 legacy=0x7f310c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 monclient: found mon.b 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310c0a3940 con 0x7f310c073470 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient: _finish_auth 0 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3110ac5640 5 monclient: authenticate success, global_id 27065 2024-08-19T12:53:50.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3110ac5640 10 monclient: _renew_subs 2024-08-19T12:53:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3110ac5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:53:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3110ac5640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f310c0a2db0 con 0x7f310c073470 2024-08-19T12:53:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3110ac5640 10 client.27065.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3110ac5640 10 monclient: _renew_subs 2024-08-19T12:53:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3110ac5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:53:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3110ac5640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f310c13e750 con 0x7f310c073470 2024-08-19T12:53:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3110ac5640 1 librados: init done 2024-08-19T12:53:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f30fc003ee0 con 0x7f310c073470 2024-08-19T12:53:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f30fc004410 con 0x7f310c073470 2024-08-19T12:53:50.529 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3108d99640 4 set_mon_vals no callback set 2024-08-19T12:53:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.509+0000 7f3102ffd640 10 monclient: dump: 2024-08-19T12:53:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.530 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.531 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.510+0000 7f3110ac5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.510+0000 7f3110ac5640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f30f00065c0 con 0x7f30f0002860 2024-08-19T12:53:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.510+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f30fc0045b0 con 0x7f310c073470 2024-08-19T12:53:50.531 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.510+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(567..567 src has 350..567) ==== 5849+0+0 (unknown 4291685512 0 0) 0x7f30fc04e7d0 con 0x7f310c073470 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.510+0000 7f3102ffd640 10 client.27065.objecter ms_dispatch 0x7f310c000b90 osd_map(567..567 src has 350..567) 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.510+0000 7f3102ffd640 3 client.27065.objecter handle_osd_map got epochs [567,567] > 0 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.510+0000 7f3102ffd640 3 client.27065.objecter handle_osd_map decoding full epoch 567 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.510+0000 7f3102ffd640 20 client.27065.objecter dump_active .. 0 homeless 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.510+0000 7f310ad9d640 10 monclient: get_auth_request con 0x7f30ec041230 auth_method 0 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 10 client.27065.objecter ms_handle_connect 0x7f30f0002860 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 10 client.27065.objecter resend_mon_ops 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3109d9b640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.26:6789/0 conn(0x7f30f0002860 legacy=0x7f30f0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3901579569 0 0) 0x7f30f00065c0 con 0x7f30f0002860 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 10 monclient(hunting): my global_id is 194121 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f310c09e030 con 0x7f30f0002860 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3700249426 0 0) 0x7f310c09e030 con 0x7f30f0002860 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30f00065c0 con 0x7f30f0002860 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f30f4001c60 con 0x7f30f0002860 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 10 monclient: dump: 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.533 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1630441534 0 0) 0x7f30f00065c0 con 0x7f30f0002860 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30f000a160 con 0x7f30f0002860 2024-08-19T12:53:50.534 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.512+0000 7f3102ffd640 10 client.27065.objecter ms_handle_connect 0x7f30ec041230 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.513+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f30f40057c0 con 0x7f30f0002860 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.513+0000 7f3102ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.513+0000 7f3102ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.513+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.26:6789/0 conn(0x7f30f0002860 legacy=0x7f30f0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.513+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.26:6789/0 conn(0x7f30f0002860 legacy=0x7f30f0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.519+0000 7f3110ac5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.519+0000 7f3110ac5640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f30f0009e50 con 0x7f30f0009110 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.521+0000 7f3109d9b640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.26:6789/0 conn(0x7f30f0009110 legacy=0x7f30f00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.521+0000 7f3102ffd640 10 client.27065.objecter ms_handle_connect 0x7f30f0009110 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.521+0000 7f3102ffd640 10 client.27065.objecter resend_mon_ops 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.521+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1228681362 0 0) 0x7f30f0009e50 con 0x7f30f0009110 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.521+0000 7f3102ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.535 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.521+0000 7f3102ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.521+0000 7f3102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.521+0000 7f3102ffd640 10 monclient(hunting): my global_id is 194130 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.521+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30f00065c0 con 0x7f30f0009110 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 89901908 0 0) 0x7f30f00065c0 con 0x7f30f0009110 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30f0009e50 con 0x7f30f0009110 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f30f4003d40 con 0x7f30f0009110 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 10 monclient: dump: 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.536 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1159906257 0 0) 0x7f30f0009e50 con 0x7f30f0009110 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f30f000c3a0 con 0x7f30f0009110 2024-08-19T12:53:50.537 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7546 (unknown 0 0 2080954619) 0x7f30f4003fc0 con 0x7f30f0009110 2024-08-19T12:53:50.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:50.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:50.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.26:6789/0 conn(0x7f30f0009110 legacy=0x7f30f00095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.538 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3102ffd640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.26:6789/0 conn(0x7f30f0009110 legacy=0x7f30f00095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:50.538 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:50.538 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:50.538 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:50.538 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:53:50.538 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 15444, 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.539 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:50.540 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:50.541 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:50.542 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:50.543 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:50.544 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.545 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.546 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:50.547 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:50.547 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.547 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.547 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.547 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:50.547 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:50.547 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:50.549 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:50.550 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.551 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.552 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:50.553 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.556 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.557 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:50.558 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:50.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3110ac5640 10 librados: watch_flush enter 2024-08-19T12:53:50.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3110ac5640 10 librados: watch_flush exit 2024-08-19T12:53:50.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3110ac5640 20 client.27065.objecter shutdown clearing up homeless session... 2024-08-19T12:53:50.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3110ac5640 10 client.27065.objecter successfully canceled tick 2024-08-19T12:53:50.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3110ac5640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.104:6800/899887506 conn(0x7f30ec041230 legacy=0x7f30ec0436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3110ac5640 10 monclient: shutdown 2024-08-19T12:53:50.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3110ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:50.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3110ac5640 1 -- 172.21.15.26:0/2699966737 >> v1:172.21.15.26:6790/0 conn(0x7f310c073470 legacy=0x7f310c0a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f310ad9d640 1 -- 172.21.15.26:0/2699966737 reap_dead start 2024-08-19T12:53:50.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f310ad9d640 1 -- 172.21.15.26:0/2699966737 reap_dead start 2024-08-19T12:53:50.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3110ac5640 1 -- 172.21.15.26:0/2699966737 shutdown_connections 2024-08-19T12:53:50.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3110ac5640 1 -- 172.21.15.26:0/2699966737 >> 172.21.15.26:0/2699966737 conn(0x7f310c0abff0 msgr2=0x7f310c1338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:50.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3110ac5640 1 -- 172.21.15.26:0/2699966737 shutdown_connections 2024-08-19T12:53:50.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.522+0000 7f3110ac5640 1 -- 172.21.15.26:0/2699966737 wait complete. 2024-08-19T12:53:50.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.523+0000 7f3110ac5640 1 librados: shutdown 2024-08-19T12:53:50.560 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:50.561 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:53:50.561 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:53:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.660+0000 7fec910da640 10 monclient: get_monmap_and_config 2024-08-19T12:53:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.660+0000 7fec910da640 10 monclient: build_initial_monmap 2024-08-19T12:53:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.660+0000 7fec910da640 10 monclient: monmap: 2024-08-19T12:53:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:50.661058+0000 2024-08-19T12:53:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:50.661058+0000 2024-08-19T12:53:50.661 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:50.662 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:50.662 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.660+0000 7fec910da640 1 Processor -- start 2024-08-19T12:53:50.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.660+0000 7fec910da640 1 -- start start 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.660+0000 7fec910da640 10 monclient: init 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.661+0000 7fec910da640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.661+0000 7fec910da640 10 monclient: _add_conns ranks=[1,7,5,8,4,3,6,2,0] 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.661+0000 7fec910da640 10 monclient(hunting): picked mon.noname-f con 0x7fec8c06e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.661+0000 7fec910da640 10 monclient(hunting): picked mon.noname-i con 0x7fec8c0a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.661+0000 7fec910da640 10 monclient(hunting): picked mon.noname-h con 0x7fec8c0a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.661+0000 7fec910da640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec8c136790 con 0x7fec8c06e7b0 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.661+0000 7fec910da640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec8c09ddf0 con 0x7fec8c0a5d50 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.661+0000 7fec910da640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec8c09e0d0 con 0x7fec8c0a5920 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.661+0000 7fec910da640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.661+0000 7fec910da640 10 monclient(hunting): authenticate will time out at 1624.748372s 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.661+0000 7fec8b59e640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7fec8c0a5d50 0x7fec8c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:38630/0 (socket says 172.21.15.26:38630) 2024-08-19T12:53:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.661+0000 7fec8b59e640 1 -- 172.21.15.26:0/936566071 learned_addr learned my addr 172.21.15.26:0/936566071 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.663+0000 7fec8b59e640 1 -- 172.21.15.26:0/936566071 >> v1:172.21.15.104:6791/0 conn(0x7fec8c0a5d50 legacy=0x7fec8c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.663+0000 7fec8a59c640 1 -- 172.21.15.26:0/936566071 >> v1:172.21.15.104:6792/0 conn(0x7fec8c0a5920 legacy=0x7fec8c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.663+0000 7fec8ad9d640 1 -- 172.21.15.26:0/936566071 >> v1:172.21.15.104:6789/0 conn(0x7fec8c06e7b0 legacy=0x7fec8c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2941377501 0 0) 0x7fec840017d0 con 0x7fec8c0a5d50 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient(hunting): my global_id is 184343 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec74003520 con 0x7fec8c0a5d50 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1792990577 0 0) 0x7fec8c136790 con 0x7fec8c06e7b0 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient(hunting): my global_id is 184330 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec840017d0 con 0x7fec8c06e7b0 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4010857020 0 0) 0x7fec8c09e0d0 con 0x7fec8c0a5920 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient(hunting): my global_id is 136087 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec8c136790 con 0x7fec8c0a5920 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3667897103 0 0) 0x7fec840017d0 con 0x7fec8c06e7b0 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 >> v1:172.21.15.104:6792/0 conn(0x7fec8c0a5920 legacy=0x7fec8c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 >> v1:172.21.15.104:6791/0 conn(0x7fec8c0a5d50 legacy=0x7fec8c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 monclient: found mon.noname-f 2024-08-19T12:53:50.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8c09e3b0 con 0x7fec8c06e7b0 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient: _finish_auth 0 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fec7c001a40 con 0x7fec8c06e7b0 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec910da640 5 monclient: authenticate success, global_id 184330 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec910da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient: dump: 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.671 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.672 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec910da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec7c001d10 con 0x7fec8c06e7b0 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 1 -- 172.21.15.26:0/936566071 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fec7c003950 con 0x7fec8c06e7b0 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec89d9b640 10 monclient: dump: 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.673 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec910da640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec910da640 4 set_mon_vals no callback set 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec910da640 10 monclient: shutdown 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec910da640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:50.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec910da640 1 -- 172.21.15.26:0/936566071 >> v1:172.21.15.104:6789/0 conn(0x7fec8c06e7b0 legacy=0x7fec8c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.664+0000 7fec910da640 1 -- 172.21.15.26:0/936566071 shutdown_connections 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.665+0000 7fec910da640 1 -- 172.21.15.26:0/936566071 >> 172.21.15.26:0/936566071 conn(0x7fec8c0a8740 msgr2=0x7fec8c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.665+0000 7fec910da640 1 -- 172.21.15.26:0/936566071 shutdown_connections 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.665+0000 7fec910da640 1 -- 172.21.15.26:0/936566071 wait complete. 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.665+0000 7fec910da640 10 monclient: build_initial_monmap 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.665+0000 7fec910da640 10 monclient: monmap: 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.675 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.665+0000 7fec910da640 1 librados: starting msgr at 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.665+0000 7fec910da640 1 librados: starting objecter 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 1 Processor -- start 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 1 -- start start 2024-08-19T12:53:50.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 1 librados: setting wanted keys 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 1 librados: calling monclient init 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 10 monclient: init 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 10 monclient: _add_conns ranks=[0,8,5,4,3,1,7,6,2] 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 10 monclient(hunting): picked mon.noname-a con 0x7fec8c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 10 monclient(hunting): picked mon.noname-e con 0x7fec8c0a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 10 monclient(hunting): picked mon.noname-h con 0x7fec8c0a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec8c13be30 con 0x7fec8c06e7b0 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec8c13b470 con 0x7fec8c0a5d50 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec8c134970 con 0x7fec8c0a5920 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec910da640 10 monclient(hunting): authenticate will time out at 1624.753834s 2024-08-19T12:53:50.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec8a59c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fec8c0a5920 0x7fec8c1381f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40610/0 (socket says 172.21.15.26:40610) 2024-08-19T12:53:50.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.666+0000 7fec8a59c640 1 -- 172.21.15.26:0/1075484586 learned_addr learned my addr 172.21.15.26:0/1075484586 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:50.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec8ad9d640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.26:6789/0 conn(0x7fec8c06e7b0 legacy=0x7fec8c137ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec8a59c640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.26:6793/0 conn(0x7fec8c0a5920 legacy=0x7fec8c1381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 client.?.objecter ms_handle_connect 0x7fec8c06e7b0 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 client.?.objecter ms_handle_connect 0x7fec8c0a5920 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec8b59e640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.104:6791/0 conn(0x7fec8c0a5d50 legacy=0x7fec8c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 client.?.objecter ms_handle_connect 0x7fec8c0a5d50 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1047796489 0 0) 0x7fec8c09e3b0 con 0x7fec8c06e7b0 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): my global_id is 194139 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec78004eb0 con 0x7fec8c06e7b0 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883027309 0 0) 0x7fec8c134970 con 0x7fec8c0a5920 2024-08-19T12:53:50.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): my global_id is 184265 2024-08-19T12:53:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec8c09e3b0 con 0x7fec8c0a5920 2024-08-19T12:53:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1230794231 0 0) 0x7fec8c13b470 con 0x7fec8c0a5d50 2024-08-19T12:53:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): my global_id is 184352 2024-08-19T12:53:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec8c134970 con 0x7fec8c0a5d50 2024-08-19T12:53:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2993862769 0 0) 0x7fec78004eb0 con 0x7fec8c06e7b0 2024-08-19T12:53:50.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec8c13b470 con 0x7fec8c06e7b0 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fec7c003950 con 0x7fec8c06e7b0 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): dump: 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.687 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2778482984 0 0) 0x7fec8c09e3b0 con 0x7fec8c0a5920 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec78004eb0 con 0x7fec8c0a5920 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fec80001d30 con 0x7fec8c0a5920 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:50.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): dump: 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.689 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1041879169 0 0) 0x7fec8c134970 con 0x7fec8c0a5d50 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec8c09e3b0 con 0x7fec8c0a5d50 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fec84001e10 con 0x7fec8c0a5d50 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient(hunting): dump: 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.690 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3351078634 0 0) 0x7fec8c13b470 con 0x7fec8c06e7b0 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.26:6793/0 conn(0x7fec8c0a5920 legacy=0x7fec8c1381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.104:6791/0 conn(0x7fec8c0a5d50 legacy=0x7fec8c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient: _finish_hunting 0 2024-08-19T12:53:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 monclient: found mon.a 2024-08-19T12:53:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8c139110 con 0x7fec8c06e7b0 2024-08-19T12:53:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient: _finish_auth 0 2024-08-19T12:53:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient: _check_auth_tickets 2024-08-19T12:53:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec910da640 5 monclient: authenticate success, global_id 194139 2024-08-19T12:53:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec910da640 10 monclient: _renew_subs 2024-08-19T12:53:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec910da640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec910da640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fec8c139cd0 con 0x7fec8c06e7b0 2024-08-19T12:53:50.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec910da640 10 client.194139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec910da640 10 monclient: _renew_subs 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec910da640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec910da640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fec8c13abd0 con 0x7fec8c06e7b0 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec910da640 1 librados: init done 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec7c001a20 con 0x7fec8c06e7b0 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fec7c004240 con 0x7fec8c06e7b0 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec8959a640 4 set_mon_vals no callback set 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.669+0000 7fec737fe640 10 monclient: dump: 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.693 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.694 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.670+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fec7c011a80 con 0x7fec8c06e7b0 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.670+0000 7fec910da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.670+0000 7fec910da640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fec6c0065c0 con 0x7fec6c002860 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.670+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(567..567 src has 350..567) ==== 5849+0+0 (unknown 4291685512 0 0) 0x7fec7c04e280 con 0x7fec8c06e7b0 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.670+0000 7fec737fe640 10 client.194139.objecter ms_dispatch 0x7fec8c000b90 osd_map(567..567 src has 350..567) 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.670+0000 7fec737fe640 3 client.194139.objecter handle_osd_map got epochs [567,567] > 0 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.670+0000 7fec737fe640 3 client.194139.objecter handle_osd_map decoding full epoch 567 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.671+0000 7fec737fe640 20 client.194139.objecter dump_active .. 0 homeless 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.671+0000 7fec8b59e640 10 monclient: get_auth_request con 0x7fec780419f0 auth_method 0 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec8a59c640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.26:6790/0 conn(0x7fec6c002860 legacy=0x7fec6c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 10 client.194139.objecter ms_handle_connect 0x7fec6c002860 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 10 client.194139.objecter resend_mon_ops 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3935280284 0 0) 0x7fec6c0065c0 con 0x7fec6c002860 2024-08-19T12:53:50.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 10 monclient(hunting): my global_id is 27074 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec8c13b470 con 0x7fec6c002860 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 10 client.194139.objecter ms_handle_connect 0x7fec780419f0 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2189245086 0 0) 0x7fec8c13b470 con 0x7fec6c002860 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec6c0065c0 con 0x7fec6c002860 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fec80003ef0 con 0x7fec6c002860 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 10 monclient: dump: 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.696 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.697 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1895870299 0 0) 0x7fec6c0065c0 con 0x7fec6c002860 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec6c00a160 con 0x7fec6c002860 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fec800042a0 con 0x7fec6c002860 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.26:6790/0 conn(0x7fec6c002860 legacy=0x7fec6c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.673+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.26:6790/0 conn(0x7fec6c002860 legacy=0x7fec6c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.680+0000 7fec910da640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.680+0000 7fec910da640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fec6c009e50 con 0x7fec6c009110 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 10 client.194139.objecter ms_handle_connect 0x7fec6c009110 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 10 client.194139.objecter resend_mon_ops 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec8a59c640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.26:6790/0 conn(0x7fec6c009110 legacy=0x7fec6c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.698 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1258843324 0 0) 0x7fec6c009e50 con 0x7fec6c009110 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 10 monclient(hunting): my global_id is 27083 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec6c0065c0 con 0x7fec6c009110 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 934132788 0 0) 0x7fec6c0065c0 con 0x7fec6c009110 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec6c009e50 con 0x7fec6c009110 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fec80005cf0 con 0x7fec6c009110 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 10 monclient: dump: 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.699 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3000999920 0 0) 0x7fec6c009e50 con 0x7fec6c009110 2024-08-19T12:53:50.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.682+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fec6c00c3a0 con 0x7fec6c009110 2024-08-19T12:53:50.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 316423952) 0x7fec800071e0 con 0x7fec6c009110 2024-08-19T12:53:50.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec737fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:50.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec737fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:50.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.26:6790/0 conn(0x7fec6c009110 legacy=0x7fec6c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec737fe640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.26:6790/0 conn(0x7fec6c009110 legacy=0x7fec6c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:50.701 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:50.701 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 943737, 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:50.702 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:50.703 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:50.704 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:50.705 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:50.706 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:50.707 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.708 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.709 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:50.709 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:50.709 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.709 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.709 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.709 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:50.709 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.709 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.709 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.711 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.712 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.714 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:50.715 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.716 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:50.719 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.720 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:50.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.721 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:50.721 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:50.721 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:53:50.721 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.721 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.721 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:50.721 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:50.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 10 librados: watch_flush enter 2024-08-19T12:53:50.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 10 librados: watch_flush exit 2024-08-19T12:53:50.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 20 client.194139.objecter shutdown clearing up homeless session... 2024-08-19T12:53:50.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 10 client.194139.objecter successfully canceled tick 2024-08-19T12:53:50.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.104:6800/899887506 conn(0x7fec780419f0 legacy=0x7fec78043eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 10 monclient: shutdown 2024-08-19T12:53:50.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:50.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 1 -- 172.21.15.26:0/1075484586 >> v1:172.21.15.26:6789/0 conn(0x7fec8c06e7b0 legacy=0x7fec8c137ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec8b59e640 1 -- 172.21.15.26:0/1075484586 reap_dead start 2024-08-19T12:53:50.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec8b59e640 1 -- 172.21.15.26:0/1075484586 reap_dead start 2024-08-19T12:53:50.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 1 -- 172.21.15.26:0/1075484586 shutdown_connections 2024-08-19T12:53:50.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 1 -- 172.21.15.26:0/1075484586 >> 172.21.15.26:0/1075484586 conn(0x7fec8c0a8740 msgr2=0x7fec8c06dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:50.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 1 -- 172.21.15.26:0/1075484586 shutdown_connections 2024-08-19T12:53:50.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 1 -- 172.21.15.26:0/1075484586 wait complete. 2024-08-19T12:53:50.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.683+0000 7fec910da640 1 librados: shutdown 2024-08-19T12:53:50.724 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:50.724 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:53:50.725 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:53:50.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.823+0000 7f0c6edf4640 10 monclient: get_monmap_and_config 2024-08-19T12:53:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.823+0000 7f0c6edf4640 10 monclient: build_initial_monmap 2024-08-19T12:53:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.823+0000 7f0c6edf4640 10 monclient: monmap: 2024-08-19T12:53:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:50.824127+0000 2024-08-19T12:53:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:50.824127+0000 2024-08-19T12:53:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:50.824 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:50.825 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:50.825 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:50.825 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:50.825 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:50.825 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:50.825 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:50.825 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.823+0000 7f0c6edf4640 1 Processor -- start 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6edf4640 1 -- start start 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6edf4640 10 monclient: init 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6edf4640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6edf4640 10 monclient: _add_conns ranks=[3,1,7,0,5,6,2,8,4] 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6edf4640 10 monclient(hunting): picked mon.noname-g con 0x7f0c6806e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6edf4640 10 monclient(hunting): picked mon.noname-f con 0x7f0c680a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6edf4640 10 monclient(hunting): picked mon.noname-i con 0x7f0c680a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6edf4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c68136790 con 0x7f0c680a5920 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6edf4640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6809ddf0 con 0x7f0c6806e7b0 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6edf4640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6809e0d0 con 0x7f0c680a5d50 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6edf4640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6edf4640 10 monclient(hunting): authenticate will time out at 1624.911607s 2024-08-19T12:53:50.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6d391640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f0c680a5d50 0x7f0c68141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:43480/0 (socket says 172.21.15.26:43480) 2024-08-19T12:53:50.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6cb90640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f0c6806e7b0 0x7f0c68136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:52768/0 (socket says 172.21.15.26:52768) 2024-08-19T12:53:50.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c6d391640 1 -- 172.21.15.26:0/2404129118 learned_addr learned my addr 172.21.15.26:0/2404129118 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:50.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.824+0000 7f0c67fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0c680a5920 0x7f0c6813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:51336/0 (socket says 172.21.15.26:51336) 2024-08-19T12:53:50.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c6d391640 1 -- 172.21.15.26:0/2404129118 >> v1:172.21.15.104:6792/0 conn(0x7f0c680a5d50 legacy=0x7f0c68141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c67fff640 1 -- 172.21.15.26:0/2404129118 >> v1:172.21.15.104:6789/0 conn(0x7f0c680a5920 legacy=0x7f0c6813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c6cb90640 1 -- 172.21.15.26:0/2404129118 >> v1:172.21.15.104:6790/0 conn(0x7f0c6806e7b0 legacy=0x7f0c68136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3785948879 0 0) 0x7f0c6809e0d0 con 0x7f0c680a5d50 2024-08-19T12:53:50.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient(hunting): my global_id is 136096 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c58003520 con 0x7f0c680a5d50 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3316658840 0 0) 0x7f0c5c001450 con 0x7f0c6806e7b0 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient(hunting): my global_id is 194115 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c6809e0d0 con 0x7f0c6806e7b0 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2834449421 0 0) 0x7f0c68136790 con 0x7f0c680a5920 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient(hunting): my global_id is 184339 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c5c001450 con 0x7f0c680a5920 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2365644437 0 0) 0x7f0c58003520 con 0x7f0c680a5d50 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 >> v1:172.21.15.104:6790/0 conn(0x7f0c6806e7b0 legacy=0x7f0c68136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 >> v1:172.21.15.104:6789/0 conn(0x7f0c680a5920 legacy=0x7f0c6813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient: _finish_hunting 0 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 monclient: found mon.noname-i 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6809e3b0 con 0x7f0c680a5d50 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient: _finish_auth 0 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient: _check_auth_tickets 2024-08-19T12:53:50.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c60001e60 con 0x7f0c680a5d50 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c6edf4640 5 monclient: authenticate success, global_id 136096 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c6edf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient: dump: 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.834 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c6edf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c60004c20 con 0x7f0c680a5d50 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 1 -- 172.21.15.26:0/2404129118 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c600051d0 con 0x7f0c680a5d50 2024-08-19T12:53:50.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c6edf4640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c6edf4640 4 set_mon_vals no callback set 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.827+0000 7f0c677fe640 10 monclient: dump: 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.836 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.828+0000 7f0c6edf4640 10 monclient: shutdown 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.828+0000 7f0c6edf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.828+0000 7f0c6edf4640 1 -- 172.21.15.26:0/2404129118 >> v1:172.21.15.104:6792/0 conn(0x7f0c680a5d50 legacy=0x7f0c68141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.828+0000 7f0c6edf4640 1 -- 172.21.15.26:0/2404129118 shutdown_connections 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.828+0000 7f0c6edf4640 1 -- 172.21.15.26:0/2404129118 >> 172.21.15.26:0/2404129118 conn(0x7f0c680a8740 msgr2=0x7f0c680a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.828+0000 7f0c6edf4640 1 -- 172.21.15.26:0/2404129118 shutdown_connections 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.828+0000 7f0c6edf4640 1 -- 172.21.15.26:0/2404129118 wait complete. 2024-08-19T12:53:50.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.829+0000 7f0c6edf4640 10 monclient: build_initial_monmap 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.829+0000 7f0c6edf4640 10 monclient: monmap: 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:50.838 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.829+0000 7f0c6edf4640 1 librados: starting msgr at 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.829+0000 7f0c6edf4640 1 librados: starting objecter 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.829+0000 7f0c6edf4640 1 Processor -- start 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.829+0000 7f0c6edf4640 1 -- start start 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.829+0000 7f0c6edf4640 1 librados: setting wanted keys 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.829+0000 7f0c6edf4640 1 librados: calling monclient init 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.829+0000 7f0c6edf4640 10 monclient: init 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.830+0000 7f0c6edf4640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.830+0000 7f0c6edf4640 10 monclient: _add_conns ranks=[0,2,4,5,1,3,8,6,7] 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.830+0000 7f0c6edf4640 10 monclient(hunting): picked mon.noname-a con 0x7f0c6806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:50.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.830+0000 7f0c6edf4640 10 monclient(hunting): picked mon.noname-b con 0x7f0c680a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:50.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.830+0000 7f0c6edf4640 10 monclient(hunting): picked mon.noname-c con 0x7f0c680a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:50.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.830+0000 7f0c6edf4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c6813b390 con 0x7f0c6806e7b0 2024-08-19T12:53:50.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.830+0000 7f0c6edf4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c68142e30 con 0x7f0c680a5920 2024-08-19T12:53:50.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.830+0000 7f0c6edf4640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c68139030 con 0x7f0c680a5d50 2024-08-19T12:53:50.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.830+0000 7f0c6edf4640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:50.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.830+0000 7f0c6edf4640 10 monclient(hunting): authenticate will time out at 1624.917203s 2024-08-19T12:53:50.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.830+0000 7f0c6cb90640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0c6806e7b0 0x7f0c681ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43336/0 (socket says 172.21.15.26:43336) 2024-08-19T12:53:50.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.830+0000 7f0c6cb90640 1 -- 172.21.15.26:0/3972080961 learned_addr learned my addr 172.21.15.26:0/3972080961 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:50.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c6cb90640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.26:6789/0 conn(0x7f0c6806e7b0 legacy=0x7f0c681ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c67fff640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.26:6790/0 conn(0x7f0c680a5920 legacy=0x7f0c681d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 client.?.objecter ms_handle_connect 0x7f0c6806e7b0 2024-08-19T12:53:50.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:50.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 client.?.objecter ms_handle_connect 0x7f0c680a5920 2024-08-19T12:53:50.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:50.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 client.?.objecter ms_handle_connect 0x7f0c680a5d50 2024-08-19T12:53:50.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:50.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c6d391640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.26:6791/0 conn(0x7f0c680a5d50 legacy=0x7f0c681d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 955881251 0 0) 0x7f0c68142e30 con 0x7f0c680a5920 2024-08-19T12:53:50.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): my global_id is 27092 2024-08-19T12:53:50.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c4c004eb0 con 0x7f0c680a5920 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1910195638 0 0) 0x7f0c6813b390 con 0x7f0c6806e7b0 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): my global_id is 194148 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c68142e30 con 0x7f0c6806e7b0 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1941135967 0 0) 0x7f0c68139030 con 0x7f0c680a5d50 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): my global_id is 184369 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c6813b390 con 0x7f0c680a5d50 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1029835122 0 0) 0x7f0c6813b390 con 0x7f0c680a5d50 2024-08-19T12:53:50.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c68139030 con 0x7f0c680a5d50 2024-08-19T12:53:50.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c600020b0 con 0x7f0c680a5d50 2024-08-19T12:53:50.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:50.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:53:50.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): dump: 2024-08-19T12:53:50.848 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.848 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.848 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.848 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.848 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.848 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.848 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.848 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.832+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1998898094 0 0) 0x7f0c68142e30 con 0x7f0c6806e7b0 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c6813b390 con 0x7f0c6806e7b0 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3000395804 0 0) 0x7f0c4c004eb0 con 0x7f0c680a5920 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c68142e30 con 0x7f0c680a5920 2024-08-19T12:53:50.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c5c004b20 con 0x7f0c6806e7b0 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient(hunting): dump: 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.850 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c54001c70 con 0x7f0c680a5920 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient(hunting): dump: 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.851 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4075546601 0 0) 0x7f0c68139030 con 0x7f0c680a5d50 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.26:6790/0 conn(0x7f0c680a5920 legacy=0x7f0c681d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.26:6789/0 conn(0x7f0c6806e7b0 legacy=0x7f0c681ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient: _finish_hunting 0 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 1 monclient: found mon.c 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6813aaf0 con 0x7f0c680a5d50 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient: _finish_auth 0 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient: _check_auth_tickets 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c6edf4640 5 monclient: authenticate success, global_id 184369 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c6edf4640 10 monclient: _renew_subs 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c6edf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:50.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c6edf4640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c6813add0 con 0x7f0c680a5d50 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c6edf4640 10 client.184369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c6edf4640 10 monclient: _renew_subs 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c6edf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c6edf4640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c6813a730 con 0x7f0c680a5d50 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c6edf4640 1 librados: init done 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c60004c20 con 0x7f0c680a5d50 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c60001900 con 0x7f0c680a5d50 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c66ffd640 4 set_mon_vals no callback set 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.833+0000 7f0c657fa640 10 monclient: dump: 2024-08-19T12:53:50.854 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.855 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.857 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.834+0000 7f0c6edf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:50.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.834+0000 7f0c6edf4640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0c480065c0 con 0x7f0c48002860 2024-08-19T12:53:50.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.834+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f0c600119c0 con 0x7f0c680a5d50 2024-08-19T12:53:50.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.834+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(567..567 src has 350..567) ==== 5849+0+0 (unknown 4291685512 0 0) 0x7f0c6004e330 con 0x7f0c680a5d50 2024-08-19T12:53:50.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.834+0000 7f0c657fa640 10 client.184369.objecter ms_dispatch 0x7f0c68000b90 osd_map(567..567 src has 350..567) 2024-08-19T12:53:50.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.834+0000 7f0c657fa640 3 client.184369.objecter handle_osd_map got epochs [567,567] > 0 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.834+0000 7f0c657fa640 3 client.184369.objecter handle_osd_map decoding full epoch 567 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.834+0000 7f0c657fa640 20 client.184369.objecter dump_active .. 0 homeless 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.834+0000 7f0c67fff640 10 monclient: get_auth_request con 0x7f0c4c041a90 auth_method 0 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 10 client.184369.objecter ms_handle_connect 0x7f0c48002860 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c6cb90640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.26:6791/0 conn(0x7f0c48002860 legacy=0x7f0c48004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 10 client.184369.objecter resend_mon_ops 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3333685823 0 0) 0x7f0c480065c0 con 0x7f0c48002860 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 10 monclient(hunting): my global_id is 184378 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c68139030 con 0x7f0c48002860 2024-08-19T12:53:50.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3080550098 0 0) 0x7f0c68139030 con 0x7f0c48002860 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c480065c0 con 0x7f0c48002860 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c5c0052a0 con 0x7f0c48002860 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.836+0000 7f0c657fa640 10 monclient: dump: 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.859 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.837+0000 7f0c657fa640 10 client.184369.objecter ms_handle_connect 0x7f0c4c041a90 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.837+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1566373483 0 0) 0x7f0c480065c0 con 0x7f0c48002860 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.837+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c4800a160 con 0x7f0c48002860 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.837+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0c5c007480 con 0x7f0c48002860 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.837+0000 7f0c657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:50.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.837+0000 7f0c657fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:50.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.837+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.26:6791/0 conn(0x7f0c48002860 legacy=0x7f0c48004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.837+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.26:6791/0 conn(0x7f0c48002860 legacy=0x7f0c48004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:50.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.843+0000 7f0c6edf4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:50.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.843+0000 7f0c6edf4640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0c48009e50 con 0x7f0c48009110 2024-08-19T12:53:50.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.846+0000 7f0c657fa640 10 client.184369.objecter ms_handle_connect 0x7f0c48009110 2024-08-19T12:53:50.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.846+0000 7f0c657fa640 10 client.184369.objecter resend_mon_ops 2024-08-19T12:53:50.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.846+0000 7f0c6cb90640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.26:6791/0 conn(0x7f0c48009110 legacy=0x7f0c480095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:50.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.917+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2818839078 0 0) 0x7f0c48009e50 con 0x7f0c48009110 2024-08-19T12:53:50.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.917+0000 7f0c657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:50.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.917+0000 7f0c657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:50.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.917+0000 7f0c657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:50.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.917+0000 7f0c657fa640 10 monclient(hunting): my global_id is 184387 2024-08-19T12:53:50.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.917+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c480065c0 con 0x7f0c48009110 2024-08-19T12:53:50.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.925+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2023395493 0 0) 0x7f0c480065c0 con 0x7f0c48009110 2024-08-19T12:53:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.925+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c48009e50 con 0x7f0c48009110 2024-08-19T12:53:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.925+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0c5c005b40 con 0x7f0c48009110 2024-08-19T12:53:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.925+0000 7f0c657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.925+0000 7f0c657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.925+0000 7f0c657fa640 10 monclient: dump: 2024-08-19T12:53:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:50.927 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.925+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2665948975 0 0) 0x7f0c48009e50 con 0x7f0c48009110 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c4800c3f0 con 0x7f0c48009110 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 2328226980) 0x7f0c5c007730 con 0x7f0c48009110 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:50.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c657fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:50.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.26:6791/0 conn(0x7f0c48009110 legacy=0x7f0c480095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c657fa640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.26:6791/0 conn(0x7f0c48009110 legacy=0x7f0c480095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:50.929 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:50.929 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:50.929 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:50.929 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:50.929 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:50.929 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 156391, 2024-08-19T12:53:50.929 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:50.930 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.931 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:50.933 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:50.934 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:50.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.938 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:50.940 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:50.941 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.941 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:50.941 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.941 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:50.941 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:50.941 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.942 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.942 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.942 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:50.942 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:50.942 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:50.942 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:50.945 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:50.946 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:50.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:50.948 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:50.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:50.949 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:50.949 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:50.949 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:50.949 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:50.949 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:50.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:50.949 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:50.949 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:50.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c6edf4640 10 librados: watch_flush enter 2024-08-19T12:53:50.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c6edf4640 10 librados: watch_flush exit 2024-08-19T12:53:50.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c6edf4640 20 client.184369.objecter shutdown clearing up homeless session... 2024-08-19T12:53:50.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c6edf4640 10 client.184369.objecter successfully canceled tick 2024-08-19T12:53:50.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c6edf4640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.104:6800/899887506 conn(0x7f0c4c041a90 legacy=0x7f0c4c043f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c6edf4640 10 monclient: shutdown 2024-08-19T12:53:50.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c6edf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:50.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c6edf4640 1 -- 172.21.15.26:0/3972080961 >> v1:172.21.15.26:6791/0 conn(0x7f0c680a5d50 legacy=0x7f0c681d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:50.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c6d391640 1 -- 172.21.15.26:0/3972080961 reap_dead start 2024-08-19T12:53:50.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c6d391640 1 -- 172.21.15.26:0/3972080961 reap_dead start 2024-08-19T12:53:50.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c6edf4640 1 -- 172.21.15.26:0/3972080961 shutdown_connections 2024-08-19T12:53:50.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.926+0000 7f0c6edf4640 1 -- 172.21.15.26:0/3972080961 >> 172.21.15.26:0/3972080961 conn(0x7f0c680a8740 msgr2=0x7f0c680aa690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:50.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.927+0000 7f0c6edf4640 1 -- 172.21.15.26:0/3972080961 shutdown_connections 2024-08-19T12:53:50.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.927+0000 7f0c6edf4640 1 -- 172.21.15.26:0/3972080961 wait complete. 2024-08-19T12:53:50.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:50.927+0000 7f0c6edf4640 1 librados: shutdown 2024-08-19T12:53:50.952 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:50.952 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:53:50.953 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:53:51.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.052+0000 7f657042f640 10 monclient: get_monmap_and_config 2024-08-19T12:53:51.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.052+0000 7f657042f640 10 monclient: build_initial_monmap 2024-08-19T12:53:51.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.052+0000 7f657042f640 10 monclient: monmap: 2024-08-19T12:53:51.053 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:51.053 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.053 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:51.053346+0000 2024-08-19T12:53:51.053 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:51.053346+0000 2024-08-19T12:53:51.053 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:51.053 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:51.054 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 1 Processor -- start 2024-08-19T12:53:51.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 1 -- start start 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 10 monclient: init 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 10 monclient: _add_conns ranks=[3,8,1,4,7,6,2,5,0] 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 10 monclient(hunting): picked mon.noname-g con 0x7f656806e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 10 monclient(hunting): picked mon.noname-e con 0x7f65680a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 10 monclient(hunting): picked mon.noname-f con 0x7f65680a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6568136790 con 0x7f65680a5d50 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656809ddf0 con 0x7f656806e7b0 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656809e0d0 con 0x7f65680a5920 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f657042f640 10 monclient(hunting): authenticate will time out at 1625.140835s 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f656d9ca640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f65680a5920 0x7f656813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40614/0 (socket says 172.21.15.26:40614) 2024-08-19T12:53:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.053+0000 7f656d9ca640 1 -- 172.21.15.26:0/3664437824 learned_addr learned my addr 172.21.15.26:0/3664437824 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:51.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d9ca640 1 -- 172.21.15.26:0/3664437824 >> v1:172.21.15.26:6793/0 conn(0x7f65680a5920 legacy=0x7f656813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656e1cb640 1 -- 172.21.15.26:0/3664437824 >> v1:172.21.15.104:6790/0 conn(0x7f656806e7b0 legacy=0x7f6568136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656e9cc640 1 -- 172.21.15.26:0/3664437824 >> v1:172.21.15.104:6789/0 conn(0x7f65680a5d50 legacy=0x7f6568141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1632122634 0 0) 0x7f656809e0d0 con 0x7f65680a5920 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient(hunting): my global_id is 184274 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6554003520 con 0x7f65680a5920 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4201060736 0 0) 0x7f65640014a0 con 0x7f656806e7b0 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient(hunting): my global_id is 194124 2024-08-19T12:53:51.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f656809e0d0 con 0x7f656806e7b0 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4161390556 0 0) 0x7f6568136790 con 0x7f65680a5d50 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient(hunting): my global_id is 184348 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65640014a0 con 0x7f65680a5d50 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 798500421 0 0) 0x7f6554003520 con 0x7f65680a5920 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 >> v1:172.21.15.104:6790/0 conn(0x7f656806e7b0 legacy=0x7f6568136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 >> v1:172.21.15.104:6789/0 conn(0x7f65680a5d50 legacy=0x7f6568141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient: _finish_hunting 0 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 monclient: found mon.noname-e 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:51.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656809e3b0 con 0x7f65680a5920 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient: _finish_auth 0 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient: _check_auth_tickets 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6558001ae0 con 0x7f65680a5920 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient: dump: 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.063 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f657042f640 5 monclient: authenticate success, global_id 184274 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f657042f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65580048a0 con 0x7f65680a5920 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 1 -- 172.21.15.26:0/3664437824 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6558004e10 con 0x7f65680a5920 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f657042f640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f657042f640 4 set_mon_vals no callback set 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.056+0000 7f656d1c9640 10 monclient: dump: 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.065 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.057+0000 7f657042f640 10 monclient: shutdown 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.057+0000 7f657042f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.057+0000 7f657042f640 1 -- 172.21.15.26:0/3664437824 >> v1:172.21.15.26:6793/0 conn(0x7f65680a5920 legacy=0x7f656813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.057+0000 7f657042f640 1 -- 172.21.15.26:0/3664437824 shutdown_connections 2024-08-19T12:53:51.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.057+0000 7f657042f640 1 -- 172.21.15.26:0/3664437824 >> 172.21.15.26:0/3664437824 conn(0x7f65680a8740 msgr2=0x7f65680a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.057+0000 7f657042f640 1 -- 172.21.15.26:0/3664437824 shutdown_connections 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.057+0000 7f657042f640 1 -- 172.21.15.26:0/3664437824 wait complete. 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.057+0000 7f657042f640 10 monclient: build_initial_monmap 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.057+0000 7f657042f640 10 monclient: monmap: 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:51.067 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 1 librados: starting msgr at 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 1 librados: starting objecter 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 1 Processor -- start 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 1 -- start start 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 1 librados: setting wanted keys 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 1 librados: calling monclient init 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 10 monclient: init 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 10 monclient: _add_conns ranks=[5,4,0,7,3,1,8,2,6] 2024-08-19T12:53:51.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 10 monclient(hunting): picked mon.noname-h con 0x7f656806e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:53:51.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 10 monclient(hunting): picked mon.noname-c con 0x7f65680a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:51.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 10 monclient(hunting): picked mon.noname-a con 0x7f65680a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:51.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656813b6c0 con 0x7f65680a5d50 2024-08-19T12:53:51.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6568134980 con 0x7f65680a5920 2024-08-19T12:53:51.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65681345c0 con 0x7f656806e7b0 2024-08-19T12:53:51.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:51.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f657042f640 10 monclient(hunting): authenticate will time out at 1625.145734s 2024-08-19T12:53:51.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f656d9ca640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f65680a5920 0x7f6568137ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59186/0 (socket says 172.21.15.26:59186) 2024-08-19T12:53:51.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.058+0000 7f656d9ca640 1 -- 172.21.15.26:0/334955532 learned_addr learned my addr 172.21.15.26:0/334955532 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:51.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.060+0000 7f656d9ca640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.26:6791/0 conn(0x7f65680a5920 legacy=0x7f6568137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.060+0000 7f655effd640 10 client.?.objecter ms_handle_connect 0x7f65680a5920 2024-08-19T12:53:51.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.060+0000 7f655effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.060+0000 7f655effd640 10 client.?.objecter ms_handle_connect 0x7f65680a5d50 2024-08-19T12:53:51.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.060+0000 7f655effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.060+0000 7f656e9cc640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.26:6789/0 conn(0x7f65680a5d50 legacy=0x7f65681d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f656e1cb640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.104:6791/0 conn(0x7f656806e7b0 legacy=0x7f6568137490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 client.?.objecter ms_handle_connect 0x7f656806e7b0 2024-08-19T12:53:51.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2044408334 0 0) 0x7f65640014a0 con 0x7f65680a5d50 2024-08-19T12:53:51.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): my global_id is 194157 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f654c004eb0 con 0x7f65680a5d50 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 704930249 0 0) 0x7f656809e3b0 con 0x7f65680a5920 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): my global_id is 184396 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65640014a0 con 0x7f65680a5920 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4187054827 0 0) 0x7f656809e0d0 con 0x7f656806e7b0 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): my global_id is 184361 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f656809e3b0 con 0x7f656806e7b0 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1830174859 0 0) 0x7f654c004eb0 con 0x7f65680a5d50 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f656809e0d0 con 0x7f65680a5d50 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6560002070 con 0x7f65680a5d50 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): dump: 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.078 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1671591140 0 0) 0x7f65640014a0 con 0x7f65680a5920 2024-08-19T12:53:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f654c004eb0 con 0x7f65680a5920 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6558004d90 con 0x7f65680a5920 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): dump: 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.080 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2014640649 0 0) 0x7f656809e3b0 con 0x7f656806e7b0 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65640014a0 con 0x7f656806e7b0 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6564003950 con 0x7f656806e7b0 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient(hunting): dump: 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.081 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.082 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1242966655 0 0) 0x7f656809e0d0 con 0x7f65680a5d50 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.104:6791/0 conn(0x7f656806e7b0 legacy=0x7f6568137490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.26:6791/0 conn(0x7f65680a5920 legacy=0x7f6568137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient: _finish_hunting 0 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 monclient: found mon.a 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656813b2a0 con 0x7f65680a5d50 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient: _finish_auth 0 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient: _check_auth_tickets 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f657042f640 5 monclient: authenticate success, global_id 194157 2024-08-19T12:53:51.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f657042f640 10 monclient: _renew_subs 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f657042f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f657042f640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65681382b0 con 0x7f65680a5d50 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f657042f640 10 client.194157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f657042f640 10 monclient: _renew_subs 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f657042f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f657042f640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6568142a10 con 0x7f65680a5d50 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f657042f640 1 librados: init done 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6560003db0 con 0x7f65680a5d50 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f65600042a0 con 0x7f65680a5d50 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f656c9c8640 4 set_mon_vals no callback set 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:51.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.061+0000 7f655effd640 10 monclient: dump: 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.085 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.062+0000 7f657042f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.062+0000 7f657042f640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f65500065c0 con 0x7f6550002860 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.062+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6560004440 con 0x7f65680a5d50 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.063+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(568..568 src has 350..568) ==== 5862+0+0 (unknown 2537974883 0 0) 0x7f656004e5d0 con 0x7f65680a5d50 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.063+0000 7f655effd640 10 client.194157.objecter ms_dispatch 0x7f6568000b90 osd_map(568..568 src has 350..568) 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.063+0000 7f655effd640 3 client.194157.objecter handle_osd_map got epochs [568,568] > 0 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.063+0000 7f655effd640 3 client.194157.objecter handle_osd_map decoding full epoch 568 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.063+0000 7f655effd640 20 client.194157.objecter dump_active .. 0 homeless 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.063+0000 7f656d9ca640 10 monclient: get_auth_request con 0x7f654c041930 auth_method 0 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.064+0000 7f655effd640 10 client.194157.objecter ms_handle_connect 0x7f6550002860 2024-08-19T12:53:51.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.064+0000 7f655effd640 10 client.194157.objecter resend_mon_ops 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.064+0000 7f656e1cb640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.26:6792/0 conn(0x7f6550002860 legacy=0x7f6550004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 225399845 0 0) 0x7f65500065c0 con 0x7f6550002860 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 10 monclient(hunting): my global_id is 194109 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f656809e0d0 con 0x7f6550002860 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1883489141 0 0) 0x7f656809e0d0 con 0x7f6550002860 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65500065c0 con 0x7f6550002860 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6564003f30 con 0x7f6550002860 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 10 monclient: dump: 2024-08-19T12:53:51.087 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.088 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2501766550 0 0) 0x7f65500065c0 con 0x7f6550002860 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f655000a160 con 0x7f6550002860 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 10 client.194157.objecter ms_handle_connect 0x7f654c041930 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f65640041e0 con 0x7f6550002860 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.26:6792/0 conn(0x7f6550002860 legacy=0x7f6550004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.065+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.26:6792/0 conn(0x7f6550002860 legacy=0x7f6550004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.071+0000 7f657042f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.072+0000 7f657042f640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6550009e70 con 0x7f6550009130 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f656e1cb640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.26:6792/0 conn(0x7f6550009130 legacy=0x7f65500095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 10 client.194157.objecter ms_handle_connect 0x7f6550009130 2024-08-19T12:53:51.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 10 client.194157.objecter resend_mon_ops 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 132836148 0 0) 0x7f6550009e70 con 0x7f6550009130 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 10 monclient(hunting): my global_id is 194118 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65500065c0 con 0x7f6550009130 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2749209492 0 0) 0x7f65500065c0 con 0x7f6550009130 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6550009e70 con 0x7f6550009130 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6564005b00 con 0x7f6550009130 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 10 monclient: dump: 2024-08-19T12:53:51.090 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.091 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.092 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1484730992 0 0) 0x7f6550009e70 con 0x7f6550009130 2024-08-19T12:53:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f655000c410 con 0x7f6550009130 2024-08-19T12:53:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 117102862) 0x7f6564005ef0 con 0x7f6550009130 2024-08-19T12:53:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:51.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.26:6792/0 conn(0x7f6550009130 legacy=0x7f65500095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.092 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:51.092 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 16001, 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:51.093 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.094 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.095 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:51.096 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.097 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.098 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.099 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.101 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:51.102 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.103 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:51.105 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.106 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:51.106 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:51.106 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.106 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.106 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.106 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:51.106 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:51.107 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:51.107 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:51.107 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.108 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.109 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.109 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:51.109 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:51.109 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.109 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.109 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.109 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:51.109 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:51.109 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:51.110 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.110 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:51.110 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:51.110 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.111 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:51.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.112 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.112 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.112 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:51.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.112 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.112 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:51.112 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:51.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.074+0000 7f655effd640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.26:6792/0 conn(0x7f6550009130 legacy=0x7f65500095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 10 librados: watch_flush enter 2024-08-19T12:53:51.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 10 librados: watch_flush exit 2024-08-19T12:53:51.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 20 client.194157.objecter shutdown clearing up homeless session... 2024-08-19T12:53:51.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 10 client.194157.objecter successfully canceled tick 2024-08-19T12:53:51.114 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.104:6800/899887506 conn(0x7f654c041930 legacy=0x7f654c043df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 10 monclient: shutdown 2024-08-19T12:53:51.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:51.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 1 -- 172.21.15.26:0/334955532 >> v1:172.21.15.26:6789/0 conn(0x7f65680a5d50 legacy=0x7f65681d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f656e9cc640 1 -- 172.21.15.26:0/334955532 reap_dead start 2024-08-19T12:53:51.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f656e9cc640 1 -- 172.21.15.26:0/334955532 reap_dead start 2024-08-19T12:53:51.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 1 -- 172.21.15.26:0/334955532 shutdown_connections 2024-08-19T12:53:51.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 1 -- 172.21.15.26:0/334955532 >> 172.21.15.26:0/334955532 conn(0x7f65680a8740 msgr2=0x7f65680a51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:51.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 1 -- 172.21.15.26:0/334955532 shutdown_connections 2024-08-19T12:53:51.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 1 -- 172.21.15.26:0/334955532 wait complete. 2024-08-19T12:53:51.115 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.075+0000 7f657042f640 1 librados: shutdown 2024-08-19T12:53:51.115 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:51.116 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:53:51.116 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:53:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.215+0000 7f67f654c640 10 monclient: get_monmap_and_config 2024-08-19T12:53:51.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.215+0000 7f67f654c640 10 monclient: build_initial_monmap 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.215+0000 7f67f654c640 10 monclient: monmap: 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:51.215938+0000 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:51.215938+0000 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:51.216 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:51.217 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:51.217 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:51.217 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.215+0000 7f67f654c640 1 Processor -- start 2024-08-19T12:53:51.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.215+0000 7f67f654c640 1 -- start start 2024-08-19T12:53:51.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.215+0000 7f67f654c640 10 monclient: init 2024-08-19T12:53:51.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.216+0000 7f67f654c640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:51.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.216+0000 7f67f654c640 10 monclient: _add_conns ranks=[7,2,5,0,4,8,6,3,1] 2024-08-19T12:53:51.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.216+0000 7f67f654c640 10 monclient(hunting): picked mon.noname-i con 0x7f67f0073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:51.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.216+0000 7f67f654c640 10 monclient(hunting): picked mon.noname-b con 0x7f67f00a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:51.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.216+0000 7f67f654c640 10 monclient(hunting): picked mon.noname-h con 0x7f67f013f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:53:51.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.216+0000 7f67f654c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f009dae0 con 0x7f67f00a57a0 2024-08-19T12:53:51.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.216+0000 7f67f654c640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f0136910 con 0x7f67f013f2e0 2024-08-19T12:53:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.216+0000 7f67f654c640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f009dd50 con 0x7f67f0073470 2024-08-19T12:53:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.216+0000 7f67f654c640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.216+0000 7f67f654c640 10 monclient(hunting): authenticate will time out at 1625.303503s 2024-08-19T12:53:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.216+0000 7f67e7fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f67f00a57a0 0x7f67f00a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47804/0 (socket says 172.21.15.26:47804) 2024-08-19T12:53:51.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.216+0000 7f67e7fff640 1 -- 172.21.15.26:0/1891035878 learned_addr learned my addr 172.21.15.26:0/1891035878 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.218+0000 7f67e7fff640 1 -- 172.21.15.26:0/1891035878 >> v1:172.21.15.26:6790/0 conn(0x7f67f00a57a0 legacy=0x7f67f00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.218+0000 7f67ef7fe640 1 -- 172.21.15.26:0/1891035878 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1386008455 0 0) 0x7f67f009dae0 con 0x7f67f00a57a0 2024-08-19T12:53:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.218+0000 7f67ef7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.218+0000 7f67ef7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.218+0000 7f67ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.218+0000 7f67ef7fe640 10 monclient(hunting): my global_id is 27101 2024-08-19T12:53:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 1 -- 172.21.15.26:0/1891035878 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67dc003520 con 0x7f67f00a57a0 2024-08-19T12:53:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67effff640 1 -- 172.21.15.26:0/1891035878 >> v1:172.21.15.104:6792/0 conn(0x7f67f0073470 legacy=0x7f67f0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67f4ae9640 1 -- 172.21.15.26:0/1891035878 >> v1:172.21.15.104:6791/0 conn(0x7f67f013f2e0 legacy=0x7f67f01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 1 -- 172.21.15.26:0/1891035878 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 672023679 0 0) 0x7f67f009dd50 con 0x7f67f0073470 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient(hunting): my global_id is 136105 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 1 -- 172.21.15.26:0/1891035878 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67f009dae0 con 0x7f67f0073470 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 1 -- 172.21.15.26:0/1891035878 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2006965656 0 0) 0x7f67dc003520 con 0x7f67f00a57a0 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 1 -- 172.21.15.26:0/1891035878 >> v1:172.21.15.104:6792/0 conn(0x7f67f0073470 legacy=0x7f67f0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 1 -- 172.21.15.26:0/1891035878 >> v1:172.21.15.104:6791/0 conn(0x7f67f013f2e0 legacy=0x7f67f01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 1 monclient: found mon.noname-b 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:51.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 1 -- 172.21.15.26:0/1891035878 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f009e030 con 0x7f67f00a57a0 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient: _finish_auth 0 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67f654c640 5 monclient: authenticate success, global_id 27101 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67f654c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 1 -- 172.21.15.26:0/1891035878 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f67d8001ae0 con 0x7f67f00a57a0 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient: dump: 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.225 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67f654c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 1 -- 172.21.15.26:0/1891035878 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f67d80048a0 con 0x7f67f00a57a0 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 1 -- 172.21.15.26:0/1891035878 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f67d8004e10 con 0x7f67f00a57a0 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67f654c640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67f654c640 4 set_mon_vals no callback set 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67ef7fe640 10 monclient: dump: 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.227 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67f654c640 10 monclient: shutdown 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67f654c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:51.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67f654c640 1 -- 172.21.15.26:0/1891035878 >> v1:172.21.15.26:6790/0 conn(0x7f67f00a57a0 legacy=0x7f67f00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67f654c640 1 -- 172.21.15.26:0/1891035878 shutdown_connections 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.219+0000 7f67f654c640 1 -- 172.21.15.26:0/1891035878 >> 172.21.15.26:0/1891035878 conn(0x7f67f00abff0 msgr2=0x7f67f00ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.220+0000 7f67f654c640 1 -- 172.21.15.26:0/1891035878 shutdown_connections 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.220+0000 7f67f654c640 1 -- 172.21.15.26:0/1891035878 wait complete. 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.220+0000 7f67f654c640 10 monclient: build_initial_monmap 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.220+0000 7f67f654c640 10 monclient: monmap: 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:51.230 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:51.230 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:51.230 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:51.230 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:51.230 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:51.230 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:51.230 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:51.230 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.220+0000 7f67f654c640 1 librados: starting msgr at 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.220+0000 7f67f654c640 1 librados: starting objecter 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.220+0000 7f67f654c640 1 Processor -- start 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.220+0000 7f67f654c640 1 -- start start 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.220+0000 7f67f654c640 1 librados: setting wanted keys 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.220+0000 7f67f654c640 1 librados: calling monclient init 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.220+0000 7f67f654c640 10 monclient: init 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67f654c640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67f654c640 10 monclient: _add_conns ranks=[8,1,0,2,4,3,5,7,6] 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67f654c640 10 monclient(hunting): picked mon.noname-e con 0x7f67f0073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67f654c640 10 monclient(hunting): picked mon.noname-f con 0x7f67f00a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67f654c640 10 monclient(hunting): picked mon.noname-a con 0x7f67f013f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67f654c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f0132420 con 0x7f67f013f2e0 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67f654c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f0142b80 con 0x7f67f00a57a0 2024-08-19T12:53:51.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67f654c640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f01317f0 con 0x7f67f0073470 2024-08-19T12:53:51.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67f654c640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:51.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67f654c640 10 monclient(hunting): authenticate will time out at 1625.308370s 2024-08-19T12:53:51.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67effff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f67f0073470 0x7f67f012dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40628/0 (socket says 172.21.15.26:40628) 2024-08-19T12:53:51.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67f4ae9640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f67f013f2e0 0x7f67f01d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43350/0 (socket says 172.21.15.26:43350) 2024-08-19T12:53:51.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.221+0000 7f67effff640 1 -- 172.21.15.26:0/1217406338 learned_addr learned my addr 172.21.15.26:0/1217406338 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:51.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67effff640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.26:6793/0 conn(0x7f67f0073470 legacy=0x7f67f012dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67f4ae9640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.26:6789/0 conn(0x7f67f013f2e0 legacy=0x7f67f01d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 client.?.objecter ms_handle_connect 0x7f67f0073470 2024-08-19T12:53:51.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 client.?.objecter ms_handle_connect 0x7f67f013f2e0 2024-08-19T12:53:51.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 client.?.objecter ms_handle_connect 0x7f67f00a57a0 2024-08-19T12:53:51.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67e7fff640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.104:6789/0 conn(0x7f67f00a57a0 legacy=0x7f67f012e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1816665268 0 0) 0x7f67f01317f0 con 0x7f67f0073470 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 monclient(hunting): my global_id is 184283 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67d0004eb0 con 0x7f67f0073470 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2996454408 0 0) 0x7f67f0132420 con 0x7f67f013f2e0 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 monclient(hunting): my global_id is 194166 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67f01317f0 con 0x7f67f013f2e0 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4203464758 0 0) 0x7f67f009e030 con 0x7f67f00a57a0 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 10 monclient(hunting): my global_id is 184357 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.223+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67f0132420 con 0x7f67f00a57a0 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2221287713 0 0) 0x7f67d0004eb0 con 0x7f67f0073470 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67f009e030 con 0x7f67f0073470 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f67e0003610 con 0x7f67f0073470 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient(hunting): dump: 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.240 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2606408537 0 0) 0x7f67f01317f0 con 0x7f67f013f2e0 2024-08-19T12:53:51.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67d0004eb0 con 0x7f67f013f2e0 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 943660474 0 0) 0x7f67f0132420 con 0x7f67f00a57a0 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67f01317f0 con 0x7f67f00a57a0 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f67e8002030 con 0x7f67f013f2e0 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient(hunting): dump: 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.242 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f67d8004d90 con 0x7f67f00a57a0 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient(hunting): dump: 2024-08-19T12:53:51.243 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3108339544 0 0) 0x7f67f009e030 con 0x7f67f0073470 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.104:6789/0 conn(0x7f67f00a57a0 legacy=0x7f67f012e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.26:6789/0 conn(0x7f67f013f2e0 legacy=0x7f67f01d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient: _finish_hunting 0 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 monclient: found mon.e 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f012e8f0 con 0x7f67f0073470 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient: _finish_auth 0 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient: _check_auth_tickets 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:51.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67f654c640 5 monclient: authenticate success, global_id 184283 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67f654c640 10 monclient: _renew_subs 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67f654c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67f654c640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f67f0131510 con 0x7f67f0073470 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67f654c640 10 client.184283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67f654c640 10 monclient: _renew_subs 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67f654c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67f654c640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f67f0132fe0 con 0x7f67f0073470 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67f654c640 1 librados: init done 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f67e0003bf0 con 0x7f67f0073470 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f67e00040e0 con 0x7f67f0073470 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67eeffd640 4 set_mon_vals no callback set 2024-08-19T12:53:51.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.224+0000 7f67ed7fa640 10 monclient: dump: 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.247 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.225+0000 7f67f654c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.225+0000 7f67f654c640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f67d40065c0 con 0x7f67d4002860 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.225+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f67e0004280 con 0x7f67f0073470 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.225+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(568..568 src has 350..568) ==== 5862+0+0 (unknown 2537974883 0 0) 0x7f67e004e460 con 0x7f67f0073470 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.225+0000 7f67ed7fa640 10 client.184283.objecter ms_dispatch 0x7f67f0000b90 osd_map(568..568 src has 350..568) 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.225+0000 7f67ed7fa640 3 client.184283.objecter handle_osd_map got epochs [568,568] > 0 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.225+0000 7f67ed7fa640 3 client.184283.objecter handle_osd_map decoding full epoch 568 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.225+0000 7f67ed7fa640 20 client.184283.objecter dump_active .. 0 homeless 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.225+0000 7f67f4ae9640 10 monclient: get_auth_request con 0x7f67d0041970 auth_method 0 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 10 client.184283.objecter ms_handle_connect 0x7f67d4002860 2024-08-19T12:53:51.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 10 client.184283.objecter resend_mon_ops 2024-08-19T12:53:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67e7fff640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.26:6793/0 conn(0x7f67d4002860 legacy=0x7f67d4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 253051763 0 0) 0x7f67d40065c0 con 0x7f67d4002860 2024-08-19T12:53:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 10 monclient(hunting): my global_id is 184292 2024-08-19T12:53:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67f009e030 con 0x7f67d4002860 2024-08-19T12:53:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2280454883 0 0) 0x7f67f009e030 con 0x7f67d4002860 2024-08-19T12:53:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67d40065c0 con 0x7f67d4002860 2024-08-19T12:53:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f67d8001580 con 0x7f67d4002860 2024-08-19T12:53:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.227+0000 7f67ed7fa640 10 monclient: dump: 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.250 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.228+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2353842077 0 0) 0x7f67d40065c0 con 0x7f67d4002860 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.228+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67d400a160 con 0x7f67d4002860 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.228+0000 7f67ed7fa640 10 client.184283.objecter ms_handle_connect 0x7f67d0041970 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.228+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f67d8001d30 con 0x7f67d4002860 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.228+0000 7f67ed7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.228+0000 7f67ed7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.228+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.26:6793/0 conn(0x7f67d4002860 legacy=0x7f67d4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.228+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.26:6793/0 conn(0x7f67d4002860 legacy=0x7f67d4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.234+0000 7f67f654c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.234+0000 7f67f654c640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f67d4009e70 con 0x7f67d4009130 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.236+0000 7f67e7fff640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.26:6793/0 conn(0x7f67d4009130 legacy=0x7f67d40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.236+0000 7f67ed7fa640 10 client.184283.objecter ms_handle_connect 0x7f67d4009130 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.236+0000 7f67ed7fa640 10 client.184283.objecter resend_mon_ops 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.236+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 990226946 0 0) 0x7f67d4009e70 con 0x7f67d4009130 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.236+0000 7f67ed7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.236+0000 7f67ed7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.236+0000 7f67ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.236+0000 7f67ed7fa640 10 monclient(hunting): my global_id is 184301 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.236+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67d40065c0 con 0x7f67d4009130 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 784214402 0 0) 0x7f67d40065c0 con 0x7f67d4009130 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67d4009e70 con 0x7f67d4009130 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f67d8005a90 con 0x7f67d4009130 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 10 monclient: dump: 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.253 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.254 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.254 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.254 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2809609502 0 0) 0x7f67d4009e70 con 0x7f67d4009130 2024-08-19T12:53:51.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f67d400c410 con 0x7f67d4009130 2024-08-19T12:53:51.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 226800831) 0x7f67d80048a0 con 0x7f67d4009130 2024-08-19T12:53:51.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:51.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:51.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.26:6793/0 conn(0x7f67d4009130 legacy=0x7f67d40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67ed7fa640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.26:6793/0 conn(0x7f67d4009130 legacy=0x7f67d40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 156703, 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:51.255 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.256 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.257 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:51.258 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:51.259 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:51.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:51.262 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:51.262 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.262 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.262 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.262 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:51.262 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:51.262 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.265 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.265 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:51.265 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.266 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:51.266 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:51.266 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.266 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.266 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:51.267 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.268 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.269 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.269 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:51.269 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:51.269 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.269 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.269 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:51.270 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.273 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:51.274 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.276 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.277 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.278 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:51.278 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.278 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.278 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:51.278 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:51.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f654c640 10 librados: watch_flush enter 2024-08-19T12:53:51.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f654c640 10 librados: watch_flush exit 2024-08-19T12:53:51.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f654c640 20 client.184283.objecter shutdown clearing up homeless session... 2024-08-19T12:53:51.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f654c640 10 client.184283.objecter successfully canceled tick 2024-08-19T12:53:51.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f654c640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.104:6800/899887506 conn(0x7f67d0041970 legacy=0x7f67d0043e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f654c640 10 monclient: shutdown 2024-08-19T12:53:51.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f654c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:51.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f654c640 1 -- 172.21.15.26:0/1217406338 >> v1:172.21.15.26:6793/0 conn(0x7f67f0073470 legacy=0x7f67f012dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f4ae9640 1 -- 172.21.15.26:0/1217406338 reap_dead start 2024-08-19T12:53:51.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f4ae9640 1 -- 172.21.15.26:0/1217406338 reap_dead start 2024-08-19T12:53:51.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f654c640 1 -- 172.21.15.26:0/1217406338 shutdown_connections 2024-08-19T12:53:51.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f654c640 1 -- 172.21.15.26:0/1217406338 >> 172.21.15.26:0/1217406338 conn(0x7f67f00abff0 msgr2=0x7f67f0133f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:51.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f654c640 1 -- 172.21.15.26:0/1217406338 shutdown_connections 2024-08-19T12:53:51.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.237+0000 7f67f654c640 1 -- 172.21.15.26:0/1217406338 wait complete. 2024-08-19T12:53:51.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.238+0000 7f67f654c640 1 librados: shutdown 2024-08-19T12:53:51.281 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:51.281 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:53:51.281 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:53:51.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.380+0000 7feb9fe27640 10 monclient: get_monmap_and_config 2024-08-19T12:53:51.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.380+0000 7feb9fe27640 10 monclient: build_initial_monmap 2024-08-19T12:53:51.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.380+0000 7feb9fe27640 10 monclient: monmap: 2024-08-19T12:53:51.381 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:51.381 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.381 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:51.381425+0000 2024-08-19T12:53:51.381 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:51.381425+0000 2024-08-19T12:53:51.381 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:51.382 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.382 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:51.382 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:51.382 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:51.382 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:51.382 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:51.382 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:51.382 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:51.382 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:51.382 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:51.382 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 1 Processor -- start 2024-08-19T12:53:51.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 1 -- start start 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 10 monclient: init 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 10 monclient: _add_conns ranks=[8,5,2,6,3,4,1,0,7] 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 10 monclient(hunting): picked mon.noname-e con 0x7feb9806e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 10 monclient(hunting): picked mon.noname-h con 0x7feb980a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 10 monclient(hunting): picked mon.noname-b con 0x7feb980a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb98136790 con 0x7feb980a5d50 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb9809ddf0 con 0x7feb980a5920 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb9809e0d0 con 0x7feb9806e7b0 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9fe27640 10 monclient(hunting): authenticate will time out at 1625.468870s 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9e3c4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7feb980a5d50 0x7feb98141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47814/0 (socket says 172.21.15.26:47814) 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.382+0000 7feb9e3c4640 1 -- 172.21.15.26:0/2130330963 learned_addr learned my addr 172.21.15.26:0/2130330963 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:51.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.381+0000 7feb9dbc3640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7feb9806e7b0 0x7feb98136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40668/0 (socket says 172.21.15.26:40668) 2024-08-19T12:53:51.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9e3c4640 1 -- 172.21.15.26:0/2130330963 >> v1:172.21.15.26:6790/0 conn(0x7feb980a5d50 legacy=0x7feb98141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9dbc3640 1 -- 172.21.15.26:0/2130330963 >> v1:172.21.15.26:6793/0 conn(0x7feb9806e7b0 legacy=0x7feb98136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9d3c2640 1 -- 172.21.15.26:0/2130330963 >> v1:172.21.15.104:6791/0 conn(0x7feb980a5920 legacy=0x7feb9813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 678463908 0 0) 0x7feb98136790 con 0x7feb980a5d50 2024-08-19T12:53:51.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient(hunting): my global_id is 27110 2024-08-19T12:53:51.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb84003520 con 0x7feb980a5d50 2024-08-19T12:53:51.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1215833756 0 0) 0x7feb9809e0d0 con 0x7feb9806e7b0 2024-08-19T12:53:51.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient(hunting): my global_id is 184310 2024-08-19T12:53:51.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb98136790 con 0x7feb9806e7b0 2024-08-19T12:53:51.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633127727 0 0) 0x7feb880014a0 con 0x7feb980a5920 2024-08-19T12:53:51.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient(hunting): my global_id is 184379 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb9809e0d0 con 0x7feb980a5920 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 759141093 0 0) 0x7feb84003520 con 0x7feb980a5d50 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 >> v1:172.21.15.26:6793/0 conn(0x7feb9806e7b0 legacy=0x7feb98136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 >> v1:172.21.15.104:6791/0 conn(0x7feb980a5920 legacy=0x7feb9813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient: _finish_hunting 0 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 monclient: found mon.noname-b 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9809e3b0 con 0x7feb980a5d50 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient: _finish_auth 0 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient: _check_auth_tickets 2024-08-19T12:53:51.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9fe27640 5 monclient: authenticate success, global_id 27110 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9fe27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7feb90001e60 con 0x7feb980a5d50 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient: dump: 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.392 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.393 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.393 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.393 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.393 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.393 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.393 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.393 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.393 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9fe27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:51.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb90004c20 con 0x7feb980a5d50 2024-08-19T12:53:51.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:51.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 1 -- 172.21.15.26:0/2130330963 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7feb90005190 con 0x7feb980a5d50 2024-08-19T12:53:51.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9fe27640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:51.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9fe27640 4 set_mon_vals no callback set 2024-08-19T12:53:51.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:51.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.384+0000 7feb9cbc1640 10 monclient: dump: 2024-08-19T12:53:51.394 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.394 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.394 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.394 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.394 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.394 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.395 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.395 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.395 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.395 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.395 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.395 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.395 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.395 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.395 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.395 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.385+0000 7feb9fe27640 10 monclient: shutdown 2024-08-19T12:53:51.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.385+0000 7feb9fe27640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:51.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.385+0000 7feb9fe27640 1 -- 172.21.15.26:0/2130330963 >> v1:172.21.15.26:6790/0 conn(0x7feb980a5d50 legacy=0x7feb98141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.385+0000 7feb9fe27640 1 -- 172.21.15.26:0/2130330963 shutdown_connections 2024-08-19T12:53:51.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.385+0000 7feb9fe27640 1 -- 172.21.15.26:0/2130330963 >> 172.21.15.26:0/2130330963 conn(0x7feb980a8740 msgr2=0x7feb980a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:51.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.385+0000 7feb9fe27640 1 -- 172.21.15.26:0/2130330963 shutdown_connections 2024-08-19T12:53:51.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.385+0000 7feb9fe27640 1 -- 172.21.15.26:0/2130330963 wait complete. 2024-08-19T12:53:51.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.385+0000 7feb9fe27640 10 monclient: build_initial_monmap 2024-08-19T12:53:51.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.385+0000 7feb9fe27640 10 monclient: monmap: 2024-08-19T12:53:51.396 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:51.397 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:51.397 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:51.397 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:51.397 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:51.397 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.397 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:51.397 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:51.397 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 1 librados: starting msgr at 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 1 librados: starting objecter 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 1 Processor -- start 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 1 -- start start 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 1 librados: setting wanted keys 2024-08-19T12:53:51.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 1 librados: calling monclient init 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 10 monclient: init 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 10 monclient: _add_conns ranks=[1,8,4,6,7,2,0,3,5] 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 10 monclient(hunting): picked mon.noname-f con 0x7feb9806e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 10 monclient(hunting): picked mon.noname-e con 0x7feb980a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 10 monclient(hunting): picked mon.noname-c con 0x7feb980a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb980a31f0 con 0x7feb9806e7b0 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb980a19f0 con 0x7feb980a5d50 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb980a25b0 con 0x7feb980a5920 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9fe27640 10 monclient(hunting): authenticate will time out at 1625.473740s 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9d3c2640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7feb980a5920 0x7feb980a0280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40672/0 (socket says 172.21.15.26:40672) 2024-08-19T12:53:51.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.386+0000 7feb9d3c2640 1 -- 172.21.15.26:0/790299468 learned_addr learned my addr 172.21.15.26:0/790299468 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:51.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb9d3c2640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.26:6793/0 conn(0x7feb980a5920 legacy=0x7feb980a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 client.?.objecter ms_handle_connect 0x7feb980a5920 2024-08-19T12:53:51.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 client.?.objecter ms_handle_connect 0x7feb980a5d50 2024-08-19T12:53:51.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb9e3c4640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.26:6791/0 conn(0x7feb980a5d50 legacy=0x7feb981d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 client.?.objecter ms_handle_connect 0x7feb9806e7b0 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb9dbc3640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.104:6789/0 conn(0x7feb9806e7b0 legacy=0x7feb9809fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1391345890 0 0) 0x7feb980a25b0 con 0x7feb980a5920 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): my global_id is 184319 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb7c004eb0 con 0x7feb980a5920 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1346245320 0 0) 0x7feb980a19f0 con 0x7feb980a5d50 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): my global_id is 184405 2024-08-19T12:53:51.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb980a25b0 con 0x7feb980a5d50 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1506421895 0 0) 0x7feb980a31f0 con 0x7feb9806e7b0 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): my global_id is 184366 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb980a19f0 con 0x7feb9806e7b0 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1691385810 0 0) 0x7feb7c004eb0 con 0x7feb980a5920 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb980a31f0 con 0x7feb980a5920 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7feb88003950 con 0x7feb980a5920 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): dump: 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.410 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.410 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.410 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.410 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.410 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.410 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.410 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.410 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.410 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2981031080 0 0) 0x7feb980a25b0 con 0x7feb980a5d50 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb7c004eb0 con 0x7feb980a5d50 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7feb900020b0 con 0x7feb980a5d50 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): dump: 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.411 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.412 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 362873069 0 0) 0x7feb980a19f0 con 0x7feb9806e7b0 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb980a25b0 con 0x7feb9806e7b0 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7feb940047e0 con 0x7feb9806e7b0 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient(hunting): dump: 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.413 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.414 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.414 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.414 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.414 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.414 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.414 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.414 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.414 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.414 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1723368720 0 0) 0x7feb980a31f0 con 0x7feb980a5920 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.26:6791/0 conn(0x7feb980a5d50 legacy=0x7feb981d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.104:6789/0 conn(0x7feb9806e7b0 legacy=0x7feb9809fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 monclient: found mon.e 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb980a0990 con 0x7feb980a5920 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient: _finish_auth 0 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb9fe27640 5 monclient: authenticate success, global_id 184319 2024-08-19T12:53:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb9fe27640 10 monclient: _renew_subs 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb9fe27640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.389+0000 7feb9fe27640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feb980a0c20 con 0x7feb980a5920 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb9fe27640 10 client.184319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb9fe27640 10 monclient: _renew_subs 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb9fe27640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb9fe27640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7feb981d7b90 con 0x7feb980a5920 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb9fe27640 1 librados: init done 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb88003c20 con 0x7feb980a5920 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb8e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7feb88004190 con 0x7feb980a5920 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb8ffff640 4 set_mon_vals no callback set 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb8e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:51.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb8e7fc640 10 monclient: dump: 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.417 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb9fe27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb9fe27640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feb800065c0 con 0x7feb80002860 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.390+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7feb88011a80 con 0x7feb980a5920 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.391+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(568..568 src has 350..568) ==== 5862+0+0 (unknown 2537974883 0 0) 0x7feb8804e4f0 con 0x7feb980a5920 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.391+0000 7feb8e7fc640 10 client.184319.objecter ms_dispatch 0x7feb98000b90 osd_map(568..568 src has 350..568) 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.391+0000 7feb8e7fc640 3 client.184319.objecter handle_osd_map got epochs [568,568] > 0 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.391+0000 7feb8e7fc640 3 client.184319.objecter handle_osd_map decoding full epoch 568 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.391+0000 7feb8e7fc640 20 client.184319.objecter dump_active .. 0 homeless 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.391+0000 7feb9e3c4640 10 monclient: get_auth_request con 0x7feb7c041b40 auth_method 0 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb9dbc3640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.104:6789/0 conn(0x7feb80002860 legacy=0x7feb80004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 10 client.184319.objecter ms_handle_connect 0x7feb80002860 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 10 client.184319.objecter resend_mon_ops 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3696432741 0 0) 0x7feb800065c0 con 0x7feb80002860 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 10 monclient(hunting): my global_id is 184375 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb980a31f0 con 0x7feb80002860 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 10 client.184319.objecter ms_handle_connect 0x7feb7c041b40 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2522318310 0 0) 0x7feb980a31f0 con 0x7feb80002860 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb800065c0 con 0x7feb80002860 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7feb94004eb0 con 0x7feb80002860 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:51.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 10 monclient: dump: 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.420 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 101175378 0 0) 0x7feb800065c0 con 0x7feb80002860 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.393+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb8000a160 con 0x7feb80002860 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.394+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7feb940063f0 con 0x7feb80002860 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.394+0000 7feb8e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.394+0000 7feb8e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.394+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.104:6789/0 conn(0x7feb80002860 legacy=0x7feb80004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.394+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.104:6789/0 conn(0x7feb80002860 legacy=0x7feb80004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.400+0000 7feb9fe27640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.400+0000 7feb9fe27640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feb80009e70 con 0x7feb80009130 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.402+0000 7feb9dbc3640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.104:6789/0 conn(0x7feb80009130 legacy=0x7feb800095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.402+0000 7feb8e7fc640 10 client.184319.objecter ms_handle_connect 0x7feb80009130 2024-08-19T12:53:51.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.402+0000 7feb8e7fc640 10 client.184319.objecter resend_mon_ops 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3482513119 0 0) 0x7feb80009e70 con 0x7feb80009130 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 10 monclient(hunting): my global_id is 184384 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb800065c0 con 0x7feb80009130 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2992635475 0 0) 0x7feb800065c0 con 0x7feb80009130 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb80009e70 con 0x7feb80009130 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7feb94005460 con 0x7feb80009130 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 10 monclient: dump: 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.422 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.423 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4180169979 0 0) 0x7feb80009e70 con 0x7feb80009130 2024-08-19T12:53:51.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feb8000c3f0 con 0x7feb80009130 2024-08-19T12:53:51.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 826300465) 0x7feb94005810 con 0x7feb80009130 2024-08-19T12:53:51.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:51.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:51.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.104:6789/0 conn(0x7feb80009130 legacy=0x7feb800095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.403+0000 7feb8e7fc640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.104:6789/0 conn(0x7feb80009130 legacy=0x7feb800095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.424 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:51.424 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:51.425 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:51.425 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:51.425 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:51.425 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 156872, 2024-08-19T12:53:51.425 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:51.425 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:51.425 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:51.425 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.426 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.427 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:51.428 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:51.429 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.430 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:51.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.431 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:51.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.431 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:51.431 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:51.431 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.431 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.431 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.431 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:51.432 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.434 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:51.435 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:51.435 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.435 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.435 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.435 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:51.435 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:51.435 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.435 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.435 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.436 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:51.436 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.436 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.436 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.436 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:51.436 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.437 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.438 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.439 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:51.440 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.441 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.445 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:51.445 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.445 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.445 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.446 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.447 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.448 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:51.448 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.448 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.448 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:51.448 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:51.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 10 librados: watch_flush enter 2024-08-19T12:53:51.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 10 librados: watch_flush exit 2024-08-19T12:53:51.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 20 client.184319.objecter shutdown clearing up homeless session... 2024-08-19T12:53:51.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 10 client.184319.objecter successfully canceled tick 2024-08-19T12:53:51.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.104:6800/899887506 conn(0x7feb7c041b40 legacy=0x7feb7c044000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 10 monclient: shutdown 2024-08-19T12:53:51.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:51.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 1 -- 172.21.15.26:0/790299468 >> v1:172.21.15.26:6793/0 conn(0x7feb980a5920 legacy=0x7feb980a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9e3c4640 1 -- 172.21.15.26:0/790299468 reap_dead start 2024-08-19T12:53:51.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9e3c4640 1 -- 172.21.15.26:0/790299468 reap_dead start 2024-08-19T12:53:51.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 1 -- 172.21.15.26:0/790299468 shutdown_connections 2024-08-19T12:53:51.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 1 -- 172.21.15.26:0/790299468 >> 172.21.15.26:0/790299468 conn(0x7feb980a8740 msgr2=0x7feb98135150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:51.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 1 -- 172.21.15.26:0/790299468 shutdown_connections 2024-08-19T12:53:51.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 1 -- 172.21.15.26:0/790299468 wait complete. 2024-08-19T12:53:51.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.404+0000 7feb9fe27640 1 librados: shutdown 2024-08-19T12:53:51.451 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:51.451 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:53:51.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:53:51.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.551+0000 7f5698b0d640 10 monclient: get_monmap_and_config 2024-08-19T12:53:51.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.551+0000 7f5698b0d640 10 monclient: build_initial_monmap 2024-08-19T12:53:51.552 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.551+0000 7f5698b0d640 10 monclient: monmap: 2024-08-19T12:53:51.552 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:51.552 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.552 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:51.552283+0000 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:51.552283+0000 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:51.553 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.551+0000 7f5698b0d640 1 Processor -- start 2024-08-19T12:53:51.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f5698b0d640 1 -- start start 2024-08-19T12:53:51.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f5698b0d640 10 monclient: init 2024-08-19T12:53:51.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f5698b0d640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:51.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f5698b0d640 10 monclient: _add_conns ranks=[0,6,1,8,3,4,5,7,2] 2024-08-19T12:53:51.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f5698b0d640 10 monclient(hunting): picked mon.noname-a con 0x7f569406e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:51.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f5698b0d640 10 monclient(hunting): picked mon.noname-d con 0x7f56940a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:53:51.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f5698b0d640 10 monclient(hunting): picked mon.noname-f con 0x7f56940a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:51.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f5698b0d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5694136790 con 0x7f569406e7b0 2024-08-19T12:53:51.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f5698b0d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f569409ddf0 con 0x7f56940a5d50 2024-08-19T12:53:51.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f5698b0d640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f569409e0d0 con 0x7f56940a5920 2024-08-19T12:53:51.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f5698b0d640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:51.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f5698b0d640 10 monclient(hunting): authenticate will time out at 1625.639737s 2024-08-19T12:53:51.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f569259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f569406e7b0 0x7f5694136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43354/0 (socket says 172.21.15.26:43354) 2024-08-19T12:53:51.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.552+0000 7f569259c640 1 -- 172.21.15.26:0/4015745238 learned_addr learned my addr 172.21.15.26:0/4015745238 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:51.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569259c640 1 -- 172.21.15.26:0/4015745238 >> v1:172.21.15.26:6789/0 conn(0x7f569406e7b0 legacy=0x7f5694136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f5691d9b640 1 -- 172.21.15.26:0/4015745238 >> v1:172.21.15.26:6792/0 conn(0x7f56940a5920 legacy=0x7f569413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f5691d9b640 1 -- 172.21.15.26:0/4015745238 >> v1:172.21.15.26:6792/0 conn(0x7f56940a5920 legacy=0x7f569413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-08-19T12:53:51.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f5691d9b640 1 -- 172.21.15.26:0/4015745238 >> v1:172.21.15.26:6792/0 conn(0x7f56940a5920 legacy=0x7f569413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:53:51.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f5691d9b640 1 --1- 172.21.15.26:0/4015745238 >> v1:172.21.15.26:6792/0 conn(0x7f56940a5920 0x7f569413e2f0 :-1 s=OPENED pgs=3516 cs=1 l=1).handle_message read tag failed 2024-08-19T12:53:51.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f5691d9b640 1 --1- 172.21.15.26:0/4015745238 >> v1:172.21.15.26:6792/0 conn(0x7f56940a5920 0x7f569413e2f0 :-1 s=OPENED pgs=3516 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:53:51.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6792/0 2024-08-19T12:53:51.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 1 -- 172.21.15.26:0/4015745238 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3779752142 0 0) 0x7f5694136790 con 0x7f569406e7b0 2024-08-19T12:53:51.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient(hunting): my global_id is 194175 2024-08-19T12:53:51.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f5692d9d640 1 -- 172.21.15.26:0/4015745238 >> v1:172.21.15.104:6789/0 conn(0x7f56940a5d50 legacy=0x7f5694141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 1 -- 172.21.15.26:0/4015745238 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f567c003520 con 0x7f569406e7b0 2024-08-19T12:53:51.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 1 -- 172.21.15.26:0/4015745238 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 775119771 0 0) 0x7f567c003520 con 0x7f569406e7b0 2024-08-19T12:53:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 1 -- 172.21.15.26:0/4015745238 >> v1:172.21.15.26:6792/0 conn(0x7f56940a5920 legacy=0x7f569413e2f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 1 -- 172.21.15.26:0/4015745238 >> v1:172.21.15.104:6789/0 conn(0x7f56940a5d50 legacy=0x7f5694141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient: _finish_hunting 0 2024-08-19T12:53:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 1 monclient: found mon.noname-a 2024-08-19T12:53:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:53:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 1 -- 172.21.15.26:0/4015745238 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f569409e3b0 con 0x7f569406e7b0 2024-08-19T12:53:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient: _finish_auth 0 2024-08-19T12:53:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient: _check_auth_tickets 2024-08-19T12:53:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 1 -- 172.21.15.26:0/4015745238 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5684001ae0 con 0x7f569406e7b0 2024-08-19T12:53:51.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f5698b0d640 5 monclient: authenticate success, global_id 194175 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f5698b0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient: dump: 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f5698b0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 1 -- 172.21.15.26:0/4015745238 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f56840048a0 con 0x7f569406e7b0 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 1 -- 172.21.15.26:0/4015745238 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5684004e10 con 0x7f569406e7b0 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f5698b0d640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f5698b0d640 4 set_mon_vals no callback set 2024-08-19T12:53:51.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.555+0000 7f569159a640 10 monclient: dump: 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.573 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.556+0000 7f5698b0d640 10 monclient: shutdown 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.556+0000 7f5698b0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.556+0000 7f5698b0d640 1 -- 172.21.15.26:0/4015745238 >> v1:172.21.15.26:6789/0 conn(0x7f569406e7b0 legacy=0x7f5694136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.556+0000 7f5698b0d640 1 -- 172.21.15.26:0/4015745238 shutdown_connections 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.556+0000 7f5698b0d640 1 -- 172.21.15.26:0/4015745238 >> 172.21.15.26:0/4015745238 conn(0x7f56940a8740 msgr2=0x7f56940a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.556+0000 7f5698b0d640 1 -- 172.21.15.26:0/4015745238 shutdown_connections 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.556+0000 7f5698b0d640 1 -- 172.21.15.26:0/4015745238 wait complete. 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.556+0000 7f5698b0d640 10 monclient: build_initial_monmap 2024-08-19T12:53:51.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.556+0000 7f5698b0d640 10 monclient: monmap: 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:51.575 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 1 librados: starting msgr at 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 1 librados: starting objecter 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 1 Processor -- start 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 1 -- start start 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 1 librados: setting wanted keys 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 1 librados: calling monclient init 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 10 monclient: init 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 10 monclient: _add_conns ranks=[0,2,6,5,1,3,8,7,4] 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 10 monclient(hunting): picked mon.noname-a con 0x7f569406e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 10 monclient(hunting): picked mon.noname-b con 0x7f56940a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:51.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 10 monclient(hunting): picked mon.noname-d con 0x7f56940a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56940a1840 con 0x7f569406e7b0 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5694134c70 con 0x7f56940a5920 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56940a5070 con 0x7f56940a5d50 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5698b0d640 10 monclient(hunting): authenticate will time out at 1625.644771s 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5691d9b640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f56940a5920 0x7f56941d3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47826/0 (socket says 172.21.15.26:47826) 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f569259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f569406e7b0 0x7f56940a4960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43356/0 (socket says 172.21.15.26:43356) 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.557+0000 7f5691d9b640 1 -- 172.21.15.26:0/2984441994 learned_addr learned my addr 172.21.15.26:0/2984441994 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5692d9d640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.26:6792/0 conn(0x7f56940a5d50 legacy=0x7f56941d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5691d9b640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.26:6790/0 conn(0x7f56940a5920 legacy=0x7f56941d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f569259c640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.26:6789/0 conn(0x7f569406e7b0 legacy=0x7f56940a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 client.?.objecter ms_handle_connect 0x7f56940a5d50 2024-08-19T12:53:51.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 client.?.objecter ms_handle_connect 0x7f56940a5920 2024-08-19T12:53:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 client.?.objecter ms_handle_connect 0x7f569406e7b0 2024-08-19T12:53:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4029508680 0 0) 0x7f56940a5070 con 0x7f56940a5d50 2024-08-19T12:53:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): my global_id is 194127 2024-08-19T12:53:51.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5674004eb0 con 0x7f56940a5d50 2024-08-19T12:53:51.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1142522133 0 0) 0x7f56940a1840 con 0x7f569406e7b0 2024-08-19T12:53:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): my global_id is 194184 2024-08-19T12:53:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56940a5070 con 0x7f569406e7b0 2024-08-19T12:53:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2351393301 0 0) 0x7f569409e0d0 con 0x7f56940a5920 2024-08-19T12:53:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): my global_id is 27119 2024-08-19T12:53:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56940a1840 con 0x7f56940a5920 2024-08-19T12:53:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2291389196 0 0) 0x7f5674004eb0 con 0x7f56940a5d50 2024-08-19T12:53:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f569409e0d0 con 0x7f56940a5d50 2024-08-19T12:53:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f568c001890 con 0x7f56940a5d50 2024-08-19T12:53:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:53:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): dump: 2024-08-19T12:53:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.582 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 439503467 0 0) 0x7f56940a5070 con 0x7f569406e7b0 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5674004eb0 con 0x7f569406e7b0 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5684004d90 con 0x7f569406e7b0 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): dump: 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.583 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2343585457 0 0) 0x7f56940a1840 con 0x7f56940a5920 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56940a5070 con 0x7f56940a5920 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5688003610 con 0x7f56940a5920 2024-08-19T12:53:51.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient(hunting): dump: 2024-08-19T12:53:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.585 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1141251345 0 0) 0x7f569409e0d0 con 0x7f56940a5d50 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.26:6790/0 conn(0x7f56940a5920 legacy=0x7f56941d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.26:6789/0 conn(0x7f569406e7b0 legacy=0x7f56940a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 monclient: found mon.d 2024-08-19T12:53:51.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56940a10f0 con 0x7f56940a5d50 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient: _finish_auth 0 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5698b0d640 5 monclient: authenticate success, global_id 194127 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5698b0d640 10 monclient: _renew_subs 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5698b0d640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5698b0d640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f56940a5280 con 0x7f56940a5d50 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5698b0d640 10 client.194127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5698b0d640 10 monclient: _renew_subs 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5698b0d640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5698b0d640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f56940a1c50 con 0x7f56940a5d50 2024-08-19T12:53:51.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5698b0d640 1 librados: init done 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f568c003c40 con 0x7f56940a5d50 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f568c004130 con 0x7f56940a5d50 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5690d99640 4 set_mon_vals no callback set 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.560+0000 7f5682ffd640 10 monclient: dump: 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.588 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.561+0000 7f5698b0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.561+0000 7f5698b0d640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f56780065c0 con 0x7f5678002860 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.561+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f568c004390 con 0x7f56940a5d50 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.562+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(568..568 src has 350..568) ==== 5862+0+0 (unknown 2537974883 0 0) 0x7f568c04e480 con 0x7f56940a5d50 2024-08-19T12:53:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.562+0000 7f5682ffd640 10 client.194127.objecter ms_dispatch 0x7f5694000b90 osd_map(568..568 src has 350..568) 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.562+0000 7f5682ffd640 3 client.194127.objecter handle_osd_map got epochs [568,568] > 0 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.562+0000 7f5682ffd640 3 client.194127.objecter handle_osd_map decoding full epoch 568 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.562+0000 7f5682ffd640 20 client.194127.objecter dump_active .. 0 homeless 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.562+0000 7f5691d9b640 10 monclient: get_auth_request con 0x7f56740419c0 auth_method 0 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 10 client.194127.objecter ms_handle_connect 0x7f5678002860 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f569259c640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.104:6790/0 conn(0x7f5678002860 legacy=0x7f5678004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 10 client.194127.objecter resend_mon_ops 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 894090670 0 0) 0x7f56780065c0 con 0x7f5678002860 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 10 monclient(hunting): my global_id is 194133 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f569409e0d0 con 0x7f5678002860 2024-08-19T12:53:51.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 10 client.194127.objecter ms_handle_connect 0x7f56740419c0 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2273636943 0 0) 0x7f569409e0d0 con 0x7f5678002860 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56780065c0 con 0x7f5678002860 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5684001320 con 0x7f5678002860 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 10 monclient: dump: 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.591 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2942975031 0 0) 0x7f56780065c0 con 0x7f5678002860 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.564+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f567800a160 con 0x7f5678002860 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.565+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f56840048a0 con 0x7f5678002860 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.565+0000 7f5682ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.565+0000 7f5682ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:51.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.565+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.104:6790/0 conn(0x7f5678002860 legacy=0x7f5678004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.565+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.104:6790/0 conn(0x7f5678002860 legacy=0x7f5678004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.571+0000 7f5698b0d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:51.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.571+0000 7f5698b0d640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5678009e70 con 0x7f5678009130 2024-08-19T12:53:51.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.573+0000 7f5682ffd640 10 client.194127.objecter ms_handle_connect 0x7f5678009130 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.573+0000 7f5682ffd640 10 client.194127.objecter resend_mon_ops 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.573+0000 7f569259c640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.104:6790/0 conn(0x7f5678009130 legacy=0x7f56780095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1373827767 0 0) 0x7f5678009e70 con 0x7f5678009130 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 10 monclient(hunting): my global_id is 194142 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56780065c0 con 0x7f5678009130 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1934995784 0 0) 0x7f56780065c0 con 0x7f5678009130 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5678009e70 con 0x7f5678009130 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f56840059c0 con 0x7f5678009130 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:53:51.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 10 monclient: dump: 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.596 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.597 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.597 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3413995867 0 0) 0x7f5678009e70 con 0x7f5678009130 2024-08-19T12:53:51.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f567800c3f0 con 0x7f5678009130 2024-08-19T12:53:51.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 2559142182) 0x7f5684006f70 con 0x7f5678009130 2024-08-19T12:53:51.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:51.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:51.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.104:6790/0 conn(0x7f5678009130 legacy=0x7f56780095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.574+0000 7f5682ffd640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.104:6790/0 conn(0x7f5678009130 legacy=0x7f56780095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.597 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:51.597 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:51.598 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:51.598 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:51.598 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:51.598 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 16497, 2024-08-19T12:53:51.598 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:51.598 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:51.598 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:51.598 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:51.598 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:51.598 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:51.598 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:51.598 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.599 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.600 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.600 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.600 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.600 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.600 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.600 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:51.600 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:51.600 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.600 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.601 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.601 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.601 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.601 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.601 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.601 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.601 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.601 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.601 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.602 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.602 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:51.602 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:51.602 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:51.602 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:51.602 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:51.602 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:51.602 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:51.602 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:51.602 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:51.602 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:51.603 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:51.603 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:51.603 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:51.603 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:51.603 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:51.603 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:51.603 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:51.603 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.603 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.603 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.603 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:51.604 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.605 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.606 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:51.608 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.609 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.614 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.615 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.616 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.617 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.618 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:51.618 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:51.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 10 librados: watch_flush enter 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 10 librados: watch_flush exit 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 20 client.194127.objecter shutdown clearing up homeless session... 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 10 client.194127.objecter successfully canceled tick 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.104:6800/899887506 conn(0x7f56740419c0 legacy=0x7f5674043e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 10 monclient: shutdown 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 1 -- 172.21.15.26:0/2984441994 >> v1:172.21.15.26:6792/0 conn(0x7f56940a5d50 legacy=0x7f56941d69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5692d9d640 1 -- 172.21.15.26:0/2984441994 reap_dead start 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5692d9d640 1 -- 172.21.15.26:0/2984441994 reap_dead start 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 1 -- 172.21.15.26:0/2984441994 shutdown_connections 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 1 -- 172.21.15.26:0/2984441994 >> 172.21.15.26:0/2984441994 conn(0x7f56940a8740 msgr2=0x7f56940aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 1 -- 172.21.15.26:0/2984441994 shutdown_connections 2024-08-19T12:53:51.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 1 -- 172.21.15.26:0/2984441994 wait complete. 2024-08-19T12:53:51.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.575+0000 7f5698b0d640 1 librados: shutdown 2024-08-19T12:53:51.621 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:51.621 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:53:51.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:53:51.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.721+0000 7f790f7f2640 10 monclient: get_monmap_and_config 2024-08-19T12:53:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.721+0000 7f790f7f2640 10 monclient: build_initial_monmap 2024-08-19T12:53:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.721+0000 7f790f7f2640 10 monclient: monmap: 2024-08-19T12:53:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:51.721902+0000 2024-08-19T12:53:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:51.721902+0000 2024-08-19T12:53:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:51.722 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:51.723 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:51.723 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:51.723 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:51.723 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:51.723 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:51.723 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.721+0000 7f790f7f2640 1 Processor -- start 2024-08-19T12:53:51.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.721+0000 7f790f7f2640 1 -- start start 2024-08-19T12:53:51.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.721+0000 7f790f7f2640 10 monclient: init 2024-08-19T12:53:51.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.722+0000 7f790f7f2640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:51.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.722+0000 7f790f7f2640 10 monclient: _add_conns ranks=[2,3,5,0,4,1,7,6,8] 2024-08-19T12:53:51.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.722+0000 7f790f7f2640 10 monclient(hunting): picked mon.noname-b con 0x7f79080a68e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:51.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.722+0000 7f790f7f2640 10 monclient(hunting): picked mon.noname-g con 0x7f79080a73d0 addr v1:172.21.15.104:6790/0 2024-08-19T12:53:51.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.722+0000 7f790f7f2640 10 monclient(hunting): picked mon.noname-h con 0x7f790813c8f0 addr v1:172.21.15.104:6791/0 2024-08-19T12:53:51.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.722+0000 7f790f7f2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79080a0390 con 0x7f79080a68e0 2024-08-19T12:53:51.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.722+0000 7f790f7f2640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79080a0670 con 0x7f79080a73d0 2024-08-19T12:53:51.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.722+0000 7f790f7f2640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79080a0950 con 0x7f790813c8f0 2024-08-19T12:53:51.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.722+0000 7f790f7f2640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:51.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.722+0000 7f790f7f2640 10 monclient(hunting): authenticate will time out at 1625.809496s 2024-08-19T12:53:51.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.722+0000 7f790d58e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f79080a68e0 0x7f79080a6cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47832/0 (socket says 172.21.15.26:47832) 2024-08-19T12:53:51.725 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.722+0000 7f790d58e640 1 -- 172.21.15.26:0/2270808997 learned_addr learned my addr 172.21.15.26:0/2270808997 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:51.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.724+0000 7f790d58e640 1 -- 172.21.15.26:0/2270808997 >> v1:172.21.15.26:6790/0 conn(0x7f79080a68e0 legacy=0x7f79080a6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.724+0000 7f790dd8f640 1 -- 172.21.15.26:0/2270808997 >> v1:172.21.15.104:6791/0 conn(0x7f790813c8f0 legacy=0x7f790813ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.724+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2887740657 0 0) 0x7f79080a0390 con 0x7f79080a68e0 2024-08-19T12:53:51.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.724+0000 7f78fffff640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.724+0000 7f78fffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.724+0000 7f78fffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.724+0000 7f78fffff640 10 monclient(hunting): my global_id is 27128 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.724+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78f4003520 con 0x7f79080a68e0 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.724+0000 7f790cd8d640 1 -- 172.21.15.26:0/2270808997 >> v1:172.21.15.104:6790/0 conn(0x7f79080a73d0 legacy=0x7f790813b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2570467797 0 0) 0x7f79080a0950 con 0x7f790813c8f0 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient(hunting): my global_id is 184388 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79080a0390 con 0x7f790813c8f0 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3806047661 0 0) 0x7f79080a0670 con 0x7f79080a73d0 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient(hunting): my global_id is 194151 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79080a0950 con 0x7f79080a73d0 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3490344656 0 0) 0x7f78f4003520 con 0x7f79080a68e0 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 >> v1:172.21.15.104:6791/0 conn(0x7f790813c8f0 legacy=0x7f790813ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 >> v1:172.21.15.104:6790/0 conn(0x7f79080a73d0 legacy=0x7f790813b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient: _finish_hunting 0 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 1 monclient: found mon.noname-b 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79080742e0 con 0x7f79080a68e0 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient: _finish_auth 0 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient: _check_auth_tickets 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:51.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7904001ae0 con 0x7f79080a68e0 2024-08-19T12:53:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f790f7f2640 5 monclient: authenticate success, global_id 27128 2024-08-19T12:53:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f790f7f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:53:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient: dump: 2024-08-19T12:53:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.732 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f790f7f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79040048a0 con 0x7f79080a68e0 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 1 -- 172.21.15.26:0/2270808997 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7904004e10 con 0x7f79080a68e0 2024-08-19T12:53:51.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f78fffff640 10 monclient: dump: 2024-08-19T12:53:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.734 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f790f7f2640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f790f7f2640 4 set_mon_vals no callback set 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f790f7f2640 10 monclient: shutdown 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f790f7f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f790f7f2640 1 -- 172.21.15.26:0/2270808997 >> v1:172.21.15.26:6790/0 conn(0x7f79080a68e0 legacy=0x7f79080a6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f790f7f2640 1 -- 172.21.15.26:0/2270808997 shutdown_connections 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.725+0000 7f790f7f2640 1 -- 172.21.15.26:0/2270808997 >> 172.21.15.26:0/2270808997 conn(0x7f79080a44d0 msgr2=0x7f79080a48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.726+0000 7f790f7f2640 1 -- 172.21.15.26:0/2270808997 shutdown_connections 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.726+0000 7f790f7f2640 1 -- 172.21.15.26:0/2270808997 wait complete. 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.726+0000 7f790f7f2640 10 monclient: build_initial_monmap 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.726+0000 7f790f7f2640 10 monclient: monmap: 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.726+0000 7f790f7f2640 1 librados: starting msgr at 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.726+0000 7f790f7f2640 1 librados: starting objecter 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.726+0000 7f790f7f2640 1 Processor -- start 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.726+0000 7f790f7f2640 1 -- start start 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.726+0000 7f790f7f2640 1 librados: setting wanted keys 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.726+0000 7f790f7f2640 1 librados: calling monclient init 2024-08-19T12:53:51.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.726+0000 7f790f7f2640 10 monclient: init 2024-08-19T12:53:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.727+0000 7f790f7f2640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.727+0000 7f790f7f2640 10 monclient: _add_conns ranks=[6,1,3,0,8,2,4,5,7] 2024-08-19T12:53:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.727+0000 7f790f7f2640 10 monclient(hunting): picked mon.noname-d con 0x7f79080a68e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:53:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.727+0000 7f790f7f2640 10 monclient(hunting): picked mon.noname-f con 0x7f79080a73d0 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.727+0000 7f790f7f2640 10 monclient(hunting): picked mon.noname-g con 0x7f790813c8f0 addr v1:172.21.15.104:6790/0 2024-08-19T12:53:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.727+0000 7f790f7f2640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7908127a10 con 0x7f79080a73d0 2024-08-19T12:53:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.727+0000 7f790f7f2640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7908127cf0 con 0x7f790813c8f0 2024-08-19T12:53:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.727+0000 7f790f7f2640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7908072f30 con 0x7f79080a68e0 2024-08-19T12:53:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.727+0000 7f790f7f2640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.727+0000 7f790f7f2640 10 monclient(hunting): authenticate will time out at 1625.814405s 2024-08-19T12:53:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.727+0000 7f790d58e640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f79080a68e0 0x7f7908138d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:36060/0 (socket says 172.21.15.26:36060) 2024-08-19T12:53:51.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.727+0000 7f790d58e640 1 -- 172.21.15.26:0/1773360736 learned_addr learned my addr 172.21.15.26:0/1773360736 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f790d58e640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.26:6792/0 conn(0x7f79080a68e0 legacy=0x7f7908138d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 client.?.objecter ms_handle_connect 0x7f79080a68e0 2024-08-19T12:53:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f790dd8f640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.104:6790/0 conn(0x7f790813c8f0 legacy=0x7f79081d60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 client.?.objecter ms_handle_connect 0x7f790813c8f0 2024-08-19T12:53:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 client.?.objecter ms_handle_connect 0x7f79080a73d0 2024-08-19T12:53:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f790cd8d640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.104:6789/0 conn(0x7f79080a73d0 legacy=0x7f7908139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3961851856 0 0) 0x7f7908072f30 con 0x7f79080a68e0 2024-08-19T12:53:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 monclient(hunting): my global_id is 194136 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78ec004eb0 con 0x7f79080a68e0 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 436012704 0 0) 0x7f7908127cf0 con 0x7f790813c8f0 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 monclient(hunting): my global_id is 194160 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7908072f30 con 0x7f790813c8f0 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1446163808 0 0) 0x7f7908127a10 con 0x7f79080a73d0 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 10 monclient(hunting): my global_id is 184402 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.729+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7908127cf0 con 0x7f79080a73d0 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2917142103 0 0) 0x7f78ec004eb0 con 0x7f79080a68e0 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7908127a10 con 0x7f79080a68e0 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7904004d90 con 0x7f79080a68e0 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient(hunting): dump: 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.745 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2121642094 0 0) 0x7f7908072f30 con 0x7f790813c8f0 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78ec004eb0 con 0x7f790813c8f0 2024-08-19T12:53:51.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79000020b0 con 0x7f790813c8f0 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient(hunting): dump: 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.747 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4099433624 0 0) 0x7f7908127a10 con 0x7f79080a68e0 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.104:6790/0 conn(0x7f790813c8f0 legacy=0x7f79081d60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.104:6789/0 conn(0x7f79080a73d0 legacy=0x7f7908139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient: _finish_hunting 0 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 monclient: found mon.d 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:53:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7908127730 con 0x7f79080a68e0 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient: _finish_auth 0 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient: _check_auth_tickets 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f790f7f2640 5 monclient: authenticate success, global_id 194136 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f790f7f2640 10 monclient: _renew_subs 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f790f7f2640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f790f7f2640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7908121e60 con 0x7f79080a68e0 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7904001d30 con 0x7f79080a68e0 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7904001a80 con 0x7f79080a68e0 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78fdffb640 10 monclient: dump: 2024-08-19T12:53:51.749 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.750 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f78ff7fe640 4 set_mon_vals no callback set 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f790f7f2640 10 client.194136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f790f7f2640 10 monclient: _renew_subs 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f790f7f2640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f790f7f2640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f790809dd10 con 0x7f79080a68e0 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.730+0000 7f790f7f2640 1 librados: init done 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.731+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f7904011a00 con 0x7f79080a68e0 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.731+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(568..568 src has 350..568) ==== 5862+0+0 (unknown 2537974883 0 0) 0x7f790404e3c0 con 0x7f79080a68e0 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.731+0000 7f78fdffb640 10 client.194136.objecter ms_dispatch 0x7f7908000b90 osd_map(568..568 src has 350..568) 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.731+0000 7f78fdffb640 3 client.194136.objecter handle_osd_map got epochs [568,568] > 0 2024-08-19T12:53:51.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.731+0000 7f78fdffb640 3 client.194136.objecter handle_osd_map decoding full epoch 568 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.731+0000 7f790f7f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.731+0000 7f790f7f2640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f79080008d0 con 0x7f790802e340 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.731+0000 7f78fdffb640 20 client.194136.objecter dump_active .. 0 homeless 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.731+0000 7f790cd8d640 10 monclient: get_auth_request con 0x7f78ec041700 auth_method 0 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.733+0000 7f78fdffb640 10 client.194136.objecter ms_handle_connect 0x7f790802e340 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.733+0000 7f78fdffb640 10 client.194136.objecter resend_mon_ops 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.733+0000 7f790dd8f640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.104:6791/0 conn(0x7f790802e340 legacy=0x7f7908063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.733+0000 7f78fdffb640 10 client.194136.objecter ms_handle_connect 0x7f78ec041700 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.733+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4161522190 0 0) 0x7f79080008d0 con 0x7f790802e340 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.733+0000 7f78fdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.733+0000 7f78fdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.733+0000 7f78fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.733+0000 7f78fdffb640 10 monclient(hunting): my global_id is 184397 2024-08-19T12:53:51.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7908127a10 con 0x7f790802e340 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1145306123 0 0) 0x7f7908127a10 con 0x7f790802e340 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79080008d0 con 0x7f790802e340 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7900004020 con 0x7f790802e340 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 10 monclient: dump: 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.753 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1405836835 0 0) 0x7f79080008d0 con 0x7f790802e340 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f790802e720 con 0x7f790802e340 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7900005b40 con 0x7f790802e340 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:51.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.104:6791/0 conn(0x7f790802e340 legacy=0x7f7908063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.734+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.104:6791/0 conn(0x7f790802e340 legacy=0x7f7908063a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.741+0000 7f790f7f2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:51.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.741+0000 7f790f7f2640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7908064190 con 0x7f7908071c30 2024-08-19T12:53:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f790dd8f640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.104:6791/0 conn(0x7f7908071c30 legacy=0x7f79081db800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 10 client.194136.objecter ms_handle_connect 0x7f7908071c30 2024-08-19T12:53:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 10 client.194136.objecter resend_mon_ops 2024-08-19T12:53:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 830771574 0 0) 0x7f7908064190 con 0x7f7908071c30 2024-08-19T12:53:51.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 10 monclient(hunting): my global_id is 184406 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79080008d0 con 0x7f7908071c30 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3605830726 0 0) 0x7f79080008d0 con 0x7f7908071c30 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7908064190 con 0x7f7908071c30 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7900005df0 con 0x7f7908071c30 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.743+0000 7f78fdffb640 10 monclient: dump: 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.757 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3814303143 0 0) 0x7f7908064190 con 0x7f7908071c30 2024-08-19T12:53:51.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7908121bc0 con 0x7f7908071c30 2024-08-19T12:53:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 752248971) 0x7f7900007220 con 0x7f7908071c30 2024-08-19T12:53:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f78fdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f78fdffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.104:6791/0 conn(0x7f7908071c30 legacy=0x7f79081db800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f78fdffb640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.104:6791/0 conn(0x7f7908071c30 legacy=0x7f79081db800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.759 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:51.759 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:51.759 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:51.759 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:51.759 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 157214, 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.760 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.761 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.762 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:51.763 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.764 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.765 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:51.766 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:51.767 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.767 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.767 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.767 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:51.767 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.767 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.767 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.768 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:51.768 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:51.768 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.768 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.768 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.768 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:51.768 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:51.768 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.769 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:51.772 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:51.775 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:51.776 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.777 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:51.779 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:51.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790f7f2640 10 librados: watch_flush enter 2024-08-19T12:53:51.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790f7f2640 10 librados: watch_flush exit 2024-08-19T12:53:51.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790f7f2640 20 client.194136.objecter shutdown clearing up homeless session... 2024-08-19T12:53:51.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790f7f2640 10 client.194136.objecter successfully canceled tick 2024-08-19T12:53:51.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790f7f2640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.104:6800/899887506 conn(0x7f78ec041700 legacy=0x7f78ec043bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790f7f2640 10 monclient: shutdown 2024-08-19T12:53:51.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790f7f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:51.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790f7f2640 1 -- 172.21.15.26:0/1773360736 >> v1:172.21.15.26:6792/0 conn(0x7f79080a68e0 legacy=0x7f7908138d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790dd8f640 1 -- 172.21.15.26:0/1773360736 reap_dead start 2024-08-19T12:53:51.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790dd8f640 1 -- 172.21.15.26:0/1773360736 reap_dead start 2024-08-19T12:53:51.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790f7f2640 1 -- 172.21.15.26:0/1773360736 shutdown_connections 2024-08-19T12:53:51.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790f7f2640 1 -- 172.21.15.26:0/1773360736 >> 172.21.15.26:0/1773360736 conn(0x7f79080a44d0 msgr2=0x7f790806f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:51.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790f7f2640 1 -- 172.21.15.26:0/1773360736 shutdown_connections 2024-08-19T12:53:51.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.744+0000 7f790f7f2640 1 -- 172.21.15.26:0/1773360736 wait complete. 2024-08-19T12:53:51.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.745+0000 7f790f7f2640 1 librados: shutdown 2024-08-19T12:53:51.783 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:51.783 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:53:51.783 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:53:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.882+0000 7f59dfc38640 10 monclient: get_monmap_and_config 2024-08-19T12:53:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.882+0000 7f59dfc38640 10 monclient: build_initial_monmap 2024-08-19T12:53:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.882+0000 7f59dfc38640 10 monclient: monmap: 2024-08-19T12:53:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:51.883270+0000 2024-08-19T12:53:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:51.883270+0000 2024-08-19T12:53:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:51.884 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:51.884 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.882+0000 7f59dfc38640 1 Processor -- start 2024-08-19T12:53:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dfc38640 1 -- start start 2024-08-19T12:53:51.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dfc38640 10 monclient: init 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dfc38640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dfc38640 10 monclient: _add_conns ranks=[5,6,2,0,3,8,7,4,1] 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dfc38640 10 monclient(hunting): picked mon.noname-h con 0x7f59d8073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dfc38640 10 monclient(hunting): picked mon.noname-d con 0x7f59d80a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dfc38640 10 monclient(hunting): picked mon.noname-b con 0x7f59d813f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dfc38640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59d809dae0 con 0x7f59d813f2e0 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dfc38640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59d8136910 con 0x7f59d8073470 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dfc38640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59d809dd50 con 0x7f59d80a57a0 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dfc38640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dfc38640 10 monclient(hunting): authenticate will time out at 1625.970724s 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59de1d5640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f59d813f2e0 0x7f59d81416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47844/0 (socket says 172.21.15.26:47844) 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59de1d5640 1 -- 172.21.15.26:0/2779436033 learned_addr learned my addr 172.21.15.26:0/2779436033 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:51.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.883+0000 7f59dd1d3640 1 --1- 172.21.15.26:0/2779436033 >> v1:172.21.15.26:6792/0 conn(0x7f59d80a57a0 0x7f59d80a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:36066/0 (socket says 172.21.15.26:36066) 2024-08-19T12:53:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.885+0000 7f59de1d5640 1 -- 172.21.15.26:0/2779436033 >> v1:172.21.15.26:6790/0 conn(0x7f59d813f2e0 legacy=0x7f59d81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dd1d3640 1 -- 172.21.15.26:0/2779436033 >> v1:172.21.15.26:6792/0 conn(0x7f59d80a57a0 legacy=0x7f59d80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1740442907 0 0) 0x7f59d809dae0 con 0x7f59d813f2e0 2024-08-19T12:53:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient(hunting): my global_id is 27137 2024-08-19T12:53:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59c0003520 con 0x7f59d813f2e0 2024-08-19T12:53:51.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dd9d4640 1 -- 172.21.15.26:0/2779436033 >> v1:172.21.15.104:6791/0 conn(0x7f59d8073470 legacy=0x7f59d8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2172840683 0 0) 0x7f59d809dd50 con 0x7f59d80a57a0 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient(hunting): my global_id is 194145 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59d809dae0 con 0x7f59d80a57a0 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2500118939 0 0) 0x7f59d8136910 con 0x7f59d8073470 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient(hunting): my global_id is 184415 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59d809dd50 con 0x7f59d8073470 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 813031697 0 0) 0x7f59d809dae0 con 0x7f59d80a57a0 2024-08-19T12:53:51.891 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 >> v1:172.21.15.104:6791/0 conn(0x7f59d8073470 legacy=0x7f59d8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 >> v1:172.21.15.26:6790/0 conn(0x7f59d813f2e0 legacy=0x7f59d81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient: _finish_hunting 0 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 monclient: found mon.noname-d 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d809e030 con 0x7f59d80a57a0 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient: _finish_auth 0 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient: _check_auth_tickets 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f59cc001a40 con 0x7f59d80a57a0 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient: dump: 2024-08-19T12:53:51.892 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.893 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dfc38640 5 monclient: authenticate success, global_id 194145 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dfc38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59cc001d10 con 0x7f59d80a57a0 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 1 -- 172.21.15.26:0/2779436033 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f59cc004b20 con 0x7f59d80a57a0 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dfc38640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dfc38640 4 set_mon_vals no callback set 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.886+0000 7f59dc9d2640 10 monclient: dump: 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.894 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.895 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.887+0000 7f59dfc38640 10 monclient: shutdown 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.887+0000 7f59dfc38640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.887+0000 7f59dfc38640 1 -- 172.21.15.26:0/2779436033 >> v1:172.21.15.26:6792/0 conn(0x7f59d80a57a0 legacy=0x7f59d80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.887+0000 7f59dfc38640 1 -- 172.21.15.26:0/2779436033 shutdown_connections 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.887+0000 7f59dfc38640 1 -- 172.21.15.26:0/2779436033 >> 172.21.15.26:0/2779436033 conn(0x7f59d80abff0 msgr2=0x7f59d80ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.887+0000 7f59dfc38640 1 -- 172.21.15.26:0/2779436033 shutdown_connections 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.887+0000 7f59dfc38640 1 -- 172.21.15.26:0/2779436033 wait complete. 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.887+0000 7f59dfc38640 10 monclient: build_initial_monmap 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.887+0000 7f59dfc38640 10 monclient: monmap: 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:51.896 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.887+0000 7f59dfc38640 1 librados: starting msgr at 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.887+0000 7f59dfc38640 1 librados: starting objecter 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 1 Processor -- start 2024-08-19T12:53:51.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 1 -- start start 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 1 librados: setting wanted keys 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 1 librados: calling monclient init 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 10 monclient: init 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 10 monclient: _add_conns ranks=[6,0,2,8,7,3,5,4,1] 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 10 monclient(hunting): picked mon.noname-d con 0x7f59d8073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 10 monclient(hunting): picked mon.noname-a con 0x7f59d80a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 10 monclient(hunting): picked mon.noname-b con 0x7f59d813f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59d80a5440 con 0x7f59d80a57a0 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59d80a3980 con 0x7f59d813f2e0 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59d80a2d90 con 0x7f59d8073470 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dfc38640 10 monclient(hunting): authenticate will time out at 1625.975624s 2024-08-19T12:53:51.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dd9d4640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f59d8073470 0x7f59d80a1f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:36072/0 (socket says 172.21.15.26:36072) 2024-08-19T12:53:51.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.888+0000 7f59dd9d4640 1 -- 172.21.15.26:0/1507142835 learned_addr learned my addr 172.21.15.26:0/1507142835 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.890+0000 7f59dd1d3640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.26:6789/0 conn(0x7f59d80a57a0 legacy=0x7f59d80a2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.890+0000 7f59d67fc640 10 client.?.objecter ms_handle_connect 0x7f59d80a57a0 2024-08-19T12:53:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.890+0000 7f59d67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.890+0000 7f59dd9d4640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.26:6792/0 conn(0x7f59d8073470 legacy=0x7f59d80a1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.890+0000 7f59d67fc640 10 client.?.objecter ms_handle_connect 0x7f59d8073470 2024-08-19T12:53:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.890+0000 7f59d67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.890+0000 7f59de1d5640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.26:6790/0 conn(0x7f59d813f2e0 legacy=0x7f59d81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.890+0000 7f59d67fc640 10 client.?.objecter ms_handle_connect 0x7f59d813f2e0 2024-08-19T12:53:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1896055540 0 0) 0x7f59d80a5440 con 0x7f59d80a57a0 2024-08-19T12:53:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): my global_id is 194193 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59c4004eb0 con 0x7f59d80a57a0 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3392181538 0 0) 0x7f59d80a3980 con 0x7f59d813f2e0 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): my global_id is 27146 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59d80a5440 con 0x7f59d813f2e0 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 482726177 0 0) 0x7f59d80a2d90 con 0x7f59d8073470 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): my global_id is 194154 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59d80a3980 con 0x7f59d8073470 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1431710348 0 0) 0x7f59c4004eb0 con 0x7f59d80a57a0 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59d80a2d90 con 0x7f59d80a57a0 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f59cc0019c0 con 0x7f59d80a57a0 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): dump: 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.906 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1783300995 0 0) 0x7f59d80a3980 con 0x7f59d8073470 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59c4004eb0 con 0x7f59d8073470 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2934084772 0 0) 0x7f59d80a5440 con 0x7f59d813f2e0 2024-08-19T12:53:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59d80a3980 con 0x7f59d813f2e0 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f59c8003990 con 0x7f59d8073470 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): dump: 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.908 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f59d0001d30 con 0x7f59d813f2e0 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient(hunting): dump: 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.909 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2912944902 0 0) 0x7f59d80a2d90 con 0x7f59d80a57a0 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.26:6792/0 conn(0x7f59d8073470 legacy=0x7f59d80a1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.26:6790/0 conn(0x7f59d813f2e0 legacy=0x7f59d81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient: _finish_hunting 0 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 monclient: found mon.a 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d81371a0 con 0x7f59d80a57a0 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient: _finish_auth 0 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient: _check_auth_tickets 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59dfc38640 5 monclient: authenticate success, global_id 194193 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59dfc38640 10 monclient: _renew_subs 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59dfc38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59dfc38640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f59d80a47b0 con 0x7f59d80a57a0 2024-08-19T12:53:51.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59dfc38640 10 client.194193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59dfc38640 10 monclient: _renew_subs 2024-08-19T12:53:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59dfc38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59dfc38640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f59d813e880 con 0x7f59d80a57a0 2024-08-19T12:53:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59dfc38640 1 librados: init done 2024-08-19T12:53:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59cc001d10 con 0x7f59d80a57a0 2024-08-19T12:53:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f59cc004bc0 con 0x7f59d80a57a0 2024-08-19T12:53:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d7fff640 4 set_mon_vals no callback set 2024-08-19T12:53:51.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.891+0000 7f59d67fc640 10 monclient: dump: 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.913 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.914 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.892+0000 7f59dfc38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.892+0000 7f59dfc38640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f59b80065c0 con 0x7f59b8002860 2024-08-19T12:53:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.892+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f59cc0116d0 con 0x7f59d80a57a0 2024-08-19T12:53:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.893+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(568..568 src has 350..568) ==== 5862+0+0 (unknown 2537974883 0 0) 0x7f59cc04e320 con 0x7f59d80a57a0 2024-08-19T12:53:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.893+0000 7f59d67fc640 10 client.194193.objecter ms_dispatch 0x7f59d8000b90 osd_map(568..568 src has 350..568) 2024-08-19T12:53:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.893+0000 7f59d67fc640 3 client.194193.objecter handle_osd_map got epochs [568,568] > 0 2024-08-19T12:53:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.893+0000 7f59d67fc640 3 client.194193.objecter handle_osd_map decoding full epoch 568 2024-08-19T12:53:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.893+0000 7f59d67fc640 20 client.194193.objecter dump_active .. 0 homeless 2024-08-19T12:53:51.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.893+0000 7f59de1d5640 10 monclient: get_auth_request con 0x7f59c4041a80 auth_method 0 2024-08-19T12:53:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 10 client.194193.objecter ms_handle_connect 0x7f59b8002860 2024-08-19T12:53:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59dd9d4640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.104:6792/0 conn(0x7f59b8002860 legacy=0x7f59b8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 10 client.194193.objecter resend_mon_ops 2024-08-19T12:53:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 655484967 0 0) 0x7f59b80065c0 con 0x7f59b8002860 2024-08-19T12:53:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 10 monclient(hunting): my global_id is 136114 2024-08-19T12:53:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59d80a2d90 con 0x7f59b8002860 2024-08-19T12:53:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 10 client.194193.objecter ms_handle_connect 0x7f59c4041a80 2024-08-19T12:53:51.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1068424046 0 0) 0x7f59d80a2d90 con 0x7f59b8002860 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59b80065c0 con 0x7f59b8002860 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f59c8004120 con 0x7f59b8002860 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 10 monclient: dump: 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.916 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2487041614 0 0) 0x7f59b80065c0 con 0x7f59b8002860 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.895+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59b800a160 con 0x7f59b8002860 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.896+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f59c8005b40 con 0x7f59b8002860 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.896+0000 7f59d67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.896+0000 7f59d67fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.896+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.104:6792/0 conn(0x7f59b8002860 legacy=0x7f59b8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.896+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.104:6792/0 conn(0x7f59b8002860 legacy=0x7f59b8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.902+0000 7f59dfc38640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.902+0000 7f59dfc38640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f59b8009e50 con 0x7f59b8009110 2024-08-19T12:53:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.904+0000 7f59dd9d4640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.104:6792/0 conn(0x7f59b8009110 legacy=0x7f59b80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.904+0000 7f59d67fc640 10 client.194193.objecter ms_handle_connect 0x7f59b8009110 2024-08-19T12:53:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.904+0000 7f59d67fc640 10 client.194193.objecter resend_mon_ops 2024-08-19T12:53:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.904+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1092574783 0 0) 0x7f59b8009e50 con 0x7f59b8009110 2024-08-19T12:53:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.904+0000 7f59d67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:51.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.904+0000 7f59d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.904+0000 7f59d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.904+0000 7f59d67fc640 10 monclient(hunting): my global_id is 136123 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.904+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59b80065c0 con 0x7f59b8009110 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1861767255 0 0) 0x7f59b80065c0 con 0x7f59b8009110 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59b8009e50 con 0x7f59b8009110 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f59c80047b0 con 0x7f59b8009110 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 10 monclient: dump: 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1756317738 0 0) 0x7f59b8009e50 con 0x7f59b8009110 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f59b800c3a0 con 0x7f59b8009110 2024-08-19T12:53:51.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7545 (unknown 0 0 2682256007) 0x7f59c80080f0 con 0x7f59b8009110 2024-08-19T12:53:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.104:6792/0 conn(0x7f59b8009110 legacy=0x7f59b80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59d67fc640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.104:6792/0 conn(0x7f59b8009110 legacy=0x7f59b80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:51.922 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:51.922 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:51.922 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:51.922 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:51.922 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:51.922 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 802263, 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:51.923 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.924 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.925 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:51.926 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:51.927 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:51.930 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.932 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:51.936 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.938 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.938 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:51.938 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:51.938 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.938 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.938 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.938 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:51.938 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.939 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:51.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.942 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:51.942 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:51.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:51.942 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:51.942 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:51.942 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:51.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:51.942 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:51.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:51.942 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:51.942 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:51.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59dfc38640 10 librados: watch_flush enter 2024-08-19T12:53:51.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59dfc38640 10 librados: watch_flush exit 2024-08-19T12:53:51.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59dfc38640 20 client.194193.objecter shutdown clearing up homeless session... 2024-08-19T12:53:51.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59dfc38640 10 client.194193.objecter successfully canceled tick 2024-08-19T12:53:51.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59dfc38640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.104:6800/899887506 conn(0x7f59c4041a80 legacy=0x7f59c4043f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59dfc38640 10 monclient: shutdown 2024-08-19T12:53:51.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59dfc38640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:51.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59dfc38640 1 -- 172.21.15.26:0/1507142835 >> v1:172.21.15.26:6789/0 conn(0x7f59d80a57a0 legacy=0x7f59d80a2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:51.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.905+0000 7f59de1d5640 1 -- 172.21.15.26:0/1507142835 reap_dead start 2024-08-19T12:53:51.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.906+0000 7f59de1d5640 1 -- 172.21.15.26:0/1507142835 reap_dead start 2024-08-19T12:53:51.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.906+0000 7f59dfc38640 1 -- 172.21.15.26:0/1507142835 shutdown_connections 2024-08-19T12:53:51.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.906+0000 7f59dfc38640 1 -- 172.21.15.26:0/1507142835 >> 172.21.15.26:0/1507142835 conn(0x7f59d80abff0 msgr2=0x7f59d8134670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:51.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.906+0000 7f59dfc38640 1 -- 172.21.15.26:0/1507142835 shutdown_connections 2024-08-19T12:53:51.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.906+0000 7f59dfc38640 1 -- 172.21.15.26:0/1507142835 wait complete. 2024-08-19T12:53:51.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:51.906+0000 7f59dfc38640 1 librados: shutdown 2024-08-19T12:53:51.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:51.946 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T12:53:51.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T12:53:52.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.056+0000 7fe5c4298640 10 monclient: get_monmap_and_config 2024-08-19T12:53:52.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.056+0000 7fe5c4298640 10 monclient: build_initial_monmap 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.056+0000 7fe5c4298640 10 monclient: monmap: 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:52.056952+0000 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:52.056952+0000 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:52.057 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:52.058 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.056+0000 7fe5c4298640 1 Processor -- start 2024-08-19T12:53:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.056+0000 7fe5c4298640 1 -- start start 2024-08-19T12:53:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.056+0000 7fe5c4298640 10 monclient: init 2024-08-19T12:53:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c4298640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c4298640 10 monclient: _add_conns ranks=[4,6,2,5,1,0,8,7,3] 2024-08-19T12:53:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c4298640 10 monclient(hunting): picked mon.noname-c con 0x7fe5bc073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:52.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c4298640 10 monclient(hunting): picked mon.noname-d con 0x7fe5bc0a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:53:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c4298640 10 monclient(hunting): picked mon.noname-b con 0x7fe5bc13f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c4298640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5bc09dae0 con 0x7fe5bc13f2e0 2024-08-19T12:53:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c4298640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5bc136910 con 0x7fe5bc073470 2024-08-19T12:53:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c4298640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5bc09dd50 con 0x7fe5bc0a57a0 2024-08-19T12:53:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c4298640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c4298640 10 monclient(hunting): authenticate will time out at 1626.144423s 2024-08-19T12:53:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c2034640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fe5bc073470 0x7fe5bc136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59208/0 (socket says 172.21.15.26:59208) 2024-08-19T12:53:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c2835640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fe5bc13f2e0 0x7fe5bc1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47866/0 (socket says 172.21.15.26:47866) 2024-08-19T12:53:52.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.057+0000 7fe5c2034640 1 -- 172.21.15.26:0/3562184674 learned_addr learned my addr 172.21.15.26:0/3562184674 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:52.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c2034640 1 -- 172.21.15.26:0/3562184674 >> v1:172.21.15.26:6791/0 conn(0x7fe5bc073470 legacy=0x7fe5bc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1833640 1 -- 172.21.15.26:0/3562184674 >> v1:172.21.15.26:6792/0 conn(0x7fe5bc0a57a0 legacy=0x7fe5bc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c2835640 1 -- 172.21.15.26:0/3562184674 >> v1:172.21.15.26:6790/0 conn(0x7fe5bc13f2e0 legacy=0x7fe5bc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1092912111 0 0) 0x7fe5bc136910 con 0x7fe5bc073470 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 10 monclient(hunting): _negotiate 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 10 monclient(hunting): my global_id is 184414 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5a8003520 con 0x7fe5bc073470 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3633838449 0 0) 0x7fe5bc09dd50 con 0x7fe5bc0a57a0 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 10 monclient(hunting): _negotiate 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:52.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 10 monclient(hunting): my global_id is 194163 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5bc136910 con 0x7fe5bc0a57a0 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009892251 0 0) 0x7fe5bc09dae0 con 0x7fe5bc13f2e0 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 10 monclient(hunting): _negotiate 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 10 monclient(hunting): my global_id is 27155 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.059+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5bc09dd50 con 0x7fe5bc13f2e0 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 179610975 0 0) 0x7fe5a8003520 con 0x7fe5bc073470 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 >> v1:172.21.15.26:6792/0 conn(0x7fe5bc0a57a0 legacy=0x7fe5bc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 >> v1:172.21.15.26:6790/0 conn(0x7fe5bc13f2e0 legacy=0x7fe5bc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 10 monclient: _finish_hunting 0 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 1 monclient: found mon.noname-c 2024-08-19T12:53:52.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:53:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5bc09e030 con 0x7fe5bc073470 2024-08-19T12:53:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 10 monclient: _finish_auth 0 2024-08-19T12:53:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 10 monclient: _check_auth_tickets 2024-08-19T12:53:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe5b8001ae0 con 0x7fe5bc073470 2024-08-19T12:53:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c4298640 5 monclient: authenticate success, global_id 184414 2024-08-19T12:53:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c4298640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:52.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:53:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 10 monclient: dump: 2024-08-19T12:53:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:52.072 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c4298640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5b80048a0 con 0x7fe5bc073470 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 1 -- 172.21.15.26:0/3562184674 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe5b8004e10 con 0x7fe5bc073470 2024-08-19T12:53:52.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c4298640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c4298640 4 set_mon_vals no callback set 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c1032640 10 monclient: dump: 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:52.074 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c4298640 10 monclient: shutdown 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c4298640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c4298640 1 -- 172.21.15.26:0/3562184674 >> v1:172.21.15.26:6791/0 conn(0x7fe5bc073470 legacy=0x7fe5bc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c4298640 1 -- 172.21.15.26:0/3562184674 shutdown_connections 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c4298640 1 -- 172.21.15.26:0/3562184674 >> 172.21.15.26:0/3562184674 conn(0x7fe5bc0abff0 msgr2=0x7fe5bc0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c4298640 1 -- 172.21.15.26:0/3562184674 shutdown_connections 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.060+0000 7fe5c4298640 1 -- 172.21.15.26:0/3562184674 wait complete. 2024-08-19T12:53:52.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.061+0000 7fe5c4298640 10 monclient: build_initial_monmap 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.061+0000 7fe5c4298640 10 monclient: monmap: 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.061+0000 7fe5c4298640 1 librados: starting msgr at 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.061+0000 7fe5c4298640 1 librados: starting objecter 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.061+0000 7fe5c4298640 1 Processor -- start 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.061+0000 7fe5c4298640 1 -- start start 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.061+0000 7fe5c4298640 1 librados: setting wanted keys 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.061+0000 7fe5c4298640 1 librados: calling monclient init 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.061+0000 7fe5c4298640 10 monclient: init 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c4298640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c4298640 10 monclient: _add_conns ranks=[8,4,0,2,7,6,3,5,1] 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c4298640 10 monclient(hunting): picked mon.noname-e con 0x7fe5bc073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:52.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c4298640 10 monclient(hunting): picked mon.noname-c con 0x7fe5bc0a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c4298640 10 monclient(hunting): picked mon.noname-a con 0x7fe5bc13f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c4298640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5bc12e9e0 con 0x7fe5bc13f2e0 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c4298640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5bc06d8f0 con 0x7fe5bc0a57a0 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c4298640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5bc1423d0 con 0x7fe5bc073470 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c4298640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c4298640 10 monclient(hunting): authenticate will time out at 1626.149378s 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c1833640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fe5bc0a57a0 0x7fe5bc12e2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59216/0 (socket says 172.21.15.26:59216) 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c2034640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fe5bc073470 0x7fe5bc12dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40688/0 (socket says 172.21.15.26:40688) 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.062+0000 7fe5c1833640 1 -- 172.21.15.26:0/105577681 learned_addr learned my addr 172.21.15.26:0/105577681 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5c2835640 1 -- 172.21.15.26:0/105577681 >> v1:172.21.15.26:6789/0 conn(0x7fe5bc13f2e0 legacy=0x7fe5bc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5c2034640 1 -- 172.21.15.26:0/105577681 >> v1:172.21.15.26:6793/0 conn(0x7fe5bc073470 legacy=0x7fe5bc12dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 client.?.objecter ms_handle_connect 0x7fe5bc13f2e0 2024-08-19T12:53:52.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 client.?.objecter ms_handle_connect 0x7fe5bc073470 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 client.?.objecter ms_handle_connect 0x7fe5bc0a57a0 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5c1833640 1 -- 172.21.15.26:0/105577681 >> v1:172.21.15.26:6791/0 conn(0x7fe5bc0a57a0 legacy=0x7fe5bc12e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2219330642 0 0) 0x7fe5bc1423d0 con 0x7fe5bc073470 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 monclient(hunting): my global_id is 184328 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5a0004eb0 con 0x7fe5bc073470 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1006300772 0 0) 0x7fe5bc06d8f0 con 0x7fe5bc0a57a0 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:52.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 monclient(hunting): my global_id is 184423 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5bc1423d0 con 0x7fe5bc0a57a0 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 727789677 0 0) 0x7fe5bc12e9e0 con 0x7fe5bc13f2e0 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 10 monclient(hunting): my global_id is 194202 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5bc06d8f0 con 0x7fe5bc13f2e0 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.064+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2950931270 0 0) 0x7fe5a0004eb0 con 0x7fe5bc073470 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5bc12e9e0 con 0x7fe5bc073470 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe5b8004d90 con 0x7fe5bc073470 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:52.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient(hunting): dump: 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:52.081 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1392740521 0 0) 0x7fe5bc1423d0 con 0x7fe5bc0a57a0 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5a0004eb0 con 0x7fe5bc0a57a0 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 691960599 0 0) 0x7fe5bc06d8f0 con 0x7fe5bc13f2e0 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5bc1423d0 con 0x7fe5bc13f2e0 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe5ac003610 con 0x7fe5bc0a57a0 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient(hunting): dump: 2024-08-19T12:53:52.082 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:52.083 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe5b4001fd0 con 0x7fe5bc13f2e0 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient(hunting): dump: 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:52.084 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3636790444 0 0) 0x7fe5bc12e9e0 con 0x7fe5bc073470 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 >> v1:172.21.15.26:6791/0 conn(0x7fe5bc0a57a0 legacy=0x7fe5bc12e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 >> v1:172.21.15.26:6789/0 conn(0x7fe5bc13f2e0 legacy=0x7fe5bc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 monclient: found mon.e 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:52.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5bc12f340 con 0x7fe5bc073470 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient: _finish_auth 0 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5c4298640 5 monclient: authenticate success, global_id 184328 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5c4298640 10 monclient: _renew_subs 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5c4298640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5c4298640 1 -- 172.21.15.26:0/105577681 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe5bc1300d0 con 0x7fe5bc073470 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5c4298640 10 client.184328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5c4298640 10 monclient: _renew_subs 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5c4298640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5c4298640 1 -- 172.21.15.26:0/105577681 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fe5bc141d10 con 0x7fe5bc073470 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5c4298640 1 librados: init done 2024-08-19T12:53:52.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5b8001d30 con 0x7fe5bc073470 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe5b80048a0 con 0x7fe5bc073470 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5c0831640 4 set_mon_vals no callback set 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.065+0000 7fe5b2ffd640 10 monclient: dump: 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:52.087 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.066+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe5b8011a00 con 0x7fe5bc073470 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.066+0000 7fe5c4298640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.066+0000 7fe5c4298640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.066+0000 7fe5c4298640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:52.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.066+0000 7fe5c4298640 1 -- 172.21.15.26:0/105577681 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5a4005080 con 0x7fe5bc073470 2024-08-19T12:53:52.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.066+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(569..569 src has 350..569) ==== 5862+0+0 (unknown 3549424396 0 0) 0x7fe5b804e2b0 con 0x7fe5bc073470 2024-08-19T12:53:52.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.066+0000 7fe5b2ffd640 10 client.184328.objecter ms_dispatch 0x7fe5bc000b90 osd_map(569..569 src has 350..569) 2024-08-19T12:53:52.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.066+0000 7fe5b2ffd640 3 client.184328.objecter handle_osd_map got epochs [569,569] > 0 2024-08-19T12:53:52.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.066+0000 7fe5b2ffd640 3 client.184328.objecter handle_osd_map decoding full epoch 569 2024-08-19T12:53:52.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.066+0000 7fe5b2ffd640 20 client.184328.objecter dump_active .. 0 homeless 2024-08-19T12:53:52.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.066+0000 7fe5c1833640 10 monclient: get_auth_request con 0x7fe5a0041a80 auth_method 0 2024-08-19T12:53:52.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.068+0000 7fe5b2ffd640 10 client.184328.objecter ms_handle_connect 0x7fe5a0041a80 2024-08-19T12:53:52.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.070+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fe5b80174b0 con 0x7fe5bc073470 2024-08-19T12:53:52.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.070+0000 7fe5b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:52.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.070+0000 7fe5b2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:52.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.326+0000 7fe5c4298640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T12:53:52.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.326+0000 7fe5c4298640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T12:53:52.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.326+0000 7fe5c4298640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:52.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.326+0000 7fe5c4298640 1 -- 172.21.15.26:0/105577681 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe5a4005c40 con 0x7fe5bc073470 2024-08-19T12:53:52.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.328+0000 7fe5b2ffd640 1 -- 172.21.15.26:0/105577681 <== mon.8 v1:172.21.15.26:6793/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fe5b8016db0 con 0x7fe5bc073470 2024-08-19T12:53:52.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.328+0000 7fe5b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T12:53:52.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.328+0000 7fe5b2ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:52.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.330+0000 7fe5c4298640 10 librados: watch_flush enter 2024-08-19T12:53:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.331+0000 7fe5c4298640 10 librados: watch_flush exit 2024-08-19T12:53:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.331+0000 7fe5c4298640 20 client.184328.objecter shutdown clearing up homeless session... 2024-08-19T12:53:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.331+0000 7fe5c4298640 10 client.184328.objecter successfully canceled tick 2024-08-19T12:53:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.331+0000 7fe5c4298640 1 -- 172.21.15.26:0/105577681 >> v1:172.21.15.104:6800/899887506 conn(0x7fe5a0041a80 legacy=0x7fe5a0043f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.331+0000 7fe5c4298640 10 monclient: shutdown 2024-08-19T12:53:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.331+0000 7fe5c4298640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.331+0000 7fe5c4298640 1 -- 172.21.15.26:0/105577681 >> v1:172.21.15.26:6793/0 conn(0x7fe5bc073470 legacy=0x7fe5bc12dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.331+0000 7fe5c4298640 1 -- 172.21.15.26:0/105577681 shutdown_connections 2024-08-19T12:53:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.331+0000 7fe5c4298640 1 -- 172.21.15.26:0/105577681 >> 172.21.15.26:0/105577681 conn(0x7fe5bc0abff0 msgr2=0x7fe5bc0a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.331+0000 7fe5c4298640 1 -- 172.21.15.26:0/105577681 shutdown_connections 2024-08-19T12:53:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.331+0000 7fe5c4298640 1 -- 172.21.15.26:0/105577681 wait complete. 2024-08-19T12:53:52.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:52.331+0000 7fe5c4298640 1 librados: shutdown 2024-08-19T12:53:52.342 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T12:53:52.501 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.496+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.501 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.496+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=13,logm=45} crc {auth=540576284,config=1097430493,health=3315045273,kv=1344821656,logm=1440695642}) 2024-08-19T12:53:52.501 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.496+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=13,logm=47} crc {auth=1466210678,config=1097430493,health=1974364611,kv=1344821656,logm=1170265503}) 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.496+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.496+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=13,logm=45} crc {auth=540576284,config=1097430493,health=3315045273,kv=1344821656,logm=1440695642}) 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.496+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=13,logm=45} crc {auth=540576284,config=1097430493,health=3779923945,kv=1344821656,logm=3022207299}) 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=13,logm=45} crc {auth=540576284,config=1097430493,health=3315045273,kv=1344821656,logm=1440695642}) 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=29,config=2,health=10,kv=13,logm=46} crc {auth=693500605,config=1097430493,health=954801684,kv=1344821656,logm=3485243457}) 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=13,logm=45} crc {auth=540576284,config=1097430493,health=3315045273,kv=1344821656,logm=1440695642}) 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=29,config=2,health=12,kv=13,logm=44} crc {auth=693500605,config=1097430493,health=3315045273,kv=701615758,logm=2258761297}) 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=13,logm=45} crc {auth=540576284,config=1097430493,health=3315045273,kv=1344821656,logm=1440695642}) 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=29,config=2,health=12,kv=12,logm=45} crc {auth=693500605,config=1097430493,health=3315045273,kv=2859555996,logm=1649557837}) 2024-08-19T12:53:52.502 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=13,logm=45} crc {auth=540576284,config=1097430493,health=3315045273,kv=1344821656,logm=1440695642}) 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=25,config=1,health=11,kv=13,logm=50} crc {auth=2171751157,config=1228700967,health=1934421924,kv=1344821656,logm=3739165604}) 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=13,logm=45} crc {auth=540576284,config=1097430493,health=3315045273,kv=1344821656,logm=1440695642}) 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=28,config=2,health=12,kv=13,logm=45} crc {auth=3113934736,config=1097430493,health=3315045273,kv=1344821656,logm=688557740}) 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=13,logm=45} crc {auth=540576284,config=1097430493,health=3315045273,kv=1344821656,logm=1440695642}) 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.497+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=28,config=2,health=11,kv=12,logm=47} crc {auth=2295424485,config=1097430493,health=1934421924,kv=3900667136,logm=2994063562}) 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.498+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.498+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853464327}) 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.498+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3073959036}) 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.498+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.503 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.498+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853464327}) 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.498+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2505786879}) 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.498+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.498+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853464327}) 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.498+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=242886660}) 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.498+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.498+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853464327}) 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.498+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=148351484}) 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.499+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.499+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853464327}) 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.499+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3813246397}) 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.499+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.499+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853464327}) 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.499+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2168619665}) 2024-08-19T12:53:52.504 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.499+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.505 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.499+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853464327}) 2024-08-19T12:53:52.505 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.499+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2640359232}) 2024-08-19T12:53:52.505 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.499+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.505 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.499+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853464327}) 2024-08-19T12:53:52.505 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.499+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3963897228}) 2024-08-19T12:53:52.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434137984}) 2024-08-19T12:53:52.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500577560}) 2024-08-19T12:53:52.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.929 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434137984}) 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3306892449}) 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434137984}) 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3264799380}) 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434137984}) 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3000277727}) 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434137984}) 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1896619451}) 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434137984}) 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4132413952}) 2024-08-19T12:53:52.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434137984}) 2024-08-19T12:53:52.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.927+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1594308151}) 2024-08-19T12:53:52.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434137984}) 2024-08-19T12:53:52.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2751188404}) 2024-08-19T12:53:52.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166149839}) 2024-08-19T12:53:52.936 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2099271872}) 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166149839}) 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532331854}) 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166149839}) 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4024405689}) 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166149839}) 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=407597951}) 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166149839}) 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2910309377}) 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.937 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166149839}) 2024-08-19T12:53:52.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2085461945}) 2024-08-19T12:53:52.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166149839}) 2024-08-19T12:53:52.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2646037790}) 2024-08-19T12:53:52.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166149839}) 2024-08-19T12:53:52.938 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.935+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4130246467}) 2024-08-19T12:53:52.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255084448}) 2024-08-19T12:53:52.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2610482664}) 2024-08-19T12:53:52.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255084448}) 2024-08-19T12:53:52.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1437918195}) 2024-08-19T12:53:52.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255084448}) 2024-08-19T12:53:52.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3004678668}) 2024-08-19T12:53:52.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255084448}) 2024-08-19T12:53:52.949 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1179328275}) 2024-08-19T12:53:52.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255084448}) 2024-08-19T12:53:52.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2913374610}) 2024-08-19T12:53:52.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.944+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255084448}) 2024-08-19T12:53:52.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.944+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=193636307}) 2024-08-19T12:53:52.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.944+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.944+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255084448}) 2024-08-19T12:53:52.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.944+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2797763771}) 2024-08-19T12:53:52.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.944+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.944+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255084448}) 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.944+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3852557475}) 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372879760}) 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1869699698}) 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372879760}) 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2173823323}) 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372879760}) 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2230390952}) 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372879760}) 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3924264132}) 2024-08-19T12:53:52.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372879760}) 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1832911940}) 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372879760}) 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=918215785}) 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372879760}) 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4002189752}) 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372879760}) 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.945+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2003313425}) 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597094938}) 2024-08-19T12:53:52.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3529850034}) 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597094938}) 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=149511799}) 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597094938}) 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=711751946}) 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597094938}) 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3685315636}) 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597094938}) 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2132515552}) 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.953 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597094938}) 2024-08-19T12:53:52.954 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2444516233}) 2024-08-19T12:53:52.954 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.954 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597094938}) 2024-08-19T12:53:52.954 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3607529405}) 2024-08-19T12:53:52.954 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:52.954 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597094938}) 2024-08-19T12:53:52.954 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.946+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3447879044}) 2024-08-19T12:53:53.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269851825}) 2024-08-19T12:53:53.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=536138781}) 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269851825}) 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1456574258}) 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269851825}) 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4237911726}) 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269851825}) 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=885782154}) 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269851825}) 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3886517240}) 2024-08-19T12:53:53.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269851825}) 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1399219631}) 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269851825}) 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1854448779}) 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269851825}) 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.996+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3838330613}) 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787601097}) 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1963809086}) 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787601097}) 2024-08-19T12:53:53.016 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606013561}) 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787601097}) 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1617411134}) 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787601097}) 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3233387775}) 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787601097}) 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2606083413}) 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787601097}) 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=921841174}) 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.017 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787601097}) 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1344266506}) 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787601097}) 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.998+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=510121974}) 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795557112}) 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1883767368}) 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795557112}) 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=366924762}) 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795557112}) 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1243629191}) 2024-08-19T12:53:53.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795557112}) 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1544253596}) 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795557112}) 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1450086706}) 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795557112}) 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3446281912}) 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795557112}) 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1343605724}) 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795557112}) 2024-08-19T12:53:53.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:52.999+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3313281576}) 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680533138}) 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1400315826}) 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680533138}) 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=918703388}) 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680533138}) 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3365098289}) 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680533138}) 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=366718142}) 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680533138}) 2024-08-19T12:53:53.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=698946134}) 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680533138}) 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1470280006}) 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680533138}) 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1446639000}) 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680533138}) 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4225708756}) 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245544038}) 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1520032977}) 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.021 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245544038}) 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752241307}) 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245544038}) 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=203688736}) 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245544038}) 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2255918784}) 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245544038}) 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1741656109}) 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245544038}) 2024-08-19T12:53:53.022 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3111171008}) 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245544038}) 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4214763910}) 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245544038}) 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.002+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=293561960}) 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.003+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.003+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565401292}) 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.003+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2175679411}) 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.003+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565401292}) 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3798491101}) 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.023 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565401292}) 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3775670815}) 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565401292}) 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2587685038}) 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565401292}) 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2266942927}) 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565401292}) 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1912898561}) 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565401292}) 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=287325448}) 2024-08-19T12:53:53.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565401292}) 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.004+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1746232457}) 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124191201}) 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4052571277}) 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124191201}) 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=878709569}) 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124191201}) 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2710055433}) 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124191201}) 2024-08-19T12:53:53.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2629887282}) 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124191201}) 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1341596284}) 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124191201}) 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1862381536}) 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124191201}) 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3693090853}) 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124191201}) 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.005+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3850721953}) 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.006+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.006+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928530757}) 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.006+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2497717847}) 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.006+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.006+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928530757}) 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.006+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4229988557}) 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.006+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.006+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928530757}) 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.006+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3088246762}) 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.006+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.006+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928530757}) 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=673699944}) 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928530757}) 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3153780027}) 2024-08-19T12:53:53.027 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928530757}) 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1435525534}) 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928530757}) 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=97347595}) 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928530757}) 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.007+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4266809276}) 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259434756}) 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146127530}) 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259434756}) 2024-08-19T12:53:53.028 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913470213}) 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259434756}) 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2394715436}) 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259434756}) 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2221468443}) 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259434756}) 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1103216381}) 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259434756}) 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3290230550}) 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259434756}) 2024-08-19T12:53:53.029 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2028853097}) 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259434756}) 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.008+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1181020395}) 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484683343}) 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2311125178}) 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484683343}) 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3948140874}) 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484683343}) 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3254677939}) 2024-08-19T12:53:53.030 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484683343}) 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=140258259}) 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484683343}) 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2801757394}) 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484683343}) 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3609958698}) 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484683343}) 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1154640820}) 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484683343}) 2024-08-19T12:53:53.031 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.010+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1106440740}) 2024-08-19T12:53:53.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207958528}) 2024-08-19T12:53:53.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1940296236}) 2024-08-19T12:53:53.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.054 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207958528}) 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1799848982}) 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207958528}) 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2892240880}) 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207958528}) 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=449308507}) 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207958528}) 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4162349936}) 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.015+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207958528}) 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1221554522}) 2024-08-19T12:53:53.055 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207958528}) 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=181040878}) 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207958528}) 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1114517558}) 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584093204}) 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3623634105}) 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584093204}) 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3744759847}) 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584093204}) 2024-08-19T12:53:53.056 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2711534543}) 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584093204}) 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=166507677}) 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584093204}) 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2224629326}) 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584093204}) 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=129367529}) 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584093204}) 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3501352332}) 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.057 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584093204}) 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.018+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3626035795}) 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942316601}) 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3747597682}) 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942316601}) 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3795128819}) 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942316601}) 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2539119222}) 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942316601}) 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=712183130}) 2024-08-19T12:53:53.058 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942316601}) 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=919829027}) 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942316601}) 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2885522701}) 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942316601}) 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4015139100}) 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942316601}) 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.019+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1102989942}) 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448122517}) 2024-08-19T12:53:53.059 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=229133563}) 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448122517}) 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1766325577}) 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448122517}) 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3875047132}) 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448122517}) 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1120608038}) 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448122517}) 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=703977994}) 2024-08-19T12:53:53.060 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448122517}) 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=864414605}) 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448122517}) 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3221343168}) 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448122517}) 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.021+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1496057941}) 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704131360}) 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3090150419}) 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704131360}) 2024-08-19T12:53:53.061 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=852914412}) 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704131360}) 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3075411006}) 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704131360}) 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3643996237}) 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704131360}) 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4265982186}) 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704131360}) 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2431244763}) 2024-08-19T12:53:53.062 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704131360}) 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2540650801}) 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704131360}) 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3882859902}) 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776199955}) 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3166465010}) 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776199955}) 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232985397}) 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776199955}) 2024-08-19T12:53:53.063 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=49885504}) 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776199955}) 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=430392075}) 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776199955}) 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=910206094}) 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776199955}) 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4267862842}) 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776199955}) 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=691364106}) 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.064 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776199955}) 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.024+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2820975657}) 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774043038}) 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1504303997}) 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774043038}) 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2249070337}) 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774043038}) 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=563700989}) 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774043038}) 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1521089730}) 2024-08-19T12:53:53.065 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774043038}) 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2947094310}) 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774043038}) 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1191983713}) 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774043038}) 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3919011717}) 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774043038}) 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=811065677}) 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.066 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440646362}) 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1008522608}) 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440646362}) 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3621374235}) 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440646362}) 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3670738843}) 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440646362}) 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1191593325}) 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440646362}) 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=703893881}) 2024-08-19T12:53:53.067 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.068 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440646362}) 2024-08-19T12:53:53.068 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3970727116}) 2024-08-19T12:53:53.068 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.068 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440646362}) 2024-08-19T12:53:53.068 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1845631700}) 2024-08-19T12:53:53.068 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.068 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440646362}) 2024-08-19T12:53:53.068 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.027+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3237151521}) 2024-08-19T12:53:53.068 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.068 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690928761}) 2024-08-19T12:53:53.068 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4231763888}) 2024-08-19T12:53:53.068 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.069 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690928761}) 2024-08-19T12:53:53.069 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=763171022}) 2024-08-19T12:53:53.069 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.069 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690928761}) 2024-08-19T12:53:53.069 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1947998916}) 2024-08-19T12:53:53.069 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.069 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690928761}) 2024-08-19T12:53:53.069 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2405840982}) 2024-08-19T12:53:53.069 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.069 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690928761}) 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1356227270}) 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690928761}) 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=115834035}) 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690928761}) 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2722942795}) 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690928761}) 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.028+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=493457436}) 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.029+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.029+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307769649}) 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.029+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2811162709}) 2024-08-19T12:53:53.070 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.029+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.029+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307769649}) 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.029+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1967588531}) 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307769649}) 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=895594796}) 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307769649}) 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1708161668}) 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307769649}) 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=934643180}) 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307769649}) 2024-08-19T12:53:53.071 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3224446929}) 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307769649}) 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2892154923}) 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307769649}) 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.030+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3321837245}) 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480688204}) 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3044722199}) 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480688204}) 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3991317626}) 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.072 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480688204}) 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3811296101}) 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480688204}) 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=968814525}) 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480688204}) 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4188484300}) 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480688204}) 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1182001385}) 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480688204}) 2024-08-19T12:53:53.073 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=914852760}) 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480688204}) 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.031+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2573899623}) 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340858361}) 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3414947803}) 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340858361}) 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3328543379}) 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340858361}) 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4113714032}) 2024-08-19T12:53:53.074 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340858361}) 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2849855403}) 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340858361}) 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=406882630}) 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340858361}) 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1612216175}) 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340858361}) 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3569045815}) 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340858361}) 2024-08-19T12:53:53.075 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.032+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4224155761}) 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121911356}) 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=813055695}) 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121911356}) 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4245411644}) 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121911356}) 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2014227538}) 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121911356}) 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2564394384}) 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121911356}) 2024-08-19T12:53:53.076 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2758902409}) 2024-08-19T12:53:53.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.033+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121911356}) 2024-08-19T12:53:53.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2892703749}) 2024-08-19T12:53:53.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121911356}) 2024-08-19T12:53:53.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1599307189}) 2024-08-19T12:53:53.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121911356}) 2024-08-19T12:53:53.077 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=709076789}) 2024-08-19T12:53:53.181 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.181 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742648064}) 2024-08-19T12:53:53.181 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1598190511}) 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742648064}) 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1171701032}) 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742648064}) 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3004264233}) 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742648064}) 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1493206429}) 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742648064}) 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2885919371}) 2024-08-19T12:53:53.182 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742648064}) 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2263877327}) 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742648064}) 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2151266326}) 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742648064}) 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.177+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1552992572}) 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498113705}) 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2417936960}) 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498113705}) 2024-08-19T12:53:53.183 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2847282875}) 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498113705}) 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3535804877}) 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498113705}) 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3787396463}) 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498113705}) 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2677518585}) 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498113705}) 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=460574931}) 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.184 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498113705}) 2024-08-19T12:53:53.185 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4225052347}) 2024-08-19T12:53:53.185 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.185 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498113705}) 2024-08-19T12:53:53.185 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.179+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1865676289}) 2024-08-19T12:53:53.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704281566}) 2024-08-19T12:53:53.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1810062273}) 2024-08-19T12:53:53.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704281566}) 2024-08-19T12:53:53.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1941690517}) 2024-08-19T12:53:53.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704281566}) 2024-08-19T12:53:53.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=425256389}) 2024-08-19T12:53:53.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.242 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704281566}) 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3804424771}) 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704281566}) 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3109734037}) 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.229+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704281566}) 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.230+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1935529482}) 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.230+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.230+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704281566}) 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.230+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2515745368}) 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.230+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.230+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704281566}) 2024-08-19T12:53:53.243 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.230+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1622680510}) 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603695811}) 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3059035869}) 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603695811}) 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614543176}) 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603695811}) 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=78809641}) 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603695811}) 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1470657693}) 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.244 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603695811}) 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=725851343}) 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603695811}) 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2057658156}) 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603695811}) 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2239546169}) 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603695811}) 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.231+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4271308511}) 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832039225}) 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2678068472}) 2024-08-19T12:53:53.245 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832039225}) 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910356157}) 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832039225}) 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3942855158}) 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832039225}) 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=588895640}) 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832039225}) 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1900524729}) 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832039225}) 2024-08-19T12:53:53.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3294745323}) 2024-08-19T12:53:53.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832039225}) 2024-08-19T12:53:53.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3837482980}) 2024-08-19T12:53:53.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832039225}) 2024-08-19T12:53:53.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.232+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=635057161}) 2024-08-19T12:53:53.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.233+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.233+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957381169}) 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.233+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2262823471}) 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.233+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.233+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957381169}) 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.233+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2799957024}) 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.233+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.233+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957381169}) 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.233+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4234257334}) 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.233+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957381169}) 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3885601945}) 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957381169}) 2024-08-19T12:53:53.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=894475415}) 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957381169}) 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4045867603}) 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957381169}) 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1558369648}) 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957381169}) 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.234+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4024999836}) 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740035156}) 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1394830546}) 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.249 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740035156}) 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3402837993}) 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740035156}) 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1861433831}) 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740035156}) 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4191000789}) 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740035156}) 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2827899949}) 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740035156}) 2024-08-19T12:53:53.250 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4011399656}) 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740035156}) 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=747646742}) 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740035156}) 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.235+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3418350060}) 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323129433}) 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1785095184}) 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323129433}) 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=551230936}) 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.251 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323129433}) 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2720035554}) 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323129433}) 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1219920933}) 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323129433}) 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4059610357}) 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323129433}) 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1531775436}) 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323129433}) 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1578850424}) 2024-08-19T12:53:53.252 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323129433}) 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.236+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=201632467}) 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062312574}) 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787263521}) 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062312574}) 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1219174745}) 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062312574}) 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=151230047}) 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.253 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062312574}) 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=512379469}) 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062312574}) 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1839577284}) 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062312574}) 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1489956566}) 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062312574}) 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3216498693}) 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062312574}) 2024-08-19T12:53:53.254 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.238+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=214347059}) 2024-08-19T12:53:53.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491346695}) 2024-08-19T12:53:53.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3042947608}) 2024-08-19T12:53:53.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491346695}) 2024-08-19T12:53:53.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1584542153}) 2024-08-19T12:53:53.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491346695}) 2024-08-19T12:53:53.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=828158590}) 2024-08-19T12:53:53.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491346695}) 2024-08-19T12:53:53.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1068807945}) 2024-08-19T12:53:53.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491346695}) 2024-08-19T12:53:53.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2170243905}) 2024-08-19T12:53:53.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491346695}) 2024-08-19T12:53:53.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3736881764}) 2024-08-19T12:53:53.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491346695}) 2024-08-19T12:53:53.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1144385948}) 2024-08-19T12:53:53.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491346695}) 2024-08-19T12:53:53.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1218025194}) 2024-08-19T12:53:53.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724396460}) 2024-08-19T12:53:53.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3484289834}) 2024-08-19T12:53:53.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724396460}) 2024-08-19T12:53:53.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1006965503}) 2024-08-19T12:53:53.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724396460}) 2024-08-19T12:53:53.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2605147530}) 2024-08-19T12:53:53.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724396460}) 2024-08-19T12:53:53.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3053779656}) 2024-08-19T12:53:53.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724396460}) 2024-08-19T12:53:53.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=433108434}) 2024-08-19T12:53:53.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724396460}) 2024-08-19T12:53:53.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1354445591}) 2024-08-19T12:53:53.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724396460}) 2024-08-19T12:53:53.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2446850883}) 2024-08-19T12:53:53.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724396460}) 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3045312603}) 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582440889}) 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1509697863}) 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582440889}) 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1535094443}) 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582440889}) 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3882899705}) 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582440889}) 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4221864600}) 2024-08-19T12:53:53.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582440889}) 2024-08-19T12:53:53.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=841599796}) 2024-08-19T12:53:53.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582440889}) 2024-08-19T12:53:53.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3084193179}) 2024-08-19T12:53:53.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582440889}) 2024-08-19T12:53:53.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1256779213}) 2024-08-19T12:53:53.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582440889}) 2024-08-19T12:53:53.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=791709593}) 2024-08-19T12:53:53.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648098914}) 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272867859}) 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648098914}) 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3643638355}) 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648098914}) 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=188778844}) 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648098914}) 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2153660166}) 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648098914}) 2024-08-19T12:53:53.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2003875150}) 2024-08-19T12:53:53.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648098914}) 2024-08-19T12:53:53.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3822161243}) 2024-08-19T12:53:53.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648098914}) 2024-08-19T12:53:53.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=655475560}) 2024-08-19T12:53:53.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648098914}) 2024-08-19T12:53:53.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1645430898}) 2024-08-19T12:53:53.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879691213}) 2024-08-19T12:53:53.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=398234407}) 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879691213}) 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3344269249}) 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879691213}) 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1492757751}) 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879691213}) 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3674051832}) 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879691213}) 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2223236103}) 2024-08-19T12:53:53.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879691213}) 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1345427848}) 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879691213}) 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2440981191}) 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879691213}) 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=840356854}) 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421029832}) 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1630819138}) 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421029832}) 2024-08-19T12:53:53.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2591860054}) 2024-08-19T12:53:53.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421029832}) 2024-08-19T12:53:53.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1900834433}) 2024-08-19T12:53:53.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421029832}) 2024-08-19T12:53:53.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1625434450}) 2024-08-19T12:53:53.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421029832}) 2024-08-19T12:53:53.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3806256613}) 2024-08-19T12:53:53.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421029832}) 2024-08-19T12:53:53.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1762097174}) 2024-08-19T12:53:53.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421029832}) 2024-08-19T12:53:53.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1930505918}) 2024-08-19T12:53:53.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421029832}) 2024-08-19T12:53:53.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=577611230}) 2024-08-19T12:53:53.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489047648}) 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=56460369}) 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489047648}) 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3215262637}) 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489047648}) 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=339071528}) 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489047648}) 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1318878702}) 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489047648}) 2024-08-19T12:53:53.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1837670680}) 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489047648}) 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=624682409}) 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489047648}) 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2235928465}) 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489047648}) 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.256+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3060469413}) 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892129215}) 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533609671}) 2024-08-19T12:53:53.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892129215}) 2024-08-19T12:53:53.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3269998409}) 2024-08-19T12:53:53.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892129215}) 2024-08-19T12:53:53.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1535002132}) 2024-08-19T12:53:53.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892129215}) 2024-08-19T12:53:53.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2799228770}) 2024-08-19T12:53:53.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892129215}) 2024-08-19T12:53:53.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=450259651}) 2024-08-19T12:53:53.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892129215}) 2024-08-19T12:53:53.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=329145181}) 2024-08-19T12:53:53.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892129215}) 2024-08-19T12:53:53.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2728644988}) 2024-08-19T12:53:53.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892129215}) 2024-08-19T12:53:53.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.257+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3806067808}) 2024-08-19T12:53:53.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.258+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.258+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196625490}) 2024-08-19T12:53:53.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.258+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=787166715}) 2024-08-19T12:53:53.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196625490}) 2024-08-19T12:53:53.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1938324017}) 2024-08-19T12:53:53.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196625490}) 2024-08-19T12:53:53.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=301869778}) 2024-08-19T12:53:53.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196625490}) 2024-08-19T12:53:53.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4075336318}) 2024-08-19T12:53:53.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196625490}) 2024-08-19T12:53:53.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1871798247}) 2024-08-19T12:53:53.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196625490}) 2024-08-19T12:53:53.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1866043800}) 2024-08-19T12:53:53.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196625490}) 2024-08-19T12:53:53.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2165922325}) 2024-08-19T12:53:53.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196625490}) 2024-08-19T12:53:53.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.259+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=685547674}) 2024-08-19T12:53:53.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461005179}) 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=868588425}) 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461005179}) 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1967480328}) 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461005179}) 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4010876377}) 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461005179}) 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1717173314}) 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461005179}) 2024-08-19T12:53:53.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=773020656}) 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461005179}) 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1724563531}) 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461005179}) 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1681223533}) 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461005179}) 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.260+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=290552905}) 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589837383}) 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254670829}) 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589837383}) 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3535134225}) 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589837383}) 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3877376146}) 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589837383}) 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2174313203}) 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589837383}) 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=353981140}) 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589837383}) 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=862209893}) 2024-08-19T12:53:53.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589837383}) 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1926868849}) 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589837383}) 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.261+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2961903352}) 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.262+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.262+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172986893}) 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.262+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1395274619}) 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.262+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.262+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172986893}) 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.262+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1083272659}) 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172986893}) 2024-08-19T12:53:53.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2873291155}) 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172986893}) 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1677026424}) 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172986893}) 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1197882027}) 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172986893}) 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2798550305}) 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172986893}) 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4235114048}) 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172986893}) 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.263+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4097943889}) 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522699145}) 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1104974755}) 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522699145}) 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4118411476}) 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522699145}) 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=773953517}) 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522699145}) 2024-08-19T12:53:53.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=472078571}) 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522699145}) 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4098904913}) 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522699145}) 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2172762864}) 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522699145}) 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2898270836}) 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522699145}) 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.264+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=20765025}) 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.265+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.265+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339668661}) 2024-08-19T12:53:53.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.265+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2936360378}) 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.265+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.265+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339668661}) 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.265+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2384963315}) 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.265+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.265+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339668661}) 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.265+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1291493909}) 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.265+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.265+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339668661}) 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1072384568}) 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339668661}) 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3282097494}) 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339668661}) 2024-08-19T12:53:53.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=921705365}) 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339668661}) 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=727470793}) 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339668661}) 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3581652168}) 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034351994}) 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2303868365}) 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034351994}) 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4193222185}) 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034351994}) 2024-08-19T12:53:53.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1160036853}) 2024-08-19T12:53:53.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034351994}) 2024-08-19T12:53:53.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2757675361}) 2024-08-19T12:53:53.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034351994}) 2024-08-19T12:53:53.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1598929828}) 2024-08-19T12:53:53.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034351994}) 2024-08-19T12:53:53.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1288990951}) 2024-08-19T12:53:53.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034351994}) 2024-08-19T12:53:53.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=437035706}) 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034351994}) 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3679899823}) 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537850189}) 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3290760048}) 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537850189}) 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2782175084}) 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537850189}) 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=755503850}) 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537850189}) 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2702074862}) 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537850189}) 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1641615028}) 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537850189}) 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3895369655}) 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537850189}) 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2301061332}) 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537850189}) 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1850803227}) 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.269+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384408732}) 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050224176}) 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384408732}) 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1633671827}) 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384408732}) 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2779135334}) 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384408732}) 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3818315196}) 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384408732}) 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=412276958}) 2024-08-19T12:53:53.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384408732}) 2024-08-19T12:53:53.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1795056756}) 2024-08-19T12:53:53.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384408732}) 2024-08-19T12:53:53.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1012958842}) 2024-08-19T12:53:53.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384408732}) 2024-08-19T12:53:53.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.270+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2344013522}) 2024-08-19T12:53:53.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600422375}) 2024-08-19T12:53:53.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3790502420}) 2024-08-19T12:53:53.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600422375}) 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1538571585}) 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600422375}) 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3587519503}) 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600422375}) 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=48752706}) 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600422375}) 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2461140770}) 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600422375}) 2024-08-19T12:53:53.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3191565598}) 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600422375}) 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=58167408}) 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600422375}) 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.271+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3908102373}) 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275577108}) 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3529044991}) 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275577108}) 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=430337635}) 2024-08-19T12:53:53.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275577108}) 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1089025382}) 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275577108}) 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2653375481}) 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275577108}) 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=929402650}) 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.273+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.273+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275577108}) 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.273+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2013159381}) 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.273+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.273+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275577108}) 2024-08-19T12:53:53.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.273+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3152885425}) 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.273+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.273+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275577108}) 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.273+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1408917242}) 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684674323}) 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1919615124}) 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684674323}) 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1033236329}) 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684674323}) 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3918497526}) 2024-08-19T12:53:53.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684674323}) 2024-08-19T12:53:53.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2210103456}) 2024-08-19T12:53:53.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684674323}) 2024-08-19T12:53:53.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4275169125}) 2024-08-19T12:53:53.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684674323}) 2024-08-19T12:53:53.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=652233544}) 2024-08-19T12:53:53.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684674323}) 2024-08-19T12:53:53.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=553631577}) 2024-08-19T12:53:53.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684674323}) 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.274+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2228055685}) 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041225410}) 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2117617851}) 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041225410}) 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=854048958}) 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041225410}) 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2392516717}) 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041225410}) 2024-08-19T12:53:53.344 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=63618604}) 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041225410}) 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=142961581}) 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041225410}) 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1813779550}) 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041225410}) 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4223496935}) 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041225410}) 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.276+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2548993613}) 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.345 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86751766}) 2024-08-19T12:53:53.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1432072580}) 2024-08-19T12:53:53.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86751766}) 2024-08-19T12:53:53.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4283621670}) 2024-08-19T12:53:53.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86751766}) 2024-08-19T12:53:53.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1364942073}) 2024-08-19T12:53:53.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86751766}) 2024-08-19T12:53:53.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=731742893}) 2024-08-19T12:53:53.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86751766}) 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4084600869}) 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86751766}) 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=155944658}) 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86751766}) 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4059201786}) 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86751766}) 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1074831549}) 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151604646}) 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2293256962}) 2024-08-19T12:53:53.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151604646}) 2024-08-19T12:53:53.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1374930839}) 2024-08-19T12:53:53.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151604646}) 2024-08-19T12:53:53.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3801840002}) 2024-08-19T12:53:53.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151604646}) 2024-08-19T12:53:53.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1803666216}) 2024-08-19T12:53:53.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.278+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.279+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151604646}) 2024-08-19T12:53:53.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.279+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3315346149}) 2024-08-19T12:53:53.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.279+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.279+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151604646}) 2024-08-19T12:53:53.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.279+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3618090678}) 2024-08-19T12:53:53.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.279+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.279+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151604646}) 2024-08-19T12:53:53.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.279+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3577540018}) 2024-08-19T12:53:53.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.279+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.279+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151604646}) 2024-08-19T12:53:53.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.279+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=565754598}) 2024-08-19T12:53:53.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818318423}) 2024-08-19T12:53:53.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1365276902}) 2024-08-19T12:53:53.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818318423}) 2024-08-19T12:53:53.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3108243339}) 2024-08-19T12:53:53.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818318423}) 2024-08-19T12:53:53.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1874407702}) 2024-08-19T12:53:53.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818318423}) 2024-08-19T12:53:53.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3519999378}) 2024-08-19T12:53:53.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818318423}) 2024-08-19T12:53:53.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=733122502}) 2024-08-19T12:53:53.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818318423}) 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3073978208}) 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818318423}) 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2763699375}) 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818318423}) 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2177345389}) 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664769812}) 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=422665379}) 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664769812}) 2024-08-19T12:53:53.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2112023923}) 2024-08-19T12:53:53.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664769812}) 2024-08-19T12:53:53.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3998083904}) 2024-08-19T12:53:53.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664769812}) 2024-08-19T12:53:53.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2280663936}) 2024-08-19T12:53:53.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664769812}) 2024-08-19T12:53:53.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3880074296}) 2024-08-19T12:53:53.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664769812}) 2024-08-19T12:53:53.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2819249311}) 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664769812}) 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4110566723}) 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664769812}) 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4016383644}) 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355855473}) 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=368331635}) 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355855473}) 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1488246414}) 2024-08-19T12:53:53.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355855473}) 2024-08-19T12:53:53.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2187425775}) 2024-08-19T12:53:53.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355855473}) 2024-08-19T12:53:53.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2085940037}) 2024-08-19T12:53:53.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355855473}) 2024-08-19T12:53:53.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2538659751}) 2024-08-19T12:53:53.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355855473}) 2024-08-19T12:53:53.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1947162650}) 2024-08-19T12:53:53.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355855473}) 2024-08-19T12:53:53.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2379449712}) 2024-08-19T12:53:53.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355855473}) 2024-08-19T12:53:53.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4115810528}) 2024-08-19T12:53:53.355 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:53:53.356 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:53:53.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.453+0000 7faee5615640 10 monclient: get_monmap_and_config 2024-08-19T12:53:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.453+0000 7faee5615640 10 monclient: build_initial_monmap 2024-08-19T12:53:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.453+0000 7faee5615640 10 monclient: monmap: 2024-08-19T12:53:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:53.453995+0000 2024-08-19T12:53:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:53.453995+0000 2024-08-19T12:53:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.454 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:53.455 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:53.455 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.453+0000 7faee5615640 1 Processor -- start 2024-08-19T12:53:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.453+0000 7faee5615640 1 -- start start 2024-08-19T12:53:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.453+0000 7faee5615640 10 monclient: init 2024-08-19T12:53:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faee5615640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faee5615640 10 monclient: _add_conns ranks=[4,0,6,8,7,3,2,1,5] 2024-08-19T12:53:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faee5615640 10 monclient(hunting): picked mon.noname-c con 0x7faee006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faee5615640 10 monclient(hunting): picked mon.noname-a con 0x7faee00a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faee5615640 10 monclient(hunting): picked mon.noname-d con 0x7faee00a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:53:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faee5615640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faee0136790 con 0x7faee00a5920 2024-08-19T12:53:53.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faee5615640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faee009ddf0 con 0x7faee006e7b0 2024-08-19T12:53:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faee5615640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faee009e0d0 con 0x7faee00a5d50 2024-08-19T12:53:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faee5615640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faee5615640 10 monclient(hunting): authenticate will time out at 1627.541315s 2024-08-19T12:53:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faedf7fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7faee00a5d50 0x7faee0141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:36082/0 (socket says 172.21.15.26:36082) 2024-08-19T12:53:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faedeffd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7faee006e7b0 0x7faee0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59232/0 (socket says 172.21.15.26:59232) 2024-08-19T12:53:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faedf7fe640 1 -- 172.21.15.26:0/3985222631 learned_addr learned my addr 172.21.15.26:0/3985222631 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.454+0000 7faede7fc640 1 --1- 172.21.15.26:0/3985222631 >> v1:172.21.15.26:6789/0 conn(0x7faee00a5920 0x7faee013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43386/0 (socket says 172.21.15.26:43386) 2024-08-19T12:53:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faede7fc640 1 -- 172.21.15.26:0/3985222631 >> v1:172.21.15.26:6789/0 conn(0x7faee00a5920 legacy=0x7faee013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faedeffd640 1 -- 172.21.15.26:0/3985222631 >> v1:172.21.15.26:6791/0 conn(0x7faee006e7b0 legacy=0x7faee0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faedf7fe640 1 -- 172.21.15.26:0/3985222631 >> v1:172.21.15.26:6792/0 conn(0x7faee00a5d50 legacy=0x7faee0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749796054 0 0) 0x7faee0136790 con 0x7faee00a5920 2024-08-19T12:53:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 10 monclient(hunting): my global_id is 194211 2024-08-19T12:53:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faecc003520 con 0x7faee00a5920 2024-08-19T12:53:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1775387187 0 0) 0x7faed00017d0 con 0x7faee006e7b0 2024-08-19T12:53:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 10 monclient(hunting): my global_id is 184432 2024-08-19T12:53:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faee0136790 con 0x7faee006e7b0 2024-08-19T12:53:53.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3884854171 0 0) 0x7faee009e0d0 con 0x7faee00a5d50 2024-08-19T12:53:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 10 monclient(hunting): my global_id is 194172 2024-08-19T12:53:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.456+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faed00017d0 con 0x7faee00a5d50 2024-08-19T12:53:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1906503360 0 0) 0x7faee0136790 con 0x7faee006e7b0 2024-08-19T12:53:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 >> v1:172.21.15.26:6792/0 conn(0x7faee00a5d50 legacy=0x7faee0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 >> v1:172.21.15.26:6789/0 conn(0x7faee00a5920 legacy=0x7faee013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 10 monclient: _finish_hunting 0 2024-08-19T12:53:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 1 monclient: found mon.noname-c 2024-08-19T12:53:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:53:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee009e3b0 con 0x7faee006e7b0 2024-08-19T12:53:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 10 monclient: _finish_auth 0 2024-08-19T12:53:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 10 monclient: _check_auth_tickets 2024-08-19T12:53:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faed0002010 con 0x7faee006e7b0 2024-08-19T12:53:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faee5615640 5 monclient: authenticate success, global_id 184432 2024-08-19T12:53:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:53:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 10 monclient: dump: 2024-08-19T12:53:53.465 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.466 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.467 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.467 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.467 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.467 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faee5615640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:53.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faed0004be0 con 0x7faee006e7b0 2024-08-19T12:53:53.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:53.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 1 -- 172.21.15.26:0/3985222631 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faed0005150 con 0x7faee006e7b0 2024-08-19T12:53:53.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faee5615640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:53.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faee5615640 4 set_mon_vals no callback set 2024-08-19T12:53:53.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:53.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:53.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faeddffb640 10 monclient: dump: 2024-08-19T12:53:53.468 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.468 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.468 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.468 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.468 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.468 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.468 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.468 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.468 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.469 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.469 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.469 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.469 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.469 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.469 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.469 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faee5615640 10 monclient: shutdown 2024-08-19T12:53:53.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faee5615640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:53.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faee5615640 1 -- 172.21.15.26:0/3985222631 >> v1:172.21.15.26:6791/0 conn(0x7faee006e7b0 legacy=0x7faee0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faee5615640 1 -- 172.21.15.26:0/3985222631 shutdown_connections 2024-08-19T12:53:53.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faee5615640 1 -- 172.21.15.26:0/3985222631 >> 172.21.15.26:0/3985222631 conn(0x7faee00a8740 msgr2=0x7faee00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:53.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faee5615640 1 -- 172.21.15.26:0/3985222631 shutdown_connections 2024-08-19T12:53:53.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.457+0000 7faee5615640 1 -- 172.21.15.26:0/3985222631 wait complete. 2024-08-19T12:53:53.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.458+0000 7faee5615640 10 monclient: build_initial_monmap 2024-08-19T12:53:53.470 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.458+0000 7faee5615640 10 monclient: monmap: 2024-08-19T12:53:53.470 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:53.470 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:53.470 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:53.470 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:53.470 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:53.470 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.470 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:53.471 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:53.471 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:53.471 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:53.471 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:53.471 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:53.471 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:53.471 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:53.471 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:53.471 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.458+0000 7faee5615640 1 librados: starting msgr at 2024-08-19T12:53:53.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.458+0000 7faee5615640 1 librados: starting objecter 2024-08-19T12:53:53.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.458+0000 7faee5615640 1 Processor -- start 2024-08-19T12:53:53.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.458+0000 7faee5615640 1 -- start start 2024-08-19T12:53:53.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.458+0000 7faee5615640 1 librados: setting wanted keys 2024-08-19T12:53:53.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.458+0000 7faee5615640 1 librados: calling monclient init 2024-08-19T12:53:53.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.458+0000 7faee5615640 10 monclient: init 2024-08-19T12:53:53.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.459+0000 7faee5615640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:53.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.459+0000 7faee5615640 10 monclient: _add_conns ranks=[4,7,8,5,3,1,2,6,0] 2024-08-19T12:53:53.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.459+0000 7faee5615640 10 monclient(hunting): picked mon.noname-c con 0x7faee006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:53.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.459+0000 7faee5615640 10 monclient(hunting): picked mon.noname-i con 0x7faee00a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:53.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.459+0000 7faee5615640 10 monclient(hunting): picked mon.noname-e con 0x7faee00a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:53.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.459+0000 7faee5615640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faee0142ec0 con 0x7faee006e7b0 2024-08-19T12:53:53.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.459+0000 7faee5615640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faee0138ff0 con 0x7faee00a5920 2024-08-19T12:53:53.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.459+0000 7faee5615640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faee013b350 con 0x7faee00a5d50 2024-08-19T12:53:53.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.459+0000 7faee5615640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:53.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.459+0000 7faee5615640 10 monclient(hunting): authenticate will time out at 1627.546310s 2024-08-19T12:53:53.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.459+0000 7faedeffd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7faee006e7b0 0x7faee01378e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59234/0 (socket says 172.21.15.26:59234) 2024-08-19T12:53:53.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.459+0000 7faedeffd640 1 -- 172.21.15.26:0/2347783160 learned_addr learned my addr 172.21.15.26:0/2347783160 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:53.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faedeffd640 1 -- 172.21.15.26:0/2347783160 >> v1:172.21.15.26:6791/0 conn(0x7faee006e7b0 legacy=0x7faee01378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 client.?.objecter ms_handle_connect 0x7faee006e7b0 2024-08-19T12:53:53.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faedf7fe640 1 -- 172.21.15.26:0/2347783160 >> v1:172.21.15.26:6793/0 conn(0x7faee00a5d50 legacy=0x7faee01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:53.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 client.?.objecter ms_handle_connect 0x7faee00a5d50 2024-08-19T12:53:53.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:53.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3647370674 0 0) 0x7faee009e3b0 con 0x7faee006e7b0 2024-08-19T12:53:53.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): my global_id is 184441 2024-08-19T12:53:53.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faec0004eb0 con 0x7faee006e7b0 2024-08-19T12:53:53.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 203134761 0 0) 0x7faee013b350 con 0x7faee00a5d50 2024-08-19T12:53:53.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): my global_id is 184337 2024-08-19T12:53:53.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faee009e3b0 con 0x7faee00a5d50 2024-08-19T12:53:53.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 client.?.objecter ms_handle_connect 0x7faee00a5920 2024-08-19T12:53:53.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:53.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faede7fc640 1 -- 172.21.15.26:0/2347783160 >> v1:172.21.15.104:6792/0 conn(0x7faee00a5920 legacy=0x7faee0137ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 544566460 0 0) 0x7faec0004eb0 con 0x7faee006e7b0 2024-08-19T12:53:53.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faee013b350 con 0x7faee006e7b0 2024-08-19T12:53:53.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faed00050d0 con 0x7faee006e7b0 2024-08-19T12:53:53.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:53.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:53:53.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): dump: 2024-08-19T12:53:53.479 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.479 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.479 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.479 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.479 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.479 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.479 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.480 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.480 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.480 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.480 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.480 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.480 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.480 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.480 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.480 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:53.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1102736747 0 0) 0x7faee009e3b0 con 0x7faee00a5d50 2024-08-19T12:53:53.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faec0004eb0 con 0x7faee00a5d50 2024-08-19T12:53:53.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faed80017f0 con 0x7faee00a5d50 2024-08-19T12:53:53.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.461+0000 7faed77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:53.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:53.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 10 monclient(hunting): dump: 2024-08-19T12:53:53.481 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.481 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.481 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.481 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.481 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.481 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.481 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.482 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.482 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.482 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.482 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.482 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.482 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.482 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.482 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.482 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 904575927 0 0) 0x7faee013b350 con 0x7faee006e7b0 2024-08-19T12:53:53.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 >> v1:172.21.15.26:6793/0 conn(0x7faee00a5d50 legacy=0x7faee01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 >> v1:172.21.15.104:6792/0 conn(0x7faee00a5920 legacy=0x7faee0137ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 10 monclient: _finish_hunting 0 2024-08-19T12:53:53.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 1 monclient: found mon.c 2024-08-19T12:53:53.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:53.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:53.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee013aab0 con 0x7faee006e7b0 2024-08-19T12:53:53.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 10 monclient: _finish_auth 0 2024-08-19T12:53:53.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 10 monclient: _check_auth_tickets 2024-08-19T12:53:53.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:53.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faee5615640 5 monclient: authenticate success, global_id 184441 2024-08-19T12:53:53.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faee5615640 10 monclient: _renew_subs 2024-08-19T12:53:53.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faee5615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:53.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faee5615640 1 -- 172.21.15.26:0/2347783160 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faee013a780 con 0x7faee006e7b0 2024-08-19T12:53:53.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faee5615640 10 client.184441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:53.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faee5615640 10 monclient: _renew_subs 2024-08-19T12:53:53.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faee5615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:53.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faee5615640 1 -- 172.21.15.26:0/2347783160 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7faee0138700 con 0x7faee006e7b0 2024-08-19T12:53:53.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faee5615640 1 librados: init done 2024-08-19T12:53:53.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faed0002030 con 0x7faee006e7b0 2024-08-19T12:53:53.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:53.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7faed0005a80 con 0x7faee006e7b0 2024-08-19T12:53:53.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:53.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faedd7fa640 4 set_mon_vals no callback set 2024-08-19T12:53:53.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:53.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.462+0000 7faed77fe640 10 monclient: dump: 2024-08-19T12:53:53.485 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.485 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.485 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.485 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.485 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.485 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.485 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.485 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.485 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.485 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.486 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.486 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.486 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.486 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.486 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.486 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.463+0000 7faee5615640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:53.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.463+0000 7faee5615640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:53.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.463+0000 7faee5615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:53.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.463+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7faed0011ee0 con 0x7faee006e7b0 2024-08-19T12:53:53.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.463+0000 7faee5615640 1 -- 172.21.15.26:0/2347783160 --> v1:172.21.15.26:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faec4005080 con 0x7faee006e7b0 2024-08-19T12:53:53.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.463+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(570..570 src has 350..570) ==== 5857+0+0 (unknown 2949121101 0 0) 0x7faed004e7e0 con 0x7faee006e7b0 2024-08-19T12:53:53.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.463+0000 7faed77fe640 10 client.184441.objecter ms_dispatch 0x7faee0000b90 osd_map(570..570 src has 350..570) 2024-08-19T12:53:53.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.463+0000 7faed77fe640 3 client.184441.objecter handle_osd_map got epochs [570,570] > 0 2024-08-19T12:53:53.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.463+0000 7faed77fe640 3 client.184441.objecter handle_osd_map decoding full epoch 570 2024-08-19T12:53:53.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.463+0000 7faed77fe640 20 client.184441.objecter dump_active .. 0 homeless 2024-08-19T12:53:53.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.463+0000 7faede7fc640 10 monclient: get_auth_request con 0x7faec0041390 auth_method 0 2024-08-19T12:53:53.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.466+0000 7faed77fe640 10 client.184441.objecter ms_handle_connect 0x7faec0041390 2024-08-19T12:53:53.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.468+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.4 v1:172.21.15.26:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7faee0142ec0 con 0x7faee006e7b0 2024-08-19T12:53:53.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.468+0000 7faed77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:53.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.468+0000 7faed77fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:53.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.722+0000 7faee5615640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:53:53.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.722+0000 7faee5615640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:53:53.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.722+0000 7faee5615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:53.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.722+0000 7faee5615640 1 -- 172.21.15.26:0/2347783160 --> v1:172.21.15.26:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faec4005370 con 0x7faee006e7b0 2024-08-19T12:53:53.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.723+0000 7faed77fe640 1 -- 172.21.15.26:0/2347783160 <== mon.4 v1:172.21.15.26:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2730 (unknown 254746657 0 93964516) 0x7faee009ddf0 con 0x7faee006e7b0 2024-08-19T12:53:53.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.723+0000 7faed77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:53:53.724 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.723+0000 7faed77fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:53.724 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":110,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:53:53.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.724+0000 7faee5615640 10 librados: watch_flush enter 2024-08-19T12:53:53.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.725+0000 7faee5615640 10 librados: watch_flush exit 2024-08-19T12:53:53.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.725+0000 7faee5615640 20 client.184441.objecter shutdown clearing up homeless session... 2024-08-19T12:53:53.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.725+0000 7faee5615640 10 client.184441.objecter successfully canceled tick 2024-08-19T12:53:53.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.725+0000 7faee5615640 1 -- 172.21.15.26:0/2347783160 >> v1:172.21.15.104:6800/899887506 conn(0x7faec0041390 legacy=0x7faec0043850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.725+0000 7faee5615640 10 monclient: shutdown 2024-08-19T12:53:53.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.725+0000 7faee5615640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:53.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.725+0000 7faee5615640 1 -- 172.21.15.26:0/2347783160 >> v1:172.21.15.26:6791/0 conn(0x7faee006e7b0 legacy=0x7faee01378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.725+0000 7faee5615640 1 -- 172.21.15.26:0/2347783160 shutdown_connections 2024-08-19T12:53:53.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.725+0000 7faee5615640 1 -- 172.21.15.26:0/2347783160 >> 172.21.15.26:0/2347783160 conn(0x7faee00a8740 msgr2=0x7faee00aa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:53.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.725+0000 7faee5615640 1 -- 172.21.15.26:0/2347783160 shutdown_connections 2024-08-19T12:53:53.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.725+0000 7faee5615640 1 -- 172.21.15.26:0/2347783160 wait complete. 2024-08-19T12:53:53.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.725+0000 7faee5615640 1 librados: shutdown 2024-08-19T12:53:53.737 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:53:53.737 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T12:53:53.737 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:53:53.737 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:53:53.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.836+0000 7ff66eabc640 10 monclient: get_monmap_and_config 2024-08-19T12:53:53.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.836+0000 7ff66eabc640 10 monclient: build_initial_monmap 2024-08-19T12:53:53.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.836+0000 7ff66eabc640 10 monclient: monmap: 2024-08-19T12:53:53.837 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:53.837 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.837 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:53.837378+0000 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:53.837378+0000 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:53.838 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 1 Processor -- start 2024-08-19T12:53:53.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 1 -- start start 2024-08-19T12:53:53.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 10 monclient: init 2024-08-19T12:53:53.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:53.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 10 monclient: _add_conns ranks=[1,6,4,3,2,7,5,0,8] 2024-08-19T12:53:53.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 10 monclient(hunting): picked mon.noname-f con 0x7ff668073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:53.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 10 monclient(hunting): picked mon.noname-d con 0x7ff6680a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:53:53.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 10 monclient(hunting): picked mon.noname-c con 0x7ff66813f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:53.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff66809dae0 con 0x7ff668073470 2024-08-19T12:53:53.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff668136910 con 0x7ff66813f2e0 2024-08-19T12:53:53.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff66809dd50 con 0x7ff6680a57a0 2024-08-19T12:53:53.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:53.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff66eabc640 10 monclient(hunting): authenticate will time out at 1627.924822s 2024-08-19T12:53:53.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff667fff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7ff6680a57a0 0x7ff6680a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:36098/0 (socket says 172.21.15.26:36098) 2024-08-19T12:53:53.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.837+0000 7ff667fff640 1 -- 172.21.15.26:0/2288218579 learned_addr learned my addr 172.21.15.26:0/2288218579 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:53.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff66d059640 1 -- 172.21.15.26:0/2288218579 >> v1:172.21.15.26:6791/0 conn(0x7ff66813f2e0 legacy=0x7ff6681416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff667fff640 1 -- 172.21.15.26:0/2288218579 >> v1:172.21.15.26:6792/0 conn(0x7ff6680a57a0 legacy=0x7ff6680a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff66c858640 1 -- 172.21.15.26:0/2288218579 >> v1:172.21.15.104:6789/0 conn(0x7ff668073470 legacy=0x7ff668136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2795642702 0 0) 0x7ff668136910 con 0x7ff66813f2e0 2024-08-19T12:53:53.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient(hunting): my global_id is 184450 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff658003520 con 0x7ff66813f2e0 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 204633901 0 0) 0x7ff66809dd50 con 0x7ff6680a57a0 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient(hunting): my global_id is 194181 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff668136910 con 0x7ff6680a57a0 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 219438647 0 0) 0x7ff66809dae0 con 0x7ff668073470 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient(hunting): my global_id is 184411 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff66809dd50 con 0x7ff668073470 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1246423846 0 0) 0x7ff658003520 con 0x7ff66813f2e0 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 >> v1:172.21.15.26:6792/0 conn(0x7ff6680a57a0 legacy=0x7ff6680a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 >> v1:172.21.15.104:6789/0 conn(0x7ff668073470 legacy=0x7ff668136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient: _finish_hunting 0 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 monclient: found mon.noname-c 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66809e030 con 0x7ff66813f2e0 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient: _finish_auth 0 2024-08-19T12:53:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient: _check_auth_tickets 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff66eabc640 5 monclient: authenticate success, global_id 184450 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff660001eb0 con 0x7ff66813f2e0 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient: dump: 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.854 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff66eabc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff660003990 con 0x7ff66813f2e0 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:53.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 1 -- 172.21.15.26:0/2288218579 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff660003f00 con 0x7ff66813f2e0 2024-08-19T12:53:53.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff66eabc640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:53.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff66eabc640 4 set_mon_vals no callback set 2024-08-19T12:53:53.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:53.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:53.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.840+0000 7ff6677fe640 10 monclient: dump: 2024-08-19T12:53:53.856 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.856 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.856 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.856 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.856 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.856 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.856 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.857 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.857 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.857 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.857 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.857 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.857 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.857 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.857 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.857 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.841+0000 7ff66eabc640 10 monclient: shutdown 2024-08-19T12:53:53.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.841+0000 7ff66eabc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:53.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.841+0000 7ff66eabc640 1 -- 172.21.15.26:0/2288218579 >> v1:172.21.15.26:6791/0 conn(0x7ff66813f2e0 legacy=0x7ff6681416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.841+0000 7ff66eabc640 1 -- 172.21.15.26:0/2288218579 shutdown_connections 2024-08-19T12:53:53.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.841+0000 7ff66eabc640 1 -- 172.21.15.26:0/2288218579 >> 172.21.15.26:0/2288218579 conn(0x7ff6680abff0 msgr2=0x7ff6680ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:53.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.841+0000 7ff66eabc640 1 -- 172.21.15.26:0/2288218579 shutdown_connections 2024-08-19T12:53:53.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.841+0000 7ff66eabc640 1 -- 172.21.15.26:0/2288218579 wait complete. 2024-08-19T12:53:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.841+0000 7ff66eabc640 10 monclient: build_initial_monmap 2024-08-19T12:53:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.841+0000 7ff66eabc640 10 monclient: monmap: 2024-08-19T12:53:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.859 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:53.860 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:53.860 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:53.860 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:53.860 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:53.860 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:53.860 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:53.860 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:53.860 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:53.860 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 1 librados: starting msgr at 2024-08-19T12:53:53.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 1 librados: starting objecter 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 1 Processor -- start 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 1 -- start start 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 1 librados: setting wanted keys 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 1 librados: calling monclient init 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 10 monclient: init 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 10 monclient: _add_conns ranks=[8,2,0,7,6,4,5,1,3] 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 10 monclient(hunting): picked mon.noname-e con 0x7ff668073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 10 monclient(hunting): picked mon.noname-b con 0x7ff6680a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 10 monclient(hunting): picked mon.noname-a con 0x7ff66813f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6680a3420 con 0x7ff66813f2e0 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6680a3970 con 0x7ff6680a57a0 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff66813f070 con 0x7ff668073470 2024-08-19T12:53:53.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:53.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.842+0000 7ff66eabc640 10 monclient(hunting): authenticate will time out at 1627.929999s 2024-08-19T12:53:53.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.843+0000 7ff66c858640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7ff668073470 0x7ff6681ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40704/0 (socket says 172.21.15.26:40704) 2024-08-19T12:53:53.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.843+0000 7ff667fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7ff6680a57a0 0x7ff6681ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47878/0 (socket says 172.21.15.26:47878) 2024-08-19T12:53:53.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.843+0000 7ff66c858640 1 -- 172.21.15.26:0/2275219373 learned_addr learned my addr 172.21.15.26:0/2275219373 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:53.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff66c858640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.26:6793/0 conn(0x7ff668073470 legacy=0x7ff6681ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 client.?.objecter ms_handle_connect 0x7ff668073470 2024-08-19T12:53:53.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:53.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff667fff640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.26:6790/0 conn(0x7ff6680a57a0 legacy=0x7ff6681ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 client.?.objecter ms_handle_connect 0x7ff6680a57a0 2024-08-19T12:53:53.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:53.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 client.?.objecter ms_handle_connect 0x7ff66813f2e0 2024-08-19T12:53:53.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff66d059640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.26:6789/0 conn(0x7ff66813f2e0 legacy=0x7ff6681d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2874211361 0 0) 0x7ff6680a3420 con 0x7ff66813f2e0 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): my global_id is 194220 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff64c004eb0 con 0x7ff66813f2e0 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2338908411 0 0) 0x7ff6680a3970 con 0x7ff6680a57a0 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): my global_id is 27164 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6680a3420 con 0x7ff6680a57a0 2024-08-19T12:53:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2785694312 0 0) 0x7ff66809dd50 con 0x7ff668073470 2024-08-19T12:53:53.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): my global_id is 184346 2024-08-19T12:53:53.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6680a3970 con 0x7ff668073470 2024-08-19T12:53:53.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1502838650 0 0) 0x7ff64c004eb0 con 0x7ff66813f2e0 2024-08-19T12:53:53.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff66809dd50 con 0x7ff66813f2e0 2024-08-19T12:53:53.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff660003e80 con 0x7ff66813f2e0 2024-08-19T12:53:53.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:53.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:53:53.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): dump: 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.865 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.866 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.866 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:53.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3030865244 0 0) 0x7ff6680a3420 con 0x7ff6680a57a0 2024-08-19T12:53:53.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff64c004eb0 con 0x7ff6680a57a0 2024-08-19T12:53:53.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 948105666 0 0) 0x7ff6680a3970 con 0x7ff668073470 2024-08-19T12:53:53.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6680a3420 con 0x7ff668073470 2024-08-19T12:53:53.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff6540047e0 con 0x7ff6680a57a0 2024-08-19T12:53:53.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:53.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:53.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): dump: 2024-08-19T12:53:53.866 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.867 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.867 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.867 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.867 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.867 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.867 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.867 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.867 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.867 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff65c003610 con 0x7ff668073470 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient(hunting): dump: 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.868 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 70073984 0 0) 0x7ff66809dd50 con 0x7ff66813f2e0 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.26:6793/0 conn(0x7ff668073470 legacy=0x7ff6681ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.26:6790/0 conn(0x7ff6680a57a0 legacy=0x7ff6681ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient: _finish_hunting 0 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 monclient: found mon.a 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.845+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6681cef20 con 0x7ff66813f2e0 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff6657fa640 10 monclient: _finish_auth 0 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff6657fa640 10 monclient: _check_auth_tickets 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff6657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff66eabc640 5 monclient: authenticate success, global_id 194220 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff66eabc640 10 monclient: _renew_subs 2024-08-19T12:53:53.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff66eabc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff66eabc640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff668135230 con 0x7ff66813f2e0 2024-08-19T12:53:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff66eabc640 10 client.194220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff66eabc640 10 monclient: _renew_subs 2024-08-19T12:53:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff66eabc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff66eabc640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6680a4770 con 0x7ff66813f2e0 2024-08-19T12:53:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff66eabc640 1 librados: init done 2024-08-19T12:53:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff660003990 con 0x7ff66813f2e0 2024-08-19T12:53:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff6657fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff660004610 con 0x7ff66813f2e0 2024-08-19T12:53:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff6657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:53.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff6657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff666ffd640 4 set_mon_vals no callback set 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff6657fa640 10 monclient: dump: 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.872 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.846+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff660011e00 con 0x7ff66813f2e0 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.847+0000 7ff66eabc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.847+0000 7ff66eabc640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff6480065c0 con 0x7ff648002860 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.847+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(570..570 src has 350..570) ==== 5857+0+0 (unknown 2949121101 0 0) 0x7ff66004e490 con 0x7ff66813f2e0 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.847+0000 7ff6657fa640 10 client.194220.objecter ms_dispatch 0x7ff668000b90 osd_map(570..570 src has 350..570) 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.847+0000 7ff6657fa640 3 client.194220.objecter handle_osd_map got epochs [570,570] > 0 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.847+0000 7ff6657fa640 3 client.194220.objecter handle_osd_map decoding full epoch 570 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.847+0000 7ff6657fa640 20 client.194220.objecter dump_active .. 0 homeless 2024-08-19T12:53:53.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.847+0000 7ff667fff640 10 monclient: get_auth_request con 0x7ff64c041980 auth_method 0 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff66c858640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.26:6789/0 conn(0x7ff648002860 legacy=0x7ff648004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 10 client.194220.objecter ms_handle_connect 0x7ff648002860 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 10 client.194220.objecter resend_mon_ops 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2623487078 0 0) 0x7ff6480065c0 con 0x7ff648002860 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 10 monclient(hunting): my global_id is 194229 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff66809dd50 con 0x7ff648002860 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2417143795 0 0) 0x7ff66809dd50 con 0x7ff648002860 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6480065c0 con 0x7ff648002860 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 10 client.194220.objecter ms_handle_connect 0x7ff64c041980 2024-08-19T12:53:53.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff65c003d20 con 0x7ff648002860 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 10 monclient: dump: 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.875 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.876 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.876 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.876 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.876 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.876 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2515820701 0 0) 0x7ff6480065c0 con 0x7ff648002860 2024-08-19T12:53:53.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.849+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff64800a160 con 0x7ff648002860 2024-08-19T12:53:53.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.850+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff65c004060 con 0x7ff648002860 2024-08-19T12:53:53.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.850+0000 7ff6657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:53.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.850+0000 7ff6657fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:53.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.850+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.26:6789/0 conn(0x7ff648002860 legacy=0x7ff648004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.850+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.26:6789/0 conn(0x7ff648002860 legacy=0x7ff648004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:53.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.856+0000 7ff66eabc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:53.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.856+0000 7ff66eabc640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff648009e50 con 0x7ff648009110 2024-08-19T12:53:53.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.858+0000 7ff66c858640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.26:6789/0 conn(0x7ff648009110 legacy=0x7ff6480095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:53.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.858+0000 7ff6657fa640 10 client.194220.objecter ms_handle_connect 0x7ff648009110 2024-08-19T12:53:53.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.858+0000 7ff6657fa640 10 client.194220.objecter resend_mon_ops 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.858+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4143216553 0 0) 0x7ff648009e50 con 0x7ff648009110 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.858+0000 7ff6657fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.858+0000 7ff6657fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.858+0000 7ff6657fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.858+0000 7ff6657fa640 10 monclient(hunting): my global_id is 194238 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.858+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6480065c0 con 0x7ff648009110 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4248808278 0 0) 0x7ff6480065c0 con 0x7ff648009110 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff648009e50 con 0x7ff648009110 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff65c005b00 con 0x7ff648009110 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 10 monclient: dump: 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:53.879 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:53.880 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.880 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:53.880 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:53.880 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:53.880 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:53.880 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:53.880 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:53.880 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:53.880 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:53.880 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:53.880 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:53.880 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:53.881 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:53.881 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:53.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4111128312 0 0) 0x7ff648009e50 con 0x7ff648009110 2024-08-19T12:53:53.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff64800c3f0 con 0x7ff648009110 2024-08-19T12:53:53.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7546 (unknown 0 0 1278195705) 0x7ff65c007d90 con 0x7ff648009110 2024-08-19T12:53:53.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:53.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:53.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.26:6789/0 conn(0x7ff648009110 legacy=0x7ff6480095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff6657fa640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.26:6789/0 conn(0x7ff648009110 legacy=0x7ff6480095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:53.881 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:53.881 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:53.882 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:53.882 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:53:53.882 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:53.882 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 18781, 2024-08-19T12:53:53.882 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:53.882 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:53.882 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:53.882 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:53.882 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:53.882 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:53.882 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:53.882 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:53.883 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:53.883 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:53.883 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:53.883 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:53:53.883 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:53.883 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:53.883 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:53.883 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:53.883 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:53.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:53.883 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:53.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:53.884 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:53.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:53.885 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:53.885 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:53.885 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:53.885 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:53.885 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:53.885 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:53.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:53.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.885 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:53.885 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:53.885 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:53.886 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:53.886 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:53.886 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:53.886 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:53.886 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:53.886 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:53.886 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:53.886 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:53.886 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:53.886 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:53.886 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:53.887 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:53.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:53.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:53.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:53.889 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:53.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:53.889 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:53.889 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:53.889 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:53.889 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:53.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.890 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:53.890 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:53.890 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:53.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:53.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:53.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:53.891 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:53.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:53.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:53.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:53.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:53.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:53.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:53.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:53.892 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:53.892 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:53.893 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:53.893 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:53.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.893 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:53.893 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:53.893 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:53.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:53.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.894 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:53.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:53.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:53.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:53.897 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:53.898 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:53.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.900 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:53.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:53.900 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:53.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:53.900 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:53.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:53.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:53.900 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:53.900 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:53.901 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:53.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.901 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:53.901 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:53.901 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:53.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:53.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:53.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:53.902 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:53.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:53.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:53.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:53.902 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:53.902 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:53.903 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:53.903 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:53.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:53.903 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:53.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.903 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:53.903 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:53.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.903 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:53.904 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:53.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff66eabc640 10 librados: watch_flush enter 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff66eabc640 10 librados: watch_flush exit 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff66eabc640 20 client.194220.objecter shutdown clearing up homeless session... 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff66eabc640 10 client.194220.objecter successfully canceled tick 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff66eabc640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.104:6800/899887506 conn(0x7ff64c041980 legacy=0x7ff64c043e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff66eabc640 10 monclient: shutdown 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff66eabc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff66eabc640 1 -- 172.21.15.26:0/2275219373 >> v1:172.21.15.26:6789/0 conn(0x7ff66813f2e0 legacy=0x7ff6681d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff66d059640 1 -- 172.21.15.26:0/2275219373 reap_dead start 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.859+0000 7ff66d059640 1 -- 172.21.15.26:0/2275219373 reap_dead start 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.860+0000 7ff66eabc640 1 -- 172.21.15.26:0/2275219373 shutdown_connections 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.860+0000 7ff66eabc640 1 -- 172.21.15.26:0/2275219373 >> 172.21.15.26:0/2275219373 conn(0x7ff6680abff0 msgr2=0x7ff668134870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.860+0000 7ff66eabc640 1 -- 172.21.15.26:0/2275219373 shutdown_connections 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.860+0000 7ff66eabc640 1 -- 172.21.15.26:0/2275219373 wait complete. 2024-08-19T12:53:53.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:53.860+0000 7ff66eabc640 1 librados: shutdown 2024-08-19T12:53:53.908 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:53.908 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:53:53.908 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:53:53.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646584844}) 2024-08-19T12:53:53.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1923809402}) 2024-08-19T12:53:53.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646584844}) 2024-08-19T12:53:53.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2726291135}) 2024-08-19T12:53:53.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.944 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646584844}) 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1407337008}) 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646584844}) 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=39634980}) 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646584844}) 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2431958938}) 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646584844}) 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2082832673}) 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.941+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.942+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646584844}) 2024-08-19T12:53:53.945 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.942+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3882218933}) 2024-08-19T12:53:53.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.942+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.942+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646584844}) 2024-08-19T12:53:53.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.942+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3714911428}) 2024-08-19T12:53:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46603224}) 2024-08-19T12:53:53.989 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=751173614}) 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46603224}) 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753480474}) 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46603224}) 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2334774385}) 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46603224}) 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1110200777}) 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46603224}) 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2602134585}) 2024-08-19T12:53:53.990 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46603224}) 2024-08-19T12:53:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3900732329}) 2024-08-19T12:53:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46603224}) 2024-08-19T12:53:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2512716349}) 2024-08-19T12:53:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46603224}) 2024-08-19T12:53:53.991 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:53.987+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1109391301}) 2024-08-19T12:53:54.013 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455504570}) 2024-08-19T12:53:54.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2699341840}) 2024-08-19T12:53:54.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455504570}) 2024-08-19T12:53:54.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1659149502}) 2024-08-19T12:53:54.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455504570}) 2024-08-19T12:53:54.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1269959202}) 2024-08-19T12:53:54.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455504570}) 2024-08-19T12:53:54.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1046566987}) 2024-08-19T12:53:54.014 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455504570}) 2024-08-19T12:53:54.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3510809885}) 2024-08-19T12:53:54.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455504570}) 2024-08-19T12:53:54.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1780985379}) 2024-08-19T12:53:54.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455504570}) 2024-08-19T12:53:54.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=268542757}) 2024-08-19T12:53:54.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455504570}) 2024-08-19T12:53:54.015 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.012+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=513172352}) 2024-08-19T12:53:54.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761241211}) 2024-08-19T12:53:54.018 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2189166231}) 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761241211}) 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1827014857}) 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761241211}) 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1853455369}) 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761241211}) 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3004561418}) 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761241211}) 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3230010754}) 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.019 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761241211}) 2024-08-19T12:53:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2618840147}) 2024-08-19T12:53:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761241211}) 2024-08-19T12:53:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2607089364}) 2024-08-19T12:53:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761241211}) 2024-08-19T12:53:54.020 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.016+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3694393885}) 2024-08-19T12:53:54.024 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867264468}) 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2449985531}) 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867264468}) 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1383556368}) 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867264468}) 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2487189988}) 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867264468}) 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3102403997}) 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.025 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867264468}) 2024-08-19T12:53:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=365115624}) 2024-08-19T12:53:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867264468}) 2024-08-19T12:53:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1901630903}) 2024-08-19T12:53:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867264468}) 2024-08-19T12:53:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=526732596}) 2024-08-19T12:53:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867264468}) 2024-08-19T12:53:54.026 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.022+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3815532696}) 2024-08-19T12:53:54.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.040+0000 7ff4fba9f640 10 monclient: get_monmap_and_config 2024-08-19T12:53:54.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.040+0000 7ff4fba9f640 10 monclient: build_initial_monmap 2024-08-19T12:53:54.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.040+0000 7ff4fba9f640 10 monclient: monmap: 2024-08-19T12:53:54.041 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:54.041 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.041 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:54.040943+0000 2024-08-19T12:53:54.041 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:54.040943+0000 2024-08-19T12:53:54.041 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:54.041 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.040+0000 7ff4fba9f640 1 Processor -- start 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.040+0000 7ff4fba9f640 1 -- start start 2024-08-19T12:53:54.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.040+0000 7ff4fba9f640 10 monclient: init 2024-08-19T12:53:54.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.041+0000 7ff4fba9f640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:54.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.041+0000 7ff4fba9f640 10 monclient: _add_conns ranks=[4,1,7,0,6,8,5,3,2] 2024-08-19T12:53:54.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.041+0000 7ff4fba9f640 10 monclient(hunting): picked mon.noname-c con 0x7ff4f4073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:54.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.041+0000 7ff4fba9f640 10 monclient(hunting): picked mon.noname-f con 0x7ff4f40a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:54.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.041+0000 7ff4fba9f640 10 monclient(hunting): picked mon.noname-i con 0x7ff4f413f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:54.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.041+0000 7ff4fba9f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4f409dae0 con 0x7ff4f40a57a0 2024-08-19T12:53:54.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.041+0000 7ff4fba9f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4f4136910 con 0x7ff4f4073470 2024-08-19T12:53:54.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.041+0000 7ff4fba9f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4f409dd50 con 0x7ff4f413f2e0 2024-08-19T12:53:54.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.041+0000 7ff4fba9f640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:54.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.041+0000 7ff4fba9f640 10 monclient(hunting): authenticate will time out at 1628.128378s 2024-08-19T12:53:54.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.041+0000 7ff4f983b640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7ff4f4073470 0x7ff4f4136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59258/0 (socket says 172.21.15.26:59258) 2024-08-19T12:53:54.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.041+0000 7ff4f983b640 1 -- 172.21.15.26:0/3264584692 learned_addr learned my addr 172.21.15.26:0/3264584692 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:54.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.043+0000 7ff4f983b640 1 -- 172.21.15.26:0/3264584692 >> v1:172.21.15.26:6791/0 conn(0x7ff4f4073470 legacy=0x7ff4f4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.043+0000 7ff4f8839640 1 -- 172.21.15.26:0/3264584692 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2809295381 0 0) 0x7ff4f4136910 con 0x7ff4f4073470 2024-08-19T12:53:54.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.043+0000 7ff4f8839640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.043+0000 7ff4f8839640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.043+0000 7ff4f8839640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.043+0000 7ff4f8839640 10 monclient(hunting): my global_id is 184459 2024-08-19T12:53:54.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.043+0000 7ff4f8839640 1 -- 172.21.15.26:0/3264584692 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4e0003520 con 0x7ff4f4073470 2024-08-19T12:53:54.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.043+0000 7ff4fa03c640 1 -- 172.21.15.26:0/3264584692 >> v1:172.21.15.104:6792/0 conn(0x7ff4f413f2e0 legacy=0x7ff4f41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.043+0000 7ff4f903a640 1 -- 172.21.15.26:0/3264584692 >> v1:172.21.15.104:6789/0 conn(0x7ff4f40a57a0 legacy=0x7ff4f40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 1 -- 172.21.15.26:0/3264584692 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2092721582 0 0) 0x7ff4e0003520 con 0x7ff4f4073470 2024-08-19T12:53:54.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 1 -- 172.21.15.26:0/3264584692 >> v1:172.21.15.104:6792/0 conn(0x7ff4f413f2e0 legacy=0x7ff4f41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 1 -- 172.21.15.26:0/3264584692 >> v1:172.21.15.104:6789/0 conn(0x7ff4f40a57a0 legacy=0x7ff4f40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 10 monclient: _finish_hunting 0 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 1 monclient: found mon.noname-c 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 1 -- 172.21.15.26:0/3264584692 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f409e030 con 0x7ff4f4073470 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 10 monclient: _finish_auth 0 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 10 monclient: _check_auth_tickets 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 1 -- 172.21.15.26:0/3264584692 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4e8001ae0 con 0x7ff4f4073470 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 10 monclient: dump: 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.049 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.050 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4fba9f640 5 monclient: authenticate success, global_id 184459 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4fba9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 1 -- 172.21.15.26:0/3264584692 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff4e80048a0 con 0x7ff4f4073470 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4fba9f640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 1 -- 172.21.15.26:0/3264584692 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4e8004dd0 con 0x7ff4f4073470 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4fba9f640 4 set_mon_vals no callback set 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4f8839640 10 monclient: dump: 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.051 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4fba9f640 10 monclient: shutdown 2024-08-19T12:53:54.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4fba9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4fba9f640 1 -- 172.21.15.26:0/3264584692 >> v1:172.21.15.26:6791/0 conn(0x7ff4f4073470 legacy=0x7ff4f4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4fba9f640 1 -- 172.21.15.26:0/3264584692 shutdown_connections 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4fba9f640 1 -- 172.21.15.26:0/3264584692 >> 172.21.15.26:0/3264584692 conn(0x7ff4f40abff0 msgr2=0x7ff4f40ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4fba9f640 1 -- 172.21.15.26:0/3264584692 shutdown_connections 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.044+0000 7ff4fba9f640 1 -- 172.21.15.26:0/3264584692 wait complete. 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.045+0000 7ff4fba9f640 10 monclient: build_initial_monmap 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.045+0000 7ff4fba9f640 10 monclient: monmap: 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.053 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.045+0000 7ff4fba9f640 1 librados: starting msgr at 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.045+0000 7ff4fba9f640 1 librados: starting objecter 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.045+0000 7ff4fba9f640 1 Processor -- start 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.045+0000 7ff4fba9f640 1 -- start start 2024-08-19T12:53:54.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.045+0000 7ff4fba9f640 1 librados: setting wanted keys 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.045+0000 7ff4fba9f640 1 librados: calling monclient init 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.045+0000 7ff4fba9f640 10 monclient: init 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.046+0000 7ff4fba9f640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.046+0000 7ff4fba9f640 10 monclient: _add_conns ranks=[0,1,2,5,7,6,3,8,4] 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.046+0000 7ff4fba9f640 10 monclient(hunting): picked mon.noname-a con 0x7ff4f4073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.046+0000 7ff4fba9f640 10 monclient(hunting): picked mon.noname-f con 0x7ff4f40a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.046+0000 7ff4fba9f640 10 monclient(hunting): picked mon.noname-b con 0x7ff4f413f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.046+0000 7ff4fba9f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4f40a3a80 con 0x7ff4f4073470 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.046+0000 7ff4fba9f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4f40a45e0 con 0x7ff4f40a57a0 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.046+0000 7ff4fba9f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4f413edb0 con 0x7ff4f413f2e0 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.046+0000 7ff4fba9f640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.046+0000 7ff4fba9f640 10 monclient(hunting): authenticate will time out at 1628.133264s 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.046+0000 7ff4f983b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff4f4073470 0x7ff4f41ce370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43428/0 (socket says 172.21.15.26:43428) 2024-08-19T12:53:54.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.046+0000 7ff4f983b640 1 -- 172.21.15.26:0/1594283397 learned_addr learned my addr 172.21.15.26:0/1594283397 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:54.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4f983b640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.26:6789/0 conn(0x7ff4f4073470 legacy=0x7ff4f41ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 client.?.objecter ms_handle_connect 0x7ff4f4073470 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 client.?.objecter ms_handle_connect 0x7ff4f413f2e0 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4fa03c640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.26:6790/0 conn(0x7ff4f413f2e0 legacy=0x7ff4f41d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 client.?.objecter ms_handle_connect 0x7ff4f40a57a0 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4f903a640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.104:6789/0 conn(0x7ff4f40a57a0 legacy=0x7ff4f41d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1759051776 0 0) 0x7ff4f413edb0 con 0x7ff4f413f2e0 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): my global_id is 27173 2024-08-19T12:53:54.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4d8004ee0 con 0x7ff4f413f2e0 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2677575666 0 0) 0x7ff4f40a3a80 con 0x7ff4f4073470 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): my global_id is 194247 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4f413edb0 con 0x7ff4f4073470 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1859062951 0 0) 0x7ff4f40a45e0 con 0x7ff4f40a57a0 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): my global_id is 184429 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4f40a3a80 con 0x7ff4f40a57a0 2024-08-19T12:53:54.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3627664250 0 0) 0x7ff4f413edb0 con 0x7ff4f4073470 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4f40a45e0 con 0x7ff4f4073470 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4e8004d90 con 0x7ff4f4073470 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): dump: 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.063 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1945381323 0 0) 0x7ff4d8004ee0 con 0x7ff4f413f2e0 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4f413edb0 con 0x7ff4f413f2e0 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.048+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4ec001cb0 con 0x7ff4f413f2e0 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient(hunting): dump: 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.065 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2283285361 0 0) 0x7ff4f40a3a80 con 0x7ff4f40a57a0 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4d8004ee0 con 0x7ff4f40a57a0 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4f0001fc0 con 0x7ff4f40a57a0 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient(hunting): dump: 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.066 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1755421739 0 0) 0x7ff4f40a45e0 con 0x7ff4f4073470 2024-08-19T12:53:54.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.26:6790/0 conn(0x7ff4f413f2e0 legacy=0x7ff4f41d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.104:6789/0 conn(0x7ff4f40a57a0 legacy=0x7ff4f41d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient: _finish_hunting 0 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 1 monclient: found mon.a 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f406d8f0 con 0x7ff4f4073470 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient: _finish_auth 0 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient: _check_auth_tickets 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4fba9f640 5 monclient: authenticate success, global_id 194247 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4fba9f640 10 monclient: _renew_subs 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4fba9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4fba9f640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f4137ee0 con 0x7ff4f4073470 2024-08-19T12:53:54.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4fba9f640 10 client.194247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4fba9f640 10 monclient: _renew_subs 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4fba9f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4fba9f640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff4f40a5180 con 0x7ff4f4073470 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4fba9f640 1 librados: init done 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff4e80048a0 con 0x7ff4f4073470 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4e8001a80 con 0x7ff4f4073470 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e7fff640 4 set_mon_vals no callback set 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.049+0000 7ff4e67fc640 10 monclient: dump: 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.069 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.050+0000 7ff4fba9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.050+0000 7ff4fba9f640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff4dc0065c0 con 0x7ff4dc002860 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.050+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ff4e80119c0 con 0x7ff4f4073470 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.050+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(571..571 src has 350..571) ==== 5857+0+0 (unknown 2999744453 0 0) 0x7ff4e804e430 con 0x7ff4f4073470 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.050+0000 7ff4e67fc640 10 client.194247.objecter ms_dispatch 0x7ff4f4000b90 osd_map(571..571 src has 350..571) 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.050+0000 7ff4e67fc640 3 client.194247.objecter handle_osd_map got epochs [571,571] > 0 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.050+0000 7ff4e67fc640 3 client.194247.objecter handle_osd_map decoding full epoch 571 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.050+0000 7ff4e67fc640 20 client.194247.objecter dump_active .. 0 homeless 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.050+0000 7ff4fa03c640 10 monclient: get_auth_request con 0x7ff4d80418c0 auth_method 0 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4f903a640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.26:6790/0 conn(0x7ff4dc002860 legacy=0x7ff4dc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 10 client.194247.objecter ms_handle_connect 0x7ff4dc002860 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 10 client.194247.objecter resend_mon_ops 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4241971923 0 0) 0x7ff4dc0065c0 con 0x7ff4dc002860 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 10 monclient(hunting): my global_id is 27182 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4f40a45e0 con 0x7ff4dc002860 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1805668410 0 0) 0x7ff4f40a45e0 con 0x7ff4dc002860 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4dc0065c0 con 0x7ff4dc002860 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4f0003f00 con 0x7ff4dc002860 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 10 monclient: dump: 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.072 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 192082632 0 0) 0x7ff4dc0065c0 con 0x7ff4dc002860 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.052+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4dc00a160 con 0x7ff4dc002860 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.053+0000 7ff4e67fc640 10 client.194247.objecter ms_handle_connect 0x7ff4d80418c0 2024-08-19T12:53:54.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.053+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff4f0005b40 con 0x7ff4dc002860 2024-08-19T12:53:54.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.053+0000 7ff4e67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:54.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.053+0000 7ff4e67fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:54.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.053+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.26:6790/0 conn(0x7ff4dc002860 legacy=0x7ff4dc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.053+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.26:6790/0 conn(0x7ff4dc002860 legacy=0x7ff4dc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:54.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.059+0000 7ff4fba9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:54.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.059+0000 7ff4fba9f640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff4dc009e50 con 0x7ff4dc009110 2024-08-19T12:53:54.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.061+0000 7ff4f903a640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.26:6790/0 conn(0x7ff4dc009110 legacy=0x7ff4dc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.061+0000 7ff4e67fc640 10 client.194247.objecter ms_handle_connect 0x7ff4dc009110 2024-08-19T12:53:54.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.061+0000 7ff4e67fc640 10 client.194247.objecter resend_mon_ops 2024-08-19T12:53:54.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.061+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2421221753 0 0) 0x7ff4dc009e50 con 0x7ff4dc009110 2024-08-19T12:53:54.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.061+0000 7ff4e67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.061+0000 7ff4e67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.061+0000 7ff4e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.061+0000 7ff4e67fc640 10 monclient(hunting): my global_id is 27191 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.061+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4dc0065c0 con 0x7ff4dc009110 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1459847826 0 0) 0x7ff4dc0065c0 con 0x7ff4dc009110 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4dc009e50 con 0x7ff4dc009110 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ff4f00047d0 con 0x7ff4dc009110 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 10 monclient: dump: 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.076 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 136270884 0 0) 0x7ff4dc009e50 con 0x7ff4dc009110 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff4dc00c3f0 con 0x7ff4dc009110 2024-08-19T12:53:54.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 1983689533) 0x7ff4f0005f40 con 0x7ff4dc009110 2024-08-19T12:53:54.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:54.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:54.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.26:6790/0 conn(0x7ff4dc009110 legacy=0x7ff4dc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4e67fc640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.26:6790/0 conn(0x7ff4dc009110 legacy=0x7ff4dc0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:54.078 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:54.078 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:54.078 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:54.078 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:54.078 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:54.078 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 947117, 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:53:54.079 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.080 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:54.082 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.083 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:54.084 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:54.087 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.088 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:54.090 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:54.092 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.095 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.096 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.097 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:54.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.097 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.097 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.097 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:53:54.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.097 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:54.097 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 10 librados: watch_flush enter 2024-08-19T12:53:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 10 librados: watch_flush exit 2024-08-19T12:53:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 20 client.194247.objecter shutdown clearing up homeless session... 2024-08-19T12:53:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 10 client.194247.objecter successfully canceled tick 2024-08-19T12:53:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.104:6800/899887506 conn(0x7ff4d80418c0 legacy=0x7ff4d8043d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 10 monclient: shutdown 2024-08-19T12:53:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:54.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 1 -- 172.21.15.26:0/1594283397 >> v1:172.21.15.26:6789/0 conn(0x7ff4f4073470 legacy=0x7ff4f41ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fa03c640 1 -- 172.21.15.26:0/1594283397 reap_dead start 2024-08-19T12:53:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fa03c640 1 -- 172.21.15.26:0/1594283397 reap_dead start 2024-08-19T12:53:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 1 -- 172.21.15.26:0/1594283397 shutdown_connections 2024-08-19T12:53:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 1 -- 172.21.15.26:0/1594283397 >> 172.21.15.26:0/1594283397 conn(0x7ff4f40abff0 msgr2=0x7ff4f4133a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 1 -- 172.21.15.26:0/1594283397 shutdown_connections 2024-08-19T12:53:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 1 -- 172.21.15.26:0/1594283397 wait complete. 2024-08-19T12:53:54.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.062+0000 7ff4fba9f640 1 librados: shutdown 2024-08-19T12:53:54.100 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:54.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:53:54.101 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:53:54.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19292010}) 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703580742}) 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19292010}) 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1106239647}) 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19292010}) 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1176711546}) 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19292010}) 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3884451233}) 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19292010}) 2024-08-19T12:53:54.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=13530697}) 2024-08-19T12:53:54.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19292010}) 2024-08-19T12:53:54.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1450085820}) 2024-08-19T12:53:54.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19292010}) 2024-08-19T12:53:54.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3704781452}) 2024-08-19T12:53:54.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19292010}) 2024-08-19T12:53:54.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.102+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3428333922}) 2024-08-19T12:53:54.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.203+0000 7f4f9a341640 10 monclient: get_monmap_and_config 2024-08-19T12:53:54.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.203+0000 7f4f9a341640 10 monclient: build_initial_monmap 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.203+0000 7f4f9a341640 10 monclient: monmap: 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:54.204797+0000 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:54.204797+0000 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:54.205 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:54.206 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:54.206 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:54.206 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:54.206 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.204+0000 7f4f9a341640 1 Processor -- start 2024-08-19T12:53:54.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.204+0000 7f4f9a341640 1 -- start start 2024-08-19T12:53:54.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.204+0000 7f4f9a341640 10 monclient: init 2024-08-19T12:53:54.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f9a341640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:54.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f9a341640 10 monclient: _add_conns ranks=[3,4,7,5,2,0,6,1,8] 2024-08-19T12:53:54.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f9a341640 10 monclient(hunting): picked mon.noname-g con 0x7f4f9406e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:53:54.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f9a341640 10 monclient(hunting): picked mon.noname-c con 0x7f4f940a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:54.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f9a341640 10 monclient(hunting): picked mon.noname-i con 0x7f4f940a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:54.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f9a341640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f94136790 con 0x7f4f9406e7b0 2024-08-19T12:53:54.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f9a341640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9409ddf0 con 0x7f4f940a5920 2024-08-19T12:53:54.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f9a341640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9409e0d0 con 0x7f4f940a5d50 2024-08-19T12:53:54.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f9a341640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:54.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f9a341640 10 monclient(hunting): authenticate will time out at 1628.292296s 2024-08-19T12:53:54.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f937fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f4f940a5920 0x7f4f9413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59264/0 (socket says 172.21.15.26:59264) 2024-08-19T12:53:54.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f93fff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f4f9406e7b0 0x7f4f94136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:52838/0 (socket says 172.21.15.26:52838) 2024-08-19T12:53:54.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.205+0000 7f4f937fe640 1 -- 172.21.15.26:0/1233071915 learned_addr learned my addr 172.21.15.26:0/1233071915 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:54.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f937fe640 1 -- 172.21.15.26:0/1233071915 >> v1:172.21.15.26:6791/0 conn(0x7f4f940a5920 legacy=0x7f4f9413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f988de640 1 -- 172.21.15.26:0/1233071915 >> v1:172.21.15.104:6792/0 conn(0x7f4f940a5d50 legacy=0x7f4f94141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f93fff640 1 -- 172.21.15.26:0/1233071915 >> v1:172.21.15.104:6790/0 conn(0x7f4f9406e7b0 legacy=0x7f4f94136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129634821 0 0) 0x7f4f9409e0d0 con 0x7f4f940a5d50 2024-08-19T12:53:54.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 10 monclient(hunting): my global_id is 136150 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f7c003520 con 0x7f4f940a5d50 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2740657766 0 0) 0x7f4f88001450 con 0x7f4f940a5920 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 10 monclient(hunting): my global_id is 184468 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f9409e0d0 con 0x7f4f940a5920 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3607750376 0 0) 0x7f4f94136790 con 0x7f4f9406e7b0 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.213 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 10 monclient(hunting): my global_id is 194169 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f88001450 con 0x7f4f9406e7b0 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.207+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2916487326 0 0) 0x7f4f7c003520 con 0x7f4f940a5d50 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 >> v1:172.21.15.26:6791/0 conn(0x7f4f940a5920 legacy=0x7f4f9413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 >> v1:172.21.15.104:6790/0 conn(0x7f4f9406e7b0 legacy=0x7f4f94136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 1 monclient: found mon.noname-i 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9409e3b0 con 0x7f4f940a5d50 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 10 monclient: _finish_auth 0 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:53:54.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4f8c001eb0 con 0x7f4f940a5d50 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 10 monclient: dump: 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.215 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f9a341640 5 monclient: authenticate success, global_id 136150 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f9a341640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f8c003990 con 0x7f4f940a5d50 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f9a341640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 1 -- 172.21.15.26:0/1233071915 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4f8c003f00 con 0x7f4f940a5d50 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f9a341640 4 set_mon_vals no callback set 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f92ffd640 10 monclient: dump: 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.217 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f9a341640 10 monclient: shutdown 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f9a341640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f9a341640 1 -- 172.21.15.26:0/1233071915 >> v1:172.21.15.104:6792/0 conn(0x7f4f940a5d50 legacy=0x7f4f94141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f9a341640 1 -- 172.21.15.26:0/1233071915 shutdown_connections 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f9a341640 1 -- 172.21.15.26:0/1233071915 >> 172.21.15.26:0/1233071915 conn(0x7f4f940a8740 msgr2=0x7f4f940a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f9a341640 1 -- 172.21.15.26:0/1233071915 shutdown_connections 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.208+0000 7f4f9a341640 1 -- 172.21.15.26:0/1233071915 wait complete. 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.209+0000 7f4f9a341640 10 monclient: build_initial_monmap 2024-08-19T12:53:54.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.209+0000 7f4f9a341640 10 monclient: monmap: 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:54.219 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.209+0000 7f4f9a341640 1 librados: starting msgr at 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.209+0000 7f4f9a341640 1 librados: starting objecter 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.209+0000 7f4f9a341640 1 Processor -- start 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.209+0000 7f4f9a341640 1 -- start start 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.209+0000 7f4f9a341640 1 librados: setting wanted keys 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.209+0000 7f4f9a341640 1 librados: calling monclient init 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.209+0000 7f4f9a341640 10 monclient: init 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.210+0000 7f4f9a341640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.210+0000 7f4f9a341640 10 monclient: _add_conns ranks=[0,7,8,2,3,4,6,5,1] 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.210+0000 7f4f9a341640 10 monclient(hunting): picked mon.noname-a con 0x7f4f9406e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:54.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.210+0000 7f4f9a341640 10 monclient(hunting): picked mon.noname-i con 0x7f4f940a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:54.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.210+0000 7f4f9a341640 10 monclient(hunting): picked mon.noname-e con 0x7f4f940a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:54.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.210+0000 7f4f9a341640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f940a0af0 con 0x7f4f9406e7b0 2024-08-19T12:53:54.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.210+0000 7f4f9a341640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f940a0810 con 0x7f4f940a5920 2024-08-19T12:53:54.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.210+0000 7f4f9a341640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9409f8a0 con 0x7f4f940a5d50 2024-08-19T12:53:54.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.210+0000 7f4f9a341640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:54.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.210+0000 7f4f9a341640 10 monclient(hunting): authenticate will time out at 1628.297356s 2024-08-19T12:53:54.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.210+0000 7f4f93fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4f9406e7b0 0x7f4f94131ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43444/0 (socket says 172.21.15.26:43444) 2024-08-19T12:53:54.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.210+0000 7f4f93fff640 1 -- 172.21.15.26:0/2243577350 learned_addr learned my addr 172.21.15.26:0/2243577350 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:54.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.212+0000 7f4f93fff640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.26:6789/0 conn(0x7f4f9406e7b0 legacy=0x7f4f94131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.212+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:53:54.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.212+0000 7f4f988de640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.26:6793/0 conn(0x7f4f940a5d50 legacy=0x7f4f94132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.212+0000 7f4f937fe640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.104:6792/0 conn(0x7f4f940a5920 legacy=0x7f4f94132600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 client.?.objecter ms_handle_connect 0x7f4f9406e7b0 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 client.?.objecter ms_handle_connect 0x7f4f940a5d50 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 client.?.objecter ms_handle_connect 0x7f4f940a5920 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2302200425 0 0) 0x7f4f9409f8a0 con 0x7f4f940a5d50 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): my global_id is 184355 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f80004eb0 con 0x7f4f940a5d50 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 106776604 0 0) 0x7f4f940a0af0 con 0x7f4f9406e7b0 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): my global_id is 194256 2024-08-19T12:53:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f9409f8a0 con 0x7f4f9406e7b0 2024-08-19T12:53:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 167967644 0 0) 0x7f4f940a0810 con 0x7f4f940a5920 2024-08-19T12:53:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): my global_id is 136159 2024-08-19T12:53:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f940a0af0 con 0x7f4f940a5920 2024-08-19T12:53:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4040507741 0 0) 0x7f4f80004eb0 con 0x7f4f940a5d50 2024-08-19T12:53:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f940a0810 con 0x7f4f940a5d50 2024-08-19T12:53:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4f8c003e80 con 0x7f4f940a5d50 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): dump: 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.229 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.230 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.230 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.230 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.230 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.230 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.230 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.214+0000 7f4f90ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:54.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 140288086 0 0) 0x7f4f9409f8a0 con 0x7f4f9406e7b0 2024-08-19T12:53:54.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f80004eb0 con 0x7f4f9406e7b0 2024-08-19T12:53:54.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4f84001ce0 con 0x7f4f9406e7b0 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient(hunting): dump: 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.231 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 898968300 0 0) 0x7f4f940a0af0 con 0x7f4f940a5920 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f9409f8a0 con 0x7f4f940a5920 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4f88001a50 con 0x7f4f940a5920 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient(hunting): dump: 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.232 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.233 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3362053724 0 0) 0x7f4f940a0810 con 0x7f4f940a5d50 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.104:6792/0 conn(0x7f4f940a5920 legacy=0x7f4f94132600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.26:6789/0 conn(0x7f4f9406e7b0 legacy=0x7f4f94131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 monclient: found mon.e 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f941da2d0 con 0x7f4f940a5d50 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient: _finish_auth 0 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f9a341640 5 monclient: authenticate success, global_id 184355 2024-08-19T12:53:54.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f9a341640 10 monclient: _renew_subs 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f9a341640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f9a341640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4f941d72a0 con 0x7f4f940a5d50 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f9a341640 10 client.184355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f9a341640 10 monclient: _renew_subs 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f9a341640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f9a341640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f4f941d7710 con 0x7f4f940a5d50 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f9a341640 1 librados: init done 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f8c003990 con 0x7f4f940a5d50 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4f8c001950 con 0x7f4f940a5d50 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f927fc640 4 set_mon_vals no callback set 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:54.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.215+0000 7f4f90ff9640 10 monclient: dump: 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.236 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.216+0000 7f4f9a341640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.216+0000 7f4f9a341640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4f740065c0 con 0x7f4f74002860 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.216+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f4f8c004610 con 0x7f4f940a5d50 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.216+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(571..571 src has 350..571) ==== 5857+0+0 (unknown 2999744453 0 0) 0x7f4f8c04e540 con 0x7f4f940a5d50 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.216+0000 7f4f90ff9640 10 client.184355.objecter ms_dispatch 0x7f4f94000b90 osd_map(571..571 src has 350..571) 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.216+0000 7f4f90ff9640 3 client.184355.objecter handle_osd_map got epochs [571,571] > 0 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.216+0000 7f4f90ff9640 3 client.184355.objecter handle_osd_map decoding full epoch 571 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.216+0000 7f4f90ff9640 20 client.184355.objecter dump_active .. 0 homeless 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.216+0000 7f4f937fe640 10 monclient: get_auth_request con 0x7f4f80041910 auth_method 0 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f93fff640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.26:6791/0 conn(0x7f4f74002860 legacy=0x7f4f74004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 10 client.184355.objecter ms_handle_connect 0x7f4f74002860 2024-08-19T12:53:54.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 10 client.184355.objecter resend_mon_ops 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 340785671 0 0) 0x7f4f740065c0 con 0x7f4f74002860 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 10 monclient(hunting): my global_id is 184477 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f940a0810 con 0x7f4f74002860 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1232758232 0 0) 0x7f4f940a0810 con 0x7f4f74002860 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f740065c0 con 0x7f4f74002860 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4f84005110 con 0x7f4f74002860 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 10 monclient: dump: 2024-08-19T12:53:54.238 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.239 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.240 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 10 client.184355.objecter ms_handle_connect 0x7f4f80041910 2024-08-19T12:53:54.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2514232355 0 0) 0x7f4f740065c0 con 0x7f4f74002860 2024-08-19T12:53:54.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.218+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f7400a160 con 0x7f4f74002860 2024-08-19T12:53:54.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.219+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4f84007370 con 0x7f4f74002860 2024-08-19T12:53:54.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.219+0000 7f4f90ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:54.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.219+0000 7f4f90ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:54.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.219+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.26:6791/0 conn(0x7f4f74002860 legacy=0x7f4f74004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.219+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.26:6791/0 conn(0x7f4f74002860 legacy=0x7f4f74004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:54.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.225+0000 7f4f9a341640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:54.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.225+0000 7f4f9a341640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4f74009e70 con 0x7f4f74009130 2024-08-19T12:53:54.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.227+0000 7f4f93fff640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.26:6791/0 conn(0x7f4f74009130 legacy=0x7f4f740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.227+0000 7f4f90ff9640 10 client.184355.objecter ms_handle_connect 0x7f4f74009130 2024-08-19T12:53:54.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.227+0000 7f4f90ff9640 10 client.184355.objecter resend_mon_ops 2024-08-19T12:53:54.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692816227 0 0) 0x7f4f74009e70 con 0x7f4f74009130 2024-08-19T12:53:54.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 10 monclient(hunting): my global_id is 184486 2024-08-19T12:53:54.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f740065c0 con 0x7f4f74009130 2024-08-19T12:53:54.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1069486072 0 0) 0x7f4f740065c0 con 0x7f4f74009130 2024-08-19T12:53:54.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f74009e70 con 0x7f4f74009130 2024-08-19T12:53:54.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4f84006ca0 con 0x7f4f74009130 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 10 monclient: dump: 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.243 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.244 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 672955347 0 0) 0x7f4f74009e70 con 0x7f4f74009130 2024-08-19T12:53:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4f7400c3f0 con 0x7f4f74009130 2024-08-19T12:53:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 2923413522) 0x7f4f84006f40 con 0x7f4f74009130 2024-08-19T12:53:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.26:6791/0 conn(0x7f4f74009130 legacy=0x7f4f740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f90ff9640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.26:6791/0 conn(0x7f4f74009130 legacy=0x7f4f740095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:54.245 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 159693, 2024-08-19T12:53:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:53:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:53:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:54.265 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:54.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f9a341640 10 librados: watch_flush enter 2024-08-19T12:53:54.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f9a341640 10 librados: watch_flush exit 2024-08-19T12:53:54.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f9a341640 20 client.184355.objecter shutdown clearing up homeless session... 2024-08-19T12:53:54.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.228+0000 7f4f9a341640 10 client.184355.objecter successfully canceled tick 2024-08-19T12:53:54.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.229+0000 7f4f9a341640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.104:6800/899887506 conn(0x7f4f80041910 legacy=0x7f4f80043dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.229+0000 7f4f9a341640 10 monclient: shutdown 2024-08-19T12:53:54.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.229+0000 7f4f9a341640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:54.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.229+0000 7f4f9a341640 1 -- 172.21.15.26:0/2243577350 >> v1:172.21.15.26:6793/0 conn(0x7f4f940a5d50 legacy=0x7f4f94132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.229+0000 7f4f988de640 1 -- 172.21.15.26:0/2243577350 reap_dead start 2024-08-19T12:53:54.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.229+0000 7f4f988de640 1 -- 172.21.15.26:0/2243577350 reap_dead start 2024-08-19T12:53:54.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.229+0000 7f4f9a341640 1 -- 172.21.15.26:0/2243577350 shutdown_connections 2024-08-19T12:53:54.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.229+0000 7f4f9a341640 1 -- 172.21.15.26:0/2243577350 >> 172.21.15.26:0/2243577350 conn(0x7f4f940a8740 msgr2=0x7f4f9406d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:54.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.229+0000 7f4f9a341640 1 -- 172.21.15.26:0/2243577350 shutdown_connections 2024-08-19T12:53:54.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.229+0000 7f4f9a341640 1 -- 172.21.15.26:0/2243577350 wait complete. 2024-08-19T12:53:54.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.229+0000 7f4f9a341640 1 librados: shutdown 2024-08-19T12:53:54.269 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:54.269 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:53:54.269 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:53:54.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017486669}) 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=581253819}) 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017486669}) 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2776439181}) 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017486669}) 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2741036476}) 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017486669}) 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3944860372}) 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017486669}) 2024-08-19T12:53:54.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2910509467}) 2024-08-19T12:53:54.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017486669}) 2024-08-19T12:53:54.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=467346372}) 2024-08-19T12:53:54.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017486669}) 2024-08-19T12:53:54.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2268048735}) 2024-08-19T12:53:54.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017486669}) 2024-08-19T12:53:54.279 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.275+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1714084464}) 2024-08-19T12:53:54.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.281 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2628002274}) 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3882355270}) 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2628002274}) 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=805306439}) 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2628002274}) 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523378167}) 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2628002274}) 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1422319988}) 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2628002274}) 2024-08-19T12:53:54.282 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1580054008}) 2024-08-19T12:53:54.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2628002274}) 2024-08-19T12:53:54.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3969463101}) 2024-08-19T12:53:54.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2628002274}) 2024-08-19T12:53:54.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3202733112}) 2024-08-19T12:53:54.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2628002274}) 2024-08-19T12:53:54.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.277+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2396961770}) 2024-08-19T12:53:54.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661318809}) 2024-08-19T12:53:54.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3874156623}) 2024-08-19T12:53:54.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661318809}) 2024-08-19T12:53:54.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3355918}) 2024-08-19T12:53:54.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661318809}) 2024-08-19T12:53:54.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3395352498}) 2024-08-19T12:53:54.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661318809}) 2024-08-19T12:53:54.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1819528790}) 2024-08-19T12:53:54.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661318809}) 2024-08-19T12:53:54.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1027652848}) 2024-08-19T12:53:54.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661318809}) 2024-08-19T12:53:54.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=333858903}) 2024-08-19T12:53:54.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661318809}) 2024-08-19T12:53:54.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1645619364}) 2024-08-19T12:53:54.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661318809}) 2024-08-19T12:53:54.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3443232573}) 2024-08-19T12:53:54.300 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.300 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733945524}) 2024-08-19T12:53:54.300 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978018055}) 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733945524}) 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1152442663}) 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733945524}) 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1533694879}) 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733945524}) 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4239718688}) 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733945524}) 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1438721971}) 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.301 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733945524}) 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1635649338}) 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733945524}) 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1114792077}) 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733945524}) 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3938735022}) 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109629832}) 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552092157}) 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109629832}) 2024-08-19T12:53:54.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3601723436}) 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109629832}) 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3948319807}) 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109629832}) 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3681900942}) 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109629832}) 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=31066237}) 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109629832}) 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2587582416}) 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109629832}) 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=182445093}) 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109629832}) 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.292+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=447905942}) 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131519126}) 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4201179181}) 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131519126}) 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2365835420}) 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131519126}) 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2650197412}) 2024-08-19T12:53:54.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131519126}) 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=303485511}) 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131519126}) 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2062105718}) 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131519126}) 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2108427674}) 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131519126}) 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1473987982}) 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131519126}) 2024-08-19T12:53:54.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=300546900}) 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291466162}) 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=809791652}) 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291466162}) 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2509970124}) 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291466162}) 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2232840728}) 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291466162}) 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1620939606}) 2024-08-19T12:53:54.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291466162}) 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1136395998}) 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291466162}) 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=851456488}) 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291466162}) 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3660859416}) 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291466162}) 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3129224327}) 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741378678}) 2024-08-19T12:53:54.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2560177645}) 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741378678}) 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2217412503}) 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741378678}) 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1101124550}) 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741378678}) 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1041824347}) 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741378678}) 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1146396769}) 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741378678}) 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3834936002}) 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741378678}) 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2004050974}) 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741378678}) 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2667991865}) 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283957475}) 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4170222996}) 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283957475}) 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1672539087}) 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283957475}) 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2609125844}) 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283957475}) 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3771068763}) 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283957475}) 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=975243471}) 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283957475}) 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=486780772}) 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283957475}) 2024-08-19T12:53:54.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=707163847}) 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2283957475}) 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1841733580}) 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2858350565}) 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146048535}) 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2858350565}) 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=600508904}) 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2858350565}) 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1111385874}) 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2858350565}) 2024-08-19T12:53:54.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=694251514}) 2024-08-19T12:53:54.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2858350565}) 2024-08-19T12:53:54.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3153788308}) 2024-08-19T12:53:54.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2858350565}) 2024-08-19T12:53:54.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3910678284}) 2024-08-19T12:53:54.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2858350565}) 2024-08-19T12:53:54.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2795571578}) 2024-08-19T12:53:54.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2858350565}) 2024-08-19T12:53:54.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=260018401}) 2024-08-19T12:53:54.395 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.395 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614617726}) 2024-08-19T12:53:54.395 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1652436034}) 2024-08-19T12:53:54.395 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.395 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614617726}) 2024-08-19T12:53:54.395 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1502041590}) 2024-08-19T12:53:54.395 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.395 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614617726}) 2024-08-19T12:53:54.395 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=125481595}) 2024-08-19T12:53:54.395 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.396 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614617726}) 2024-08-19T12:53:54.396 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1461604667}) 2024-08-19T12:53:54.396 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.396 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614617726}) 2024-08-19T12:53:54.396 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2405465417}) 2024-08-19T12:53:54.396 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.396 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614617726}) 2024-08-19T12:53:54.396 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3044486384}) 2024-08-19T12:53:54.396 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.396 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614617726}) 2024-08-19T12:53:54.396 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1898926810}) 2024-08-19T12:53:54.396 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614617726}) 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2720954173}) 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776597516}) 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1631068877}) 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776597516}) 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2137191880}) 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776597516}) 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2870208541}) 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776597516}) 2024-08-19T12:53:54.397 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=369567192}) 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776597516}) 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=279850959}) 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.310+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776597516}) 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.310+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4048249365}) 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.310+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.310+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776597516}) 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.310+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4118980289}) 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.310+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.310+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776597516}) 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.310+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1600369184}) 2024-08-19T12:53:54.398 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984360965}) 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2563877595}) 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984360965}) 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4102208335}) 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984360965}) 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=495117892}) 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984360965}) 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3562920106}) 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984360965}) 2024-08-19T12:53:54.399 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1230356534}) 2024-08-19T12:53:54.400 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.400 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984360965}) 2024-08-19T12:53:54.400 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=755245150}) 2024-08-19T12:53:54.400 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.400 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984360965}) 2024-08-19T12:53:54.400 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1477952111}) 2024-08-19T12:53:54.400 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.400 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984360965}) 2024-08-19T12:53:54.400 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1417562002}) 2024-08-19T12:53:54.400 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.400 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995958804}) 2024-08-19T12:53:54.400 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2123341098}) 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995958804}) 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2254436101}) 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995958804}) 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1240630253}) 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995958804}) 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1430877935}) 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995958804}) 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1879612801}) 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.401 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995958804}) 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=328826101}) 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995958804}) 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2956110405}) 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995958804}) 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.312+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3967375212}) 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953819299}) 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1809570218}) 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953819299}) 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3236553099}) 2024-08-19T12:53:54.402 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953819299}) 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4126946161}) 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953819299}) 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=613435426}) 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953819299}) 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1158635692}) 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953819299}) 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1764808688}) 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953819299}) 2024-08-19T12:53:54.403 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=32555280}) 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953819299}) 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.314+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1434913910}) 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206165854}) 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1537754849}) 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206165854}) 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3243282706}) 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206165854}) 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2835436183}) 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.404 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206165854}) 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=604165384}) 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206165854}) 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1573564336}) 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206165854}) 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1811019122}) 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206165854}) 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4143460509}) 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206165854}) 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1195358406}) 2024-08-19T12:53:54.405 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980861508}) 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2429816807}) 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980861508}) 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3708991352}) 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980861508}) 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3554478199}) 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980861508}) 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=649210572}) 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.316+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980861508}) 2024-08-19T12:53:54.406 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1264355481}) 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980861508}) 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2957592249}) 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980861508}) 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1213814235}) 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980861508}) 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3386167176}) 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267211356}) 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1029710605}) 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.407 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267211356}) 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2218203686}) 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267211356}) 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3754164245}) 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267211356}) 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3086175158}) 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267211356}) 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1368160808}) 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267211356}) 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1628226817}) 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.408 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267211356}) 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1178146587}) 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267211356}) 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=747832839}) 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619267192}) 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3718480525}) 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619267192}) 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3263818978}) 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619267192}) 2024-08-19T12:53:54.409 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=612502076}) 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619267192}) 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=716907811}) 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619267192}) 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1555817046}) 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619267192}) 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=460735642}) 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619267192}) 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3499460522}) 2024-08-19T12:53:54.410 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619267192}) 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2329857476}) 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148050870}) 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1100368253}) 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148050870}) 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1334129013}) 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148050870}) 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=501024964}) 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.411 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148050870}) 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2493254388}) 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148050870}) 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3156607199}) 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148050870}) 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1631446943}) 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148050870}) 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2565726591}) 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148050870}) 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=352280954}) 2024-08-19T12:53:54.412 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866390322}) 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2250037547}) 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866390322}) 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1775949378}) 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866390322}) 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2681185063}) 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866390322}) 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3054377241}) 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866390322}) 2024-08-19T12:53:54.413 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1429293758}) 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866390322}) 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3335244611}) 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866390322}) 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3416373065}) 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866390322}) 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1944903430}) 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3183689539}) 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4007633174}) 2024-08-19T12:53:54.414 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3183689539}) 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2712365168}) 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3183689539}) 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4008070633}) 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3183689539}) 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2586630779}) 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3183689539}) 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=488876178}) 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3183689539}) 2024-08-19T12:53:54.415 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=107248798}) 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3183689539}) 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4109250466}) 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3183689539}) 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=491531972}) 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140577614}) 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2119596819}) 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140577614}) 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2299055410}) 2024-08-19T12:53:54.416 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140577614}) 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=709035760}) 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140577614}) 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=568817777}) 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140577614}) 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=858903905}) 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140577614}) 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3580702678}) 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140577614}) 2024-08-19T12:53:54.417 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2045109703}) 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140577614}) 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2905191314}) 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265794271}) 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1675596750}) 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265794271}) 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3554659890}) 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265794271}) 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2594123399}) 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.418 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265794271}) 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2953374908}) 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265794271}) 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=42859339}) 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265794271}) 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3392401185}) 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265794271}) 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2404724920}) 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.328+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265794271}) 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.328+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3706176880}) 2024-08-19T12:53:54.419 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475964870}) 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2133837269}) 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475964870}) 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4163655151}) 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475964870}) 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2036380329}) 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475964870}) 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3601191366}) 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475964870}) 2024-08-19T12:53:54.420 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4044253198}) 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475964870}) 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2140568550}) 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475964870}) 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2085576469}) 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475964870}) 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=206597532}) 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110378365}) 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3772013466}) 2024-08-19T12:53:54.421 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110378365}) 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=29902832}) 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110378365}) 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3227314201}) 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110378365}) 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3193050372}) 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110378365}) 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2938488102}) 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110378365}) 2024-08-19T12:53:54.422 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2709084197}) 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110378365}) 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3794359885}) 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110378365}) 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.330+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=564035465}) 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216136412}) 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1424658614}) 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216136412}) 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2744312841}) 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.423 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216136412}) 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=778909426}) 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216136412}) 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2100329510}) 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216136412}) 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1760317568}) 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216136412}) 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1327560869}) 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216136412}) 2024-08-19T12:53:54.424 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2648786342}) 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216136412}) 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.332+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=197250878}) 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439278064}) 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=682836565}) 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439278064}) 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022743739}) 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439278064}) 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1777839104}) 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.425 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439278064}) 2024-08-19T12:53:54.426 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2851780135}) 2024-08-19T12:53:54.426 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.426 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439278064}) 2024-08-19T12:53:54.426 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2297516313}) 2024-08-19T12:53:54.426 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.426 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439278064}) 2024-08-19T12:53:54.426 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2717534603}) 2024-08-19T12:53:54.426 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.426 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439278064}) 2024-08-19T12:53:54.426 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2961649140}) 2024-08-19T12:53:54.426 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.426 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439278064}) 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.333+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=519114132}) 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745826603}) 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3899800129}) 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745826603}) 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1644561138}) 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745826603}) 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2656900229}) 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745826603}) 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3988390214}) 2024-08-19T12:53:54.427 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745826603}) 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1205007892}) 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745826603}) 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1319044258}) 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.334+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745826603}) 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.335+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1412555595}) 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.335+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.335+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745826603}) 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.335+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3177812383}) 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321289209}) 2024-08-19T12:53:54.428 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1125814056}) 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321289209}) 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526017993}) 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321289209}) 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4289435675}) 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321289209}) 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=374067597}) 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321289209}) 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=722276011}) 2024-08-19T12:53:54.429 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321289209}) 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4185961332}) 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321289209}) 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4290922195}) 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3321289209}) 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.336+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2722638545}) 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115273336}) 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2564163985}) 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.430 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115273336}) 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3091066921}) 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115273336}) 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2881917443}) 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115273336}) 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3611496743}) 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115273336}) 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2542634782}) 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115273336}) 2024-08-19T12:53:54.431 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=124738326}) 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115273336}) 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1718262576}) 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115273336}) 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3960736934}) 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412473123}) 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2781012158}) 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412473123}) 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3610269559}) 2024-08-19T12:53:54.432 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412473123}) 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1014778406}) 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412473123}) 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3863399570}) 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412473123}) 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1308003082}) 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412473123}) 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3094475965}) 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412473123}) 2024-08-19T12:53:54.433 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1565788612}) 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412473123}) 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1115324046}) 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165547953}) 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131410140}) 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165547953}) 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3300513335}) 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165547953}) 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4027103479}) 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.434 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165547953}) 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1549601319}) 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165547953}) 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=900596203}) 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165547953}) 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=115478499}) 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165547953}) 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3141790646}) 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.341+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165547953}) 2024-08-19T12:53:54.435 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.341+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1720249098}) 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601899706}) 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3267735345}) 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601899706}) 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243696470}) 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601899706}) 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2881995044}) 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601899706}) 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=620441442}) 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.436 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601899706}) 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1077318032}) 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601899706}) 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3578258218}) 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601899706}) 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=700879525}) 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601899706}) 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3854573263}) 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625690055}) 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4234883614}) 2024-08-19T12:53:54.437 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625690055}) 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461282544}) 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625690055}) 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3254667491}) 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625690055}) 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=125070860}) 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625690055}) 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1697520519}) 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.438 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625690055}) 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1277170529}) 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625690055}) 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=598521515}) 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625690055}) 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3723152367}) 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.353+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.353+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281778351}) 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.353+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181235233}) 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.353+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.353+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281778351}) 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.353+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290360014}) 2024-08-19T12:53:54.439 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281778351}) 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=387272913}) 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281778351}) 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3152795432}) 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281778351}) 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1888976533}) 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281778351}) 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3765918037}) 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281778351}) 2024-08-19T12:53:54.440 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=273864873}) 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281778351}) 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.354+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4044507838}) 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117223044}) 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2452746198}) 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117223044}) 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907803359}) 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117223044}) 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2866446763}) 2024-08-19T12:53:54.441 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117223044}) 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=227081653}) 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117223044}) 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2960719938}) 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117223044}) 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4172721630}) 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117223044}) 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2437298841}) 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117223044}) 2024-08-19T12:53:54.442 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.355+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=903088458}) 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803992594}) 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=990077914}) 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803992594}) 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=412906686}) 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803992594}) 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3706525851}) 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803992594}) 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2966929199}) 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.443 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803992594}) 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2522894946}) 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803992594}) 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2809182588}) 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803992594}) 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=989693773}) 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=803992594}) 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.357+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2509041910}) 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510123625}) 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3558591032}) 2024-08-19T12:53:54.444 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510123625}) 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4113244578}) 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510123625}) 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3754478625}) 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510123625}) 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3076899601}) 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510123625}) 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.358+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3166695743}) 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.359+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.359+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510123625}) 2024-08-19T12:53:54.445 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.359+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2923613839}) 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.359+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.359+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510123625}) 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.359+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3241135558}) 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.359+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.359+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510123625}) 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.359+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2634533605}) 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867488383}) 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=621454528}) 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867488383}) 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1004861127}) 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.446 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867488383}) 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3477699891}) 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867488383}) 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=752944883}) 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867488383}) 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2774796292}) 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867488383}) 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3724995488}) 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867488383}) 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3018596427}) 2024-08-19T12:53:54.447 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867488383}) 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.360+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2104479243}) 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673302847}) 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1180862188}) 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673302847}) 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=632660347}) 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673302847}) 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2999393741}) 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673302847}) 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3784758885}) 2024-08-19T12:53:54.448 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.361+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.362+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673302847}) 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.362+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3694479545}) 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.362+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.362+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673302847}) 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.362+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2086683890}) 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.362+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.362+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673302847}) 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.362+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1100230704}) 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.362+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.362+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673302847}) 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.362+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1372520322}) 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096220343}) 2024-08-19T12:53:54.449 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2971446451}) 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096220343}) 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2723458321}) 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096220343}) 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3793545308}) 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096220343}) 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2620372789}) 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096220343}) 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=898353156}) 2024-08-19T12:53:54.450 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096220343}) 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=415314946}) 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096220343}) 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4103802113}) 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096220343}) 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.363+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=862081045}) 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.364+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.364+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094829432}) 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.364+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2236479492}) 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.364+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.364+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094829432}) 2024-08-19T12:53:54.451 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.364+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190349161}) 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094829432}) 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1899019042}) 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094829432}) 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1916482071}) 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094829432}) 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2956986260}) 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094829432}) 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2505767765}) 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094829432}) 2024-08-19T12:53:54.452 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1024910086}) 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094829432}) 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.365+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2614642479}) 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632628781}) 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1099831797}) 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632628781}) 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4245270923}) 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632628781}) 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=309610858}) 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.453 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632628781}) 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1426807291}) 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632628781}) 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4062133909}) 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632628781}) 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2627059132}) 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632628781}) 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2901656239}) 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632628781}) 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.366+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1178009084}) 2024-08-19T12:53:54.454 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.375+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.375+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982011261}) 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.375+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=550319543}) 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.375+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.375+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982011261}) 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=900614635}) 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982011261}) 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2258587875}) 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982011261}) 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1395534780}) 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.455 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982011261}) 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1404648271}) 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982011261}) 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1857247986}) 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982011261}) 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=110263134}) 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982011261}) 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.376+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4119699702}) 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46582827}) 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=243587380}) 2024-08-19T12:53:54.456 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46582827}) 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1260898144}) 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46582827}) 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3199050873}) 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46582827}) 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2725862707}) 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46582827}) 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1411758988}) 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46582827}) 2024-08-19T12:53:54.457 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2856629002}) 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46582827}) 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=825320749}) 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.377+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46582827}) 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.378+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1459073474}) 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795463265}) 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2010789588}) 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795463265}) 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3138074820}) 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.458 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795463265}) 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4018204550}) 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795463265}) 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=843209053}) 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795463265}) 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3993847096}) 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.379+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795463265}) 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.380+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=497832406}) 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.380+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.380+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795463265}) 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.380+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3750436232}) 2024-08-19T12:53:54.459 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.380+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.380+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795463265}) 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.380+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2588157406}) 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956259741}) 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2993281257}) 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956259741}) 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2142956020}) 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956259741}) 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3714611462}) 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956259741}) 2024-08-19T12:53:54.460 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=432317897}) 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956259741}) 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2442669260}) 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956259741}) 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2020112985}) 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956259741}) 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2624850683}) 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956259741}) 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.381+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=485329437}) 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.382+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.461 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124917214}) 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2467338707}) 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124917214}) 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1201553167}) 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124917214}) 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1598863705}) 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124917214}) 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=617498839}) 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124917214}) 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1009036022}) 2024-08-19T12:53:54.462 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124917214}) 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2308600408}) 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124917214}) 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3148978439}) 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124917214}) 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.383+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4120986763}) 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297397569}) 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2815325996}) 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297397569}) 2024-08-19T12:53:54.463 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1634155674}) 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297397569}) 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1237112360}) 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297397569}) 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3403955359}) 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297397569}) 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3908366906}) 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297397569}) 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3787066697}) 2024-08-19T12:53:54.464 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297397569}) 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=166060669}) 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297397569}) 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.385+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1672869369}) 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.386+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.386+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041498819}) 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.386+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1815368125}) 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.386+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.386+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041498819}) 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.386+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4236705044}) 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.386+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.386+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041498819}) 2024-08-19T12:53:54.465 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.386+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=937297608}) 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.386+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041498819}) 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2025358845}) 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041498819}) 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1393767222}) 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041498819}) 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4027046481}) 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041498819}) 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=817335914}) 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.466 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041498819}) 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.387+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3981920140}) 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894211296}) 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=556860775}) 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894211296}) 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3693549610}) 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894211296}) 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=292208812}) 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894211296}) 2024-08-19T12:53:54.467 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4034684555}) 2024-08-19T12:53:54.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894211296}) 2024-08-19T12:53:54.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3707856538}) 2024-08-19T12:53:54.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894211296}) 2024-08-19T12:53:54.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=441759933}) 2024-08-19T12:53:54.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894211296}) 2024-08-19T12:53:54.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3292899358}) 2024-08-19T12:53:54.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894211296}) 2024-08-19T12:53:54.468 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.388+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1853150874}) 2024-08-19T12:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.395+0000 7f78dcd28640 10 monclient: get_monmap_and_config 2024-08-19T12:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.395+0000 7f78dcd28640 10 monclient: build_initial_monmap 2024-08-19T12:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.395+0000 7f78dcd28640 10 monclient: monmap: 2024-08-19T12:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:54.396400+0000 2024-08-19T12:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:54.396400+0000 2024-08-19T12:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:54.478 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 1 Processor -- start 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 1 -- start start 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 10 monclient: init 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 10 monclient: _add_conns ranks=[8,4,7,5,6,0,1,3,2] 2024-08-19T12:53:54.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 10 monclient(hunting): picked mon.noname-e con 0x7f78d8073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 10 monclient(hunting): picked mon.noname-c con 0x7f78d80a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 10 monclient(hunting): picked mon.noname-i con 0x7f78d813f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d809dae0 con 0x7f78d80a57a0 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d8136910 con 0x7f78d813f2e0 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d809dd50 con 0x7f78d8073470 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78dcd28640 10 monclient(hunting): authenticate will time out at 1628.483889s 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78d659c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f78d8073470 0x7f78d8136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40716/0 (socket says 172.21.15.26:40716) 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.396+0000 7f78d659c640 1 -- 172.21.15.26:0/3591202511 learned_addr learned my addr 172.21.15.26:0/3591202511 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d5d9b640 1 -- 172.21.15.26:0/3591202511 >> v1:172.21.15.26:6791/0 conn(0x7f78d80a57a0 legacy=0x7f78d80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d659c640 1 -- 172.21.15.26:0/3591202511 >> v1:172.21.15.26:6793/0 conn(0x7f78d8073470 legacy=0x7f78d8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 1 -- 172.21.15.26:0/3591202511 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1449507737 0 0) 0x7f78d809dae0 con 0x7f78d80a57a0 2024-08-19T12:53:54.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient(hunting): my global_id is 184495 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 1 -- 172.21.15.26:0/3591202511 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c4003520 con 0x7f78d80a57a0 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 1 -- 172.21.15.26:0/3591202511 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1983283843 0 0) 0x7f78d809dd50 con 0x7f78d8073470 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient(hunting): my global_id is 184364 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 1 -- 172.21.15.26:0/3591202511 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78d809dae0 con 0x7f78d8073470 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d6d9d640 1 -- 172.21.15.26:0/3591202511 >> v1:172.21.15.104:6792/0 conn(0x7f78d813f2e0 legacy=0x7f78d81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 1 -- 172.21.15.26:0/3591202511 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1183170826 0 0) 0x7f78c4003520 con 0x7f78d80a57a0 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 1 -- 172.21.15.26:0/3591202511 >> v1:172.21.15.26:6793/0 conn(0x7f78d8073470 legacy=0x7f78d8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 1 -- 172.21.15.26:0/3591202511 >> v1:172.21.15.104:6792/0 conn(0x7f78d813f2e0 legacy=0x7f78d81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient: _finish_hunting 0 2024-08-19T12:53:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 1 monclient: found mon.noname-c 2024-08-19T12:53:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:53:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 1 -- 172.21.15.26:0/3591202511 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d809e030 con 0x7f78d80a57a0 2024-08-19T12:53:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient: _finish_auth 0 2024-08-19T12:53:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient: _check_auth_tickets 2024-08-19T12:53:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 1 -- 172.21.15.26:0/3591202511 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f78c0001b30 con 0x7f78d80a57a0 2024-08-19T12:53:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78dcd28640 5 monclient: authenticate success, global_id 184495 2024-08-19T12:53:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78dcd28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:54.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient: dump: 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.483 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78dcd28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 1 -- 172.21.15.26:0/3591202511 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78c0003610 con 0x7f78d80a57a0 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 1 -- 172.21.15.26:0/3591202511 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f78c0003b80 con 0x7f78d80a57a0 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78dcd28640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78dcd28640 4 set_mon_vals no callback set 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.399+0000 7f78d559a640 10 monclient: dump: 2024-08-19T12:53:54.484 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.485 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.486 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.486 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.400+0000 7f78dcd28640 10 monclient: shutdown 2024-08-19T12:53:54.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.400+0000 7f78dcd28640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:54.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.400+0000 7f78dcd28640 1 -- 172.21.15.26:0/3591202511 >> v1:172.21.15.26:6791/0 conn(0x7f78d80a57a0 legacy=0x7f78d80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.400+0000 7f78dcd28640 1 -- 172.21.15.26:0/3591202511 shutdown_connections 2024-08-19T12:53:54.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.400+0000 7f78dcd28640 1 -- 172.21.15.26:0/3591202511 >> 172.21.15.26:0/3591202511 conn(0x7f78d80abff0 msgr2=0x7f78d80ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:54.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.400+0000 7f78dcd28640 1 -- 172.21.15.26:0/3591202511 shutdown_connections 2024-08-19T12:53:54.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.400+0000 7f78dcd28640 1 -- 172.21.15.26:0/3591202511 wait complete. 2024-08-19T12:53:54.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 10 monclient: build_initial_monmap 2024-08-19T12:53:54.486 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 10 monclient: monmap: 2024-08-19T12:53:54.486 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:54.487 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 1 librados: starting msgr at 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 1 librados: starting objecter 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 1 Processor -- start 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 1 -- start start 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 1 librados: setting wanted keys 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 1 librados: calling monclient init 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 10 monclient: init 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 10 monclient: _add_conns ranks=[2,1,4,0,3,8,6,5,7] 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 10 monclient(hunting): picked mon.noname-b con 0x7f78d8073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 10 monclient(hunting): picked mon.noname-f con 0x7f78d80a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 10 monclient(hunting): picked mon.noname-c con 0x7f78d813f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:54.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d8136d70 con 0x7f78d80a57a0 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d80a3d30 con 0x7f78d8073470 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d80a3890 con 0x7f78d813f2e0 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78dcd28640 10 monclient(hunting): authenticate will time out at 1628.488916s 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78d659c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f78d8073470 0x7f78d80a2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47914/0 (socket says 172.21.15.26:47914) 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.401+0000 7f78d659c640 1 -- 172.21.15.26:0/1482441101 learned_addr learned my addr 172.21.15.26:0/1482441101 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78d6d9d640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.26:6791/0 conn(0x7f78d813f2e0 legacy=0x7f78d81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 client.?.objecter ms_handle_connect 0x7f78d813f2e0 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78d659c640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.26:6790/0 conn(0x7f78d8073470 legacy=0x7f78d80a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 client.?.objecter ms_handle_connect 0x7f78d8073470 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3141958244 0 0) 0x7f78d80a3890 con 0x7f78d813f2e0 2024-08-19T12:53:54.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): my global_id is 184504 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78b8004eb0 con 0x7f78d813f2e0 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3748732083 0 0) 0x7f78d80a3d30 con 0x7f78d8073470 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): my global_id is 27200 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78d80a3890 con 0x7f78d8073470 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78d5d9b640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.104:6789/0 conn(0x7f78d80a57a0 legacy=0x7f78d80a2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 client.?.objecter ms_handle_connect 0x7f78d80a57a0 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2547460503 0 0) 0x7f78b8004eb0 con 0x7f78d813f2e0 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78d80a3d30 con 0x7f78d813f2e0 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f78d0001cb0 con 0x7f78d813f2e0 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): dump: 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.491 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 950941456 0 0) 0x7f78d80a3890 con 0x7f78d8073470 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78b8004eb0 con 0x7f78d8073470 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f78c8004b60 con 0x7f78d8073470 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): dump: 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.493 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355114701 0 0) 0x7f78d8136d70 con 0x7f78d80a57a0 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient(hunting): my global_id is 184438 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78d80a3890 con 0x7f78d80a57a0 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3932991277 0 0) 0x7f78d80a3d30 con 0x7f78d813f2e0 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.26:6790/0 conn(0x7f78d8073470 legacy=0x7f78d80a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.104:6789/0 conn(0x7f78d80a57a0 legacy=0x7f78d80a2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient: _finish_hunting 0 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 monclient: found mon.c 2024-08-19T12:53:54.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d80a5350 con 0x7f78d813f2e0 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient: _finish_auth 0 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient: _check_auth_tickets 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78dcd28640 5 monclient: authenticate success, global_id 184504 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78dcd28640 10 monclient: _renew_subs 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78dcd28640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78dcd28640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78d80a3050 con 0x7f78d813f2e0 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78dcd28640 10 client.184504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78dcd28640 10 monclient: _renew_subs 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78dcd28640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78dcd28640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f78d81d7870 con 0x7f78d813f2e0 2024-08-19T12:53:54.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78dcd28640 1 librados: init done 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78d00039f0 con 0x7f78d813f2e0 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f78d0003ee0 con 0x7f78d813f2e0 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78d4d99640 4 set_mon_vals no callback set 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.404+0000 7f78ceffd640 10 monclient: dump: 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.496 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.405+0000 7f78dcd28640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.405+0000 7f78dcd28640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f78bc0065c0 con 0x7f78bc002860 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.405+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f78d0004080 con 0x7f78d813f2e0 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.405+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(571..571 src has 350..571) ==== 5857+0+0 (unknown 2999744453 0 0) 0x7f78d004e140 con 0x7f78d813f2e0 2024-08-19T12:53:54.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.405+0000 7f78ceffd640 10 client.184504.objecter ms_dispatch 0x7f78d8000b90 osd_map(571..571 src has 350..571) 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.405+0000 7f78ceffd640 3 client.184504.objecter handle_osd_map got epochs [571,571] > 0 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.405+0000 7f78ceffd640 3 client.184504.objecter handle_osd_map decoding full epoch 571 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.405+0000 7f78ceffd640 20 client.184504.objecter dump_active .. 0 homeless 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.406+0000 7f78d5d9b640 10 monclient: get_auth_request con 0x7f78b8041690 auth_method 0 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.407+0000 7f78ceffd640 10 client.184504.objecter ms_handle_connect 0x7f78bc002860 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.407+0000 7f78ceffd640 10 client.184504.objecter resend_mon_ops 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.407+0000 7f78d659c640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.26:6792/0 conn(0x7f78bc002860 legacy=0x7f78bc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.407+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1431848696 0 0) 0x7f78bc0065c0 con 0x7f78bc002860 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.407+0000 7f78ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.407+0000 7f78ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.407+0000 7f78ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.407+0000 7f78ceffd640 10 monclient(hunting): my global_id is 194190 2024-08-19T12:53:54.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.407+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78d80a3d30 con 0x7f78bc002860 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2156800456 0 0) 0x7f78d80a3d30 con 0x7f78bc002860 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78bc0065c0 con 0x7f78bc002860 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f78c8005300 con 0x7f78bc002860 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 10 monclient: dump: 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.499 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1271545560 0 0) 0x7f78bc0065c0 con 0x7f78bc002860 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78bc00a160 con 0x7f78bc002860 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 10 client.184504.objecter ms_handle_connect 0x7f78b8041690 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f78c8007540 con 0x7f78bc002860 2024-08-19T12:53:54.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.26:6792/0 conn(0x7f78bc002860 legacy=0x7f78bc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.408+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.26:6792/0 conn(0x7f78bc002860 legacy=0x7f78bc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.414+0000 7f78dcd28640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.414+0000 7f78dcd28640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f78bc009e70 con 0x7f78bc009130 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 10 client.184504.objecter ms_handle_connect 0x7f78bc009130 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 10 client.184504.objecter resend_mon_ops 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78d659c640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.26:6792/0 conn(0x7f78bc009130 legacy=0x7f78bc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1708774368 0 0) 0x7f78bc009e70 con 0x7f78bc009130 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 10 monclient(hunting): my global_id is 194199 2024-08-19T12:53:54.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78bc0065c0 con 0x7f78bc009130 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1700295546 0 0) 0x7f78bc0065c0 con 0x7f78bc009130 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78bc009e70 con 0x7f78bc009130 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f78c8006d70 con 0x7f78bc009130 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 10 monclient: dump: 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.502 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2393535928 0 0) 0x7f78bc009e70 con 0x7f78bc009130 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.417+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f78bc00c3f0 con 0x7f78bc009130 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 3501673221) 0x7f78c8007090 con 0x7f78bc009130 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78ceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:54.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78ceffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:54.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.26:6792/0 conn(0x7f78bc009130 legacy=0x7f78bc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78ceffd640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.26:6792/0 conn(0x7f78bc009130 legacy=0x7f78bc0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:54.504 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:54.504 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:54.504 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:54.504 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:54.504 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:54.504 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 19344, 2024-08-19T12:53:54.505 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:54.505 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:54.505 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:54.505 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:54.505 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:54.505 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:54.505 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:54.505 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:54.505 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:54.505 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:54.505 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.505 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.506 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.507 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:54.508 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.509 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:54.510 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.511 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.512 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.513 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:54.514 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.515 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:54.516 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.518 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.519 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:54.520 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.522 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:54.522 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:54.522 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.522 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.522 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.522 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.522 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.522 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:54.522 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:54.523 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:54.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 10 librados: watch_flush enter 2024-08-19T12:53:54.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 10 librados: watch_flush exit 2024-08-19T12:53:54.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 20 client.184504.objecter shutdown clearing up homeless session... 2024-08-19T12:53:54.525 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 10 client.184504.objecter successfully canceled tick 2024-08-19T12:53:54.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.104:6800/899887506 conn(0x7f78b8041690 legacy=0x7f78b8043b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 10 monclient: shutdown 2024-08-19T12:53:54.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:54.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 1 -- 172.21.15.26:0/1482441101 >> v1:172.21.15.26:6791/0 conn(0x7f78d813f2e0 legacy=0x7f78d81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78d6d9d640 1 -- 172.21.15.26:0/1482441101 reap_dead start 2024-08-19T12:53:54.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78d6d9d640 1 -- 172.21.15.26:0/1482441101 reap_dead start 2024-08-19T12:53:54.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 1 -- 172.21.15.26:0/1482441101 shutdown_connections 2024-08-19T12:53:54.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 1 -- 172.21.15.26:0/1482441101 >> 172.21.15.26:0/1482441101 conn(0x7f78d80abff0 msgr2=0x7f78d81338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:54.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 1 -- 172.21.15.26:0/1482441101 shutdown_connections 2024-08-19T12:53:54.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 1 -- 172.21.15.26:0/1482441101 wait complete. 2024-08-19T12:53:54.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.418+0000 7f78dcd28640 1 librados: shutdown 2024-08-19T12:53:54.527 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:54.527 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:53:54.527 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:53:54.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.665+0000 7f297ee92640 10 monclient: get_monmap_and_config 2024-08-19T12:53:54.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.665+0000 7f297ee92640 10 monclient: build_initial_monmap 2024-08-19T12:53:54.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.665+0000 7f297ee92640 10 monclient: monmap: 2024-08-19T12:53:54.666 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:54.666 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.666 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:54.666341+0000 2024-08-19T12:53:54.666 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:54.666341+0000 2024-08-19T12:53:54.667 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:54.667 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.667 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:54.667 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:54.667 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:54.667 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:54.667 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:54.667 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:54.667 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:54.667 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:54.667 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:54.667 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.668 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 1 Processor -- start 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 1 -- start start 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 10 monclient: init 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 10 monclient: _add_conns ranks=[1,8,2,0,3,4,5,7,6] 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 10 monclient(hunting): picked mon.noname-f con 0x7f297806e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 10 monclient(hunting): picked mon.noname-e con 0x7f29780a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 10 monclient(hunting): picked mon.noname-b con 0x7f29780a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2978136790 con 0x7f297806e7b0 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f297809ddf0 con 0x7f29780a5d50 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f297809e0d0 con 0x7f29780a5920 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297ee92640 10 monclient(hunting): authenticate will time out at 1628.753783s 2024-08-19T12:53:54.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297d42f640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f29780a5d50 0x7f2978141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47920/0 (socket says 172.21.15.26:47920) 2024-08-19T12:53:54.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f297d42f640 1 -- 172.21.15.26:0/1973792979 learned_addr learned my addr 172.21.15.26:0/1973792979 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:54.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.666+0000 7f2977fff640 1 --1- 172.21.15.26:0/1973792979 >> v1:172.21.15.26:6793/0 conn(0x7f29780a5920 0x7f297813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40732/0 (socket says 172.21.15.26:40732) 2024-08-19T12:53:54.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f297d42f640 1 -- 172.21.15.26:0/1973792979 >> v1:172.21.15.26:6790/0 conn(0x7f29780a5d50 legacy=0x7f2978141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f2977fff640 1 -- 172.21.15.26:0/1973792979 >> v1:172.21.15.26:6793/0 conn(0x7f29780a5920 legacy=0x7f297813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.673 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f297cc2e640 1 -- 172.21.15.26:0/1973792979 >> v1:172.21.15.104:6789/0 conn(0x7f297806e7b0 legacy=0x7f2978136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3671741286 0 0) 0x7f29700017d0 con 0x7f29780a5d50 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient(hunting): my global_id is 27209 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2968003520 con 0x7f29780a5d50 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2231385818 0 0) 0x7f297809e0d0 con 0x7f29780a5920 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient(hunting): my global_id is 184373 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29700017d0 con 0x7f29780a5920 2024-08-19T12:53:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3362342952 0 0) 0x7f2978136790 con 0x7f297806e7b0 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient(hunting): my global_id is 184447 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f297809e0d0 con 0x7f297806e7b0 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2547785250 0 0) 0x7f29700017d0 con 0x7f29780a5920 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 >> v1:172.21.15.26:6790/0 conn(0x7f29780a5d50 legacy=0x7f2978141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 >> v1:172.21.15.104:6789/0 conn(0x7f297806e7b0 legacy=0x7f2978136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient: _finish_hunting 0 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 monclient: found mon.noname-e 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:53:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f297809e3b0 con 0x7f29780a5920 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient: _finish_auth 0 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient: _check_auth_tickets 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f297ee92640 5 monclient: authenticate success, global_id 184373 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2964001ac0 con 0x7f29780a5920 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f297ee92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient: dump: 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.676 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f297ee92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f29640047e0 con 0x7f29780a5920 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:54.677 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 1 -- 172.21.15.26:0/1973792979 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2964004d50 con 0x7f29780a5920 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f29777fe640 10 monclient: dump: 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.678 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f297ee92640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.669+0000 7f297ee92640 4 set_mon_vals no callback set 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.670+0000 7f297ee92640 10 monclient: shutdown 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.670+0000 7f297ee92640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.670+0000 7f297ee92640 1 -- 172.21.15.26:0/1973792979 >> v1:172.21.15.26:6793/0 conn(0x7f29780a5920 legacy=0x7f297813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.670+0000 7f297ee92640 1 -- 172.21.15.26:0/1973792979 shutdown_connections 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.670+0000 7f297ee92640 1 -- 172.21.15.26:0/1973792979 >> 172.21.15.26:0/1973792979 conn(0x7f29780a8740 msgr2=0x7f29780a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:54.679 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.670+0000 7f297ee92640 1 -- 172.21.15.26:0/1973792979 shutdown_connections 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.670+0000 7f297ee92640 1 -- 172.21.15.26:0/1973792979 wait complete. 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.670+0000 7f297ee92640 10 monclient: build_initial_monmap 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.670+0000 7f297ee92640 10 monclient: monmap: 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:54.680 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 1 librados: starting msgr at 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 1 librados: starting objecter 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 1 Processor -- start 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 1 -- start start 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 1 librados: setting wanted keys 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 1 librados: calling monclient init 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 10 monclient: init 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 10 monclient: _add_conns ranks=[7,2,8,0,4,3,6,1,5] 2024-08-19T12:53:54.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 10 monclient(hunting): picked mon.noname-i con 0x7f297806e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:54.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 10 monclient(hunting): picked mon.noname-b con 0x7f29780a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:54.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 10 monclient(hunting): picked mon.noname-e con 0x7f29780a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:54.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29781239f0 con 0x7f29780a5920 2024-08-19T12:53:54.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2978124550 con 0x7f297806e7b0 2024-08-19T12:53:54.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29781254b0 con 0x7f29780a5d50 2024-08-19T12:53:54.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:54.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297ee92640 10 monclient(hunting): authenticate will time out at 1628.758887s 2024-08-19T12:53:54.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f2977fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f29780a5920 0x7f29781d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47926/0 (socket says 172.21.15.26:47926) 2024-08-19T12:53:54.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f297d42f640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f29780a5d50 0x7f29781d6bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40738/0 (socket says 172.21.15.26:40738) 2024-08-19T12:53:54.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.671+0000 7f2977fff640 1 -- 172.21.15.26:0/569330008 learned_addr learned my addr 172.21.15.26:0/569330008 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:54.688 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f297d42f640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.26:6793/0 conn(0x7f29780a5d50 legacy=0x7f29781d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f2977fff640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.26:6790/0 conn(0x7f29780a5920 legacy=0x7f29781d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 client.?.objecter ms_handle_connect 0x7f29780a5d50 2024-08-19T12:53:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 client.?.objecter ms_handle_connect 0x7f29780a5920 2024-08-19T12:53:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f297cc2e640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.104:6792/0 conn(0x7f297806e7b0 legacy=0x7f29781ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 client.?.objecter ms_handle_connect 0x7f297806e7b0 2024-08-19T12:53:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4224326609 0 0) 0x7f29781254b0 con 0x7f29780a5d50 2024-08-19T12:53:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): my global_id is 184382 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f295c004eb0 con 0x7f29780a5d50 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1033688962 0 0) 0x7f29781239f0 con 0x7f29780a5920 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): my global_id is 27218 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29781254b0 con 0x7f29780a5920 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 197491994 0 0) 0x7f2978124550 con 0x7f297806e7b0 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.690 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): my global_id is 136177 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29781239f0 con 0x7f297806e7b0 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2720384275 0 0) 0x7f295c004eb0 con 0x7f29780a5d50 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2978124550 con 0x7f29780a5d50 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2970001700 con 0x7f29780a5d50 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): dump: 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.691 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 169251271 0 0) 0x7f29781254b0 con 0x7f29780a5920 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f295c004eb0 con 0x7f29780a5920 2024-08-19T12:53:54.692 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3317887679 0 0) 0x7f29781239f0 con 0x7f297806e7b0 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29781254b0 con 0x7f297806e7b0 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2964004cd0 con 0x7f29780a5920 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): dump: 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f296c003610 con 0x7f297806e7b0 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient(hunting): dump: 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.694 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2505524100 0 0) 0x7f2978124550 con 0x7f29780a5d50 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.104:6792/0 conn(0x7f297806e7b0 legacy=0x7f29781ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.26:6790/0 conn(0x7f29780a5920 legacy=0x7f29781d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient: _finish_hunting 0 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 monclient: found mon.e 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2978125790 con 0x7f29780a5d50 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient: _finish_auth 0 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient: _check_auth_tickets 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f297ee92640 5 monclient: authenticate success, global_id 184382 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f297ee92640 10 monclient: _renew_subs 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f297ee92640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:54.696 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f297ee92640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29781232a0 con 0x7f29780a5d50 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f297ee92640 10 client.184382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f297ee92640 10 monclient: _renew_subs 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f297ee92640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f297ee92640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f29781266a0 con 0x7f29780a5d50 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f297ee92640 1 librados: init done 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2970004f90 con 0x7f29780a5d50 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2970005480 con 0x7f29780a5d50 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f29757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.674+0000 7f2976ffd640 4 set_mon_vals no callback set 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.675+0000 7f29757fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.675+0000 7f29757fa640 10 monclient: dump: 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.697 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.698 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.675+0000 7f297ee92640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.675+0000 7f297ee92640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f29580065c0 con 0x7f2958002860 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.675+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f29700118e0 con 0x7f29780a5d50 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.676+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(571..571 src has 350..571) ==== 5857+0+0 (unknown 2999744453 0 0) 0x7f297004e580 con 0x7f29780a5d50 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.676+0000 7f29757fa640 10 client.184382.objecter ms_dispatch 0x7f2978000b90 osd_map(571..571 src has 350..571) 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.676+0000 7f29757fa640 3 client.184382.objecter handle_osd_map got epochs [571,571] > 0 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.676+0000 7f29757fa640 3 client.184382.objecter handle_osd_map decoding full epoch 571 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.676+0000 7f29757fa640 20 client.184382.objecter dump_active .. 0 homeless 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.676+0000 7f2977fff640 10 monclient: get_auth_request con 0x7f295c041a00 auth_method 0 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.677+0000 7f29757fa640 10 client.184382.objecter ms_handle_connect 0x7f2958002860 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.677+0000 7f297cc2e640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.26:6793/0 conn(0x7f2958002860 legacy=0x7f2958004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.677+0000 7f29757fa640 10 client.184382.objecter resend_mon_ops 2024-08-19T12:53:54.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2233636441 0 0) 0x7f29580065c0 con 0x7f2958002860 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 10 monclient(hunting): my global_id is 184391 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2978124550 con 0x7f2958002860 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2753394129 0 0) 0x7f2978124550 con 0x7f2958002860 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29580065c0 con 0x7f2958002860 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f296c003d20 con 0x7f2958002860 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 10 monclient: dump: 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.700 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1907892309 0 0) 0x7f29580065c0 con 0x7f2958002860 2024-08-19T12:53:54.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f295800a160 con 0x7f2958002860 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 10 client.184382.objecter ms_handle_connect 0x7f295c041a00 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f296c003f90 con 0x7f2958002860 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.26:6793/0 conn(0x7f2958002860 legacy=0x7f2958004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.678+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.26:6793/0 conn(0x7f2958002860 legacy=0x7f2958004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.684+0000 7f297ee92640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.684+0000 7f297ee92640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2958009e70 con 0x7f2958009130 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f297cc2e640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.26:6793/0 conn(0x7f2958009130 legacy=0x7f29580095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 10 client.184382.objecter ms_handle_connect 0x7f2958009130 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 10 client.184382.objecter resend_mon_ops 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2185752359 0 0) 0x7f2958009e70 con 0x7f2958009130 2024-08-19T12:53:54.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 10 monclient(hunting): my global_id is 184400 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29580065c0 con 0x7f2958009130 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3160975812 0 0) 0x7f29580065c0 con 0x7f2958009130 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2958009e70 con 0x7f2958009130 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f296c0043b0 con 0x7f2958009130 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 10 monclient: dump: 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.703 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.704 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1755626972 0 0) 0x7f2958009e70 con 0x7f2958009130 2024-08-19T12:53:54.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f295800c410 con 0x7f2958009130 2024-08-19T12:53:54.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 2467248249) 0x7f296c005b80 con 0x7f2958009130 2024-08-19T12:53:54.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:54.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:54.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.26:6793/0 conn(0x7f2958009130 legacy=0x7f29580095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.687+0000 7f29757fa640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.26:6793/0 conn(0x7f2958009130 legacy=0x7f29580095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:54.705 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:54.705 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:54.705 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:54.705 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 160153, 2024-08-19T12:53:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:54.706 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.707 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.708 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:54.709 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:54.711 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:54.713 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:54.714 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.716 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.717 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.719 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:54.720 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:54.722 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.723 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.724 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.725 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:54.725 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 10 librados: watch_flush enter 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 10 librados: watch_flush exit 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 20 client.184382.objecter shutdown clearing up homeless session... 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 10 client.184382.objecter successfully canceled tick 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.104:6800/899887506 conn(0x7f295c041a00 legacy=0x7f295c043ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 10 monclient: shutdown 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 1 -- 172.21.15.26:0/569330008 >> v1:172.21.15.26:6793/0 conn(0x7f29780a5d50 legacy=0x7f29781d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297d42f640 1 -- 172.21.15.26:0/569330008 reap_dead start 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297d42f640 1 -- 172.21.15.26:0/569330008 reap_dead start 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 1 -- 172.21.15.26:0/569330008 shutdown_connections 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 1 -- 172.21.15.26:0/569330008 >> 172.21.15.26:0/569330008 conn(0x7f29780a8740 msgr2=0x7f2978135120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:54.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 1 -- 172.21.15.26:0/569330008 shutdown_connections 2024-08-19T12:53:54.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 1 -- 172.21.15.26:0/569330008 wait complete. 2024-08-19T12:53:54.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.688+0000 7f297ee92640 1 librados: shutdown 2024-08-19T12:53:54.728 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:54.728 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:53:54.728 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:53:54.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.827+0000 7f621d227640 10 monclient: get_monmap_and_config 2024-08-19T12:53:54.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.827+0000 7f621d227640 10 monclient: build_initial_monmap 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.827+0000 7f621d227640 10 monclient: monmap: 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:54.827869+0000 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:54.827869+0000 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:54.829 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:54.829 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:54.829 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.827+0000 7f621d227640 1 Processor -- start 2024-08-19T12:53:54.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.827+0000 7f621d227640 1 -- start start 2024-08-19T12:53:54.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.827+0000 7f621d227640 10 monclient: init 2024-08-19T12:53:54.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.828+0000 7f621d227640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:54.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.828+0000 7f621d227640 10 monclient: _add_conns ranks=[5,1,6,2,3,7,8,0,4] 2024-08-19T12:53:54.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.828+0000 7f621d227640 10 monclient(hunting): picked mon.noname-h con 0x7f621806e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:53:54.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.828+0000 7f621d227640 10 monclient(hunting): picked mon.noname-f con 0x7f62180a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:54.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.828+0000 7f621d227640 10 monclient(hunting): picked mon.noname-d con 0x7f62180a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:53:54.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.828+0000 7f621d227640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6218136790 con 0x7f62180a5920 2024-08-19T12:53:54.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.828+0000 7f621d227640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f621809ddf0 con 0x7f621806e7b0 2024-08-19T12:53:54.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.828+0000 7f621d227640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f621809e0d0 con 0x7f62180a5d50 2024-08-19T12:53:54.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.828+0000 7f621d227640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:54.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.828+0000 7f621d227640 10 monclient(hunting): authenticate will time out at 1628.915213s 2024-08-19T12:53:54.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.828+0000 7f621759e640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f62180a5d50 0x7f6218141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:36164/0 (socket says 172.21.15.26:36164) 2024-08-19T12:53:54.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.828+0000 7f621759e640 1 -- 172.21.15.26:0/2134300131 learned_addr learned my addr 172.21.15.26:0/2134300131 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f621759e640 1 -- 172.21.15.26:0/2134300131 >> v1:172.21.15.26:6792/0 conn(0x7f62180a5d50 legacy=0x7f6218141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f621659c640 1 -- 172.21.15.26:0/2134300131 >> v1:172.21.15.104:6789/0 conn(0x7f62180a5920 legacy=0x7f621813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6216d9d640 1 -- 172.21.15.26:0/2134300131 >> v1:172.21.15.104:6791/0 conn(0x7f621806e7b0 legacy=0x7f6218136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890409450 0 0) 0x7f621809e0d0 con 0x7f62180a5d50 2024-08-19T12:53:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 10 monclient(hunting): my global_id is 194208 2024-08-19T12:53:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6204003520 con 0x7f62180a5d50 2024-08-19T12:53:54.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2093485873 0 0) 0x7f6218136790 con 0x7f62180a5920 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 10 monclient(hunting): my global_id is 184456 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f621809e0d0 con 0x7f62180a5920 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 950525420 0 0) 0x7f62080014a0 con 0x7f621806e7b0 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 10 monclient(hunting): my global_id is 184424 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.830+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6218136790 con 0x7f621806e7b0 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 945704432 0 0) 0x7f621809e0d0 con 0x7f62180a5920 2024-08-19T12:53:54.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 >> v1:172.21.15.26:6792/0 conn(0x7f62180a5d50 legacy=0x7f6218141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 >> v1:172.21.15.104:6791/0 conn(0x7f621806e7b0 legacy=0x7f6218136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 1 monclient: found mon.noname-f 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621809e3b0 con 0x7f62180a5920 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 10 monclient: _finish_auth 0 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6200001bd0 con 0x7f62180a5920 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f621d227640 5 monclient: authenticate success, global_id 184456 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 10 monclient: dump: 2024-08-19T12:53:54.837 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.838 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f621d227640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6200003610 con 0x7f62180a5920 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f621d227640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 1 -- 172.21.15.26:0/2134300131 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6200003b80 con 0x7f62180a5920 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f621d227640 4 set_mon_vals no callback set 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f6215d9b640 10 monclient: dump: 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.839 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f621d227640 10 monclient: shutdown 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f621d227640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f621d227640 1 -- 172.21.15.26:0/2134300131 >> v1:172.21.15.104:6789/0 conn(0x7f62180a5920 legacy=0x7f621813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f621d227640 1 -- 172.21.15.26:0/2134300131 shutdown_connections 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f621d227640 1 -- 172.21.15.26:0/2134300131 >> 172.21.15.26:0/2134300131 conn(0x7f62180a8740 msgr2=0x7f62180a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f621d227640 1 -- 172.21.15.26:0/2134300131 shutdown_connections 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.831+0000 7f621d227640 1 -- 172.21.15.26:0/2134300131 wait complete. 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.832+0000 7f621d227640 10 monclient: build_initial_monmap 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.832+0000 7f621d227640 10 monclient: monmap: 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:54.841 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.832+0000 7f621d227640 1 librados: starting msgr at 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.832+0000 7f621d227640 1 librados: starting objecter 2024-08-19T12:53:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.832+0000 7f621d227640 1 Processor -- start 2024-08-19T12:53:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 1 -- start start 2024-08-19T12:53:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 1 librados: setting wanted keys 2024-08-19T12:53:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 1 librados: calling monclient init 2024-08-19T12:53:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 10 monclient: init 2024-08-19T12:53:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 10 monclient: _add_conns ranks=[1,2,4,8,3,0,5,6,7] 2024-08-19T12:53:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 10 monclient(hunting): picked mon.noname-f con 0x7f621806e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 10 monclient(hunting): picked mon.noname-b con 0x7f62180a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 10 monclient(hunting): picked mon.noname-c con 0x7f62180a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:54.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62181328f0 con 0x7f621806e7b0 2024-08-19T12:53:54.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6218123970 con 0x7f62180a5920 2024-08-19T12:53:54.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62181347c0 con 0x7f62180a5d50 2024-08-19T12:53:54.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:54.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621d227640 10 monclient(hunting): authenticate will time out at 1628.920481s 2024-08-19T12:53:54.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621659c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f62180a5920 0x7f62181d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47930/0 (socket says 172.21.15.26:47930) 2024-08-19T12:53:54.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.833+0000 7f621659c640 1 -- 172.21.15.26:0/2453627953 learned_addr learned my addr 172.21.15.26:0/2453627953 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:54.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f621759e640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.26:6791/0 conn(0x7f62180a5d50 legacy=0x7f62181d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 client.?.objecter ms_handle_connect 0x7f62180a5d50 2024-08-19T12:53:54.848 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f621659c640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.26:6790/0 conn(0x7f62180a5920 legacy=0x7f62181d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 client.?.objecter ms_handle_connect 0x7f62180a5920 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f6216d9d640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.104:6789/0 conn(0x7f621806e7b0 legacy=0x7f62181ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 client.?.objecter ms_handle_connect 0x7f621806e7b0 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3014803218 0 0) 0x7f62181347c0 con 0x7f62180a5d50 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 monclient(hunting): my global_id is 184513 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61f8004eb0 con 0x7f62180a5d50 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3542225676 0 0) 0x7f6218123970 con 0x7f62180a5920 2024-08-19T12:53:54.849 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 10 monclient(hunting): my global_id is 27227 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62181347c0 con 0x7f62180a5920 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.835+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2994868075 0 0) 0x7f62181328f0 con 0x7f621806e7b0 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): my global_id is 184465 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6218123970 con 0x7f621806e7b0 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2269389768 0 0) 0x7f62181347c0 con 0x7f62180a5920 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62181328f0 con 0x7f62180a5920 2024-08-19T12:53:54.850 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 690810694 0 0) 0x7f61f8004eb0 con 0x7f62180a5d50 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f62181347c0 con 0x7f62180a5d50 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2861958458 0 0) 0x7f6218123970 con 0x7f621806e7b0 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61f8004eb0 con 0x7f621806e7b0 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6200001b90 con 0x7f62180a5920 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): dump: 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.851 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f62100020b0 con 0x7f62180a5d50 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:54.852 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): dump: 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.853 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f62080039f0 con 0x7f621806e7b0 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient(hunting): dump: 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.854 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1364655650 0 0) 0x7f61f8004eb0 con 0x7f621806e7b0 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.26:6791/0 conn(0x7f62180a5d50 legacy=0x7f62181d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.26:6790/0 conn(0x7f62180a5920 legacy=0x7f62181d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 monclient: found mon.f 2024-08-19T12:53:54.855 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6218142ec0 con 0x7f621806e7b0 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f621d227640 5 monclient: authenticate success, global_id 184465 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f621d227640 10 monclient: _renew_subs 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f621d227640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f621d227640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6218123690 con 0x7f621806e7b0 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f621d227640 10 client.184465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f621d227640 10 monclient: _renew_subs 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f621d227640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f621d227640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f62181290c0 con 0x7f621806e7b0 2024-08-19T12:53:54.856 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f621d227640 1 librados: init done 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f62080013e0 con 0x7f621806e7b0 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6208003ed0 con 0x7f621806e7b0 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f621559a640 4 set_mon_vals no callback set 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.836+0000 7f620f7fe640 10 monclient: dump: 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.857 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.837+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6208004070 con 0x7f621806e7b0 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.837+0000 7f621d227640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.837+0000 7f621d227640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f61fc0065c0 con 0x7f61fc002860 2024-08-19T12:53:54.858 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.837+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(571..571 src has 350..571) ==== 5857+0+0 (unknown 2999744453 0 0) 0x7f620804e140 con 0x7f621806e7b0 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.837+0000 7f620f7fe640 10 client.184465.objecter ms_dispatch 0x7f6218000b90 osd_map(571..571 src has 350..571) 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.837+0000 7f620f7fe640 3 client.184465.objecter handle_osd_map got epochs [571,571] > 0 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.837+0000 7f620f7fe640 3 client.184465.objecter handle_osd_map decoding full epoch 571 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.837+0000 7f620f7fe640 20 client.184465.objecter dump_active .. 0 homeless 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.837+0000 7f621759e640 10 monclient: get_auth_request con 0x7f61f8041a80 auth_method 0 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.839+0000 7f620f7fe640 10 client.184465.objecter ms_handle_connect 0x7f61fc002860 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.839+0000 7f621659c640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.104:6789/0 conn(0x7f61fc002860 legacy=0x7f61fc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.839+0000 7f620f7fe640 10 client.184465.objecter resend_mon_ops 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1759809611 0 0) 0x7f61fc0065c0 con 0x7f61fc002860 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 10 monclient(hunting): my global_id is 184474 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61f8004eb0 con 0x7f61fc002860 2024-08-19T12:53:54.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 10 client.184465.objecter ms_handle_connect 0x7f61f8041a80 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2995235449 0 0) 0x7f61f8004eb0 con 0x7f61fc002860 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61fc0065c0 con 0x7f61fc002860 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6200003610 con 0x7f61fc002860 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 10 monclient: dump: 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.860 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1483147338 0 0) 0x7f61fc0065c0 con 0x7f61fc002860 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61fc00a160 con 0x7f61fc002860 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f62000057c0 con 0x7f61fc002860 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:54.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.104:6789/0 conn(0x7f61fc002860 legacy=0x7f61fc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.840+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.104:6789/0 conn(0x7f61fc002860 legacy=0x7f61fc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:54.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.847+0000 7f621d227640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:54.862 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.847+0000 7f621d227640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f61fc009e70 con 0x7f61fc009130 2024-08-19T12:53:54.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.849+0000 7f621659c640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.104:6789/0 conn(0x7f61fc009130 legacy=0x7f61fc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:54.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.849+0000 7f620f7fe640 10 client.184465.objecter ms_handle_connect 0x7f61fc009130 2024-08-19T12:53:54.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.849+0000 7f620f7fe640 10 client.184465.objecter resend_mon_ops 2024-08-19T12:53:54.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.849+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4261836835 0 0) 0x7f61fc009e70 con 0x7f61fc009130 2024-08-19T12:53:54.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.849+0000 7f620f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:54.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.849+0000 7f620f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:54.863 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.849+0000 7f620f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.849+0000 7f620f7fe640 10 monclient(hunting): my global_id is 184483 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.849+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61fc0065c0 con 0x7f61fc009130 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2698067710 0 0) 0x7f61fc0065c0 con 0x7f61fc009130 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f61fc009e70 con 0x7f61fc009130 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6200006080 con 0x7f61fc009130 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 10 monclient: dump: 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:54.864 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4019547796 0 0) 0x7f61fc009e70 con 0x7f61fc009130 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f61fc00c410 con 0x7f61fc009130 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 3307865498) 0x7f6200008300 con 0x7f61fc009130 2024-08-19T12:53:54.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:54.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:54.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.104:6789/0 conn(0x7f61fc009130 legacy=0x7f61fc0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f620f7fe640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.104:6789/0 conn(0x7f61fc009130 legacy=0x7f61fc0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:54.866 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:54.866 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:54.866 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:54.866 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:54.866 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:54.866 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 160319, 2024-08-19T12:53:54.866 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 4, 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:54.867 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.868 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:54.870 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:54.871 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.872 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:54.874 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.874 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.874 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:54.874 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:54.874 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.874 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.874 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:54.875 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.876 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:54.878 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.879 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:54.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.883 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.884 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:53:54.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.885 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:54.885 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:53:54.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:54.885 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:54.885 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:54.885 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:54.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:54.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:54.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:54.885 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:54.885 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:54.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621d227640 10 librados: watch_flush enter 2024-08-19T12:53:54.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621d227640 10 librados: watch_flush exit 2024-08-19T12:53:54.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621d227640 20 client.184465.objecter shutdown clearing up homeless session... 2024-08-19T12:53:54.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621d227640 10 client.184465.objecter successfully canceled tick 2024-08-19T12:53:54.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621d227640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.104:6800/899887506 conn(0x7f61f8041a80 legacy=0x7f61f8043f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621d227640 10 monclient: shutdown 2024-08-19T12:53:54.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621d227640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:54.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621d227640 1 -- 172.21.15.26:0/2453627953 >> v1:172.21.15.104:6789/0 conn(0x7f621806e7b0 legacy=0x7f62181ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:54.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621759e640 1 -- 172.21.15.26:0/2453627953 reap_dead start 2024-08-19T12:53:54.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621759e640 1 -- 172.21.15.26:0/2453627953 reap_dead start 2024-08-19T12:53:54.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621d227640 1 -- 172.21.15.26:0/2453627953 shutdown_connections 2024-08-19T12:53:54.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621d227640 1 -- 172.21.15.26:0/2453627953 >> 172.21.15.26:0/2453627953 conn(0x7f62180a8740 msgr2=0x7f621806d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:54.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621d227640 1 -- 172.21.15.26:0/2453627953 shutdown_connections 2024-08-19T12:53:54.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.850+0000 7f621d227640 1 -- 172.21.15.26:0/2453627953 wait complete. 2024-08-19T12:53:54.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:54.851+0000 7f621d227640 1 librados: shutdown 2024-08-19T12:53:54.888 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:54.889 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:53:54.889 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:53:54.955 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2808395994}) 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=665322160}) 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2808395994}) 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276058225}) 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2808395994}) 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1921075371}) 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2808395994}) 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1888866547}) 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.956 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2808395994}) 2024-08-19T12:53:54.957 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=367482883}) 2024-08-19T12:53:54.957 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.957 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2808395994}) 2024-08-19T12:53:54.957 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1530109162}) 2024-08-19T12:53:54.957 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.957 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2808395994}) 2024-08-19T12:53:54.957 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4071080739}) 2024-08-19T12:53:54.957 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:54.957 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2808395994}) 2024-08-19T12:53:54.957 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:54.954+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1800918268}) 2024-08-19T12:53:55.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.003 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628808255}) 2024-08-19T12:53:55.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=789618571}) 2024-08-19T12:53:55.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628808255}) 2024-08-19T12:53:55.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862166794}) 2024-08-19T12:53:55.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628808255}) 2024-08-19T12:53:55.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1415848913}) 2024-08-19T12:53:55.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628808255}) 2024-08-19T12:53:55.004 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=619734722}) 2024-08-19T12:53:55.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628808255}) 2024-08-19T12:53:55.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1882579899}) 2024-08-19T12:53:55.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628808255}) 2024-08-19T12:53:55.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2443806887}) 2024-08-19T12:53:55.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628808255}) 2024-08-19T12:53:55.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2148146670}) 2024-08-19T12:53:55.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628808255}) 2024-08-19T12:53:55.005 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.001+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=745730269}) 2024-08-19T12:53:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.008+0000 7f4c233e2640 10 monclient: get_monmap_and_config 2024-08-19T12:53:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.008+0000 7f4c233e2640 10 monclient: build_initial_monmap 2024-08-19T12:53:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.008+0000 7f4c233e2640 10 monclient: monmap: 2024-08-19T12:53:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:55.009403+0000 2024-08-19T12:53:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:55.009403+0000 2024-08-19T12:53:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:55.010 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:55.010 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:55.010 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:55.010 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:55.010 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:55.010 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:55.010 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.009+0000 7f4c233e2640 1 Processor -- start 2024-08-19T12:53:55.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.009+0000 7f4c233e2640 1 -- start start 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.009+0000 7f4c233e2640 10 monclient: init 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.009+0000 7f4c233e2640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.009+0000 7f4c233e2640 10 monclient: _add_conns ranks=[2,6,3,0,8,5,4,1,7] 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.010+0000 7f4c233e2640 10 monclient(hunting): picked mon.noname-b con 0x7f4c1c073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.010+0000 7f4c233e2640 10 monclient(hunting): picked mon.noname-d con 0x7f4c1c0a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.010+0000 7f4c233e2640 10 monclient(hunting): picked mon.noname-g con 0x7f4c1c13f2e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.010+0000 7f4c233e2640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c1c09dae0 con 0x7f4c1c073470 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.010+0000 7f4c233e2640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c1c136910 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.010+0000 7f4c233e2640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c1c09dd50 con 0x7f4c1c0a57a0 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.010+0000 7f4c233e2640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.010+0000 7f4c233e2640 10 monclient(hunting): authenticate will time out at 1629.097134s 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.010+0000 7f4c2197f640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f4c1c13f2e0 0x7f4c1c1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:52862/0 (socket says 172.21.15.26:52862) 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.010+0000 7f4c2117e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f4c1c073470 0x7f4c1c136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:47946/0 (socket says 172.21.15.26:47946) 2024-08-19T12:53:55.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.010+0000 7f4c2097d640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f4c1c0a57a0 0x7f4c1c0a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:36168/0 (socket says 172.21.15.26:36168) 2024-08-19T12:53:55.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.010+0000 7f4c2197f640 1 -- 172.21.15.26:0/810973113 learned_addr learned my addr 172.21.15.26:0/810973113 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:55.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c2117e640 1 -- 172.21.15.26:0/810973113 >> v1:172.21.15.26:6790/0 conn(0x7f4c1c073470 legacy=0x7f4c1c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c2197f640 1 -- 172.21.15.26:0/810973113 >> v1:172.21.15.104:6790/0 conn(0x7f4c1c13f2e0 legacy=0x7f4c1c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c2097d640 1 -- 172.21.15.26:0/810973113 >> v1:172.21.15.26:6792/0 conn(0x7f4c1c0a57a0 legacy=0x7f4c1c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 820573248 0 0) 0x7f4c1c136910 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient(hunting): my global_id is 194178 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c08003520 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3541052307 0 0) 0x7f4c1c09dd50 con 0x7f4c1c0a57a0 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient(hunting): my global_id is 194217 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c1c136910 con 0x7f4c1c0a57a0 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 79916838 0 0) 0x7f4c1c09dae0 con 0x7f4c1c073470 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient(hunting): my global_id is 27236 2024-08-19T12:53:55.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c1c09dd50 con 0x7f4c1c073470 2024-08-19T12:53:55.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3462686302 0 0) 0x7f4c08003520 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 >> v1:172.21.15.26:6792/0 conn(0x7f4c1c0a57a0 legacy=0x7f4c1c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 >> v1:172.21.15.26:6790/0 conn(0x7f4c1c073470 legacy=0x7f4c1c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient: _finish_hunting 0 2024-08-19T12:53:55.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 1 monclient: found mon.noname-g 2024-08-19T12:53:55.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:55.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:53:55.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1c09e030 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient: _finish_auth 0 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient: _check_auth_tickets 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c14001d90 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c13fff640 10 monclient: dump: 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.019 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c233e2640 5 monclient: authenticate success, global_id 194178 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.012+0000 7f4c233e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4c14002060 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c13fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c13fff640 1 -- 172.21.15.26:0/810973113 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c14004fa0 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c13fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c13fff640 10 monclient: dump: 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.021 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c233e2640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c233e2640 4 set_mon_vals no callback set 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c233e2640 10 monclient: shutdown 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c233e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c233e2640 1 -- 172.21.15.26:0/810973113 >> v1:172.21.15.104:6790/0 conn(0x7f4c1c13f2e0 legacy=0x7f4c1c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c233e2640 1 -- 172.21.15.26:0/810973113 shutdown_connections 2024-08-19T12:53:55.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c233e2640 1 -- 172.21.15.26:0/810973113 >> 172.21.15.26:0/810973113 conn(0x7f4c1c0abff0 msgr2=0x7f4c1c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c233e2640 1 -- 172.21.15.26:0/810973113 shutdown_connections 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.013+0000 7f4c233e2640 1 -- 172.21.15.26:0/810973113 wait complete. 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.014+0000 7f4c233e2640 10 monclient: build_initial_monmap 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.014+0000 7f4c233e2640 10 monclient: monmap: 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:55.023 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.014+0000 7f4c233e2640 1 librados: starting msgr at 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.014+0000 7f4c233e2640 1 librados: starting objecter 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.014+0000 7f4c233e2640 1 Processor -- start 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.014+0000 7f4c233e2640 1 -- start start 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.014+0000 7f4c233e2640 1 librados: setting wanted keys 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.014+0000 7f4c233e2640 1 librados: calling monclient init 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.014+0000 7f4c233e2640 10 monclient: init 2024-08-19T12:53:55.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.015+0000 7f4c233e2640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:55.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.015+0000 7f4c233e2640 10 monclient: _add_conns ranks=[3,0,4,1,7,5,8,6,2] 2024-08-19T12:53:55.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.015+0000 7f4c233e2640 10 monclient(hunting): picked mon.noname-g con 0x7f4c1c073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:53:55.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.015+0000 7f4c233e2640 10 monclient(hunting): picked mon.noname-a con 0x7f4c1c0a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:55.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.015+0000 7f4c233e2640 10 monclient(hunting): picked mon.noname-c con 0x7f4c1c13f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:55.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.015+0000 7f4c233e2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c1c0a3a70 con 0x7f4c1c0a57a0 2024-08-19T12:53:55.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.015+0000 7f4c233e2640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c1c135400 con 0x7f4c1c073470 2024-08-19T12:53:55.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.015+0000 7f4c233e2640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c1c1350b0 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.015+0000 7f4c233e2640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:55.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.015+0000 7f4c233e2640 10 monclient(hunting): authenticate will time out at 1629.102427s 2024-08-19T12:53:55.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.015+0000 7f4c2097d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f4c1c0a57a0 0x7f4c1c0a26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43478/0 (socket says 172.21.15.26:43478) 2024-08-19T12:53:55.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.015+0000 7f4c2097d640 1 -- 172.21.15.26:0/12443085 learned_addr learned my addr 172.21.15.26:0/12443085 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:55.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c2097d640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.26:6789/0 conn(0x7f4c1c0a57a0 legacy=0x7f4c1c0a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c11ffb640 10 client.?.objecter ms_handle_connect 0x7f4c1c0a57a0 2024-08-19T12:53:55.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c11ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:55.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c2197f640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.26:6791/0 conn(0x7f4c1c13f2e0 legacy=0x7f4c1c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c11ffb640 10 client.?.objecter ms_handle_connect 0x7f4c1c13f2e0 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c11ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c2097d640 1 --1- 172.21.15.26:0/12443085 >> v1:172.21.15.26:6789/0 conn(0x7f4c1c0a57a0 0x7f4c1c0a26e0 :-1 s=OPENED pgs=3934 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f4c1c0a3a70 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4028016249 0 0) 0x7f4c1c1350b0 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c11ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c11ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c11ffb640 10 monclient(hunting): my global_id is 184522 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c00004eb0 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c11ffb640 10 client.?.objecter ms_handle_connect 0x7f4c1c073470 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c11ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.017+0000 7f4c2117e640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.104:6790/0 conn(0x7f4c1c073470 legacy=0x7f4c1c0a1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1497457763 0 0) 0x7f4c00004eb0 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c1c1350b0 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c14001db0 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient(hunting): dump: 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.030 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.031 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.031 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.031 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.031 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.031 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.031 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.031 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:55.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2779086480 0 0) 0x7f4c1c135400 con 0x7f4c1c073470 2024-08-19T12:53:55.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient(hunting): my global_id is 194187 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c00004eb0 con 0x7f4c1c073470 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3927471983 0 0) 0x7f4c1c1350b0 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.104:6790/0 conn(0x7f4c1c073470 legacy=0x7f4c1c0a1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.26:6789/0 conn(0x7f4c1c0a57a0 legacy=0x7f4c1c0a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 monclient: found mon.c 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1c13edb0 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient: _finish_auth 0 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:53:55.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c233e2640 5 monclient: authenticate success, global_id 184522 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2894524811 0 0) 0x7f4c1c0a3a70 con 0x7f4c1c0a57a0 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c233e2640 10 monclient: _renew_subs 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c233e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c233e2640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4c1c137210 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4c14005150 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c14005640 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c11ffb640 10 monclient: dump: 2024-08-19T12:53:55.033 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.034 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c137fe640 4 set_mon_vals no callback set 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c233e2640 10 client.184522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c233e2640 10 monclient: _renew_subs 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c233e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c233e2640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f4c1c13e980 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.018+0000 7f4c233e2640 1 librados: init done 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.019+0000 7f4c233e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.019+0000 7f4c233e2640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4c1c0008d0 con 0x7f4c1c142df0 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.019+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f4c14011b60 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.019+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(571..571 src has 350..571) ==== 5857+0+0 (unknown 2999744453 0 0) 0x7f4c1404e480 con 0x7f4c1c13f2e0 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.019+0000 7f4c11ffb640 10 client.184522.objecter ms_dispatch 0x7f4c1c000b90 osd_map(571..571 src has 350..571) 2024-08-19T12:53:55.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.019+0000 7f4c11ffb640 3 client.184522.objecter handle_osd_map got epochs [571,571] > 0 2024-08-19T12:53:55.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.019+0000 7f4c11ffb640 3 client.184522.objecter handle_osd_map decoding full epoch 571 2024-08-19T12:53:55.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.019+0000 7f4c2097d640 10 monclient: get_auth_request con 0x7f4c00040f60 auth_method 0 2024-08-19T12:53:55.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.019+0000 7f4c11ffb640 20 client.184522.objecter dump_active .. 0 homeless 2024-08-19T12:53:55.037 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.037 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482459446}) 2024-08-19T12:53:55.037 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2837550613}) 2024-08-19T12:53:55.037 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.037 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482459446}) 2024-08-19T12:53:55.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3586865015}) 2024-08-19T12:53:55.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482459446}) 2024-08-19T12:53:55.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2665131069}) 2024-08-19T12:53:55.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482459446}) 2024-08-19T12:53:55.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=424511588}) 2024-08-19T12:53:55.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482459446}) 2024-08-19T12:53:55.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=459674259}) 2024-08-19T12:53:55.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.038 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482459446}) 2024-08-19T12:53:55.039 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3019844555}) 2024-08-19T12:53:55.039 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.039 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482459446}) 2024-08-19T12:53:55.039 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=156884816}) 2024-08-19T12:53:55.039 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.039 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482459446}) 2024-08-19T12:53:55.039 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.020+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=321862105}) 2024-08-19T12:53:55.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.021+0000 7f4c2117e640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.104:6790/0 conn(0x7f4c1c142df0 legacy=0x7f4c1c09e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.021+0000 7f4c11ffb640 10 client.184522.objecter ms_handle_connect 0x7f4c1c142df0 2024-08-19T12:53:55.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.021+0000 7f4c11ffb640 10 client.184522.objecter resend_mon_ops 2024-08-19T12:53:55.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.022+0000 7f4c11ffb640 10 client.184522.objecter ms_handle_connect 0x7f4c00040f60 2024-08-19T12:53:55.041 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.041 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738777100}) 2024-08-19T12:53:55.041 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1512684279}) 2024-08-19T12:53:55.041 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.041 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738777100}) 2024-08-19T12:53:55.041 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3372191830}) 2024-08-19T12:53:55.041 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.041 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738777100}) 2024-08-19T12:53:55.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3580226796}) 2024-08-19T12:53:55.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738777100}) 2024-08-19T12:53:55.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3671789536}) 2024-08-19T12:53:55.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738777100}) 2024-08-19T12:53:55.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2199395722}) 2024-08-19T12:53:55.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738777100}) 2024-08-19T12:53:55.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2825172300}) 2024-08-19T12:53:55.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.042 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738777100}) 2024-08-19T12:53:55.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1091281770}) 2024-08-19T12:53:55.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738777100}) 2024-08-19T12:53:55.043 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.025+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=770886648}) 2024-08-19T12:53:55.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2186713390 0 0) 0x7f4c1c0008d0 con 0x7f4c1c142df0 2024-08-19T12:53:55.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 10 monclient(hunting): my global_id is 194196 2024-08-19T12:53:55.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c1c0a3a70 con 0x7f4c1c142df0 2024-08-19T12:53:55.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3640761511 0 0) 0x7f4c1c0a3a70 con 0x7f4c1c142df0 2024-08-19T12:53:55.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c1c0008d0 con 0x7f4c1c142df0 2024-08-19T12:53:55.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c18001c10 con 0x7f4c1c142df0 2024-08-19T12:53:55.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 10 monclient: dump: 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.046 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.047 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.047 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.047 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.047 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.047 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 5782330 0 0) 0x7f4c1c0008d0 con 0x7f4c1c142df0 2024-08-19T12:53:55.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.029+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c1c137630 con 0x7f4c1c142df0 2024-08-19T12:53:55.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.030+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4c18006860 con 0x7f4c1c142df0 2024-08-19T12:53:55.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.030+0000 7f4c11ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:55.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.030+0000 7f4c11ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:55.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.030+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.104:6790/0 conn(0x7f4c1c142df0 legacy=0x7f4c1c09e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.030+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.104:6790/0 conn(0x7f4c1c142df0 legacy=0x7f4c1c09e5f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.036+0000 7f4c233e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.037+0000 7f4c233e2640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4c1c1377e0 con 0x7f4c1c02e1d0 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 10 client.184522.objecter ms_handle_connect 0x7f4c1c02e1d0 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c2117e640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.104:6790/0 conn(0x7f4c1c02e1d0 legacy=0x7f4c1c09ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 10 client.184522.objecter resend_mon_ops 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 658261013 0 0) 0x7f4c1c1377e0 con 0x7f4c1c02e1d0 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 10 monclient(hunting): my global_id is 194205 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c1c0008d0 con 0x7f4c1c02e1d0 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 672032945 0 0) 0x7f4c1c0008d0 con 0x7f4c1c02e1d0 2024-08-19T12:53:55.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4c1c1377e0 con 0x7f4c1c02e1d0 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4c18004fc0 con 0x7f4c1c02e1d0 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 10 monclient: dump: 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.049 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.050 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.050 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.050 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.050 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.050 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.050 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1486914931 0 0) 0x7f4c1c1377e0 con 0x7f4c1c02e1d0 2024-08-19T12:53:55.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.039+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4c1c02e5b0 con 0x7f4c1c02e1d0 2024-08-19T12:53:55.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 3954637637) 0x7f4c180064d0 con 0x7f4c1c02e1d0 2024-08-19T12:53:55.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c11ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:55.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c11ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:55.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.104:6790/0 conn(0x7f4c1c02e1d0 legacy=0x7f4c1c09ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c11ffb640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.104:6790/0 conn(0x7f4c1c02e1d0 legacy=0x7f4c1c09ed00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:55.051 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:55.051 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:55.051 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:55.051 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:55.051 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:55.051 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 19962, 2024-08-19T12:53:55.051 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:55.051 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:55.052 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:55.053 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:55.054 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:55.054 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:55.054 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:55.054 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:55.054 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:55.054 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:55.054 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:55.054 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:55.054 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:55.054 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:55.054 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:55.055 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:55.056 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.057 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:55.058 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:55.059 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.059 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.059 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.059 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:55.059 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.059 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.059 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.059 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:55.059 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:55.059 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.060 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.062 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.063 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.064 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.065 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.066 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:55.067 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.068 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.069 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.070 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:55.071 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:55.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c233e2640 10 librados: watch_flush enter 2024-08-19T12:53:55.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c233e2640 10 librados: watch_flush exit 2024-08-19T12:53:55.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c233e2640 20 client.184522.objecter shutdown clearing up homeless session... 2024-08-19T12:53:55.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c233e2640 10 client.184522.objecter successfully canceled tick 2024-08-19T12:53:55.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c233e2640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.104:6800/899887506 conn(0x7f4c00040f60 legacy=0x7f4c00043420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c233e2640 10 monclient: shutdown 2024-08-19T12:53:55.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c233e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:55.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c233e2640 1 -- 172.21.15.26:0/12443085 >> v1:172.21.15.26:6791/0 conn(0x7f4c1c13f2e0 legacy=0x7f4c1c1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c2197f640 1 -- 172.21.15.26:0/12443085 reap_dead start 2024-08-19T12:53:55.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c2197f640 1 -- 172.21.15.26:0/12443085 reap_dead start 2024-08-19T12:53:55.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c233e2640 1 -- 172.21.15.26:0/12443085 shutdown_connections 2024-08-19T12:53:55.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.040+0000 7f4c233e2640 1 -- 172.21.15.26:0/12443085 >> 172.21.15.26:0/12443085 conn(0x7f4c1c0abff0 msgr2=0x7f4c1c133a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:55.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.041+0000 7f4c233e2640 1 -- 172.21.15.26:0/12443085 shutdown_connections 2024-08-19T12:53:55.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.041+0000 7f4c233e2640 1 -- 172.21.15.26:0/12443085 wait complete. 2024-08-19T12:53:55.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.041+0000 7f4c233e2640 1 librados: shutdown 2024-08-19T12:53:55.073 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:55.074 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:53:55.074 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:53:55.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.089+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.089+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791673261}) 2024-08-19T12:53:55.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3885578950}) 2024-08-19T12:53:55.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791673261}) 2024-08-19T12:53:55.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=157275108}) 2024-08-19T12:53:55.091 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791673261}) 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4281888555}) 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791673261}) 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1438703436}) 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791673261}) 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1119252498}) 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791673261}) 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2789471454}) 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791673261}) 2024-08-19T12:53:55.092 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=120212053}) 2024-08-19T12:53:55.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791673261}) 2024-08-19T12:53:55.093 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3582564162}) 2024-08-19T12:53:55.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.095 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559179529}) 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110134479}) 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559179529}) 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1285395543}) 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559179529}) 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3661765682}) 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559179529}) 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3873808679}) 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559179529}) 2024-08-19T12:53:55.096 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2462054095}) 2024-08-19T12:53:55.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559179529}) 2024-08-19T12:53:55.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=182501256}) 2024-08-19T12:53:55.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559179529}) 2024-08-19T12:53:55.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=379785637}) 2024-08-19T12:53:55.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559179529}) 2024-08-19T12:53:55.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1724852206}) 2024-08-19T12:53:55.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761821428}) 2024-08-19T12:53:55.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2958113057}) 2024-08-19T12:53:55.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761821428}) 2024-08-19T12:53:55.109 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=544304642}) 2024-08-19T12:53:55.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761821428}) 2024-08-19T12:53:55.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=363481774}) 2024-08-19T12:53:55.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761821428}) 2024-08-19T12:53:55.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=744039383}) 2024-08-19T12:53:55.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761821428}) 2024-08-19T12:53:55.112 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3510312121}) 2024-08-19T12:53:55.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761821428}) 2024-08-19T12:53:55.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1560342448}) 2024-08-19T12:53:55.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761821428}) 2024-08-19T12:53:55.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=167492084}) 2024-08-19T12:53:55.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761821428}) 2024-08-19T12:53:55.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2450455395}) 2024-08-19T12:53:55.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365438113}) 2024-08-19T12:53:55.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3904249211}) 2024-08-19T12:53:55.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365438113}) 2024-08-19T12:53:55.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=785916073}) 2024-08-19T12:53:55.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365438113}) 2024-08-19T12:53:55.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1181002926}) 2024-08-19T12:53:55.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365438113}) 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1772950952}) 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365438113}) 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=35283164}) 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365438113}) 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1096848032}) 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365438113}) 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2700674948}) 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365438113}) 2024-08-19T12:53:55.120 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.114+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3331896877}) 2024-08-19T12:53:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.196+0000 7f74e4cb3640 10 monclient: get_monmap_and_config 2024-08-19T12:53:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.196+0000 7f74e4cb3640 10 monclient: build_initial_monmap 2024-08-19T12:53:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.196+0000 7f74e4cb3640 10 monclient: monmap: 2024-08-19T12:53:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:55.197 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:55.197618+0000 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:55.197618+0000 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:55.198 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.197+0000 7f74e4cb3640 1 Processor -- start 2024-08-19T12:53:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.197+0000 7f74e4cb3640 1 -- start start 2024-08-19T12:53:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.197+0000 7f74e4cb3640 10 monclient: init 2024-08-19T12:53:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.197+0000 7f74e4cb3640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.197+0000 7f74e4cb3640 10 monclient: _add_conns ranks=[5,7,1,0,6,4,8,3,2] 2024-08-19T12:53:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.197+0000 7f74e4cb3640 10 monclient(hunting): picked mon.noname-h con 0x7f74e006e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:53:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.197+0000 7f74e4cb3640 10 monclient(hunting): picked mon.noname-i con 0x7f74e00a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:53:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.197+0000 7f74e4cb3640 10 monclient(hunting): picked mon.noname-f con 0x7f74e00a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.197+0000 7f74e4cb3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74e0136790 con 0x7f74e00a5d50 2024-08-19T12:53:55.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.198+0000 7f74e4cb3640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74e009ddf0 con 0x7f74e006e7b0 2024-08-19T12:53:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.198+0000 7f74e4cb3640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74e009e0d0 con 0x7f74e00a5920 2024-08-19T12:53:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.198+0000 7f74e4cb3640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.198+0000 7f74e4cb3640 10 monclient(hunting): authenticate will time out at 1629.285038s 2024-08-19T12:53:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.198+0000 7f74ded9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f74e00a5d50 0x7f74e0141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:51508/0 (socket says 172.21.15.26:51508) 2024-08-19T12:53:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.198+0000 7f74ddd9b640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f74e00a5920 0x7f74e013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:43588/0 (socket says 172.21.15.26:43588) 2024-08-19T12:53:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.198+0000 7f74ded9d640 1 -- 172.21.15.26:0/1041897731 learned_addr learned my addr 172.21.15.26:0/1041897731 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74de59c640 1 -- 172.21.15.26:0/1041897731 >> v1:172.21.15.104:6791/0 conn(0x7f74e006e7b0 legacy=0x7f74e0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74ddd9b640 1 -- 172.21.15.26:0/1041897731 >> v1:172.21.15.104:6792/0 conn(0x7f74e00a5920 legacy=0x7f74e013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74ded9d640 1 -- 172.21.15.26:0/1041897731 >> v1:172.21.15.104:6789/0 conn(0x7f74e00a5d50 legacy=0x7f74e0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1064238336 0 0) 0x7f74d00014a0 con 0x7f74e006e7b0 2024-08-19T12:53:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 10 monclient(hunting): my global_id is 184433 2024-08-19T12:53:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74c8003520 con 0x7f74e006e7b0 2024-08-19T12:53:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1405388136 0 0) 0x7f74e009e0d0 con 0x7f74e00a5920 2024-08-19T12:53:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 10 monclient(hunting): my global_id is 136186 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74d00014a0 con 0x7f74e00a5920 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2050024136 0 0) 0x7f74e0136790 con 0x7f74e00a5d50 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 10 monclient(hunting): my global_id is 184492 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.200+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74e009e0d0 con 0x7f74e00a5d50 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 711283354 0 0) 0x7f74c8003520 con 0x7f74e006e7b0 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 >> v1:172.21.15.104:6792/0 conn(0x7f74e00a5920 legacy=0x7f74e013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 >> v1:172.21.15.104:6789/0 conn(0x7f74e00a5d50 legacy=0x7f74e0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 10 monclient: _finish_hunting 0 2024-08-19T12:53:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 1 monclient: found mon.noname-h 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e009e3b0 con 0x7f74e006e7b0 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 10 monclient: _finish_auth 0 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 10 monclient: _check_auth_tickets 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f74d0003610 con 0x7f74e006e7b0 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 10 monclient: dump: 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.208 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f74d8001e00 con 0x7f74e00a5d50 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74e4cb3640 5 monclient: authenticate success, global_id 184433 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74e4cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f74d0003830 con 0x7f74e006e7b0 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 1 -- 172.21.15.26:0/1041897731 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f74d0003de0 con 0x7f74e006e7b0 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74e4cb3640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74e4cb3640 4 set_mon_vals no callback set 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74dd59a640 10 monclient: dump: 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.210 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.211 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.212 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74e4cb3640 10 monclient: shutdown 2024-08-19T12:53:55.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74e4cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:55.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74e4cb3640 1 -- 172.21.15.26:0/1041897731 >> v1:172.21.15.104:6791/0 conn(0x7f74e006e7b0 legacy=0x7f74e0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74e4cb3640 1 -- 172.21.15.26:0/1041897731 shutdown_connections 2024-08-19T12:53:55.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74e4cb3640 1 -- 172.21.15.26:0/1041897731 >> 172.21.15.26:0/1041897731 conn(0x7f74e00a8740 msgr2=0x7f74e00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:55.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74e4cb3640 1 -- 172.21.15.26:0/1041897731 shutdown_connections 2024-08-19T12:53:55.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.201+0000 7f74e4cb3640 1 -- 172.21.15.26:0/1041897731 wait complete. 2024-08-19T12:53:55.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.202+0000 7f74e4cb3640 10 monclient: build_initial_monmap 2024-08-19T12:53:55.212 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.202+0000 7f74e4cb3640 10 monclient: monmap: 2024-08-19T12:53:55.212 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:55.212 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:55.213 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.202+0000 7f74e4cb3640 1 librados: starting msgr at 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.202+0000 7f74e4cb3640 1 librados: starting objecter 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.202+0000 7f74e4cb3640 1 Processor -- start 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.202+0000 7f74e4cb3640 1 -- start start 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.202+0000 7f74e4cb3640 1 librados: setting wanted keys 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.202+0000 7f74e4cb3640 1 librados: calling monclient init 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.202+0000 7f74e4cb3640 10 monclient: init 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.203+0000 7f74e4cb3640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.203+0000 7f74e4cb3640 10 monclient: _add_conns ranks=[4,3,1,2,5,6,8,0,7] 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.203+0000 7f74e4cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f74e006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.203+0000 7f74e4cb3640 10 monclient(hunting): picked mon.noname-g con 0x7f74e00a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:53:55.214 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.203+0000 7f74e4cb3640 10 monclient(hunting): picked mon.noname-f con 0x7f74e00a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:55.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.203+0000 7f74e4cb3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74e013aaf0 con 0x7f74e00a5d50 2024-08-19T12:53:55.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.203+0000 7f74e4cb3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74e0142e30 con 0x7f74e00a5920 2024-08-19T12:53:55.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.203+0000 7f74e4cb3640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74e0139030 con 0x7f74e006e7b0 2024-08-19T12:53:55.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.203+0000 7f74e4cb3640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:55.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.203+0000 7f74e4cb3640 10 monclient(hunting): authenticate will time out at 1629.290403s 2024-08-19T12:53:55.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.203+0000 7f74de59c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f74e006e7b0 0x7f74e0137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59334/0 (socket says 172.21.15.26:59334) 2024-08-19T12:53:55.215 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.203+0000 7f74de59c640 1 -- 172.21.15.26:0/987226363 learned_addr learned my addr 172.21.15.26:0/987226363 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:55.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74de59c640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.26:6791/0 conn(0x7f74e006e7b0 legacy=0x7f74e0137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 client.?.objecter ms_handle_connect 0x7f74e006e7b0 2024-08-19T12:53:55.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:55.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 client.?.objecter ms_handle_connect 0x7f74e00a5d50 2024-08-19T12:53:55.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:55.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ded9d640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.104:6789/0 conn(0x7f74e00a5d50 legacy=0x7f74e01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 client.?.objecter ms_handle_connect 0x7f74e00a5920 2024-08-19T12:53:55.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:55.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ddd9b640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.104:6790/0 conn(0x7f74e00a5920 legacy=0x7f74e0137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1610817982 0 0) 0x7f74e0139030 con 0x7f74e006e7b0 2024-08-19T12:53:55.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 monclient(hunting): my global_id is 184531 2024-08-19T12:53:55.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74c0004eb0 con 0x7f74e006e7b0 2024-08-19T12:53:55.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3894519521 0 0) 0x7f74d8002090 con 0x7f74e00a5d50 2024-08-19T12:53:55.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 monclient(hunting): my global_id is 184501 2024-08-19T12:53:55.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74e0139030 con 0x7f74e00a5d50 2024-08-19T12:53:55.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3798024829 0 0) 0x7f74e0142e30 con 0x7f74e00a5920 2024-08-19T12:53:55.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 10 monclient(hunting): my global_id is 194214 2024-08-19T12:53:55.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.205+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74d8002090 con 0x7f74e00a5920 2024-08-19T12:53:55.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 248740989 0 0) 0x7f74e0139030 con 0x7f74e00a5d50 2024-08-19T12:53:55.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74e0142e30 con 0x7f74e00a5d50 2024-08-19T12:53:55.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f74d80016a0 con 0x7f74e00a5d50 2024-08-19T12:53:55.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:55.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:53:55.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient(hunting): dump: 2024-08-19T12:53:55.224 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.224 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.225 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.225 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.225 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.225 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.225 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.225 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.225 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.225 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.225 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.225 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.225 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.225 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2700004905 0 0) 0x7f74c0004eb0 con 0x7f74e006e7b0 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74e0139030 con 0x7f74e006e7b0 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2178757040 0 0) 0x7f74d8002090 con 0x7f74e00a5920 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74c0004eb0 con 0x7f74e00a5920 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f74d00040a0 con 0x7f74e006e7b0 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient(hunting): dump: 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.226 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.227 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f74d40047e0 con 0x7f74e00a5920 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient(hunting): dump: 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.228 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1867334849 0 0) 0x7f74e0142e30 con 0x7f74e00a5d50 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.26:6791/0 conn(0x7f74e006e7b0 legacy=0x7f74e0137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.104:6790/0 conn(0x7f74e00a5920 legacy=0x7f74e0137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient: _finish_hunting 0 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 monclient: found mon.f 2024-08-19T12:53:55.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e01384a0 con 0x7f74e00a5d50 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient: _finish_auth 0 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient: _check_auth_tickets 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74e4cb3640 5 monclient: authenticate success, global_id 184501 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74e4cb3640 10 monclient: _renew_subs 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74e4cb3640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74e4cb3640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f74e013b390 con 0x7f74e00a5d50 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74e4cb3640 10 client.184501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74e4cb3640 10 monclient: _renew_subs 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74e4cb3640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:53:55.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74e4cb3640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f74e0142a70 con 0x7f74e00a5d50 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74e4cb3640 1 librados: init done 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f74d8005150 con 0x7f74e00a5d50 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f74d8001ce0 con 0x7f74e00a5d50 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74dcd99640 4 set_mon_vals no callback set 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.206+0000 7f74ceffd640 10 monclient: dump: 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.207+0000 7f74e4cb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.207+0000 7f74e4cb3640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f74c40065c0 con 0x7f74c4002860 2024-08-19T12:53:55.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.207+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f74d8011a00 con 0x7f74e00a5d50 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.207+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(573..573 src has 350..573) ==== 5969+0+0 (unknown 1669907522 0 0) 0x7f74d804e790 con 0x7f74e00a5d50 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.207+0000 7f74ceffd640 10 client.184501.objecter ms_dispatch 0x7f74e0000b90 osd_map(573..573 src has 350..573) 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.207+0000 7f74ceffd640 3 client.184501.objecter handle_osd_map got epochs [573,573] > 0 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.207+0000 7f74ceffd640 3 client.184501.objecter handle_osd_map decoding full epoch 573 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.207+0000 7f74ceffd640 20 client.184501.objecter dump_active .. 0 homeless 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.207+0000 7f74ddd9b640 10 monclient: get_auth_request con 0x7f74c0041a40 auth_method 0 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74de59c640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.104:6791/0 conn(0x7f74c4002860 legacy=0x7f74c4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 10 client.184501.objecter ms_handle_connect 0x7f74c4002860 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 10 client.184501.objecter resend_mon_ops 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 315939586 0 0) 0x7f74c40065c0 con 0x7f74c4002860 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 10 monclient(hunting): my global_id is 184442 2024-08-19T12:53:55.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74e0142e30 con 0x7f74c4002860 2024-08-19T12:53:55.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3758188429 0 0) 0x7f74e0142e30 con 0x7f74c4002860 2024-08-19T12:53:55.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74c40065c0 con 0x7f74c4002860 2024-08-19T12:53:55.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f74d00039f0 con 0x7f74c4002860 2024-08-19T12:53:55.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:53:55.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 10 monclient: dump: 2024-08-19T12:53:55.234 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.234 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.234 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.234 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.234 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 446403820 0 0) 0x7f74c40065c0 con 0x7f74c4002860 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f74c400a160 con 0x7f74c4002860 2024-08-19T12:53:55.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.209+0000 7f74ceffd640 10 client.184501.objecter ms_handle_connect 0x7f74c0041a40 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.210+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f74d0003c10 con 0x7f74c4002860 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.210+0000 7f74ceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.210+0000 7f74ceffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.210+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.104:6791/0 conn(0x7f74c4002860 legacy=0x7f74c4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.210+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.104:6791/0 conn(0x7f74c4002860 legacy=0x7f74c4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.216+0000 7f74e4cb3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.216+0000 7f74e4cb3640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f74c4009e50 con 0x7f74c4009110 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.218+0000 7f74de59c640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.104:6791/0 conn(0x7f74c4009110 legacy=0x7f74c40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.218+0000 7f74ceffd640 10 client.184501.objecter ms_handle_connect 0x7f74c4009110 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.218+0000 7f74ceffd640 10 client.184501.objecter resend_mon_ops 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.218+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 720500221 0 0) 0x7f74c4009e50 con 0x7f74c4009110 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.218+0000 7f74ceffd640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.218+0000 7f74ceffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.218+0000 7f74ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.218+0000 7f74ceffd640 10 monclient(hunting): my global_id is 184451 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.218+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74c40065c0 con 0x7f74c4009110 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1509541892 0 0) 0x7f74c40065c0 con 0x7f74c4009110 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74c4009e50 con 0x7f74c4009110 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f74d00039f0 con 0x7f74c4009110 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 10 monclient: dump: 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.237 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3994266031 0 0) 0x7f74c4009e50 con 0x7f74c4009110 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f74c400c3f0 con 0x7f74c4009110 2024-08-19T12:53:55.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 3367437743) 0x7f74d0005de0 con 0x7f74c4009110 2024-08-19T12:53:55.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:55.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:55.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.104:6791/0 conn(0x7f74c4009110 legacy=0x7f74c40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74ceffd640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.104:6791/0 conn(0x7f74c4009110 legacy=0x7f74c40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:55.239 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:55.239 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 160689, 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:53:55.240 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:55.241 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:55.242 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:55.243 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:55.244 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.247 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.249 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:55.250 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:55.251 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:55.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.253 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:55.256 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.257 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:53:55.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.259 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:55.260 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:55.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74e4cb3640 10 librados: watch_flush enter 2024-08-19T12:53:55.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74e4cb3640 10 librados: watch_flush exit 2024-08-19T12:53:55.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74e4cb3640 20 client.184501.objecter shutdown clearing up homeless session... 2024-08-19T12:53:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.219+0000 7f74e4cb3640 10 client.184501.objecter successfully canceled tick 2024-08-19T12:53:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.220+0000 7f74e4cb3640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.104:6800/899887506 conn(0x7f74c0041a40 legacy=0x7f74c0043f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.220+0000 7f74e4cb3640 10 monclient: shutdown 2024-08-19T12:53:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.220+0000 7f74e4cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.220+0000 7f74e4cb3640 1 -- 172.21.15.26:0/987226363 >> v1:172.21.15.104:6789/0 conn(0x7f74e00a5d50 legacy=0x7f74e01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.220+0000 7f74ded9d640 1 -- 172.21.15.26:0/987226363 reap_dead start 2024-08-19T12:53:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.220+0000 7f74ded9d640 1 -- 172.21.15.26:0/987226363 reap_dead start 2024-08-19T12:53:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.220+0000 7f74e4cb3640 1 -- 172.21.15.26:0/987226363 shutdown_connections 2024-08-19T12:53:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.220+0000 7f74e4cb3640 1 -- 172.21.15.26:0/987226363 >> 172.21.15.26:0/987226363 conn(0x7f74e00a8740 msgr2=0x7f74e00aa580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.220+0000 7f74e4cb3640 1 -- 172.21.15.26:0/987226363 shutdown_connections 2024-08-19T12:53:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.220+0000 7f74e4cb3640 1 -- 172.21.15.26:0/987226363 wait complete. 2024-08-19T12:53:55.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.220+0000 7f74e4cb3640 1 librados: shutdown 2024-08-19T12:53:55.263 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:55.263 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:53:55.263 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:53:55.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241672692}) 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4204704657}) 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241672692}) 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753966667}) 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241672692}) 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3126450740}) 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241672692}) 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3245175164}) 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241672692}) 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3184023058}) 2024-08-19T12:53:55.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241672692}) 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1200655333}) 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241672692}) 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3062533944}) 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241672692}) 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.337+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4285483411}) 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000991627}) 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2456033900}) 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000991627}) 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2225091126}) 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000991627}) 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3315179592}) 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000991627}) 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3845717232}) 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000991627}) 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2495365994}) 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000991627}) 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3311552334}) 2024-08-19T12:53:55.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.338+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000991627}) 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2583899432}) 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000991627}) 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.339+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3926517486}) 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648633140}) 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2114168840}) 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648633140}) 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420212291}) 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648633140}) 2024-08-19T12:53:55.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4154673404}) 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648633140}) 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3090742365}) 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648633140}) 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3034619239}) 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648633140}) 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1311400754}) 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648633140}) 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1795914014}) 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648633140}) 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.340+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4032861948}) 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.341+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.341+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290221839}) 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.341+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397818296}) 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.341+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.341+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290221839}) 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.341+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202512322}) 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.341+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.341+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290221839}) 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1483003060}) 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290221839}) 2024-08-19T12:53:55.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2950339070}) 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290221839}) 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1507428105}) 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290221839}) 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1784716985}) 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290221839}) 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1805780812}) 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290221839}) 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.342+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3866214732}) 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272038979}) 2024-08-19T12:53:55.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3664247991}) 2024-08-19T12:53:55.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272038979}) 2024-08-19T12:53:55.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1604800640}) 2024-08-19T12:53:55.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272038979}) 2024-08-19T12:53:55.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3118835164}) 2024-08-19T12:53:55.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272038979}) 2024-08-19T12:53:55.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=296152316}) 2024-08-19T12:53:55.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272038979}) 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1041387220}) 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272038979}) 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3836910541}) 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272038979}) 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2554631312}) 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272038979}) 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.343+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2462126449}) 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2312802916}) 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3819254362}) 2024-08-19T12:53:55.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2312802916}) 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626412920}) 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2312802916}) 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1430951899}) 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2312802916}) 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3752193230}) 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2312802916}) 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2923982156}) 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2312802916}) 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1106109790}) 2024-08-19T12:53:55.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2312802916}) 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2638410463}) 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2312802916}) 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.344+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1189987337}) 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.345+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748247637}) 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2314576280}) 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748247637}) 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442363912}) 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748247637}) 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3222070144}) 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748247637}) 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4070024023}) 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748247637}) 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3095074216}) 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748247637}) 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=914417229}) 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748247637}) 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3125962157}) 2024-08-19T12:53:55.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748247637}) 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.346+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1506404251}) 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572378474}) 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1906664980}) 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572378474}) 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526407680}) 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572378474}) 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1691830682}) 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572378474}) 2024-08-19T12:53:55.363 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3780486146}) 2024-08-19T12:53:55.364 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.364 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572378474}) 2024-08-19T12:53:55.364 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1786392691}) 2024-08-19T12:53:55.364 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.364 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572378474}) 2024-08-19T12:53:55.364 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3149880882}) 2024-08-19T12:53:55.364 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.364 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572378474}) 2024-08-19T12:53:55.364 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2952952862}) 2024-08-19T12:53:55.364 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.364 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572378474}) 2024-08-19T12:53:55.364 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.347+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4246904867}) 2024-08-19T12:53:55.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.404+0000 7fd7155b8640 10 monclient: get_monmap_and_config 2024-08-19T12:53:55.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.404+0000 7fd7155b8640 10 monclient: build_initial_monmap 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.404+0000 7fd7155b8640 10 monclient: monmap: 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:53:55.405783+0000 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:53:55.405783+0000 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:55.406 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:55.407 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:55.407 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:55.407 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:55.407 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.405+0000 7fd7155b8640 1 Processor -- start 2024-08-19T12:53:55.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.405+0000 7fd7155b8640 1 -- start start 2024-08-19T12:53:55.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.405+0000 7fd7155b8640 10 monclient: init 2024-08-19T12:53:55.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd7155b8640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:55.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd7155b8640 10 monclient: _add_conns ranks=[4,8,1,0,2,5,7,3,6] 2024-08-19T12:53:55.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd7155b8640 10 monclient(hunting): picked mon.noname-c con 0x7fd71006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:53:55.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd7155b8640 10 monclient(hunting): picked mon.noname-e con 0x7fd7100a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:53:55.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd7155b8640 10 monclient(hunting): picked mon.noname-f con 0x7fd7100a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd7155b8640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd710136790 con 0x7fd7100a5d50 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd7155b8640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71009ddf0 con 0x7fd71006e7b0 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd7155b8640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71009e0d0 con 0x7fd7100a5920 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd7155b8640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd7155b8640 10 monclient(hunting): authenticate will time out at 1629.493242s 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd70effd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fd71006e7b0 0x7fd710136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59350/0 (socket says 172.21.15.26:59350) 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd70e7fc640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fd7100a5920 0x7fd71013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40768/0 (socket says 172.21.15.26:40768) 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.406+0000 7fd70effd640 1 -- 172.21.15.26:0/3991695039 learned_addr learned my addr 172.21.15.26:0/3991695039 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70e7fc640 1 -- 172.21.15.26:0/3991695039 >> v1:172.21.15.26:6793/0 conn(0x7fd7100a5920 legacy=0x7fd71013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70effd640 1 -- 172.21.15.26:0/3991695039 >> v1:172.21.15.26:6791/0 conn(0x7fd71006e7b0 legacy=0x7fd710136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70f7fe640 1 -- 172.21.15.26:0/3991695039 >> v1:172.21.15.104:6789/0 conn(0x7fd7100a5d50 legacy=0x7fd710141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3860896351 0 0) 0x7fd71009e0d0 con 0x7fd7100a5920 2024-08-19T12:53:55.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient(hunting): my global_id is 184409 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6fc003520 con 0x7fd7100a5920 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3783742732 0 0) 0x7fd7080017d0 con 0x7fd71006e7b0 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient(hunting): my global_id is 184540 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd71009e0d0 con 0x7fd71006e7b0 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2664089049 0 0) 0x7fd710136790 con 0x7fd7100a5d50 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient(hunting): my global_id is 184510 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7080017d0 con 0x7fd7100a5d50 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1794441942 0 0) 0x7fd6fc003520 con 0x7fd7100a5920 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70f7fe640 1 --1- 172.21.15.26:0/3991695039 >> v1:172.21.15.104:6789/0 conn(0x7fd7100a5d50 0x7fd710141a20 :-1 s=OPENED pgs=3737 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fd700001ad0 mon_map magic: 0 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 >> v1:172.21.15.26:6791/0 conn(0x7fd71006e7b0 legacy=0x7fd710136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 >> v1:172.21.15.104:6789/0 conn(0x7fd7100a5d50 legacy=0x7fd710141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 monclient: found mon.noname-e 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71009e3b0 con 0x7fd7100a5920 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient: _finish_auth 0 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:53:55.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd6f8001ae0 con 0x7fd7100a5920 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd7155b8640 5 monclient: authenticate success, global_id 184409 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd7155b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient: dump: 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.416 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd7155b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd700001ad0 con 0x7fd7100a5d50 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.408+0000 7fd70dffb640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd6f80047e0 con 0x7fd7100a5920 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd70dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd70dffb640 1 -- 172.21.15.26:0/3991695039 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd6f8004d50 con 0x7fd7100a5920 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd70dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd70dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd70dffb640 10 monclient: dump: 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.418 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd7155b8640 10 monclient: get_monmap_and_config success 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd7155b8640 4 set_mon_vals no callback set 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd7155b8640 10 monclient: shutdown 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd7155b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd7155b8640 1 -- 172.21.15.26:0/3991695039 >> v1:172.21.15.26:6793/0 conn(0x7fd7100a5920 legacy=0x7fd71013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd7155b8640 1 -- 172.21.15.26:0/3991695039 shutdown_connections 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd7155b8640 1 -- 172.21.15.26:0/3991695039 >> 172.21.15.26:0/3991695039 conn(0x7fd7100a8740 msgr2=0x7fd7100a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:55.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd7155b8640 1 -- 172.21.15.26:0/3991695039 shutdown_connections 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.409+0000 7fd7155b8640 1 -- 172.21.15.26:0/3991695039 wait complete. 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.410+0000 7fd7155b8640 10 monclient: build_initial_monmap 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.410+0000 7fd7155b8640 10 monclient: monmap: 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:53:55.420 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.410+0000 7fd7155b8640 1 librados: starting msgr at 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.410+0000 7fd7155b8640 1 librados: starting objecter 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.410+0000 7fd7155b8640 1 Processor -- start 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.410+0000 7fd7155b8640 1 -- start start 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.410+0000 7fd7155b8640 1 librados: setting wanted keys 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.410+0000 7fd7155b8640 1 librados: calling monclient init 2024-08-19T12:53:55.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.410+0000 7fd7155b8640 10 monclient: init 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd7155b8640 10 monclient: _reopen_session rank -1 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd7155b8640 10 monclient: _add_conns ranks=[5,0,6,1,3,2,4,7,8] 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd7155b8640 10 monclient(hunting): picked mon.noname-h con 0x7fd71006e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd7155b8640 10 monclient(hunting): picked mon.noname-a con 0x7fd7100a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd7155b8640 10 monclient(hunting): picked mon.noname-d con 0x7fd7100a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd7155b8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd710142ec0 con 0x7fd7100a5920 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd7155b8640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd710138fc0 con 0x7fd71006e7b0 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd7155b8640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71013aa80 con 0x7fd7100a5d50 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd7155b8640 10 monclient(hunting): _renew_subs 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd7155b8640 10 monclient(hunting): authenticate will time out at 1629.498131s 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd70e7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd7100a5920 0x7fd7101d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:43490/0 (socket says 172.21.15.26:43490) 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd70f7fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fd7100a5d50 0x7fd7101d6bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:36176/0 (socket says 172.21.15.26:36176) 2024-08-19T12:53:55.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.411+0000 7fd70e7fc640 1 -- 172.21.15.26:0/4126581742 learned_addr learned my addr 172.21.15.26:0/4126581742 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:53:55.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd70e7fc640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.26:6789/0 conn(0x7fd7100a5920 legacy=0x7fd7101d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd70f7fe640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.26:6792/0 conn(0x7fd7100a5d50 legacy=0x7fd7101d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 client.?.objecter ms_handle_connect 0x7fd7100a5920 2024-08-19T12:53:55.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:55.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 client.?.objecter ms_handle_connect 0x7fd7100a5d50 2024-08-19T12:53:55.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:55.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd70effd640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.104:6791/0 conn(0x7fd71006e7b0 legacy=0x7fd7101ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 client.?.objecter ms_handle_connect 0x7fd71006e7b0 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 400478834 0 0) 0x7fd710142ec0 con 0x7fd7100a5920 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): my global_id is 194274 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6f0004eb0 con 0x7fd7100a5920 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 98394143 0 0) 0x7fd71013aa80 con 0x7fd7100a5d50 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): my global_id is 194226 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd710142ec0 con 0x7fd7100a5d50 2024-08-19T12:53:55.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2028191518 0 0) 0x7fd710138fc0 con 0x7fd71006e7b0 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): my global_id is 184460 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd71013aa80 con 0x7fd71006e7b0 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3820267206 0 0) 0x7fd710142ec0 con 0x7fd7100a5d50 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd710138fc0 con 0x7fd7100a5d50 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd700003950 con 0x7fd7100a5d50 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): dump: 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.430 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.431 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3686754873 0 0) 0x7fd6f0004eb0 con 0x7fd7100a5920 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd710142ec0 con 0x7fd7100a5920 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd6f8004cd0 con 0x7fd7100a5920 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 10 monclient(hunting): dump: 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.432 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.413+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3863264738 0 0) 0x7fd710138fc0 con 0x7fd7100a5d50 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.104:6791/0 conn(0x7fd71006e7b0 legacy=0x7fd7101ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.26:6789/0 conn(0x7fd7100a5920 legacy=0x7fd7101d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 10 monclient: _finish_hunting 0 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 1 monclient: found mon.d 2024-08-19T12:53:55.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71013ad60 con 0x7fd7100a5d50 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 10 monclient: _finish_auth 0 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 10 monclient: _check_auth_tickets 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7155b8640 5 monclient: authenticate success, global_id 194226 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7155b8640 10 monclient: _renew_subs 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7155b8640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7155b8640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd71013a650 con 0x7fd7100a5d50 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7155b8640 10 client.194226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7155b8640 10 monclient: _renew_subs 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7155b8640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:53:55.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7155b8640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fd7101386b0 con 0x7fd7100a5d50 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7155b8640 1 librados: init done 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd700003af0 con 0x7fd7100a5d50 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd700004020 con 0x7fd7100a5d50 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd70d7fa640 4 set_mon_vals no callback set 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7077fe640 10 monclient: dump: 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.435 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7155b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.414+0000 7fd7155b8640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd6f40065c0 con 0x7fd6f4002860 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.415+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd7000042c0 con 0x7fd7100a5d50 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.415+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(573..573 src has 350..573) ==== 5969+0+0 (unknown 1669907522 0 0) 0x7fd70004e170 con 0x7fd7100a5d50 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.415+0000 7fd7077fe640 10 client.194226.objecter ms_dispatch 0x7fd710000b90 osd_map(573..573 src has 350..573) 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.415+0000 7fd7077fe640 3 client.194226.objecter handle_osd_map got epochs [573,573] > 0 2024-08-19T12:53:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.415+0000 7fd7077fe640 3 client.194226.objecter handle_osd_map decoding full epoch 573 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.415+0000 7fd7077fe640 20 client.194226.objecter dump_active .. 0 homeless 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.415+0000 7fd70e7fc640 10 monclient: get_auth_request con 0x7fd6f0041500 auth_method 0 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd70effd640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.104:6792/0 conn(0x7fd6f4002860 legacy=0x7fd6f4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 10 client.194226.objecter ms_handle_connect 0x7fd6f4002860 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 10 client.194226.objecter resend_mon_ops 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2233714706 0 0) 0x7fd6f40065c0 con 0x7fd6f4002860 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 10 monclient(hunting): my global_id is 136195 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd710138fc0 con 0x7fd6f4002860 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2992522399 0 0) 0x7fd710138fc0 con 0x7fd6f4002860 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6f40065c0 con 0x7fd6f4002860 2024-08-19T12:53:55.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd708005270 con 0x7fd6f4002860 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 10 monclient: dump: 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 10 client.194226.objecter ms_handle_connect 0x7fd6f0041500 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1830854209 0 0) 0x7fd6f40065c0 con 0x7fd6f4002860 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.417+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6f400a160 con 0x7fd6f4002860 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.418+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd7080075b0 con 0x7fd6f4002860 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.418+0000 7fd7077fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.418+0000 7fd7077fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.418+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.104:6792/0 conn(0x7fd6f4002860 legacy=0x7fd6f4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.418+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.104:6792/0 conn(0x7fd6f4002860 legacy=0x7fd6f4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.424+0000 7fd7155b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.424+0000 7fd7155b8640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd6f4009e70 con 0x7fd6f4009130 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.426+0000 7fd70effd640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.104:6792/0 conn(0x7fd6f4009130 legacy=0x7fd6f40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.426+0000 7fd7077fe640 10 client.194226.objecter ms_handle_connect 0x7fd6f4009130 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.426+0000 7fd7077fe640 10 client.194226.objecter resend_mon_ops 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.426+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1698926003 0 0) 0x7fd6f4009e70 con 0x7fd6f4009130 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.426+0000 7fd7077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.426+0000 7fd7077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.426+0000 7fd7077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.426+0000 7fd7077fe640 10 monclient(hunting): my global_id is 136204 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.426+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6f40065c0 con 0x7fd6f4009130 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2500297625 0 0) 0x7fd6f40065c0 con 0x7fd6f4009130 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6f4009e70 con 0x7fd6f4009130 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd708005c90 con 0x7fd6f4009130 2024-08-19T12:53:55.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 10 monclient: dump: 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:53:55.441 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:53:55.442 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:53:55.442 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:53:55.442 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:53:55.442 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:53:55.442 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:53:55.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1714956463 0 0) 0x7fd6f4009e70 con 0x7fd6f4009130 2024-08-19T12:53:55.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6f400c410 con 0x7fd6f4009130 2024-08-19T12:53:55.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7545 (unknown 0 0 1045228569) 0x7fd7080070c0 con 0x7fd6f4009130 2024-08-19T12:53:55.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:53:55.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:53:55.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.104:6792/0 conn(0x7fd6f4009130 legacy=0x7fd6f40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7077fe640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.104:6792/0 conn(0x7fd6f4009130 legacy=0x7fd6f40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:53:55.443 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:53:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:53:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 110, 2024-08-19T12:53:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 805785, 2024-08-19T12:53:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:53:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:53:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:53:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:53:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:55.444 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:53:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:53:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:53:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:53:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:53:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:53:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:53:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:53:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:53:55.461 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:53:55.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7155b8640 10 librados: watch_flush enter 2024-08-19T12:53:55.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7155b8640 10 librados: watch_flush exit 2024-08-19T12:53:55.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7155b8640 20 client.194226.objecter shutdown clearing up homeless session... 2024-08-19T12:53:55.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7155b8640 10 client.194226.objecter successfully canceled tick 2024-08-19T12:53:55.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7155b8640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.104:6800/899887506 conn(0x7fd6f0041500 legacy=0x7fd6f00439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7155b8640 10 monclient: shutdown 2024-08-19T12:53:55.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7155b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:53:55.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7155b8640 1 -- 172.21.15.26:0/4126581742 >> v1:172.21.15.26:6792/0 conn(0x7fd7100a5d50 legacy=0x7fd7101d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:53:55.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd70f7fe640 1 -- 172.21.15.26:0/4126581742 reap_dead start 2024-08-19T12:53:55.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd70f7fe640 1 -- 172.21.15.26:0/4126581742 reap_dead start 2024-08-19T12:53:55.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7155b8640 1 -- 172.21.15.26:0/4126581742 shutdown_connections 2024-08-19T12:53:55.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.427+0000 7fd7155b8640 1 -- 172.21.15.26:0/4126581742 >> 172.21.15.26:0/4126581742 conn(0x7fd7100a8740 msgr2=0x7fd7100ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:53:55.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.428+0000 7fd7155b8640 1 -- 172.21.15.26:0/4126581742 shutdown_connections 2024-08-19T12:53:55.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.428+0000 7fd7155b8640 1 -- 172.21.15.26:0/4126581742 wait complete. 2024-08-19T12:53:55.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:53:55.428+0000 7fd7155b8640 1 librados: shutdown 2024-08-19T12:53:55.465 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:53:55.465 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['f'] 2024-08-19T12:53:55.465 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['e'] 2024-08-19T12:53:55.465 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f 2024-08-19T12:53:55.465 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-08-19T12:53:55.465 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-08-19T12:53:55.465 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:53:55.479 INFO:tasks.ceph.mon.f:Stopped 2024-08-19T12:53:55.479 INFO:tasks.mon_thrash:Sending STOP to mon e 2024-08-19T12:53:55.479 INFO:tasks.ceph.mon.e:Sent signal 19 2024-08-19T12:53:55.479 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:53:55.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417903645}) 2024-08-19T12:53:55.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3581335563}) 2024-08-19T12:53:55.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417903645}) 2024-08-19T12:53:55.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2819336078}) 2024-08-19T12:53:55.964 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417903645}) 2024-08-19T12:53:55.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=763065588}) 2024-08-19T12:53:55.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417903645}) 2024-08-19T12:53:55.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3186070677}) 2024-08-19T12:53:55.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417903645}) 2024-08-19T12:53:55.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2545788145}) 2024-08-19T12:53:55.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417903645}) 2024-08-19T12:53:55.965 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1916919108}) 2024-08-19T12:53:55.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417903645}) 2024-08-19T12:53:55.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3292677299}) 2024-08-19T12:53:55.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:53:55.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417903645}) 2024-08-19T12:53:55.966 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:53:55.961+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2589526089}) 2024-08-19T12:54:10.480 INFO:tasks.mon_thrash:Sending CONT to mon e 2024-08-19T12:54:10.481 INFO:tasks.ceph.mon.e:Sent signal 18 2024-08-19T12:54:10.481 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-08-19T12:54:10.481 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:54:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.585+0000 7f444523f640 10 monclient: get_monmap_and_config 2024-08-19T12:54:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.585+0000 7f444523f640 10 monclient: build_initial_monmap 2024-08-19T12:54:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.585+0000 7f444523f640 10 monclient: monmap: 2024-08-19T12:54:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:10.587 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:54:10.587044+0000 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:54:10.587044+0000 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:10.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.585+0000 7f444523f640 1 Processor -- start 2024-08-19T12:54:10.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.585+0000 7f444523f640 1 -- start start 2024-08-19T12:54:10.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.585+0000 7f444523f640 10 monclient: init 2024-08-19T12:54:10.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.586+0000 7f444523f640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:10.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.586+0000 7f444523f640 10 monclient: _add_conns ranks=[6,7,5,2,4,0,3,8,1] 2024-08-19T12:54:10.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.586+0000 7f444523f640 10 monclient(hunting): picked mon.noname-d con 0x7f4440073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:54:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.586+0000 7f444523f640 10 monclient(hunting): picked mon.noname-i con 0x7f44400a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:54:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.586+0000 7f444523f640 10 monclient(hunting): picked mon.noname-h con 0x7f444013f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:54:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.586+0000 7f444523f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f444009dae0 con 0x7f444013f2e0 2024-08-19T12:54:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.586+0000 7f444523f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4440136910 con 0x7f4440073470 2024-08-19T12:54:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.586+0000 7f444523f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f444009dd50 con 0x7f44400a57a0 2024-08-19T12:54:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.586+0000 7f444523f640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.586+0000 7f444523f640 10 monclient(hunting): authenticate will time out at 1644.674439s 2024-08-19T12:54:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.586+0000 7f443ed9d640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f4440073470 0x7f4440136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:42720/0 (socket says 172.21.15.26:42720) 2024-08-19T12:54:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.586+0000 7f443ed9d640 1 -- 172.21.15.26:0/2136529379 learned_addr learned my addr 172.21.15.26:0/2136529379 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.588+0000 7f443ed9d640 1 -- 172.21.15.26:0/2136529379 >> v1:172.21.15.26:6792/0 conn(0x7f4440073470 legacy=0x7f4440136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.588+0000 7f443f59e640 1 -- 172.21.15.26:0/2136529379 >> v1:172.21.15.104:6791/0 conn(0x7f444013f2e0 legacy=0x7f44401416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:10.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:10.588+0000 7f443e59c640 1 -- 172.21.15.26:0/2136529379 >> v1:172.21.15.104:6792/0 conn(0x7f44400a57a0 legacy=0x7f44400a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:13.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 10 monclient(hunting): tick 2024-08-19T12:54:13.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 1 monclient(hunting): continuing hunt 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 1 -- 172.21.15.26:0/2136529379 >> v1:172.21.15.104:6792/0 conn(0x7f44400a57a0 legacy=0x7f44400a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 1 -- 172.21.15.26:0/2136529379 >> v1:172.21.15.26:6792/0 conn(0x7f4440073470 legacy=0x7f4440136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 1 -- 172.21.15.26:0/2136529379 >> v1:172.21.15.104:6791/0 conn(0x7f444013f2e0 legacy=0x7f44401416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 10 monclient: _add_conns ranks=[0,8,3,2,4,1] 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 10 monclient(hunting): picked mon.noname-a con 0x7f442c003960 addr v1:172.21.15.26:6789/0 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 10 monclient(hunting): picked mon.noname-e con 0x7f442c0075f0 addr v1:172.21.15.26:6793/0 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 10 monclient(hunting): picked mon.noname-g con 0x7f442c00b360 addr v1:172.21.15.104:6790/0 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 1 -- 172.21.15.26:0/2136529379 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f444009e030 con 0x7f442c003960 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 1 -- 172.21.15.26:0/2136529379 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f442c010180 con 0x7f442c00b360 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 1 -- 172.21.15.26:0/2136529379 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f442c011400 con 0x7f442c0075f0 2024-08-19T12:54:13.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.587+0000 7f443cd99640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:13.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.589+0000 7f443e59c640 1 -- 172.21.15.26:0/2136529379 >> v1:172.21.15.26:6793/0 conn(0x7f442c0075f0 legacy=0x7f442c009a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:13.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.589+0000 7f443ed9d640 1 -- 172.21.15.26:0/2136529379 >> v1:172.21.15.26:6789/0 conn(0x7f442c003960 legacy=0x7f442c005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:13.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:13.589+0000 7f443f59e640 1 -- 172.21.15.26:0/2136529379 >> v1:172.21.15.104:6790/0 conn(0x7f442c00b360 legacy=0x7f442c00d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:15.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.491+0000 7f443dd9b640 1 -- 172.21.15.26:0/2136529379 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 909717730 0 0) 0x7f442c010180 con 0x7f442c00b360 2024-08-19T12:54:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.491+0000 7f443dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:54:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.491+0000 7f443dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.491+0000 7f443dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.491+0000 7f443dd9b640 10 monclient(hunting): my global_id is 194223 2024-08-19T12:54:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.491+0000 7f443dd9b640 1 -- 172.21.15.26:0/2136529379 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4428003520 con 0x7f442c00b360 2024-08-19T12:54:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 1 -- 172.21.15.26:0/2136529379 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 524123027 0 0) 0x7f4428003520 con 0x7f442c00b360 2024-08-19T12:54:15.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 1 -- 172.21.15.26:0/2136529379 >> v1:172.21.15.26:6793/0 conn(0x7f442c0075f0 legacy=0x7f442c009a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 1 -- 172.21.15.26:0/2136529379 >> v1:172.21.15.26:6789/0 conn(0x7f442c003960 legacy=0x7f442c005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 10 monclient: _finish_hunting 0 2024-08-19T12:54:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 1 monclient: found mon.noname-g 2024-08-19T12:54:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:54:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 1 -- 172.21.15.26:0/2136529379 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442c012700 con 0x7f442c00b360 2024-08-19T12:54:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 10 monclient: _finish_auth 0 2024-08-19T12:54:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 10 monclient: _check_auth_tickets 2024-08-19T12:54:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 1 -- 172.21.15.26:0/2136529379 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4438001eb0 con 0x7f442c00b360 2024-08-19T12:54:15.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:54:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443f59e640 1 -- 172.21.15.26:0/2136529379 reap_dead start 2024-08-19T12:54:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 10 monclient: dump: 2024-08-19T12:54:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:15.496 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:15.497 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:15.498 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f444523f640 5 monclient: authenticate success, global_id 194223 2024-08-19T12:54:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f444523f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 1 -- 172.21.15.26:0/2136529379 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4438003990 con 0x7f442c00b360 2024-08-19T12:54:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 1 -- 172.21.15.26:0/2136529379 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4438003e80 con 0x7f442c00b360 2024-08-19T12:54:15.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:54:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f443dd9b640 10 monclient: dump: 2024-08-19T12:54:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:15.500 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f444523f640 10 monclient: get_monmap_and_config success 2024-08-19T12:54:15.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.492+0000 7f444523f640 4 set_mon_vals no callback set 2024-08-19T12:54:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.493+0000 7f444523f640 10 monclient: shutdown 2024-08-19T12:54:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.493+0000 7f444523f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:15.502 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.493+0000 7f444523f640 1 -- 172.21.15.26:0/2136529379 >> v1:172.21.15.104:6790/0 conn(0x7f442c00b360 legacy=0x7f442c00d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.493+0000 7f444523f640 1 -- 172.21.15.26:0/2136529379 shutdown_connections 2024-08-19T12:54:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.493+0000 7f444523f640 1 -- 172.21.15.26:0/2136529379 >> 172.21.15.26:0/2136529379 conn(0x7f44400abff0 msgr2=0x7f44400ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.493+0000 7f444523f640 1 -- 172.21.15.26:0/2136529379 shutdown_connections 2024-08-19T12:54:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.493+0000 7f444523f640 1 -- 172.21.15.26:0/2136529379 wait complete. 2024-08-19T12:54:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.495+0000 7f444523f640 10 monclient: build_initial_monmap 2024-08-19T12:54:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.495+0000 7f444523f640 10 monclient: monmap: 2024-08-19T12:54:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:54:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:54:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:54:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.495+0000 7f444523f640 1 librados: starting msgr at 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.495+0000 7f444523f640 1 librados: starting objecter 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.496+0000 7f444523f640 1 Processor -- start 2024-08-19T12:54:15.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.496+0000 7f444523f640 1 -- start start 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.496+0000 7f444523f640 1 librados: setting wanted keys 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.496+0000 7f444523f640 1 librados: calling monclient init 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.496+0000 7f444523f640 10 monclient: init 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.497+0000 7f444523f640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.497+0000 7f444523f640 10 monclient: _add_conns ranks=[3,6,2,5,4,8,1,7,0] 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.497+0000 7f444523f640 10 monclient(hunting): picked mon.noname-g con 0x7f442c00b360 addr v1:172.21.15.104:6790/0 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.497+0000 7f444523f640 10 monclient(hunting): picked mon.noname-d con 0x7f444009ed00 addr v1:172.21.15.26:6792/0 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.497+0000 7f444523f640 10 monclient(hunting): picked mon.noname-b con 0x7f444009f140 addr v1:172.21.15.26:6790/0 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.497+0000 7f444523f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44400a0670 con 0x7f444009f140 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.497+0000 7f444523f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44400a1a30 con 0x7f442c00b360 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.497+0000 7f444523f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44400aaba0 con 0x7f444009ed00 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.497+0000 7f444523f640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.497+0000 7f444523f640 10 monclient(hunting): authenticate will time out at 1649.584337s 2024-08-19T12:54:15.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.497+0000 7f443e59c640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f444009ed00 0x7f44401d36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:42730/0 (socket says 172.21.15.26:42730) 2024-08-19T12:54:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.497+0000 7f443e59c640 1 -- 172.21.15.26:0/15901573 learned_addr learned my addr 172.21.15.26:0/15901573 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.499+0000 7f443e59c640 1 -- 172.21.15.26:0/15901573 >> v1:172.21.15.26:6792/0 conn(0x7f444009ed00 legacy=0x7f44401d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.499+0000 7f443f59e640 1 -- 172.21.15.26:0/15901573 >> v1:172.21.15.26:6790/0 conn(0x7f444009f140 legacy=0x7f44401d6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.499+0000 7f44277fe640 10 client.?.objecter ms_handle_connect 0x7f444009ed00 2024-08-19T12:54:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.499+0000 7f44277fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.499+0000 7f44277fe640 10 client.?.objecter ms_handle_connect 0x7f444009f140 2024-08-19T12:54:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.499+0000 7f44277fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.499+0000 7f443ed9d640 1 -- 172.21.15.26:0/15901573 >> v1:172.21.15.104:6790/0 conn(0x7f442c00b360 legacy=0x7f444009e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.499+0000 7f44277fe640 10 client.?.objecter ms_handle_connect 0x7f442c00b360 2024-08-19T12:54:15.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.499+0000 7f44277fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2280081313 0 0) 0x7f44400a1a30 con 0x7f442c00b360 2024-08-19T12:54:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 10 monclient(hunting): _negotiate 2024-08-19T12:54:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 10 monclient(hunting): my global_id is 194232 2024-08-19T12:54:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f442c006040 con 0x7f442c00b360 2024-08-19T12:54:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2181848574 0 0) 0x7f44400a0670 con 0x7f444009f140 2024-08-19T12:54:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 10 monclient(hunting): _negotiate 2024-08-19T12:54:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 10 monclient(hunting): my global_id is 27245 2024-08-19T12:54:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44400a1a30 con 0x7f444009f140 2024-08-19T12:54:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2897289872 0 0) 0x7f442c006040 con 0x7f442c00b360 2024-08-19T12:54:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f44400a0670 con 0x7f442c00b360 2024-08-19T12:54:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4430001bc0 con 0x7f442c00b360 2024-08-19T12:54:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:54:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 10 monclient(hunting): dump: 2024-08-19T12:54:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.519 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.513+0000 7f44277fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:54:15.520 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1163932450 0 0) 0x7f44400a1a30 con 0x7f444009f140 2024-08-19T12:54:15.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f442c006040 con 0x7f444009f140 2024-08-19T12:54:15.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4438003e80 con 0x7f444009f140 2024-08-19T12:54:15.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:15.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:54:15.521 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 10 monclient(hunting): dump: 2024-08-19T12:54:15.521 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:15.521 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:15.521 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.521 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.522 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1358126009 0 0) 0x7f44400a0670 con 0x7f442c00b360 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 >> v1:172.21.15.26:6792/0 conn(0x7f444009ed00 legacy=0x7f44401d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 >> v1:172.21.15.26:6790/0 conn(0x7f444009f140 legacy=0x7f44401d6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 10 monclient: _finish_hunting 0 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 1 monclient: found mon.g 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444009f6d0 con 0x7f442c00b360 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 10 monclient: _finish_auth 0 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 10 monclient: _check_auth_tickets 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f444523f640 5 monclient: authenticate success, global_id 194232 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f444523f640 10 monclient: _renew_subs 2024-08-19T12:54:15.523 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f444523f640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f444523f640 1 -- 172.21.15.26:0/15901573 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f44400a1750 con 0x7f442c00b360 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f444523f640 10 client.194232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f444523f640 10 monclient: _renew_subs 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f444523f640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f444523f640 1 -- 172.21.15.26:0/15901573 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f44400a0230 con 0x7f442c00b360 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f444523f640 1 librados: init done 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.3 v1:172.21.15.104:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4430004a90 con 0x7f442c00b360 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.3 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4430005000 con 0x7f442c00b360 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f443d59a640 4 set_mon_vals no callback set 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.514+0000 7f44277fe640 10 monclient: dump: 2024-08-19T12:54:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:15.525 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:15.526 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.515+0000 7f444523f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.515+0000 7f444523f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.515+0000 7f444523f640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:54:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.515+0000 7f444523f640 1 -- 172.21.15.26:0/15901573 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4420005080 con 0x7f442c00b360 2024-08-19T12:54:15.526 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.515+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.3 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f44300051a0 con 0x7f442c00b360 2024-08-19T12:54:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.515+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.3 v1:172.21.15.104:6790/0 8 ==== osd_map(573..573 src has 350..573) ==== 5969+0+0 (unknown 1669907522 0 0) 0x7f443004e2e0 con 0x7f442c00b360 2024-08-19T12:54:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.515+0000 7f44277fe640 10 client.194232.objecter ms_dispatch 0x7f4440000b90 osd_map(573..573 src has 350..573) 2024-08-19T12:54:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.515+0000 7f44277fe640 3 client.194232.objecter handle_osd_map got epochs [573,573] > 0 2024-08-19T12:54:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.515+0000 7f44277fe640 3 client.194232.objecter handle_osd_map decoding full epoch 573 2024-08-19T12:54:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.516+0000 7f44277fe640 20 client.194232.objecter dump_active .. 0 homeless 2024-08-19T12:54:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.516+0000 7f443e59c640 10 monclient: get_auth_request con 0x7f442c042e40 auth_method 0 2024-08-19T12:54:15.527 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.518+0000 7f44277fe640 10 client.194232.objecter ms_handle_connect 0x7f442c042e40 2024-08-19T12:54:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.520+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.3 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f4430017490 con 0x7f442c00b360 2024-08-19T12:54:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.520+0000 7f44277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:15.528 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.520+0000 7f44277fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:54:15.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.779+0000 7f444523f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:54:15.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.780+0000 7f444523f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:54:15.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.780+0000 7f444523f640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:54:15.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.780+0000 7f444523f640 1 -- 172.21.15.26:0/15901573 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4420005370 con 0x7f442c00b360 2024-08-19T12:54:15.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.780+0000 7f44277fe640 1 -- 172.21.15.26:0/15901573 <== mon.3 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2724 (unknown 254746657 0 760018620) 0x7f4430016d90 con 0x7f442c00b360 2024-08-19T12:54:15.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.780+0000 7f44277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:54:15.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.780+0000 7f44277fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:54:15.782 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":116,"quorum":[0,2,3,4,5,6,7,8],"quorum_names":["a","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:54:15.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.782+0000 7f444523f640 10 librados: watch_flush enter 2024-08-19T12:54:15.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.782+0000 7f444523f640 10 librados: watch_flush exit 2024-08-19T12:54:15.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.782+0000 7f444523f640 20 client.194232.objecter shutdown clearing up homeless session... 2024-08-19T12:54:15.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.782+0000 7f444523f640 10 client.194232.objecter successfully canceled tick 2024-08-19T12:54:15.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.783+0000 7f444523f640 1 -- 172.21.15.26:0/15901573 >> v1:172.21.15.104:6800/899887506 conn(0x7f442c042e40 legacy=0x7f442c045300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.783+0000 7f444523f640 10 monclient: shutdown 2024-08-19T12:54:15.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.783+0000 7f444523f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:15.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.783+0000 7f444523f640 1 -- 172.21.15.26:0/15901573 >> v1:172.21.15.104:6790/0 conn(0x7f442c00b360 legacy=0x7f444009e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.783+0000 7f444523f640 1 -- 172.21.15.26:0/15901573 shutdown_connections 2024-08-19T12:54:15.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.783+0000 7f444523f640 1 -- 172.21.15.26:0/15901573 >> 172.21.15.26:0/15901573 conn(0x7f44400abff0 msgr2=0x7f44400ab3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:15.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.783+0000 7f444523f640 1 -- 172.21.15.26:0/15901573 shutdown_connections 2024-08-19T12:54:15.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.783+0000 7f444523f640 1 -- 172.21.15.26:0/15901573 wait complete. 2024-08-19T12:54:15.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.783+0000 7f444523f640 1 librados: shutdown 2024-08-19T12:54:15.794 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-08-19T12:54:15.794 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-08-19T12:54:15.795 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:54:15.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.894+0000 7f84ad1b1640 10 monclient: get_monmap_and_config 2024-08-19T12:54:15.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.894+0000 7f84ad1b1640 10 monclient: build_initial_monmap 2024-08-19T12:54:15.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.894+0000 7f84ad1b1640 10 monclient: monmap: 2024-08-19T12:54:15.895 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:15.895 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:15.895 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:54:15.894945+0000 2024-08-19T12:54:15.895 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:54:15.894945+0000 2024-08-19T12:54:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:15.896 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:15.897 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.894+0000 7f84ad1b1640 1 Processor -- start 2024-08-19T12:54:15.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.894+0000 7f84ad1b1640 1 -- start start 2024-08-19T12:54:15.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.894+0000 7f84ad1b1640 10 monclient: init 2024-08-19T12:54:15.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84ad1b1640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:15.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84ad1b1640 10 monclient: _add_conns ranks=[6,5,2,1,3,8,0,7,4] 2024-08-19T12:54:15.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84ad1b1640 10 monclient(hunting): picked mon.noname-d con 0x7f84a8073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:54:15.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84ad1b1640 10 monclient(hunting): picked mon.noname-h con 0x7f84a80a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:54:15.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84ad1b1640 10 monclient(hunting): picked mon.noname-b con 0x7f84a813f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:54:15.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84ad1b1640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84a809dae0 con 0x7f84a813f2e0 2024-08-19T12:54:15.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84ad1b1640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84a8136910 con 0x7f84a80a57a0 2024-08-19T12:54:15.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84ad1b1640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84a809dd50 con 0x7f84a8073470 2024-08-19T12:54:15.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84ad1b1640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:15.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84ad1b1640 10 monclient(hunting): authenticate will time out at 1649.982237s 2024-08-19T12:54:15.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84a759e640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f84a813f2e0 0x7f84a81416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41316/0 (socket says 172.21.15.26:41316) 2024-08-19T12:54:15.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84a759e640 1 -- 172.21.15.26:0/1308231821 learned_addr learned my addr 172.21.15.26:0/1308231821 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:15.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.895+0000 7f84a6d9d640 1 --1- 172.21.15.26:0/1308231821 >> v1:172.21.15.26:6792/0 conn(0x7f84a8073470 0x7f84a8136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:42738/0 (socket says 172.21.15.26:42738) 2024-08-19T12:54:15.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a759e640 1 -- 172.21.15.26:0/1308231821 >> v1:172.21.15.26:6790/0 conn(0x7f84a813f2e0 legacy=0x7f84a81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:15.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a6d9d640 1 -- 172.21.15.26:0/1308231821 >> v1:172.21.15.26:6792/0 conn(0x7f84a8073470 legacy=0x7f84a8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:15.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f849e59c640 1 -- 172.21.15.26:0/1308231821 >> v1:172.21.15.104:6791/0 conn(0x7f84a80a57a0 legacy=0x7f84a80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:15.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 1 -- 172.21.15.26:0/1308231821 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 550528978 0 0) 0x7f84a809dd50 con 0x7f84a8073470 2024-08-19T12:54:15.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 10 monclient(hunting): _negotiate 2024-08-19T12:54:15.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:15.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:15.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 10 monclient(hunting): my global_id is 194235 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 1 -- 172.21.15.26:0/1308231821 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8494003520 con 0x7f84a8073470 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 1 -- 172.21.15.26:0/1308231821 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2837123586 0 0) 0x7f8494003520 con 0x7f84a8073470 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 1 -- 172.21.15.26:0/1308231821 >> v1:172.21.15.104:6791/0 conn(0x7f84a80a57a0 legacy=0x7f84a80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 1 -- 172.21.15.26:0/1308231821 >> v1:172.21.15.26:6790/0 conn(0x7f84a813f2e0 legacy=0x7f84a81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 10 monclient: _finish_hunting 0 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 1 monclient: found mon.noname-d 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 1 -- 172.21.15.26:0/1308231821 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a809e030 con 0x7f84a8073470 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 10 monclient: _finish_auth 0 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 10 monclient: _check_auth_tickets 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:15.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 1 -- 172.21.15.26:0/1308231821 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8498001ae0 con 0x7f84a8073470 2024-08-19T12:54:15.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84ad1b1640 5 monclient: authenticate success, global_id 194235 2024-08-19T12:54:15.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84ad1b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:15.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:15.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:54:15.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.897+0000 7f84a659c640 10 monclient: dump: 2024-08-19T12:54:15.906 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:15.906 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:15.906 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.906 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.907 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:15.907 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.907 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:15.907 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:15.907 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:15.907 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:15.907 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:15.907 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:15.907 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:15.907 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:15.907 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84ad1b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84a659c640 1 -- 172.21.15.26:0/1308231821 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f84980047e0 con 0x7f84a8073470 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84a659c640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84a659c640 1 -- 172.21.15.26:0/1308231821 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8498004d50 con 0x7f84a8073470 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84ad1b1640 10 monclient: get_monmap_and_config success 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84ad1b1640 4 set_mon_vals no callback set 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84a659c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84a659c640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84a659c640 10 monclient: dump: 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.908 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84ad1b1640 10 monclient: shutdown 2024-08-19T12:54:15.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84ad1b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84ad1b1640 1 -- 172.21.15.26:0/1308231821 >> v1:172.21.15.26:6792/0 conn(0x7f84a8073470 legacy=0x7f84a8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84ad1b1640 1 -- 172.21.15.26:0/1308231821 shutdown_connections 2024-08-19T12:54:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84ad1b1640 1 -- 172.21.15.26:0/1308231821 >> 172.21.15.26:0/1308231821 conn(0x7f84a80abff0 msgr2=0x7f84a80ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84ad1b1640 1 -- 172.21.15.26:0/1308231821 shutdown_connections 2024-08-19T12:54:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.898+0000 7f84ad1b1640 1 -- 172.21.15.26:0/1308231821 wait complete. 2024-08-19T12:54:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.899+0000 7f84ad1b1640 10 monclient: build_initial_monmap 2024-08-19T12:54:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.899+0000 7f84ad1b1640 10 monclient: monmap: 2024-08-19T12:54:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:54:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:54:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:54:15.910 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.899+0000 7f84ad1b1640 1 librados: starting msgr at 2024-08-19T12:54:15.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.899+0000 7f84ad1b1640 1 librados: starting objecter 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.899+0000 7f84ad1b1640 1 Processor -- start 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.899+0000 7f84ad1b1640 1 -- start start 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.899+0000 7f84ad1b1640 1 librados: setting wanted keys 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.899+0000 7f84ad1b1640 1 librados: calling monclient init 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.899+0000 7f84ad1b1640 10 monclient: init 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f84ad1b1640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f84ad1b1640 10 monclient: _add_conns ranks=[1,0,8,2,7,6,5,3,4] 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f84ad1b1640 10 monclient(hunting): picked mon.noname-f con 0x7f84a8073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f84ad1b1640 10 monclient(hunting): picked mon.noname-a con 0x7f84a80a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f84ad1b1640 10 monclient(hunting): picked mon.noname-e con 0x7f84a813f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f84ad1b1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84a813edb0 con 0x7f84a80a57a0 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f84ad1b1640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84a80a3b10 con 0x7f84a8073470 2024-08-19T12:54:15.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f84ad1b1640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84a81372a0 con 0x7f84a813f2e0 2024-08-19T12:54:15.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f84ad1b1640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:15.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f84ad1b1640 10 monclient(hunting): authenticate will time out at 1649.987432s 2024-08-19T12:54:15.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f84a6d9d640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f84a8073470 legacy=0x7f84a81ce3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:54:15.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f849e59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f84a80a57a0 0x7f84a81d3340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53868/0 (socket says 172.21.15.26:53868) 2024-08-19T12:54:15.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.900+0000 7f849e59c640 1 -- 172.21.15.26:0/3091282976 learned_addr learned my addr 172.21.15.26:0/3091282976 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:15.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.902+0000 7f849e59c640 1 -- 172.21.15.26:0/3091282976 >> v1:172.21.15.26:6789/0 conn(0x7f84a80a57a0 legacy=0x7f84a81d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:15.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.902+0000 7f849ffff640 10 client.?.objecter ms_handle_connect 0x7f84a80a57a0 2024-08-19T12:54:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.902+0000 7f849ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.902+0000 7f849ffff640 10 client.?.objecter ms_handle_connect 0x7f84a813f2e0 2024-08-19T12:54:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.902+0000 7f849ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.902+0000 7f84a759e640 1 -- 172.21.15.26:0/3091282976 >> v1:172.21.15.26:6793/0 conn(0x7f84a813f2e0 legacy=0x7f84a81d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.902+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2587440435 0 0) 0x7f84a8136910 con 0x7f84a80a57a0 2024-08-19T12:54:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.902+0000 7f849ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:54:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.902+0000 7f849ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:15.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.902+0000 7f849ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.902+0000 7f849ffff640 10 monclient(hunting): my global_id is 194283 2024-08-19T12:54:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.902+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8488004eb0 con 0x7f84a80a57a0 2024-08-19T12:54:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 493088091 0 0) 0x7f8488004eb0 con 0x7f84a80a57a0 2024-08-19T12:54:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f84a8136910 con 0x7f84a80a57a0 2024-08-19T12:54:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8490003610 con 0x7f84a80a57a0 2024-08-19T12:54:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:54:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 10 monclient(hunting): dump: 2024-08-19T12:54:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:15.919 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:15.921 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:54:15.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2227430127 0 0) 0x7f84a8136910 con 0x7f84a80a57a0 2024-08-19T12:54:15.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 >> v1:172.21.15.26:6793/0 conn(0x7f84a813f2e0 legacy=0x7f84a81d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 >> v1:172.21.15.104:6789/0 conn(0x7f84a8073470 legacy=0x7f84a81ce3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:54:15.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 10 monclient: _finish_hunting 0 2024-08-19T12:54:15.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 1 monclient: found mon.a 2024-08-19T12:54:15.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:15.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:54:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a80a4670 con 0x7f84a80a57a0 2024-08-19T12:54:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 10 monclient: _finish_auth 0 2024-08-19T12:54:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 10 monclient: _check_auth_tickets 2024-08-19T12:54:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f84ad1b1640 5 monclient: authenticate success, global_id 194283 2024-08-19T12:54:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f84ad1b1640 10 monclient: _renew_subs 2024-08-19T12:54:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f84ad1b1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:54:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f84ad1b1640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f84a80a5210 con 0x7f84a80a57a0 2024-08-19T12:54:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f84ad1b1640 10 client.194283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:54:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f84ad1b1640 10 monclient: _renew_subs 2024-08-19T12:54:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f84ad1b1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:54:15.922 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f84ad1b1640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f84a80a3380 con 0x7f84a80a57a0 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f84ad1b1640 1 librados: init done 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8490003c00 con 0x7f84a80a57a0 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f8490004130 con 0x7f84a80a57a0 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f84a5d9b640 4 set_mon_vals no callback set 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.903+0000 7f849ffff640 10 monclient: dump: 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.923 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:15.924 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.924 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:15.924 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:15.924 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:15.924 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:15.924 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:15.924 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:15.924 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:15.924 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:15.924 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:15.924 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.904+0000 7f84ad1b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.904+0000 7f84ad1b1640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f848c0065c0 con 0x7f848c002860 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.904+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f84900042d0 con 0x7f84a80a57a0 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.904+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(573..573 src has 350..573) ==== 5969+0+0 (unknown 1669907522 0 0) 0x7f849004e5f0 con 0x7f84a80a57a0 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.904+0000 7f849ffff640 10 client.194283.objecter ms_dispatch 0x7f84a8000b90 osd_map(573..573 src has 350..573) 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.904+0000 7f849ffff640 3 client.194283.objecter handle_osd_map got epochs [573,573] > 0 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.904+0000 7f849ffff640 3 client.194283.objecter handle_osd_map decoding full epoch 573 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.904+0000 7f849ffff640 20 client.194283.objecter dump_active .. 0 homeless 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.904+0000 7f84a759e640 10 monclient: get_auth_request con 0x7f8488040e90 auth_method 0 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 10 client.194283.objecter ms_handle_connect 0x7f848c002860 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 10 client.194283.objecter resend_mon_ops 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f84a6d9d640 1 -- 172.21.15.26:0/3091282976 >> v1:172.21.15.26:6789/0 conn(0x7f848c002860 legacy=0x7f848c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:15.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 828105870 0 0) 0x7f848c0065c0 con 0x7f848c002860 2024-08-19T12:54:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:54:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 10 monclient(hunting): my global_id is 194292 2024-08-19T12:54:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84a8136910 con 0x7f848c002860 2024-08-19T12:54:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1308850138 0 0) 0x7f84a8136910 con 0x7f848c002860 2024-08-19T12:54:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f848c0065c0 con 0x7f848c002860 2024-08-19T12:54:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f84980047e0 con 0x7f848c002860 2024-08-19T12:54:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:54:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 10 monclient: dump: 2024-08-19T12:54:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:15.927 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2982254772 0 0) 0x7f848c0065c0 con 0x7f848c002860 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.906+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f848c00a160 con 0x7f848c002860 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.907+0000 7f849ffff640 10 client.194283.objecter ms_handle_connect 0x7f8488040e90 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.907+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f84980047e0 con 0x7f848c002860 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.907+0000 7f849ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.907+0000 7f849ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.907+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 >> v1:172.21.15.26:6789/0 conn(0x7f848c002860 legacy=0x7f848c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.907+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 >> v1:172.21.15.26:6789/0 conn(0x7f848c002860 legacy=0x7f848c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.913+0000 7f84ad1b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.913+0000 7f84ad1b1640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f848c009e50 con 0x7f848c009110 2024-08-19T12:54:15.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.915+0000 7f84a6d9d640 1 -- 172.21.15.26:0/3091282976 >> v1:172.21.15.26:6789/0 conn(0x7f848c009110 legacy=0x7f848c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:15.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.915+0000 7f849ffff640 10 client.194283.objecter ms_handle_connect 0x7f848c009110 2024-08-19T12:54:15.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.915+0000 7f849ffff640 10 client.194283.objecter resend_mon_ops 2024-08-19T12:54:15.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.917+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1814279861 0 0) 0x7f848c009e50 con 0x7f848c009110 2024-08-19T12:54:15.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.917+0000 7f849ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:54:15.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.917+0000 7f849ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:15.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.917+0000 7f849ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:15.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.917+0000 7f849ffff640 10 monclient(hunting): my global_id is 194301 2024-08-19T12:54:15.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f848c0065c0 con 0x7f848c009110 2024-08-19T12:54:15.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1368973347 0 0) 0x7f848c0065c0 con 0x7f848c009110 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f848c009e50 con 0x7f848c009110 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f84980053d0 con 0x7f848c009110 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 10 monclient: dump: 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:15.931 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 165123274 0 0) 0x7f848c009e50 con 0x7f848c009110 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f848c00c3f0 con 0x7f848c009110 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7535 (unknown 0 0 3580844612) 0x7f8498007d80 con 0x7f848c009110 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 >> v1:172.21.15.26:6789/0 conn(0x7f848c009110 legacy=0x7f848c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f849ffff640 1 -- 172.21.15.26:0/3091282976 >> v1:172.21.15.26:6789/0 conn(0x7f848c009110 legacy=0x7f848c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:15.933 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:54:15.933 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:54:15.933 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:54:15.933 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:54:15.933 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 116, 2024-08-19T12:54:15.933 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 40841, 2024-08-19T12:54:15.933 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:54:15.933 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:54:15.933 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:15.934 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:15.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:15.935 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:15.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:15.935 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:15.935 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:15.935 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:15.935 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:15.935 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:15.935 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:15.935 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:15.935 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:54:15.935 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.936 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:54:15.937 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:54:15.938 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:54:15.938 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:15.938 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:54:15.938 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:15.938 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:15.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:15.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:15.938 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:15.938 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:15.938 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:15.938 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:15.938 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:15.939 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:15.939 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:15.939 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:15.939 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:54:15.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:54:15.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.939 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:54:15.939 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:54:15.939 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:15.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:15.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:54:15.941 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:54:15.942 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:15.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:15.943 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:15.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:54:15.943 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:54:15.943 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:15.943 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:15.943 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:15.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.943 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:54:15.943 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:15.944 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:15.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.945 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:54:15.945 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:54:15.945 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:15.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:15.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.945 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:15.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:54:15.945 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:15.945 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:15.945 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:54:15.949 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:54:15.949 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:15.949 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:15.949 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:15.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.949 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:54:15.949 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:54:15.949 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:15.949 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:15.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.949 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:15.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:54:15.950 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:15.950 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:15.950 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:15.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:54:15.950 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:54:15.950 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:15.950 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:15.950 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:15.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:54:15.951 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:54:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:54:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:15.954 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:54:15.954 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:54:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f84ad1b1640 10 librados: watch_flush enter 2024-08-19T12:54:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f84ad1b1640 10 librados: watch_flush exit 2024-08-19T12:54:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f84ad1b1640 20 client.194283.objecter shutdown clearing up homeless session... 2024-08-19T12:54:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f84ad1b1640 10 client.194283.objecter successfully canceled tick 2024-08-19T12:54:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f84ad1b1640 1 -- 172.21.15.26:0/3091282976 >> v1:172.21.15.104:6800/899887506 conn(0x7f8488040e90 legacy=0x7f8488043350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f84ad1b1640 10 monclient: shutdown 2024-08-19T12:54:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f84ad1b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f84ad1b1640 1 -- 172.21.15.26:0/3091282976 >> v1:172.21.15.26:6789/0 conn(0x7f84a80a57a0 legacy=0x7f84a81d3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.918+0000 7f84a759e640 1 -- 172.21.15.26:0/3091282976 reap_dead start 2024-08-19T12:54:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.919+0000 7f84a759e640 1 -- 172.21.15.26:0/3091282976 reap_dead start 2024-08-19T12:54:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.919+0000 7f84ad1b1640 1 -- 172.21.15.26:0/3091282976 shutdown_connections 2024-08-19T12:54:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.919+0000 7f84ad1b1640 1 -- 172.21.15.26:0/3091282976 >> 172.21.15.26:0/3091282976 conn(0x7f84a80abff0 msgr2=0x7f84a8133a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:15.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.919+0000 7f84ad1b1640 1 -- 172.21.15.26:0/3091282976 shutdown_connections 2024-08-19T12:54:15.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.919+0000 7f84ad1b1640 1 -- 172.21.15.26:0/3091282976 wait complete. 2024-08-19T12:54:15.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:15.919+0000 7f84ad1b1640 1 librados: shutdown 2024-08-19T12:54:15.957 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:54:15.957 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-08-19T12:54:15.957 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:54:16.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.056+0000 7fee33c9f640 10 monclient: get_monmap_and_config 2024-08-19T12:54:16.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.056+0000 7fee33c9f640 10 monclient: build_initial_monmap 2024-08-19T12:54:16.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.056+0000 7fee33c9f640 10 monclient: monmap: 2024-08-19T12:54:16.057 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:16.057 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.057 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:54:16.057492+0000 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:54:16.057492+0000 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:16.058 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.057+0000 7fee33c9f640 1 Processor -- start 2024-08-19T12:54:16.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.057+0000 7fee33c9f640 1 -- start start 2024-08-19T12:54:16.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.057+0000 7fee33c9f640 10 monclient: init 2024-08-19T12:54:16.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.057+0000 7fee33c9f640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:16.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.057+0000 7fee33c9f640 10 monclient: _add_conns ranks=[4,2,8,7,5,0,6,1,3] 2024-08-19T12:54:16.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.057+0000 7fee33c9f640 10 monclient(hunting): picked mon.noname-c con 0x7fee2c06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:54:16.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.057+0000 7fee33c9f640 10 monclient(hunting): picked mon.noname-b con 0x7fee2c0a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:54:16.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.057+0000 7fee33c9f640 10 monclient(hunting): picked mon.noname-e con 0x7fee2c0a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.057+0000 7fee33c9f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2c136790 con 0x7fee2c0a5920 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.057+0000 7fee33c9f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2c09ddf0 con 0x7fee2c06e7b0 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.058+0000 7fee33c9f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2c09e0d0 con 0x7fee2c0a5d50 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.058+0000 7fee33c9f640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.058+0000 7fee33c9f640 10 monclient(hunting): authenticate will time out at 1650.145048s 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.058+0000 7fee31a3b640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fee2c06e7b0 0x7fee2c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36660/0 (socket says 172.21.15.26:36660) 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.058+0000 7fee31a3b640 1 -- 172.21.15.26:0/582533561 learned_addr learned my addr 172.21.15.26:0/582533561 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.058+0000 7fee3223c640 1 --1- 172.21.15.26:0/582533561 >> v1:172.21.15.26:6793/0 conn(0x7fee2c0a5d50 0x7fee2c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:52556/0 (socket says 172.21.15.26:52556) 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.060+0000 7fee3223c640 1 -- 172.21.15.26:0/582533561 >> v1:172.21.15.26:6793/0 conn(0x7fee2c0a5d50 legacy=0x7fee2c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.060+0000 7fee3223c640 1 -- 172.21.15.26:0/582533561 >> v1:172.21.15.26:6793/0 conn(0x7fee2c0a5d50 legacy=0x7fee2c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.060+0000 7fee3223c640 1 --1- 172.21.15.26:0/582533561 >> v1:172.21.15.26:6793/0 conn(0x7fee2c0a5d50 0x7fee2c141a20 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.060+0000 7fee3123a640 1 -- 172.21.15.26:0/582533561 >> v1:172.21.15.26:6790/0 conn(0x7fee2c0a5920 legacy=0x7fee2c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.060+0000 7fee31a3b640 1 -- 172.21.15.26:0/582533561 >> v1:172.21.15.26:6791/0 conn(0x7fee2c06e7b0 legacy=0x7fee2c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.185+0000 7fee30a39640 1 -- 172.21.15.26:0/582533561 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3994340283 0 0) 0x7fee2c136790 con 0x7fee2c0a5920 2024-08-19T12:54:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.185+0000 7fee30a39640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.185+0000 7fee30a39640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.185+0000 7fee30a39640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.185+0000 7fee30a39640 10 monclient(hunting): my global_id is 27254 2024-08-19T12:54:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.185+0000 7fee30a39640 1 -- 172.21.15.26:0/582533561 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee14003520 con 0x7fee2c0a5920 2024-08-19T12:54:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.185+0000 7fee30a39640 1 -- 172.21.15.26:0/582533561 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1245217774 0 0) 0x7fee14003520 con 0x7fee2c0a5920 2024-08-19T12:54:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 1 -- 172.21.15.26:0/582533561 >> v1:172.21.15.26:6793/0 conn(0x7fee2c0a5d50 legacy=0x7fee2c141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:54:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 1 -- 172.21.15.26:0/582533561 >> v1:172.21.15.26:6791/0 conn(0x7fee2c06e7b0 legacy=0x7fee2c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 10 monclient: _finish_hunting 0 2024-08-19T12:54:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 1 monclient: found mon.noname-b 2024-08-19T12:54:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:16.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 1 -- 172.21.15.26:0/582533561 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c09e3b0 con 0x7fee2c0a5920 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 10 monclient: _finish_auth 0 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 10 monclient: _check_auth_tickets 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 1 -- 172.21.15.26:0/582533561 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee20001b30 con 0x7fee2c0a5920 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 10 monclient: dump: 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.189 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.190 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.190 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.190 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.190 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.190 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.190 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.190 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.190 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.190 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.190 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.190 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee33c9f640 5 monclient: authenticate success, global_id 27254 2024-08-19T12:54:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 1 -- 172.21.15.26:0/582533561 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4126990901 0 0) 0x7fee1c0017d0 con 0x7fee2c06e7b0 2024-08-19T12:54:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:54:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee33c9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 1 -- 172.21.15.26:0/582533561 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee20003610 con 0x7fee2c0a5920 2024-08-19T12:54:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 1 -- 172.21.15.26:0/582533561 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee20003b80 con 0x7fee2c0a5920 2024-08-19T12:54:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee33c9f640 10 monclient: get_monmap_and_config success 2024-08-19T12:54:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee33c9f640 4 set_mon_vals no callback set 2024-08-19T12:54:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.186+0000 7fee30a39640 10 monclient: dump: 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.193 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.187+0000 7fee33c9f640 10 monclient: shutdown 2024-08-19T12:54:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.187+0000 7fee33c9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.187+0000 7fee33c9f640 1 -- 172.21.15.26:0/582533561 >> v1:172.21.15.26:6790/0 conn(0x7fee2c0a5920 legacy=0x7fee2c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.187+0000 7fee33c9f640 1 -- 172.21.15.26:0/582533561 shutdown_connections 2024-08-19T12:54:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.187+0000 7fee33c9f640 1 -- 172.21.15.26:0/582533561 >> 172.21.15.26:0/582533561 conn(0x7fee2c0a8740 msgr2=0x7fee2c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.187+0000 7fee33c9f640 1 -- 172.21.15.26:0/582533561 shutdown_connections 2024-08-19T12:54:16.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.187+0000 7fee33c9f640 1 -- 172.21.15.26:0/582533561 wait complete. 2024-08-19T12:54:16.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.187+0000 7fee33c9f640 10 monclient: build_initial_monmap 2024-08-19T12:54:16.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.187+0000 7fee33c9f640 10 monclient: monmap: 2024-08-19T12:54:16.195 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:16.195 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:16.196 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 1 librados: starting msgr at 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 1 librados: starting objecter 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 1 Processor -- start 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 1 -- start start 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 1 librados: setting wanted keys 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 1 librados: calling monclient init 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 10 monclient: init 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 10 monclient: _add_conns ranks=[4,2,1,3,7,0,5,6,8] 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 10 monclient(hunting): picked mon.noname-c con 0x7fee2c06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 10 monclient(hunting): picked mon.noname-b con 0x7fee2c0a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 10 monclient(hunting): picked mon.noname-f con 0x7fee2c0a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:54:16.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2c134c80 con 0x7fee2c0a5d50 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2c142e50 con 0x7fee2c0a5920 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2c0a0880 con 0x7fee2c06e7b0 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee33c9f640 10 monclient(hunting): authenticate will time out at 1650.275702s 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee3223c640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7fee2c0a5d50 legacy=0x7fee2c132cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee31a3b640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fee2c06e7b0 0x7fee2c131eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36670/0 (socket says 172.21.15.26:36670) 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee3123a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fee2c0a5920 0x7fee2c1325c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41334/0 (socket says 172.21.15.26:41334) 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.188+0000 7fee31a3b640 1 -- 172.21.15.26:0/1631202358 learned_addr learned my addr 172.21.15.26:0/1631202358 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.190+0000 7fee31a3b640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.26:6791/0 conn(0x7fee2c06e7b0 legacy=0x7fee2c131eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.190+0000 7fee3123a640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.26:6790/0 conn(0x7fee2c0a5920 legacy=0x7fee2c1325c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.190+0000 7fee2a7fc640 10 client.?.objecter ms_handle_connect 0x7fee2c06e7b0 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.190+0000 7fee2a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:16.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.190+0000 7fee2a7fc640 10 client.?.objecter ms_handle_connect 0x7fee2c0a5920 2024-08-19T12:54:16.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.190+0000 7fee2a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:16.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.389+0000 7fee3223c640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.104:6789/0 conn(0x7fee2c0a5d50 legacy=0x7fee2c132cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:54:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 125814931 0 0) 0x7fee2c09e3b0 con 0x7fee2c0a5920 2024-08-19T12:54:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): my global_id is 27263 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee18004eb0 con 0x7fee2c0a5920 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2622971544 0 0) 0x7fee2c0a0880 con 0x7fee2c06e7b0 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): my global_id is 184558 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee2c09e3b0 con 0x7fee2c06e7b0 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3483968770 0 0) 0x7fee2c09e3b0 con 0x7fee2c06e7b0 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee2c0a0880 con 0x7fee2c06e7b0 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee1c004b60 con 0x7fee2c06e7b0 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:16.413 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): dump: 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.414 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2126139507 0 0) 0x7fee18004eb0 con 0x7fee2c0a5920 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee2c09e3b0 con 0x7fee2c0a5920 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee20003b00 con 0x7fee2c0a5920 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.409+0000 7fee2a7fc640 10 monclient(hunting): dump: 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.415 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.416 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1682441129 0 0) 0x7fee2c0a0880 con 0x7fee2c06e7b0 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.26:6790/0 conn(0x7fee2c0a5920 legacy=0x7fee2c1325c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.104:6789/0 conn(0x7fee2c0a5d50 legacy=0x7fee2c132cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 1 monclient: found mon.c 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c09f640 con 0x7fee2c06e7b0 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:54:16.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee33c9f640 5 monclient: authenticate success, global_id 184558 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee33c9f640 10 monclient: _renew_subs 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee33c9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee33c9f640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fee2c1d72a0 con 0x7fee2c06e7b0 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee33c9f640 10 client.184558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee33c9f640 10 monclient: _renew_subs 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee33c9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee33c9f640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fee2c1d7650 con 0x7fee2c06e7b0 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee33c9f640 1 librados: init done 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee1c0050e0 con 0x7fee2c06e7b0 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2bfff640 4 set_mon_vals no callback set 2024-08-19T12:54:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee1c005610 con 0x7fee2c06e7b0 2024-08-19T12:54:16.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:54:16.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.410+0000 7fee2a7fc640 10 monclient: dump: 2024-08-19T12:54:16.419 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.419 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.419 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.419 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.419 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.419 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.419 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.419 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.419 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.420 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.420 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.420 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.420 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.420 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.420 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.420 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.411+0000 7fee33c9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:16.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.411+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fee1c0057b0 con 0x7fee2c06e7b0 2024-08-19T12:54:16.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.411+0000 7fee33c9f640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fee0c0065c0 con 0x7fee0c002860 2024-08-19T12:54:16.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.411+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(574..574 src has 350..574) ==== 5913+0+0 (unknown 1558564169 0 0) 0x7fee1c04e700 con 0x7fee2c06e7b0 2024-08-19T12:54:16.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.411+0000 7fee2a7fc640 10 client.184558.objecter ms_dispatch 0x7fee2c000b90 osd_map(574..574 src has 350..574) 2024-08-19T12:54:16.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.411+0000 7fee2a7fc640 3 client.184558.objecter handle_osd_map got epochs [574,574] > 0 2024-08-19T12:54:16.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.411+0000 7fee2a7fc640 3 client.184558.objecter handle_osd_map decoding full epoch 574 2024-08-19T12:54:16.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.411+0000 7fee2a7fc640 20 client.184558.objecter dump_active .. 0 homeless 2024-08-19T12:54:16.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.412+0000 7fee3223c640 10 monclient: get_auth_request con 0x7fee18041510 auth_method 0 2024-08-19T12:54:16.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.413+0000 7fee3123a640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.26:6790/0 conn(0x7fee0c002860 legacy=0x7fee0c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.413+0000 7fee2a7fc640 10 client.184558.objecter ms_handle_connect 0x7fee0c002860 2024-08-19T12:54:16.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.413+0000 7fee2a7fc640 10 client.184558.objecter resend_mon_ops 2024-08-19T12:54:16.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.413+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2455877048 0 0) 0x7fee0c0065c0 con 0x7fee0c002860 2024-08-19T12:54:16.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.413+0000 7fee2a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.413+0000 7fee2a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.413+0000 7fee2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.413+0000 7fee2a7fc640 10 monclient(hunting): my global_id is 27272 2024-08-19T12:54:16.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.413+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee2c0a0880 con 0x7fee0c002860 2024-08-19T12:54:16.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.413+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 424550670 0 0) 0x7fee2c0a0880 con 0x7fee0c002860 2024-08-19T12:54:16.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee0c0065c0 con 0x7fee0c002860 2024-08-19T12:54:16.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee20003fe0 con 0x7fee0c002860 2024-08-19T12:54:16.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 10 monclient: dump: 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.424 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 697129905 0 0) 0x7fee0c0065c0 con 0x7fee0c002860 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee0c00a160 con 0x7fee0c002860 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fee20004220 con 0x7fee0c002860 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.26:6790/0 conn(0x7fee0c002860 legacy=0x7fee0c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.26:6790/0 conn(0x7fee0c002860 legacy=0x7fee0c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.414+0000 7fee2a7fc640 10 client.184558.objecter ms_handle_connect 0x7fee18041510 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.420+0000 7fee33c9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:54:16.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.420+0000 7fee33c9f640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fee0c009e50 con 0x7fee0c009110 2024-08-19T12:54:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee3123a640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.26:6790/0 conn(0x7fee0c009110 legacy=0x7fee0c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 10 client.184558.objecter ms_handle_connect 0x7fee0c009110 2024-08-19T12:54:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 10 client.184558.objecter resend_mon_ops 2024-08-19T12:54:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2235695629 0 0) 0x7fee0c009e50 con 0x7fee0c009110 2024-08-19T12:54:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 10 monclient(hunting): my global_id is 27281 2024-08-19T12:54:16.427 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee0c0065c0 con 0x7fee0c009110 2024-08-19T12:54:16.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2378096242 0 0) 0x7fee0c0065c0 con 0x7fee0c009110 2024-08-19T12:54:16.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee0c009e50 con 0x7fee0c009110 2024-08-19T12:54:16.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fee20005b00 con 0x7fee0c009110 2024-08-19T12:54:16.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:54:16.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 10 monclient: dump: 2024-08-19T12:54:16.428 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.428 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.428 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.429 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1552101869 0 0) 0x7fee0c009e50 con 0x7fee0c009110 2024-08-19T12:54:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.423+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fee0c00c3f0 con 0x7fee0c009110 2024-08-19T12:54:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7701 (unknown 0 0 980615720) 0x7fee20007e20 con 0x7fee0c009110 2024-08-19T12:54:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee2a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:54:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee2a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:54:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.26:6790/0 conn(0x7fee0c009110 legacy=0x7fee0c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee2a7fc640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.26:6790/0 conn(0x7fee0c009110 legacy=0x7fee0c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:16.430 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:54:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:54:16.430 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:54:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:54:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 116, 2024-08-19T12:54:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 969478, 2024-08-19T12:54:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:54:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:54:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:54:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:54:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:54:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:54:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:54:16.431 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:54:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:54:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:54:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:54:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:54:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:16.432 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:54:16.433 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:54:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:16.434 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:54:16.435 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:16.436 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:54:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:54:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:54:16.437 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.438 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.440 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.441 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.442 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.444 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:54:16.445 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.446 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.447 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:16.450 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.451 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.451 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:54:16.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.451 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:16.451 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:16.451 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:54:16.451 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.451 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.451 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:54:16.451 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:54:16.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 10 librados: watch_flush enter 2024-08-19T12:54:16.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 10 librados: watch_flush exit 2024-08-19T12:54:16.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 20 client.184558.objecter shutdown clearing up homeless session... 2024-08-19T12:54:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 10 client.184558.objecter successfully canceled tick 2024-08-19T12:54:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.104:6800/899887506 conn(0x7fee18041510 legacy=0x7fee180439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 10 monclient: shutdown 2024-08-19T12:54:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 1 -- 172.21.15.26:0/1631202358 >> v1:172.21.15.26:6791/0 conn(0x7fee2c06e7b0 legacy=0x7fee2c131eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee3223c640 1 -- 172.21.15.26:0/1631202358 reap_dead start 2024-08-19T12:54:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee3223c640 1 -- 172.21.15.26:0/1631202358 reap_dead start 2024-08-19T12:54:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 1 -- 172.21.15.26:0/1631202358 shutdown_connections 2024-08-19T12:54:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 1 -- 172.21.15.26:0/1631202358 >> 172.21.15.26:0/1631202358 conn(0x7fee2c0a8740 msgr2=0x7fee2c06d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 1 -- 172.21.15.26:0/1631202358 shutdown_connections 2024-08-19T12:54:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 1 -- 172.21.15.26:0/1631202358 wait complete. 2024-08-19T12:54:16.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.424+0000 7fee33c9f640 1 librados: shutdown 2024-08-19T12:54:16.455 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:54:16.455 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-08-19T12:54:16.455 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:54:16.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.556+0000 7f7da0b98640 10 monclient: get_monmap_and_config 2024-08-19T12:54:16.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.556+0000 7f7da0b98640 10 monclient: build_initial_monmap 2024-08-19T12:54:16.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.556+0000 7f7da0b98640 10 monclient: monmap: 2024-08-19T12:54:16.557 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:16.557 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.557 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:54:16.557473+0000 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:54:16.557473+0000 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:16.558 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 1 Processor -- start 2024-08-19T12:54:16.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 1 -- start start 2024-08-19T12:54:16.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 10 monclient: init 2024-08-19T12:54:16.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:16.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 10 monclient: _add_conns ranks=[3,7,2,5,4,8,0,1,6] 2024-08-19T12:54:16.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 10 monclient(hunting): picked mon.noname-g con 0x7f7d9c06e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:54:16.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 10 monclient(hunting): picked mon.noname-i con 0x7f7d9c0a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:54:16.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 10 monclient(hunting): picked mon.noname-b con 0x7f7d9c0a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:54:16.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d9c136790 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d9c09ddf0 con 0x7f7d9c06e7b0 2024-08-19T12:54:16.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d9c09e0d0 con 0x7f7d9c0a5920 2024-08-19T12:54:16.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:16.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7da0b98640 10 monclient(hunting): authenticate will time out at 1650.644803s 2024-08-19T12:54:16.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7d9ad9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7d9c0a5d50 0x7f7d9c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41354/0 (socket says 172.21.15.26:41354) 2024-08-19T12:54:16.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.557+0000 7f7d9ad9d640 1 -- 172.21.15.26:0/2257757789 learned_addr learned my addr 172.21.15.26:0/2257757789 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:16.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d9ad9d640 1 -- 172.21.15.26:0/2257757789 >> v1:172.21.15.26:6790/0 conn(0x7f7d9c0a5d50 legacy=0x7f7d9c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 1 -- 172.21.15.26:0/2257757789 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2436498315 0 0) 0x7f7d9c136790 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient(hunting): my global_id is 27290 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 1 -- 172.21.15.26:0/2257757789 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d88003520 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d9a59c640 1 -- 172.21.15.26:0/2257757789 >> v1:172.21.15.104:6790/0 conn(0x7f7d9c06e7b0 legacy=0x7f7d9c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d91d9b640 1 -- 172.21.15.26:0/2257757789 >> v1:172.21.15.104:6792/0 conn(0x7f7d9c0a5920 legacy=0x7f7d9c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 1 -- 172.21.15.26:0/2257757789 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3778192769 0 0) 0x7f7d88003520 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 1 -- 172.21.15.26:0/2257757789 >> v1:172.21.15.104:6792/0 conn(0x7f7d9c0a5920 legacy=0x7f7d9c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 1 -- 172.21.15.26:0/2257757789 >> v1:172.21.15.104:6790/0 conn(0x7f7d9c06e7b0 legacy=0x7f7d9c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 1 monclient: found mon.noname-b 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:16.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:54:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 1 -- 172.21.15.26:0/2257757789 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9c09e3b0 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient: _finish_auth 0 2024-08-19T12:54:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:54:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7da0b98640 5 monclient: authenticate success, global_id 27290 2024-08-19T12:54:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 1 -- 172.21.15.26:0/2257757789 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7d94001e60 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7da0b98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:54:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient: dump: 2024-08-19T12:54:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.567 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7da0b98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 1 -- 172.21.15.26:0/2257757789 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d94004c20 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 1 -- 172.21.15.26:0/2257757789 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7d94005190 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7da0b98640 10 monclient: get_monmap_and_config success 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7da0b98640 4 set_mon_vals no callback set 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.560+0000 7f7d99d9b640 10 monclient: dump: 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.568 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.561+0000 7f7da0b98640 10 monclient: shutdown 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.561+0000 7f7da0b98640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.561+0000 7f7da0b98640 1 -- 172.21.15.26:0/2257757789 >> v1:172.21.15.26:6790/0 conn(0x7f7d9c0a5d50 legacy=0x7f7d9c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.561+0000 7f7da0b98640 1 -- 172.21.15.26:0/2257757789 shutdown_connections 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.561+0000 7f7da0b98640 1 -- 172.21.15.26:0/2257757789 >> 172.21.15.26:0/2257757789 conn(0x7f7d9c0a8740 msgr2=0x7f7d9c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.561+0000 7f7da0b98640 1 -- 172.21.15.26:0/2257757789 shutdown_connections 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.561+0000 7f7da0b98640 1 -- 172.21.15.26:0/2257757789 wait complete. 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.561+0000 7f7da0b98640 10 monclient: build_initial_monmap 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.561+0000 7f7da0b98640 10 monclient: monmap: 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:54:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.561+0000 7f7da0b98640 1 librados: starting msgr at 2024-08-19T12:54:16.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.561+0000 7f7da0b98640 1 librados: starting objecter 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 1 Processor -- start 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 1 -- start start 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 1 librados: setting wanted keys 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 1 librados: calling monclient init 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 10 monclient: init 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 10 monclient: _add_conns ranks=[4,3,8,2,5,6,7,0,1] 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 10 monclient(hunting): picked mon.noname-c con 0x7f7d9c06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 10 monclient(hunting): picked mon.noname-g con 0x7f7d9c0a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 10 monclient(hunting): picked mon.noname-e con 0x7f7d9c0a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d9c139270 con 0x7f7d9c0a5920 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d9c1349a0 con 0x7f7d9c06e7b0 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d9c1345e0 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:16.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7da0b98640 10 monclient(hunting): authenticate will time out at 1650.649763s 2024-08-19T12:54:16.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7d9a59c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f7d9c06e7b0 0x7f7d9c1327c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36676/0 (socket says 172.21.15.26:36676) 2024-08-19T12:54:16.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.562+0000 7f7d9a59c640 1 -- 172.21.15.26:0/2232604802 learned_addr learned my addr 172.21.15.26:0/2232604802 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:16.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d9a59c640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.26:6791/0 conn(0x7f7d9c06e7b0 legacy=0x7f7d9c1327c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d9ad9d640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.26:6793/0 conn(0x7f7d9c0a5d50 legacy=0x7f7d9c1d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 client.?.objecter ms_handle_connect 0x7f7d9c06e7b0 2024-08-19T12:54:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 client.?.objecter ms_handle_connect 0x7f7d9c0a5d50 2024-08-19T12:54:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 client.?.objecter ms_handle_connect 0x7f7d9c0a5920 2024-08-19T12:54:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d91d9b640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.104:6790/0 conn(0x7f7d9c0a5920 legacy=0x7f7d9c1d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d9a59c640 1 --1- 172.21.15.26:0/2232604802 >> v1:172.21.15.26:6791/0 conn(0x7f7d9c06e7b0 0x7f7d9c1327c0 :-1 s=OPENED pgs=3626 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f7d9c1349a0 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:54:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1425566352 0 0) 0x7f7d9c1345e0 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): my global_id is 184418 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d7c004eb0 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2637747625 0 0) 0x7f7d9c139270 con 0x7f7d9c0a5920 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): my global_id is 194250 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d9c1345e0 con 0x7f7d9c0a5920 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2250535892 0 0) 0x7f7d7c004eb0 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d9c139270 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7d940020b0 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): dump: 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.580 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4255790570 0 0) 0x7f7d9c1345e0 con 0x7f7d9c0a5920 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d7c004eb0 con 0x7f7d9c0a5920 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7d84001c80 con 0x7f7d9c0a5920 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient(hunting): dump: 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.582 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2999345099 0 0) 0x7f7d9c139270 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.26:6791/0 conn(0x7f7d9c06e7b0 legacy=0x7f7d9c1327c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.104:6790/0 conn(0x7f7d9c0a5920 legacy=0x7f7d9c1d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient: _finish_hunting 0 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 monclient: found mon.e 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9c13b5d0 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient: _finish_auth 0 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient: _check_auth_tickets 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 288107788 0 0) 0x7f7d9c1349a0 con 0x7f7d9c06e7b0 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7da0b98640 5 monclient: authenticate success, global_id 184418 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7da0b98640 10 monclient: _renew_subs 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7da0b98640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7da0b98640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7d9c139dd0 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7da0b98640 10 client.184418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7da0b98640 10 monclient: _renew_subs 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7da0b98640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7da0b98640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f7d9c1387d0 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7da0b98640 1 librados: init done 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d94004c20 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7d94005700 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d9959a640 4 set_mon_vals no callback set 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.565+0000 7f7d937fe640 10 monclient: dump: 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.585 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.566+0000 7f7da0b98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.566+0000 7f7da0b98640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7d800065c0 con 0x7f7d80002860 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.566+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f7d94011b60 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.566+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(574..574 src has 350..574) ==== 5913+0+0 (unknown 1558564169 0 0) 0x7f7d9404e5d0 con 0x7f7d9c0a5d50 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.566+0000 7f7d937fe640 10 client.184418.objecter ms_dispatch 0x7f7d9c000b90 osd_map(574..574 src has 350..574) 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.566+0000 7f7d937fe640 3 client.184418.objecter handle_osd_map got epochs [574,574] > 0 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.566+0000 7f7d937fe640 3 client.184418.objecter handle_osd_map decoding full epoch 574 2024-08-19T12:54:16.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.567+0000 7f7d937fe640 20 client.184418.objecter dump_active .. 0 homeless 2024-08-19T12:54:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.567+0000 7f7d91d9b640 10 monclient: get_auth_request con 0x7f7d7c041670 auth_method 0 2024-08-19T12:54:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.568+0000 7f7d937fe640 10 client.184418.objecter ms_handle_connect 0x7f7d80002860 2024-08-19T12:54:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.568+0000 7f7d9a59c640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.26:6791/0 conn(0x7f7d80002860 legacy=0x7f7d80004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.568+0000 7f7d937fe640 10 client.184418.objecter resend_mon_ops 2024-08-19T12:54:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.568+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 420754176 0 0) 0x7f7d800065c0 con 0x7f7d80002860 2024-08-19T12:54:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.568+0000 7f7d937fe640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.568+0000 7f7d937fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.568+0000 7f7d937fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.568+0000 7f7d937fe640 10 monclient(hunting): my global_id is 184576 2024-08-19T12:54:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.568+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d9c1349a0 con 0x7f7d80002860 2024-08-19T12:54:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3479491805 0 0) 0x7f7d9c1349a0 con 0x7f7d80002860 2024-08-19T12:54:16.587 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d800065c0 con 0x7f7d80002860 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7d8c003610 con 0x7f7d80002860 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 10 monclient: dump: 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.588 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1384827195 0 0) 0x7f7d800065c0 con 0x7f7d80002860 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d8000a160 con 0x7f7d80002860 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 10 client.184418.objecter ms_handle_connect 0x7f7d7c041670 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7d8c0057c0 con 0x7f7d80002860 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:54:16.589 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.26:6791/0 conn(0x7f7d80002860 legacy=0x7f7d80004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.569+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.26:6791/0 conn(0x7f7d80002860 legacy=0x7f7d80004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:16.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.575+0000 7f7da0b98640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:54:16.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.575+0000 7f7da0b98640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7d80009e70 con 0x7f7d80009130 2024-08-19T12:54:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 10 client.184418.objecter ms_handle_connect 0x7f7d80009130 2024-08-19T12:54:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d9a59c640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.26:6791/0 conn(0x7f7d80009130 legacy=0x7f7d800095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 10 client.184418.objecter resend_mon_ops 2024-08-19T12:54:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2386821800 0 0) 0x7f7d80009e70 con 0x7f7d80009130 2024-08-19T12:54:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 10 monclient(hunting): my global_id is 184585 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d800065c0 con 0x7f7d80009130 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2039493962 0 0) 0x7f7d800065c0 con 0x7f7d80009130 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d80009e70 con 0x7f7d80009130 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7d8c003e10 con 0x7f7d80009130 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 10 monclient: dump: 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.592 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2982920880 0 0) 0x7f7d80009e70 con 0x7f7d80009130 2024-08-19T12:54:16.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7d8000c410 con 0x7f7d80009130 2024-08-19T12:54:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7701 (unknown 0 0 4060486563) 0x7f7d8c004180 con 0x7f7d80009130 2024-08-19T12:54:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:54:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:54:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.26:6791/0 conn(0x7f7d80009130 legacy=0x7f7d800095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7d937fe640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.26:6791/0 conn(0x7f7d80009130 legacy=0x7f7d800095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:16.594 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:54:16.594 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:54:16.594 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:54:16.594 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:54:16.594 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 116, 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 182044, 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:54:16.595 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:16.596 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.597 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:54:16.597 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:54:16.597 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:16.597 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:16.597 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:16.597 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:16.597 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:16.597 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:16.597 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:16.597 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:16.597 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:16.597 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:54:16.598 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:16.599 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.600 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.601 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:54:16.601 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.601 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.601 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:54:16.601 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:54:16.601 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.602 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.602 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.602 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:54:16.602 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:54:16.602 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.602 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.602 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.602 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:54:16.602 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:54:16.603 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.604 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.606 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:54:16.607 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.608 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.609 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:54:16.612 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.613 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:16.614 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.615 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:54:16.615 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:54:16.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.578+0000 7f7da0b98640 10 librados: watch_flush enter 2024-08-19T12:54:16.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7da0b98640 10 librados: watch_flush exit 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7da0b98640 20 client.184418.objecter shutdown clearing up homeless session... 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7da0b98640 10 client.184418.objecter successfully canceled tick 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7da0b98640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.104:6800/899887506 conn(0x7f7d7c041670 legacy=0x7f7d7c043b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7da0b98640 10 monclient: shutdown 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7da0b98640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7da0b98640 1 -- 172.21.15.26:0/2232604802 >> v1:172.21.15.26:6793/0 conn(0x7f7d9c0a5d50 legacy=0x7f7d9c1d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7d9ad9d640 1 -- 172.21.15.26:0/2232604802 reap_dead start 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7d9ad9d640 1 -- 172.21.15.26:0/2232604802 reap_dead start 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7da0b98640 1 -- 172.21.15.26:0/2232604802 shutdown_connections 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7da0b98640 1 -- 172.21.15.26:0/2232604802 >> 172.21.15.26:0/2232604802 conn(0x7f7d9c0a8740 msgr2=0x7f7d9c135120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7da0b98640 1 -- 172.21.15.26:0/2232604802 shutdown_connections 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7da0b98640 1 -- 172.21.15.26:0/2232604802 wait complete. 2024-08-19T12:54:16.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.579+0000 7f7da0b98640 1 librados: shutdown 2024-08-19T12:54:16.618 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:54:16.618 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon d 2024-08-19T12:54:16.618 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:54:16.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.717+0000 7f51f7620640 10 monclient: get_monmap_and_config 2024-08-19T12:54:16.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.717+0000 7f51f7620640 10 monclient: build_initial_monmap 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.717+0000 7f51f7620640 10 monclient: monmap: 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:54:16.718797+0000 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:54:16.718797+0000 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:16.719 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:16.720 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:16.720 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:16.720 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:16.720 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.718+0000 7f51f7620640 1 Processor -- start 2024-08-19T12:54:16.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.718+0000 7f51f7620640 1 -- start start 2024-08-19T12:54:16.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.718+0000 7f51f7620640 10 monclient: init 2024-08-19T12:54:16.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.719+0000 7f51f7620640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:16.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.719+0000 7f51f7620640 10 monclient: _add_conns ranks=[2,4,5,1,6,8,0,3,7] 2024-08-19T12:54:16.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.719+0000 7f51f7620640 10 monclient(hunting): picked mon.noname-b con 0x7f51f006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:54:16.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.719+0000 7f51f7620640 10 monclient(hunting): picked mon.noname-c con 0x7f51f00a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:54:16.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.719+0000 7f51f7620640 10 monclient(hunting): picked mon.noname-h con 0x7f51f00a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:54:16.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.719+0000 7f51f7620640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51f0136790 con 0x7f51f006e7b0 2024-08-19T12:54:16.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.719+0000 7f51f7620640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51f009ddf0 con 0x7f51f00a5920 2024-08-19T12:54:16.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.719+0000 7f51f7620640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51f009e0d0 con 0x7f51f00a5d50 2024-08-19T12:54:16.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.719+0000 7f51f7620640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:16.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.719+0000 7f51f7620640 10 monclient(hunting): authenticate will time out at 1650.806302s 2024-08-19T12:54:16.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.719+0000 7f51f53bc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f51f006e7b0 0x7f51f0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41358/0 (socket says 172.21.15.26:41358) 2024-08-19T12:54:16.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.719+0000 7f51f53bc640 1 -- 172.21.15.26:0/1617346550 learned_addr learned my addr 172.21.15.26:0/1617346550 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:16.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51f4bbb640 1 -- 172.21.15.26:0/1617346550 >> v1:172.21.15.26:6791/0 conn(0x7f51f00a5920 legacy=0x7f51f013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51f53bc640 1 -- 172.21.15.26:0/1617346550 >> v1:172.21.15.26:6790/0 conn(0x7f51f006e7b0 legacy=0x7f51f0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 1 -- 172.21.15.26:0/1617346550 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867508928 0 0) 0x7f51e4001820 con 0x7f51f00a5920 2024-08-19T12:54:16.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 10 monclient(hunting): my global_id is 184594 2024-08-19T12:54:16.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 1 -- 172.21.15.26:0/1617346550 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51d8003520 con 0x7f51f00a5920 2024-08-19T12:54:16.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 1 -- 172.21.15.26:0/1617346550 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1781310132 0 0) 0x7f51f0136790 con 0x7f51f006e7b0 2024-08-19T12:54:16.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.726 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 10 monclient(hunting): my global_id is 27299 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 1 -- 172.21.15.26:0/1617346550 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51e4001820 con 0x7f51f006e7b0 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51f5bbd640 1 -- 172.21.15.26:0/1617346550 >> v1:172.21.15.104:6791/0 conn(0x7f51f00a5d50 legacy=0x7f51f0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 1 -- 172.21.15.26:0/1617346550 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3379970373 0 0) 0x7f51d8003520 con 0x7f51f00a5920 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 1 -- 172.21.15.26:0/1617346550 >> v1:172.21.15.104:6791/0 conn(0x7f51f00a5d50 legacy=0x7f51f0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 1 -- 172.21.15.26:0/1617346550 >> v1:172.21.15.26:6790/0 conn(0x7f51f006e7b0 legacy=0x7f51f0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 10 monclient: _finish_hunting 0 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 1 monclient: found mon.noname-c 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 1 -- 172.21.15.26:0/1617346550 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f009e3b0 con 0x7f51f00a5920 2024-08-19T12:54:16.727 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.721+0000 7f51effff640 10 monclient: _finish_auth 0 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51effff640 10 monclient: _check_auth_tickets 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51effff640 1 -- 172.21.15.26:0/1617346550 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f51e4002080 con 0x7f51f00a5920 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51f7620640 5 monclient: authenticate success, global_id 184594 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51f7620640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51effff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51effff640 10 monclient: dump: 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.728 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51f7620640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51effff640 1 -- 172.21.15.26:0/1617346550 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f51e4003a80 con 0x7f51f00a5920 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51effff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:16.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51f7620640 10 monclient: get_monmap_and_config success 2024-08-19T12:54:16.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51effff640 1 -- 172.21.15.26:0/1617346550 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f51e4003fb0 con 0x7f51f00a5920 2024-08-19T12:54:16.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51f7620640 4 set_mon_vals no callback set 2024-08-19T12:54:16.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51effff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:54:16.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51effff640 10 monclient: dump: 2024-08-19T12:54:16.730 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.730 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.730 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.730 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51f7620640 10 monclient: shutdown 2024-08-19T12:54:16.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51f7620640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51f7620640 1 -- 172.21.15.26:0/1617346550 >> v1:172.21.15.26:6791/0 conn(0x7f51f00a5920 legacy=0x7f51f013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51f7620640 1 -- 172.21.15.26:0/1617346550 shutdown_connections 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51f7620640 1 -- 172.21.15.26:0/1617346550 >> 172.21.15.26:0/1617346550 conn(0x7f51f00a8740 msgr2=0x7f51f00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51f7620640 1 -- 172.21.15.26:0/1617346550 shutdown_connections 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.722+0000 7f51f7620640 1 -- 172.21.15.26:0/1617346550 wait complete. 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.723+0000 7f51f7620640 10 monclient: build_initial_monmap 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.723+0000 7f51f7620640 10 monclient: monmap: 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.732 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.723+0000 7f51f7620640 1 librados: starting msgr at 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.723+0000 7f51f7620640 1 librados: starting objecter 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.723+0000 7f51f7620640 1 Processor -- start 2024-08-19T12:54:16.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.723+0000 7f51f7620640 1 -- start start 2024-08-19T12:54:16.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.723+0000 7f51f7620640 1 librados: setting wanted keys 2024-08-19T12:54:16.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.723+0000 7f51f7620640 1 librados: calling monclient init 2024-08-19T12:54:16.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.723+0000 7f51f7620640 10 monclient: init 2024-08-19T12:54:16.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.724+0000 7f51f7620640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:16.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.724+0000 7f51f7620640 10 monclient: _add_conns ranks=[2,8,5,1,4,7,0,6,3] 2024-08-19T12:54:16.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.724+0000 7f51f7620640 10 monclient(hunting): picked mon.noname-b con 0x7f51f006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:54:16.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.724+0000 7f51f7620640 10 monclient(hunting): picked mon.noname-e con 0x7f51f00a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:54:16.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.724+0000 7f51f7620640 10 monclient(hunting): picked mon.noname-h con 0x7f51f00a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:54:16.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.724+0000 7f51f7620640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51f0123be0 con 0x7f51f006e7b0 2024-08-19T12:54:16.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.724+0000 7f51f7620640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51f01346c0 con 0x7f51f00a5d50 2024-08-19T12:54:16.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.724+0000 7f51f7620640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51f0125680 con 0x7f51f00a5920 2024-08-19T12:54:16.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.724+0000 7f51f7620640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:16.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.724+0000 7f51f7620640 10 monclient(hunting): authenticate will time out at 1650.811099s 2024-08-19T12:54:16.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.724+0000 7f51f53bc640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f51f006e7b0 0x7f51f0141eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41364/0 (socket says 172.21.15.26:41364) 2024-08-19T12:54:16.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.724+0000 7f51f53bc640 1 -- 172.21.15.26:0/3655268156 learned_addr learned my addr 172.21.15.26:0/3655268156 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:16.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51f53bc640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.26:6790/0 conn(0x7f51f006e7b0 legacy=0x7f51f0141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51f4bbb640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.26:6793/0 conn(0x7f51f00a5920 legacy=0x7f51f01d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 client.?.objecter ms_handle_connect 0x7f51f006e7b0 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 client.?.objecter ms_handle_connect 0x7f51f00a5920 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 959532715 0 0) 0x7f51f0123be0 con 0x7f51f006e7b0 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): my global_id is 27308 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51dc004eb0 con 0x7f51f006e7b0 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2716771283 0 0) 0x7f51f0125680 con 0x7f51f00a5920 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): my global_id is 184427 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51f0123be0 con 0x7f51f00a5920 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51f5bbd640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.104:6791/0 conn(0x7f51f00a5d50 legacy=0x7f51f01d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 client.?.objecter ms_handle_connect 0x7f51f00a5d50 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1788159908 0 0) 0x7f51dc004eb0 con 0x7f51f006e7b0 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51f0125680 con 0x7f51f006e7b0 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f51e80047e0 con 0x7f51f006e7b0 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): dump: 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.741 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.742 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3950886344 0 0) 0x7f51f0123be0 con 0x7f51f00a5920 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51dc004eb0 con 0x7f51f00a5920 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f51e4003f70 con 0x7f51f00a5920 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): dump: 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.743 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1870241056 0 0) 0x7f51f01346c0 con 0x7f51f00a5d50 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient(hunting): my global_id is 184478 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51f0123be0 con 0x7f51f00a5d50 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3473148303 0 0) 0x7f51f0125680 con 0x7f51f006e7b0 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.26:6793/0 conn(0x7f51f00a5920 legacy=0x7f51f01d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.104:6791/0 conn(0x7f51f00a5d50 legacy=0x7f51f01d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient: _finish_hunting 0 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 monclient: found mon.b 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f0125f20 con 0x7f51f006e7b0 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient: _finish_auth 0 2024-08-19T12:54:16.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 10 monclient: _check_auth_tickets 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51f7620640 5 monclient: authenticate success, global_id 27308 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51f7620640 10 monclient: _renew_subs 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.726+0000 7f51f7620640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51f7620640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f51f0124720 con 0x7f51f006e7b0 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f51e8004c30 con 0x7f51f006e7b0 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51edffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f51e8005120 con 0x7f51f006e7b0 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51edffb640 10 monclient: dump: 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.746 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.747 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51f7620640 10 client.27308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51f7620640 10 monclient: _renew_subs 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51f7620640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51ef7fe640 4 set_mon_vals no callback set 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51f7620640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f51f01232a0 con 0x7f51f006e7b0 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51f7620640 1 librados: init done 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.727+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f51e8011580 con 0x7f51f006e7b0 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.728+0000 7f51f7620640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.728+0000 7f51f7620640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f51f0136740 con 0x7f51f002e340 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.728+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(574..574 src has 350..574) ==== 5913+0+0 (unknown 1558564169 0 0) 0x7f51e804e2b0 con 0x7f51f006e7b0 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.728+0000 7f51edffb640 10 client.27308.objecter ms_dispatch 0x7f51f0000b90 osd_map(574..574 src has 350..574) 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.728+0000 7f51edffb640 3 client.27308.objecter handle_osd_map got epochs [574,574] > 0 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.728+0000 7f51edffb640 3 client.27308.objecter handle_osd_map decoding full epoch 574 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.728+0000 7f51edffb640 20 client.27308.objecter dump_active .. 0 homeless 2024-08-19T12:54:16.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.728+0000 7f51f5bbd640 10 monclient: get_auth_request con 0x7f51dc041680 auth_method 0 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 10 client.27308.objecter ms_handle_connect 0x7f51f002e340 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 10 client.27308.objecter resend_mon_ops 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51f4bbb640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.26:6792/0 conn(0x7f51f002e340 legacy=0x7f51f0063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2743265399 0 0) 0x7f51dc004eb0 con 0x7f51f002e340 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 10 monclient(hunting): my global_id is 194244 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51f0125680 con 0x7f51f002e340 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3503520049 0 0) 0x7f51f0125680 con 0x7f51f002e340 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51dc004eb0 con 0x7f51f002e340 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f51e4004630 con 0x7f51f002e340 2024-08-19T12:54:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 10 monclient: dump: 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.750 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 10 client.27308.objecter ms_handle_connect 0x7f51dc041680 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3525254025 0 0) 0x7f51dc004eb0 con 0x7f51f002e340 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.730+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51f0134a80 con 0x7f51f002e340 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.731+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f51e40017e0 con 0x7f51f002e340 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.731+0000 7f51edffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.731+0000 7f51edffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.731+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.26:6792/0 conn(0x7f51f002e340 legacy=0x7f51f0063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.731+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.26:6792/0 conn(0x7f51f002e340 legacy=0x7f51f0063a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:16.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.737+0000 7f51f7620640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:54:16.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.737+0000 7f51f7620640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f51f00a1fe0 con 0x7f51f01425c0 2024-08-19T12:54:16.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.739+0000 7f51f4bbb640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.26:6792/0 conn(0x7f51f01425c0 legacy=0x7f51f00a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.739+0000 7f51edffb640 10 client.27308.objecter ms_handle_connect 0x7f51f01425c0 2024-08-19T12:54:16.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.739+0000 7f51edffb640 10 client.27308.objecter resend_mon_ops 2024-08-19T12:54:16.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2352083800 0 0) 0x7f51f00a1fe0 con 0x7f51f01425c0 2024-08-19T12:54:16.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 10 monclient(hunting): my global_id is 194253 2024-08-19T12:54:16.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51dc004eb0 con 0x7f51f01425c0 2024-08-19T12:54:16.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2550101678 0 0) 0x7f51dc004eb0 con 0x7f51f01425c0 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51f00a1fe0 con 0x7f51f01425c0 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f51e4005e80 con 0x7f51f01425c0 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 10 monclient: dump: 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.754 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1039978753 0 0) 0x7f51f00a1fe0 con 0x7f51f01425c0 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f51f002e720 con 0x7f51f01425c0 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 3142947951) 0x7f51e4007d80 con 0x7f51f01425c0 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:54:16.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.26:6792/0 conn(0x7f51f01425c0 legacy=0x7f51f00a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51edffb640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.26:6792/0 conn(0x7f51f01425c0 legacy=0x7f51f00a18d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:16.756 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:54:16.756 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:54:16.756 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:54:16.756 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:54:16.756 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 116, 2024-08-19T12:54:16.756 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 41668, 2024-08-19T12:54:16.756 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:54:16.756 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:54:16.756 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:16.757 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:16.758 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:54:16.759 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:16.760 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:54:16.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:54:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.765 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.767 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:54:16.768 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:54:16.771 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:16.774 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:16.775 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.775 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.775 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:54:16.775 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:54:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51f7620640 10 librados: watch_flush enter 2024-08-19T12:54:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51f7620640 10 librados: watch_flush exit 2024-08-19T12:54:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51f7620640 20 client.27308.objecter shutdown clearing up homeless session... 2024-08-19T12:54:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.740+0000 7f51f7620640 10 client.27308.objecter successfully canceled tick 2024-08-19T12:54:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.741+0000 7f51f7620640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.104:6800/899887506 conn(0x7f51dc041680 legacy=0x7f51dc043b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.741+0000 7f51f7620640 10 monclient: shutdown 2024-08-19T12:54:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.741+0000 7f51f7620640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.741+0000 7f51f7620640 1 -- 172.21.15.26:0/3655268156 >> v1:172.21.15.26:6790/0 conn(0x7f51f006e7b0 legacy=0x7f51f0141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.741+0000 7f51f5bbd640 1 -- 172.21.15.26:0/3655268156 reap_dead start 2024-08-19T12:54:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.741+0000 7f51f5bbd640 1 -- 172.21.15.26:0/3655268156 reap_dead start 2024-08-19T12:54:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.741+0000 7f51f7620640 1 -- 172.21.15.26:0/3655268156 shutdown_connections 2024-08-19T12:54:16.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.741+0000 7f51f7620640 1 -- 172.21.15.26:0/3655268156 >> 172.21.15.26:0/3655268156 conn(0x7f51f00a8740 msgr2=0x7f51f006dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:16.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.741+0000 7f51f7620640 1 -- 172.21.15.26:0/3655268156 shutdown_connections 2024-08-19T12:54:16.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.741+0000 7f51f7620640 1 -- 172.21.15.26:0/3655268156 wait complete. 2024-08-19T12:54:16.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.741+0000 7f51f7620640 1 librados: shutdown 2024-08-19T12:54:16.778 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:54:16.778 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-08-19T12:54:16.778 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:54:16.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.877+0000 7f2c4ffde640 10 monclient: get_monmap_and_config 2024-08-19T12:54:16.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.877+0000 7f2c4ffde640 10 monclient: build_initial_monmap 2024-08-19T12:54:16.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.877+0000 7f2c4ffde640 10 monclient: monmap: 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:54:16.878245+0000 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:54:16.878245+0000 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:16.879 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:16.880 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:16.880 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.877+0000 7f2c4ffde640 1 Processor -- start 2024-08-19T12:54:16.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4ffde640 1 -- start start 2024-08-19T12:54:16.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4ffde640 10 monclient: init 2024-08-19T12:54:16.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4ffde640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:16.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4ffde640 10 monclient: _add_conns ranks=[2,6,0,5,7,4,3,8,1] 2024-08-19T12:54:16.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4ffde640 10 monclient(hunting): picked mon.noname-b con 0x7f2c4806e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:54:16.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4ffde640 10 monclient(hunting): picked mon.noname-d con 0x7f2c480a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:54:16.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4ffde640 10 monclient(hunting): picked mon.noname-a con 0x7f2c480a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4ffde640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c48136790 con 0x7f2c480a5d50 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4ffde640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c4809ddf0 con 0x7f2c4806e7b0 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4ffde640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c4809e0d0 con 0x7f2c480a5920 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4ffde640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4ffde640 10 monclient(hunting): authenticate will time out at 1650.965693s 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4d579640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f2c480a5920 0x7f2c4813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:42772/0 (socket says 172.21.15.26:42772) 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4dd7a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2c4806e7b0 0x7f2c48136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41372/0 (socket says 172.21.15.26:41372) 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4e57b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2c480a5d50 0x7f2c48141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53888/0 (socket says 172.21.15.26:53888) 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.878+0000 7f2c4d579640 1 -- 172.21.15.26:0/950419823 learned_addr learned my addr 172.21.15.26:0/950419823 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.880+0000 7f2c4d579640 1 -- 172.21.15.26:0/950419823 >> v1:172.21.15.26:6792/0 conn(0x7f2c480a5920 legacy=0x7f2c4813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.880+0000 7f2c4e57b640 1 -- 172.21.15.26:0/950419823 >> v1:172.21.15.26:6789/0 conn(0x7f2c480a5d50 legacy=0x7f2c48141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4dd7a640 1 -- 172.21.15.26:0/950419823 >> v1:172.21.15.26:6790/0 conn(0x7f2c4806e7b0 legacy=0x7f2c48136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 846735297 0 0) 0x7f2c4809e0d0 con 0x7f2c480a5920 2024-08-19T12:54:16.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient(hunting): my global_id is 194262 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c30003520 con 0x7f2c480a5920 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 578723966 0 0) 0x7f2c48136790 con 0x7f2c480a5d50 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient(hunting): my global_id is 194310 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c4809e0d0 con 0x7f2c480a5d50 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2241014697 0 0) 0x7f2c38001700 con 0x7f2c4806e7b0 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient(hunting): my global_id is 27317 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c48136790 con 0x7f2c4806e7b0 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3693480520 0 0) 0x7f2c4809e0d0 con 0x7f2c480a5d50 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 >> v1:172.21.15.26:6792/0 conn(0x7f2c480a5920 legacy=0x7f2c4813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 >> v1:172.21.15.26:6790/0 conn(0x7f2c4806e7b0 legacy=0x7f2c48136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient: _finish_hunting 0 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 monclient: found mon.noname-a 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4809e3b0 con 0x7f2c480a5d50 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient: _finish_auth 0 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient: _check_auth_tickets 2024-08-19T12:54:16.886 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c40001ac0 con 0x7f2c480a5d50 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4ffde640 5 monclient: authenticate success, global_id 194310 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4ffde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient: dump: 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.887 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4ffde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c400047e0 con 0x7f2c480a5d50 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 1 -- 172.21.15.26:0/950419823 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c40004d50 con 0x7f2c480a5d50 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4ffde640 10 monclient: get_monmap_and_config success 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4ffde640 4 set_mon_vals no callback set 2024-08-19T12:54:16.888 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.881+0000 7f2c4cd78640 10 monclient: dump: 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.889 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.882+0000 7f2c4ffde640 10 monclient: shutdown 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.882+0000 7f2c4ffde640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.882+0000 7f2c4ffde640 1 -- 172.21.15.26:0/950419823 >> v1:172.21.15.26:6789/0 conn(0x7f2c480a5d50 legacy=0x7f2c48141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.882+0000 7f2c4ffde640 1 -- 172.21.15.26:0/950419823 shutdown_connections 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.882+0000 7f2c4ffde640 1 -- 172.21.15.26:0/950419823 >> 172.21.15.26:0/950419823 conn(0x7f2c480a8740 msgr2=0x7f2c480a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.882+0000 7f2c4ffde640 1 -- 172.21.15.26:0/950419823 shutdown_connections 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.882+0000 7f2c4ffde640 1 -- 172.21.15.26:0/950419823 wait complete. 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.882+0000 7f2c4ffde640 10 monclient: build_initial_monmap 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.882+0000 7f2c4ffde640 10 monclient: monmap: 2024-08-19T12:54:16.890 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:16.891 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.882+0000 7f2c4ffde640 1 librados: starting msgr at 2024-08-19T12:54:16.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.882+0000 7f2c4ffde640 1 librados: starting objecter 2024-08-19T12:54:16.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 1 Processor -- start 2024-08-19T12:54:16.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 1 -- start start 2024-08-19T12:54:16.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 1 librados: setting wanted keys 2024-08-19T12:54:16.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 1 librados: calling monclient init 2024-08-19T12:54:16.892 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 10 monclient: init 2024-08-19T12:54:16.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:16.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 10 monclient: _add_conns ranks=[4,6,5,8,2,1,7,3,0] 2024-08-19T12:54:16.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 10 monclient(hunting): picked mon.noname-c con 0x7f2c4806e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:54:16.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 10 monclient(hunting): picked mon.noname-d con 0x7f2c480a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 10 monclient(hunting): picked mon.noname-h con 0x7f2c480a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c4813bd40 con 0x7f2c4806e7b0 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c48139000 con 0x7f2c480a5d50 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c48139b60 con 0x7f2c480a5920 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4ffde640 10 monclient(hunting): authenticate will time out at 1650.970667s 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4dd7a640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f2c4806e7b0 0x7f2c481ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36724/0 (socket says 172.21.15.26:36724) 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.883+0000 7f2c4dd7a640 1 -- 172.21.15.26:0/1192924518 learned_addr learned my addr 172.21.15.26:0/1192924518 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.885+0000 7f2c4d579640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.26:6792/0 conn(0x7f2c480a5920 legacy=0x7f2c481d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.885+0000 7f2c4dd7a640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.26:6791/0 conn(0x7f2c4806e7b0 legacy=0x7f2c481ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.885+0000 7f2c467fc640 10 client.?.objecter ms_handle_connect 0x7f2c480a5920 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.885+0000 7f2c467fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:16.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.885+0000 7f2c467fc640 10 client.?.objecter ms_handle_connect 0x7f2c4806e7b0 2024-08-19T12:54:16.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.885+0000 7f2c467fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:16.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1368165839 0 0) 0x7f2c48139b60 con 0x7f2c480a5920 2024-08-19T12:54:16.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): my global_id is 194271 2024-08-19T12:54:16.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c34004eb0 con 0x7f2c480a5920 2024-08-19T12:54:16.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1440088013 0 0) 0x7f2c48136790 con 0x7f2c4806e7b0 2024-08-19T12:54:16.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.899 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): my global_id is 184603 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c48139b60 con 0x7f2c4806e7b0 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 client.?.objecter ms_handle_connect 0x7f2c480a5d50 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c4e57b640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.104:6791/0 conn(0x7f2c480a5d50 legacy=0x7f2c481d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 709996106 0 0) 0x7f2c34004eb0 con 0x7f2c480a5920 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c48136790 con 0x7f2c480a5920 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c3c0047e0 con 0x7f2c480a5920 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): dump: 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.900 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.901 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4216939898 0 0) 0x7f2c48139b60 con 0x7f2c4806e7b0 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c34004eb0 con 0x7f2c4806e7b0 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c38004b60 con 0x7f2c4806e7b0 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): dump: 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.902 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4292705489 0 0) 0x7f2c48139000 con 0x7f2c480a5d50 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient(hunting): my global_id is 184487 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c48139b60 con 0x7f2c480a5d50 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 92987810 0 0) 0x7f2c48136790 con 0x7f2c480a5920 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.104:6791/0 conn(0x7f2c480a5d50 legacy=0x7f2c481d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.26:6791/0 conn(0x7f2c4806e7b0 legacy=0x7f2c481ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient: _finish_hunting 0 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 monclient: found mon.d 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4813b360 con 0x7f2c480a5920 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient: _finish_auth 0 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient: _check_auth_tickets 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c4ffde640 5 monclient: authenticate success, global_id 194271 2024-08-19T12:54:16.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c4ffde640 10 monclient: _renew_subs 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c4ffde640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c4ffde640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2c4813aac0 con 0x7f2c480a5920 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c4ffde640 10 client.194271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c4ffde640 10 monclient: _renew_subs 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c4ffde640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c4ffde640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f2c4813a790 con 0x7f2c480a5920 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c4ffde640 1 librados: init done 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c3c004c30 con 0x7f2c480a5920 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c3c005160 con 0x7f2c480a5920 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c47fff640 4 set_mon_vals no callback set 2024-08-19T12:54:16.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.886+0000 7f2c467fc640 10 monclient: dump: 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.906 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.887+0000 7f2c4ffde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.887+0000 7f2c4ffde640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2c280065c0 con 0x7f2c28002860 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.887+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2c3c005300 con 0x7f2c480a5920 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.887+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(574..574 src has 350..574) ==== 5913+0+0 (unknown 1558564169 0 0) 0x7f2c3c04e360 con 0x7f2c480a5920 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.887+0000 7f2c467fc640 10 client.194271.objecter ms_dispatch 0x7f2c48000b90 osd_map(574..574 src has 350..574) 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.887+0000 7f2c467fc640 3 client.194271.objecter handle_osd_map got epochs [574,574] > 0 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.887+0000 7f2c467fc640 3 client.194271.objecter handle_osd_map decoding full epoch 574 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.887+0000 7f2c467fc640 20 client.194271.objecter dump_active .. 0 homeless 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.888+0000 7f2c4e57b640 10 monclient: get_auth_request con 0x7f2c340416b0 auth_method 0 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.889+0000 7f2c4dd7a640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.26:6793/0 conn(0x7f2c28002860 legacy=0x7f2c28004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.889+0000 7f2c467fc640 10 client.194271.objecter ms_handle_connect 0x7f2c28002860 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.889+0000 7f2c467fc640 10 client.194271.objecter resend_mon_ops 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.889+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 393444020 0 0) 0x7f2c280065c0 con 0x7f2c28002860 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.889+0000 7f2c467fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.889+0000 7f2c467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.889+0000 7f2c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.889+0000 7f2c467fc640 10 monclient(hunting): my global_id is 184436 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.889+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c48136790 con 0x7f2c28002860 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2556757134 0 0) 0x7f2c48136790 con 0x7f2c28002860 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c280065c0 con 0x7f2c28002860 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c380052b0 con 0x7f2c28002860 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 10 monclient: dump: 2024-08-19T12:54:16.908 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.909 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.910 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1793149773 0 0) 0x7f2c280065c0 con 0x7f2c28002860 2024-08-19T12:54:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c2800a160 con 0x7f2c28002860 2024-08-19T12:54:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 10 client.194271.objecter ms_handle_connect 0x7f2c340416b0 2024-08-19T12:54:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2c38005b90 con 0x7f2c28002860 2024-08-19T12:54:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:54:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.26:6793/0 conn(0x7f2c28002860 legacy=0x7f2c28004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.890+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.26:6793/0 conn(0x7f2c28002860 legacy=0x7f2c28004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.896+0000 7f2c4ffde640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:54:16.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.896+0000 7f2c4ffde640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2c28009e50 con 0x7f2c28009110 2024-08-19T12:54:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 10 client.194271.objecter ms_handle_connect 0x7f2c28009110 2024-08-19T12:54:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c4dd7a640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.26:6793/0 conn(0x7f2c28009110 legacy=0x7f2c280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 10 client.194271.objecter resend_mon_ops 2024-08-19T12:54:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1906256267 0 0) 0x7f2c28009e50 con 0x7f2c28009110 2024-08-19T12:54:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 10 monclient(hunting): my global_id is 184445 2024-08-19T12:54:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c280065c0 con 0x7f2c28009110 2024-08-19T12:54:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3153771251 0 0) 0x7f2c280065c0 con 0x7f2c28009110 2024-08-19T12:54:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c28009e50 con 0x7f2c28009110 2024-08-19T12:54:16.912 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c38005850 con 0x7f2c28009110 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 10 monclient: dump: 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:16.913 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4180345686 0 0) 0x7f2c28009e50 con 0x7f2c28009110 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.899+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2c2800c3f0 con 0x7f2c28009110 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7701 (unknown 0 0 392395734) 0x7f2c38006080 con 0x7f2c28009110 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c467fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c467fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.26:6793/0 conn(0x7f2c28009110 legacy=0x7f2c280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c467fc640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.26:6793/0 conn(0x7f2c28009110 legacy=0x7f2c280095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:16.915 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:54:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:54:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:54:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:54:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 116, 2024-08-19T12:54:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 182366, 2024-08-19T12:54:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:54:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:54:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:54:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:54:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:54:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:54:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:54:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:54:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:54:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:54:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:54:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:54:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:54:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:54:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:54:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:54:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:54:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:54:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:54:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:54:16.935 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:54:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 10 librados: watch_flush enter 2024-08-19T12:54:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 10 librados: watch_flush exit 2024-08-19T12:54:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 20 client.194271.objecter shutdown clearing up homeless session... 2024-08-19T12:54:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 10 client.194271.objecter successfully canceled tick 2024-08-19T12:54:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.104:6800/899887506 conn(0x7f2c340416b0 legacy=0x7f2c34043b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 10 monclient: shutdown 2024-08-19T12:54:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 1 -- 172.21.15.26:0/1192924518 >> v1:172.21.15.26:6792/0 conn(0x7f2c480a5920 legacy=0x7f2c481d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4e57b640 1 -- 172.21.15.26:0/1192924518 reap_dead start 2024-08-19T12:54:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4e57b640 1 -- 172.21.15.26:0/1192924518 reap_dead start 2024-08-19T12:54:16.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 1 -- 172.21.15.26:0/1192924518 shutdown_connections 2024-08-19T12:54:16.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 1 -- 172.21.15.26:0/1192924518 >> 172.21.15.26:0/1192924518 conn(0x7f2c480a8740 msgr2=0x7f2c480aa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:16.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 1 -- 172.21.15.26:0/1192924518 shutdown_connections 2024-08-19T12:54:16.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 1 -- 172.21.15.26:0/1192924518 wait complete. 2024-08-19T12:54:16.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:16.900+0000 7f2c4ffde640 1 librados: shutdown 2024-08-19T12:54:16.938 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:54:16.938 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-08-19T12:54:16.938 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:54:17.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.037+0000 7fecfe6ff640 10 monclient: get_monmap_and_config 2024-08-19T12:54:17.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.037+0000 7fecfe6ff640 10 monclient: build_initial_monmap 2024-08-19T12:54:17.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.037+0000 7fecfe6ff640 10 monclient: monmap: 2024-08-19T12:54:17.038 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:17.039 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.039 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:54:17.038506+0000 2024-08-19T12:54:17.039 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:54:17.038506+0000 2024-08-19T12:54:17.039 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:17.039 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.039 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:17.039 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:17.039 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:17.039 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:17.039 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:17.039 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:17.039 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:17.040 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:17.040 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:17.040 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.038+0000 7fecfe6ff640 1 Processor -- start 2024-08-19T12:54:17.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.038+0000 7fecfe6ff640 1 -- start start 2024-08-19T12:54:17.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.038+0000 7fecfe6ff640 10 monclient: init 2024-08-19T12:54:17.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.038+0000 7fecfe6ff640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:17.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.038+0000 7fecfe6ff640 10 monclient: _add_conns ranks=[4,5,2,1,3,6,0,8,7] 2024-08-19T12:54:17.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.038+0000 7fecfe6ff640 10 monclient(hunting): picked mon.noname-c con 0x7fecf806e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:54:17.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.038+0000 7fecfe6ff640 10 monclient(hunting): picked mon.noname-h con 0x7fecf80a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:54:17.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.038+0000 7fecfe6ff640 10 monclient(hunting): picked mon.noname-b con 0x7fecf80a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:54:17.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.038+0000 7fecfe6ff640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf8136790 con 0x7fecf80a5d50 2024-08-19T12:54:17.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.038+0000 7fecfe6ff640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf809ddf0 con 0x7fecf806e7b0 2024-08-19T12:54:17.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.038+0000 7fecfe6ff640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf809e0d0 con 0x7fecf80a5920 2024-08-19T12:54:17.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.038+0000 7fecfe6ff640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:17.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.039+0000 7fecfe6ff640 10 monclient(hunting): authenticate will time out at 1651.126027s 2024-08-19T12:54:17.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.039+0000 7fecf7fff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fecf806e7b0 0x7fecf8136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36736/0 (socket says 172.21.15.26:36736) 2024-08-19T12:54:17.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.039+0000 7fecf7fff640 1 -- 172.21.15.26:0/1680975414 learned_addr learned my addr 172.21.15.26:0/1680975414 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:17.042 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.039+0000 7fecfcc9c640 1 --1- 172.21.15.26:0/1680975414 >> v1:172.21.15.26:6790/0 conn(0x7fecf80a5d50 0x7fecf8141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41386/0 (socket says 172.21.15.26:41386) 2024-08-19T12:54:17.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf7fff640 1 -- 172.21.15.26:0/1680975414 >> v1:172.21.15.26:6791/0 conn(0x7fecf806e7b0 legacy=0x7fecf8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecfcc9c640 1 -- 172.21.15.26:0/1680975414 >> v1:172.21.15.26:6790/0 conn(0x7fecf80a5d50 legacy=0x7fecf8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 1 -- 172.21.15.26:0/1680975414 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3176092660 0 0) 0x7fece8001450 con 0x7fecf806e7b0 2024-08-19T12:54:17.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient(hunting): my global_id is 184612 2024-08-19T12:54:17.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 1 -- 172.21.15.26:0/1680975414 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fece0003520 con 0x7fecf806e7b0 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 1 -- 172.21.15.26:0/1680975414 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2062207862 0 0) 0x7fecf8136790 con 0x7fecf80a5d50 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient(hunting): my global_id is 27326 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 1 -- 172.21.15.26:0/1680975414 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fece8001450 con 0x7fecf80a5d50 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf77fe640 1 -- 172.21.15.26:0/1680975414 >> v1:172.21.15.104:6791/0 conn(0x7fecf80a5920 legacy=0x7fecf813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 1 -- 172.21.15.26:0/1680975414 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3947479202 0 0) 0x7fece0003520 con 0x7fecf806e7b0 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 1 -- 172.21.15.26:0/1680975414 >> v1:172.21.15.104:6791/0 conn(0x7fecf80a5920 legacy=0x7fecf813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 1 -- 172.21.15.26:0/1680975414 >> v1:172.21.15.26:6790/0 conn(0x7fecf80a5d50 legacy=0x7fecf8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 1 monclient: found mon.noname-c 2024-08-19T12:54:17.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 1 -- 172.21.15.26:0/1680975414 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf809e3b0 con 0x7fecf806e7b0 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient: _finish_auth 0 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 1 -- 172.21.15.26:0/1680975414 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fece8001cb0 con 0x7fecf806e7b0 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecfe6ff640 5 monclient: authenticate success, global_id 184612 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecfe6ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient: dump: 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.048 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.049 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecfe6ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 1 -- 172.21.15.26:0/1680975414 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fece8004950 con 0x7fecf806e7b0 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 1 -- 172.21.15.26:0/1680975414 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fece8004ec0 con 0x7fecf806e7b0 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecfe6ff640 10 monclient: get_monmap_and_config success 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecfe6ff640 4 set_mon_vals no callback set 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.041+0000 7fecf6ffd640 10 monclient: dump: 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.050 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.042+0000 7fecfe6ff640 10 monclient: shutdown 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.042+0000 7fecfe6ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.042+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1680975414 >> v1:172.21.15.26:6791/0 conn(0x7fecf806e7b0 legacy=0x7fecf8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.042+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1680975414 shutdown_connections 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.042+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1680975414 >> 172.21.15.26:0/1680975414 conn(0x7fecf80a8740 msgr2=0x7fecf80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.042+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1680975414 shutdown_connections 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.042+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1680975414 wait complete. 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.042+0000 7fecfe6ff640 10 monclient: build_initial_monmap 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.042+0000 7fecfe6ff640 10 monclient: monmap: 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.052 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 1 librados: starting msgr at 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 1 librados: starting objecter 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 1 Processor -- start 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 1 -- start start 2024-08-19T12:54:17.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 1 librados: setting wanted keys 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 1 librados: calling monclient init 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 10 monclient: init 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 10 monclient: _add_conns ranks=[5,8,0,4,2,3,6,1,7] 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 10 monclient(hunting): picked mon.noname-h con 0x7fecf806e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 10 monclient(hunting): picked mon.noname-e con 0x7fecf80a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 10 monclient(hunting): picked mon.noname-a con 0x7fecf80a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf813b490 con 0x7fecf80a5d50 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf8139130 con 0x7fecf806e7b0 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf813abf0 con 0x7fecf80a5920 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.043+0000 7fecfe6ff640 10 monclient(hunting): authenticate will time out at 1651.130946s 2024-08-19T12:54:17.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.044+0000 7fecf77fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fecf80a5920 0x7fecf81d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:52594/0 (socket says 172.21.15.26:52594) 2024-08-19T12:54:17.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.044+0000 7fecf77fe640 1 -- 172.21.15.26:0/1459542223 learned_addr learned my addr 172.21.15.26:0/1459542223 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:17.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf77fe640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.26:6793/0 conn(0x7fecf80a5920 legacy=0x7fecf81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 client.?.objecter ms_handle_connect 0x7fecf80a5920 2024-08-19T12:54:17.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:17.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecfcc9c640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.26:6789/0 conn(0x7fecf80a5d50 legacy=0x7fecf81d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 client.?.objecter ms_handle_connect 0x7fecf80a5d50 2024-08-19T12:54:17.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:17.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2413287849 0 0) 0x7fecf813abf0 con 0x7fecf80a5920 2024-08-19T12:54:17.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): my global_id is 184454 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf7fff640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.104:6791/0 conn(0x7fecf806e7b0 legacy=0x7fecf81ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fece4004eb0 con 0x7fecf80a5920 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 client.?.objecter ms_handle_connect 0x7fecf806e7b0 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2026564292 0 0) 0x7fecf813b490 con 0x7fecf80a5d50 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): my global_id is 194319 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecf813abf0 con 0x7fecf80a5d50 2024-08-19T12:54:17.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 447762146 0 0) 0x7fecf8139130 con 0x7fecf806e7b0 2024-08-19T12:54:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): my global_id is 184505 2024-08-19T12:54:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecf813b490 con 0x7fecf806e7b0 2024-08-19T12:54:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 359579517 0 0) 0x7fece4004eb0 con 0x7fecf80a5920 2024-08-19T12:54:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecf8139130 con 0x7fecf80a5920 2024-08-19T12:54:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fecec003610 con 0x7fecf80a5920 2024-08-19T12:54:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:54:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): dump: 2024-08-19T12:54:17.062 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.063 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.064 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.064 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:54:17.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3318173100 0 0) 0x7fecf813abf0 con 0x7fecf80a5d50 2024-08-19T12:54:17.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fece4004eb0 con 0x7fecf80a5d50 2024-08-19T12:54:17.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fecf0002090 con 0x7fecf80a5d50 2024-08-19T12:54:17.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:17.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:54:17.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): dump: 2024-08-19T12:54:17.064 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.064 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.065 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 145872576 0 0) 0x7fecf813b490 con 0x7fecf806e7b0 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecf813abf0 con 0x7fecf806e7b0 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fece8001c30 con 0x7fecf806e7b0 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient(hunting): dump: 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.066 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4045953156 0 0) 0x7fecf8139130 con 0x7fecf80a5920 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.104:6791/0 conn(0x7fecf806e7b0 legacy=0x7fecf81ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.26:6789/0 conn(0x7fecf80a5d50 legacy=0x7fecf81d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:54:17.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 monclient: found mon.e 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf813aed0 con 0x7fecf80a5920 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient: _finish_auth 0 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.046+0000 7fecf4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecfe6ff640 5 monclient: authenticate success, global_id 184454 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecfe6ff640 10 monclient: _renew_subs 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecfe6ff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fecf813a8a0 con 0x7fecf80a5920 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecfe6ff640 10 client.184454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecfe6ff640 10 monclient: _renew_subs 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecfe6ff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:54:17.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fecf81386a0 con 0x7fecf80a5920 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecfe6ff640 1 librados: init done 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fecec003b90 con 0x7fecf80a5920 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecf4ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fecec0040c0 con 0x7fecf80a5920 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecf4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecf4ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecf67fc640 4 set_mon_vals no callback set 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecf4ff9640 10 monclient: dump: 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.069 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecfe6ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fecd80065c0 con 0x7fecd8002860 2024-08-19T12:54:17.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.047+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fecec004260 con 0x7fecf80a5920 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.048+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(574..574 src has 350..574) ==== 5913+0+0 (unknown 1558564169 0 0) 0x7fecec04e400 con 0x7fecf80a5920 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.048+0000 7fecf4ff9640 10 client.184454.objecter ms_dispatch 0x7fecf8000b90 osd_map(574..574 src has 350..574) 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.048+0000 7fecf4ff9640 3 client.184454.objecter handle_osd_map got epochs [574,574] > 0 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.048+0000 7fecf4ff9640 3 client.184454.objecter handle_osd_map decoding full epoch 574 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.048+0000 7fecfcc9c640 10 monclient: get_auth_request con 0x7fece40418d0 auth_method 0 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.048+0000 7fecf4ff9640 20 client.184454.objecter dump_active .. 0 homeless 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf7fff640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.104:6790/0 conn(0x7fecd8002860 legacy=0x7fecd8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 10 client.184454.objecter ms_handle_connect 0x7fecd8002860 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 10 client.184454.objecter resend_mon_ops 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4272381718 0 0) 0x7fecd80065c0 con 0x7fecd8002860 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 10 monclient(hunting): my global_id is 194259 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecf8139130 con 0x7fecd8002860 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3215690024 0 0) 0x7fecf8139130 con 0x7fecd8002860 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecd80065c0 con 0x7fecd8002860 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fece8004ec0 con 0x7fecd8002860 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 10 monclient: dump: 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.072 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 10 client.184454.objecter ms_handle_connect 0x7fece40418d0 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 424674012 0 0) 0x7fecd80065c0 con 0x7fecd8002860 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.050+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fecd800a160 con 0x7fecd8002860 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.051+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fece8005200 con 0x7fecd8002860 2024-08-19T12:54:17.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.051+0000 7fecf4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:17.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.051+0000 7fecf4ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:54:17.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.051+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.104:6790/0 conn(0x7fecd8002860 legacy=0x7fecd8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.051+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.104:6790/0 conn(0x7fecd8002860 legacy=0x7fecd8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:17.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.057+0000 7fecfe6ff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:54:17.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.057+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fecd8009e70 con 0x7fecd8009130 2024-08-19T12:54:17.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.059+0000 7fecf7fff640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.104:6790/0 conn(0x7fecd8009130 legacy=0x7fecd80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.059+0000 7fecf4ff9640 10 client.184454.objecter ms_handle_connect 0x7fecd8009130 2024-08-19T12:54:17.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.059+0000 7fecf4ff9640 10 client.184454.objecter resend_mon_ops 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3349742527 0 0) 0x7fecd8009e70 con 0x7fecd8009130 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 10 monclient(hunting): my global_id is 194268 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecd80065c0 con 0x7fecd8009130 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 414116215 0 0) 0x7fecd80065c0 con 0x7fecd8009130 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecd8009e70 con 0x7fecd8009130 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fece8005960 con 0x7fecd8009130 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 10 monclient: dump: 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.076 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.077 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3888283928 0 0) 0x7fecd8009e70 con 0x7fecd8009130 2024-08-19T12:54:17.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.060+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fecd800c3f0 con 0x7fecd8009130 2024-08-19T12:54:17.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7533 (unknown 0 0 1841655025) 0x7fece8007ee0 con 0x7fecd8009130 2024-08-19T12:54:17.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecf4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:54:17.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecf4ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:54:17.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.104:6790/0 conn(0x7fecd8009130 legacy=0x7fecd80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecf4ff9640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.104:6790/0 conn(0x7fecd8009130 legacy=0x7fecd80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:17.078 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:54:17.078 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:54:17.078 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 116, 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 41983, 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.079 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:17.080 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:17.081 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:54:17.082 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:17.083 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.085 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:54:17.086 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:54:17.090 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:54:17.092 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:54:17.092 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.092 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.092 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.093 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:54:17.094 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:54:17.094 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:54:17.094 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:54:17.094 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.097 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:54:17.098 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:54:17.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.098 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:17.098 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:17.098 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:17.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.098 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.098 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:54:17.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.098 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:17.098 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:17.098 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:54:17.099 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.099 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.099 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:54:17.099 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:54:17.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 10 librados: watch_flush enter 2024-08-19T12:54:17.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 10 librados: watch_flush exit 2024-08-19T12:54:17.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 20 client.184454.objecter shutdown clearing up homeless session... 2024-08-19T12:54:17.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 10 client.184454.objecter successfully canceled tick 2024-08-19T12:54:17.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.104:6800/899887506 conn(0x7fece40418d0 legacy=0x7fece4043d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 10 monclient: shutdown 2024-08-19T12:54:17.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:17.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1459542223 >> v1:172.21.15.26:6793/0 conn(0x7fecf80a5920 legacy=0x7fecf81d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfcc9c640 1 -- 172.21.15.26:0/1459542223 reap_dead start 2024-08-19T12:54:17.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfcc9c640 1 -- 172.21.15.26:0/1459542223 reap_dead start 2024-08-19T12:54:17.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1459542223 shutdown_connections 2024-08-19T12:54:17.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1459542223 >> 172.21.15.26:0/1459542223 conn(0x7fecf80a8740 msgr2=0x7fecf80aa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:17.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1459542223 shutdown_connections 2024-08-19T12:54:17.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 1 -- 172.21.15.26:0/1459542223 wait complete. 2024-08-19T12:54:17.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.061+0000 7fecfe6ff640 1 librados: shutdown 2024-08-19T12:54:17.102 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:54:17.102 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-08-19T12:54:17.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:54:17.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.204+0000 7febea848640 10 monclient: get_monmap_and_config 2024-08-19T12:54:17.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.204+0000 7febea848640 10 monclient: build_initial_monmap 2024-08-19T12:54:17.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.204+0000 7febea848640 10 monclient: monmap: 2024-08-19T12:54:17.205 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:17.205 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.205 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:54:17.205246+0000 2024-08-19T12:54:17.205 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:54:17.205246+0000 2024-08-19T12:54:17.205 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:17.205 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.206 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:17.206 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:17.206 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:17.206 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:17.206 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:17.206 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:17.206 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:17.206 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:17.206 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:17.206 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 1 Processor -- start 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 1 -- start start 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 10 monclient: init 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 10 monclient: _add_conns ranks=[4,6,1,3,8,7,2,5,0] 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 10 monclient(hunting): picked mon.noname-c con 0x7febe406e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 10 monclient(hunting): picked mon.noname-d con 0x7febe40a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 10 monclient(hunting): picked mon.noname-f con 0x7febe40a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe4136790 con 0x7febe40a5d50 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe409ddf0 con 0x7febe406e7b0 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe409e0d0 con 0x7febe40a5920 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febea848640 10 monclient(hunting): authenticate will time out at 1651.292707s 2024-08-19T12:54:17.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febe37fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7febe40a5920 0x7febe413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:42782/0 (socket says 172.21.15.26:42782) 2024-08-19T12:54:17.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febe37fe640 1 -- 172.21.15.26:0/85971554 learned_addr learned my addr 172.21.15.26:0/85971554 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:17.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febe3fff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7febe406e7b0 0x7febe4136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36746/0 (socket says 172.21.15.26:36746) 2024-08-19T12:54:17.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.205+0000 7febe8de5640 1 -- 172.21.15.26:0/85971554 >> v1:172.21.15.104:6789/0 conn(0x7febe40a5d50 legacy=0x7febe4141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:54:17.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.207+0000 7febe3fff640 1 -- 172.21.15.26:0/85971554 >> v1:172.21.15.26:6791/0 conn(0x7febe406e7b0 legacy=0x7febe4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.207+0000 7febe37fe640 1 -- 172.21.15.26:0/85971554 >> v1:172.21.15.26:6792/0 conn(0x7febe40a5920 legacy=0x7febe413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 1 -- 172.21.15.26:0/85971554 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2527896146 0 0) 0x7febe409e0d0 con 0x7febe40a5920 2024-08-19T12:54:17.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient(hunting): my global_id is 194280 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 1 -- 172.21.15.26:0/85971554 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febcc003520 con 0x7febe40a5920 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 1 -- 172.21.15.26:0/85971554 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114596312 0 0) 0x7febd4001450 con 0x7febe406e7b0 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient(hunting): my global_id is 184621 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 1 -- 172.21.15.26:0/85971554 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febe409e0d0 con 0x7febe406e7b0 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 1 -- 172.21.15.26:0/85971554 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 37916862 0 0) 0x7febcc003520 con 0x7febe40a5920 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 1 -- 172.21.15.26:0/85971554 >> v1:172.21.15.26:6791/0 conn(0x7febe406e7b0 legacy=0x7febe4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 1 -- 172.21.15.26:0/85971554 >> v1:172.21.15.104:6789/0 conn(0x7febe40a5d50 legacy=0x7febe4141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:54:17.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 1 monclient: found mon.noname-d 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 1 -- 172.21.15.26:0/85971554 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe409e3b0 con 0x7febe40a5920 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient: _finish_auth 0 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febea848640 5 monclient: authenticate success, global_id 194280 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 1 -- 172.21.15.26:0/85971554 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7febd8001ae0 con 0x7febe40a5920 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febea848640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient: dump: 2024-08-19T12:54:17.220 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.221 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febea848640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 1 -- 172.21.15.26:0/85971554 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febd80048a0 con 0x7febe40a5920 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 1 -- 172.21.15.26:0/85971554 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7febd8004e10 con 0x7febe40a5920 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febea848640 10 monclient: get_monmap_and_config success 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febea848640 4 set_mon_vals no callback set 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.208+0000 7febe2ffd640 10 monclient: dump: 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.222 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.209+0000 7febea848640 10 monclient: shutdown 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.209+0000 7febea848640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.209+0000 7febea848640 1 -- 172.21.15.26:0/85971554 >> v1:172.21.15.26:6792/0 conn(0x7febe40a5920 legacy=0x7febe413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.209+0000 7febea848640 1 -- 172.21.15.26:0/85971554 shutdown_connections 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.209+0000 7febea848640 1 -- 172.21.15.26:0/85971554 >> 172.21.15.26:0/85971554 conn(0x7febe40a8740 msgr2=0x7febe40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.209+0000 7febea848640 1 -- 172.21.15.26:0/85971554 shutdown_connections 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.209+0000 7febea848640 1 -- 172.21.15.26:0/85971554 wait complete. 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.209+0000 7febea848640 10 monclient: build_initial_monmap 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.209+0000 7febea848640 10 monclient: monmap: 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:17.224 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.209+0000 7febea848640 1 librados: starting msgr at 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.209+0000 7febea848640 1 librados: starting objecter 2024-08-19T12:54:17.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.209+0000 7febea848640 1 Processor -- start 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 1 -- start start 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 1 librados: setting wanted keys 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 1 librados: calling monclient init 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 10 monclient: init 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 10 monclient: _add_conns ranks=[7,2,0,4,3,1,5,8,6] 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 10 monclient(hunting): picked mon.noname-i con 0x7febe406e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 10 monclient(hunting): picked mon.noname-b con 0x7febe40a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 10 monclient(hunting): picked mon.noname-a con 0x7febe40a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe41345d0 con 0x7febe40a5d50 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe4125bd0 con 0x7febe40a5920 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febe4125330 con 0x7febe406e7b0 2024-08-19T12:54:17.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febea848640 10 monclient(hunting): authenticate will time out at 1651.297711s 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febe37fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7febe40a5920 0x7febe4122c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:41388/0 (socket says 172.21.15.26:41388) 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.210+0000 7febe37fe640 1 -- 172.21.15.26:0/4159479365 learned_addr learned my addr 172.21.15.26:0/4159479365 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.212+0000 7febe37fe640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.26:6790/0 conn(0x7febe40a5920 legacy=0x7febe4122c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.212+0000 7febe8de5640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.26:6789/0 conn(0x7febe40a5d50 legacy=0x7febe41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.212+0000 7febe0ff9640 10 client.?.objecter ms_handle_connect 0x7febe40a5920 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.212+0000 7febe0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.212+0000 7febe0ff9640 10 client.?.objecter ms_handle_connect 0x7febe40a5d50 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.212+0000 7febe0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1185788456 0 0) 0x7febe4125bd0 con 0x7febe40a5920 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): my global_id is 27335 2024-08-19T12:54:17.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febd0004eb0 con 0x7febe40a5920 2024-08-19T12:54:17.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe3fff640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.104:6792/0 conn(0x7febe406e7b0 legacy=0x7febe4122500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 client.?.objecter ms_handle_connect 0x7febe406e7b0 2024-08-19T12:54:17.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:17.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3081576972 0 0) 0x7febdc004060 con 0x7febe40a5d50 2024-08-19T12:54:17.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): my global_id is 194328 2024-08-19T12:54:17.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febe4125bd0 con 0x7febe40a5d50 2024-08-19T12:54:17.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1899486272 0 0) 0x7febd0004eb0 con 0x7febe40a5920 2024-08-19T12:54:17.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febdc004060 con 0x7febe40a5920 2024-08-19T12:54:17.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2263909488 0 0) 0x7febe4125bd0 con 0x7febe40a5d50 2024-08-19T12:54:17.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febd0004eb0 con 0x7febe40a5d50 2024-08-19T12:54:17.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 213766050 0 0) 0x7febe4125330 con 0x7febe406e7b0 2024-08-19T12:54:17.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): my global_id is 136222 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febe4125bd0 con 0x7febe406e7b0 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7febd8004d90 con 0x7febe40a5920 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): dump: 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.230 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.231 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:54:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7febdc004a10 con 0x7febe40a5d50 2024-08-19T12:54:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:54:17.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient(hunting): dump: 2024-08-19T12:54:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.232 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2510424243 0 0) 0x7febd0004eb0 con 0x7febe40a5d50 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.104:6792/0 conn(0x7febe406e7b0 legacy=0x7febe4122500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.26:6790/0 conn(0x7febe40a5920 legacy=0x7febe4122c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 monclient: found mon.a 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:54:17.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe4125610 con 0x7febe40a5d50 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient: _finish_auth 0 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febea848640 5 monclient: authenticate success, global_id 194328 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febea848640 10 monclient: _renew_subs 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febea848640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febea848640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7febe4124f90 con 0x7febe40a5d50 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febea848640 10 client.194328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febea848640 10 monclient: _renew_subs 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febea848640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febea848640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7febe4123320 con 0x7febe40a5d50 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febea848640 1 librados: init done 2024-08-19T12:54:17.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febdc004f90 con 0x7febe40a5d50 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7febdc0054c0 con 0x7febe40a5d50 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe27fc640 4 set_mon_vals no callback set 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.213+0000 7febe0ff9640 10 monclient: dump: 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.235 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.214+0000 7febea848640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.214+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7febdc005660 con 0x7febe40a5d50 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.214+0000 7febea848640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7febc40065c0 con 0x7febc4002860 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.214+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(574..574 src has 350..574) ==== 5913+0+0 (unknown 1558564169 0 0) 0x7febdc04e730 con 0x7febe40a5d50 2024-08-19T12:54:17.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.214+0000 7febe0ff9640 10 client.194328.objecter ms_dispatch 0x7febe4000b90 osd_map(574..574 src has 350..574) 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.214+0000 7febe0ff9640 3 client.194328.objecter handle_osd_map got epochs [574,574] > 0 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.214+0000 7febe0ff9640 3 client.194328.objecter handle_osd_map decoding full epoch 574 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.214+0000 7febe0ff9640 20 client.194328.objecter dump_active .. 0 homeless 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.214+0000 7febe37fe640 10 monclient: get_auth_request con 0x7febd0041680 auth_method 0 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.216+0000 7febe3fff640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.104:6791/0 conn(0x7febc4002860 legacy=0x7febc4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 10 client.194328.objecter ms_handle_connect 0x7febc4002860 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 10 client.194328.objecter resend_mon_ops 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 963760270 0 0) 0x7febc40065c0 con 0x7febc4002860 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 10 monclient(hunting): my global_id is 184514 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febd0004eb0 con 0x7febc4002860 2024-08-19T12:54:17.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 10 client.194328.objecter ms_handle_connect 0x7febd0041680 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2399177414 0 0) 0x7febd0004eb0 con 0x7febc4002860 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febc40065c0 con 0x7febc4002860 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7febd4001500 con 0x7febc4002860 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 10 monclient: dump: 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3404139478 0 0) 0x7febc40065c0 con 0x7febc4002860 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7febc400a160 con 0x7febc4002860 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.217+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7febd4004880 con 0x7febc4002860 2024-08-19T12:54:17.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.218+0000 7febe0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:17.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.218+0000 7febe0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:54:17.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.218+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.104:6791/0 conn(0x7febc4002860 legacy=0x7febc4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.218+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.104:6791/0 conn(0x7febc4002860 legacy=0x7febc4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:17.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.224+0000 7febea848640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:54:17.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.224+0000 7febea848640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7febc4009e50 con 0x7febc4009110 2024-08-19T12:54:17.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.226+0000 7febe3fff640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.104:6791/0 conn(0x7febc4009110 legacy=0x7febc40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.226+0000 7febe0ff9640 10 client.194328.objecter ms_handle_connect 0x7febc4009110 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.226+0000 7febe0ff9640 10 client.194328.objecter resend_mon_ops 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2724022485 0 0) 0x7febc4009e50 con 0x7febc4009110 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 10 monclient(hunting): my global_id is 184523 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febc40065c0 con 0x7febc4009110 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2543501462 0 0) 0x7febc40065c0 con 0x7febc4009110 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febc4009e50 con 0x7febc4009110 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7febd4005060 con 0x7febc4009110 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:54:17.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 10 monclient: dump: 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.244 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.245 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 262738278 0 0) 0x7febc4009e50 con 0x7febc4009110 2024-08-19T12:54:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.227+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7febc400c3f0 con 0x7febc4009110 2024-08-19T12:54:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7868 (unknown 0 0 1981721473) 0x7febd40073c0 con 0x7febc4009110 2024-08-19T12:54:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febe0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:54:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febe0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:54:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.104:6791/0 conn(0x7febc4009110 legacy=0x7febc40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febe0ff9640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.104:6791/0 conn(0x7febc4009110 legacy=0x7febc40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:17.246 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:54:17.246 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:54:17.246 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:54:17.246 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:54:17.246 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 116, 2024-08-19T12:54:17.246 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 182697, 2024-08-19T12:54:17.246 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:54:17.246 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:54:17.246 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:54:17.246 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:54:17.246 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:17.247 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:17.248 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:54:17.249 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:17.250 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:54:17.251 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:17.251 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:17.251 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:17.251 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:17.251 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:17.251 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:17.251 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:17.251 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:17.251 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:17.251 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:17.251 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:17.251 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:54:17.255 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.256 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:54:17.258 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.258 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:54:17.258 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:54:17.258 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.258 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.258 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:54:17.259 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:54:17.260 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:54:17.261 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:54:17.261 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.261 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.261 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.261 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:54:17.261 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.262 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:17.264 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:17.265 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:54:17.266 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:54:17.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 10 librados: watch_flush enter 2024-08-19T12:54:17.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 10 librados: watch_flush exit 2024-08-19T12:54:17.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 20 client.194328.objecter shutdown clearing up homeless session... 2024-08-19T12:54:17.268 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 10 client.194328.objecter successfully canceled tick 2024-08-19T12:54:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.104:6800/899887506 conn(0x7febd0041680 legacy=0x7febd0043b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 10 monclient: shutdown 2024-08-19T12:54:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 1 -- 172.21.15.26:0/4159479365 >> v1:172.21.15.26:6789/0 conn(0x7febe40a5d50 legacy=0x7febe41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febe8de5640 1 -- 172.21.15.26:0/4159479365 reap_dead start 2024-08-19T12:54:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febe8de5640 1 -- 172.21.15.26:0/4159479365 reap_dead start 2024-08-19T12:54:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 1 -- 172.21.15.26:0/4159479365 shutdown_connections 2024-08-19T12:54:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 1 -- 172.21.15.26:0/4159479365 >> 172.21.15.26:0/4159479365 conn(0x7febe40a8740 msgr2=0x7febe4132e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 1 -- 172.21.15.26:0/4159479365 shutdown_connections 2024-08-19T12:54:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 1 -- 172.21.15.26:0/4159479365 wait complete. 2024-08-19T12:54:17.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.228+0000 7febea848640 1 librados: shutdown 2024-08-19T12:54:17.270 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:54:17.270 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-08-19T12:54:17.270 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:54:17.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.368+0000 7f2967d72640 10 monclient: get_monmap_and_config 2024-08-19T12:54:17.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.368+0000 7f2967d72640 10 monclient: build_initial_monmap 2024-08-19T12:54:17.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.368+0000 7f2967d72640 10 monclient: monmap: 2024-08-19T12:54:17.369 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:17.370 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.370 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:54:17.369636+0000 2024-08-19T12:54:17.370 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:54:17.369636+0000 2024-08-19T12:54:17.370 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:17.370 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.370 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:17.370 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:17.370 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:17.370 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:17.370 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:17.371 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:17.371 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:17.371 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:17.371 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:17.371 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.369+0000 7f2967d72640 1 Processor -- start 2024-08-19T12:54:17.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.369+0000 7f2967d72640 1 -- start start 2024-08-19T12:54:17.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.369+0000 7f2967d72640 10 monclient: init 2024-08-19T12:54:17.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f2967d72640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:17.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f2967d72640 10 monclient: _add_conns ranks=[4,1,8,5,0,7,2,3,6] 2024-08-19T12:54:17.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f2967d72640 10 monclient(hunting): picked mon.noname-c con 0x7f2960073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:54:17.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f2967d72640 10 monclient(hunting): picked mon.noname-f con 0x7f29600a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:54:17.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f2967d72640 10 monclient(hunting): picked mon.noname-e con 0x7f296013f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:54:17.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f2967d72640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f296009dae0 con 0x7f29600a57a0 2024-08-19T12:54:17.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f2967d72640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2960136910 con 0x7f2960073470 2024-08-19T12:54:17.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f2967d72640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f296009dd50 con 0x7f296013f2e0 2024-08-19T12:54:17.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f2967d72640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:17.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f2967d72640 10 monclient(hunting): authenticate will time out at 1651.457211s 2024-08-19T12:54:17.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f2965b0e640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f2960073470 0x7f2960136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36754/0 (socket says 172.21.15.26:36754) 2024-08-19T12:54:17.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f2965b0e640 1 -- 172.21.15.26:0/2916618654 learned_addr learned my addr 172.21.15.26:0/2916618654 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:17.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.370+0000 7f296530d640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f29600a57a0 legacy=0x7f29600a5b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:54:17.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f296630f640 1 -- 172.21.15.26:0/2916618654 >> v1:172.21.15.26:6793/0 conn(0x7f296013f2e0 legacy=0x7f29601416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2965b0e640 1 -- 172.21.15.26:0/2916618654 >> v1:172.21.15.26:6791/0 conn(0x7f2960073470 legacy=0x7f2960136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 1 -- 172.21.15.26:0/2916618654 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449838803 0 0) 0x7f2960136910 con 0x7f2960073470 2024-08-19T12:54:17.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient(hunting): my global_id is 184630 2024-08-19T12:54:17.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 1 -- 172.21.15.26:0/2916618654 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2948003520 con 0x7f2960073470 2024-08-19T12:54:17.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 1 -- 172.21.15.26:0/2916618654 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1654991287 0 0) 0x7f296009dd50 con 0x7f296013f2e0 2024-08-19T12:54:17.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient(hunting): my global_id is 184463 2024-08-19T12:54:17.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 1 -- 172.21.15.26:0/2916618654 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2960136910 con 0x7f296013f2e0 2024-08-19T12:54:17.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 1 -- 172.21.15.26:0/2916618654 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3955805148 0 0) 0x7f2948003520 con 0x7f2960073470 2024-08-19T12:54:17.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 1 -- 172.21.15.26:0/2916618654 >> v1:172.21.15.26:6793/0 conn(0x7f296013f2e0 legacy=0x7f29601416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 1 -- 172.21.15.26:0/2916618654 >> v1:172.21.15.104:6789/0 conn(0x7f29600a57a0 legacy=0x7f29600a5b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:54:17.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient: _finish_hunting 0 2024-08-19T12:54:17.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 1 monclient: found mon.noname-c 2024-08-19T12:54:17.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:17.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:54:17.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 1 -- 172.21.15.26:0/2916618654 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f296009e030 con 0x7f2960073470 2024-08-19T12:54:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient: _finish_auth 0 2024-08-19T12:54:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient: _check_auth_tickets 2024-08-19T12:54:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 1 -- 172.21.15.26:0/2916618654 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2958001a10 con 0x7f2960073470 2024-08-19T12:54:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2967d72640 5 monclient: authenticate success, global_id 184630 2024-08-19T12:54:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:54:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2964b0c640 10 monclient: dump: 2024-08-19T12:54:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.381 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.382 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.382 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.382 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.382 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.382 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.382 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.382 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.382 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.382 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.382 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.382 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.382 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.372+0000 7f2967d72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2964b0c640 1 -- 172.21.15.26:0/2916618654 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2958001ce0 con 0x7f2960073470 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2964b0c640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2964b0c640 1 -- 172.21.15.26:0/2916618654 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2958004be0 con 0x7f2960073470 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2964b0c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2964b0c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2964b0c640 10 monclient: dump: 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.383 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2967d72640 10 monclient: get_monmap_and_config success 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2967d72640 4 set_mon_vals no callback set 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2967d72640 10 monclient: shutdown 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2967d72640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:17.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2967d72640 1 -- 172.21.15.26:0/2916618654 >> v1:172.21.15.26:6791/0 conn(0x7f2960073470 legacy=0x7f2960136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2967d72640 1 -- 172.21.15.26:0/2916618654 shutdown_connections 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2967d72640 1 -- 172.21.15.26:0/2916618654 >> 172.21.15.26:0/2916618654 conn(0x7f29600abff0 msgr2=0x7f29600ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2967d72640 1 -- 172.21.15.26:0/2916618654 shutdown_connections 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.373+0000 7f2967d72640 1 -- 172.21.15.26:0/2916618654 wait complete. 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.374+0000 7f2967d72640 10 monclient: build_initial_monmap 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.374+0000 7f2967d72640 10 monclient: monmap: 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:54:17.385 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:54:17.386 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:54:17.386 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:54:17.386 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:54:17.386 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:54:17.386 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:54:17.386 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:54:17.386 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:54:17.386 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.374+0000 7f2967d72640 1 librados: starting msgr at 2024-08-19T12:54:17.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.374+0000 7f2967d72640 1 librados: starting objecter 2024-08-19T12:54:17.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.374+0000 7f2967d72640 1 Processor -- start 2024-08-19T12:54:17.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.374+0000 7f2967d72640 1 -- start start 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.374+0000 7f2967d72640 1 librados: setting wanted keys 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.374+0000 7f2967d72640 1 librados: calling monclient init 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.374+0000 7f2967d72640 10 monclient: init 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.374+0000 7f2967d72640 10 monclient: _reopen_session rank -1 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.375+0000 7f2967d72640 10 monclient: _add_conns ranks=[3,0,1,7,2,8,4,6,5] 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.375+0000 7f2967d72640 10 monclient(hunting): picked mon.noname-g con 0x7f2960073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.375+0000 7f2967d72640 10 monclient(hunting): picked mon.noname-a con 0x7f29600a57a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.375+0000 7f2967d72640 10 monclient(hunting): picked mon.noname-f con 0x7f296013f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.375+0000 7f2967d72640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f296012f2a0 con 0x7f29600a57a0 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.375+0000 7f2967d72640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2960142bc0 con 0x7f296013f2e0 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.375+0000 7f2967d72640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2960131600 con 0x7f2960073470 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.375+0000 7f2967d72640 10 monclient(hunting): _renew_subs 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.375+0000 7f2967d72640 10 monclient(hunting): authenticate will time out at 1651.462084s 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.375+0000 7f296530d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f29600a57a0 0x7f29601d2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:53914/0 (socket says 172.21.15.26:53914) 2024-08-19T12:54:17.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.375+0000 7f296530d640 1 -- 172.21.15.26:0/3240334148 learned_addr learned my addr 172.21.15.26:0/3240334148 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:54:17.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.375+0000 7f296630f640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.104:6789/0 conn(0x7f296013f2e0 legacy=0x7f29601d61f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:54:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f296530d640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.26:6789/0 conn(0x7f29600a57a0 legacy=0x7f29601d2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 client.?.objecter ms_handle_connect 0x7f29600a57a0 2024-08-19T12:54:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 client.?.objecter ms_handle_connect 0x7f2960073470 2024-08-19T12:54:17.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f2965b0e640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.104:6790/0 conn(0x7f2960073470 legacy=0x7f2960132a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2593826255 0 0) 0x7f296012f2a0 con 0x7f29600a57a0 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 monclient(hunting): my global_id is 194337 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f294c004eb0 con 0x7f29600a57a0 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968334330 0 0) 0x7f2960131600 con 0x7f2960073470 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 monclient(hunting): my global_id is 194277 2024-08-19T12:54:17.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f296012f2a0 con 0x7f2960073470 2024-08-19T12:54:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 870895713 0 0) 0x7f294c004eb0 con 0x7f29600a57a0 2024-08-19T12:54:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2960131600 con 0x7f29600a57a0 2024-08-19T12:54:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f29540047a0 con 0x7f29600a57a0 2024-08-19T12:54:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:54:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.377+0000 7f295e7fc640 10 monclient(hunting): dump: 2024-08-19T12:54:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.396 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1797101793 0 0) 0x7f296012f2a0 con 0x7f2960073470 2024-08-19T12:54:17.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f294c004eb0 con 0x7f2960073470 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2958004b20 con 0x7f2960073470 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 10 monclient(hunting): dump: 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.398 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4255975226 0 0) 0x7f2960131600 con 0x7f29600a57a0 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.104:6790/0 conn(0x7f2960073470 legacy=0x7f2960132a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.104:6789/0 conn(0x7f296013f2e0 legacy=0x7f29601d61f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 1 monclient: found mon.a 2024-08-19T12:54:17.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2960130d60 con 0x7f29600a57a0 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 10 monclient: _finish_auth 0 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f2967d72640 5 monclient: authenticate success, global_id 194337 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f2967d72640 10 monclient: _renew_subs 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f2967d72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f2967d72640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2960130040 con 0x7f29600a57a0 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f2967d72640 10 client.194337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f2967d72640 10 monclient: _renew_subs 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f2967d72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f2967d72640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2960133140 con 0x7f29600a57a0 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f2967d72640 1 librados: init done 2024-08-19T12:54:17.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2954004cd0 con 0x7f29600a57a0 2024-08-19T12:54:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:54:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f29540051c0 con 0x7f29600a57a0 2024-08-19T12:54:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295ffff640 4 set_mon_vals no callback set 2024-08-19T12:54:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:54:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f295e7fc640 10 monclient: dump: 2024-08-19T12:54:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.401 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f2967d72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:17.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.378+0000 7f2967d72640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f29400065c0 con 0x7f2940002860 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.379+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2954005410 con 0x7f29600a57a0 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.379+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(574..574 src has 350..574) ==== 5913+0+0 (unknown 1558564169 0 0) 0x7f295404e5a0 con 0x7f29600a57a0 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.379+0000 7f295e7fc640 10 client.194337.objecter ms_dispatch 0x7f2960000b90 osd_map(574..574 src has 350..574) 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.379+0000 7f295e7fc640 3 client.194337.objecter handle_osd_map got epochs [574,574] > 0 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.379+0000 7f295e7fc640 3 client.194337.objecter handle_osd_map decoding full epoch 574 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.379+0000 7f296630f640 10 monclient: get_auth_request con 0x7f294c0414f0 auth_method 0 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.379+0000 7f295e7fc640 20 client.194337.objecter dump_active .. 0 homeless 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f2965b0e640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.104:6792/0 conn(0x7f2940002860 legacy=0x7f2940004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 10 client.194337.objecter ms_handle_connect 0x7f2940002860 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 10 client.194337.objecter resend_mon_ops 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 104255668 0 0) 0x7f29400065c0 con 0x7f2940002860 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 10 monclient(hunting): my global_id is 136231 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2960131600 con 0x7f2940002860 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2831317127 0 0) 0x7f2960131600 con 0x7f2940002860 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29400065c0 con 0x7f2940002860 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 10 client.194337.objecter ms_handle_connect 0x7f294c0414f0 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f29580014b0 con 0x7f2940002860 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 10 monclient: dump: 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.404 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.405 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1899864779 0 0) 0x7f29400065c0 con 0x7f2940002860 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.381+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f294000a160 con 0x7f2940002860 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.382+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2958005260 con 0x7f2940002860 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.382+0000 7f295e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.382+0000 7f295e7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.382+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.104:6792/0 conn(0x7f2940002860 legacy=0x7f2940004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.382+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.104:6792/0 conn(0x7f2940002860 legacy=0x7f2940004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.388+0000 7f2967d72640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.388+0000 7f2967d72640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2940009e70 con 0x7f2940009130 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f2965b0e640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.104:6792/0 conn(0x7f2940009130 legacy=0x7f29400095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 10 client.194337.objecter ms_handle_connect 0x7f2940009130 2024-08-19T12:54:17.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 10 client.194337.objecter resend_mon_ops 2024-08-19T12:54:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 737200692 0 0) 0x7f2940009e70 con 0x7f2940009130 2024-08-19T12:54:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:54:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:54:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:54:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 10 monclient(hunting): my global_id is 136240 2024-08-19T12:54:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29400065c0 con 0x7f2940009130 2024-08-19T12:54:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1310550430 0 0) 0x7f29400065c0 con 0x7f2940009130 2024-08-19T12:54:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2940009e70 con 0x7f2940009130 2024-08-19T12:54:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f29580059c0 con 0x7f2940009130 2024-08-19T12:54:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:54:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:54:17.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 10 monclient: dump: 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:54:17.408 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:54:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:54:17.409 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:54:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3061347634 0 0) 0x7f2940009e70 con 0x7f2940009130 2024-08-19T12:54:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.391+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f294000c410 con 0x7f2940009130 2024-08-19T12:54:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7534 (unknown 0 0 3558888911) 0x7f2958008710 con 0x7f2940009130 2024-08-19T12:54:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f295e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:54:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f295e7fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:54:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.104:6792/0 conn(0x7f2940009130 legacy=0x7f29400095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f295e7fc640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.104:6792/0 conn(0x7f2940009130 legacy=0x7f29400095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:54:17.410 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:54:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:54:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:54:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:54:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 116, 2024-08-19T12:54:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 827749, 2024-08-19T12:54:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:54:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:54:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:54:17.410 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:17.411 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:17.412 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:54:17.413 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:54:17.414 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.415 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.416 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.417 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.418 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:54:17.419 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.420 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.422 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:54:17.423 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.424 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:54:17.427 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:54:17.428 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:54:17.428 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:54:17.428 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:54:17.428 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.428 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.428 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:54:17.428 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:54:17.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.428 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:17.428 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:17.428 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:54:17.429 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.429 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:54:17.429 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:54:17.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:54:17.429 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:54:17.429 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:54:17.429 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:54:17.429 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:54:17.429 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:54:17.429 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:54:17.429 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:54:17.429 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:54:17.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 10 librados: watch_flush enter 2024-08-19T12:54:17.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 10 librados: watch_flush exit 2024-08-19T12:54:17.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 20 client.194337.objecter shutdown clearing up homeless session... 2024-08-19T12:54:17.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 10 client.194337.objecter successfully canceled tick 2024-08-19T12:54:17.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.104:6800/899887506 conn(0x7f294c0414f0 legacy=0x7f294c0439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 10 monclient: shutdown 2024-08-19T12:54:17.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:54:17.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 1 -- 172.21.15.26:0/3240334148 >> v1:172.21.15.26:6789/0 conn(0x7f29600a57a0 legacy=0x7f29601d2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:54:17.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f296630f640 1 -- 172.21.15.26:0/3240334148 reap_dead start 2024-08-19T12:54:17.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f296630f640 1 -- 172.21.15.26:0/3240334148 reap_dead start 2024-08-19T12:54:17.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 1 -- 172.21.15.26:0/3240334148 shutdown_connections 2024-08-19T12:54:17.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 1 -- 172.21.15.26:0/3240334148 >> 172.21.15.26:0/3240334148 conn(0x7f29600abff0 msgr2=0x7f29600a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:54:17.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 1 -- 172.21.15.26:0/3240334148 shutdown_connections 2024-08-19T12:54:17.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 1 -- 172.21.15.26:0/3240334148 wait complete. 2024-08-19T12:54:17.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:54:17.392+0000 7f2967d72640 1 librados: shutdown 2024-08-19T12:54:17.433 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-08-19T12:54:17.433 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T12:54:26.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:54:26.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (33955 ms) 2024-08-19T12:54:26.103 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-08-19T12:54:26.104 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:54:26.104 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6282 ms) 2024-08-19T12:54:26.104 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-08-19T12:54:26.104 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:54:26.104 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (4725 ms) 2024-08-19T12:54:26.104 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-08-19T12:54:26.104 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:54:26.104 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26018 ms) 2024-08-19T12:54:26.104 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-08-19T12:54:26.105 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:54:26.105 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14083 ms) 2024-08-19T12:54:26.105 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-08-19T12:54:26.105 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11920 ms) 2024-08-19T12:54:26.105 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-08-19T12:54:26.105 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-08-19T12:54:26.105 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: 00000010 69 73 |is| 2024-08-19T12:54:26.105 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: 00000012 2024-08-19T12:54:26.105 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12024 ms) 2024-08-19T12:54:26.105 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-08-19T12:54:26.105 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: require_osd_release = squid 2024-08-19T12:54:26.105 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5970 ms) 2024-08-19T12:54:26.106 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (677274 ms total) 2024-08-19T12:54:26.106 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: 2024-08-19T12:54:26.106 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-08-19T12:54:26.106 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-08-19T12:54:26.106 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1016 ms) 2024-08-19T12:54:26.106 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-08-19T12:54:26.106 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (46964 ms) 2024-08-19T12:54:26.106 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-08-19T12:54:26.106 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (19490 ms) 2024-08-19T12:54:26.106 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-08-19T12:54:26.106 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (7 ms) 2024-08-19T12:54:26.106 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (67477 ms total) 2024-08-19T12:54:26.107 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: 2024-08-19T12:54:26.107 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-08-19T12:54:26.107 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-08-19T12:54:26.107 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7054 ms) 2024-08-19T12:54:26.107 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-08-19T12:54:26.107 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8939 ms) 2024-08-19T12:54:26.107 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-08-19T12:54:26.107 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: waiting for scrub... 2024-08-19T12:54:26.107 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: done waiting 2024-08-19T12:54:26.107 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (23398 ms) 2024-08-19T12:54:26.107 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-08-19T12:54:26.108 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10948 ms) 2024-08-19T12:54:26.108 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-08-19T12:54:26.108 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7759 ms) 2024-08-19T12:54:26.108 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-08-19T12:54:26.108 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8068 ms) 2024-08-19T12:54:26.108 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-08-19T12:54:26.108 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (44744 ms) 2024-08-19T12:54:26.108 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-08-19T12:55:47.434 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.f 2024-08-19T12:55:47.435 INFO:tasks.ceph.mon.f:Restarting daemon 2024-08-19T12:55:47.435 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-08-19T12:55:47.440 INFO:tasks.ceph.mon.f:Started 2024-08-19T12:55:47.440 INFO:tasks.mon_thrash:Sending STOP to mon e 2024-08-19T12:55:47.440 INFO:tasks.ceph.mon.e:Sent signal 19 2024-08-19T12:55:47.440 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:56:02.441 INFO:tasks.mon_thrash:Sending CONT to mon e 2024-08-19T12:56:02.442 INFO:tasks.ceph.mon.e:Sent signal 18 2024-08-19T12:56:02.442 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:56:02.443 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:56:02.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.561+0000 7fe1b80e5640 10 monclient: get_monmap_and_config 2024-08-19T12:56:02.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.561+0000 7fe1b80e5640 10 monclient: build_initial_monmap 2024-08-19T12:56:02.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.561+0000 7fe1b80e5640 10 monclient: monmap: 2024-08-19T12:56:02.562 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:02.562 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:02.562486+0000 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:02.562486+0000 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:02.563 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:02.564 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:02.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 1 Processor -- start 2024-08-19T12:56:02.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 1 -- start start 2024-08-19T12:56:02.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 10 monclient: init 2024-08-19T12:56:02.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:02.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 10 monclient: _add_conns ranks=[3,5,0,7,8,6,4,1,2] 2024-08-19T12:56:02.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 10 monclient(hunting): picked mon.noname-g con 0x7fe1b006e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:02.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 10 monclient(hunting): picked mon.noname-h con 0x7fe1b00a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:02.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 10 monclient(hunting): picked mon.noname-a con 0x7fe1b00a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:02.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b0136790 con 0x7fe1b00a5d50 2024-08-19T12:56:02.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b009ddf0 con 0x7fe1b006e7b0 2024-08-19T12:56:02.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b009e0d0 con 0x7fe1b00a5920 2024-08-19T12:56:02.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:02.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.562+0000 7fe1b80e5640 10 monclient(hunting): authenticate will time out at 1756.649971s 2024-08-19T12:56:02.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.563+0000 7fe1b6682640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe1b00a5d50 0x7fe1b0141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52566/0 (socket says 172.21.15.26:52566) 2024-08-19T12:56:02.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.563+0000 7fe1b6682640 1 -- 172.21.15.26:0/3573480098 learned_addr learned my addr 172.21.15.26:0/3573480098 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:02.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b6682640 1 -- 172.21.15.26:0/3573480098 >> v1:172.21.15.26:6789/0 conn(0x7fe1b00a5d50 legacy=0x7fe1b0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:02.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b5e81640 1 -- 172.21.15.26:0/3573480098 >> v1:172.21.15.104:6790/0 conn(0x7fe1b006e7b0 legacy=0x7fe1b0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:02.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b5680640 1 -- 172.21.15.26:0/3573480098 >> v1:172.21.15.104:6791/0 conn(0x7fe1b00a5920 legacy=0x7fe1b013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:02.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4059833130 0 0) 0x7fe1b0136790 con 0x7fe1b00a5d50 2024-08-19T12:56:02.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient(hunting): _negotiate 2024-08-19T12:56:02.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:02.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:02.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient(hunting): my global_id is 194346 2024-08-19T12:56:02.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe19c003520 con 0x7fe1b00a5d50 2024-08-19T12:56:02.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 475387789 0 0) 0x7fe1ac0014a0 con 0x7fe1b006e7b0 2024-08-19T12:56:02.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient(hunting): _negotiate 2024-08-19T12:56:02.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:02.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:02.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient(hunting): my global_id is 194286 2024-08-19T12:56:02.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1b0136790 con 0x7fe1b006e7b0 2024-08-19T12:56:02.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1888436873 0 0) 0x7fe1b009e0d0 con 0x7fe1b00a5920 2024-08-19T12:56:02.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient(hunting): _negotiate 2024-08-19T12:56:02.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:02.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:02.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient(hunting): my global_id is 184532 2024-08-19T12:56:02.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1ac0014a0 con 0x7fe1b00a5920 2024-08-19T12:56:02.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 20326041 0 0) 0x7fe19c003520 con 0x7fe1b00a5d50 2024-08-19T12:56:02.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 >> v1:172.21.15.104:6791/0 conn(0x7fe1b00a5920 legacy=0x7fe1b013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:02.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 >> v1:172.21.15.104:6790/0 conn(0x7fe1b006e7b0 legacy=0x7fe1b0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:02.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient: _finish_hunting 0 2024-08-19T12:56:02.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 monclient: found mon.noname-a 2024-08-19T12:56:02.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b009e3b0 con 0x7fe1b00a5d50 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient: _finish_auth 0 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient: _check_auth_tickets 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a8001e60 con 0x7fe1b00a5d50 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b80e5640 5 monclient: authenticate success, global_id 194346 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b80e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient: dump: 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:02.577 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:02.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b80e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1a8004c20 con 0x7fe1b00a5d50 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 1 -- 172.21.15.26:0/3573480098 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a8005190 con 0x7fe1b00a5d50 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b80e5640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b80e5640 4 set_mon_vals no callback set 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.565+0000 7fe1b4e7f640 10 monclient: dump: 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:02.579 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.566+0000 7fe1b80e5640 10 monclient: shutdown 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.566+0000 7fe1b80e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.566+0000 7fe1b80e5640 1 -- 172.21.15.26:0/3573480098 >> v1:172.21.15.26:6789/0 conn(0x7fe1b00a5d50 legacy=0x7fe1b0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.566+0000 7fe1b80e5640 1 -- 172.21.15.26:0/3573480098 shutdown_connections 2024-08-19T12:56:02.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.566+0000 7fe1b80e5640 1 -- 172.21.15.26:0/3573480098 >> 172.21.15.26:0/3573480098 conn(0x7fe1b00a8740 msgr2=0x7fe1b00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.566+0000 7fe1b80e5640 1 -- 172.21.15.26:0/3573480098 shutdown_connections 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.566+0000 7fe1b80e5640 1 -- 172.21.15.26:0/3573480098 wait complete. 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 10 monclient: build_initial_monmap 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 10 monclient: monmap: 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:02.581 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 1 librados: starting msgr at 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 1 librados: starting objecter 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 1 Processor -- start 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 1 -- start start 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 1 librados: setting wanted keys 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 1 librados: calling monclient init 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 10 monclient: init 2024-08-19T12:56:02.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:02.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 10 monclient: _add_conns ranks=[7,8,6,1,0,4,5,3,2] 2024-08-19T12:56:02.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 10 monclient(hunting): picked mon.noname-i con 0x7fe1b006e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:02.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 10 monclient(hunting): picked mon.noname-e con 0x7fe1b00a5920 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:02.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 10 monclient(hunting): picked mon.noname-d con 0x7fe1b00a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:02.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b0125640 con 0x7fe1b00a5d50 2024-08-19T12:56:02.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b0125bd0 con 0x7fe1b006e7b0 2024-08-19T12:56:02.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b0125eb0 con 0x7fe1b00a5920 2024-08-19T12:56:02.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:02.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.567+0000 7fe1b80e5640 10 monclient(hunting): authenticate will time out at 1756.654962s 2024-08-19T12:56:02.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.568+0000 7fe1b5680640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fe1b00a5920 0x7fe1b01d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:57902/0 (socket says 172.21.15.26:57902) 2024-08-19T12:56:02.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.568+0000 7fe1b5680640 1 -- 172.21.15.26:0/1063370280 learned_addr learned my addr 172.21.15.26:0/1063370280 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:02.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1b5680640 1 -- 172.21.15.26:0/1063370280 >> v1:172.21.15.26:6793/0 conn(0x7fe1b00a5920 legacy=0x7fe1b01d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:02.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 client.?.objecter ms_handle_connect 0x7fe1b00a5920 2024-08-19T12:56:02.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:02.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1b6682640 1 -- 172.21.15.26:0/1063370280 >> v1:172.21.15.26:6792/0 conn(0x7fe1b00a5d50 legacy=0x7fe1b01d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:02.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 client.?.objecter ms_handle_connect 0x7fe1b00a5d50 2024-08-19T12:56:02.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:02.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1b5e81640 1 -- 172.21.15.26:0/1063370280 >> v1:172.21.15.104:6792/0 conn(0x7fe1b006e7b0 legacy=0x7fe1b0142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:02.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 client.?.objecter ms_handle_connect 0x7fe1b006e7b0 2024-08-19T12:56:02.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:02.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2160363527 0 0) 0x7fe1b0125640 con 0x7fe1b00a5d50 2024-08-19T12:56:02.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:02.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:02.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:02.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): my global_id is 194289 2024-08-19T12:56:02.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe194004eb0 con 0x7fe1b00a5d50 2024-08-19T12:56:02.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1042365639 0 0) 0x7fe1b0125eb0 con 0x7fe1b00a5920 2024-08-19T12:56:02.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:02.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:02.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:02.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): my global_id is 184472 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1b0125640 con 0x7fe1b00a5920 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3043826105 0 0) 0x7fe1b0125bd0 con 0x7fe1b006e7b0 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): my global_id is 136249 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1b0125eb0 con 0x7fe1b006e7b0 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1732247248 0 0) 0x7fe194004eb0 con 0x7fe1b00a5d50 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1b0125bd0 con 0x7fe1b00a5d50 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a80020b0 con 0x7fe1b00a5d50 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): dump: 2024-08-19T12:56:02.586 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:02.587 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:02.587 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:02.587 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:02.587 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:02.587 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:02.587 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:02.587 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:02.590 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:02.590 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:02.590 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:02.590 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:02.590 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:02.590 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:02.590 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:02.590 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:02.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:02.590 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1775986298 0 0) 0x7fe1b0125640 con 0x7fe1b00a5920 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe194004eb0 con 0x7fe1b00a5920 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a0003610 con 0x7fe1b00a5920 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): dump: 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1540480761 0 0) 0x7fe1b0125eb0 con 0x7fe1b006e7b0 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1b0125640 con 0x7fe1b006e7b0 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1ac003950 con 0x7fe1b006e7b0 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:02.592 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient(hunting): dump: 2024-08-19T12:56:02.593 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:02.593 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:02.593 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:02.593 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:02.593 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:02.593 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:02.593 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:02.593 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:02.593 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:02.593 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:02.593 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3520451806 0 0) 0x7fe1b0125bd0 con 0x7fe1b00a5d50 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 >> v1:172.21.15.26:6793/0 conn(0x7fe1b00a5920 legacy=0x7fe1b01d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 >> v1:172.21.15.104:6792/0 conn(0x7fe1b006e7b0 legacy=0x7fe1b0142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient: _finish_hunting 0 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 monclient: found mon.d 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:02.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b01258f0 con 0x7fe1b00a5d50 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient: _finish_auth 0 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 10 monclient: _check_auth_tickets 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1b80e5640 5 monclient: authenticate success, global_id 194289 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1b80e5640 10 monclient: _renew_subs 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1b80e5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.570+0000 7fe1b80e5640 1 -- 172.21.15.26:0/1063370280 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b0142b10 con 0x7fe1b00a5d50 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1b80e5640 10 client.194289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1b80e5640 10 monclient: _renew_subs 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1b80e5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1b80e5640 1 -- 172.21.15.26:0/1063370280 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1b0123d70 con 0x7fe1b00a5d50 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1b80e5640 1 librados: init done 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1a8004c20 con 0x7fe1b00a5d50 2024-08-19T12:56:02.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1a67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe1a8001ec0 con 0x7fe1b00a5d50 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1a7fff640 4 set_mon_vals no callback set 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1a67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1a67fc640 10 monclient: dump: 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:02.596 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe1a80119c0 con 0x7fe1b00a5d50 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1b80e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1b80e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1b80e5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.571+0000 7fe1b80e5640 1 -- 172.21.15.26:0/1063370280 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe198005080 con 0x7fe1b00a5d50 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.572+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(652..652 src has 350..652) ==== 5913+0+0 (unknown 2030158540 0 0) 0x7fe1a804e470 con 0x7fe1b00a5d50 2024-08-19T12:56:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.572+0000 7fe1a67fc640 10 client.194289.objecter ms_dispatch 0x7fe1b0000b90 osd_map(652..652 src has 350..652) 2024-08-19T12:56:02.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.572+0000 7fe1a67fc640 3 client.194289.objecter handle_osd_map got epochs [652,652] > 0 2024-08-19T12:56:02.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.572+0000 7fe1a67fc640 3 client.194289.objecter handle_osd_map decoding full epoch 652 2024-08-19T12:56:02.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.572+0000 7fe1b5e81640 10 monclient: get_auth_request con 0x7fe194041b50 auth_method 0 2024-08-19T12:56:02.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.572+0000 7fe1a67fc640 20 client.194289.objecter dump_active .. 0 homeless 2024-08-19T12:56:02.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.574+0000 7fe1a67fc640 10 client.194289.objecter ms_handle_connect 0x7fe194041b50 2024-08-19T12:56:02.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.576+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fe1a801b480 con 0x7fe1b00a5d50 2024-08-19T12:56:02.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.576+0000 7fe1a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:02.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.576+0000 7fe1a67fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:02.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.832+0000 7fe1b80e5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:56:02.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.832+0000 7fe1b80e5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:56:02.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.832+0000 7fe1b80e5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:02.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.832+0000 7fe1b80e5640 1 -- 172.21.15.26:0/1063370280 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe198005370 con 0x7fe1b00a5d50 2024-08-19T12:56:02.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.833+0000 7fe1a67fc640 1 -- 172.21.15.26:0/1063370280 <== mon.6 v1:172.21.15.26:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2730 (unknown 254746657 0 303400706) 0x7fe1a801ff30 con 0x7fe1b00a5d50 2024-08-19T12:56:02.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.833+0000 7fe1a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:56:02.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.833+0000 7fe1a67fc640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:02.834 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":124,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:56:02.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 10 librados: watch_flush enter 2024-08-19T12:56:02.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 10 librados: watch_flush exit 2024-08-19T12:56:02.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 20 client.194289.objecter shutdown clearing up homeless session... 2024-08-19T12:56:02.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 10 client.194289.objecter successfully canceled tick 2024-08-19T12:56:02.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 1 -- 172.21.15.26:0/1063370280 >> v1:172.21.15.104:6800/899887506 conn(0x7fe194041b50 legacy=0x7fe194044010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:02.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 10 monclient: shutdown 2024-08-19T12:56:02.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:02.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 1 -- 172.21.15.26:0/1063370280 >> v1:172.21.15.26:6792/0 conn(0x7fe1b00a5d50 legacy=0x7fe1b01d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:02.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 1 -- 172.21.15.26:0/1063370280 shutdown_connections 2024-08-19T12:56:02.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 1 -- 172.21.15.26:0/1063370280 >> 172.21.15.26:0/1063370280 conn(0x7fe1b00a8740 msgr2=0x7fe1b00aa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:02.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 1 -- 172.21.15.26:0/1063370280 shutdown_connections 2024-08-19T12:56:02.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 1 -- 172.21.15.26:0/1063370280 wait complete. 2024-08-19T12:56:02.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.835+0000 7fe1b80e5640 1 librados: shutdown 2024-08-19T12:56:02.846 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:56:02.846 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:56:02.846 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:56:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.945+0000 7f3a1548b640 10 monclient: get_monmap_and_config 2024-08-19T12:56:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.945+0000 7f3a1548b640 10 monclient: build_initial_monmap 2024-08-19T12:56:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.945+0000 7f3a1548b640 10 monclient: monmap: 2024-08-19T12:56:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:02.946750+0000 2024-08-19T12:56:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:02.946750+0000 2024-08-19T12:56:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:02.947 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:02.948 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.946+0000 7f3a1548b640 1 Processor -- start 2024-08-19T12:56:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.946+0000 7f3a1548b640 1 -- start start 2024-08-19T12:56:02.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.946+0000 7f3a1548b640 10 monclient: init 2024-08-19T12:56:02.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.946+0000 7f3a1548b640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:02.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.946+0000 7f3a1548b640 10 monclient: _add_conns ranks=[4,5,1,7,3,2,6,8,0] 2024-08-19T12:56:02.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.946+0000 7f3a1548b640 10 monclient(hunting): picked mon.noname-c con 0x7f3a1006e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:02.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.947+0000 7f3a1548b640 10 monclient(hunting): picked mon.noname-h con 0x7f3a100a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:02.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.947+0000 7f3a1548b640 10 monclient(hunting): picked mon.noname-f con 0x7f3a100a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:02.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.947+0000 7f3a1548b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a10136790 con 0x7f3a100a5d50 2024-08-19T12:56:02.950 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.947+0000 7f3a1548b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a1009ddf0 con 0x7f3a1006e7b0 2024-08-19T12:56:02.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.947+0000 7f3a1548b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a1009e0d0 con 0x7f3a100a5920 2024-08-19T12:56:02.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.947+0000 7f3a1548b640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:02.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.947+0000 7f3a1548b640 10 monclient(hunting): authenticate will time out at 1757.034066s 2024-08-19T12:56:02.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.947+0000 7f3a0effd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f3a1006e7b0 0x7f3a10136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59952/0 (socket says 172.21.15.26:59952) 2024-08-19T12:56:02.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.947+0000 7f3a0effd640 1 -- 172.21.15.26:0/43945901 learned_addr learned my addr 172.21.15.26:0/43945901 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:02.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.949+0000 7f3a0effd640 1 -- 172.21.15.26:0/43945901 >> v1:172.21.15.26:6791/0 conn(0x7f3a1006e7b0 legacy=0x7f3a10136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:02.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.949+0000 7f3a0f7fe640 1 -- 172.21.15.26:0/43945901 >> v1:172.21.15.104:6789/0 conn(0x7f3a100a5d50 legacy=0x7f3a10141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:02.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:02.949+0000 7f3a0e7fc640 1 -- 172.21.15.26:0/43945901 >> v1:172.21.15.104:6791/0 conn(0x7f3a100a5920 legacy=0x7f3a1013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.406+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3238103092 0 0) 0x7f3a080017d0 con 0x7f3a1006e7b0 2024-08-19T12:56:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.406+0000 7f3a0dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.406+0000 7f3a0dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.406+0000 7f3a0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.406+0000 7f3a0dffb640 10 monclient(hunting): my global_id is 184639 2024-08-19T12:56:03.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.406+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39fc003520 con 0x7f3a1006e7b0 2024-08-19T12:56:03.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.409+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 925719178 0 0) 0x7f3a10136790 con 0x7f3a100a5d50 2024-08-19T12:56:03.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.409+0000 7f3a0dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.409+0000 7f3a0dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.409+0000 7f3a0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.409+0000 7f3a0dffb640 10 monclient(hunting): my global_id is 204103 2024-08-19T12:56:03.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.409+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a080017d0 con 0x7f3a100a5d50 2024-08-19T12:56:03.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.410+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3765703694 0 0) 0x7f3a1009e0d0 con 0x7f3a100a5920 2024-08-19T12:56:03.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.410+0000 7f3a0dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.410+0000 7f3a0dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.410+0000 7f3a0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.410+0000 7f3a0dffb640 10 monclient(hunting): my global_id is 184541 2024-08-19T12:56:03.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.410+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a10136790 con 0x7f3a100a5920 2024-08-19T12:56:03.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 496070193 0 0) 0x7f3a10136790 con 0x7f3a100a5920 2024-08-19T12:56:03.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 >> v1:172.21.15.26:6791/0 conn(0x7f3a1006e7b0 legacy=0x7f3a10136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 >> v1:172.21.15.104:6789/0 conn(0x7f3a100a5d50 legacy=0x7f3a10141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:56:03.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 1 monclient: found mon.noname-h 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1009e3b0 con 0x7f3a100a5920 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 10 monclient: _finish_auth 0 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f39f8001a00 con 0x7f3a100a5920 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 10 monclient: dump: 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.606 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.607 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.607 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.607 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.607 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.607 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.607 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.607 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.607 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.607 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.610 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a1548b640 5 monclient: authenticate success, global_id 184541 2024-08-19T12:56:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a1548b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f39f8001c50 con 0x7f3a100a5920 2024-08-19T12:56:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:03.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 1 -- 172.21.15.26:0/43945901 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f39f8003950 con 0x7f3a100a5920 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a0dffb640 10 monclient: dump: 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.611 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a1548b640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.603+0000 7f3a1548b640 4 set_mon_vals no callback set 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.604+0000 7f3a1548b640 10 monclient: shutdown 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.604+0000 7f3a1548b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.604+0000 7f3a1548b640 1 -- 172.21.15.26:0/43945901 >> v1:172.21.15.104:6791/0 conn(0x7f3a100a5920 legacy=0x7f3a1013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.604+0000 7f3a1548b640 1 -- 172.21.15.26:0/43945901 shutdown_connections 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.604+0000 7f3a1548b640 1 -- 172.21.15.26:0/43945901 >> 172.21.15.26:0/43945901 conn(0x7f3a100a8740 msgr2=0x7f3a100a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.604+0000 7f3a1548b640 1 -- 172.21.15.26:0/43945901 shutdown_connections 2024-08-19T12:56:03.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.604+0000 7f3a1548b640 1 -- 172.21.15.26:0/43945901 wait complete. 2024-08-19T12:56:03.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.605+0000 7f3a1548b640 10 monclient: build_initial_monmap 2024-08-19T12:56:03.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.605+0000 7f3a1548b640 10 monclient: monmap: 2024-08-19T12:56:03.619 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:03.619 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:03.619 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:03.619 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.605+0000 7f3a1548b640 1 librados: starting msgr at 2024-08-19T12:56:03.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.605+0000 7f3a1548b640 1 librados: starting objecter 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.605+0000 7f3a1548b640 1 Processor -- start 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 1 -- start start 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 1 librados: setting wanted keys 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 1 librados: calling monclient init 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 10 monclient: init 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 10 monclient: _add_conns ranks=[3,6,8,4,2,5,1,0,7] 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 10 monclient(hunting): picked mon.noname-g con 0x7f3a1006e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 10 monclient(hunting): picked mon.noname-d con 0x7f3a100a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 10 monclient(hunting): picked mon.noname-e con 0x7f3a100a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a1013b2f0 con 0x7f3a1006e7b0 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a10138350 con 0x7f3a100a5920 2024-08-19T12:56:03.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a1013b010 con 0x7f3a100a5d50 2024-08-19T12:56:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a1548b640 10 monclient(hunting): authenticate will time out at 1757.693613s 2024-08-19T12:56:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a0e7fc640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f3a100a5920 0x7f3a10137c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34028/0 (socket says 172.21.15.26:34028) 2024-08-19T12:56:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a0f7fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f3a100a5d50 0x7f3a101d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:57904/0 (socket says 172.21.15.26:57904) 2024-08-19T12:56:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.606+0000 7f3a0e7fc640 1 -- 172.21.15.26:0/3892566155 learned_addr learned my addr 172.21.15.26:0/3892566155 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.608+0000 7f3a0e7fc640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.26:6792/0 conn(0x7f3a100a5920 legacy=0x7f3a10137c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.608+0000 7f3a077fe640 10 client.?.objecter ms_handle_connect 0x7f3a100a5920 2024-08-19T12:56:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.608+0000 7f3a077fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.608+0000 7f3a077fe640 10 client.?.objecter ms_handle_connect 0x7f3a100a5d50 2024-08-19T12:56:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.608+0000 7f3a077fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.608+0000 7f3a0f7fe640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.26:6793/0 conn(0x7f3a100a5d50 legacy=0x7f3a101d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a0effd640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.104:6790/0 conn(0x7f3a1006e7b0 legacy=0x7f3a10137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 client.?.objecter ms_handle_connect 0x7f3a1006e7b0 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2250609359 0 0) 0x7f3a10138350 con 0x7f3a100a5920 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): my global_id is 194298 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39f0004ee0 con 0x7f3a100a5920 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 506918704 0 0) 0x7f3a1013b010 con 0x7f3a100a5d50 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): my global_id is 184481 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a10138350 con 0x7f3a100a5d50 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1731424019 0 0) 0x7f3a1013b2f0 con 0x7f3a1006e7b0 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): my global_id is 194295 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a1013b010 con 0x7f3a1006e7b0 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2473825401 0 0) 0x7f39f0004ee0 con 0x7f3a100a5920 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a1013b2f0 con 0x7f3a100a5920 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f39f8003950 con 0x7f3a100a5920 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): dump: 2024-08-19T12:56:03.624 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.625 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.626 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:03.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1879415165 0 0) 0x7f3a10138350 con 0x7f3a100a5d50 2024-08-19T12:56:03.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39f0004ee0 con 0x7f3a100a5d50 2024-08-19T12:56:03.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3a00003610 con 0x7f3a100a5d50 2024-08-19T12:56:03.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:03.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:03.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): dump: 2024-08-19T12:56:03.627 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.627 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.627 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.627 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.627 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.627 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.627 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.627 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.627 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.627 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1520426433 0 0) 0x7f3a1013b010 con 0x7f3a1006e7b0 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a10138350 con 0x7f3a1006e7b0 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3a08001e50 con 0x7f3a1006e7b0 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient(hunting): dump: 2024-08-19T12:56:03.628 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.629 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4026732143 0 0) 0x7f3a1013b2f0 con 0x7f3a100a5920 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.26:6793/0 conn(0x7f3a100a5d50 legacy=0x7f3a101d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.104:6790/0 conn(0x7f3a1006e7b0 legacy=0x7f3a10137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient: _finish_hunting 0 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 monclient: found mon.d 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a10139d60 con 0x7f3a100a5920 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient: _finish_auth 0 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient: _check_auth_tickets 2024-08-19T12:56:03.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a1548b640 5 monclient: authenticate success, global_id 194298 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a1548b640 10 monclient: _renew_subs 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a1548b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a1548b640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3a10142a60 con 0x7f3a100a5920 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a1548b640 10 client.194298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a1548b640 10 monclient: _renew_subs 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a1548b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a1548b640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f3a101d7b90 con 0x7f3a100a5920 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a1548b640 1 librados: init done 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f39f8003f40 con 0x7f3a100a5920 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f39f80057c0 con 0x7f3a100a5920 2024-08-19T12:56:03.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a0d7fa640 4 set_mon_vals no callback set 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.609+0000 7f3a077fe640 10 monclient: dump: 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.610+0000 7f3a1548b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.610+0000 7f3a1548b640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f39f40065c0 con 0x7f39f4002860 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.610+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f39f8011c20 con 0x7f3a100a5920 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.611+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(653..653 src has 350..653) ==== 5913+0+0 (unknown 1656795161 0 0) 0x7f39f804e380 con 0x7f3a100a5920 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.611+0000 7f3a077fe640 10 client.194298.objecter ms_dispatch 0x7f3a10000b90 osd_map(653..653 src has 350..653) 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.611+0000 7f3a077fe640 3 client.194298.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.611+0000 7f3a077fe640 3 client.194298.objecter handle_osd_map decoding full epoch 653 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.611+0000 7f3a077fe640 20 client.194298.objecter dump_active .. 0 homeless 2024-08-19T12:56:03.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.611+0000 7f3a0f7fe640 10 monclient: get_auth_request con 0x7f39f0041870 auth_method 0 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 10 client.194298.objecter ms_handle_connect 0x7f39f4002860 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 10 client.194298.objecter resend_mon_ops 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a0effd640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.26:6789/0 conn(0x7f39f4002860 legacy=0x7f39f4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3071459963 0 0) 0x7f39f40065c0 con 0x7f39f4002860 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 10 monclient(hunting): my global_id is 194355 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a1013b2f0 con 0x7f39f4002860 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3225144950 0 0) 0x7f3a1013b2f0 con 0x7f39f4002860 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39f40065c0 con 0x7f39f4002860 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3a080052e0 con 0x7f39f4002860 2024-08-19T12:56:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 10 monclient: dump: 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.635 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1581914591 0 0) 0x7f39f40065c0 con 0x7f39f4002860 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39f400a160 con 0x7f39f4002860 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.613+0000 7f3a077fe640 10 client.194298.objecter ms_handle_connect 0x7f39f0041870 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.614+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3a08005480 con 0x7f39f4002860 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.614+0000 7f3a077fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.614+0000 7f3a077fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.614+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.26:6789/0 conn(0x7f39f4002860 legacy=0x7f39f4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.614+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.26:6789/0 conn(0x7f39f4002860 legacy=0x7f39f4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:03.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.620+0000 7f3a1548b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:03.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.620+0000 7f3a1548b640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f39f4009e70 con 0x7f39f4009130 2024-08-19T12:56:03.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.622+0000 7f3a0effd640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.26:6789/0 conn(0x7f39f4009130 legacy=0x7f39f40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.622+0000 7f3a077fe640 10 client.194298.objecter ms_handle_connect 0x7f39f4009130 2024-08-19T12:56:03.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.622+0000 7f3a077fe640 10 client.194298.objecter resend_mon_ops 2024-08-19T12:56:03.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319023304 0 0) 0x7f39f4009e70 con 0x7f39f4009130 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 10 monclient(hunting): my global_id is 194364 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39f40065c0 con 0x7f39f4009130 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1531654248 0 0) 0x7f39f40065c0 con 0x7f39f4009130 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39f4009e70 con 0x7f39f4009130 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3a08005d10 con 0x7f39f4009130 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 10 monclient: dump: 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.639 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.640 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 271753751 0 0) 0x7f39f4009e70 con 0x7f39f4009130 2024-08-19T12:56:03.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f39f400c3f0 con 0x7f39f4009130 2024-08-19T12:56:03.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7547 (unknown 0 0 930601430) 0x7f3a08007250 con 0x7f39f4009130 2024-08-19T12:56:03.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:03.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:03.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.26:6789/0 conn(0x7f39f4009130 legacy=0x7f39f40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.623+0000 7f3a077fe640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.26:6789/0 conn(0x7f39f4009130 legacy=0x7f39f40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:03.641 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:03.641 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:03.641 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 148545, 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:03.642 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:03.643 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:03.644 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:03.645 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:03.646 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.647 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.648 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:03.650 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:03.650 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.650 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.650 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.650 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:03.650 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:03.650 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:03.652 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.655 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:03.655 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.655 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:03.655 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:03.655 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.655 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.655 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.657 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.658 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.659 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:03.660 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:03.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.661 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:03.661 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:03.661 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:03.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.661 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.661 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:03.661 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:03.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 10 librados: watch_flush enter 2024-08-19T12:56:03.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 10 librados: watch_flush exit 2024-08-19T12:56:03.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 20 client.194298.objecter shutdown clearing up homeless session... 2024-08-19T12:56:03.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 10 client.194298.objecter successfully canceled tick 2024-08-19T12:56:03.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.104:6800/899887506 conn(0x7f39f0041870 legacy=0x7f39f0043d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 10 monclient: shutdown 2024-08-19T12:56:03.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:03.663 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 1 -- 172.21.15.26:0/3892566155 >> v1:172.21.15.26:6792/0 conn(0x7f3a100a5920 legacy=0x7f3a10137c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a0f7fe640 1 -- 172.21.15.26:0/3892566155 reap_dead start 2024-08-19T12:56:03.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a0f7fe640 1 -- 172.21.15.26:0/3892566155 reap_dead start 2024-08-19T12:56:03.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 1 -- 172.21.15.26:0/3892566155 shutdown_connections 2024-08-19T12:56:03.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 1 -- 172.21.15.26:0/3892566155 >> 172.21.15.26:0/3892566155 conn(0x7f3a100a8740 msgr2=0x7f3a100ab030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:03.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 1 -- 172.21.15.26:0/3892566155 shutdown_connections 2024-08-19T12:56:03.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 1 -- 172.21.15.26:0/3892566155 wait complete. 2024-08-19T12:56:03.664 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.624+0000 7f3a1548b640 1 librados: shutdown 2024-08-19T12:56:03.664 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:03.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:56:03.665 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:56:03.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.768+0000 7f13ddc20640 10 monclient: get_monmap_and_config 2024-08-19T12:56:03.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.768+0000 7f13ddc20640 10 monclient: build_initial_monmap 2024-08-19T12:56:03.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.768+0000 7f13ddc20640 10 monclient: monmap: 2024-08-19T12:56:03.769 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:03.769 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.769 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:03.768962+0000 2024-08-19T12:56:03.769 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:03.768962+0000 2024-08-19T12:56:03.769 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:03.769 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:03.770 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:03.770 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.768+0000 7f13ddc20640 1 Processor -- start 2024-08-19T12:56:03.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.768+0000 7f13ddc20640 1 -- start start 2024-08-19T12:56:03.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.768+0000 7f13ddc20640 10 monclient: init 2024-08-19T12:56:03.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.769+0000 7f13ddc20640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:03.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.769+0000 7f13ddc20640 10 monclient: _add_conns ranks=[4,1,8,5,3,2,0,7,6] 2024-08-19T12:56:03.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.769+0000 7f13ddc20640 10 monclient(hunting): picked mon.noname-c con 0x7f13d8073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:03.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.769+0000 7f13ddc20640 10 monclient(hunting): picked mon.noname-f con 0x7f13d80a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.769+0000 7f13ddc20640 10 monclient(hunting): picked mon.noname-e con 0x7f13d813f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.769+0000 7f13ddc20640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d809dae0 con 0x7f13d80a57a0 2024-08-19T12:56:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.769+0000 7f13ddc20640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d8136910 con 0x7f13d8073470 2024-08-19T12:56:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.769+0000 7f13ddc20640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d809dd50 con 0x7f13d813f2e0 2024-08-19T12:56:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.769+0000 7f13ddc20640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.769+0000 7f13ddc20640 10 monclient(hunting): authenticate will time out at 1757.856488s 2024-08-19T12:56:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.769+0000 7f13d77fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f13d8073470 0x7f13d8136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59954/0 (socket says 172.21.15.26:59954) 2024-08-19T12:56:03.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.769+0000 7f13d77fe640 1 -- 172.21.15.26:0/1616534782 learned_addr learned my addr 172.21.15.26:0/1616534782 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:03.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d7fff640 1 -- 172.21.15.26:0/1616534782 >> v1:172.21.15.26:6793/0 conn(0x7f13d813f2e0 legacy=0x7f13d81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d77fe640 1 -- 172.21.15.26:0/1616534782 >> v1:172.21.15.26:6791/0 conn(0x7f13d8073470 legacy=0x7f13d8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d6ffd640 1 -- 172.21.15.26:0/1616534782 >> v1:172.21.15.104:6789/0 conn(0x7f13d80a57a0 legacy=0x7f13d80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2983074875 0 0) 0x7f13d8136910 con 0x7f13d8073470 2024-08-19T12:56:03.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d67fc640 10 monclient(hunting): my global_id is 184648 2024-08-19T12:56:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13c0003520 con 0x7f13d8073470 2024-08-19T12:56:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4185831365 0 0) 0x7f13d809dd50 con 0x7f13d813f2e0 2024-08-19T12:56:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d67fc640 10 monclient(hunting): my global_id is 184490 2024-08-19T12:56:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.771+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d8136910 con 0x7f13d813f2e0 2024-08-19T12:56:03.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3932852351 0 0) 0x7f13d809dae0 con 0x7f13d80a57a0 2024-08-19T12:56:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient(hunting): my global_id is 204112 2024-08-19T12:56:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d809dd50 con 0x7f13d80a57a0 2024-08-19T12:56:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 298884702 0 0) 0x7f13c0003520 con 0x7f13d8073470 2024-08-19T12:56:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 >> v1:172.21.15.26:6793/0 conn(0x7f13d813f2e0 legacy=0x7f13d81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 >> v1:172.21.15.104:6789/0 conn(0x7f13d80a57a0 legacy=0x7f13d80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient: _finish_hunting 0 2024-08-19T12:56:03.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 1 monclient: found mon.noname-c 2024-08-19T12:56:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:56:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d809e030 con 0x7f13d8073470 2024-08-19T12:56:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient: _finish_auth 0 2024-08-19T12:56:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient: _check_auth_tickets 2024-08-19T12:56:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f13c8001ae0 con 0x7f13d8073470 2024-08-19T12:56:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:56:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient: dump: 2024-08-19T12:56:03.781 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.782 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.783 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13ddc20640 5 monclient: authenticate success, global_id 184648 2024-08-19T12:56:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13ddc20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f13c80048a0 con 0x7f13d8073470 2024-08-19T12:56:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 1 -- 172.21.15.26:0/1616534782 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f13c8004e10 con 0x7f13d8073470 2024-08-19T12:56:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:03.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13d67fc640 10 monclient: dump: 2024-08-19T12:56:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.784 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.785 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13ddc20640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13ddc20640 4 set_mon_vals no callback set 2024-08-19T12:56:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13ddc20640 10 monclient: shutdown 2024-08-19T12:56:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13ddc20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13ddc20640 1 -- 172.21.15.26:0/1616534782 >> v1:172.21.15.26:6791/0 conn(0x7f13d8073470 legacy=0x7f13d8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13ddc20640 1 -- 172.21.15.26:0/1616534782 shutdown_connections 2024-08-19T12:56:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13ddc20640 1 -- 172.21.15.26:0/1616534782 >> 172.21.15.26:0/1616534782 conn(0x7f13d80abff0 msgr2=0x7f13d80ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:03.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13ddc20640 1 -- 172.21.15.26:0/1616534782 shutdown_connections 2024-08-19T12:56:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.772+0000 7f13ddc20640 1 -- 172.21.15.26:0/1616534782 wait complete. 2024-08-19T12:56:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.773+0000 7f13ddc20640 10 monclient: build_initial_monmap 2024-08-19T12:56:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.773+0000 7f13ddc20640 10 monclient: monmap: 2024-08-19T12:56:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:03.786 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:03.787 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.773+0000 7f13ddc20640 1 librados: starting msgr at 2024-08-19T12:56:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.773+0000 7f13ddc20640 1 librados: starting objecter 2024-08-19T12:56:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.773+0000 7f13ddc20640 1 Processor -- start 2024-08-19T12:56:03.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 1 -- start start 2024-08-19T12:56:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 1 librados: setting wanted keys 2024-08-19T12:56:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 1 librados: calling monclient init 2024-08-19T12:56:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 10 monclient: init 2024-08-19T12:56:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 10 monclient: _add_conns ranks=[8,3,6,1,5,0,2,4,7] 2024-08-19T12:56:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 10 monclient(hunting): picked mon.noname-e con 0x7f13d8073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 10 monclient(hunting): picked mon.noname-g con 0x7f13d80a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 10 monclient(hunting): picked mon.noname-d con 0x7f13d813f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d8131f00 con 0x7f13d80a57a0 2024-08-19T12:56:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d81cef20 con 0x7f13d813f2e0 2024-08-19T12:56:03.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d8131800 con 0x7f13d8073470 2024-08-19T12:56:03.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:03.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13ddc20640 10 monclient(hunting): authenticate will time out at 1757.861579s 2024-08-19T12:56:03.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13d77fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f13d8073470 0x7f13d81ce100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:57928/0 (socket says 172.21.15.26:57928) 2024-08-19T12:56:03.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.774+0000 7f13d77fe640 1 -- 172.21.15.26:0/604252853 learned_addr learned my addr 172.21.15.26:0/604252853 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:03.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13d77fe640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.26:6793/0 conn(0x7f13d8073470 legacy=0x7f13d81ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 client.?.objecter ms_handle_connect 0x7f13d8073470 2024-08-19T12:56:03.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:03.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 client.?.objecter ms_handle_connect 0x7f13d813f2e0 2024-08-19T12:56:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13d7fff640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.26:6792/0 conn(0x7f13d813f2e0 legacy=0x7f13d81d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 237274374 0 0) 0x7f13d8131800 con 0x7f13d8073470 2024-08-19T12:56:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 monclient(hunting): my global_id is 184499 2024-08-19T12:56:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13c4004eb0 con 0x7f13d8073470 2024-08-19T12:56:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 client.?.objecter ms_handle_connect 0x7f13d80a57a0 2024-08-19T12:56:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13d6ffd640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.104:6790/0 conn(0x7f13d80a57a0 legacy=0x7f13d81ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1010228709 0 0) 0x7f13d81cef20 con 0x7f13d813f2e0 2024-08-19T12:56:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 10 monclient(hunting): my global_id is 194307 2024-08-19T12:56:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.776+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d8131800 con 0x7f13d813f2e0 2024-08-19T12:56:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 810479911 0 0) 0x7f13d809dd50 con 0x7f13d80a57a0 2024-08-19T12:56:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient(hunting): my global_id is 194304 2024-08-19T12:56:03.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d81cef20 con 0x7f13d80a57a0 2024-08-19T12:56:03.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2918785939 0 0) 0x7f13c4004eb0 con 0x7f13d8073470 2024-08-19T12:56:03.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13d809dd50 con 0x7f13d8073470 2024-08-19T12:56:03.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f13c8004d90 con 0x7f13d8073470 2024-08-19T12:56:03.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:03.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:56:03.798 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient(hunting): dump: 2024-08-19T12:56:03.798 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.798 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.798 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.798 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.798 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.799 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1120581521 0 0) 0x7f13d8131800 con 0x7f13d813f2e0 2024-08-19T12:56:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13c4004eb0 con 0x7f13d813f2e0 2024-08-19T12:56:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f13d0003990 con 0x7f13d813f2e0 2024-08-19T12:56:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient(hunting): dump: 2024-08-19T12:56:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.800 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.801 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3118260159 0 0) 0x7f13d809dd50 con 0x7f13d8073470 2024-08-19T12:56:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.26:6792/0 conn(0x7f13d813f2e0 legacy=0x7f13d81d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.104:6790/0 conn(0x7f13d80a57a0 legacy=0x7f13d81ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient: _finish_hunting 0 2024-08-19T12:56:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 monclient: found mon.e 2024-08-19T12:56:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d8131ae0 con 0x7f13d8073470 2024-08-19T12:56:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient: _finish_auth 0 2024-08-19T12:56:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient: _check_auth_tickets 2024-08-19T12:56:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13ddc20640 5 monclient: authenticate success, global_id 184499 2024-08-19T12:56:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13ddc20640 10 monclient: _renew_subs 2024-08-19T12:56:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13ddc20640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13ddc20640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f13d81331f0 con 0x7f13d8073470 2024-08-19T12:56:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13ddc20640 10 client.184499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13ddc20640 10 monclient: _renew_subs 2024-08-19T12:56:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13ddc20640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13ddc20640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f13d81d7a80 con 0x7f13d8073470 2024-08-19T12:56:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13ddc20640 1 librados: init done 2024-08-19T12:56:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f13c8001d30 con 0x7f13d8073470 2024-08-19T12:56:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f13c80048a0 con 0x7f13d8073470 2024-08-19T12:56:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13d5ffb640 4 set_mon_vals no callback set 2024-08-19T12:56:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.777+0000 7f13bffff640 10 monclient: dump: 2024-08-19T12:56:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.804 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.805 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.805 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.805 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.805 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.805 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.778+0000 7f13ddc20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:03.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.778+0000 7f13ddc20640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f13b80065c0 con 0x7f13b8002860 2024-08-19T12:56:03.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.778+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f13c8011a00 con 0x7f13d8073470 2024-08-19T12:56:03.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.778+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(653..653 src has 350..653) ==== 5913+0+0 (unknown 1656795161 0 0) 0x7f13c804e3f0 con 0x7f13d8073470 2024-08-19T12:56:03.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.778+0000 7f13bffff640 10 client.184499.objecter ms_dispatch 0x7f13d8000b90 osd_map(653..653 src has 350..653) 2024-08-19T12:56:03.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.778+0000 7f13bffff640 3 client.184499.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T12:56:03.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.778+0000 7f13bffff640 3 client.184499.objecter handle_osd_map decoding full epoch 653 2024-08-19T12:56:03.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.778+0000 7f13bffff640 20 client.184499.objecter dump_active .. 0 homeless 2024-08-19T12:56:03.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.778+0000 7f13d7fff640 10 monclient: get_auth_request con 0x7f13c4041800 auth_method 0 2024-08-19T12:56:03.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13d6ffd640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.26:6790/0 conn(0x7f13b8002860 legacy=0x7f13b8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 10 client.184499.objecter ms_handle_connect 0x7f13b8002860 2024-08-19T12:56:03.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 10 client.184499.objecter resend_mon_ops 2024-08-19T12:56:03.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2330879479 0 0) 0x7f13b80065c0 con 0x7f13b8002860 2024-08-19T12:56:03.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 10 monclient(hunting): my global_id is 27344 2024-08-19T12:56:03.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d809dd50 con 0x7f13b8002860 2024-08-19T12:56:03.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3706098742 0 0) 0x7f13d809dd50 con 0x7f13b8002860 2024-08-19T12:56:03.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13b80065c0 con 0x7f13b8002860 2024-08-19T12:56:03.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f13cc001d40 con 0x7f13b8002860 2024-08-19T12:56:03.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:03.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 10 monclient: dump: 2024-08-19T12:56:03.807 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.807 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.808 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.808 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.808 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.808 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.808 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.808 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.808 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.808 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.808 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.808 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.808 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.808 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.809 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.809 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2111105004 0 0) 0x7f13b80065c0 con 0x7f13b8002860 2024-08-19T12:56:03.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.780+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13b800a160 con 0x7f13b8002860 2024-08-19T12:56:03.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.781+0000 7f13bffff640 10 client.184499.objecter ms_handle_connect 0x7f13c4041800 2024-08-19T12:56:03.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.781+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f13cc0057c0 con 0x7f13b8002860 2024-08-19T12:56:03.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.781+0000 7f13bffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:03.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.781+0000 7f13bffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:03.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.781+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.26:6790/0 conn(0x7f13b8002860 legacy=0x7f13b8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.781+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.26:6790/0 conn(0x7f13b8002860 legacy=0x7f13b8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:03.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.787+0000 7f13ddc20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:03.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.787+0000 7f13ddc20640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f13b8009e70 con 0x7f13b8009130 2024-08-19T12:56:03.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.789+0000 7f13d6ffd640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.26:6790/0 conn(0x7f13b8009130 legacy=0x7f13b80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.789+0000 7f13bffff640 10 client.184499.objecter ms_handle_connect 0x7f13b8009130 2024-08-19T12:56:03.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.789+0000 7f13bffff640 10 client.184499.objecter resend_mon_ops 2024-08-19T12:56:03.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3242703438 0 0) 0x7f13b8009e70 con 0x7f13b8009130 2024-08-19T12:56:03.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 10 monclient(hunting): my global_id is 27353 2024-08-19T12:56:03.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13b80065c0 con 0x7f13b8009130 2024-08-19T12:56:03.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 613799895 0 0) 0x7f13b80065c0 con 0x7f13b8009130 2024-08-19T12:56:03.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13b8009e70 con 0x7f13b8009130 2024-08-19T12:56:03.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f13cc004100 con 0x7f13b8009130 2024-08-19T12:56:03.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:03.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 10 monclient: dump: 2024-08-19T12:56:03.811 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.811 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.811 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.811 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.811 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.811 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.811 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.811 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.812 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.812 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.812 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.812 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.812 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.812 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.812 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.812 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1646925981 0 0) 0x7f13b8009e70 con 0x7f13b8009130 2024-08-19T12:56:03.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f13b800c410 con 0x7f13b8009130 2024-08-19T12:56:03.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7713 (unknown 0 0 189409186) 0x7f13cc005990 con 0x7f13b8009130 2024-08-19T12:56:03.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:03.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:03.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.26:6790/0 conn(0x7f13b8009130 legacy=0x7f13b80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13bffff640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.26:6790/0 conn(0x7f13b8009130 legacy=0x7f13b80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:03.813 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:03.813 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 1076845, 2024-08-19T12:56:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:03.814 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:56:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:03.815 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:03.816 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:03.817 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:03.818 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:03.819 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:03.820 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.822 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:03.824 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:03.825 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.826 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.830 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.831 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:03.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:03.833 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:03.834 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.835 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:56:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:03.836 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:56:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.837 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:03.838 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:03.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.790+0000 7f13ddc20640 10 librados: watch_flush enter 2024-08-19T12:56:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13ddc20640 10 librados: watch_flush exit 2024-08-19T12:56:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13ddc20640 20 client.184499.objecter shutdown clearing up homeless session... 2024-08-19T12:56:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13ddc20640 10 client.184499.objecter successfully canceled tick 2024-08-19T12:56:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13ddc20640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.104:6800/899887506 conn(0x7f13c4041800 legacy=0x7f13c4043cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13ddc20640 10 monclient: shutdown 2024-08-19T12:56:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13ddc20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13ddc20640 1 -- 172.21.15.26:0/604252853 >> v1:172.21.15.26:6793/0 conn(0x7f13d8073470 legacy=0x7f13d81ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13d7fff640 1 -- 172.21.15.26:0/604252853 reap_dead start 2024-08-19T12:56:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13d7fff640 1 -- 172.21.15.26:0/604252853 reap_dead start 2024-08-19T12:56:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13ddc20640 1 -- 172.21.15.26:0/604252853 shutdown_connections 2024-08-19T12:56:03.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13ddc20640 1 -- 172.21.15.26:0/604252853 >> 172.21.15.26:0/604252853 conn(0x7f13d80abff0 msgr2=0x7f13d812c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13ddc20640 1 -- 172.21.15.26:0/604252853 shutdown_connections 2024-08-19T12:56:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13ddc20640 1 -- 172.21.15.26:0/604252853 wait complete. 2024-08-19T12:56:03.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.791+0000 7f13ddc20640 1 librados: shutdown 2024-08-19T12:56:03.841 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:03.841 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:56:03.842 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:56:03.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.940+0000 7f382b2ec640 10 monclient: get_monmap_and_config 2024-08-19T12:56:03.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.940+0000 7f382b2ec640 10 monclient: build_initial_monmap 2024-08-19T12:56:03.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.940+0000 7f382b2ec640 10 monclient: monmap: 2024-08-19T12:56:03.941 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:03.941 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.941 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:03.940910+0000 2024-08-19T12:56:03.941 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:03.940910+0000 2024-08-19T12:56:03.941 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:03.941 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.941 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:03.941 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:03.942 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:03.942 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:03.942 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:03.942 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:03.942 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:03.942 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:03.942 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:03.942 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.954 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.940+0000 7f382b2ec640 1 Processor -- start 2024-08-19T12:56:03.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.940+0000 7f382b2ec640 1 -- start start 2024-08-19T12:56:03.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.940+0000 7f382b2ec640 10 monclient: init 2024-08-19T12:56:03.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f382b2ec640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:03.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f382b2ec640 10 monclient: _add_conns ranks=[1,6,2,8,5,7,0,4,3] 2024-08-19T12:56:03.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f382b2ec640 10 monclient(hunting): picked mon.noname-f con 0x7f3824073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:03.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f382b2ec640 10 monclient(hunting): picked mon.noname-d con 0x7f38240a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:03.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f382b2ec640 10 monclient(hunting): picked mon.noname-b con 0x7f382413f2e0 addr v1:172.21.15.26:6790/0 2024-08-19T12:56:03.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f382b2ec640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f382409dae0 con 0x7f3824073470 2024-08-19T12:56:03.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f382b2ec640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3824136910 con 0x7f382413f2e0 2024-08-19T12:56:03.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f382b2ec640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f382409dd50 con 0x7f38240a57a0 2024-08-19T12:56:03.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f382b2ec640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:03.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f382b2ec640 10 monclient(hunting): authenticate will time out at 1758.028443s 2024-08-19T12:56:03.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f3828887640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f38240a57a0 0x7f38240a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34046/0 (socket says 172.21.15.26:34046) 2024-08-19T12:56:03.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f3828887640 1 -- 172.21.15.26:0/1477321881 learned_addr learned my addr 172.21.15.26:0/1477321881 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:03.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.941+0000 7f3829889640 1 --1- 172.21.15.26:0/1477321881 >> v1:172.21.15.26:6790/0 conn(0x7f382413f2e0 0x7f38241416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55132/0 (socket says 172.21.15.26:55132) 2024-08-19T12:56:03.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f3829889640 1 -- 172.21.15.26:0/1477321881 >> v1:172.21.15.26:6790/0 conn(0x7f382413f2e0 legacy=0x7f38241416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f3828887640 1 -- 172.21.15.26:0/1477321881 >> v1:172.21.15.26:6792/0 conn(0x7f38240a57a0 legacy=0x7f38240a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f3829088640 1 -- 172.21.15.26:0/1477321881 >> v1:172.21.15.104:6789/0 conn(0x7f3824073470 legacy=0x7f3824136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2084883966 0 0) 0x7f3824136910 con 0x7f382413f2e0 2024-08-19T12:56:03.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 10 monclient(hunting): my global_id is 27362 2024-08-19T12:56:03.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3810003520 con 0x7f382413f2e0 2024-08-19T12:56:03.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1999239194 0 0) 0x7f382409dd50 con 0x7f38240a57a0 2024-08-19T12:56:03.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 10 monclient(hunting): my global_id is 194316 2024-08-19T12:56:03.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3824136910 con 0x7f38240a57a0 2024-08-19T12:56:03.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 364241027 0 0) 0x7f382409dae0 con 0x7f3824073470 2024-08-19T12:56:03.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 10 monclient(hunting): my global_id is 204121 2024-08-19T12:56:03.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.943+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f382409dd50 con 0x7f3824073470 2024-08-19T12:56:03.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3314294466 0 0) 0x7f3810003520 con 0x7f382413f2e0 2024-08-19T12:56:03.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 >> v1:172.21.15.26:6792/0 conn(0x7f38240a57a0 legacy=0x7f38240a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 >> v1:172.21.15.104:6789/0 conn(0x7f3824073470 legacy=0x7f3824136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:56:03.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 1 monclient: found mon.noname-b 2024-08-19T12:56:03.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:03.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:56:03.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382409e030 con 0x7f382413f2e0 2024-08-19T12:56:03.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 10 monclient: _finish_auth 0 2024-08-19T12:56:03.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:56:03.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:03.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f381c001e60 con 0x7f382413f2e0 2024-08-19T12:56:03.959 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f382b2ec640 5 monclient: authenticate success, global_id 27362 2024-08-19T12:56:03.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f382b2ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:03.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:56:03.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 10 monclient: dump: 2024-08-19T12:56:03.960 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.960 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.960 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.960 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.960 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.960 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.960 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.961 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f382b2ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f381c004b60 con 0x7f382413f2e0 2024-08-19T12:56:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:03.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 1 -- 172.21.15.26:0/1477321881 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f381c0050d0 con 0x7f382413f2e0 2024-08-19T12:56:03.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f382b2ec640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:03.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f382b2ec640 4 set_mon_vals no callback set 2024-08-19T12:56:03.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:03.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f381bfff640 10 monclient: dump: 2024-08-19T12:56:03.962 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.963 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.963 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.963 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.963 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.963 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.963 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.963 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.963 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.963 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.963 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.964 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.964 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.964 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.964 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.964 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f382b2ec640 10 monclient: shutdown 2024-08-19T12:56:03.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f382b2ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:03.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f382b2ec640 1 -- 172.21.15.26:0/1477321881 >> v1:172.21.15.26:6790/0 conn(0x7f382413f2e0 legacy=0x7f38241416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f382b2ec640 1 -- 172.21.15.26:0/1477321881 shutdown_connections 2024-08-19T12:56:03.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f382b2ec640 1 -- 172.21.15.26:0/1477321881 >> 172.21.15.26:0/1477321881 conn(0x7f38240abff0 msgr2=0x7f38240ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:03.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f382b2ec640 1 -- 172.21.15.26:0/1477321881 shutdown_connections 2024-08-19T12:56:03.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.944+0000 7f382b2ec640 1 -- 172.21.15.26:0/1477321881 wait complete. 2024-08-19T12:56:03.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.945+0000 7f382b2ec640 10 monclient: build_initial_monmap 2024-08-19T12:56:03.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.945+0000 7f382b2ec640 10 monclient: monmap: 2024-08-19T12:56:03.965 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:03.965 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:03.965 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:03.965 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:03.965 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:03.965 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.965 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:03.965 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:03.966 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:03.966 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:03.966 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:03.966 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:03.966 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:03.966 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:03.966 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:03.966 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.945+0000 7f382b2ec640 1 librados: starting msgr at 2024-08-19T12:56:03.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.945+0000 7f382b2ec640 1 librados: starting objecter 2024-08-19T12:56:03.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.945+0000 7f382b2ec640 1 Processor -- start 2024-08-19T12:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.945+0000 7f382b2ec640 1 -- start start 2024-08-19T12:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.945+0000 7f382b2ec640 1 librados: setting wanted keys 2024-08-19T12:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.945+0000 7f382b2ec640 1 librados: calling monclient init 2024-08-19T12:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.945+0000 7f382b2ec640 10 monclient: init 2024-08-19T12:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f382b2ec640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f382b2ec640 10 monclient: _add_conns ranks=[1,7,5,4,6,8,3,2,0] 2024-08-19T12:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f382b2ec640 10 monclient(hunting): picked mon.noname-f con 0x7f3824073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f382b2ec640 10 monclient(hunting): picked mon.noname-i con 0x7f38240a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f382b2ec640 10 monclient(hunting): picked mon.noname-h con 0x7f382413f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f382b2ec640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f382412f580 con 0x7f3824073470 2024-08-19T12:56:03.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f382b2ec640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3824142b60 con 0x7f382413f2e0 2024-08-19T12:56:03.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f382b2ec640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3824131040 con 0x7f38240a57a0 2024-08-19T12:56:03.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f382b2ec640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:03.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f382b2ec640 10 monclient(hunting): authenticate will time out at 1758.033260s 2024-08-19T12:56:03.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f3829088640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f3824073470 0x7f382412dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:57252/0 (socket says 172.21.15.26:57252) 2024-08-19T12:56:03.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f3828887640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f38240a57a0 0x7f382412e3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:39900/0 (socket says 172.21.15.26:39900) 2024-08-19T12:56:03.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f3829889640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f382413f2e0 0x7f38241d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:35258/0 (socket says 172.21.15.26:35258) 2024-08-19T12:56:03.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.946+0000 7f3829088640 1 -- 172.21.15.26:0/741194549 learned_addr learned my addr 172.21.15.26:0/741194549 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:03.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3828887640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.104:6792/0 conn(0x7f38240a57a0 legacy=0x7f382412e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3829889640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.104:6791/0 conn(0x7f382413f2e0 legacy=0x7f38241d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 client.?.objecter ms_handle_connect 0x7f38240a57a0 2024-08-19T12:56:03.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:03.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 client.?.objecter ms_handle_connect 0x7f3824073470 2024-08-19T12:56:03.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:03.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 client.?.objecter ms_handle_connect 0x7f382413f2e0 2024-08-19T12:56:03.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:03.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3829088640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.104:6789/0 conn(0x7f3824073470 legacy=0x7f382412dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3877843171 0 0) 0x7f3824142b60 con 0x7f382413f2e0 2024-08-19T12:56:03.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 monclient(hunting): my global_id is 184550 2024-08-19T12:56:03.969 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3808004eb0 con 0x7f382413f2e0 2024-08-19T12:56:03.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2630189869 0 0) 0x7f3824131040 con 0x7f38240a57a0 2024-08-19T12:56:03.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 monclient(hunting): my global_id is 136258 2024-08-19T12:56:03.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3824142b60 con 0x7f38240a57a0 2024-08-19T12:56:03.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3932533889 0 0) 0x7f382412f580 con 0x7f3824073470 2024-08-19T12:56:03.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 10 monclient(hunting): my global_id is 204130 2024-08-19T12:56:03.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.948+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3824131040 con 0x7f3824073470 2024-08-19T12:56:03.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1351596202 0 0) 0x7f3808004eb0 con 0x7f382413f2e0 2024-08-19T12:56:03.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f382412f580 con 0x7f382413f2e0 2024-08-19T12:56:03.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f381c0020b0 con 0x7f382413f2e0 2024-08-19T12:56:03.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:03.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:56:03.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient(hunting): dump: 2024-08-19T12:56:03.971 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.971 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.971 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.972 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.973 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3480421695 0 0) 0x7f3824131040 con 0x7f3824073470 2024-08-19T12:56:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3808004eb0 con 0x7f3824073470 2024-08-19T12:56:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3813925449 0 0) 0x7f3824142b60 con 0x7f38240a57a0 2024-08-19T12:56:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3824131040 con 0x7f38240a57a0 2024-08-19T12:56:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3820003610 con 0x7f3824073470 2024-08-19T12:56:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:03.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient(hunting): dump: 2024-08-19T12:56:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.974 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.975 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f38140047e0 con 0x7f38240a57a0 2024-08-19T12:56:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient(hunting): dump: 2024-08-19T12:56:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.975 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.976 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.977 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3123831724 0 0) 0x7f382412f580 con 0x7f382413f2e0 2024-08-19T12:56:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.104:6792/0 conn(0x7f38240a57a0 legacy=0x7f382412e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.104:6789/0 conn(0x7f3824073470 legacy=0x7f382412dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:56:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 monclient: found mon.h 2024-08-19T12:56:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:56:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3824131320 con 0x7f382413f2e0 2024-08-19T12:56:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient: _finish_auth 0 2024-08-19T12:56:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:56:03.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f382b2ec640 5 monclient: authenticate success, global_id 184550 2024-08-19T12:56:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f382b2ec640 10 monclient: _renew_subs 2024-08-19T12:56:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f382b2ec640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:56:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f382b2ec640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3824131d60 con 0x7f382413f2e0 2024-08-19T12:56:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f381c005300 con 0x7f382413f2e0 2024-08-19T12:56:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f381c0057f0 con 0x7f382413f2e0 2024-08-19T12:56:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f3819ffb640 10 monclient: dump: 2024-08-19T12:56:03.978 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.979 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.980 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f382b2ec640 10 client.184550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f382b2ec640 10 monclient: _renew_subs 2024-08-19T12:56:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f382b2ec640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:56:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f382b2ec640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f382412eab0 con 0x7f382413f2e0 2024-08-19T12:56:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f381b7fe640 4 set_mon_vals no callback set 2024-08-19T12:56:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.949+0000 7f382b2ec640 1 librados: init done 2024-08-19T12:56:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.950+0000 7f382b2ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.950+0000 7f382b2ec640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f38240ab7d0 con 0x7f3824142df0 2024-08-19T12:56:03.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.950+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f381c011c50 con 0x7f382413f2e0 2024-08-19T12:56:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.950+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(653..653 src has 350..653) ==== 5913+0+0 (unknown 1656795161 0 0) 0x7f381c04e470 con 0x7f382413f2e0 2024-08-19T12:56:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.950+0000 7f3819ffb640 10 client.184550.objecter ms_dispatch 0x7f3824000b90 osd_map(653..653 src has 350..653) 2024-08-19T12:56:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.950+0000 7f3819ffb640 3 client.184550.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T12:56:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.950+0000 7f3819ffb640 3 client.184550.objecter handle_osd_map decoding full epoch 653 2024-08-19T12:56:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.950+0000 7f3819ffb640 20 client.184550.objecter dump_active .. 0 homeless 2024-08-19T12:56:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.950+0000 7f3828887640 10 monclient: get_auth_request con 0x7f3808041c20 auth_method 0 2024-08-19T12:56:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3829088640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.26:6791/0 conn(0x7f3824142df0 legacy=0x7f3824134750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 10 client.184550.objecter ms_handle_connect 0x7f3824142df0 2024-08-19T12:56:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 10 client.184550.objecter resend_mon_ops 2024-08-19T12:56:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1435459321 0 0) 0x7f3808004eb0 con 0x7f3824142df0 2024-08-19T12:56:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 10 monclient(hunting): my global_id is 184657 2024-08-19T12:56:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f382412f580 con 0x7f3824142df0 2024-08-19T12:56:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2634763860 0 0) 0x7f382412f580 con 0x7f3824142df0 2024-08-19T12:56:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3808004eb0 con 0x7f3824142df0 2024-08-19T12:56:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3820003d20 con 0x7f3824142df0 2024-08-19T12:56:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.952+0000 7f3819ffb640 10 monclient: dump: 2024-08-19T12:56:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.983 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.984 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.984 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.953+0000 7f3819ffb640 10 client.184550.objecter ms_handle_connect 0x7f3808041c20 2024-08-19T12:56:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.953+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1286502411 0 0) 0x7f3808004eb0 con 0x7f3824142df0 2024-08-19T12:56:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.953+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3824131690 con 0x7f3824142df0 2024-08-19T12:56:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.953+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3820003fa0 con 0x7f3824142df0 2024-08-19T12:56:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.953+0000 7f3819ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.953+0000 7f3819ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.953+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.26:6791/0 conn(0x7f3824142df0 legacy=0x7f3824134750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.953+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.26:6791/0 conn(0x7f3824142df0 legacy=0x7f3824134750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:03.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.959+0000 7f382b2ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:03.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.959+0000 7f382b2ec640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f382412f150 con 0x7f38241300e0 2024-08-19T12:56:03.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.961+0000 7f3829088640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.26:6791/0 conn(0x7f38241300e0 legacy=0x7f3824134e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:03.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.961+0000 7f3819ffb640 10 client.184550.objecter ms_handle_connect 0x7f38241300e0 2024-08-19T12:56:03.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.961+0000 7f3819ffb640 10 client.184550.objecter resend_mon_ops 2024-08-19T12:56:03.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.961+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1369521291 0 0) 0x7f382412f150 con 0x7f38241300e0 2024-08-19T12:56:03.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.961+0000 7f3819ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:03.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.961+0000 7f3819ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:03.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.961+0000 7f3819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:03.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.961+0000 7f3819ffb640 10 monclient(hunting): my global_id is 184666 2024-08-19T12:56:03.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.961+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3808004eb0 con 0x7f38241300e0 2024-08-19T12:56:03.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4112279237 0 0) 0x7f3808004eb0 con 0x7f38241300e0 2024-08-19T12:56:03.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f382412f150 con 0x7f38241300e0 2024-08-19T12:56:03.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3820004450 con 0x7f38241300e0 2024-08-19T12:56:03.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:03.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:03.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 10 monclient: dump: 2024-08-19T12:56:03.989 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:03.989 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:03.989 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:03.990 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:03.991 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1830679782 0 0) 0x7f382412f150 con 0x7f38241300e0 2024-08-19T12:56:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f38241304c0 con 0x7f38241300e0 2024-08-19T12:56:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 2563390904) 0x7f3820005d50 con 0x7f38241300e0 2024-08-19T12:56:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.26:6791/0 conn(0x7f38241300e0 legacy=0x7f3824134e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:03.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3819ffb640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.26:6791/0 conn(0x7f38241300e0 legacy=0x7f3824134e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:03.992 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 289427, 2024-08-19T12:56:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:03.992 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:56:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:03.993 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:03.994 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:03.995 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:03.996 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:03.997 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:03.998 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:03.998 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:03.998 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:03.998 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:03.998 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:03.998 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:03.998 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:03.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.000 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:04.001 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.004 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.005 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:04.006 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:04.006 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:04.006 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.006 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.007 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.007 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:04.007 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:04.007 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.007 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.007 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:04.007 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:04.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:04.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:04.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:04.008 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.009 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.009 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:04.009 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:04.009 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:04.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:04.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:04.010 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.010 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.010 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:04.011 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:04.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:04.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:04.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:56:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.014 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:56:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.016 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:04.016 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:04.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f382b2ec640 10 librados: watch_flush enter 2024-08-19T12:56:04.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f382b2ec640 10 librados: watch_flush exit 2024-08-19T12:56:04.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f382b2ec640 20 client.184550.objecter shutdown clearing up homeless session... 2024-08-19T12:56:04.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f382b2ec640 10 client.184550.objecter successfully canceled tick 2024-08-19T12:56:04.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f382b2ec640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.104:6800/899887506 conn(0x7f3808041c20 legacy=0x7f38080440e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f382b2ec640 10 monclient: shutdown 2024-08-19T12:56:04.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f382b2ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:04.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f382b2ec640 1 -- 172.21.15.26:0/741194549 >> v1:172.21.15.104:6791/0 conn(0x7f382413f2e0 legacy=0x7f38241d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3829889640 1 -- 172.21.15.26:0/741194549 reap_dead start 2024-08-19T12:56:04.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f3829889640 1 -- 172.21.15.26:0/741194549 reap_dead start 2024-08-19T12:56:04.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f382b2ec640 1 -- 172.21.15.26:0/741194549 shutdown_connections 2024-08-19T12:56:04.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f382b2ec640 1 -- 172.21.15.26:0/741194549 >> 172.21.15.26:0/741194549 conn(0x7f38240abff0 msgr2=0x7f38240a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:04.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.962+0000 7f382b2ec640 1 -- 172.21.15.26:0/741194549 shutdown_connections 2024-08-19T12:56:04.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.963+0000 7f382b2ec640 1 -- 172.21.15.26:0/741194549 wait complete. 2024-08-19T12:56:04.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:03.963+0000 7f382b2ec640 1 librados: shutdown 2024-08-19T12:56:04.019 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:04.019 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:56:04.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:56:04.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.118+0000 7fb217986640 10 monclient: get_monmap_and_config 2024-08-19T12:56:04.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.118+0000 7fb217986640 10 monclient: build_initial_monmap 2024-08-19T12:56:04.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.118+0000 7fb217986640 10 monclient: monmap: 2024-08-19T12:56:04.120 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:04.120 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.120 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:04.119650+0000 2024-08-19T12:56:04.120 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:04.119650+0000 2024-08-19T12:56:04.120 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:04.120 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.120 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:04.120 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:04.120 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:04.120 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:04.121 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:04.121 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:04.121 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:04.121 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:04.121 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:04.121 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.119+0000 7fb217986640 1 Processor -- start 2024-08-19T12:56:04.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.119+0000 7fb217986640 1 -- start start 2024-08-19T12:56:04.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.119+0000 7fb217986640 10 monclient: init 2024-08-19T12:56:04.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.120+0000 7fb217986640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:04.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.120+0000 7fb217986640 10 monclient: _add_conns ranks=[8,3,1,4,2,0,5,7,6] 2024-08-19T12:56:04.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.120+0000 7fb217986640 10 monclient(hunting): picked mon.noname-e con 0x7fb21006e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:04.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.120+0000 7fb217986640 10 monclient(hunting): picked mon.noname-g con 0x7fb2100a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:04.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.120+0000 7fb217986640 10 monclient(hunting): picked mon.noname-f con 0x7fb2100a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:04.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.120+0000 7fb217986640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb210136790 con 0x7fb2100a5d50 2024-08-19T12:56:04.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.120+0000 7fb217986640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb21009ddf0 con 0x7fb2100a5920 2024-08-19T12:56:04.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.120+0000 7fb217986640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb21009e0d0 con 0x7fb21006e7b0 2024-08-19T12:56:04.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.120+0000 7fb217986640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:04.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.120+0000 7fb217986640 10 monclient(hunting): authenticate will time out at 1758.207143s 2024-08-19T12:56:04.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.120+0000 7fb215722640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fb21006e7b0 0x7fb210136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:57944/0 (socket says 172.21.15.26:57944) 2024-08-19T12:56:04.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.120+0000 7fb215722640 1 -- 172.21.15.26:0/379703708 learned_addr learned my addr 172.21.15.26:0/379703708 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:04.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.122+0000 7fb215722640 1 -- 172.21.15.26:0/379703708 >> v1:172.21.15.26:6793/0 conn(0x7fb21006e7b0 legacy=0x7fb210136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.122+0000 7fb214f21640 1 -- 172.21.15.26:0/379703708 >> v1:172.21.15.104:6790/0 conn(0x7fb2100a5920 legacy=0x7fb21013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.122+0000 7fb215f23640 1 -- 172.21.15.26:0/379703708 >> v1:172.21.15.104:6789/0 conn(0x7fb2100a5d50 legacy=0x7fb210141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.122+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3093957030 0 0) 0x7fb21009e0d0 con 0x7fb21006e7b0 2024-08-19T12:56:04.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.122+0000 7fb20ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.122+0000 7fb20ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.122+0000 7fb20ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.122+0000 7fb20ffff640 10 monclient(hunting): my global_id is 184508 2024-08-19T12:56:04.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.122+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1f8003520 con 0x7fb21006e7b0 2024-08-19T12:56:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3663128273 0 0) 0x7fb210136790 con 0x7fb2100a5d50 2024-08-19T12:56:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient(hunting): my global_id is 204139 2024-08-19T12:56:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb21009e0d0 con 0x7fb2100a5d50 2024-08-19T12:56:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1730761416 0 0) 0x7fb2040014a0 con 0x7fb2100a5920 2024-08-19T12:56:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient(hunting): my global_id is 194313 2024-08-19T12:56:04.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb210136790 con 0x7fb2100a5920 2024-08-19T12:56:04.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2035514153 0 0) 0x7fb1f8003520 con 0x7fb21006e7b0 2024-08-19T12:56:04.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 >> v1:172.21.15.104:6790/0 conn(0x7fb2100a5920 legacy=0x7fb21013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 >> v1:172.21.15.104:6789/0 conn(0x7fb2100a5d50 legacy=0x7fb210141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient: _finish_hunting 0 2024-08-19T12:56:04.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 1 monclient: found mon.noname-e 2024-08-19T12:56:04.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:04.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:56:04.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb21009e3b0 con 0x7fb21006e7b0 2024-08-19T12:56:04.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient: _finish_auth 0 2024-08-19T12:56:04.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient: _check_auth_tickets 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb200001e60 con 0x7fb21006e7b0 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient: dump: 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.131 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.132 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.132 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.132 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.132 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.132 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.132 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.132 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.132 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb217986640 5 monclient: authenticate success, global_id 184508 2024-08-19T12:56:04.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb217986640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:04.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb2000020b0 con 0x7fb21006e7b0 2024-08-19T12:56:04.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:04.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 1 -- 172.21.15.26:0/379703708 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb200004f60 con 0x7fb21006e7b0 2024-08-19T12:56:04.133 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb217986640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:04.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb217986640 4 set_mon_vals no callback set 2024-08-19T12:56:04.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:04.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb20ffff640 10 monclient: dump: 2024-08-19T12:56:04.134 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.134 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.134 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.134 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.134 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.135 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb217986640 10 monclient: shutdown 2024-08-19T12:56:04.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb217986640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:04.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb217986640 1 -- 172.21.15.26:0/379703708 >> v1:172.21.15.26:6793/0 conn(0x7fb21006e7b0 legacy=0x7fb210136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb217986640 1 -- 172.21.15.26:0/379703708 shutdown_connections 2024-08-19T12:56:04.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb217986640 1 -- 172.21.15.26:0/379703708 >> 172.21.15.26:0/379703708 conn(0x7fb2100a8740 msgr2=0x7fb2100a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:04.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb217986640 1 -- 172.21.15.26:0/379703708 shutdown_connections 2024-08-19T12:56:04.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.123+0000 7fb217986640 1 -- 172.21.15.26:0/379703708 wait complete. 2024-08-19T12:56:04.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.124+0000 7fb217986640 10 monclient: build_initial_monmap 2024-08-19T12:56:04.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.124+0000 7fb217986640 10 monclient: monmap: 2024-08-19T12:56:04.136 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:04.136 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:04.136 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:04.136 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:04.137 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:04.137 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.137 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:04.137 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:04.137 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:04.137 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:04.137 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:04.137 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:04.137 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:04.137 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:04.137 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:04.138 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.124+0000 7fb217986640 1 librados: starting msgr at 2024-08-19T12:56:04.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.124+0000 7fb217986640 1 librados: starting objecter 2024-08-19T12:56:04.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.124+0000 7fb217986640 1 Processor -- start 2024-08-19T12:56:04.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.124+0000 7fb217986640 1 -- start start 2024-08-19T12:56:04.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.124+0000 7fb217986640 1 librados: setting wanted keys 2024-08-19T12:56:04.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.124+0000 7fb217986640 1 librados: calling monclient init 2024-08-19T12:56:04.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.124+0000 7fb217986640 10 monclient: init 2024-08-19T12:56:04.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.125+0000 7fb217986640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:04.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.125+0000 7fb217986640 10 monclient: _add_conns ranks=[0,3,5,7,6,8,4,1,2] 2024-08-19T12:56:04.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.125+0000 7fb217986640 10 monclient(hunting): picked mon.noname-a con 0x7fb21006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:04.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.125+0000 7fb217986640 10 monclient(hunting): picked mon.noname-g con 0x7fb2100a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:04.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.125+0000 7fb217986640 10 monclient(hunting): picked mon.noname-h con 0x7fb2100a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:04.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.125+0000 7fb217986640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2100a2f60 con 0x7fb21006e7b0 2024-08-19T12:56:04.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.125+0000 7fb217986640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2100a3240 con 0x7fb2100a5920 2024-08-19T12:56:04.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.125+0000 7fb217986640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2100a3520 con 0x7fb2100a5d50 2024-08-19T12:56:04.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.125+0000 7fb217986640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:04.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.125+0000 7fb217986640 10 monclient(hunting): authenticate will time out at 1758.212126s 2024-08-19T12:56:04.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.125+0000 7fb215722640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb21006e7b0 0x7fb210132a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52584/0 (socket says 172.21.15.26:52584) 2024-08-19T12:56:04.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.125+0000 7fb215722640 1 -- 172.21.15.26:0/93446264 learned_addr learned my addr 172.21.15.26:0/93446264 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb215722640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.26:6789/0 conn(0x7fb21006e7b0 legacy=0x7fb210132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 client.?.objecter ms_handle_connect 0x7fb21006e7b0 2024-08-19T12:56:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 client.?.objecter ms_handle_connect 0x7fb2100a5920 2024-08-19T12:56:04.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb214f21640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.104:6790/0 conn(0x7fb2100a5920 legacy=0x7fb2100a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4112762253 0 0) 0x7fb2100a2f60 con 0x7fb21006e7b0 2024-08-19T12:56:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): my global_id is 194373 2024-08-19T12:56:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb215f23640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.104:6791/0 conn(0x7fb2100a5d50 legacy=0x7fb2101d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1fc004eb0 con 0x7fb21006e7b0 2024-08-19T12:56:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 client.?.objecter ms_handle_connect 0x7fb2100a5d50 2024-08-19T12:56:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:04.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2848887643 0 0) 0x7fb2100a3240 con 0x7fb2100a5920 2024-08-19T12:56:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): my global_id is 194322 2024-08-19T12:56:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2100a2f60 con 0x7fb2100a5920 2024-08-19T12:56:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3589077029 0 0) 0x7fb2100a3520 con 0x7fb2100a5d50 2024-08-19T12:56:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): my global_id is 184559 2024-08-19T12:56:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2100a3240 con 0x7fb2100a5d50 2024-08-19T12:56:04.148 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 275971414 0 0) 0x7fb1fc004eb0 con 0x7fb21006e7b0 2024-08-19T12:56:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2100a3520 con 0x7fb21006e7b0 2024-08-19T12:56:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb200001ee0 con 0x7fb21006e7b0 2024-08-19T12:56:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:56:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): dump: 2024-08-19T12:56:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.149 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.150 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.150 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.150 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.150 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.150 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.150 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.150 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.150 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.150 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.150 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.150 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1057992452 0 0) 0x7fb2100a2f60 con 0x7fb2100a5920 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1fc004eb0 con 0x7fb2100a5920 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb204003950 con 0x7fb2100a5920 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): dump: 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.151 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.152 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.152 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.152 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.152 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.152 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.152 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.152 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.152 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.152 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1517537263 0 0) 0x7fb2100a3240 con 0x7fb2100a5d50 2024-08-19T12:56:04.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2100a2f60 con 0x7fb2100a5d50 2024-08-19T12:56:04.152 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb208001c20 con 0x7fb2100a5d50 2024-08-19T12:56:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 10 monclient(hunting): dump: 2024-08-19T12:56:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.154 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.127+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 951557892 0 0) 0x7fb2100a3520 con 0x7fb21006e7b0 2024-08-19T12:56:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.104:6791/0 conn(0x7fb2100a5d50 legacy=0x7fb2101d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.104:6790/0 conn(0x7fb2100a5920 legacy=0x7fb2100a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:56:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 1 monclient: found mon.a 2024-08-19T12:56:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:56:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2100a1200 con 0x7fb21006e7b0 2024-08-19T12:56:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 10 monclient: _finish_auth 0 2024-08-19T12:56:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:56:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb217986640 5 monclient: authenticate success, global_id 194373 2024-08-19T12:56:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb217986640 10 monclient: _renew_subs 2024-08-19T12:56:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb217986640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:56:04.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb217986640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb2100a5160 con 0x7fb21006e7b0 2024-08-19T12:56:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb200005550 con 0x7fb21006e7b0 2024-08-19T12:56:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb200005a40 con 0x7fb21006e7b0 2024-08-19T12:56:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 10 monclient: dump: 2024-08-19T12:56:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.156 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.157 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.158 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb217986640 10 client.194373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb217986640 10 monclient: _renew_subs 2024-08-19T12:56:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20f7fe640 4 set_mon_vals no callback set 2024-08-19T12:56:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb217986640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:56:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb217986640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb2101d79b0 con 0x7fb21006e7b0 2024-08-19T12:56:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb217986640 1 librados: init done 2024-08-19T12:56:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb217986640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fb200011ea0 con 0x7fb21006e7b0 2024-08-19T12:56:04.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.128+0000 7fb217986640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb2100aba20 con 0x7fb210125490 2024-08-19T12:56:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.129+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(653..653 src has 350..653) ==== 5913+0+0 (unknown 1656795161 0 0) 0x7fb20004e810 con 0x7fb21006e7b0 2024-08-19T12:56:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.129+0000 7fb20dffb640 10 client.194373.objecter ms_dispatch 0x7fb210000b90 osd_map(653..653 src has 350..653) 2024-08-19T12:56:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.129+0000 7fb20dffb640 3 client.194373.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T12:56:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.129+0000 7fb20dffb640 3 client.194373.objecter handle_osd_map decoding full epoch 653 2024-08-19T12:56:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.129+0000 7fb20dffb640 20 client.194373.objecter dump_active .. 0 homeless 2024-08-19T12:56:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.129+0000 7fb215f23640 10 monclient: get_auth_request con 0x7fb1fc041980 auth_method 0 2024-08-19T12:56:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 10 client.194373.objecter ms_handle_connect 0x7fb210125490 2024-08-19T12:56:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb214f21640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.26:6792/0 conn(0x7fb210125490 legacy=0x7fb210125870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 10 client.194373.objecter resend_mon_ops 2024-08-19T12:56:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1506685597 0 0) 0x7fb1fc004eb0 con 0x7fb210125490 2024-08-19T12:56:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 10 monclient(hunting): my global_id is 194325 2024-08-19T12:56:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2100a3520 con 0x7fb210125490 2024-08-19T12:56:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2345530823 0 0) 0x7fb2100a3520 con 0x7fb210125490 2024-08-19T12:56:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1fc004eb0 con 0x7fb210125490 2024-08-19T12:56:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb204003eb0 con 0x7fb210125490 2024-08-19T12:56:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 10 monclient: dump: 2024-08-19T12:56:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.160 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.162 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 10 client.194373.objecter ms_handle_connect 0x7fb1fc041980 2024-08-19T12:56:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2620807322 0 0) 0x7fb1fc004eb0 con 0x7fb210125490 2024-08-19T12:56:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.131+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2100a2d10 con 0x7fb210125490 2024-08-19T12:56:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.132+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb204004160 con 0x7fb210125490 2024-08-19T12:56:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.132+0000 7fb20dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.132+0000 7fb20dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.132+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.26:6792/0 conn(0x7fb210125490 legacy=0x7fb210125870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.132+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.26:6792/0 conn(0x7fb210125490 legacy=0x7fb210125870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:04.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.138+0000 7fb217986640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.138+0000 7fb217986640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb210068df0 con 0x7fb210125f80 2024-08-19T12:56:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.140+0000 7fb214f21640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.26:6792/0 conn(0x7fb210125f80 legacy=0x7fb2101dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.140+0000 7fb20dffb640 10 client.194373.objecter ms_handle_connect 0x7fb210125f80 2024-08-19T12:56:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.140+0000 7fb20dffb640 10 client.194373.objecter resend_mon_ops 2024-08-19T12:56:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.140+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4245454838 0 0) 0x7fb210068df0 con 0x7fb210125f80 2024-08-19T12:56:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.140+0000 7fb20dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.140+0000 7fb20dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.140+0000 7fb20dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.140+0000 7fb20dffb640 10 monclient(hunting): my global_id is 194334 2024-08-19T12:56:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.140+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1fc004eb0 con 0x7fb210125f80 2024-08-19T12:56:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3527101654 0 0) 0x7fb1fc004eb0 con 0x7fb210125f80 2024-08-19T12:56:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb210068df0 con 0x7fb210125f80 2024-08-19T12:56:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb204005b00 con 0x7fb210125f80 2024-08-19T12:56:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 10 monclient: dump: 2024-08-19T12:56:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.164 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.165 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1392763610 0 0) 0x7fb210068df0 con 0x7fb210125f80 2024-08-19T12:56:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb210126360 con 0x7fb210125f80 2024-08-19T12:56:04.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7545 (unknown 0 0 4066228893) 0x7fb204007d90 con 0x7fb210125f80 2024-08-19T12:56:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.26:6792/0 conn(0x7fb210125f80 legacy=0x7fb2101dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb20dffb640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.26:6792/0 conn(0x7fb210125f80 legacy=0x7fb2101dbba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:04.166 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:04.166 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 149068, 2024-08-19T12:56:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:04.167 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:56:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:04.168 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:04.169 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:04.171 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:04.172 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:04.173 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:04.174 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.176 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:04.177 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.178 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.181 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.183 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.185 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.186 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.187 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.189 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:04.189 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:04.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb217986640 10 librados: watch_flush enter 2024-08-19T12:56:04.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb217986640 10 librados: watch_flush exit 2024-08-19T12:56:04.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb217986640 20 client.194373.objecter shutdown clearing up homeless session... 2024-08-19T12:56:04.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb217986640 10 client.194373.objecter successfully canceled tick 2024-08-19T12:56:04.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb217986640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.104:6800/899887506 conn(0x7fb1fc041980 legacy=0x7fb1fc043e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb217986640 10 monclient: shutdown 2024-08-19T12:56:04.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb217986640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:04.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb217986640 1 -- 172.21.15.26:0/93446264 >> v1:172.21.15.26:6789/0 conn(0x7fb21006e7b0 legacy=0x7fb210132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb215f23640 1 -- 172.21.15.26:0/93446264 reap_dead start 2024-08-19T12:56:04.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb215f23640 1 -- 172.21.15.26:0/93446264 reap_dead start 2024-08-19T12:56:04.191 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb217986640 1 -- 172.21.15.26:0/93446264 shutdown_connections 2024-08-19T12:56:04.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb217986640 1 -- 172.21.15.26:0/93446264 >> 172.21.15.26:0/93446264 conn(0x7fb2100a8740 msgr2=0x7fb2100aa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:04.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb217986640 1 -- 172.21.15.26:0/93446264 shutdown_connections 2024-08-19T12:56:04.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.141+0000 7fb217986640 1 -- 172.21.15.26:0/93446264 wait complete. 2024-08-19T12:56:04.192 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.142+0000 7fb217986640 1 librados: shutdown 2024-08-19T12:56:04.192 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:04.192 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:56:04.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:56:04.294 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.292+0000 7efd64fc9640 10 monclient: get_monmap_and_config 2024-08-19T12:56:04.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.292+0000 7efd64fc9640 10 monclient: build_initial_monmap 2024-08-19T12:56:04.295 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.292+0000 7efd64fc9640 10 monclient: monmap: 2024-08-19T12:56:04.295 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:04.295 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.295 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:04.292941+0000 2024-08-19T12:56:04.295 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:04.292941+0000 2024-08-19T12:56:04.295 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:04.295 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.295 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:04.295 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:04.295 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:04.295 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:04.296 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:04.296 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:04.296 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:04.296 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:04.296 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:04.296 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.292+0000 7efd64fc9640 1 Processor -- start 2024-08-19T12:56:04.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.292+0000 7efd64fc9640 1 -- start start 2024-08-19T12:56:04.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.292+0000 7efd64fc9640 10 monclient: init 2024-08-19T12:56:04.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.293+0000 7efd64fc9640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:04.296 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.293+0000 7efd64fc9640 10 monclient: _add_conns ranks=[8,3,1,0,5,4,6,7,2] 2024-08-19T12:56:04.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.293+0000 7efd64fc9640 10 monclient(hunting): picked mon.noname-e con 0x7efd60073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:04.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.293+0000 7efd64fc9640 10 monclient(hunting): picked mon.noname-g con 0x7efd600a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:04.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.293+0000 7efd64fc9640 10 monclient(hunting): picked mon.noname-f con 0x7efd6013f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:04.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.293+0000 7efd64fc9640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd6009dae0 con 0x7efd6013f2e0 2024-08-19T12:56:04.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.293+0000 7efd64fc9640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd60136910 con 0x7efd600a57a0 2024-08-19T12:56:04.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.293+0000 7efd64fc9640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd6009dd50 con 0x7efd60073470 2024-08-19T12:56:04.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.293+0000 7efd64fc9640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:04.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.293+0000 7efd64fc9640 10 monclient(hunting): authenticate will time out at 1758.380269s 2024-08-19T12:56:04.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.293+0000 7efd5e59c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7efd60073470 0x7efd60136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:57958/0 (socket says 172.21.15.26:57958) 2024-08-19T12:56:04.297 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.293+0000 7efd5e59c640 1 -- 172.21.15.26:0/836857579 learned_addr learned my addr 172.21.15.26:0/836857579 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:04.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5e59c640 1 -- 172.21.15.26:0/836857579 >> v1:172.21.15.26:6793/0 conn(0x7efd60073470 legacy=0x7efd60136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd55d9b640 1 -- 172.21.15.26:0/836857579 >> v1:172.21.15.104:6790/0 conn(0x7efd600a57a0 legacy=0x7efd600a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5ed9d640 1 -- 172.21.15.26:0/836857579 >> v1:172.21.15.104:6789/0 conn(0x7efd6013f2e0 legacy=0x7efd601416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713484218 0 0) 0x7efd6009dd50 con 0x7efd60073470 2024-08-19T12:56:04.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 10 monclient(hunting): my global_id is 184517 2024-08-19T12:56:04.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd4c003520 con 0x7efd60073470 2024-08-19T12:56:04.307 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714121466 0 0) 0x7efd60136910 con 0x7efd600a57a0 2024-08-19T12:56:04.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 10 monclient(hunting): my global_id is 194331 2024-08-19T12:56:04.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd6009dd50 con 0x7efd600a57a0 2024-08-19T12:56:04.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 109882509 0 0) 0x7efd6009dae0 con 0x7efd6013f2e0 2024-08-19T12:56:04.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 10 monclient(hunting): my global_id is 204148 2024-08-19T12:56:04.308 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd60136910 con 0x7efd6013f2e0 2024-08-19T12:56:04.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.295+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2730244669 0 0) 0x7efd4c003520 con 0x7efd60073470 2024-08-19T12:56:04.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 >> v1:172.21.15.104:6790/0 conn(0x7efd600a57a0 legacy=0x7efd600a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 >> v1:172.21.15.104:6789/0 conn(0x7efd6013f2e0 legacy=0x7efd601416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 10 monclient: _finish_hunting 0 2024-08-19T12:56:04.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 1 monclient: found mon.noname-e 2024-08-19T12:56:04.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:04.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:56:04.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6009e030 con 0x7efd60073470 2024-08-19T12:56:04.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 10 monclient: _finish_auth 0 2024-08-19T12:56:04.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 10 monclient: _check_auth_tickets 2024-08-19T12:56:04.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:04.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd64fc9640 5 monclient: authenticate success, global_id 184517 2024-08-19T12:56:04.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd50001ae0 con 0x7efd60073470 2024-08-19T12:56:04.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd64fc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:04.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:56:04.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 10 monclient: dump: 2024-08-19T12:56:04.311 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.311 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.311 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.311 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.311 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.311 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.311 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.311 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.312 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.312 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.312 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.312 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.312 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.312 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.312 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.312 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd64fc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:04.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd500048a0 con 0x7efd60073470 2024-08-19T12:56:04.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:04.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 1 -- 172.21.15.26:0/836857579 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd50004e10 con 0x7efd60073470 2024-08-19T12:56:04.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd64fc9640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:04.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd64fc9640 4 set_mon_vals no callback set 2024-08-19T12:56:04.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:04.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd5dd9b640 10 monclient: dump: 2024-08-19T12:56:04.313 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.313 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.313 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.313 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.315 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd64fc9640 10 monclient: shutdown 2024-08-19T12:56:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd64fc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd64fc9640 1 -- 172.21.15.26:0/836857579 >> v1:172.21.15.26:6793/0 conn(0x7efd60073470 legacy=0x7efd60136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd64fc9640 1 -- 172.21.15.26:0/836857579 shutdown_connections 2024-08-19T12:56:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd64fc9640 1 -- 172.21.15.26:0/836857579 >> 172.21.15.26:0/836857579 conn(0x7efd600abff0 msgr2=0x7efd600ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd64fc9640 1 -- 172.21.15.26:0/836857579 shutdown_connections 2024-08-19T12:56:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.296+0000 7efd64fc9640 1 -- 172.21.15.26:0/836857579 wait complete. 2024-08-19T12:56:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.297+0000 7efd64fc9640 10 monclient: build_initial_monmap 2024-08-19T12:56:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.297+0000 7efd64fc9640 10 monclient: monmap: 2024-08-19T12:56:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:04.316 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:04.317 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.297+0000 7efd64fc9640 1 librados: starting msgr at 2024-08-19T12:56:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.297+0000 7efd64fc9640 1 librados: starting objecter 2024-08-19T12:56:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.297+0000 7efd64fc9640 1 Processor -- start 2024-08-19T12:56:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.297+0000 7efd64fc9640 1 -- start start 2024-08-19T12:56:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.297+0000 7efd64fc9640 1 librados: setting wanted keys 2024-08-19T12:56:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.297+0000 7efd64fc9640 1 librados: calling monclient init 2024-08-19T12:56:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.297+0000 7efd64fc9640 10 monclient: init 2024-08-19T12:56:04.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd64fc9640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:04.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd64fc9640 10 monclient: _add_conns ranks=[1,3,5,7,0,2,8,6,4] 2024-08-19T12:56:04.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd64fc9640 10 monclient(hunting): picked mon.noname-f con 0x7efd60073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:04.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd64fc9640 10 monclient(hunting): picked mon.noname-g con 0x7efd600a57a0 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:04.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd64fc9640 10 monclient(hunting): picked mon.noname-h con 0x7efd6013f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:04.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd64fc9640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd600a2bf0 con 0x7efd60073470 2024-08-19T12:56:04.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd64fc9640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd600a1d80 con 0x7efd600a57a0 2024-08-19T12:56:04.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd64fc9640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd6009fc50 con 0x7efd6013f2e0 2024-08-19T12:56:04.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd64fc9640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:04.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd64fc9640 10 monclient(hunting): authenticate will time out at 1758.385421s 2024-08-19T12:56:04.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd55d9b640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7efd600a57a0 0x7efd6009f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.26:39954/0 (socket says 172.21.15.26:39954) 2024-08-19T12:56:04.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd5e59c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7efd60073470 0x7efd6009ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:57270/0 (socket says 172.21.15.26:57270) 2024-08-19T12:56:04.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd5ed9d640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7efd6013f2e0 0x7efd601d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:35280/0 (socket says 172.21.15.26:35280) 2024-08-19T12:56:04.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.298+0000 7efd55d9b640 1 -- 172.21.15.26:0/3881426312 learned_addr learned my addr 172.21.15.26:0/3881426312 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:04.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.300+0000 7efd5ed9d640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.104:6791/0 conn(0x7efd6013f2e0 legacy=0x7efd601d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.300+0000 7efd577fe640 10 client.?.objecter ms_handle_connect 0x7efd6013f2e0 2024-08-19T12:56:04.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.300+0000 7efd5e59c640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.104:6789/0 conn(0x7efd60073470 legacy=0x7efd6009ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.300+0000 7efd577fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:04.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.300+0000 7efd577fe640 10 client.?.objecter ms_handle_connect 0x7efd60073470 2024-08-19T12:56:04.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.300+0000 7efd577fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:04.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.300+0000 7efd577fe640 10 client.?.objecter ms_handle_connect 0x7efd600a57a0 2024-08-19T12:56:04.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.300+0000 7efd577fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:04.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd55d9b640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.104:6790/0 conn(0x7efd600a57a0 legacy=0x7efd6009f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4038731638 0 0) 0x7efd6009fc50 con 0x7efd6013f2e0 2024-08-19T12:56:04.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): my global_id is 184568 2024-08-19T12:56:04.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd40004eb0 con 0x7efd6013f2e0 2024-08-19T12:56:04.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4203975459 0 0) 0x7efd600a2bf0 con 0x7efd60073470 2024-08-19T12:56:04.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): my global_id is 204157 2024-08-19T12:56:04.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd6009fc50 con 0x7efd60073470 2024-08-19T12:56:04.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1259552964 0 0) 0x7efd600a1d80 con 0x7efd600a57a0 2024-08-19T12:56:04.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): my global_id is 194340 2024-08-19T12:56:04.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd600a2bf0 con 0x7efd600a57a0 2024-08-19T12:56:04.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3186992218 0 0) 0x7efd40004eb0 con 0x7efd6013f2e0 2024-08-19T12:56:04.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd600a1d80 con 0x7efd6013f2e0 2024-08-19T12:56:04.321 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd580020b0 con 0x7efd6013f2e0 2024-08-19T12:56:04.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:04.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:56:04.322 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): dump: 2024-08-19T12:56:04.322 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.322 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.322 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.322 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.322 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.322 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.322 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.322 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.323 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.323 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.323 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.323 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.323 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.323 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.323 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.323 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:04.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3016045563 0 0) 0x7efd6009fc50 con 0x7efd60073470 2024-08-19T12:56:04.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd40004eb0 con 0x7efd60073470 2024-08-19T12:56:04.323 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd50004d90 con 0x7efd60073470 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): dump: 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.325 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 470819296 0 0) 0x7efd600a2bf0 con 0x7efd600a57a0 2024-08-19T12:56:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd6009fc50 con 0x7efd600a57a0 2024-08-19T12:56:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd48003610 con 0x7efd600a57a0 2024-08-19T12:56:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:56:04.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient(hunting): dump: 2024-08-19T12:56:04.326 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.326 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.326 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.326 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.326 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.326 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.326 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.326 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.326 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.326 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.327 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 670960088 0 0) 0x7efd600a1d80 con 0x7efd6013f2e0 2024-08-19T12:56:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.104:6790/0 conn(0x7efd600a57a0 legacy=0x7efd6009f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.104:6789/0 conn(0x7efd60073470 legacy=0x7efd6009ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient: _finish_hunting 0 2024-08-19T12:56:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 monclient: found mon.h 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd600a5460 con 0x7efd6013f2e0 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient: _finish_auth 0 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient: _check_auth_tickets 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd64fc9640 5 monclient: authenticate success, global_id 184568 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd64fc9640 10 monclient: _renew_subs 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd64fc9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd64fc9640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efd600a1660 con 0x7efd6013f2e0 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd64fc9640 10 client.184568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd64fc9640 10 monclient: _renew_subs 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd64fc9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:56:04.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd64fc9640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7efd601d7a50 con 0x7efd6013f2e0 2024-08-19T12:56:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd64fc9640 1 librados: init done 2024-08-19T12:56:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd58003dc0 con 0x7efd6013f2e0 2024-08-19T12:56:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd580042f0 con 0x7efd6013f2e0 2024-08-19T12:56:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd5d59a640 4 set_mon_vals no callback set 2024-08-19T12:56:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.301+0000 7efd577fe640 10 monclient: dump: 2024-08-19T12:56:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.329 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.330 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.302+0000 7efd64fc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.302+0000 7efd64fc9640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efd440065c0 con 0x7efd44002860 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.302+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7efd580045c0 con 0x7efd6013f2e0 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.303+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(653..653 src has 350..653) ==== 5913+0+0 (unknown 1656795161 0 0) 0x7efd5804e600 con 0x7efd6013f2e0 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.303+0000 7efd577fe640 10 client.184568.objecter ms_dispatch 0x7efd60000b90 osd_map(653..653 src has 350..653) 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.303+0000 7efd577fe640 3 client.184568.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.303+0000 7efd577fe640 3 client.184568.objecter handle_osd_map decoding full epoch 653 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.303+0000 7efd577fe640 20 client.184568.objecter dump_active .. 0 homeless 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.303+0000 7efd55d9b640 10 monclient: get_auth_request con 0x7efd40041bb0 auth_method 0 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.304+0000 7efd5e59c640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.26:6793/0 conn(0x7efd44002860 legacy=0x7efd44004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.304+0000 7efd577fe640 10 client.184568.objecter ms_handle_connect 0x7efd44002860 2024-08-19T12:56:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.304+0000 7efd577fe640 10 client.184568.objecter resend_mon_ops 2024-08-19T12:56:04.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.304+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 854698823 0 0) 0x7efd440065c0 con 0x7efd44002860 2024-08-19T12:56:04.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.304+0000 7efd577fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.304+0000 7efd577fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.304+0000 7efd577fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.304+0000 7efd577fe640 10 monclient(hunting): my global_id is 184526 2024-08-19T12:56:04.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.304+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd600a1d80 con 0x7efd44002860 2024-08-19T12:56:04.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2381866628 0 0) 0x7efd600a1d80 con 0x7efd44002860 2024-08-19T12:56:04.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd440065c0 con 0x7efd44002860 2024-08-19T12:56:04.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd50001580 con 0x7efd44002860 2024-08-19T12:56:04.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 10 monclient: dump: 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.333 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.334 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.334 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.334 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.334 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1229168142 0 0) 0x7efd440065c0 con 0x7efd44002860 2024-08-19T12:56:04.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd4400a160 con 0x7efd44002860 2024-08-19T12:56:04.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 10 client.184568.objecter ms_handle_connect 0x7efd40041bb0 2024-08-19T12:56:04.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7efd500048a0 con 0x7efd44002860 2024-08-19T12:56:04.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:04.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:04.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.26:6793/0 conn(0x7efd44002860 legacy=0x7efd44004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.305+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.26:6793/0 conn(0x7efd44002860 legacy=0x7efd44004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:04.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.311+0000 7efd64fc9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:04.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.311+0000 7efd64fc9640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efd44009e70 con 0x7efd44009130 2024-08-19T12:56:04.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd5e59c640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.26:6793/0 conn(0x7efd44009130 legacy=0x7efd440095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 10 client.184568.objecter ms_handle_connect 0x7efd44009130 2024-08-19T12:56:04.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 10 client.184568.objecter resend_mon_ops 2024-08-19T12:56:04.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1163106190 0 0) 0x7efd44009e70 con 0x7efd44009130 2024-08-19T12:56:04.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 10 monclient(hunting): my global_id is 184535 2024-08-19T12:56:04.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd440065c0 con 0x7efd44009130 2024-08-19T12:56:04.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2409221771 0 0) 0x7efd440065c0 con 0x7efd44009130 2024-08-19T12:56:04.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd44009e70 con 0x7efd44009130 2024-08-19T12:56:04.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7efd500059c0 con 0x7efd44009130 2024-08-19T12:56:04.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:04.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 10 monclient: dump: 2024-08-19T12:56:04.338 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.338 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.338 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.339 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.339 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.339 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.339 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.339 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.339 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.339 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.339 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.339 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.339 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.339 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.339 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.340 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 421949683 0 0) 0x7efd44009e70 con 0x7efd44009130 2024-08-19T12:56:04.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efd4400c3f0 con 0x7efd44009130 2024-08-19T12:56:04.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 2878459001) 0x7efd50005d50 con 0x7efd44009130 2024-08-19T12:56:04.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:04.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:04.340 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:04.340 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:04.341 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:04.341 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:04.341 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:04.341 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 289780, 2024-08-19T12:56:04.341 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:04.341 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:04.341 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:04.341 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:04.341 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:04.341 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:04.341 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:04.341 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:04.342 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:04.342 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:04.342 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.342 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:56:04.342 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:04.342 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:04.342 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:04.342 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:04.342 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:04.342 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:04.343 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:04.343 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:04.343 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:04.343 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:04.343 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:04.343 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:04.343 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:04.343 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:04.343 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.343 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:04.343 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:04.344 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:04.344 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:04.344 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:04.344 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:04.344 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:04.344 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:04.344 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:04.344 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:04.344 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:04.344 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:04.345 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:04.345 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.345 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:04.345 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:04.345 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:04.345 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:04.345 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:04.345 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:04.346 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:04.346 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:04.346 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:04.346 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:04.346 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:04.346 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:04.346 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:04.346 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:04.346 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:04.346 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:04.347 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:04.347 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:04.347 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:04.347 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:04.347 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:04.347 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:04.347 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:04.347 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:04.347 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:04.347 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:04.347 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:04.348 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:04.348 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.348 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:04.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.348 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:04.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.348 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:04.348 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:04.348 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:04.349 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.349 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:04.349 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:04.349 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.349 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.349 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:04.351 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:04.352 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:04.353 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.354 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.354 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:04.354 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:04.354 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.354 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.354 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.355 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:04.355 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:04.355 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:04.355 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.355 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.355 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:04.356 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:04.356 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.356 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.356 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.356 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:04.356 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:04.356 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.356 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.357 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:04.357 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:04.357 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:04.357 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.357 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.357 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.358 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:04.358 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:04.358 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:04.358 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:04.359 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:04.359 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.359 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.359 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.359 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:04.359 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:04.360 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.360 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:04.360 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.360 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:04.360 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:04.360 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.361 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.361 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.361 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:04.361 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:04.361 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.361 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:04.361 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.362 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.362 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:04.362 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:04.362 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.362 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.362 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.362 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.362 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.362 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:04.363 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:04.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.363 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:04.363 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:04.363 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:04.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.363 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.363 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:56:04.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.363 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:04.363 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:04.364 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:56:04.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.364 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.364 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:04.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.364 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:04.364 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:04.364 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:04.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.364 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.364 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:04.365 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:04.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.26:6793/0 conn(0x7efd44009130 legacy=0x7efd440095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.314+0000 7efd577fe640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.26:6793/0 conn(0x7efd44009130 legacy=0x7efd440095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:04.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 10 librados: watch_flush enter 2024-08-19T12:56:04.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 10 librados: watch_flush exit 2024-08-19T12:56:04.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 20 client.184568.objecter shutdown clearing up homeless session... 2024-08-19T12:56:04.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 10 client.184568.objecter successfully canceled tick 2024-08-19T12:56:04.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.104:6800/899887506 conn(0x7efd40041bb0 legacy=0x7efd40044070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 10 monclient: shutdown 2024-08-19T12:56:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 1 -- 172.21.15.26:0/3881426312 >> v1:172.21.15.104:6791/0 conn(0x7efd6013f2e0 legacy=0x7efd601d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd5ed9d640 1 -- 172.21.15.26:0/3881426312 reap_dead start 2024-08-19T12:56:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd5ed9d640 1 -- 172.21.15.26:0/3881426312 reap_dead start 2024-08-19T12:56:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 1 -- 172.21.15.26:0/3881426312 shutdown_connections 2024-08-19T12:56:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 1 -- 172.21.15.26:0/3881426312 >> 172.21.15.26:0/3881426312 conn(0x7efd600abff0 msgr2=0x7efd60135280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 1 -- 172.21.15.26:0/3881426312 shutdown_connections 2024-08-19T12:56:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 1 -- 172.21.15.26:0/3881426312 wait complete. 2024-08-19T12:56:04.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.315+0000 7efd64fc9640 1 librados: shutdown 2024-08-19T12:56:04.369 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:04.369 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:56:04.369 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:56:04.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.470+0000 7f65af9f4640 10 monclient: get_monmap_and_config 2024-08-19T12:56:04.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.470+0000 7f65af9f4640 10 monclient: build_initial_monmap 2024-08-19T12:56:04.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.470+0000 7f65af9f4640 10 monclient: monmap: 2024-08-19T12:56:04.471 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:04.471 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.471 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:04.471481+0000 2024-08-19T12:56:04.471 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:04.471481+0000 2024-08-19T12:56:04.472 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:04.472 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.472 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:04.472 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:04.472 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:04.472 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:04.472 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:04.472 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:04.472 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:04.472 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:04.472 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:04.472 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 1 Processor -- start 2024-08-19T12:56:04.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 1 -- start start 2024-08-19T12:56:04.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 10 monclient: init 2024-08-19T12:56:04.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:04.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 10 monclient: _add_conns ranks=[3,6,5,2,8,1,7,0,4] 2024-08-19T12:56:04.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 10 monclient(hunting): picked mon.noname-g con 0x7f65a806e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:04.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 10 monclient(hunting): picked mon.noname-d con 0x7f65a80a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:04.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 10 monclient(hunting): picked mon.noname-h con 0x7f65a80a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:04.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a8136790 con 0x7f65a806e7b0 2024-08-19T12:56:04.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a809ddf0 con 0x7f65a80a5d50 2024-08-19T12:56:04.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a809e0d0 con 0x7f65a80a5920 2024-08-19T12:56:04.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:04.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65af9f4640 10 monclient(hunting): authenticate will time out at 1758.558898s 2024-08-19T12:56:04.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.471+0000 7f65acf8f640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f65a80a5920 0x7f65a813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34080/0 (socket says 172.21.15.26:34080) 2024-08-19T12:56:04.474 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.472+0000 7f65acf8f640 1 -- 172.21.15.26:0/11864662 learned_addr learned my addr 172.21.15.26:0/11864662 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:04.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f65acf8f640 1 -- 172.21.15.26:0/11864662 >> v1:172.21.15.26:6792/0 conn(0x7f65a80a5920 legacy=0x7f65a813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f65acf8f640 1 -- 172.21.15.26:0/11864662 >> v1:172.21.15.26:6792/0 conn(0x7f65a80a5920 legacy=0x7f65a813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-08-19T12:56:04.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f65acf8f640 1 -- 172.21.15.26:0/11864662 >> v1:172.21.15.26:6792/0 conn(0x7f65a80a5920 legacy=0x7f65a813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:56:04.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f65acf8f640 1 --1- 172.21.15.26:0/11864662 >> v1:172.21.15.26:6792/0 conn(0x7f65a80a5920 0x7f65a813e2f0 :-1 s=OPENED pgs=3578 cs=1 l=1).handle_message read tag failed 2024-08-19T12:56:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f65acf8f640 1 --1- 172.21.15.26:0/11864662 >> v1:172.21.15.26:6792/0 conn(0x7f65a80a5920 0x7f65a813e2f0 :-1 s=OPENED pgs=3578 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:56:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6792/0 2024-08-19T12:56:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f65ad790640 1 -- 172.21.15.26:0/11864662 >> v1:172.21.15.104:6790/0 conn(0x7f65a806e7b0 legacy=0x7f65a8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f65adf91640 1 -- 172.21.15.26:0/11864662 >> v1:172.21.15.104:6791/0 conn(0x7f65a80a5d50 legacy=0x7f65a8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 1 -- 172.21.15.26:0/11864662 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1187874556 0 0) 0x7f65a8136790 con 0x7f65a806e7b0 2024-08-19T12:56:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient(hunting): my global_id is 194349 2024-08-19T12:56:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 1 -- 172.21.15.26:0/11864662 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6594003520 con 0x7f65a806e7b0 2024-08-19T12:56:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 1 -- 172.21.15.26:0/11864662 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3252361726 0 0) 0x7f65a0001820 con 0x7f65a80a5d50 2024-08-19T12:56:04.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient(hunting): my global_id is 184577 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 1 -- 172.21.15.26:0/11864662 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65a8136790 con 0x7f65a80a5d50 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 1 -- 172.21.15.26:0/11864662 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3803627993 0 0) 0x7f6594003520 con 0x7f65a806e7b0 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 1 -- 172.21.15.26:0/11864662 >> v1:172.21.15.26:6792/0 conn(0x7f65a80a5920 legacy=0x7f65a813e2f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 1 -- 172.21.15.26:0/11864662 >> v1:172.21.15.104:6791/0 conn(0x7f65a80a5d50 legacy=0x7f65a8141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient: _finish_hunting 0 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 1 monclient: found mon.noname-g 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.104:6790/0 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 1 -- 172.21.15.26:0/11864662 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a809e3b0 con 0x7f65a806e7b0 2024-08-19T12:56:04.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient: _finish_auth 0 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient: _check_auth_tickets 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f65af9f4640 5 monclient: authenticate success, global_id 194349 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 1 -- 172.21.15.26:0/11864662 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f65a4001b30 con 0x7f65a806e7b0 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f659ffff640 10 monclient: dump: 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.474+0000 7f65af9f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f659ffff640 1 -- 172.21.15.26:0/11864662 <== mon.3 v1:172.21.15.104:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65a4003610 con 0x7f65a806e7b0 2024-08-19T12:56:04.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f659ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:04.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f659ffff640 1 -- 172.21.15.26:0/11864662 <== mon.3 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f65a4003b80 con 0x7f65a806e7b0 2024-08-19T12:56:04.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f65af9f4640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:04.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f65af9f4640 4 set_mon_vals no callback set 2024-08-19T12:56:04.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f659ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f659ffff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:56:04.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f659ffff640 10 monclient: dump: 2024-08-19T12:56:04.483 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.483 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.483 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.483 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f65af9f4640 10 monclient: shutdown 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f65af9f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f65af9f4640 1 -- 172.21.15.26:0/11864662 >> v1:172.21.15.104:6790/0 conn(0x7f65a806e7b0 legacy=0x7f65a8136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f65af9f4640 1 -- 172.21.15.26:0/11864662 shutdown_connections 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f65af9f4640 1 -- 172.21.15.26:0/11864662 >> 172.21.15.26:0/11864662 conn(0x7f65a80a8740 msgr2=0x7f65a80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f65af9f4640 1 -- 172.21.15.26:0/11864662 shutdown_connections 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.475+0000 7f65af9f4640 1 -- 172.21.15.26:0/11864662 wait complete. 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 10 monclient: build_initial_monmap 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 10 monclient: monmap: 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:04.485 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.486 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:04.486 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:04.486 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:04.486 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:04.486 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:04.486 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 1 librados: starting msgr at 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 1 librados: starting objecter 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 1 Processor -- start 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 1 -- start start 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 1 librados: setting wanted keys 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 1 librados: calling monclient init 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 10 monclient: init 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:04.487 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 10 monclient: _add_conns ranks=[6,0,7,1,5,4,2,3,8] 2024-08-19T12:56:04.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 10 monclient(hunting): picked mon.noname-d con 0x7f65a806e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:04.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 10 monclient(hunting): picked mon.noname-a con 0x7f65a80a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:04.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 10 monclient(hunting): picked mon.noname-i con 0x7f65a80a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:04.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a813b150 con 0x7f65a80a5920 2024-08-19T12:56:04.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a8138120 con 0x7f65a806e7b0 2024-08-19T12:56:04.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a813ae70 con 0x7f65a80a5d50 2024-08-19T12:56:04.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:04.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.476+0000 7f65af9f4640 10 monclient(hunting): authenticate will time out at 1758.563961s 2024-08-19T12:56:04.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.477+0000 7f65ad790640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f65a806e7b0 0x7f65a8137300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34088/0 (socket says 172.21.15.26:34088) 2024-08-19T12:56:04.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.477+0000 7f65ad790640 1 -- 172.21.15.26:0/423466484 learned_addr learned my addr 172.21.15.26:0/423466484 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:04.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f65ad790640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.26:6792/0 conn(0x7f65a806e7b0 legacy=0x7f65a8137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f65acf8f640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.26:6789/0 conn(0x7f65a80a5920 legacy=0x7f65a8137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 client.?.objecter ms_handle_connect 0x7f65a806e7b0 2024-08-19T12:56:04.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:04.494 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 client.?.objecter ms_handle_connect 0x7f65a80a5920 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2866759171 0 0) 0x7f65a8138120 con 0x7f65a806e7b0 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): my global_id is 194343 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f658c004eb0 con 0x7f65a806e7b0 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3273626165 0 0) 0x7f65a813b150 con 0x7f65a80a5920 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): my global_id is 194382 2024-08-19T12:56:04.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65a8138120 con 0x7f65a80a5920 2024-08-19T12:56:04.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 client.?.objecter ms_handle_connect 0x7f65a80a5d50 2024-08-19T12:56:04.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f65adf91640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.104:6792/0 conn(0x7f65a80a5d50 legacy=0x7f65a81d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:04.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2103059520 0 0) 0x7f658c004eb0 con 0x7f65a806e7b0 2024-08-19T12:56:04.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65a813b150 con 0x7f65a806e7b0 2024-08-19T12:56:04.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f65a4003b00 con 0x7f65a806e7b0 2024-08-19T12:56:04.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:04.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): dump: 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.497 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2491702249 0 0) 0x7f65a8138120 con 0x7f65a80a5920 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f658c004eb0 con 0x7f65a80a5920 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6598001cc0 con 0x7f65a80a5920 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): dump: 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.498 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.499 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3012752384 0 0) 0x7f65a813ae70 con 0x7f65a80a5d50 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient(hunting): my global_id is 136267 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65a8138120 con 0x7f65a80a5d50 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3550538537 0 0) 0x7f65a813b150 con 0x7f65a806e7b0 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.104:6792/0 conn(0x7f65a80a5d50 legacy=0x7f65a81d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.26:6789/0 conn(0x7f65a80a5920 legacy=0x7f65a8137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient: _finish_hunting 0 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 monclient: found mon.d 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:04.500 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a8139bc0 con 0x7f65a806e7b0 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient: _finish_auth 0 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 10 monclient: _check_auth_tickets 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f659dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f65af9f4640 5 monclient: authenticate success, global_id 194343 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f65af9f4640 10 monclient: _renew_subs 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f65af9f4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.479+0000 7f65af9f4640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65a81428c0 con 0x7f65a806e7b0 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65a4001b50 con 0x7f65a806e7b0 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f659dffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f65a40057c0 con 0x7f65a806e7b0 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f659dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f659dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:04.501 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f659dffb640 10 monclient: dump: 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.502 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f659f7fe640 4 set_mon_vals no callback set 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f65af9f4640 10 client.194343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f65af9f4640 10 monclient: _renew_subs 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f65af9f4640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f65af9f4640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f65a81d7b90 con 0x7f65a806e7b0 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f65af9f4640 1 librados: init done 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f65af9f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f65af9f4640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f65a81345d0 con 0x7f65a80a18d0 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.480+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f65a4011c20 con 0x7f65a806e7b0 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.481+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(653..653 src has 350..653) ==== 5913+0+0 (unknown 1656795161 0 0) 0x7f65a404e340 con 0x7f65a806e7b0 2024-08-19T12:56:04.503 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.481+0000 7f659dffb640 10 client.194343.objecter ms_dispatch 0x7f65a8000b90 osd_map(653..653 src has 350..653) 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.481+0000 7f659dffb640 3 client.194343.objecter handle_osd_map got epochs [653,653] > 0 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.481+0000 7f659dffb640 3 client.194343.objecter handle_osd_map decoding full epoch 653 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.481+0000 7f659dffb640 20 client.194343.objecter dump_active .. 0 homeless 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.481+0000 7f65adf91640 10 monclient: get_auth_request con 0x7f658c041640 auth_method 0 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f65acf8f640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.104:6789/0 conn(0x7f65a80a18d0 legacy=0x7f65a80a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 10 client.194343.objecter ms_handle_connect 0x7f65a80a18d0 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 10 client.194343.objecter resend_mon_ops 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3381787365 0 0) 0x7f65a81345d0 con 0x7f65a80a18d0 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 10 monclient(hunting): my global_id is 204166 2024-08-19T12:56:04.504 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65a813b150 con 0x7f65a80a18d0 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1665412245 0 0) 0x7f65a813b150 con 0x7f65a80a18d0 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65a81345d0 con 0x7f65a80a18d0 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6598005190 con 0x7f65a80a18d0 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 10 monclient: dump: 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.505 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.506 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 10 client.194343.objecter ms_handle_connect 0x7f658c041640 2024-08-19T12:56:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3646831085 0 0) 0x7f65a81345d0 con 0x7f65a80a18d0 2024-08-19T12:56:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.483+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f65a80008d0 con 0x7f65a80a18d0 2024-08-19T12:56:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.484+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6598005520 con 0x7f65a80a18d0 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.484+0000 7f659dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.484+0000 7f659dffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.484+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.104:6789/0 conn(0x7f65a80a18d0 legacy=0x7f65a80a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.484+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.104:6789/0 conn(0x7f65a80a18d0 legacy=0x7f65a80a1cb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.490+0000 7f65af9f4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.490+0000 7f65af9f4640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f65a80a27a0 con 0x7f65a80a23c0 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.492+0000 7f65acf8f640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.104:6789/0 conn(0x7f65a80a23c0 legacy=0x7f65a81dbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.492+0000 7f659dffb640 10 client.194343.objecter ms_handle_connect 0x7f65a80a23c0 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.492+0000 7f659dffb640 10 client.194343.objecter resend_mon_ops 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2289935707 0 0) 0x7f65a80a27a0 con 0x7f65a80a23c0 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 10 monclient(hunting): my global_id is 204175 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65a81345d0 con 0x7f65a80a23c0 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3034868844 0 0) 0x7f65a81345d0 con 0x7f65a80a23c0 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65a80a27a0 con 0x7f65a80a23c0 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6598005e60 con 0x7f65a80a23c0 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 10 monclient: dump: 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.508 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2749373224 0 0) 0x7f65a80a27a0 con 0x7f65a80a23c0 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f65a81430c0 con 0x7f65a80a23c0 2024-08-19T12:56:04.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 2290489257) 0x7f6598008160 con 0x7f65a80a23c0 2024-08-19T12:56:04.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:04.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:04.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.104:6789/0 conn(0x7f65a80a23c0 legacy=0x7f65a81dbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.493+0000 7f659dffb640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.104:6789/0 conn(0x7f65a80a23c0 legacy=0x7f65a81dbd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:04.510 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:04.510 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:04.510 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:04.510 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 16967, 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:04.511 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.512 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:56:04.512 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:04.512 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:04.512 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:04.512 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:04.512 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:04.512 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:04.512 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:04.512 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:04.512 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:04.512 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:04.512 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:04.513 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:04.514 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:04.514 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:04.514 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:04.514 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:04.514 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.514 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.514 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:04.514 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:04.514 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:04.514 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:04.514 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:04.515 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.516 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.517 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.518 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:04.519 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.520 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.522 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.523 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.524 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.525 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.526 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:04.527 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.528 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.529 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:04.530 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:04.530 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:04.530 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.530 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.530 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:04.530 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:04.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 10 librados: watch_flush enter 2024-08-19T12:56:04.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 10 librados: watch_flush exit 2024-08-19T12:56:04.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 20 client.194343.objecter shutdown clearing up homeless session... 2024-08-19T12:56:04.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 10 client.194343.objecter successfully canceled tick 2024-08-19T12:56:04.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.104:6800/899887506 conn(0x7f658c041640 legacy=0x7f658c043b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.532 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 10 monclient: shutdown 2024-08-19T12:56:04.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:04.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 1 -- 172.21.15.26:0/423466484 >> v1:172.21.15.26:6792/0 conn(0x7f65a806e7b0 legacy=0x7f65a8137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65adf91640 1 -- 172.21.15.26:0/423466484 reap_dead start 2024-08-19T12:56:04.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65adf91640 1 -- 172.21.15.26:0/423466484 reap_dead start 2024-08-19T12:56:04.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 1 -- 172.21.15.26:0/423466484 shutdown_connections 2024-08-19T12:56:04.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 1 -- 172.21.15.26:0/423466484 >> 172.21.15.26:0/423466484 conn(0x7f65a80a8740 msgr2=0x7f65a80aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:04.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 1 -- 172.21.15.26:0/423466484 shutdown_connections 2024-08-19T12:56:04.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 1 -- 172.21.15.26:0/423466484 wait complete. 2024-08-19T12:56:04.533 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.494+0000 7f65af9f4640 1 librados: shutdown 2024-08-19T12:56:04.533 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:04.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:56:04.534 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:56:04.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.633+0000 7f06f8f5a640 10 monclient: get_monmap_and_config 2024-08-19T12:56:04.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.633+0000 7f06f8f5a640 10 monclient: build_initial_monmap 2024-08-19T12:56:04.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.633+0000 7f06f8f5a640 10 monclient: monmap: 2024-08-19T12:56:04.635 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:04.635 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.635 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:04.634690+0000 2024-08-19T12:56:04.636 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:04.634690+0000 2024-08-19T12:56:04.636 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:04.636 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.636 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:04.636 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:04.636 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:04.637 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:04.637 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:04.637 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:04.637 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:04.637 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:04.637 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:04.638 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.634+0000 7f06f8f5a640 1 Processor -- start 2024-08-19T12:56:04.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.634+0000 7f06f8f5a640 1 -- start start 2024-08-19T12:56:04.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.634+0000 7f06f8f5a640 10 monclient: init 2024-08-19T12:56:04.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.634+0000 7f06f8f5a640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:04.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.635+0000 7f06f8f5a640 10 monclient: _add_conns ranks=[4,6,7,1,0,8,2,5,3] 2024-08-19T12:56:04.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.635+0000 7f06f8f5a640 10 monclient(hunting): picked mon.noname-c con 0x7f06f406e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:04.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.635+0000 7f06f8f5a640 10 monclient(hunting): picked mon.noname-d con 0x7f06f40a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:04.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.635+0000 7f06f8f5a640 10 monclient(hunting): picked mon.noname-i con 0x7f06f40a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:04.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.635+0000 7f06f8f5a640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06f4136790 con 0x7f06f406e7b0 2024-08-19T12:56:04.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.635+0000 7f06f8f5a640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06f409ddf0 con 0x7f06f40a5920 2024-08-19T12:56:04.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.635+0000 7f06f8f5a640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06f409e0d0 con 0x7f06f40a5d50 2024-08-19T12:56:04.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.635+0000 7f06f8f5a640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:04.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.635+0000 7f06f8f5a640 10 monclient(hunting): authenticate will time out at 1758.722107s 2024-08-19T12:56:04.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.635+0000 7f06f259c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f06f406e7b0 0x7f06f4136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59970/0 (socket says 172.21.15.26:59970) 2024-08-19T12:56:04.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.635+0000 7f06ebfff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f06f40a5920 0x7f06f413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34102/0 (socket says 172.21.15.26:34102) 2024-08-19T12:56:04.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.635+0000 7f06f259c640 1 -- 172.21.15.26:0/1144155217 learned_addr learned my addr 172.21.15.26:0/1144155217 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:04.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.637+0000 7f06ebfff640 1 -- 172.21.15.26:0/1144155217 >> v1:172.21.15.26:6792/0 conn(0x7f06f40a5920 legacy=0x7f06f413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.637+0000 7f06f259c640 1 -- 172.21.15.26:0/1144155217 >> v1:172.21.15.26:6791/0 conn(0x7f06f406e7b0 legacy=0x7f06f4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.637+0000 7f06f2d9d640 1 -- 172.21.15.26:0/1144155217 >> v1:172.21.15.104:6792/0 conn(0x7f06f40a5d50 legacy=0x7f06f4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.907+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1660414670 0 0) 0x7f06dc001450 con 0x7f06f40a5920 2024-08-19T12:56:04.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.907+0000 7f06f1d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.907+0000 7f06f1d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.907+0000 7f06f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.907+0000 7f06f1d9b640 10 monclient(hunting): my global_id is 194352 2024-08-19T12:56:04.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.907+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06e0003520 con 0x7f06f40a5920 2024-08-19T12:56:04.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.907+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1956018809 0 0) 0x7f06f4136790 con 0x7f06f406e7b0 2024-08-19T12:56:04.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.907+0000 7f06f1d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.907+0000 7f06f1d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.907+0000 7f06f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.907+0000 7f06f1d9b640 10 monclient(hunting): my global_id is 184675 2024-08-19T12:56:04.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.907+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06dc001450 con 0x7f06f406e7b0 2024-08-19T12:56:04.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.910+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4254198527 0 0) 0x7f06f409e0d0 con 0x7f06f40a5d50 2024-08-19T12:56:04.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.910+0000 7f06f1d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.910+0000 7f06f1d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.910+0000 7f06f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.910+0000 7f06f1d9b640 10 monclient(hunting): my global_id is 136276 2024-08-19T12:56:04.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.910+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06f4136790 con 0x7f06f40a5d50 2024-08-19T12:56:04.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2337890843 0 0) 0x7f06dc001450 con 0x7f06f406e7b0 2024-08-19T12:56:04.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 >> v1:172.21.15.104:6792/0 conn(0x7f06f40a5d50 legacy=0x7f06f4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 >> v1:172.21.15.26:6792/0 conn(0x7f06f40a5920 legacy=0x7f06f413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:56:04.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 1 monclient: found mon.noname-c 2024-08-19T12:56:04.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:04.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:56:04.915 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f409e3b0 con 0x7f06f406e7b0 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 10 monclient: _finish_auth 0 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06e4001ac0 con 0x7f06f406e7b0 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 10 monclient: dump: 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.916 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f8f5a640 5 monclient: authenticate success, global_id 184675 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f8f5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f06e40048a0 con 0x7f06f406e7b0 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:04.917 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f8f5a640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f8f5a640 4 set_mon_vals no callback set 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 1 -- 172.21.15.26:0/1144155217 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06e4004d90 con 0x7f06f406e7b0 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.914+0000 7f06f1d9b640 10 monclient: dump: 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.918 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.919 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.919 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.919 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.919 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.919 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.919 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.915+0000 7f06f8f5a640 10 monclient: shutdown 2024-08-19T12:56:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.915+0000 7f06f8f5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:04.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.915+0000 7f06f8f5a640 1 -- 172.21.15.26:0/1144155217 >> v1:172.21.15.26:6791/0 conn(0x7f06f406e7b0 legacy=0x7f06f4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.915+0000 7f06f8f5a640 1 -- 172.21.15.26:0/1144155217 shutdown_connections 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.915+0000 7f06f8f5a640 1 -- 172.21.15.26:0/1144155217 >> 172.21.15.26:0/1144155217 conn(0x7f06f40a8740 msgr2=0x7f06f40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.915+0000 7f06f8f5a640 1 -- 172.21.15.26:0/1144155217 shutdown_connections 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.915+0000 7f06f8f5a640 1 -- 172.21.15.26:0/1144155217 wait complete. 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.916+0000 7f06f8f5a640 10 monclient: build_initial_monmap 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.916+0000 7f06f8f5a640 10 monclient: monmap: 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.916+0000 7f06f8f5a640 1 librados: starting msgr at 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.916+0000 7f06f8f5a640 1 librados: starting objecter 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.916+0000 7f06f8f5a640 1 Processor -- start 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.916+0000 7f06f8f5a640 1 -- start start 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.916+0000 7f06f8f5a640 1 librados: setting wanted keys 2024-08-19T12:56:04.927 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.916+0000 7f06f8f5a640 1 librados: calling monclient init 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.916+0000 7f06f8f5a640 10 monclient: init 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.917+0000 7f06f8f5a640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.917+0000 7f06f8f5a640 10 monclient: _add_conns ranks=[3,4,5,6,1,2,0,7,8] 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.917+0000 7f06f8f5a640 10 monclient(hunting): picked mon.noname-g con 0x7f06f406e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.917+0000 7f06f8f5a640 10 monclient(hunting): picked mon.noname-c con 0x7f06f40a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.917+0000 7f06f8f5a640 10 monclient(hunting): picked mon.noname-h con 0x7f06f40a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.917+0000 7f06f8f5a640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06f40a5140 con 0x7f06f406e7b0 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.917+0000 7f06f8f5a640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06f40a1060 con 0x7f06f40a5920 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.917+0000 7f06f8f5a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06f40a28d0 con 0x7f06f40a5d50 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.917+0000 7f06f8f5a640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.917+0000 7f06f8f5a640 10 monclient(hunting): authenticate will time out at 1759.004471s 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.917+0000 7f06ebfff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f06f40a5920 0x7f06f41d3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:59976/0 (socket says 172.21.15.26:59976) 2024-08-19T12:56:04.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.917+0000 7f06ebfff640 1 -- 172.21.15.26:0/3493543236 learned_addr learned my addr 172.21.15.26:0/3493543236 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06f259c640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6790/0 conn(0x7f06f406e7b0 legacy=0x7f06f41ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06ebfff640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.26:6791/0 conn(0x7f06f40a5920 legacy=0x7f06f41d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 client.?.objecter ms_handle_connect 0x7f06f406e7b0 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 client.?.objecter ms_handle_connect 0x7f06f40a5920 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 client.?.objecter ms_handle_connect 0x7f06f40a5d50 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06f2d9d640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6791/0 conn(0x7f06f40a5d50 legacy=0x7f06f41d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 605275135 0 0) 0x7f06f40a5140 con 0x7f06f406e7b0 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 monclient(hunting): my global_id is 194358 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06d4004eb0 con 0x7f06f406e7b0 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 461176784 0 0) 0x7f06f40a1060 con 0x7f06f40a5920 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.919+0000 7f06eaffd640 10 monclient(hunting): my global_id is 184684 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06f40a5140 con 0x7f06f40a5920 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1585227654 0 0) 0x7f06f40a28d0 con 0x7f06f40a5d50 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient(hunting): my global_id is 184586 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06f40a1060 con 0x7f06f40a5d50 2024-08-19T12:56:04.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f2d9d640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6791/0 conn(0x7f06f40a5d50 legacy=0x7f06f41d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f2d9d640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6791/0 conn(0x7f06f40a5d50 legacy=0x7f06f41d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f2d9d640 1 --1- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6791/0 conn(0x7f06f40a5d50 0x7f06f41d6bb0 :-1 s=OPENED pgs=3660 cs=1 l=1).handle_message read tag failed 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f2d9d640 1 --1- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6791/0 conn(0x7f06f40a5d50 0x7f06f41d6bb0 :-1 s=OPENED pgs=3660 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.104:6791/0 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 898846343 0 0) 0x7f06d4004eb0 con 0x7f06f406e7b0 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06f40a28d0 con 0x7f06f406e7b0 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06e4001ae0 con 0x7f06f406e7b0 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient(hunting): dump: 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.931 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:04.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 100577823 0 0) 0x7f06f40a5140 con 0x7f06f40a5920 2024-08-19T12:56:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06d4004eb0 con 0x7f06f40a5920 2024-08-19T12:56:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06dc001380 con 0x7f06f40a5920 2024-08-19T12:56:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient(hunting): dump: 2024-08-19T12:56:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.933 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3539116307 0 0) 0x7f06f40a28d0 con 0x7f06f406e7b0 2024-08-19T12:56:04.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6791/0 conn(0x7f06f40a5d50 legacy=0x7f06f41d6bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.26:6791/0 conn(0x7f06f40a5920 legacy=0x7f06f41d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient: _finish_hunting 0 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 monclient: found mon.g 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f40a2bb0 con 0x7f06f406e7b0 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient: _finish_auth 0 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient: _check_auth_tickets 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f8f5a640 5 monclient: authenticate success, global_id 194358 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f8f5a640 10 monclient: _renew_subs 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f8f5a640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f8f5a640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f06f40a2e90 con 0x7f06f406e7b0 2024-08-19T12:56:04.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f8f5a640 10 client.194358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f8f5a640 10 monclient: _renew_subs 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f8f5a640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.104:6790/0 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f8f5a640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f06f40a24a0 con 0x7f06f406e7b0 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f8f5a640 1 librados: init done 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f06e4005040 con 0x7f06f406e7b0 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06e4005530 con 0x7f06f406e7b0 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06f159a640 4 set_mon_vals no callback set 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.920+0000 7f06eaffd640 10 monclient: dump: 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.936 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.921+0000 7f06f8f5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:04.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.921+0000 7f06f8f5a640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f06d80065c0 con 0x7f06d8002860 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.921+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f06e4011990 con 0x7f06f406e7b0 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.921+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 8 ==== osd_map(654..654 src has 350..654) ==== 5913+0+0 (unknown 1317218150 0 0) 0x7f06e404e330 con 0x7f06f406e7b0 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.921+0000 7f06eaffd640 10 client.194358.objecter ms_dispatch 0x7f06f4000b90 osd_map(654..654 src has 350..654) 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.921+0000 7f06eaffd640 3 client.194358.objecter handle_osd_map got epochs [654,654] > 0 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.921+0000 7f06eaffd640 3 client.194358.objecter handle_osd_map decoding full epoch 654 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.921+0000 7f06eaffd640 20 client.194358.objecter dump_active .. 0 homeless 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.921+0000 7f06f2d9d640 10 monclient: get_auth_request con 0x7f06d4041770 auth_method 0 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.923+0000 7f06ebfff640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6790/0 conn(0x7f06d8002860 legacy=0x7f06d8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.923+0000 7f06eaffd640 10 client.194358.objecter ms_handle_connect 0x7f06d8002860 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.923+0000 7f06eaffd640 10 client.194358.objecter resend_mon_ops 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.923+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2041767927 0 0) 0x7f06d80065c0 con 0x7f06d8002860 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.923+0000 7f06eaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.923+0000 7f06eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.923+0000 7f06eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.923+0000 7f06eaffd640 10 monclient(hunting): my global_id is 194367 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.923+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06f40a28d0 con 0x7f06d8002860 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2865140313 0 0) 0x7f06f40a28d0 con 0x7f06d8002860 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06d80065c0 con 0x7f06d8002860 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06dc004ff0 con 0x7f06d8002860 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 10 monclient: dump: 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.939 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 10 client.194358.objecter ms_handle_connect 0x7f06d4041770 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2470398547 0 0) 0x7f06d80065c0 con 0x7f06d8002860 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06d800a160 con 0x7f06d8002860 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f06dc0050f0 con 0x7f06d8002860 2024-08-19T12:56:04.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6790/0 conn(0x7f06d8002860 legacy=0x7f06d8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.924+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6790/0 conn(0x7f06d8002860 legacy=0x7f06d8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.931+0000 7f06f8f5a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:04.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.931+0000 7f06f8f5a640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f06d8009e70 con 0x7f06d8009130 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.933+0000 7f06ebfff640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6790/0 conn(0x7f06d8009130 legacy=0x7f06d80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.933+0000 7f06eaffd640 10 client.194358.objecter ms_handle_connect 0x7f06d8009130 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.933+0000 7f06eaffd640 10 client.194358.objecter resend_mon_ops 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.933+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1302514336 0 0) 0x7f06d8009e70 con 0x7f06d8009130 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.933+0000 7f06eaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.933+0000 7f06eaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.933+0000 7f06eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.933+0000 7f06eaffd640 10 monclient(hunting): my global_id is 194376 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.933+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06d80065c0 con 0x7f06d8009130 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 767759639 0 0) 0x7f06d80065c0 con 0x7f06d8009130 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06d8009e70 con 0x7f06d8009130 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f06dc0058e0 con 0x7f06d8009130 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:04.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 10 monclient: dump: 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:04.945 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 967280227 0 0) 0x7f06d8009e70 con 0x7f06d8009130 2024-08-19T12:56:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f06d800c410 con 0x7f06d8009130 2024-08-19T12:56:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7545 (unknown 0 0 2789791291) 0x7f06dc007d30 con 0x7f06d8009130 2024-08-19T12:56:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6790/0 conn(0x7f06d8009130 legacy=0x7f06d80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.934+0000 7f06eaffd640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6790/0 conn(0x7f06d8009130 legacy=0x7f06d80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:04.945 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:04.945 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 149857, 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:04.946 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:04.947 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:04.948 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:04.949 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:04.950 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:04.951 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.952 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.954 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:04.955 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:04.957 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:04.958 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:04.960 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:04.962 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.963 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:04.964 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 5 2024-08-19T12:56:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:04.965 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:04.965 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:04.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 10 librados: watch_flush enter 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 10 librados: watch_flush exit 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 20 client.194358.objecter shutdown clearing up homeless session... 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 10 client.194358.objecter successfully canceled tick 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6800/899887506 conn(0x7f06d4041770 legacy=0x7f06d4043c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 10 monclient: shutdown 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 1 -- 172.21.15.26:0/3493543236 >> v1:172.21.15.104:6790/0 conn(0x7f06f406e7b0 legacy=0x7f06f41ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f2d9d640 1 -- 172.21.15.26:0/3493543236 reap_dead start 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f2d9d640 1 -- 172.21.15.26:0/3493543236 reap_dead start 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 1 -- 172.21.15.26:0/3493543236 shutdown_connections 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 1 -- 172.21.15.26:0/3493543236 >> 172.21.15.26:0/3493543236 conn(0x7f06f40a8740 msgr2=0x7f06f4135130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 1 -- 172.21.15.26:0/3493543236 shutdown_connections 2024-08-19T12:56:04.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 1 -- 172.21.15.26:0/3493543236 wait complete. 2024-08-19T12:56:04.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:04.935+0000 7f06f8f5a640 1 librados: shutdown 2024-08-19T12:56:04.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:04.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:56:04.968 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:56:05.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.069+0000 7f403a02f640 10 monclient: get_monmap_and_config 2024-08-19T12:56:05.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.069+0000 7f403a02f640 10 monclient: build_initial_monmap 2024-08-19T12:56:05.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.069+0000 7f403a02f640 10 monclient: monmap: 2024-08-19T12:56:05.070 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:05.070 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.070 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:05.070126+0000 2024-08-19T12:56:05.070 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:05.070126+0000 2024-08-19T12:56:05.070 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:05.070 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.070 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:05.070 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:05.070 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:05.070 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:05.071 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:05.071 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:05.071 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:05.071 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:05.071 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:05.071 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 1 Processor -- start 2024-08-19T12:56:05.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 1 -- start start 2024-08-19T12:56:05.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 10 monclient: init 2024-08-19T12:56:05.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:05.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 10 monclient: _add_conns ranks=[8,4,7,1,2,3,6,5,0] 2024-08-19T12:56:05.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 10 monclient(hunting): picked mon.noname-e con 0x7f4034073470 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:05.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 10 monclient(hunting): picked mon.noname-c con 0x7f40340a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:05.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 10 monclient(hunting): picked mon.noname-i con 0x7f403413f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:05.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403409dae0 con 0x7f40340a57a0 2024-08-19T12:56:05.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4034136910 con 0x7f403413f2e0 2024-08-19T12:56:05.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403409dd50 con 0x7f4034073470 2024-08-19T12:56:05.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:05.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f403a02f640 10 monclient(hunting): authenticate will time out at 1759.157694s 2024-08-19T12:56:05.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f40337fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f4034073470 0x7f4034136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:57984/0 (socket says 172.21.15.26:57984) 2024-08-19T12:56:05.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.070+0000 7f40337fe640 1 -- 172.21.15.26:0/3932005786 learned_addr learned my addr 172.21.15.26:0/3932005786 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:05.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.072+0000 7f40337fe640 1 -- 172.21.15.26:0/3932005786 >> v1:172.21.15.26:6793/0 conn(0x7f4034073470 legacy=0x7f4034136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.072+0000 7f4032ffd640 1 -- 172.21.15.26:0/3932005786 >> v1:172.21.15.26:6791/0 conn(0x7f40340a57a0 legacy=0x7f40340a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 1 -- 172.21.15.26:0/3932005786 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2375860042 0 0) 0x7f403409dae0 con 0x7f40340a57a0 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient(hunting): my global_id is 184693 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f4033fff640 1 -- 172.21.15.26:0/3932005786 >> v1:172.21.15.104:6792/0 conn(0x7f403413f2e0 legacy=0x7f40341416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 1 -- 172.21.15.26:0/3932005786 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f401c003520 con 0x7f40340a57a0 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 1 -- 172.21.15.26:0/3932005786 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2776272369 0 0) 0x7f403409dd50 con 0x7f4034073470 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient(hunting): my global_id is 184544 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 1 -- 172.21.15.26:0/3932005786 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f403409dae0 con 0x7f4034073470 2024-08-19T12:56:05.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 1 -- 172.21.15.26:0/3932005786 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 767636302 0 0) 0x7f401c003520 con 0x7f40340a57a0 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 1 -- 172.21.15.26:0/3932005786 >> v1:172.21.15.26:6793/0 conn(0x7f4034073470 legacy=0x7f4034136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 1 -- 172.21.15.26:0/3932005786 >> v1:172.21.15.104:6792/0 conn(0x7f403413f2e0 legacy=0x7f40341416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient: _finish_hunting 0 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 1 monclient: found mon.noname-c 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 1 -- 172.21.15.26:0/3932005786 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f403409e030 con 0x7f40340a57a0 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient: _finish_auth 0 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient: _check_auth_tickets 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 1 -- 172.21.15.26:0/3932005786 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4028001b30 con 0x7f40340a57a0 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:56:05.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient: dump: 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.080 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f403a02f640 5 monclient: authenticate success, global_id 184693 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f403a02f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 1 -- 172.21.15.26:0/3932005786 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4028003610 con 0x7f40340a57a0 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 1 -- 172.21.15.26:0/3932005786 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4028003b80 con 0x7f40340a57a0 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f40327fc640 10 monclient: dump: 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.081 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f403a02f640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.073+0000 7f403a02f640 4 set_mon_vals no callback set 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.074+0000 7f403a02f640 10 monclient: shutdown 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.074+0000 7f403a02f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.074+0000 7f403a02f640 1 -- 172.21.15.26:0/3932005786 >> v1:172.21.15.26:6791/0 conn(0x7f40340a57a0 legacy=0x7f40340a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.074+0000 7f403a02f640 1 -- 172.21.15.26:0/3932005786 shutdown_connections 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.074+0000 7f403a02f640 1 -- 172.21.15.26:0/3932005786 >> 172.21.15.26:0/3932005786 conn(0x7f40340abff0 msgr2=0x7f40340ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.074+0000 7f403a02f640 1 -- 172.21.15.26:0/3932005786 shutdown_connections 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.074+0000 7f403a02f640 1 -- 172.21.15.26:0/3932005786 wait complete. 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.074+0000 7f403a02f640 10 monclient: build_initial_monmap 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.074+0000 7f403a02f640 10 monclient: monmap: 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:05.083 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:05.084 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.074+0000 7f403a02f640 1 librados: starting msgr at 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.074+0000 7f403a02f640 1 librados: starting objecter 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.074+0000 7f403a02f640 1 Processor -- start 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 1 -- start start 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 1 librados: setting wanted keys 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 1 librados: calling monclient init 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 10 monclient: init 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 10 monclient: _add_conns ranks=[6,7,1,5,4,2,8,3,0] 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 10 monclient(hunting): picked mon.noname-d con 0x7f4034073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 10 monclient(hunting): picked mon.noname-i con 0x7f40340a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:05.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 10 monclient(hunting): picked mon.noname-f con 0x7f403413f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:05.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40340a5320 con 0x7f403413f2e0 2024-08-19T12:56:05.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4034136d30 con 0x7f4034073470 2024-08-19T12:56:05.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40340aa1d0 con 0x7f40340a57a0 2024-08-19T12:56:05.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:05.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f403a02f640 10 monclient(hunting): authenticate will time out at 1759.162631s 2024-08-19T12:56:05.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f40337fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f4034073470 0x7f40340a1f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34114/0 (socket says 172.21.15.26:34114) 2024-08-19T12:56:05.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.075+0000 7f40337fe640 1 -- 172.21.15.26:0/3824332474 learned_addr learned my addr 172.21.15.26:0/3824332474 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:05.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.077+0000 7f40337fe640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.26:6792/0 conn(0x7f4034073470 legacy=0x7f40340a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.077+0000 7f401bfff640 10 client.?.objecter ms_handle_connect 0x7f4034073470 2024-08-19T12:56:05.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.077+0000 7f401bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:05.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 client.?.objecter ms_handle_connect 0x7f403413f2e0 2024-08-19T12:56:05.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:05.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f4033fff640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.104:6789/0 conn(0x7f403413f2e0 legacy=0x7f40341d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 client.?.objecter ms_handle_connect 0x7f40340a57a0 2024-08-19T12:56:05.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:05.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f4032ffd640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.104:6792/0 conn(0x7f40340a57a0 legacy=0x7f40340a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286217764 0 0) 0x7f40340a5320 con 0x7f403413f2e0 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): my global_id is 204184 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4020004eb0 con 0x7f403413f2e0 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 782500939 0 0) 0x7f40340aa1d0 con 0x7f40340a57a0 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): my global_id is 136294 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40340a5320 con 0x7f40340a57a0 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1083152767 0 0) 0x7f4020004eb0 con 0x7f403413f2e0 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40340aa1d0 con 0x7f403413f2e0 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f402c001ff0 con 0x7f403413f2e0 2024-08-19T12:56:05.092 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): dump: 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.093 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3806167238 0 0) 0x7f40340a5320 con 0x7f40340a57a0 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4020004eb0 con 0x7f40340a57a0 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4028003b00 con 0x7f40340a57a0 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient(hunting): dump: 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.094 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3954576232 0 0) 0x7f40340aa1d0 con 0x7f403413f2e0 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.104:6792/0 conn(0x7f40340a57a0 legacy=0x7f40340a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.26:6792/0 conn(0x7f4034073470 legacy=0x7f40340a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 monclient: found mon.f 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40340a2d20 con 0x7f403413f2e0 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient: _finish_auth 0 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f403a02f640 5 monclient: authenticate success, global_id 204184 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f403a02f640 10 monclient: _renew_subs 2024-08-19T12:56:05.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f403a02f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f403a02f640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4034137010 con 0x7f403413f2e0 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f403a02f640 10 client.204184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f403a02f640 10 monclient: _renew_subs 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f403a02f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f403a02f640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f403413e660 con 0x7f403413f2e0 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f403a02f640 1 librados: init done 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f402c003c40 con 0x7f403413f2e0 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f402c0041f0 con 0x7f403413f2e0 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f4031ffb640 4 set_mon_vals no callback set 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.078+0000 7f401bfff640 10 monclient: dump: 2024-08-19T12:56:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.098 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.079+0000 7f403a02f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.079+0000 7f403a02f640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f40140065c0 con 0x7f4014002860 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.079+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f402c004490 con 0x7f403413f2e0 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.080+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(654..654 src has 350..654) ==== 5913+0+0 (unknown 1317218150 0 0) 0x7f402c04e5d0 con 0x7f403413f2e0 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.080+0000 7f401bfff640 10 client.204184.objecter ms_dispatch 0x7f4034000b90 osd_map(654..654 src has 350..654) 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.080+0000 7f401bfff640 3 client.204184.objecter handle_osd_map got epochs [654,654] > 0 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.080+0000 7f401bfff640 3 client.204184.objecter handle_osd_map decoding full epoch 654 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.080+0000 7f401bfff640 20 client.204184.objecter dump_active .. 0 homeless 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.080+0000 7f4032ffd640 10 monclient: get_auth_request con 0x7f40200414b0 auth_method 0 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.081+0000 7f401bfff640 10 client.204184.objecter ms_handle_connect 0x7f4014002860 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.081+0000 7f401bfff640 10 client.204184.objecter resend_mon_ops 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.081+0000 7f40337fe640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.104:6791/0 conn(0x7f4014002860 legacy=0x7f4014004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1150888681 0 0) 0x7f40140065c0 con 0x7f4014002860 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 10 monclient(hunting): my global_id is 184595 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40340aa1d0 con 0x7f4014002860 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1471156275 0 0) 0x7f40340aa1d0 con 0x7f4014002860 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40140065c0 con 0x7f4014002860 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4024001ab0 con 0x7f4014002860 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 10 monclient: dump: 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.100 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 10 client.204184.objecter ms_handle_connect 0x7f40200414b0 2024-08-19T12:56:05.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 106224061 0 0) 0x7f40140065c0 con 0x7f4014002860 2024-08-19T12:56:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f401400a160 con 0x7f4014002860 2024-08-19T12:56:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4024006860 con 0x7f4014002860 2024-08-19T12:56:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.104:6791/0 conn(0x7f4014002860 legacy=0x7f4014004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.082+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.104:6791/0 conn(0x7f4014002860 legacy=0x7f4014004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.089+0000 7f403a02f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:05.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.089+0000 7f403a02f640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4014009e70 con 0x7f4014009130 2024-08-19T12:56:05.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.091+0000 7f40337fe640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.104:6791/0 conn(0x7f4014009130 legacy=0x7f40140095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.091+0000 7f401bfff640 10 client.204184.objecter ms_handle_connect 0x7f4014009130 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.091+0000 7f401bfff640 10 client.204184.objecter resend_mon_ops 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.091+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 236701258 0 0) 0x7f4014009e70 con 0x7f4014009130 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.091+0000 7f401bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.091+0000 7f401bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.091+0000 7f401bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.091+0000 7f401bfff640 10 monclient(hunting): my global_id is 184604 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.091+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40140065c0 con 0x7f4014009130 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 75245727 0 0) 0x7f40140065c0 con 0x7f4014009130 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4014009e70 con 0x7f4014009130 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f40240050f0 con 0x7f4014009130 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:05.104 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 10 monclient: dump: 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.105 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.106 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.106 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.106 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2779113407 0 0) 0x7f4014009e70 con 0x7f4014009130 2024-08-19T12:56:05.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f401400c410 con 0x7f4014009130 2024-08-19T12:56:05.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7879 (unknown 0 0 3712822621) 0x7f4024006a10 con 0x7f4014009130 2024-08-19T12:56:05.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:05.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:05.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.104:6791/0 conn(0x7f4014009130 legacy=0x7f40140095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f401bfff640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.104:6791/0 conn(0x7f4014009130 legacy=0x7f40140095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:05.106 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 290561, 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:05.107 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 2, 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:05.108 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:05.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:05.110 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:05.111 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:05.112 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.113 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.114 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:05.115 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:05.118 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.120 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:05.121 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:05.123 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:56:05.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:56:05.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.126 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:05.126 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f403a02f640 10 librados: watch_flush enter 2024-08-19T12:56:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f403a02f640 10 librados: watch_flush exit 2024-08-19T12:56:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f403a02f640 20 client.204184.objecter shutdown clearing up homeless session... 2024-08-19T12:56:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f403a02f640 10 client.204184.objecter successfully canceled tick 2024-08-19T12:56:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f403a02f640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.104:6800/899887506 conn(0x7f40200414b0 legacy=0x7f4020043970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f403a02f640 10 monclient: shutdown 2024-08-19T12:56:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f403a02f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f403a02f640 1 -- 172.21.15.26:0/3824332474 >> v1:172.21.15.104:6789/0 conn(0x7f403413f2e0 legacy=0x7f40341d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f4033fff640 1 -- 172.21.15.26:0/3824332474 reap_dead start 2024-08-19T12:56:05.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.092+0000 7f4033fff640 1 -- 172.21.15.26:0/3824332474 reap_dead start 2024-08-19T12:56:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.093+0000 7f403a02f640 1 -- 172.21.15.26:0/3824332474 shutdown_connections 2024-08-19T12:56:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.093+0000 7f403a02f640 1 -- 172.21.15.26:0/3824332474 >> 172.21.15.26:0/3824332474 conn(0x7f40340abff0 msgr2=0x7f40341338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.093+0000 7f403a02f640 1 -- 172.21.15.26:0/3824332474 shutdown_connections 2024-08-19T12:56:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.093+0000 7f403a02f640 1 -- 172.21.15.26:0/3824332474 wait complete. 2024-08-19T12:56:05.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.093+0000 7f403a02f640 1 librados: shutdown 2024-08-19T12:56:05.129 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:05.129 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:56:05.129 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:56:05.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.230+0000 7f9eab09f640 10 monclient: get_monmap_and_config 2024-08-19T12:56:05.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.230+0000 7f9eab09f640 10 monclient: build_initial_monmap 2024-08-19T12:56:05.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.230+0000 7f9eab09f640 10 monclient: monmap: 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:05.231762+0000 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:05.231762+0000 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:05.233 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:05.233 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:05.233 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.231+0000 7f9eab09f640 1 Processor -- start 2024-08-19T12:56:05.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.231+0000 7f9eab09f640 1 -- start start 2024-08-19T12:56:05.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.231+0000 7f9eab09f640 10 monclient: init 2024-08-19T12:56:05.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.232+0000 7f9eab09f640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:05.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.232+0000 7f9eab09f640 10 monclient: _add_conns ranks=[1,5,2,8,7,0,6,3,4] 2024-08-19T12:56:05.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.232+0000 7f9eab09f640 10 monclient(hunting): picked mon.noname-f con 0x7f9ea406e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:05.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.232+0000 7f9eab09f640 10 monclient(hunting): picked mon.noname-h con 0x7f9ea40a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:05.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.232+0000 7f9eab09f640 10 monclient(hunting): picked mon.noname-b con 0x7f9ea40a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:56:05.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.232+0000 7f9eab09f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ea4136790 con 0x7f9ea406e7b0 2024-08-19T12:56:05.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.232+0000 7f9eab09f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ea409ddf0 con 0x7f9ea40a5d50 2024-08-19T12:56:05.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.232+0000 7f9eab09f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ea409e0d0 con 0x7f9ea40a5920 2024-08-19T12:56:05.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.232+0000 7f9eab09f640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:05.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.232+0000 7f9eab09f640 10 monclient(hunting): authenticate will time out at 1759.319252s 2024-08-19T12:56:05.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.232+0000 7f9ea963c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f9ea40a5d50 0x7f9ea4141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55148/0 (socket says 172.21.15.26:55148) 2024-08-19T12:56:05.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.232+0000 7f9ea963c640 1 -- 172.21.15.26:0/3781730453 learned_addr learned my addr 172.21.15.26:0/3781730453 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:05.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9ea963c640 1 -- 172.21.15.26:0/3781730453 >> v1:172.21.15.26:6790/0 conn(0x7f9ea40a5d50 legacy=0x7f9ea4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9bfff640 1 -- 172.21.15.26:0/3781730453 >> v1:172.21.15.104:6791/0 conn(0x7f9ea40a5920 legacy=0x7f9ea413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9ea8e3b640 1 -- 172.21.15.26:0/3781730453 >> v1:172.21.15.104:6789/0 conn(0x7f9ea406e7b0 legacy=0x7f9ea4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 1 -- 172.21.15.26:0/3781730453 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3095018802 0 0) 0x7f9e9c001450 con 0x7f9ea40a5d50 2024-08-19T12:56:05.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:05.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:05.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:05.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 10 monclient(hunting): my global_id is 27371 2024-08-19T12:56:05.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 1 -- 172.21.15.26:0/3781730453 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e8c003520 con 0x7f9ea40a5d50 2024-08-19T12:56:05.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 1 -- 172.21.15.26:0/3781730453 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1238667461 0 0) 0x7f9e8c003520 con 0x7f9ea40a5d50 2024-08-19T12:56:05.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 1 -- 172.21.15.26:0/3781730453 >> v1:172.21.15.104:6791/0 conn(0x7f9ea40a5920 legacy=0x7f9ea413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 1 -- 172.21.15.26:0/3781730453 >> v1:172.21.15.104:6789/0 conn(0x7f9ea406e7b0 legacy=0x7f9ea4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 1 monclient: found mon.noname-b 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 1 -- 172.21.15.26:0/3781730453 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea409e3b0 con 0x7f9ea40a5d50 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 10 monclient: _finish_auth 0 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 1 -- 172.21.15.26:0/3781730453 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9e9c001cb0 con 0x7f9ea40a5d50 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9e9b7fe640 10 monclient: dump: 2024-08-19T12:56:05.243 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.244 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9eab09f640 5 monclient: authenticate success, global_id 27371 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.234+0000 7f9eab09f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9e9b7fe640 1 -- 172.21.15.26:0/3781730453 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9e9c004990 con 0x7f9ea40a5d50 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9e9b7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9e9b7fe640 1 -- 172.21.15.26:0/3781730453 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9e9c004f00 con 0x7f9ea40a5d50 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9e9b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9e9b7fe640 10 monclient: dump: 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.245 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9eab09f640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9eab09f640 4 set_mon_vals no callback set 2024-08-19T12:56:05.246 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9eab09f640 10 monclient: shutdown 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9eab09f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9eab09f640 1 -- 172.21.15.26:0/3781730453 >> v1:172.21.15.26:6790/0 conn(0x7f9ea40a5d50 legacy=0x7f9ea4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9eab09f640 1 -- 172.21.15.26:0/3781730453 shutdown_connections 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9eab09f640 1 -- 172.21.15.26:0/3781730453 >> 172.21.15.26:0/3781730453 conn(0x7f9ea40a8740 msgr2=0x7f9ea40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9eab09f640 1 -- 172.21.15.26:0/3781730453 shutdown_connections 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.235+0000 7f9eab09f640 1 -- 172.21.15.26:0/3781730453 wait complete. 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.236+0000 7f9eab09f640 10 monclient: build_initial_monmap 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.236+0000 7f9eab09f640 10 monclient: monmap: 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:05.247 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:05.250 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.250 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.236+0000 7f9eab09f640 1 librados: starting msgr at 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.236+0000 7f9eab09f640 1 librados: starting objecter 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.236+0000 7f9eab09f640 1 Processor -- start 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.236+0000 7f9eab09f640 1 -- start start 2024-08-19T12:56:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.236+0000 7f9eab09f640 1 librados: setting wanted keys 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.236+0000 7f9eab09f640 1 librados: calling monclient init 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.236+0000 7f9eab09f640 10 monclient: init 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9eab09f640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9eab09f640 10 monclient: _add_conns ranks=[5,6,0,2,7,3,8,1,4] 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9eab09f640 10 monclient(hunting): picked mon.noname-h con 0x7f9ea406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9eab09f640 10 monclient(hunting): picked mon.noname-d con 0x7f9ea40a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9eab09f640 10 monclient(hunting): picked mon.noname-a con 0x7f9ea40a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9eab09f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ea40a1370 con 0x7f9ea40a5d50 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9eab09f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ea40a1ed0 con 0x7f9ea406e7b0 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9eab09f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ea4134970 con 0x7f9ea40a5920 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9eab09f640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9eab09f640 10 monclient(hunting): authenticate will time out at 1759.324103s 2024-08-19T12:56:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9e9bfff640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f9ea40a5920 0x7f9ea40a4e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34122/0 (socket says 172.21.15.26:34122) 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9e9bfff640 1 -- 172.21.15.26:0/176470373 learned_addr learned my addr 172.21.15.26:0/176470373 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.237+0000 7f9ea963c640 1 --1- 172.21.15.26:0/176470373 >> v1:172.21.15.26:6789/0 conn(0x7f9ea40a5d50 0x7f9ea41d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52594/0 (socket says 172.21.15.26:52594) 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9ea963c640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.26:6789/0 conn(0x7f9ea40a5d50 legacy=0x7f9ea41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 client.?.objecter ms_handle_connect 0x7f9ea40a5d50 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e9bfff640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.26:6792/0 conn(0x7f9ea40a5920 legacy=0x7f9ea40a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 client.?.objecter ms_handle_connect 0x7f9ea40a5920 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9ea8e3b640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.104:6791/0 conn(0x7f9ea406e7b0 legacy=0x7f9ea4132b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 client.?.objecter ms_handle_connect 0x7f9ea406e7b0 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 865701515 0 0) 0x7f9ea40a1370 con 0x7f9ea40a5d50 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:05.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 monclient(hunting): my global_id is 194391 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e90004eb0 con 0x7f9ea40a5d50 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2883731512 0 0) 0x7f9e90004eb0 con 0x7f9ea40a5d50 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ea40a1370 con 0x7f9ea40a5d50 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9e9c0013a0 con 0x7f9ea40a5d50 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 monclient(hunting): dump: 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:05.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1066277153 0 0) 0x7f9ea40a1370 con 0x7f9ea40a5d50 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.26:6792/0 conn(0x7f9ea40a5920 legacy=0x7f9ea40a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.104:6791/0 conn(0x7f9ea406e7b0 legacy=0x7f9ea4132b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 monclient: _finish_hunting 0 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 1 monclient: found mon.a 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea40a2e30 con 0x7f9ea40a5d50 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 monclient: _finish_auth 0 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 10 monclient: _check_auth_tickets 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9e997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9eab09f640 5 monclient: authenticate success, global_id 194391 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9eab09f640 10 monclient: _renew_subs 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9eab09f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:56:05.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.239+0000 7f9eab09f640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea40a5540 con 0x7f9ea40a5d50 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9eab09f640 10 client.194391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9eab09f640 10 monclient: _renew_subs 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9eab09f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9eab09f640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9ea41d7b90 con 0x7f9ea40a5d50 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9eab09f640 1 librados: init done 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9e9c004950 con 0x7f9ea40a5d50 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9e997fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9e9c005470 con 0x7f9ea40a5d50 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9e997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9e9affd640 4 set_mon_vals no callback set 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9e997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9e997fa640 10 monclient: dump: 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.257 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.258 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9eab09f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:05.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.240+0000 7f9eab09f640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9e840065c0 con 0x7f9e84002860 2024-08-19T12:56:05.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.241+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f9e9c0118d0 con 0x7f9ea40a5d50 2024-08-19T12:56:05.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.241+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(655..655 src has 350..655) ==== 5939+0+0 (unknown 580299850 0 0) 0x7f9e9c04e300 con 0x7f9ea40a5d50 2024-08-19T12:56:05.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.241+0000 7f9e997fa640 10 client.194391.objecter ms_dispatch 0x7f9ea4000b90 osd_map(655..655 src has 350..655) 2024-08-19T12:56:05.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.241+0000 7f9e997fa640 3 client.194391.objecter handle_osd_map got epochs [655,655] > 0 2024-08-19T12:56:05.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.241+0000 7f9e997fa640 3 client.194391.objecter handle_osd_map decoding full epoch 655 2024-08-19T12:56:05.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.241+0000 7f9e997fa640 20 client.194391.objecter dump_active .. 0 homeless 2024-08-19T12:56:05.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.241+0000 7f9e9bfff640 10 monclient: get_auth_request con 0x7f9e90040f30 auth_method 0 2024-08-19T12:56:05.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.243+0000 7f9ea8e3b640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.104:6792/0 conn(0x7f9e84002860 legacy=0x7f9e84004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.243+0000 7f9e997fa640 10 client.194391.objecter ms_handle_connect 0x7f9e84002860 2024-08-19T12:56:05.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.243+0000 7f9e997fa640 10 client.194391.objecter resend_mon_ops 2024-08-19T12:56:05.260 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.243+0000 7f9e997fa640 10 client.194391.objecter ms_handle_connect 0x7f9e90040f30 2024-08-19T12:56:05.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.842+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 804063052 0 0) 0x7f9e840065c0 con 0x7f9e84002860 2024-08-19T12:56:05.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.842+0000 7f9e997fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:05.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.842+0000 7f9e997fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:05.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.842+0000 7f9e997fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:05.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.842+0000 7f9e997fa640 10 monclient(hunting): my global_id is 136303 2024-08-19T12:56:05.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.842+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ea40a1370 con 0x7f9e84002860 2024-08-19T12:56:05.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.843+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1099807 0 0) 0x7f9ea40a1370 con 0x7f9e84002860 2024-08-19T12:56:05.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.843+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e840065c0 con 0x7f9e84002860 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.843+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9ea0001e90 con 0x7f9e84002860 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.843+0000 7f9e997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.843+0000 7f9e997fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.843+0000 7f9e997fa640 10 monclient: dump: 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.845 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.846 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.846 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.846 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.846 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.846 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.846 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.843+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1043467857 0 0) 0x7f9e840065c0 con 0x7f9e84002860 2024-08-19T12:56:05.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.843+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e8400a160 con 0x7f9e84002860 2024-08-19T12:56:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.844+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9ea0005b40 con 0x7f9e84002860 2024-08-19T12:56:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.844+0000 7f9e997fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.844+0000 7f9e997fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.844+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.104:6792/0 conn(0x7f9e84002860 legacy=0x7f9e84004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.844+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.104:6792/0 conn(0x7f9e84002860 legacy=0x7f9e84004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:05.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.858+0000 7f9eab09f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:05.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.858+0000 7f9eab09f640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9e84009e50 con 0x7f9e84009110 2024-08-19T12:56:05.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9ea8e3b640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.104:6792/0 conn(0x7f9e84009110 legacy=0x7f9e840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:05.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 10 client.194391.objecter ms_handle_connect 0x7f9e84009110 2024-08-19T12:56:05.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 10 client.194391.objecter resend_mon_ops 2024-08-19T12:56:05.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3422025206 0 0) 0x7f9e84009e50 con 0x7f9e84009110 2024-08-19T12:56:05.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:05.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:05.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:05.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 10 monclient(hunting): my global_id is 136312 2024-08-19T12:56:05.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e840065c0 con 0x7f9e84009110 2024-08-19T12:56:05.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4172111531 0 0) 0x7f9e840065c0 con 0x7f9e84009110 2024-08-19T12:56:05.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9e84009e50 con 0x7f9e84009110 2024-08-19T12:56:05.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9ea0004040 con 0x7f9e84009110 2024-08-19T12:56:05.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:05.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:05.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.861+0000 7f9e997fa640 10 monclient: dump: 2024-08-19T12:56:05.869 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:05.869 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:05.869 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.869 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:05.869 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:05.869 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:05.869 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:05.870 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:05.870 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:05.871 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:05.871 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:05.871 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:05.871 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:05.871 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:05.871 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:05.871 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:05.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.862+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3124223181 0 0) 0x7f9e84009e50 con 0x7f9e84009110 2024-08-19T12:56:05.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.862+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9e8400c3f0 con 0x7f9e84009110 2024-08-19T12:56:05.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.862+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7545 (unknown 0 0 4172786347) 0x7f9ea00042e0 con 0x7f9e84009110 2024-08-19T12:56:05.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.862+0000 7f9e997fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:05.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.862+0000 7f9e997fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:05.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.862+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.104:6792/0 conn(0x7f9e84009110 legacy=0x7f9e840095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.862+0000 7f9e997fa640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.104:6792/0 conn(0x7f9e84009110 legacy=0x7f9e840095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:05.873 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:05.873 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:05.873 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:05.873 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:05.873 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:05.873 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 936219, 2024-08-19T12:56:05.873 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:05.873 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:05.873 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:05.873 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:05.873 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:05.873 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 3, 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:05.874 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:05.875 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:05.875 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:05.875 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:05.875 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:05.875 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:05.875 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:05.875 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:05.875 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:05.875 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:05.877 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:05.878 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:05.879 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:05.879 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:05.879 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:05.879 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:05.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.879 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:05.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.879 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:05.879 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:05.879 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.880 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:05.881 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.881 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:05.881 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:05.881 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.881 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.881 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.881 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:05.882 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.883 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.885 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:05.886 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.889 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.890 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:05.892 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:05.893 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:05.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.862+0000 7f9eab09f640 10 librados: watch_flush enter 2024-08-19T12:56:05.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9eab09f640 10 librados: watch_flush exit 2024-08-19T12:56:05.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9eab09f640 20 client.194391.objecter shutdown clearing up homeless session... 2024-08-19T12:56:05.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9eab09f640 10 client.194391.objecter successfully canceled tick 2024-08-19T12:56:05.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9eab09f640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.104:6800/899887506 conn(0x7f9e90040f30 legacy=0x7f9e900433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9eab09f640 10 monclient: shutdown 2024-08-19T12:56:05.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9eab09f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:05.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9eab09f640 1 -- 172.21.15.26:0/176470373 >> v1:172.21.15.26:6789/0 conn(0x7f9ea40a5d50 legacy=0x7f9ea41d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:05.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9ea963c640 1 -- 172.21.15.26:0/176470373 reap_dead start 2024-08-19T12:56:05.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9ea963c640 1 -- 172.21.15.26:0/176470373 reap_dead start 2024-08-19T12:56:05.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9eab09f640 1 -- 172.21.15.26:0/176470373 shutdown_connections 2024-08-19T12:56:05.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9eab09f640 1 -- 172.21.15.26:0/176470373 >> 172.21.15.26:0/176470373 conn(0x7f9ea40a8740 msgr2=0x7f9ea40aa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:05.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9eab09f640 1 -- 172.21.15.26:0/176470373 shutdown_connections 2024-08-19T12:56:05.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9eab09f640 1 -- 172.21.15.26:0/176470373 wait complete. 2024-08-19T12:56:05.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.863+0000 7f9eab09f640 1 librados: shutdown 2024-08-19T12:56:05.897 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:05.897 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-08-19T12:56:05.897 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-08-19T12:56:06.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.999+0000 7fa31549f640 10 monclient: get_monmap_and_config 2024-08-19T12:56:06.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.999+0000 7fa31549f640 10 monclient: build_initial_monmap 2024-08-19T12:56:06.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:05.999+0000 7fa31549f640 10 monclient: monmap: 2024-08-19T12:56:06.000 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:06.001 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:06.001 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:06.000542+0000 2024-08-19T12:56:06.001 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:06.000542+0000 2024-08-19T12:56:06.001 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:06.001 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:06.002 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:06.002 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:06.002 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:06.002 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:06.002 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:06.002 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:06.003 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:06.003 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:06.003 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:06.003 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:06.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.000+0000 7fa31549f640 1 Processor -- start 2024-08-19T12:56:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.000+0000 7fa31549f640 1 -- start start 2024-08-19T12:56:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.000+0000 7fa31549f640 10 monclient: init 2024-08-19T12:56:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.000+0000 7fa31549f640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.000+0000 7fa31549f640 10 monclient: _add_conns ranks=[6,7,5,2,3,4,8,1,0] 2024-08-19T12:56:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.000+0000 7fa31549f640 10 monclient(hunting): picked mon.noname-d con 0x7fa310073470 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.000+0000 7fa31549f640 10 monclient(hunting): picked mon.noname-i con 0x7fa3100a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.000+0000 7fa31549f640 10 monclient(hunting): picked mon.noname-h con 0x7fa31013f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.000+0000 7fa31549f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa31009dae0 con 0x7fa31013f2e0 2024-08-19T12:56:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.000+0000 7fa31549f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa310136910 con 0x7fa310073470 2024-08-19T12:56:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.000+0000 7fa31549f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa31009dd50 con 0x7fa3100a57a0 2024-08-19T12:56:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.000+0000 7fa31549f640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:06.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.001+0000 7fa31549f640 10 monclient(hunting): authenticate will time out at 1760.088022s 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.001+0000 7fa30effd640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fa310073470 0x7fa310136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34132/0 (socket says 172.21.15.26:34132) 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.001+0000 7fa30effd640 1 -- 172.21.15.26:0/3494638133 learned_addr learned my addr 172.21.15.26:0/3494638133 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30effd640 1 -- 172.21.15.26:0/3494638133 >> v1:172.21.15.26:6792/0 conn(0x7fa310073470 legacy=0x7fa310136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa307fff640 1 -- 172.21.15.26:0/3494638133 >> v1:172.21.15.104:6792/0 conn(0x7fa3100a57a0 legacy=0x7fa3100a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30f7fe640 1 -- 172.21.15.26:0/3494638133 >> v1:172.21.15.104:6791/0 conn(0x7fa31013f2e0 legacy=0x7fa3101416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3199555017 0 0) 0x7fa310136910 con 0x7fa310073470 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient(hunting): my global_id is 194361 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2fc003520 con 0x7fa310073470 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 514461788 0 0) 0x7fa31009dd50 con 0x7fa3100a57a0 2024-08-19T12:56:06.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient(hunting): my global_id is 136321 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa310136910 con 0x7fa3100a57a0 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2514953216 0 0) 0x7fa31009dae0 con 0x7fa31013f2e0 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient(hunting): my global_id is 184613 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa31009dd50 con 0x7fa31013f2e0 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3767711628 0 0) 0x7fa2fc003520 con 0x7fa310073470 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 >> v1:172.21.15.104:6792/0 conn(0x7fa3100a57a0 legacy=0x7fa3100a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 >> v1:172.21.15.104:6791/0 conn(0x7fa31013f2e0 legacy=0x7fa3101416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:06.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 monclient: found mon.noname-d 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31009e030 con 0x7fa310073470 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient: _finish_auth 0 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa31549f640 5 monclient: authenticate success, global_id 194361 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa300001ae0 con 0x7fa310073470 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa31549f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient: dump: 2024-08-19T12:56:06.020 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:06.021 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa31549f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3000048a0 con 0x7fa310073470 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa31549f640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 1 -- 172.21.15.26:0/3494638133 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa300004e10 con 0x7fa310073470 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa31549f640 4 set_mon_vals no callback set 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.003+0000 7fa30e7fc640 10 monclient: dump: 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:06.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:06.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.004+0000 7fa31549f640 10 monclient: shutdown 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.004+0000 7fa31549f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.004+0000 7fa31549f640 1 -- 172.21.15.26:0/3494638133 >> v1:172.21.15.26:6792/0 conn(0x7fa310073470 legacy=0x7fa310136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.004+0000 7fa31549f640 1 -- 172.21.15.26:0/3494638133 shutdown_connections 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.004+0000 7fa31549f640 1 -- 172.21.15.26:0/3494638133 >> 172.21.15.26:0/3494638133 conn(0x7fa3100abff0 msgr2=0x7fa3100ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.004+0000 7fa31549f640 1 -- 172.21.15.26:0/3494638133 shutdown_connections 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.004+0000 7fa31549f640 1 -- 172.21.15.26:0/3494638133 wait complete. 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.004+0000 7fa31549f640 10 monclient: build_initial_monmap 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.004+0000 7fa31549f640 10 monclient: monmap: 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:06.024 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 1 librados: starting msgr at 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 1 librados: starting objecter 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 1 Processor -- start 2024-08-19T12:56:06.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 1 -- start start 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 1 librados: setting wanted keys 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 1 librados: calling monclient init 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 10 monclient: init 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 10 monclient: _add_conns ranks=[3,7,4,1,2,8,0,5,6] 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 10 monclient(hunting): picked mon.noname-g con 0x7fa310073470 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 10 monclient(hunting): picked mon.noname-i con 0x7fa3100a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 10 monclient(hunting): picked mon.noname-c con 0x7fa31013f2e0 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa31013e630 con 0x7fa310073470 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3101374d0 con 0x7fa31013f2e0 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa310133140 con 0x7fa3100a57a0 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:06.026 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa31549f640 10 monclient(hunting): authenticate will time out at 1760.092809s 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa30f7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fa31013f2e0 0x7fa3101d6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:60004/0 (socket says 172.21.15.26:60004) 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.005+0000 7fa30f7fe640 1 -- 172.21.15.26:0/4094998873 learned_addr learned my addr 172.21.15.26:0/4094998873 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.007+0000 7fa30f7fe640 1 -- 172.21.15.26:0/4094998873 >> v1:172.21.15.26:6791/0 conn(0x7fa31013f2e0 legacy=0x7fa3101d6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 client.?.objecter ms_handle_connect 0x7fa31013f2e0 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 client.?.objecter ms_handle_connect 0x7fa3100a57a0 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa307fff640 1 -- 172.21.15.26:0/4094998873 >> v1:172.21.15.104:6792/0 conn(0x7fa3100a57a0 legacy=0x7fa310132a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa30effd640 1 -- 172.21.15.26:0/4094998873 >> v1:172.21.15.104:6790/0 conn(0x7fa310073470 legacy=0x7fa310132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 client.?.objecter ms_handle_connect 0x7fa310073470 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2697942273 0 0) 0x7fa3101374d0 con 0x7fa31013f2e0 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:06.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): my global_id is 184702 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2f0004eb0 con 0x7fa31013f2e0 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1959916691 0 0) 0x7fa310133140 con 0x7fa3100a57a0 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): my global_id is 136330 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3101374d0 con 0x7fa3100a57a0 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 111190988 0 0) 0x7fa31013e630 con 0x7fa310073470 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:06.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): my global_id is 194385 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa310133140 con 0x7fa310073470 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2113864146 0 0) 0x7fa2f0004eb0 con 0x7fa31013f2e0 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa31013e630 con 0x7fa31013f2e0 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa308002070 con 0x7fa31013f2e0 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): dump: 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:06.029 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 990002759 0 0) 0x7fa310133140 con 0x7fa310073470 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa2f0004eb0 con 0x7fa310073470 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2777393749 0 0) 0x7fa3101374d0 con 0x7fa3100a57a0 2024-08-19T12:56:06.030 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa310133140 con 0x7fa3100a57a0 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa300004d90 con 0x7fa310073470 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): dump: 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:06.031 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa2f8003610 con 0x7fa3100a57a0 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient(hunting): dump: 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1015788958 0 0) 0x7fa31013e630 con 0x7fa31013f2e0 2024-08-19T12:56:06.033 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 >> v1:172.21.15.104:6792/0 conn(0x7fa3100a57a0 legacy=0x7fa310132a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 >> v1:172.21.15.104:6790/0 conn(0x7fa310073470 legacy=0x7fa310132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient: _finish_hunting 0 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 monclient: found mon.c 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3101353b0 con 0x7fa31013f2e0 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient: _finish_auth 0 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient: _check_auth_tickets 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa31549f640 5 monclient: authenticate success, global_id 184702 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa31549f640 10 monclient: _renew_subs 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa31549f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:56:06.034 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa31549f640 1 -- 172.21.15.26:0/4094998873 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa3101371f0 con 0x7fa31013f2e0 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa31549f640 10 client.184702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa31549f640 10 monclient: _renew_subs 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa31549f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa31549f640 1 -- 172.21.15.26:0/4094998873 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fa3100a3370 con 0x7fa31013f2e0 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa31549f640 1 librados: init done 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa308003db0 con 0x7fa31013f2e0 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa3080042e0 con 0x7fa31013f2e0 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa30dffb640 4 set_mon_vals no callback set 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.008+0000 7fa3077fe640 10 monclient: dump: 2024-08-19T12:56:06.035 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:06.036 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.009+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa308004580 con 0x7fa31013f2e0 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.009+0000 7fa31549f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.009+0000 7fa31549f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.009+0000 7fa31549f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.009+0000 7fa31549f640 1 -- 172.21.15.26:0/4094998873 --> v1:172.21.15.26:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f4005080 con 0x7fa31013f2e0 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.009+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(655..655 src has 350..655) ==== 5939+0+0 (unknown 580299850 0 0) 0x7fa30804e660 con 0x7fa31013f2e0 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.009+0000 7fa3077fe640 10 client.184702.objecter ms_dispatch 0x7fa310000b90 osd_map(655..655 src has 350..655) 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.009+0000 7fa3077fe640 3 client.184702.objecter handle_osd_map got epochs [655,655] > 0 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.009+0000 7fa3077fe640 3 client.184702.objecter handle_osd_map decoding full epoch 655 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.009+0000 7fa3077fe640 20 client.184702.objecter dump_active .. 0 homeless 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.009+0000 7fa30effd640 10 monclient: get_auth_request con 0x7fa2f0041960 auth_method 0 2024-08-19T12:56:06.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.012+0000 7fa3077fe640 10 client.184702.objecter ms_handle_connect 0x7fa2f0041960 2024-08-19T12:56:06.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.014+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.4 v1:172.21.15.26:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fa3080171c0 con 0x7fa31013f2e0 2024-08-19T12:56:06.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.014+0000 7fa3077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:06.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.014+0000 7fa3077fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:06.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.270+0000 7fa31549f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-08-19T12:56:06.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.270+0000 7fa31549f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-08-19T12:56:06.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.270+0000 7fa31549f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:56:06.271 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.270+0000 7fa31549f640 1 -- 172.21.15.26:0/4094998873 --> v1:172.21.15.26:6791/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa2f4005c40 con 0x7fa31013f2e0 2024-08-19T12:56:06.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.271+0000 7fa3077fe640 1 -- 172.21.15.26:0/4094998873 <== mon.4 v1:172.21.15.26:6791/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fa308020100 con 0x7fa31013f2e0 2024-08-19T12:56:06.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.271+0000 7fa3077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-08-19T12:56:06.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.271+0000 7fa3077fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:06.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=11,kv=13,logm=43} crc {auth=836115394,config=1097430493,health=1105059601,kv=701615758,logm=1859434996}) 2024-08-19T12:56:06.273 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=11,kv=13,logm=44} crc {auth=3627660434,config=1097430493,health=2375187314,kv=1344821656,logm=3173183215}) 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=11,kv=13,logm=43} crc {auth=836115394,config=1097430493,health=1105059601,kv=701615758,logm=1859434996}) 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=11,kv=13,logm=44} crc {auth=3704299254,config=1097430493,health=3302566429,kv=1344821656,logm=1589602914}) 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=11,kv=13,logm=43} crc {auth=836115394,config=1097430493,health=1105059601,kv=701615758,logm=1859434996}) 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=30,config=2,health=11,kv=11,logm=46} crc {auth=3150518457,config=1097430493,health=68801358,kv=834118746,logm=1827122080}) 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=11,kv=13,logm=43} crc {auth=836115394,config=1097430493,health=1105059601,kv=701615758,logm=1859434996}) 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=27,config=2,health=12,kv=13,logm=46} crc {auth=2778351326,config=1097430493,health=3069894875,kv=1344821656,logm=1696422715}) 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=11,kv=13,logm=43} crc {auth=836115394,config=1097430493,health=1105059601,kv=701615758,logm=1859434996}) 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=29,config=2,health=11,kv=12,logm=46} crc {auth=1435603200,config=1097430493,health=4081214817,kv=956038015,logm=4239106057}) 2024-08-19T12:56:06.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=11,kv=13,logm=43} crc {auth=836115394,config=1097430493,health=1105059601,kv=701615758,logm=1859434996}) 2024-08-19T12:56:06.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=29,config=2,health=12,kv=13,logm=44} crc {auth=188658257,config=1097430493,health=3069894875,kv=1344821656,logm=1293648943}) 2024-08-19T12:56:06.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=11,kv=13,logm=43} crc {auth=836115394,config=1097430493,health=1105059601,kv=701615758,logm=1859434996}) 2024-08-19T12:56:06.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=31,config=2,health=11,kv=13,logm=43} crc {auth=836115394,config=1097430493,health=1738207343,kv=1344821656,logm=601341811}) 2024-08-19T12:56:06.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=11,kv=13,logm=43} crc {auth=836115394,config=1097430493,health=1105059601,kv=701615758,logm=1859434996}) 2024-08-19T12:56:06.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.272+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=30,config=2,health=11,kv=13,logm=44} crc {auth=39888249,config=1097430493,health=68801358,kv=1344821656,logm=2474235811}) 2024-08-19T12:56:06.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.274+0000 7fa31549f640 10 librados: watch_flush enter 2024-08-19T12:56:06.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.274+0000 7fa31549f640 10 librados: watch_flush exit 2024-08-19T12:56:06.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.274+0000 7fa31549f640 20 client.184702.objecter shutdown clearing up homeless session... 2024-08-19T12:56:06.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.274+0000 7fa31549f640 10 client.184702.objecter successfully canceled tick 2024-08-19T12:56:06.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.274+0000 7fa31549f640 1 -- 172.21.15.26:0/4094998873 >> v1:172.21.15.104:6800/899887506 conn(0x7fa2f0041960 legacy=0x7fa2f0043e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:06.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.274+0000 7fa31549f640 10 monclient: shutdown 2024-08-19T12:56:06.279 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.274+0000 7fa31549f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:06.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.274+0000 7fa31549f640 1 -- 172.21.15.26:0/4094998873 >> v1:172.21.15.26:6791/0 conn(0x7fa31013f2e0 legacy=0x7fa3101d6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:06.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.275+0000 7fa31549f640 1 -- 172.21.15.26:0/4094998873 shutdown_connections 2024-08-19T12:56:06.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.275+0000 7fa31549f640 1 -- 172.21.15.26:0/4094998873 >> 172.21.15.26:0/4094998873 conn(0x7fa3100abff0 msgr2=0x7fa310134a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:06.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.275+0000 7fa31549f640 1 -- 172.21.15.26:0/4094998873 shutdown_connections 2024-08-19T12:56:06.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.275+0000 7fa31549f640 1 -- 172.21.15.26:0/4094998873 wait complete. 2024-08-19T12:56:06.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:06.275+0000 7fa31549f640 1 librados: shutdown 2024-08-19T12:56:06.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288868568}) 2024-08-19T12:56:06.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2986014342}) 2024-08-19T12:56:06.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288868568}) 2024-08-19T12:56:06.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4272929714}) 2024-08-19T12:56:06.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288868568}) 2024-08-19T12:56:06.283 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=984476572}) 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288868568}) 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2075352996}) 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288868568}) 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=813948362}) 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288868568}) 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2250717769}) 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288868568}) 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4280089296}) 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.284 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288868568}) 2024-08-19T12:56:06.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2971910499}) 2024-08-19T12:56:06.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21511679}) 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3084419882}) 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21511679}) 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103043516}) 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21511679}) 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2184778011}) 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21511679}) 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2206141537}) 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21511679}) 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1878103228}) 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21511679}) 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=163007221}) 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21511679}) 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=530706204}) 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21511679}) 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=591686924}) 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038900180}) 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669057001}) 2024-08-19T12:56:06.291 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038900180}) 2024-08-19T12:56:06.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237807053}) 2024-08-19T12:56:06.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038900180}) 2024-08-19T12:56:06.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3899114013}) 2024-08-19T12:56:06.292 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038900180}) 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2408614834}) 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038900180}) 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3206507829}) 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038900180}) 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1720571391}) 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038900180}) 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2485180925}) 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.293 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038900180}) 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.285+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2430754110}) 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666000759}) 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3938382694}) 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666000759}) 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=654771568}) 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666000759}) 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3686742390}) 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666000759}) 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3416499431}) 2024-08-19T12:56:06.294 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.295 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666000759}) 2024-08-19T12:56:06.295 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1738520839}) 2024-08-19T12:56:06.295 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.295 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666000759}) 2024-08-19T12:56:06.295 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3838059047}) 2024-08-19T12:56:06.295 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.295 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666000759}) 2024-08-19T12:56:06.295 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4185318271}) 2024-08-19T12:56:06.295 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.295 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666000759}) 2024-08-19T12:56:06.295 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.287+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1907610100}) 2024-08-19T12:56:06.297 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-08-19T12:56:06.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384071679}) 2024-08-19T12:56:06.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2360716649}) 2024-08-19T12:56:06.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384071679}) 2024-08-19T12:56:06.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734949252}) 2024-08-19T12:56:06.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384071679}) 2024-08-19T12:56:06.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2345752165}) 2024-08-19T12:56:06.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384071679}) 2024-08-19T12:56:06.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3125769943}) 2024-08-19T12:56:06.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384071679}) 2024-08-19T12:56:06.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1633432979}) 2024-08-19T12:56:06.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.314 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384071679}) 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2785156158}) 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384071679}) 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3774184270}) 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384071679}) 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.289+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4197561797}) 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283591152}) 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1741651816}) 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283591152}) 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3508606256}) 2024-08-19T12:56:06.315 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283591152}) 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=365867765}) 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283591152}) 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3051336001}) 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283591152}) 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2632897475}) 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283591152}) 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2110785359}) 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283591152}) 2024-08-19T12:56:06.316 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1043233054}) 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283591152}) 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.290+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=217959058}) 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939973605}) 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1963715642}) 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939973605}) 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3935134185}) 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939973605}) 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3944031732}) 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.317 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939973605}) 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=433389058}) 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939973605}) 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1734441614}) 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939973605}) 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=464670465}) 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939973605}) 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=793397788}) 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939973605}) 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.291+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1522932517}) 2024-08-19T12:56:06.318 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320308688}) 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3069332630}) 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320308688}) 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=196815281}) 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320308688}) 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2473543999}) 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320308688}) 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2217354654}) 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320308688}) 2024-08-19T12:56:06.319 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3121111317}) 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320308688}) 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2663794332}) 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320308688}) 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3846018830}) 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320308688}) 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=114925368}) 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784781275}) 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=342057076}) 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.320 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784781275}) 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3783309267}) 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784781275}) 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1711955206}) 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784781275}) 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1961634884}) 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784781275}) 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=900934053}) 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784781275}) 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3819752150}) 2024-08-19T12:56:06.321 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784781275}) 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1999193799}) 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784781275}) 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2778031987}) 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707624403}) 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755763255}) 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707624403}) 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2492047659}) 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.322 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707624403}) 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3516239235}) 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707624403}) 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3666694457}) 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707624403}) 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1864743282}) 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707624403}) 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=945481475}) 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707624403}) 2024-08-19T12:56:06.323 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3821837652}) 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707624403}) 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2169414234}) 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414398697}) 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3289105778}) 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414398697}) 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=60490206}) 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414398697}) 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2319496844}) 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.324 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414398697}) 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2357269697}) 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414398697}) 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2668234720}) 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414398697}) 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1413002472}) 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414398697}) 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=488778330}) 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414398697}) 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2289408459}) 2024-08-19T12:56:06.325 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590560509}) 2024-08-19T12:56:06.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2060931510}) 2024-08-19T12:56:06.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590560509}) 2024-08-19T12:56:06.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=65815169}) 2024-08-19T12:56:06.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590560509}) 2024-08-19T12:56:06.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3924245390}) 2024-08-19T12:56:06.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.326 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590560509}) 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2340120413}) 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590560509}) 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3844832860}) 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590560509}) 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3589124085}) 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590560509}) 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3391453814}) 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590560509}) 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2781215471}) 2024-08-19T12:56:06.327 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676442933}) 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2654377308}) 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676442933}) 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2165767632}) 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676442933}) 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=355339048}) 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676442933}) 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4282639179}) 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.328 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676442933}) 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3012810578}) 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676442933}) 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3537499986}) 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676442933}) 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2118254837}) 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676442933}) 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.300+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4163610817}) 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425034186}) 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=698137707}) 2024-08-19T12:56:06.329 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425034186}) 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2451342743}) 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425034186}) 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=166367696}) 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425034186}) 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2613550127}) 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425034186}) 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=504388968}) 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425034186}) 2024-08-19T12:56:06.330 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2472232389}) 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425034186}) 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1727210261}) 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425034186}) 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1878324363}) 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563866429}) 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1571223975}) 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563866429}) 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1020718783}) 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.331 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563866429}) 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2927037369}) 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563866429}) 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2684431051}) 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563866429}) 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=864746909}) 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563866429}) 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2588784768}) 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563866429}) 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3624058577}) 2024-08-19T12:56:06.332 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563866429}) 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3606444986}) 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263408918}) 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2530552534}) 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263408918}) 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3554426243}) 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263408918}) 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=126903276}) 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.333 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263408918}) 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=267393389}) 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263408918}) 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2567111454}) 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263408918}) 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1041771383}) 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263408918}) 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3549129454}) 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263408918}) 2024-08-19T12:56:06.334 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1100188947}) 2024-08-19T12:56:06.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134414763}) 2024-08-19T12:56:06.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2198085253}) 2024-08-19T12:56:06.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134414763}) 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2329254601}) 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134414763}) 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3851558845}) 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134414763}) 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=955487041}) 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134414763}) 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2634954735}) 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134414763}) 2024-08-19T12:56:06.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3408054760}) 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134414763}) 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1821230167}) 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134414763}) 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.315+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4265478362}) 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320693697}) 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1480312030}) 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320693697}) 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045367648}) 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320693697}) 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2528363197}) 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320693697}) 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3995681417}) 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320693697}) 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=841789327}) 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320693697}) 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2686203662}) 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320693697}) 2024-08-19T12:56:06.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2115208895}) 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320693697}) 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.317+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1006034135}) 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280349394}) 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1768764721}) 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280349394}) 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3607978372}) 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280349394}) 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3516224219}) 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280349394}) 2024-08-19T12:56:06.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=689642611}) 2024-08-19T12:56:06.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280349394}) 2024-08-19T12:56:06.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=393461840}) 2024-08-19T12:56:06.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280349394}) 2024-08-19T12:56:06.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1956566404}) 2024-08-19T12:56:06.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280349394}) 2024-08-19T12:56:06.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=298907799}) 2024-08-19T12:56:06.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280349394}) 2024-08-19T12:56:06.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.318+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1080804454}) 2024-08-19T12:56:06.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606968355}) 2024-08-19T12:56:06.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=810465221}) 2024-08-19T12:56:06.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606968355}) 2024-08-19T12:56:06.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1754375945}) 2024-08-19T12:56:06.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606968355}) 2024-08-19T12:56:06.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=176819811}) 2024-08-19T12:56:06.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606968355}) 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=569644073}) 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606968355}) 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3475946648}) 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606968355}) 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=527927406}) 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606968355}) 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2360504375}) 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606968355}) 2024-08-19T12:56:06.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.319+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3026940619}) 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087077558}) 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2468366224}) 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087077558}) 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909536307}) 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087077558}) 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1206300424}) 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087077558}) 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2308924974}) 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087077558}) 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1358525537}) 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087077558}) 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2105593401}) 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087077558}) 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=208741884}) 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087077558}) 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.321+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3662469961}) 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558516420}) 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=616217496}) 2024-08-19T12:56:06.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558516420}) 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536872918}) 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558516420}) 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2032554852}) 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558516420}) 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3578982150}) 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558516420}) 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1372836632}) 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558516420}) 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3971388268}) 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558516420}) 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2793565234}) 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558516420}) 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2279267489}) 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011908994}) 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=705496489}) 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011908994}) 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3683127051}) 2024-08-19T12:56:06.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011908994}) 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2068465952}) 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011908994}) 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=939436481}) 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011908994}) 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2091477362}) 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011908994}) 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1468834700}) 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011908994}) 2024-08-19T12:56:06.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=625080442}) 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011908994}) 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1973319348}) 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174952564}) 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2161205593}) 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174952564}) 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=743647302}) 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174952564}) 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3490177144}) 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174952564}) 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1028474063}) 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174952564}) 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2859102521}) 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174952564}) 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.325+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1408992605}) 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.325+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.325+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174952564}) 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.325+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=926159837}) 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.325+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.325+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174952564}) 2024-08-19T12:56:06.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.325+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4268253522}) 2024-08-19T12:56:06.843 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.843 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022346968}) 2024-08-19T12:56:06.843 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2356982538}) 2024-08-19T12:56:06.843 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.843 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022346968}) 2024-08-19T12:56:06.843 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3136207774}) 2024-08-19T12:56:06.843 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.843 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022346968}) 2024-08-19T12:56:06.843 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2437560807}) 2024-08-19T12:56:06.843 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022346968}) 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3903114346}) 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022346968}) 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=166831442}) 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022346968}) 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=170612918}) 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022346968}) 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1216221579}) 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.844 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022346968}) 2024-08-19T12:56:06.845 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.840+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=560792091}) 2024-08-19T12:56:06.847 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.847 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511866155}) 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1152125041}) 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511866155}) 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=471765487}) 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511866155}) 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=946438909}) 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511866155}) 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2157475635}) 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.845+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511866155}) 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.846+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=425299133}) 2024-08-19T12:56:06.848 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.846+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.849 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.846+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511866155}) 2024-08-19T12:56:06.849 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.846+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4021187816}) 2024-08-19T12:56:06.849 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.846+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.849 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.846+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511866155}) 2024-08-19T12:56:06.849 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.846+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=590646825}) 2024-08-19T12:56:06.849 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.846+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.849 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.846+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511866155}) 2024-08-19T12:56:06.849 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.846+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2560631812}) 2024-08-19T12:56:06.854 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.854 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665200709}) 2024-08-19T12:56:06.854 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2800531100}) 2024-08-19T12:56:06.854 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.854 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665200709}) 2024-08-19T12:56:06.854 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1074206900}) 2024-08-19T12:56:06.854 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665200709}) 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3813445360}) 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665200709}) 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3362305992}) 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665200709}) 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=264020478}) 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665200709}) 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1920641635}) 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665200709}) 2024-08-19T12:56:06.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3574497832}) 2024-08-19T12:56:06.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=665200709}) 2024-08-19T12:56:06.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.852+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1772068988}) 2024-08-19T12:56:06.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94687963}) 2024-08-19T12:56:06.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2665476730}) 2024-08-19T12:56:06.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94687963}) 2024-08-19T12:56:06.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3176746576}) 2024-08-19T12:56:06.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94687963}) 2024-08-19T12:56:06.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=395136242}) 2024-08-19T12:56:06.922 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94687963}) 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3138254517}) 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94687963}) 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2726503134}) 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94687963}) 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2746990866}) 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.919+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94687963}) 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.920+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2206063722}) 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.920+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.923 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.920+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94687963}) 2024-08-19T12:56:06.924 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.920+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3987915208}) 2024-08-19T12:56:06.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286206791}) 2024-08-19T12:56:06.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2112530368}) 2024-08-19T12:56:06.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286206791}) 2024-08-19T12:56:06.941 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3186307604}) 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286206791}) 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3362292938}) 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286206791}) 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3385955675}) 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286206791}) 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=683868077}) 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.937+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.938+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286206791}) 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.938+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2700515167}) 2024-08-19T12:56:06.942 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.938+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.938+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286206791}) 2024-08-19T12:56:06.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.938+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1070284817}) 2024-08-19T12:56:06.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.938+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:06.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.938+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286206791}) 2024-08-19T12:56:06.943 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:06.938+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1031457415}) 2024-08-19T12:56:07.159 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.157+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.159 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.157+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606047771}) 2024-08-19T12:56:07.159 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.157+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2350766330}) 2024-08-19T12:56:07.159 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.157+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.157+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606047771}) 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.157+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163515892}) 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.157+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.157+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606047771}) 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2923570593}) 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606047771}) 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=277954968}) 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606047771}) 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1034308119}) 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.160 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606047771}) 2024-08-19T12:56:07.161 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=247281231}) 2024-08-19T12:56:07.161 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.161 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606047771}) 2024-08-19T12:56:07.161 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1268690729}) 2024-08-19T12:56:07.161 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.161 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606047771}) 2024-08-19T12:56:07.161 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.158+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3316206963}) 2024-08-19T12:56:07.298 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:56:07.298 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:56:07.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.406+0000 7ffbdd16b640 10 monclient: get_monmap_and_config 2024-08-19T12:56:07.406 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.406+0000 7ffbdd16b640 10 monclient: build_initial_monmap 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.406+0000 7ffbdd16b640 10 monclient: monmap: 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:07.406992+0000 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:07.406992+0000 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:07.407 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:07.408 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:07.408 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:07.408 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.406+0000 7ffbdd16b640 1 Processor -- start 2024-08-19T12:56:07.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.406+0000 7ffbdd16b640 1 -- start start 2024-08-19T12:56:07.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.406+0000 7ffbdd16b640 10 monclient: init 2024-08-19T12:56:07.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.407+0000 7ffbdd16b640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:07.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.407+0000 7ffbdd16b640 10 monclient: _add_conns ranks=[4,1,7,3,6,0,2,5,8] 2024-08-19T12:56:07.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.407+0000 7ffbdd16b640 10 monclient(hunting): picked mon.noname-c con 0x7ffbd8073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:07.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.407+0000 7ffbdd16b640 10 monclient(hunting): picked mon.noname-f con 0x7ffbd80a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:07.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.407+0000 7ffbdd16b640 10 monclient(hunting): picked mon.noname-i con 0x7ffbd813f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:07.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.407+0000 7ffbdd16b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd809dae0 con 0x7ffbd80a57a0 2024-08-19T12:56:07.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.407+0000 7ffbdd16b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd8136910 con 0x7ffbd8073470 2024-08-19T12:56:07.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.407+0000 7ffbdd16b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd809dd50 con 0x7ffbd813f2e0 2024-08-19T12:56:07.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.407+0000 7ffbdd16b640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:07.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.407+0000 7ffbdd16b640 10 monclient(hunting): authenticate will time out at 1761.494400s 2024-08-19T12:56:07.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.407+0000 7ffbd6d9d640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7ffbd8073470 0x7ffbd8136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:60016/0 (socket says 172.21.15.26:60016) 2024-08-19T12:56:07.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.407+0000 7ffbd6d9d640 1 -- 172.21.15.26:0/1989654502 learned_addr learned my addr 172.21.15.26:0/1989654502 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:07.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd6d9d640 1 -- 172.21.15.26:0/1989654502 >> v1:172.21.15.26:6791/0 conn(0x7ffbd8073470 legacy=0x7ffbd8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd759e640 1 -- 172.21.15.26:0/1989654502 >> v1:172.21.15.104:6792/0 conn(0x7ffbd813f2e0 legacy=0x7ffbd81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbce59c640 1 -- 172.21.15.26:0/1989654502 >> v1:172.21.15.104:6789/0 conn(0x7ffbd80a57a0 legacy=0x7ffbd80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2560871690 0 0) 0x7ffbd8136910 con 0x7ffbd8073470 2024-08-19T12:56:07.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.414 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 10 monclient(hunting): my global_id is 184711 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbc4003520 con 0x7ffbd8073470 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 445637796 0 0) 0x7ffbd809dd50 con 0x7ffbd813f2e0 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 10 monclient(hunting): my global_id is 136339 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbd8136910 con 0x7ffbd813f2e0 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3892640128 0 0) 0x7ffbd809dae0 con 0x7ffbd80a57a0 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.415 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 10 monclient(hunting): my global_id is 204193 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.409+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbd809dd50 con 0x7ffbd80a57a0 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2611218137 0 0) 0x7ffbc4003520 con 0x7ffbd8073470 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 >> v1:172.21.15.104:6792/0 conn(0x7ffbd813f2e0 legacy=0x7ffbd81416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 >> v1:172.21.15.104:6789/0 conn(0x7ffbd80a57a0 legacy=0x7ffbd80a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 10 monclient: _finish_hunting 0 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 1 monclient: found mon.noname-c 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd809e030 con 0x7ffbd8073470 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 10 monclient: _finish_auth 0 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 10 monclient: _check_auth_tickets 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:07.416 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbdd16b640 5 monclient: authenticate success, global_id 184711 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbdd16b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffbc8001ae0 con 0x7ffbd8073470 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 10 monclient: dump: 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:07.417 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbdd16b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffbc80048a0 con 0x7ffbd8073470 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 1 -- 172.21.15.26:0/1989654502 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffbc8004e10 con 0x7ffbd8073470 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:07.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbd659c640 10 monclient: dump: 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:07.419 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbdd16b640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbdd16b640 4 set_mon_vals no callback set 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbdd16b640 10 monclient: shutdown 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbdd16b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbdd16b640 1 -- 172.21.15.26:0/1989654502 >> v1:172.21.15.26:6791/0 conn(0x7ffbd8073470 legacy=0x7ffbd8136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbdd16b640 1 -- 172.21.15.26:0/1989654502 shutdown_connections 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbdd16b640 1 -- 172.21.15.26:0/1989654502 >> 172.21.15.26:0/1989654502 conn(0x7ffbd80abff0 msgr2=0x7ffbd80ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbdd16b640 1 -- 172.21.15.26:0/1989654502 shutdown_connections 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.410+0000 7ffbdd16b640 1 -- 172.21.15.26:0/1989654502 wait complete. 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.411+0000 7ffbdd16b640 10 monclient: build_initial_monmap 2024-08-19T12:56:07.420 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.411+0000 7ffbdd16b640 10 monclient: monmap: 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:07.421 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.411+0000 7ffbdd16b640 1 librados: starting msgr at 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.411+0000 7ffbdd16b640 1 librados: starting objecter 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.411+0000 7ffbdd16b640 1 Processor -- start 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 1 -- start start 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 1 librados: setting wanted keys 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 1 librados: calling monclient init 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 10 monclient: init 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 10 monclient: _add_conns ranks=[7,4,1,0,6,8,3,2,5] 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 10 monclient(hunting): picked mon.noname-i con 0x7ffbd8073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:07.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 10 monclient(hunting): picked mon.noname-c con 0x7ffbd80a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:07.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 10 monclient(hunting): picked mon.noname-f con 0x7ffbd813f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:07.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd812f810 con 0x7ffbd813f2e0 2024-08-19T12:56:07.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd8135290 con 0x7ffbd80a57a0 2024-08-19T12:56:07.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd8131b70 con 0x7ffbd8073470 2024-08-19T12:56:07.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:07.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbdd16b640 10 monclient(hunting): authenticate will time out at 1761.499586s 2024-08-19T12:56:07.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbce59c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7ffbd80a57a0 0x7ffbd812e4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:60026/0 (socket says 172.21.15.26:60026) 2024-08-19T12:56:07.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.412+0000 7ffbce59c640 1 -- 172.21.15.26:0/2659920271 learned_addr learned my addr 172.21.15.26:0/2659920271 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:07.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbce59c640 1 -- 172.21.15.26:0/2659920271 >> v1:172.21.15.26:6791/0 conn(0x7ffbd80a57a0 legacy=0x7ffbd812e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbcffff640 10 client.?.objecter ms_handle_connect 0x7ffbd80a57a0 2024-08-19T12:56:07.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbcffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:07.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbcffff640 10 client.?.objecter ms_handle_connect 0x7ffbd8073470 2024-08-19T12:56:07.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbcffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:07.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbd6d9d640 1 -- 172.21.15.26:0/2659920271 >> v1:172.21.15.104:6792/0 conn(0x7ffbd8073470 legacy=0x7ffbd812dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbd759e640 1 -- 172.21.15.26:0/2659920271 >> v1:172.21.15.104:6789/0 conn(0x7ffbd813f2e0 legacy=0x7ffbd81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbcffff640 10 client.?.objecter ms_handle_connect 0x7ffbd813f2e0 2024-08-19T12:56:07.428 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbcffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:07.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2410266429 0 0) 0x7ffbc0001280 con 0x7ffbd80a57a0 2024-08-19T12:56:07.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbcffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbcffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbcffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbcffff640 10 monclient(hunting): my global_id is 184720 2024-08-19T12:56:07.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.414+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbb8004eb0 con 0x7ffbd80a57a0 2024-08-19T12:56:07.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1668421292 0 0) 0x7ffbd8131b70 con 0x7ffbd8073470 2024-08-19T12:56:07.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): my global_id is 136348 2024-08-19T12:56:07.429 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbc0001280 con 0x7ffbd8073470 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1865686154 0 0) 0x7ffbd812f810 con 0x7ffbd813f2e0 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): my global_id is 204202 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbd8131b70 con 0x7ffbd813f2e0 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1544759001 0 0) 0x7ffbb8004eb0 con 0x7ffbd80a57a0 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffbd812f810 con 0x7ffbd80a57a0 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffbc0003950 con 0x7ffbd80a57a0 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): dump: 2024-08-19T12:56:07.430 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:07.431 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3089696036 0 0) 0x7ffbc0001280 con 0x7ffbd8073470 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffbb8004eb0 con 0x7ffbd8073470 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffbc8004d90 con 0x7ffbd8073470 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): dump: 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.432 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 602783174 0 0) 0x7ffbd8131b70 con 0x7ffbd813f2e0 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffbc0001280 con 0x7ffbd813f2e0 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffbd0001f20 con 0x7ffbd813f2e0 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:07.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient(hunting): dump: 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:07.434 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2638527560 0 0) 0x7ffbb8004eb0 con 0x7ffbd8073470 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 >> v1:172.21.15.26:6791/0 conn(0x7ffbd80a57a0 legacy=0x7ffbd812e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 >> v1:172.21.15.104:6789/0 conn(0x7ffbd813f2e0 legacy=0x7ffbd81d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient: _finish_hunting 0 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 monclient: found mon.i 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd806d8f0 con 0x7ffbd8073470 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient: _finish_auth 0 2024-08-19T12:56:07.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient: _check_auth_tickets 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbdd16b640 5 monclient: authenticate success, global_id 136348 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbdd16b640 10 monclient: _renew_subs 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbdd16b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbdd16b640 1 -- 172.21.15.26:0/2659920271 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd8131890 con 0x7ffbd8073470 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbdd16b640 10 client.136348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbdd16b640 10 monclient: _renew_subs 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbdd16b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbdd16b640 1 -- 172.21.15.26:0/2659920271 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7ffbd8132300 con 0x7ffbd8073470 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbdd16b640 1 librados: init done 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffbc8001d30 con 0x7ffbd8073470 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7ffbc8001b00 con 0x7ffbd8073470 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:07.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbd5d9b640 4 set_mon_vals no callback set 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.415+0000 7ffbcffff640 10 monclient: dump: 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:07.437 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.416+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7ffbc8011a00 con 0x7ffbd8073470 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.416+0000 7ffbdd16b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.416+0000 7ffbdd16b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.416+0000 7ffbdd16b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.416+0000 7ffbdd16b640 1 -- 172.21.15.26:0/2659920271 --> v1:172.21.15.104:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbbc005080 con 0x7ffbd8073470 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.416+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(656..656 src has 350..656) ==== 5939+0+0 (unknown 1913693553 0 0) 0x7ffbc804e2b0 con 0x7ffbd8073470 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.416+0000 7ffbcffff640 10 client.136348.objecter ms_dispatch 0x7ffbd8000b90 osd_map(656..656 src has 350..656) 2024-08-19T12:56:07.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.416+0000 7ffbcffff640 3 client.136348.objecter handle_osd_map got epochs [656,656] > 0 2024-08-19T12:56:07.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.416+0000 7ffbcffff640 3 client.136348.objecter handle_osd_map decoding full epoch 656 2024-08-19T12:56:07.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.416+0000 7ffbcffff640 20 client.136348.objecter dump_active .. 0 homeless 2024-08-19T12:56:07.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.416+0000 7ffbce59c640 10 monclient: get_auth_request con 0x7ffbb80418c0 auth_method 0 2024-08-19T12:56:07.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.419+0000 7ffbcffff640 10 client.136348.objecter ms_handle_connect 0x7ffbb80418c0 2024-08-19T12:56:07.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.421+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.7 v1:172.21.15.104:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7ffbc8017530 con 0x7ffbd8073470 2024-08-19T12:56:07.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.421+0000 7ffbcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:07.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.421+0000 7ffbcffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:07.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.681+0000 7ffbdd16b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:56:07.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.681+0000 7ffbdd16b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:56:07.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.681+0000 7ffbdd16b640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:56:07.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.681+0000 7ffbdd16b640 1 -- 172.21.15.26:0/2659920271 --> v1:172.21.15.104:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffbbc005370 con 0x7ffbd8073470 2024-08-19T12:56:07.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.681+0000 7ffbcffff640 1 -- 172.21.15.26:0/2659920271 <== mon.7 v1:172.21.15.104:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2730 (unknown 254746657 0 3291668920) 0x7ffbc8016e30 con 0x7ffbd8073470 2024-08-19T12:56:07.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.681+0000 7ffbcffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:56:07.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.681+0000 7ffbcffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:07.682 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":124,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:56:07.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.683+0000 7ffbdd16b640 10 librados: watch_flush enter 2024-08-19T12:56:07.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.684+0000 7ffbdd16b640 10 librados: watch_flush exit 2024-08-19T12:56:07.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.684+0000 7ffbdd16b640 20 client.136348.objecter shutdown clearing up homeless session... 2024-08-19T12:56:07.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.684+0000 7ffbdd16b640 10 client.136348.objecter successfully canceled tick 2024-08-19T12:56:07.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.684+0000 7ffbdd16b640 1 -- 172.21.15.26:0/2659920271 >> v1:172.21.15.104:6800/899887506 conn(0x7ffbb80418c0 legacy=0x7ffbb8043d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.684+0000 7ffbdd16b640 10 monclient: shutdown 2024-08-19T12:56:07.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.684+0000 7ffbdd16b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:07.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.684+0000 7ffbdd16b640 1 -- 172.21.15.26:0/2659920271 >> v1:172.21.15.104:6792/0 conn(0x7ffbd8073470 legacy=0x7ffbd812dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.684+0000 7ffbdd16b640 1 -- 172.21.15.26:0/2659920271 shutdown_connections 2024-08-19T12:56:07.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.684+0000 7ffbdd16b640 1 -- 172.21.15.26:0/2659920271 >> 172.21.15.26:0/2659920271 conn(0x7ffbd80abff0 msgr2=0x7ffbd81354a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:07.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.684+0000 7ffbdd16b640 1 -- 172.21.15.26:0/2659920271 shutdown_connections 2024-08-19T12:56:07.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.684+0000 7ffbdd16b640 1 -- 172.21.15.26:0/2659920271 wait complete. 2024-08-19T12:56:07.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.684+0000 7ffbdd16b640 1 librados: shutdown 2024-08-19T12:56:07.695 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-08-19T12:56:07.696 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-08-19T12:56:07.696 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-08-19T12:56:07.696 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-08-19T12:56:07.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432789713}) 2024-08-19T12:56:07.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3350437260}) 2024-08-19T12:56:07.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432789713}) 2024-08-19T12:56:07.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3930377879}) 2024-08-19T12:56:07.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432789713}) 2024-08-19T12:56:07.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2817152376}) 2024-08-19T12:56:07.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432789713}) 2024-08-19T12:56:07.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3983482689}) 2024-08-19T12:56:07.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432789713}) 2024-08-19T12:56:07.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=846408199}) 2024-08-19T12:56:07.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432789713}) 2024-08-19T12:56:07.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2895742975}) 2024-08-19T12:56:07.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432789713}) 2024-08-19T12:56:07.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1354340318}) 2024-08-19T12:56:07.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432789713}) 2024-08-19T12:56:07.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.702+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2346654348}) 2024-08-19T12:56:07.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.796+0000 7f37e3508640 10 monclient: get_monmap_and_config 2024-08-19T12:56:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.796+0000 7f37e3508640 10 monclient: build_initial_monmap 2024-08-19T12:56:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.796+0000 7f37e3508640 10 monclient: monmap: 2024-08-19T12:56:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:07.796943+0000 2024-08-19T12:56:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:07.796943+0000 2024-08-19T12:56:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:07.797 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:07.798 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:07.798 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:07.798 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:07.798 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:07.798 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:07.798 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:07.798 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.796+0000 7f37e3508640 1 Processor -- start 2024-08-19T12:56:07.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.796+0000 7f37e3508640 1 -- start start 2024-08-19T12:56:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.796+0000 7f37e3508640 10 monclient: init 2024-08-19T12:56:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.797+0000 7f37e3508640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.797+0000 7f37e3508640 10 monclient: _add_conns ranks=[4,2,5,7,3,6,8,0,1] 2024-08-19T12:56:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.797+0000 7f37e3508640 10 monclient(hunting): picked mon.noname-c con 0x7f37dc073470 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.797+0000 7f37e3508640 10 monclient(hunting): picked mon.noname-b con 0x7f37dc0a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:56:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.797+0000 7f37e3508640 10 monclient(hunting): picked mon.noname-h con 0x7f37dc13f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.797+0000 7f37e3508640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37dc09dae0 con 0x7f37dc0a57a0 2024-08-19T12:56:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.797+0000 7f37e3508640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37dc136910 con 0x7f37dc073470 2024-08-19T12:56:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.797+0000 7f37e3508640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37dc09dd50 con 0x7f37dc13f2e0 2024-08-19T12:56:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.797+0000 7f37e3508640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:07.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.797+0000 7f37e3508640 10 monclient(hunting): authenticate will time out at 1761.884377s 2024-08-19T12:56:07.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.797+0000 7f37e12a4640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f37dc073470 0x7f37dc136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:60030/0 (socket says 172.21.15.26:60030) 2024-08-19T12:56:07.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.797+0000 7f37e12a4640 1 -- 172.21.15.26:0/152925543 learned_addr learned my addr 172.21.15.26:0/152925543 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:07.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37e0aa3640 1 -- 172.21.15.26:0/152925543 >> v1:172.21.15.26:6790/0 conn(0x7f37dc0a57a0 legacy=0x7f37dc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37e12a4640 1 -- 172.21.15.26:0/152925543 >> v1:172.21.15.26:6791/0 conn(0x7f37dc073470 legacy=0x7f37dc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2661704531 0 0) 0x7f37dc09dae0 con 0x7f37dc0a57a0 2024-08-19T12:56:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37dbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37dbfff640 10 monclient(hunting): my global_id is 27380 2024-08-19T12:56:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37e1aa5640 1 -- 172.21.15.26:0/152925543 >> v1:172.21.15.104:6791/0 conn(0x7f37dc13f2e0 legacy=0x7f37dc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37c4003520 con 0x7f37dc0a57a0 2024-08-19T12:56:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1902728015 0 0) 0x7f37dc136910 con 0x7f37dc073470 2024-08-19T12:56:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37dbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37dbfff640 10 monclient(hunting): my global_id is 184729 2024-08-19T12:56:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.799+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37dc09dae0 con 0x7f37dc073470 2024-08-19T12:56:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3967885958 0 0) 0x7f37dc09dd50 con 0x7f37dc13f2e0 2024-08-19T12:56:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient(hunting): my global_id is 184622 2024-08-19T12:56:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37dc136910 con 0x7f37dc13f2e0 2024-08-19T12:56:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2854642842 0 0) 0x7f37dc09dae0 con 0x7f37dc073470 2024-08-19T12:56:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 >> v1:172.21.15.104:6791/0 conn(0x7f37dc13f2e0 legacy=0x7f37dc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 >> v1:172.21.15.26:6790/0 conn(0x7f37dc0a57a0 legacy=0x7f37dc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient: _finish_hunting 0 2024-08-19T12:56:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 1 monclient: found mon.noname-c 2024-08-19T12:56:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:56:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37dc09e030 con 0x7f37dc073470 2024-08-19T12:56:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient: _finish_auth 0 2024-08-19T12:56:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient: _check_auth_tickets 2024-08-19T12:56:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f37d4001a40 con 0x7f37dc073470 2024-08-19T12:56:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37e3508640 5 monclient: authenticate success, global_id 184729 2024-08-19T12:56:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37e3508640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:56:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient: dump: 2024-08-19T12:56:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:07.814 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:07.815 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37e3508640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f37d4001d10 con 0x7f37dc073470 2024-08-19T12:56:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:07.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 1 -- 172.21.15.26:0/152925543 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f37d4004be0 con 0x7f37dc073470 2024-08-19T12:56:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37dbfff640 10 monclient: dump: 2024-08-19T12:56:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:07.817 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37e3508640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37e3508640 4 set_mon_vals no callback set 2024-08-19T12:56:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37e3508640 10 monclient: shutdown 2024-08-19T12:56:07.817 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37e3508640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37e3508640 1 -- 172.21.15.26:0/152925543 >> v1:172.21.15.26:6791/0 conn(0x7f37dc073470 legacy=0x7f37dc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37e3508640 1 -- 172.21.15.26:0/152925543 shutdown_connections 2024-08-19T12:56:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37e3508640 1 -- 172.21.15.26:0/152925543 >> 172.21.15.26:0/152925543 conn(0x7f37dc0abff0 msgr2=0x7f37dc0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37e3508640 1 -- 172.21.15.26:0/152925543 shutdown_connections 2024-08-19T12:56:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.800+0000 7f37e3508640 1 -- 172.21.15.26:0/152925543 wait complete. 2024-08-19T12:56:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.801+0000 7f37e3508640 10 monclient: build_initial_monmap 2024-08-19T12:56:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.801+0000 7f37e3508640 10 monclient: monmap: 2024-08-19T12:56:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:07.818 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:07.819 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:07.820 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.801+0000 7f37e3508640 1 librados: starting msgr at 2024-08-19T12:56:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.801+0000 7f37e3508640 1 librados: starting objecter 2024-08-19T12:56:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.801+0000 7f37e3508640 1 Processor -- start 2024-08-19T12:56:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.801+0000 7f37e3508640 1 -- start start 2024-08-19T12:56:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.801+0000 7f37e3508640 1 librados: setting wanted keys 2024-08-19T12:56:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.801+0000 7f37e3508640 1 librados: calling monclient init 2024-08-19T12:56:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.801+0000 7f37e3508640 10 monclient: init 2024-08-19T12:56:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.802+0000 7f37e3508640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:07.820 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.802+0000 7f37e3508640 10 monclient: _add_conns ranks=[1,7,8,4,3,6,0,5,2] 2024-08-19T12:56:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.802+0000 7f37e3508640 10 monclient(hunting): picked mon.noname-f con 0x7f37dc073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.802+0000 7f37e3508640 10 monclient(hunting): picked mon.noname-i con 0x7f37dc0a57a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.802+0000 7f37e3508640 10 monclient(hunting): picked mon.noname-e con 0x7f37dc13f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.802+0000 7f37e3508640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37dc12f810 con 0x7f37dc073470 2024-08-19T12:56:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.802+0000 7f37e3508640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37dc135290 con 0x7f37dc0a57a0 2024-08-19T12:56:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.802+0000 7f37e3508640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37dc131b70 con 0x7f37dc13f2e0 2024-08-19T12:56:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.802+0000 7f37e3508640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.802+0000 7f37e3508640 10 monclient(hunting): authenticate will time out at 1761.889369s 2024-08-19T12:56:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.802+0000 7f37e1aa5640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f37dc13f2e0 0x7f37dc1d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:58002/0 (socket says 172.21.15.26:58002) 2024-08-19T12:56:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.802+0000 7f37e1aa5640 1 -- 172.21.15.26:0/3509658697 learned_addr learned my addr 172.21.15.26:0/3509658697 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37e1aa5640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.26:6793/0 conn(0x7f37dc13f2e0 legacy=0x7f37dc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.821 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 client.?.objecter ms_handle_connect 0x7f37dc13f2e0 2024-08-19T12:56:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583462385 0 0) 0x7f37dc131b70 con 0x7f37dc13f2e0 2024-08-19T12:56:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 monclient(hunting): my global_id is 184553 2024-08-19T12:56:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37e12a4640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.104:6789/0 conn(0x7f37dc073470 legacy=0x7f37dc12dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37c8004eb0 con 0x7f37dc13f2e0 2024-08-19T12:56:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 client.?.objecter ms_handle_connect 0x7f37dc073470 2024-08-19T12:56:07.822 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 client.?.objecter ms_handle_connect 0x7f37dc0a57a0 2024-08-19T12:56:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37e0aa3640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.104:6792/0 conn(0x7f37dc0a57a0 legacy=0x7f37dc12e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2491080008 0 0) 0x7f37dc12f810 con 0x7f37dc073470 2024-08-19T12:56:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 10 monclient(hunting): my global_id is 204211 2024-08-19T12:56:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37dc131b70 con 0x7f37dc073470 2024-08-19T12:56:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.804+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 482175339 0 0) 0x7f37c8004eb0 con 0x7f37dc13f2e0 2024-08-19T12:56:07.823 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37dc12f810 con 0x7f37dc13f2e0 2024-08-19T12:56:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f37cc003990 con 0x7f37dc13f2e0 2024-08-19T12:56:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:56:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient(hunting): dump: 2024-08-19T12:56:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:07.824 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:07.825 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4133416163 0 0) 0x7f37d0001630 con 0x7f37dc0a57a0 2024-08-19T12:56:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient(hunting): my global_id is 136357 2024-08-19T12:56:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37c8004eb0 con 0x7f37dc0a57a0 2024-08-19T12:56:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 56489844 0 0) 0x7f37dc131b70 con 0x7f37dc073470 2024-08-19T12:56:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37d0001630 con 0x7f37dc073470 2024-08-19T12:56:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2496174725 0 0) 0x7f37dc12f810 con 0x7f37dc13f2e0 2024-08-19T12:56:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.104:6792/0 conn(0x7f37dc0a57a0 legacy=0x7f37dc12e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.104:6789/0 conn(0x7f37dc073470 legacy=0x7f37dc12dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:56:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 monclient: found mon.e 2024-08-19T12:56:07.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37dc06d8f0 con 0x7f37dc13f2e0 2024-08-19T12:56:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient: _finish_auth 0 2024-08-19T12:56:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:56:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37e3508640 5 monclient: authenticate success, global_id 184553 2024-08-19T12:56:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37e3508640 10 monclient: _renew_subs 2024-08-19T12:56:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37e3508640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37e3508640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f37dc131890 con 0x7f37dc13f2e0 2024-08-19T12:56:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f37cc003e90 con 0x7f37dc13f2e0 2024-08-19T12:56:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:07.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f37cc004400 con 0x7f37dc13f2e0 2024-08-19T12:56:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37d9ffb640 10 monclient: dump: 2024-08-19T12:56:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:07.828 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:07.829 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37e3508640 10 client.184553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37e3508640 10 monclient: _renew_subs 2024-08-19T12:56:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37e3508640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:07.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37e3508640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f37dc132300 con 0x7f37dc13f2e0 2024-08-19T12:56:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37daffd640 4 set_mon_vals no callback set 2024-08-19T12:56:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.805+0000 7f37e3508640 1 librados: init done 2024-08-19T12:56:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.806+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f37cc0045a0 con 0x7f37dc13f2e0 2024-08-19T12:56:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.806+0000 7f37e3508640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.806+0000 7f37e3508640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f37dc1313d0 con 0x7f37dc142bd0 2024-08-19T12:56:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.806+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(656..656 src has 350..656) ==== 5939+0+0 (unknown 1913693553 0 0) 0x7f37cc04e630 con 0x7f37dc13f2e0 2024-08-19T12:56:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.806+0000 7f37d9ffb640 10 client.184553.objecter ms_dispatch 0x7f37dc000b90 osd_map(656..656 src has 350..656) 2024-08-19T12:56:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.806+0000 7f37d9ffb640 3 client.184553.objecter handle_osd_map got epochs [656,656] > 0 2024-08-19T12:56:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.806+0000 7f37d9ffb640 3 client.184553.objecter handle_osd_map decoding full epoch 656 2024-08-19T12:56:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.806+0000 7f37e0aa3640 10 monclient: get_auth_request con 0x7f37c8041330 auth_method 0 2024-08-19T12:56:07.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.806+0000 7f37d9ffb640 20 client.184553.objecter dump_active .. 0 homeless 2024-08-19T12:56:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.808+0000 7f37e12a4640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.26:6789/0 conn(0x7f37dc142bd0 legacy=0x7f37dc13f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.808+0000 7f37d9ffb640 10 client.184553.objecter ms_handle_connect 0x7f37dc142bd0 2024-08-19T12:56:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.808+0000 7f37d9ffb640 10 client.184553.objecter resend_mon_ops 2024-08-19T12:56:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.808+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3661714434 0 0) 0x7f37dc1313d0 con 0x7f37dc142bd0 2024-08-19T12:56:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.808+0000 7f37d9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.808+0000 7f37d9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.808+0000 7f37d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.808+0000 7f37d9ffb640 10 monclient(hunting): my global_id is 194400 2024-08-19T12:56:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.808+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37dc12f810 con 0x7f37dc142bd0 2024-08-19T12:56:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.808+0000 7f37d9ffb640 10 client.184553.objecter ms_handle_connect 0x7f37c8041330 2024-08-19T12:56:07.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3546365036 0 0) 0x7f37dc12f810 con 0x7f37dc142bd0 2024-08-19T12:56:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37dc1313d0 con 0x7f37dc142bd0 2024-08-19T12:56:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f37d4001d10 con 0x7f37dc142bd0 2024-08-19T12:56:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 10 monclient: dump: 2024-08-19T12:56:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:07.833 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1986923637 0 0) 0x7f37dc1313d0 con 0x7f37dc142bd0 2024-08-19T12:56:07.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37dc0008d0 con 0x7f37dc142bd0 2024-08-19T12:56:07.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f37d4005220 con 0x7f37dc142bd0 2024-08-19T12:56:07.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:07.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:07.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.26:6789/0 conn(0x7f37dc142bd0 legacy=0x7f37dc13f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.809+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.26:6789/0 conn(0x7f37dc142bd0 legacy=0x7f37dc13f6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.815+0000 7f37e3508640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.815+0000 7f37e3508640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f37dc142fb0 con 0x7f37dc02e1d0 2024-08-19T12:56:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37e12a4640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.26:6789/0 conn(0x7f37dc02e1d0 legacy=0x7f37dc13fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 10 client.184553.objecter ms_handle_connect 0x7f37dc02e1d0 2024-08-19T12:56:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 10 client.184553.objecter resend_mon_ops 2024-08-19T12:56:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056811310 0 0) 0x7f37dc142fb0 con 0x7f37dc02e1d0 2024-08-19T12:56:07.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 10 monclient(hunting): my global_id is 194409 2024-08-19T12:56:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37dc1313d0 con 0x7f37dc02e1d0 2024-08-19T12:56:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2468038470 0 0) 0x7f37dc1313d0 con 0x7f37dc02e1d0 2024-08-19T12:56:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37dc142fb0 con 0x7f37dc02e1d0 2024-08-19T12:56:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f37d4005980 con 0x7f37dc02e1d0 2024-08-19T12:56:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 10 monclient: dump: 2024-08-19T12:56:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:07.837 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:07.838 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:07.839 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 843153817 0 0) 0x7f37dc142fb0 con 0x7f37dc02e1d0 2024-08-19T12:56:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.818+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 --> v1:172.21.15.26:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f37dc02e5b0 con 0x7f37dc02e1d0 2024-08-19T12:56:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 <== mon.0 v1:172.21.15.26:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7547 (unknown 0 0 175655199) 0x7f37d4007e30 con 0x7f37dc02e1d0 2024-08-19T12:56:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37d9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37d9ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.26:6789/0 conn(0x7f37dc02e1d0 legacy=0x7f37dc13fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37d9ffb640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.26:6789/0 conn(0x7f37dc02e1d0 legacy=0x7f37dc13fdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:07.840 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:07.840 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:07.840 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:07.840 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:56:07.840 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:07.840 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 152740, 2024-08-19T12:56:07.840 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:07.840 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:07.840 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:07.841 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:07.841 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:07.841 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:07.841 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:07.841 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:07.841 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:07.841 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:07.841 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:07.841 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:56:07.841 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:07.841 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:07.842 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:07.842 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:07.842 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:07.842 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:07.842 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:07.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:07.842 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:07.842 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:07.842 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:07.842 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:07.842 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:07.843 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:07.843 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:07.843 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:07.843 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:07.843 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:07.843 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:07.843 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:07.843 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:07.843 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:07.843 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:07.843 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:07.844 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:07.844 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:07.844 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:07.844 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:07.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:07.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.844 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:07.844 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:07.844 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:07.844 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:07.844 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:07.845 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:07.845 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:07.845 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:07.845 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:07.845 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:07.845 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:07.845 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:07.845 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:07.845 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:07.845 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:07.845 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:07.846 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:07.846 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:07.846 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:07.846 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:07.846 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:07.846 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:07.846 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:07.846 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:07.846 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:07.846 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:07.846 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:07.847 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:07.847 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:07.847 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:07.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.847 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:07.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.847 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:07.847 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:07.847 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:07.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:07.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:07.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:07.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:07.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:07.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:07.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:07.849 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:07.849 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:07.849 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:07.849 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:07.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.849 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:07.849 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:07.849 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:07.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:07.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:07.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:07.850 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:07.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:07.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:07.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:07.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:07.850 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:07.850 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:07.850 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:07.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.851 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:07.851 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:07.851 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:07.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:07.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.851 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:07.851 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:07.851 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:07.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:07.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:07.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:07.852 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:07.852 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:07.852 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:07.852 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:07.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.852 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:07.852 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:07.853 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:07.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:07.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:07.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:07.853 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:07.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:07.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:07.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:07.853 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:07.854 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:07.854 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:07.854 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:07.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.854 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:07.854 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:07.854 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:07.854 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:07.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.855 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:07.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:07.855 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:07.855 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:07.855 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:07.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:07.856 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:07.856 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:07.856 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:07.856 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:07.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.856 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:07.856 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:07.856 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:07.856 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:07.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.857 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:07.857 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:07.857 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:07.857 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:07.857 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:07.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.857 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:07.857 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:07.857 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:07.857 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:07.857 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:07.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.858 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:07.858 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:07.858 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:07.858 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:07.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:07.858 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:07.858 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:07.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:07.859 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:07.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.859 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:07.859 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:07.859 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:07.859 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:07.859 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:07.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.859 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:07.859 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:07.859 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:07.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:07.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.860 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:07.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:07.860 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:07.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:07.860 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:07.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:07.861 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:07.861 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:07.861 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:07.861 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:07.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.861 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:07.861 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:07.861 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:07.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:07.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.862 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:07.862 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:07.862 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:07.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:07.862 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:07.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.862 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:07.862 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:07.862 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:07.862 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:07.862 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:07.863 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:07.863 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:07.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.863 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:07.863 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:07.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.863 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:07.863 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:07.863 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:07.863 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:07.863 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:07.863 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:07.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:07.864 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:07.864 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:07.864 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:07.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:07.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:07.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:07.864 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:07.864 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:07.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 10 librados: watch_flush enter 2024-08-19T12:56:07.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 10 librados: watch_flush exit 2024-08-19T12:56:07.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 20 client.184553.objecter shutdown clearing up homeless session... 2024-08-19T12:56:07.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 10 client.184553.objecter successfully canceled tick 2024-08-19T12:56:07.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.104:6800/899887506 conn(0x7f37c8041330 legacy=0x7f37c80437f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 10 monclient: shutdown 2024-08-19T12:56:07.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:07.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 1 -- 172.21.15.26:0/3509658697 >> v1:172.21.15.26:6793/0 conn(0x7f37dc13f2e0 legacy=0x7f37dc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:07.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e1aa5640 1 -- 172.21.15.26:0/3509658697 reap_dead start 2024-08-19T12:56:07.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e1aa5640 1 -- 172.21.15.26:0/3509658697 reap_dead start 2024-08-19T12:56:07.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 1 -- 172.21.15.26:0/3509658697 shutdown_connections 2024-08-19T12:56:07.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 1 -- 172.21.15.26:0/3509658697 >> 172.21.15.26:0/3509658697 conn(0x7f37dc0abff0 msgr2=0x7f37dc1354a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:07.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 1 -- 172.21.15.26:0/3509658697 shutdown_connections 2024-08-19T12:56:07.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 1 -- 172.21.15.26:0/3509658697 wait complete. 2024-08-19T12:56:07.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:07.819+0000 7f37e3508640 1 librados: shutdown 2024-08-19T12:56:07.868 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:07.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-08-19T12:56:07.868 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:56:07.872 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.872 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218561154}) 2024-08-19T12:56:07.872 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2944502456}) 2024-08-19T12:56:07.872 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.872 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218561154}) 2024-08-19T12:56:07.872 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2019843074}) 2024-08-19T12:56:07.872 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.872 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218561154}) 2024-08-19T12:56:07.872 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1092023050}) 2024-08-19T12:56:07.872 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.873 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218561154}) 2024-08-19T12:56:07.873 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3612398870}) 2024-08-19T12:56:07.873 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.873 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218561154}) 2024-08-19T12:56:07.873 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1077246713}) 2024-08-19T12:56:07.873 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.873 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218561154}) 2024-08-19T12:56:07.873 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1059695379}) 2024-08-19T12:56:07.873 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.873 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218561154}) 2024-08-19T12:56:07.873 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1168030723}) 2024-08-19T12:56:07.874 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.874 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218561154}) 2024-08-19T12:56:07.874 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.851+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2064364607}) 2024-08-19T12:56:07.874 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.874 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060445145}) 2024-08-19T12:56:07.874 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3674857069}) 2024-08-19T12:56:07.874 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.874 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060445145}) 2024-08-19T12:56:07.874 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=97888274}) 2024-08-19T12:56:07.874 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.874 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060445145}) 2024-08-19T12:56:07.875 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3719408342}) 2024-08-19T12:56:07.875 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.875 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060445145}) 2024-08-19T12:56:07.875 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3107745581}) 2024-08-19T12:56:07.875 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.875 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060445145}) 2024-08-19T12:56:07.875 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2787314651}) 2024-08-19T12:56:07.875 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.875 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060445145}) 2024-08-19T12:56:07.875 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1314639332}) 2024-08-19T12:56:07.875 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.875 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060445145}) 2024-08-19T12:56:07.876 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3348604137}) 2024-08-19T12:56:07.876 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.876 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060445145}) 2024-08-19T12:56:07.876 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.855+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=49367035}) 2024-08-19T12:56:07.925 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.925 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4288534001}) 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=847599057}) 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4288534001}) 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3207338550}) 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4288534001}) 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=533414236}) 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4288534001}) 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2117317856}) 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4288534001}) 2024-08-19T12:56:07.926 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2177052860}) 2024-08-19T12:56:07.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4288534001}) 2024-08-19T12:56:07.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3139920028}) 2024-08-19T12:56:07.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4288534001}) 2024-08-19T12:56:07.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=149696395}) 2024-08-19T12:56:07.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4288534001}) 2024-08-19T12:56:07.927 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.923+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1125266242}) 2024-08-19T12:56:07.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042140533}) 2024-08-19T12:56:07.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3498594055}) 2024-08-19T12:56:07.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042140533}) 2024-08-19T12:56:07.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3662876330}) 2024-08-19T12:56:07.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042140533}) 2024-08-19T12:56:07.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3917136442}) 2024-08-19T12:56:07.930 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042140533}) 2024-08-19T12:56:07.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3458488352}) 2024-08-19T12:56:07.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042140533}) 2024-08-19T12:56:07.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3162515945}) 2024-08-19T12:56:07.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042140533}) 2024-08-19T12:56:07.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1569320279}) 2024-08-19T12:56:07.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042140533}) 2024-08-19T12:56:07.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=791309358}) 2024-08-19T12:56:07.931 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042140533}) 2024-08-19T12:56:07.932 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.928+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2172231003}) 2024-08-19T12:56:07.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243094894}) 2024-08-19T12:56:07.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1860539545}) 2024-08-19T12:56:07.950 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243094894}) 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2960809191}) 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243094894}) 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1899254086}) 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243094894}) 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2167223848}) 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243094894}) 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=995841663}) 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.951 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243094894}) 2024-08-19T12:56:07.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2439590071}) 2024-08-19T12:56:07.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243094894}) 2024-08-19T12:56:07.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1146169797}) 2024-08-19T12:56:07.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:07.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243094894}) 2024-08-19T12:56:07.952 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:07.947+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1897300016}) 2024-08-19T12:56:08.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.011+0000 7f5ac0fb1640 10 monclient: get_monmap_and_config 2024-08-19T12:56:08.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.011+0000 7f5ac0fb1640 10 monclient: build_initial_monmap 2024-08-19T12:56:08.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.011+0000 7f5ac0fb1640 10 monclient: monmap: 2024-08-19T12:56:08.013 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:08.013 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.013 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:08.012546+0000 2024-08-19T12:56:08.013 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:08.012546+0000 2024-08-19T12:56:08.013 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:08.013 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.013 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:08.013 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:08.013 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:08.014 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:08.014 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:08.014 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:08.014 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:08.014 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:08.014 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:08.014 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 1 Processor -- start 2024-08-19T12:56:08.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 1 -- start start 2024-08-19T12:56:08.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 10 monclient: init 2024-08-19T12:56:08.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:08.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 10 monclient: _add_conns ranks=[0,1,3,5,8,2,7,4,6] 2024-08-19T12:56:08.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 10 monclient(hunting): picked mon.noname-a con 0x7f5abc06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:08.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 10 monclient(hunting): picked mon.noname-f con 0x7f5abc0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:08.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 10 monclient(hunting): picked mon.noname-g con 0x7f5abc0a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:08.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5abc136790 con 0x7f5abc06e7b0 2024-08-19T12:56:08.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5abc09ddf0 con 0x7f5abc0a5920 2024-08-19T12:56:08.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5abc09e0d0 con 0x7f5abc0a5d50 2024-08-19T12:56:08.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:08.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.012+0000 7f5ac0fb1640 10 monclient(hunting): authenticate will time out at 1762.099938s 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.013+0000 7f5aba59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f5abc06e7b0 0x7f5abc136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52606/0 (socket says 172.21.15.26:52606) 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.013+0000 7f5aba59c640 1 -- 172.21.15.26:0/2034072455 learned_addr learned my addr 172.21.15.26:0/2034072455 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5aba59c640 1 -- 172.21.15.26:0/2034072455 >> v1:172.21.15.26:6789/0 conn(0x7f5abc06e7b0 legacy=0x7f5abc136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 1 -- 172.21.15.26:0/2034072455 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 26807858 0 0) 0x7f5abc136790 con 0x7f5abc06e7b0 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient(hunting): my global_id is 194418 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 1 -- 172.21.15.26:0/2034072455 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5aa4003520 con 0x7f5abc06e7b0 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab9d9b640 1 -- 172.21.15.26:0/2034072455 >> v1:172.21.15.104:6789/0 conn(0x7f5abc0a5920 legacy=0x7f5abc13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5abad9d640 1 -- 172.21.15.26:0/2034072455 >> v1:172.21.15.104:6790/0 conn(0x7f5abc0a5d50 legacy=0x7f5abc141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 1 -- 172.21.15.26:0/2034072455 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1102980120 0 0) 0x7f5aa4003520 con 0x7f5abc06e7b0 2024-08-19T12:56:08.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 1 -- 172.21.15.26:0/2034072455 >> v1:172.21.15.104:6790/0 conn(0x7f5abc0a5d50 legacy=0x7f5abc141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 1 -- 172.21.15.26:0/2034072455 >> v1:172.21.15.104:6789/0 conn(0x7f5abc0a5920 legacy=0x7f5abc13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient: _finish_hunting 0 2024-08-19T12:56:08.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 1 monclient: found mon.noname-a 2024-08-19T12:56:08.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:08.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:56:08.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 1 -- 172.21.15.26:0/2034072455 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5abc09e3b0 con 0x7f5abc06e7b0 2024-08-19T12:56:08.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient: _finish_auth 0 2024-08-19T12:56:08.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient: _check_auth_tickets 2024-08-19T12:56:08.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:08.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 1 -- 172.21.15.26:0/2034072455 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5aac001ae0 con 0x7f5abc06e7b0 2024-08-19T12:56:08.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:56:08.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient: dump: 2024-08-19T12:56:08.022 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.022 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.022 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.022 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.022 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.022 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.022 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.022 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.022 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.022 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.022 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ac0fb1640 5 monclient: authenticate success, global_id 194418 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ac0fb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 1 -- 172.21.15.26:0/2034072455 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5aac0048a0 con 0x7f5abc06e7b0 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 1 -- 172.21.15.26:0/2034072455 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5aac004e10 con 0x7f5abc06e7b0 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ac0fb1640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ac0fb1640 4 set_mon_vals no callback set 2024-08-19T12:56:08.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.015+0000 7f5ab959a640 10 monclient: dump: 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.024 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.016+0000 7f5ac0fb1640 10 monclient: shutdown 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.016+0000 7f5ac0fb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.016+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2034072455 >> v1:172.21.15.26:6789/0 conn(0x7f5abc06e7b0 legacy=0x7f5abc136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.016+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2034072455 shutdown_connections 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.016+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2034072455 >> 172.21.15.26:0/2034072455 conn(0x7f5abc0a8740 msgr2=0x7f5abc0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.016+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2034072455 shutdown_connections 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.016+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2034072455 wait complete. 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.017+0000 7f5ac0fb1640 10 monclient: build_initial_monmap 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.017+0000 7f5ac0fb1640 10 monclient: monmap: 2024-08-19T12:56:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:08.026 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.017+0000 7f5ac0fb1640 1 librados: starting msgr at 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.017+0000 7f5ac0fb1640 1 librados: starting objecter 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.017+0000 7f5ac0fb1640 1 Processor -- start 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.017+0000 7f5ac0fb1640 1 -- start start 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.017+0000 7f5ac0fb1640 1 librados: setting wanted keys 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.017+0000 7f5ac0fb1640 1 librados: calling monclient init 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.017+0000 7f5ac0fb1640 10 monclient: init 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.017+0000 7f5ac0fb1640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.017+0000 7f5ac0fb1640 10 monclient: _add_conns ranks=[4,6,5,1,2,8,0,7,3] 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.018+0000 7f5ac0fb1640 10 monclient(hunting): picked mon.noname-c con 0x7f5abc06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.018+0000 7f5ac0fb1640 10 monclient(hunting): picked mon.noname-d con 0x7f5abc0a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:08.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.018+0000 7f5ac0fb1640 10 monclient(hunting): picked mon.noname-h con 0x7f5abc0a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:08.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.018+0000 7f5ac0fb1640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5abc125820 con 0x7f5abc06e7b0 2024-08-19T12:56:08.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.018+0000 7f5ac0fb1640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5abc125260 con 0x7f5abc0a5d50 2024-08-19T12:56:08.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.018+0000 7f5ac0fb1640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5abc125540 con 0x7f5abc0a5920 2024-08-19T12:56:08.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.018+0000 7f5ac0fb1640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:08.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.018+0000 7f5ac0fb1640 10 monclient(hunting): authenticate will time out at 1762.105105s 2024-08-19T12:56:08.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.018+0000 7f5aba59c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f5abc06e7b0 0x7f5abc122560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:60036/0 (socket says 172.21.15.26:60036) 2024-08-19T12:56:08.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.018+0000 7f5aba59c640 1 -- 172.21.15.26:0/2469268440 learned_addr learned my addr 172.21.15.26:0/2469268440 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:08.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5ab9d9b640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.26:6792/0 conn(0x7f5abc0a5920 legacy=0x7f5abc122c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 client.?.objecter ms_handle_connect 0x7f5abc0a5920 2024-08-19T12:56:08.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.035 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 client.?.objecter ms_handle_connect 0x7f5abc06e7b0 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aba59c640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.26:6791/0 conn(0x7f5abc06e7b0 legacy=0x7f5abc122560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 client.?.objecter ms_handle_connect 0x7f5abc0a5d50 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5abad9d640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.104:6791/0 conn(0x7f5abc0a5d50 legacy=0x7f5abc123380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1984148611 0 0) 0x7f5abc125540 con 0x7f5abc0a5920 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): my global_id is 194370 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a9c004eb0 con 0x7f5abc0a5920 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1995718353 0 0) 0x7f5abc125820 con 0x7f5abc06e7b0 2024-08-19T12:56:08.036 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): my global_id is 184738 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5abc125540 con 0x7f5abc06e7b0 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2089140771 0 0) 0x7f5abc125260 con 0x7f5abc0a5d50 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): my global_id is 184631 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5abc125820 con 0x7f5abc0a5d50 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4091463176 0 0) 0x7f5a9c004eb0 con 0x7f5abc0a5920 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5abc125260 con 0x7f5abc0a5920 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5ab0003950 con 0x7f5abc0a5920 2024-08-19T12:56:08.037 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:56:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): dump: 2024-08-19T12:56:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.038 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.039 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.039 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.039 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.039 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.039 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.039 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.039 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:08.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2667984494 0 0) 0x7f5abc125540 con 0x7f5abc06e7b0 2024-08-19T12:56:08.039 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a9c004eb0 con 0x7f5abc06e7b0 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5aac004d90 con 0x7f5abc06e7b0 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): dump: 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.040 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1196796281 0 0) 0x7f5abc125820 con 0x7f5abc0a5d50 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5abc125540 con 0x7f5abc0a5d50 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5ab4002030 con 0x7f5abc0a5d50 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient(hunting): dump: 2024-08-19T12:56:08.041 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.042 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 376842469 0 0) 0x7f5abc125260 con 0x7f5abc0a5920 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.104:6791/0 conn(0x7f5abc0a5d50 legacy=0x7f5abc123380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.26:6791/0 conn(0x7f5abc06e7b0 legacy=0x7f5abc122560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient: _finish_hunting 0 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 monclient: found mon.d 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.020+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5abc124dc0 con 0x7f5abc0a5920 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5aaaffd640 10 monclient: _finish_auth 0 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5aaaffd640 10 monclient: _check_auth_tickets 2024-08-19T12:56:08.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5aaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5ac0fb1640 5 monclient: authenticate success, global_id 194370 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5ac0fb1640 10 monclient: _renew_subs 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5ac0fb1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5abc1d72a0 con 0x7f5abc0a5920 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5ac0fb1640 10 client.194370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5ac0fb1640 10 monclient: _renew_subs 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5ac0fb1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f5abc1d7650 con 0x7f5abc0a5920 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5ac0fb1640 1 librados: init done 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ab0003c20 con 0x7f5abc0a5920 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5aaaffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5ab00041d0 con 0x7f5abc0a5920 2024-08-19T12:56:08.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5aaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5aaaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5aabfff640 4 set_mon_vals no callback set 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5aaaffd640 10 monclient: dump: 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.045 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.046 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.046 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.046 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.046 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.046 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.021+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5ab0011a80 con 0x7f5abc0a5920 2024-08-19T12:56:08.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.022+0000 7f5ac0fb1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:08.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.022+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5aa00065c0 con 0x7f5aa0002860 2024-08-19T12:56:08.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.022+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(657..657 src has 350..657) ==== 5939+0+0 (unknown 4211950173 0 0) 0x7f5ab004e4d0 con 0x7f5abc0a5920 2024-08-19T12:56:08.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.022+0000 7f5aaaffd640 10 client.194370.objecter ms_dispatch 0x7f5abc000b90 osd_map(657..657 src has 350..657) 2024-08-19T12:56:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.022+0000 7f5aaaffd640 3 client.194370.objecter handle_osd_map got epochs [657,657] > 0 2024-08-19T12:56:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.022+0000 7f5aaaffd640 3 client.194370.objecter handle_osd_map decoding full epoch 657 2024-08-19T12:56:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.022+0000 7f5aaaffd640 20 client.194370.objecter dump_active .. 0 homeless 2024-08-19T12:56:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.022+0000 7f5abad9d640 10 monclient: get_auth_request con 0x7f5a9c041780 auth_method 0 2024-08-19T12:56:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 10 client.194370.objecter ms_handle_connect 0x7f5aa0002860 2024-08-19T12:56:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 10 client.194370.objecter resend_mon_ops 2024-08-19T12:56:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aba59c640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.26:6790/0 conn(0x7f5aa0002860 legacy=0x7f5aa0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1699731039 0 0) 0x7f5aa00065c0 con 0x7f5aa0002860 2024-08-19T12:56:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.047 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 10 monclient(hunting): my global_id is 27389 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5abc125260 con 0x7f5aa0002860 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 10 client.194370.objecter ms_handle_connect 0x7f5a9c041780 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1439540199 0 0) 0x7f5abc125260 con 0x7f5aa0002860 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5aa00065c0 con 0x7f5aa0002860 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5aac001d30 con 0x7f5aa0002860 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 10 monclient: dump: 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.048 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1992197564 0 0) 0x7f5aa00065c0 con 0x7f5aa0002860 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.024+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5aa000a160 con 0x7f5aa0002860 2024-08-19T12:56:08.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.025+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5aac005740 con 0x7f5aa0002860 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.025+0000 7f5aaaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.025+0000 7f5aaaffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.025+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.26:6790/0 conn(0x7f5aa0002860 legacy=0x7f5aa0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.025+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.26:6790/0 conn(0x7f5aa0002860 legacy=0x7f5aa0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.031+0000 7f5ac0fb1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.031+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5aa0009e70 con 0x7f5aa0009130 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.033+0000 7f5aba59c640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.26:6790/0 conn(0x7f5aa0009130 legacy=0x7f5aa00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.033+0000 7f5aaaffd640 10 client.194370.objecter ms_handle_connect 0x7f5aa0009130 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.033+0000 7f5aaaffd640 10 client.194370.objecter resend_mon_ops 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.033+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763274747 0 0) 0x7f5aa0009e70 con 0x7f5aa0009130 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.033+0000 7f5aaaffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.033+0000 7f5aaaffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.033+0000 7f5aaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.033+0000 7f5aaaffd640 10 monclient(hunting): my global_id is 27398 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.033+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5aa00065c0 con 0x7f5aa0009130 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3819460058 0 0) 0x7f5aa00065c0 con 0x7f5aa0009130 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5aa0009e70 con 0x7f5aa0009130 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5aac005e20 con 0x7f5aa0009130 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 10 monclient: dump: 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.051 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1738732182 0 0) 0x7f5aa0009e70 con 0x7f5aa0009130 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5aa000c3f0 con 0x7f5aa0009130 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7713 (unknown 0 0 1584861068) 0x7f5aac008120 con 0x7f5aa0009130 2024-08-19T12:56:08.052 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:08.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:08.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.26:6790/0 conn(0x7f5aa0009130 legacy=0x7f5aa00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5aaaffd640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.26:6790/0 conn(0x7f5aa0009130 legacy=0x7f5aa00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.053 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:08.053 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 1081089, 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:56:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.055 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.056 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:08.057 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.058 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.059 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:08.062 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:08.066 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.067 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.068 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:08.070 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:08.071 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.072 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:08.074 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.075 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 3 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.076 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.077 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:08.077 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:08.078 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.078 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045812242}) 2024-08-19T12:56:08.078 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4239749745}) 2024-08-19T12:56:08.078 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.078 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045812242}) 2024-08-19T12:56:08.078 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2673204440}) 2024-08-19T12:56:08.078 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.078 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045812242}) 2024-08-19T12:56:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5ac0fb1640 10 librados: watch_flush enter 2024-08-19T12:56:08.079 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4264688556}) 2024-08-19T12:56:08.079 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.079 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045812242}) 2024-08-19T12:56:08.079 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=437664445}) 2024-08-19T12:56:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5ac0fb1640 10 librados: watch_flush exit 2024-08-19T12:56:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5ac0fb1640 20 client.194370.objecter shutdown clearing up homeless session... 2024-08-19T12:56:08.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5ac0fb1640 10 client.194370.objecter successfully canceled tick 2024-08-19T12:56:08.081 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.081 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045812242}) 2024-08-19T12:56:08.081 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2278319832}) 2024-08-19T12:56:08.081 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.081 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045812242}) 2024-08-19T12:56:08.081 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2956087974}) 2024-08-19T12:56:08.081 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.081 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045812242}) 2024-08-19T12:56:08.081 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.104:6800/899887506 conn(0x7f5a9c041780 legacy=0x7f5a9c043c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.082 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=634758187}) 2024-08-19T12:56:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5ac0fb1640 10 monclient: shutdown 2024-08-19T12:56:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5ac0fb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2469268440 >> v1:172.21.15.26:6792/0 conn(0x7f5abc0a5920 legacy=0x7f5abc122c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.082 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.034+0000 7f5abad9d640 1 -- 172.21.15.26:0/2469268440 reap_dead start 2024-08-19T12:56:08.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.035+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.035+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045812242}) 2024-08-19T12:56:08.083 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.035+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=10178224}) 2024-08-19T12:56:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.035+0000 7f5abad9d640 1 -- 172.21.15.26:0/2469268440 reap_dead start 2024-08-19T12:56:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.035+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2469268440 shutdown_connections 2024-08-19T12:56:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.035+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2469268440 >> 172.21.15.26:0/2469268440 conn(0x7f5abc0a8740 msgr2=0x7f5abc135130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.035+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2469268440 shutdown_connections 2024-08-19T12:56:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.035+0000 7f5ac0fb1640 1 -- 172.21.15.26:0/2469268440 wait complete. 2024-08-19T12:56:08.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.035+0000 7f5ac0fb1640 1 librados: shutdown 2024-08-19T12:56:08.085 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:08.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-08-19T12:56:08.085 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:56:08.136 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.136 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277894983}) 2024-08-19T12:56:08.136 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2488523177}) 2024-08-19T12:56:08.136 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.136 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277894983}) 2024-08-19T12:56:08.136 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=441511889}) 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277894983}) 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=607847754}) 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277894983}) 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4291871917}) 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277894983}) 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=211799620}) 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277894983}) 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3569997837}) 2024-08-19T12:56:08.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277894983}) 2024-08-19T12:56:08.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=765568133}) 2024-08-19T12:56:08.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277894983}) 2024-08-19T12:56:08.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.134+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2418080264}) 2024-08-19T12:56:08.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.226+0000 7f0d1c7f4640 10 monclient: get_monmap_and_config 2024-08-19T12:56:08.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.226+0000 7f0d1c7f4640 10 monclient: build_initial_monmap 2024-08-19T12:56:08.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.226+0000 7f0d1c7f4640 10 monclient: monmap: 2024-08-19T12:56:08.227 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:08.227 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.227 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:08.226986+0000 2024-08-19T12:56:08.227 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:08.226986+0000 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.226+0000 7f0d1c7f4640 1 Processor -- start 2024-08-19T12:56:08.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.226+0000 7f0d1c7f4640 1 -- start start 2024-08-19T12:56:08.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.226+0000 7f0d1c7f4640 10 monclient: init 2024-08-19T12:56:08.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.227+0000 7f0d1c7f4640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.227+0000 7f0d1c7f4640 10 monclient: _add_conns ranks=[1,8,5,6,7,2,4,0,3] 2024-08-19T12:56:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.227+0000 7f0d1c7f4640 10 monclient(hunting): picked mon.noname-f con 0x7f0d14073470 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.227+0000 7f0d1c7f4640 10 monclient(hunting): picked mon.noname-e con 0x7f0d140a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.227+0000 7f0d1c7f4640 10 monclient(hunting): picked mon.noname-h con 0x7f0d1413f2e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.227+0000 7f0d1c7f4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d1409dae0 con 0x7f0d14073470 2024-08-19T12:56:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.227+0000 7f0d1c7f4640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d14136910 con 0x7f0d1413f2e0 2024-08-19T12:56:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.227+0000 7f0d1c7f4640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d1409dd50 con 0x7f0d140a57a0 2024-08-19T12:56:08.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.227+0000 7f0d1c7f4640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.227+0000 7f0d1c7f4640 10 monclient(hunting): authenticate will time out at 1762.314405s 2024-08-19T12:56:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.227+0000 7f0d19d8f640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f0d140a57a0 0x7f0d140a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:58014/0 (socket says 172.21.15.26:58014) 2024-08-19T12:56:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.227+0000 7f0d19d8f640 1 -- 172.21.15.26:0/2794562565 learned_addr learned my addr 172.21.15.26:0/2794562565 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.229+0000 7f0d19d8f640 1 -- 172.21.15.26:0/2794562565 >> v1:172.21.15.26:6793/0 conn(0x7f0d140a57a0 legacy=0x7f0d140a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.229+0000 7f0d1958e640 1 -- 172.21.15.26:0/2794562565 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1857803924 0 0) 0x7f0d1409dd50 con 0x7f0d140a57a0 2024-08-19T12:56:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.229+0000 7f0d1958e640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.229+0000 7f0d1958e640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.229+0000 7f0d1958e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.229+0000 7f0d1958e640 10 monclient(hunting): my global_id is 184562 2024-08-19T12:56:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.229+0000 7f0d1ad91640 1 -- 172.21.15.26:0/2794562565 >> v1:172.21.15.104:6791/0 conn(0x7f0d1413f2e0 legacy=0x7f0d141416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.229+0000 7f0d1958e640 1 -- 172.21.15.26:0/2794562565 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cfc003520 con 0x7f0d140a57a0 2024-08-19T12:56:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.229+0000 7f0d1a590640 1 -- 172.21.15.26:0/2794562565 >> v1:172.21.15.104:6789/0 conn(0x7f0d14073470 legacy=0x7f0d14136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 1 -- 172.21.15.26:0/2794562565 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1307831891 0 0) 0x7f0d14136910 con 0x7f0d1413f2e0 2024-08-19T12:56:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient(hunting): my global_id is 184640 2024-08-19T12:56:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 1 -- 172.21.15.26:0/2794562565 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d1409dd50 con 0x7f0d1413f2e0 2024-08-19T12:56:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 1 -- 172.21.15.26:0/2794562565 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1176695712 0 0) 0x7f0cfc003520 con 0x7f0d140a57a0 2024-08-19T12:56:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 1 -- 172.21.15.26:0/2794562565 >> v1:172.21.15.104:6791/0 conn(0x7f0d1413f2e0 legacy=0x7f0d141416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 1 -- 172.21.15.26:0/2794562565 >> v1:172.21.15.104:6789/0 conn(0x7f0d14073470 legacy=0x7f0d14136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient: _finish_hunting 0 2024-08-19T12:56:08.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 1 monclient: found mon.noname-e 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 1 -- 172.21.15.26:0/2794562565 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1409e030 con 0x7f0d140a57a0 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient: _finish_auth 0 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient: _check_auth_tickets 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 1 -- 172.21.15.26:0/2794562565 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0d08001ae0 con 0x7f0d140a57a0 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1c7f4640 5 monclient: authenticate success, global_id 184562 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1c7f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient: dump: 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.237 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1c7f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 1 -- 172.21.15.26:0/2794562565 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0d080048a0 con 0x7f0d140a57a0 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 1 -- 172.21.15.26:0/2794562565 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0d08004e10 con 0x7f0d140a57a0 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1958e640 10 monclient: dump: 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.239 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.240 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1c7f4640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1c7f4640 4 set_mon_vals no callback set 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1c7f4640 10 monclient: shutdown 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1c7f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/2794562565 >> v1:172.21.15.26:6793/0 conn(0x7f0d140a57a0 legacy=0x7f0d140a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/2794562565 shutdown_connections 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/2794562565 >> 172.21.15.26:0/2794562565 conn(0x7f0d140abff0 msgr2=0x7f0d140ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/2794562565 shutdown_connections 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.230+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/2794562565 wait complete. 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.231+0000 7f0d1c7f4640 10 monclient: build_initial_monmap 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.231+0000 7f0d1c7f4640 10 monclient: monmap: 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:08.241 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:08.242 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.231+0000 7f0d1c7f4640 1 librados: starting msgr at 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.231+0000 7f0d1c7f4640 1 librados: starting objecter 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.231+0000 7f0d1c7f4640 1 Processor -- start 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.231+0000 7f0d1c7f4640 1 -- start start 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.231+0000 7f0d1c7f4640 1 librados: setting wanted keys 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.231+0000 7f0d1c7f4640 1 librados: calling monclient init 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.231+0000 7f0d1c7f4640 10 monclient: init 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d1c7f4640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d1c7f4640 10 monclient: _add_conns ranks=[0,2,7,6,8,1,4,3,5] 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d1c7f4640 10 monclient(hunting): picked mon.noname-a con 0x7f0d14073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d1c7f4640 10 monclient(hunting): picked mon.noname-b con 0x7f0d140a57a0 addr v1:172.21.15.26:6790/0 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d1c7f4640 10 monclient(hunting): picked mon.noname-i con 0x7f0d1413f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d1c7f4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d14136bc0 con 0x7f0d14073470 2024-08-19T12:56:08.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d1c7f4640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d140a39b0 con 0x7f0d140a57a0 2024-08-19T12:56:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d1c7f4640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d140a2dd0 con 0x7f0d1413f2e0 2024-08-19T12:56:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d1c7f4640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d1c7f4640 10 monclient(hunting): authenticate will time out at 1762.319290s 2024-08-19T12:56:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d19d8f640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f0d140a57a0 0x7f0d140a26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:55198/0 (socket says 172.21.15.26:55198) 2024-08-19T12:56:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d1a590640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0d14073470 0x7f0d140a1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52618/0 (socket says 172.21.15.26:52618) 2024-08-19T12:56:08.244 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.232+0000 7f0d19d8f640 1 -- 172.21.15.26:0/68217473 learned_addr learned my addr 172.21.15.26:0/68217473 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d19d8f640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.26:6790/0 conn(0x7f0d140a57a0 legacy=0x7f0d140a26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d1a590640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.26:6789/0 conn(0x7f0d14073470 legacy=0x7f0d140a1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d1ad91640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.104:6792/0 conn(0x7f0d1413f2e0 legacy=0x7f0d141d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 client.?.objecter ms_handle_connect 0x7f0d140a57a0 2024-08-19T12:56:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 client.?.objecter ms_handle_connect 0x7f0d14073470 2024-08-19T12:56:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 client.?.objecter ms_handle_connect 0x7f0d1413f2e0 2024-08-19T12:56:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3975150136 0 0) 0x7f0d140a39b0 con 0x7f0d140a57a0 2024-08-19T12:56:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.251 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): my global_id is 27407 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d00004eb0 con 0x7f0d140a57a0 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2617463478 0 0) 0x7f0d140a2dd0 con 0x7f0d1413f2e0 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): my global_id is 136366 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d140a39b0 con 0x7f0d1413f2e0 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1286818294 0 0) 0x7f0d14136bc0 con 0x7f0d14073470 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): my global_id is 194427 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d140a2dd0 con 0x7f0d14073470 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3031573905 0 0) 0x7f0d00004eb0 con 0x7f0d140a57a0 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d14136bc0 con 0x7f0d140a57a0 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0d08004d90 con 0x7f0d140a57a0 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.234+0000 7f0d12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient(hunting): dump: 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.253 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1415095596 0 0) 0x7f0d140a2dd0 con 0x7f0d14073470 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d00004eb0 con 0x7f0d14073470 2024-08-19T12:56:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 384540152 0 0) 0x7f0d140a39b0 con 0x7f0d1413f2e0 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d140a2dd0 con 0x7f0d1413f2e0 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0d0c003610 con 0x7f0d14073470 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient(hunting): dump: 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.255 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0d04002070 con 0x7f0d1413f2e0 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient(hunting): dump: 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3197133236 0 0) 0x7f0d14136bc0 con 0x7f0d140a57a0 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.104:6792/0 conn(0x7f0d1413f2e0 legacy=0x7f0d141d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.26:6789/0 conn(0x7f0d14073470 legacy=0x7f0d140a1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 monclient: found mon.b 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d14136e80 con 0x7f0d140a57a0 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient: _finish_auth 0 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d1c7f4640 5 monclient: authenticate success, global_id 27407 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d1c7f4640 10 monclient: _renew_subs 2024-08-19T12:56:08.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d1c7f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0d140a5470 con 0x7f0d140a57a0 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d1c7f4640 10 client.27407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d1c7f4640 10 monclient: _renew_subs 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d1c7f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0d140a4780 con 0x7f0d140a57a0 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d1c7f4640 1 librados: init done 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0d08001d30 con 0x7f0d140a57a0 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0d080048a0 con 0x7f0d140a57a0 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d18d8d640 4 set_mon_vals no callback set 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:08.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d12ffd640 10 monclient: dump: 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.260 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.235+0000 7f0d1c7f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.236+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0cf40065c0 con 0x7f0cf4002860 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.236+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f0d08011a00 con 0x7f0d140a57a0 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.236+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(657..657 src has 350..657) ==== 5939+0+0 (unknown 4211950173 0 0) 0x7f0d0804e450 con 0x7f0d140a57a0 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.236+0000 7f0d12ffd640 10 client.27407.objecter ms_dispatch 0x7f0d14000b90 osd_map(657..657 src has 350..657) 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.236+0000 7f0d12ffd640 3 client.27407.objecter handle_osd_map got epochs [657,657] > 0 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.236+0000 7f0d12ffd640 3 client.27407.objecter handle_osd_map decoding full epoch 657 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.236+0000 7f0d1ad91640 10 monclient: get_auth_request con 0x7f0d00041bc0 auth_method 0 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.236+0000 7f0d12ffd640 20 client.27407.objecter dump_active .. 0 homeless 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d1a590640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.26:6791/0 conn(0x7f0cf4002860 legacy=0x7f0cf4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 10 client.27407.objecter ms_handle_connect 0x7f0cf4002860 2024-08-19T12:56:08.261 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 10 client.27407.objecter resend_mon_ops 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 27270488 0 0) 0x7f0cf40065c0 con 0x7f0cf4002860 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 10 monclient(hunting): my global_id is 184747 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d14136bc0 con 0x7f0cf4002860 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 740815223 0 0) 0x7f0d14136bc0 con 0x7f0cf4002860 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0cf40065c0 con 0x7f0cf4002860 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0d0c003e10 con 0x7f0cf4002860 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 10 monclient: dump: 2024-08-19T12:56:08.262 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.263 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2339339968 0 0) 0x7f0cf40065c0 con 0x7f0cf4002860 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0cf400a160 con 0x7f0cf4002860 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.238+0000 7f0d12ffd640 10 client.27407.objecter ms_handle_connect 0x7f0d00041bc0 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.239+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0d0c003ea0 con 0x7f0cf4002860 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.239+0000 7f0d12ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.239+0000 7f0d12ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.239+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.26:6791/0 conn(0x7f0cf4002860 legacy=0x7f0cf4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.239+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.26:6791/0 conn(0x7f0cf4002860 legacy=0x7f0cf4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.245+0000 7f0d1c7f4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.245+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0cf4009e70 con 0x7f0cf4009130 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d1a590640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.26:6791/0 conn(0x7f0cf4009130 legacy=0x7f0cf40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 10 client.27407.objecter ms_handle_connect 0x7f0cf4009130 2024-08-19T12:56:08.264 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 10 client.27407.objecter resend_mon_ops 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3040869181 0 0) 0x7f0cf4009e70 con 0x7f0cf4009130 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 10 monclient(hunting): my global_id is 184756 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cf40065c0 con 0x7f0cf4009130 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3893525176 0 0) 0x7f0cf40065c0 con 0x7f0cf4009130 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0cf4009e70 con 0x7f0cf4009130 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f0d0c004350 con 0x7f0cf4009130 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 10 monclient: dump: 2024-08-19T12:56:08.265 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.266 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.267 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.267 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 953407607 0 0) 0x7f0cf4009e70 con 0x7f0cf4009130 2024-08-19T12:56:08.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.247+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0cf400c410 con 0x7f0cf4009130 2024-08-19T12:56:08.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 673597734) 0x7f0d0c007bc0 con 0x7f0cf4009130 2024-08-19T12:56:08.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d12ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:08.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d12ffd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:08.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.26:6791/0 conn(0x7f0cf4009130 legacy=0x7f0cf40095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d12ffd640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.26:6791/0 conn(0x7f0cf4009130 legacy=0x7f0cf40095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.268 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:08.268 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:08.268 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:08.268 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:08.268 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:08.268 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 293713, 2024-08-19T12:56:08.268 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:08.268 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:08.268 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:08.268 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:08.268 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:08.268 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.269 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.270 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:08.271 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:08.272 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:08.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.275 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.276 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:08.276 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.276 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.276 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:08.276 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:08.276 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.276 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.276 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:08.277 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:08.278 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:08.279 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.280 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.282 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:08.282 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:08.282 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.282 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.282 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.282 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:08.282 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:08.282 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.282 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:08.284 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:08.285 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.285 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.285 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.285 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:08.285 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:08.286 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.286 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.286 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.286 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:08.286 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:08.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:08.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.289 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.289 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.289 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:08.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.289 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.289 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:08.289 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:08.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 10 librados: watch_flush enter 2024-08-19T12:56:08.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 10 librados: watch_flush exit 2024-08-19T12:56:08.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 20 client.27407.objecter shutdown clearing up homeless session... 2024-08-19T12:56:08.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 10 client.27407.objecter successfully canceled tick 2024-08-19T12:56:08.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.104:6800/899887506 conn(0x7f0d00041bc0 legacy=0x7f0d00044080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 10 monclient: shutdown 2024-08-19T12:56:08.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:08.291 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/68217473 >> v1:172.21.15.26:6790/0 conn(0x7f0d140a57a0 legacy=0x7f0d140a26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1ad91640 1 -- 172.21.15.26:0/68217473 reap_dead start 2024-08-19T12:56:08.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1ad91640 1 -- 172.21.15.26:0/68217473 reap_dead start 2024-08-19T12:56:08.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/68217473 shutdown_connections 2024-08-19T12:56:08.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/68217473 >> 172.21.15.26:0/68217473 conn(0x7f0d140abff0 msgr2=0x7f0d14134570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:08.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/68217473 shutdown_connections 2024-08-19T12:56:08.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 1 -- 172.21.15.26:0/68217473 wait complete. 2024-08-19T12:56:08.292 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.248+0000 7f0d1c7f4640 1 librados: shutdown 2024-08-19T12:56:08.292 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:08.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-08-19T12:56:08.293 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-08-19T12:56:08.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.430+0000 7fe628b7b640 10 monclient: get_monmap_and_config 2024-08-19T12:56:08.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.430+0000 7fe628b7b640 10 monclient: build_initial_monmap 2024-08-19T12:56:08.431 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.430+0000 7fe628b7b640 10 monclient: monmap: 2024-08-19T12:56:08.431 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:08.431 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.431 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:08.431571+0000 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:08.431571+0000 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:08.432 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 1 Processor -- start 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 1 -- start start 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 10 monclient: init 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 10 monclient: _add_conns ranks=[0,7,8,3,4,6,2,5,1] 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 10 monclient(hunting): picked mon.noname-a con 0x7fe62406e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 10 monclient(hunting): picked mon.noname-i con 0x7fe6240a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 10 monclient(hunting): picked mon.noname-e con 0x7fe6240a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe624136790 con 0x7fe62406e7b0 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe62409ddf0 con 0x7fe6240a5920 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe62409e0d0 con 0x7fe6240a5d50 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:08.434 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.431+0000 7fe628b7b640 10 monclient(hunting): authenticate will time out at 1762.518975s 2024-08-19T12:56:08.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.432+0000 7fe622d9d640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fe6240a5d50 0x7fe624141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:58022/0 (socket says 172.21.15.26:58022) 2024-08-19T12:56:08.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.432+0000 7fe62259c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe62406e7b0 0x7fe624136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52632/0 (socket says 172.21.15.26:52632) 2024-08-19T12:56:08.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.432+0000 7fe622d9d640 1 -- 172.21.15.26:0/1428547036 learned_addr learned my addr 172.21.15.26:0/1428547036 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:08.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe622d9d640 1 -- 172.21.15.26:0/1428547036 >> v1:172.21.15.26:6793/0 conn(0x7fe6240a5d50 legacy=0x7fe624141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe62259c640 1 -- 172.21.15.26:0/1428547036 >> v1:172.21.15.26:6789/0 conn(0x7fe62406e7b0 legacy=0x7fe624136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 1 -- 172.21.15.26:0/1428547036 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1137976561 0 0) 0x7fe624136790 con 0x7fe62406e7b0 2024-08-19T12:56:08.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient(hunting): my global_id is 194436 2024-08-19T12:56:08.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 1 -- 172.21.15.26:0/1428547036 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe610003520 con 0x7fe62406e7b0 2024-08-19T12:56:08.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 1 -- 172.21.15.26:0/1428547036 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 338483723 0 0) 0x7fe62409e0d0 con 0x7fe6240a5d50 2024-08-19T12:56:08.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient(hunting): my global_id is 184571 2024-08-19T12:56:08.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 1 -- 172.21.15.26:0/1428547036 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe624136790 con 0x7fe6240a5d50 2024-08-19T12:56:08.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe61bfff640 1 -- 172.21.15.26:0/1428547036 >> v1:172.21.15.104:6792/0 conn(0x7fe6240a5920 legacy=0x7fe62413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 1 -- 172.21.15.26:0/1428547036 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1749075097 0 0) 0x7fe610003520 con 0x7fe62406e7b0 2024-08-19T12:56:08.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 1 -- 172.21.15.26:0/1428547036 >> v1:172.21.15.26:6793/0 conn(0x7fe6240a5d50 legacy=0x7fe624141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 1 -- 172.21.15.26:0/1428547036 >> v1:172.21.15.104:6792/0 conn(0x7fe6240a5920 legacy=0x7fe62413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:56:08.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 1 monclient: found mon.noname-a 2024-08-19T12:56:08.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:08.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:56:08.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 1 -- 172.21.15.26:0/1428547036 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62409e3b0 con 0x7fe62406e7b0 2024-08-19T12:56:08.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient: _finish_auth 0 2024-08-19T12:56:08.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:56:08.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:08.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 1 -- 172.21.15.26:0/1428547036 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe614001ae0 con 0x7fe62406e7b0 2024-08-19T12:56:08.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe628b7b640 5 monclient: authenticate success, global_id 194436 2024-08-19T12:56:08.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe628b7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:08.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:56:08.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient: dump: 2024-08-19T12:56:08.441 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.442 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe628b7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 1 -- 172.21.15.26:0/1428547036 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe6140047e0 con 0x7fe62406e7b0 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 1 -- 172.21.15.26:0/1428547036 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe614004d50 con 0x7fe62406e7b0 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe621d9b640 10 monclient: dump: 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.443 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe628b7b640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.434+0000 7fe628b7b640 4 set_mon_vals no callback set 2024-08-19T12:56:08.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.435+0000 7fe628b7b640 10 monclient: shutdown 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.435+0000 7fe628b7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.435+0000 7fe628b7b640 1 -- 172.21.15.26:0/1428547036 >> v1:172.21.15.26:6789/0 conn(0x7fe62406e7b0 legacy=0x7fe624136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.435+0000 7fe628b7b640 1 -- 172.21.15.26:0/1428547036 shutdown_connections 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.435+0000 7fe628b7b640 1 -- 172.21.15.26:0/1428547036 >> 172.21.15.26:0/1428547036 conn(0x7fe6240a8740 msgr2=0x7fe6240a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.435+0000 7fe628b7b640 1 -- 172.21.15.26:0/1428547036 shutdown_connections 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.435+0000 7fe628b7b640 1 -- 172.21.15.26:0/1428547036 wait complete. 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 10 monclient: build_initial_monmap 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 10 monclient: monmap: 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:08.445 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 1 librados: starting msgr at 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 1 librados: starting objecter 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 1 Processor -- start 2024-08-19T12:56:08.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 1 -- start start 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 1 librados: setting wanted keys 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 1 librados: calling monclient init 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 10 monclient: init 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 10 monclient: _add_conns ranks=[6,1,3,8,5,4,7,0,2] 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 10 monclient(hunting): picked mon.noname-d con 0x7fe62406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 10 monclient(hunting): picked mon.noname-f con 0x7fe6240a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 10 monclient(hunting): picked mon.noname-g con 0x7fe6240a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe62413b3f0 con 0x7fe6240a5920 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe624139090 con 0x7fe6240a5d50 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe62413ab50 con 0x7fe62406e7b0 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:08.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe628b7b640 10 monclient(hunting): authenticate will time out at 1762.523956s 2024-08-19T12:56:08.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.436+0000 7fe62259c640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7fe62406e7b0 0x7fe6241ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34150/0 (socket says 172.21.15.26:34150) 2024-08-19T12:56:08.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.437+0000 7fe62259c640 1 -- 172.21.15.26:0/3976037012 learned_addr learned my addr 172.21.15.26:0/3976037012 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:08.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.439+0000 7fe62259c640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe62406e7b0 legacy=0x7fe6241ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.439+0000 7fe61affd640 10 client.?.objecter ms_handle_connect 0x7fe62406e7b0 2024-08-19T12:56:08.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.439+0000 7fe61affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.439+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3327991701 0 0) 0x7fe62413ab50 con 0x7fe62406e7b0 2024-08-19T12:56:08.453 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.439+0000 7fe61affd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.439+0000 7fe61affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.439+0000 7fe61affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.439+0000 7fe61affd640 10 monclient(hunting): my global_id is 194379 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.439+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe604004eb0 con 0x7fe62406e7b0 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.439+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.439+0000 7fe61bfff640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.104:6789/0 conn(0x7fe6240a5920 legacy=0x7fe6241d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.439+0000 7fe622d9d640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.104:6790/0 conn(0x7fe6240a5d50 legacy=0x7fe6241d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 client.?.objecter ms_handle_connect 0x7fe6240a5920 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 client.?.objecter ms_handle_connect 0x7fe6240a5d50 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 938629218 0 0) 0x7fe604004eb0 con 0x7fe62406e7b0 2024-08-19T12:56:08.454 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe62413ab50 con 0x7fe62406e7b0 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2002100851 0 0) 0x7fe62413b3f0 con 0x7fe6240a5920 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient(hunting): my global_id is 204238 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe604004eb0 con 0x7fe6240a5920 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2321508980 0 0) 0x7fe624139090 con 0x7fe6240a5d50 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient(hunting): my global_id is 194403 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe62413b3f0 con 0x7fe6240a5d50 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe614004cd0 con 0x7fe62406e7b0 2024-08-19T12:56:08.455 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient(hunting): dump: 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.456 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2362456767 0 0) 0x7fe62413ab50 con 0x7fe62406e7b0 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.104:6790/0 conn(0x7fe6240a5d50 legacy=0x7fe6241d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.104:6789/0 conn(0x7fe6240a5920 legacy=0x7fe6241d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient: _finish_hunting 0 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 monclient: found mon.d 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:08.457 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62413ae30 con 0x7fe62406e7b0 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient: _finish_auth 0 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient: _check_auth_tickets 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe628b7b640 5 monclient: authenticate success, global_id 194379 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe628b7b640 10 monclient: _renew_subs 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe628b7b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe628b7b640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe624139bf0 con 0x7fe62406e7b0 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe628b7b640 10 client.194379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe628b7b640 10 monclient: _renew_subs 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe628b7b640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe628b7b640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6241386a0 con 0x7fe62406e7b0 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe628b7b640 1 librados: init done 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe614001b00 con 0x7fe62406e7b0 2024-08-19T12:56:08.458 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe614005680 con 0x7fe62406e7b0 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe62159a640 4 set_mon_vals no callback set 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.441+0000 7fe61affd640 10 monclient: dump: 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.459 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.442+0000 7fe628b7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.442+0000 7fe628b7b640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe6080065c0 con 0x7fe608002860 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.442+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fe614011ae0 con 0x7fe62406e7b0 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.443+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(657..657 src has 350..657) ==== 5939+0+0 (unknown 4211950173 0 0) 0x7fe61404e4c0 con 0x7fe62406e7b0 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.443+0000 7fe61affd640 10 client.194379.objecter ms_dispatch 0x7fe624000b90 osd_map(657..657 src has 350..657) 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.443+0000 7fe61affd640 3 client.194379.objecter handle_osd_map got epochs [657,657] > 0 2024-08-19T12:56:08.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.443+0000 7fe61affd640 3 client.194379.objecter handle_osd_map decoding full epoch 657 2024-08-19T12:56:08.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.443+0000 7fe61affd640 20 client.194379.objecter dump_active .. 0 homeless 2024-08-19T12:56:08.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.443+0000 7fe622d9d640 10 monclient: get_auth_request con 0x7fe604041290 auth_method 0 2024-08-19T12:56:08.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61bfff640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe608002860 legacy=0x7fe608004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 10 client.194379.objecter ms_handle_connect 0x7fe608002860 2024-08-19T12:56:08.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 10 client.194379.objecter resend_mon_ops 2024-08-19T12:56:08.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 447364825 0 0) 0x7fe6080065c0 con 0x7fe608002860 2024-08-19T12:56:08.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.461 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 10 monclient(hunting): my global_id is 194388 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe62413ab50 con 0x7fe608002860 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 10 client.194379.objecter ms_handle_connect 0x7fe604041290 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 160713049 0 0) 0x7fe62413ab50 con 0x7fe608002860 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6080065c0 con 0x7fe608002860 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe60c001cb0 con 0x7fe608002860 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 10 monclient: dump: 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.462 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2406345648 0 0) 0x7fe6080065c0 con 0x7fe608002860 2024-08-19T12:56:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe60800a160 con 0x7fe608002860 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61bfff640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe608002860 legacy=0x7fe608004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61bfff640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe608002860 legacy=0x7fe608004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61bfff640 1 --1- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe608002860 0x7fe608004d00 :-1 s=OPENED pgs=3586 cs=1 l=1).handle_message read tag failed 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61bfff640 1 --1- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe608002860 0x7fe608004d00 :-1 s=OPENED pgs=3586 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe608002860 legacy=0x7fe608004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe608002860 legacy=0x7fe608004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.445+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe6080065c0 con 0x7fe60404c870 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 10 client.194379.objecter ms_handle_connect 0x7fe60404c870 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 10 client.194379.objecter resend_mon_ops 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61bfff640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe60404c870 legacy=0x7fe60404aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 245551222 0 0) 0x7fe6080065c0 con 0x7fe60404c870 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.464 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 10 monclient(hunting): my global_id is 194397 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe62413ab50 con 0x7fe60404c870 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1509041310 0 0) 0x7fe62413ab50 con 0x7fe60404c870 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6080065c0 con 0x7fe60404c870 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe60c003d40 con 0x7fe60404c870 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 10 monclient: dump: 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.465 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3002176514 0 0) 0x7fe6080065c0 con 0x7fe60404c870 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6140047e0 con 0x7fe60404c870 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe60c0041b0 con 0x7fe60404c870 2024-08-19T12:56:08.466 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:08.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:08.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe60404c870 legacy=0x7fe60404aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.467 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.448+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe60404c870 legacy=0x7fe60404aec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.454+0000 7fe628b7b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:08.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.455+0000 7fe628b7b640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe60800c4f0 con 0x7fe608009110 2024-08-19T12:56:08.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 10 client.194379.objecter ms_handle_connect 0x7fe608009110 2024-08-19T12:56:08.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 10 client.194379.objecter resend_mon_ops 2024-08-19T12:56:08.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61bfff640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe608009110 legacy=0x7fe608009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1142738524 0 0) 0x7fe60800c4f0 con 0x7fe608009110 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 10 monclient(hunting): my global_id is 194406 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6080065c0 con 0x7fe608009110 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3131632089 0 0) 0x7fe6080065c0 con 0x7fe608009110 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe60800c4f0 con 0x7fe608009110 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fe60c005b00 con 0x7fe608009110 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 10 monclient: dump: 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.469 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.470 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3205641520 0 0) 0x7fe60800c4f0 con 0x7fe608009110 2024-08-19T12:56:08.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 --> v1:172.21.15.26:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe608009ce0 con 0x7fe608009110 2024-08-19T12:56:08.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 <== mon.6 v1:172.21.15.26:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7545 (unknown 0 0 1099216633) 0x7fe60c006870 con 0x7fe608009110 2024-08-19T12:56:08.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:08.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:08.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe608009110 legacy=0x7fe608009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe61affd640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe608009110 legacy=0x7fe608009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.457+0000 7fe622d9d640 1 -- 172.21.15.26:0/3976037012 reap_dead start 2024-08-19T12:56:08.471 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:08.471 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 153384, 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:08.472 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 5, 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.473 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.474 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:08.475 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.476 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:08.477 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:08.480 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.481 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:08.483 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.484 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.485 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.486 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:08.487 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.489 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:08.490 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:08.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 10 librados: watch_flush enter 2024-08-19T12:56:08.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 10 librados: watch_flush exit 2024-08-19T12:56:08.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 20 client.194379.objecter shutdown clearing up homeless session... 2024-08-19T12:56:08.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 10 client.194379.objecter successfully canceled tick 2024-08-19T12:56:08.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.104:6800/899887506 conn(0x7fe604041290 legacy=0x7fe604043750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 10 monclient: shutdown 2024-08-19T12:56:08.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:08.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 1 -- 172.21.15.26:0/3976037012 >> v1:172.21.15.26:6792/0 conn(0x7fe62406e7b0 legacy=0x7fe6241ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 1 -- 172.21.15.26:0/3976037012 shutdown_connections 2024-08-19T12:56:08.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 1 -- 172.21.15.26:0/3976037012 >> 172.21.15.26:0/3976037012 conn(0x7fe6240a8740 msgr2=0x7fe6240aa470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:08.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 1 -- 172.21.15.26:0/3976037012 shutdown_connections 2024-08-19T12:56:08.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 1 -- 172.21.15.26:0/3976037012 wait complete. 2024-08-19T12:56:08.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.458+0000 7fe628b7b640 1 librados: shutdown 2024-08-19T12:56:08.493 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:08.494 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-08-19T12:56:08.494 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:56:08.573 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.573 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060377743}) 2024-08-19T12:56:08.573 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2181470485}) 2024-08-19T12:56:08.573 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.573 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060377743}) 2024-08-19T12:56:08.573 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742007594}) 2024-08-19T12:56:08.573 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060377743}) 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1004594449}) 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060377743}) 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3659247552}) 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060377743}) 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2622628878}) 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060377743}) 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3408170115}) 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.574 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060377743}) 2024-08-19T12:56:08.575 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2654208183}) 2024-08-19T12:56:08.575 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.575 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060377743}) 2024-08-19T12:56:08.575 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.571+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2366672453}) 2024-08-19T12:56:08.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.593+0000 7f655dcbc640 10 monclient: get_monmap_and_config 2024-08-19T12:56:08.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.594+0000 7f655dcbc640 10 monclient: build_initial_monmap 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.594+0000 7f655dcbc640 10 monclient: monmap: 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:08.594867+0000 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:08.594867+0000 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:08.595 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:08.596 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:08.596 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.594+0000 7f655dcbc640 1 Processor -- start 2024-08-19T12:56:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.594+0000 7f655dcbc640 1 -- start start 2024-08-19T12:56:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.594+0000 7f655dcbc640 10 monclient: init 2024-08-19T12:56:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f655dcbc640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:08.597 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f655dcbc640 10 monclient: _add_conns ranks=[5,0,8,4,1,3,7,6,2] 2024-08-19T12:56:08.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f655dcbc640 10 monclient(hunting): picked mon.noname-h con 0x7f655806e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:08.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f655dcbc640 10 monclient(hunting): picked mon.noname-a con 0x7f65580a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:08.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f655dcbc640 10 monclient(hunting): picked mon.noname-e con 0x7f65580a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:08.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f655dcbc640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6558136790 con 0x7f65580a5920 2024-08-19T12:56:08.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f655dcbc640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655809ddf0 con 0x7f655806e7b0 2024-08-19T12:56:08.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f655dcbc640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655809e0d0 con 0x7f65580a5d50 2024-08-19T12:56:08.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f655dcbc640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:08.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f655dcbc640 10 monclient(hunting): authenticate will time out at 1762.682301s 2024-08-19T12:56:08.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f6557fff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f65580a5d50 0x7f6558141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:58026/0 (socket says 172.21.15.26:58026) 2024-08-19T12:56:08.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f6556ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f65580a5920 0x7f655813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:52648/0 (socket says 172.21.15.26:52648) 2024-08-19T12:56:08.599 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.595+0000 7f6557fff640 1 -- 172.21.15.26:0/2202708431 learned_addr learned my addr 172.21.15.26:0/2202708431 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f6557fff640 1 -- 172.21.15.26:0/2202708431 >> v1:172.21.15.26:6793/0 conn(0x7f65580a5d50 legacy=0x7f6558141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f6556ffd640 1 -- 172.21.15.26:0/2202708431 >> v1:172.21.15.26:6789/0 conn(0x7f65580a5920 legacy=0x7f655813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65577fe640 1 -- 172.21.15.26:0/2202708431 >> v1:172.21.15.104:6791/0 conn(0x7f655806e7b0 legacy=0x7f6558136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 61671373 0 0) 0x7f655809e0d0 con 0x7f65580a5d50 2024-08-19T12:56:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 10 monclient(hunting): my global_id is 184580 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6540003520 con 0x7f65580a5d50 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1547551174 0 0) 0x7f6558136790 con 0x7f65580a5920 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 10 monclient(hunting): my global_id is 194445 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f655809e0d0 con 0x7f65580a5920 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3427213039 0 0) 0x7f65480014a0 con 0x7f655806e7b0 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 10 monclient(hunting): my global_id is 184649 2024-08-19T12:56:08.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.597+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6558136790 con 0x7f655806e7b0 2024-08-19T12:56:08.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1982236540 0 0) 0x7f655809e0d0 con 0x7f65580a5920 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 >> v1:172.21.15.26:6793/0 conn(0x7f65580a5d50 legacy=0x7f6558141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 >> v1:172.21.15.104:6791/0 conn(0x7f655806e7b0 legacy=0x7f6558136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 10 monclient: _finish_hunting 0 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 1 monclient: found mon.noname-a 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655809e3b0 con 0x7f65580a5920 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 10 monclient: _finish_auth 0 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 10 monclient: _check_auth_tickets 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f654c001970 con 0x7f65580a5920 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f655dcbc640 5 monclient: authenticate success, global_id 194445 2024-08-19T12:56:08.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f655dcbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 10 monclient: dump: 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.614 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.615 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f655dcbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f654c001c40 con 0x7f65580a5920 2024-08-19T12:56:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 1 -- 172.21.15.26:0/2202708431 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f654c004b20 con 0x7f65580a5920 2024-08-19T12:56:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f65567fc640 10 monclient: dump: 2024-08-19T12:56:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.616 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.617 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f655dcbc640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f655dcbc640 4 set_mon_vals no callback set 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f655dcbc640 10 monclient: shutdown 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f655dcbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f655dcbc640 1 -- 172.21.15.26:0/2202708431 >> v1:172.21.15.26:6789/0 conn(0x7f65580a5920 legacy=0x7f655813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f655dcbc640 1 -- 172.21.15.26:0/2202708431 shutdown_connections 2024-08-19T12:56:08.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f655dcbc640 1 -- 172.21.15.26:0/2202708431 >> 172.21.15.26:0/2202708431 conn(0x7f65580a8740 msgr2=0x7f65580a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f655dcbc640 1 -- 172.21.15.26:0/2202708431 shutdown_connections 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.598+0000 7f655dcbc640 1 -- 172.21.15.26:0/2202708431 wait complete. 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.599+0000 7f655dcbc640 10 monclient: build_initial_monmap 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.599+0000 7f655dcbc640 10 monclient: monmap: 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:08.619 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:08.620 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.599+0000 7f655dcbc640 1 librados: starting msgr at 2024-08-19T12:56:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.599+0000 7f655dcbc640 1 librados: starting objecter 2024-08-19T12:56:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.599+0000 7f655dcbc640 1 Processor -- start 2024-08-19T12:56:08.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.599+0000 7f655dcbc640 1 -- start start 2024-08-19T12:56:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.599+0000 7f655dcbc640 1 librados: setting wanted keys 2024-08-19T12:56:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.599+0000 7f655dcbc640 1 librados: calling monclient init 2024-08-19T12:56:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.599+0000 7f655dcbc640 10 monclient: init 2024-08-19T12:56:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.600+0000 7f655dcbc640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.600+0000 7f655dcbc640 10 monclient: _add_conns ranks=[6,5,7,8,3,2,4,0,1] 2024-08-19T12:56:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.600+0000 7f655dcbc640 10 monclient(hunting): picked mon.noname-d con 0x7f655806e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.600+0000 7f655dcbc640 10 monclient(hunting): picked mon.noname-h con 0x7f65580a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.600+0000 7f655dcbc640 10 monclient(hunting): picked mon.noname-i con 0x7f65580a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:08.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.600+0000 7f655dcbc640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655813b1a0 con 0x7f65580a5920 2024-08-19T12:56:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.600+0000 7f655dcbc640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655813b480 con 0x7f655806e7b0 2024-08-19T12:56:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.600+0000 7f655dcbc640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6558134580 con 0x7f65580a5d50 2024-08-19T12:56:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.600+0000 7f655dcbc640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.600+0000 7f655dcbc640 10 monclient(hunting): authenticate will time out at 1762.687343s 2024-08-19T12:56:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.600+0000 7f65577fe640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f655806e7b0 0x7f65581ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:34172/0 (socket says 172.21.15.26:34172) 2024-08-19T12:56:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.600+0000 7f65577fe640 1 -- 172.21.15.26:0/2472651109 learned_addr learned my addr 172.21.15.26:0/2472651109 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f65577fe640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.26:6792/0 conn(0x7f655806e7b0 legacy=0x7f65581ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 client.?.objecter ms_handle_connect 0x7f655806e7b0 2024-08-19T12:56:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 client.?.objecter ms_handle_connect 0x7f65580a5920 2024-08-19T12:56:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f6556ffd640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.104:6791/0 conn(0x7f65580a5920 legacy=0x7f65581d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 client.?.objecter ms_handle_connect 0x7f65580a5d50 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f6557fff640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.104:6792/0 conn(0x7f65580a5d50 legacy=0x7f65581d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085819664 0 0) 0x7f655813b480 con 0x7f655806e7b0 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): my global_id is 194415 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6544004eb0 con 0x7f655806e7b0 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1358683132 0 0) 0x7f6558134580 con 0x7f65580a5d50 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): my global_id is 136384 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f655813b480 con 0x7f65580a5d50 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3950872671 0 0) 0x7f655813b1a0 con 0x7f65580a5920 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): my global_id is 184658 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6558134580 con 0x7f65580a5920 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3420676622 0 0) 0x7f6544004eb0 con 0x7f655806e7b0 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f655813b1a0 con 0x7f655806e7b0 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6548001530 con 0x7f655806e7b0 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): dump: 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.624 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.625 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.602+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3257014193 0 0) 0x7f655813b480 con 0x7f65580a5d50 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6544004eb0 con 0x7f65580a5d50 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f65500020b0 con 0x7f65580a5d50 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient(hunting): dump: 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.626 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.627 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3046824181 0 0) 0x7f6558134580 con 0x7f65580a5920 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f655813b480 con 0x7f65580a5920 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f654c004b20 con 0x7f65580a5920 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient(hunting): dump: 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.628 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3776344489 0 0) 0x7f655813b1a0 con 0x7f655806e7b0 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.104:6792/0 conn(0x7f65580a5d50 legacy=0x7f65581d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.104:6791/0 conn(0x7f65580a5920 legacy=0x7f65581d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient: _finish_hunting 0 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 1 monclient: found mon.d 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655813aec0 con 0x7f655806e7b0 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient: _finish_auth 0 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient: _check_auth_tickets 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f655dcbc640 5 monclient: authenticate success, global_id 194415 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f655dcbc640 10 monclient: _renew_subs 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f655dcbc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f655dcbc640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65581387d0 con 0x7f655806e7b0 2024-08-19T12:56:08.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f655dcbc640 10 client.194415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f655dcbc640 10 monclient: _renew_subs 2024-08-19T12:56:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f655dcbc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:56:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f655dcbc640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f65581cebc0 con 0x7f655806e7b0 2024-08-19T12:56:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f655dcbc640 1 librados: init done 2024-08-19T12:56:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6548003b30 con 0x7f655806e7b0 2024-08-19T12:56:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6548004060 con 0x7f655806e7b0 2024-08-19T12:56:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f6555ffb640 4 set_mon_vals no callback set 2024-08-19T12:56:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:08.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.603+0000 7f653ffff640 10 monclient: dump: 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.632 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.604+0000 7f655dcbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.604+0000 7f655dcbc640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f65380065c0 con 0x7f6538002860 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.604+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6548004200 con 0x7f655806e7b0 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.604+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(657..657 src has 350..657) ==== 5939+0+0 (unknown 4211950173 0 0) 0x7f654804e430 con 0x7f655806e7b0 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.604+0000 7f653ffff640 10 client.194415.objecter ms_dispatch 0x7f6558000b90 osd_map(657..657 src has 350..657) 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.604+0000 7f653ffff640 3 client.194415.objecter handle_osd_map got epochs [657,657] > 0 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.604+0000 7f653ffff640 3 client.194415.objecter handle_osd_map decoding full epoch 657 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.604+0000 7f653ffff640 20 client.194415.objecter dump_active .. 0 homeless 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.604+0000 7f6557fff640 10 monclient: get_auth_request con 0x7f6544041aa0 auth_method 0 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f6556ffd640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.26:6793/0 conn(0x7f6538002860 legacy=0x7f6538004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 10 client.194415.objecter ms_handle_connect 0x7f6538002860 2024-08-19T12:56:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 10 client.194415.objecter resend_mon_ops 2024-08-19T12:56:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 306285549 0 0) 0x7f65380065c0 con 0x7f6538002860 2024-08-19T12:56:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 10 monclient(hunting): my global_id is 184589 2024-08-19T12:56:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f655813b1a0 con 0x7f6538002860 2024-08-19T12:56:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 428627005 0 0) 0x7f655813b1a0 con 0x7f6538002860 2024-08-19T12:56:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65380065c0 con 0x7f6538002860 2024-08-19T12:56:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f654c001bc0 con 0x7f6538002860 2024-08-19T12:56:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 10 monclient: dump: 2024-08-19T12:56:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.635 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1768079230 0 0) 0x7f65380065c0 con 0x7f6538002860 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f653800a160 con 0x7f6538002860 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.606+0000 7f653ffff640 10 client.194415.objecter ms_handle_connect 0x7f6544041aa0 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.607+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f654c0051a0 con 0x7f6538002860 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.607+0000 7f653ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.607+0000 7f653ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.607+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.26:6793/0 conn(0x7f6538002860 legacy=0x7f6538004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.607+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.26:6793/0 conn(0x7f6538002860 legacy=0x7f6538004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.613+0000 7f655dcbc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.613+0000 7f655dcbc640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6538009e30 con 0x7f6538009110 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f6556ffd640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.26:6793/0 conn(0x7f6538009110 legacy=0x7f6538009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 10 client.194415.objecter ms_handle_connect 0x7f6538009110 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 10 client.194415.objecter resend_mon_ops 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 776266211 0 0) 0x7f6538009e30 con 0x7f6538009110 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 10 monclient(hunting): my global_id is 184598 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65380065c0 con 0x7f6538009110 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4172629856 0 0) 0x7f65380065c0 con 0x7f6538009110 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6538009e30 con 0x7f6538009110 2024-08-19T12:56:08.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f654c005990 con 0x7f6538009110 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.615+0000 7f653ffff640 10 monclient: dump: 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.640 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.641 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 970841512 0 0) 0x7f6538009e30 con 0x7f6538009110 2024-08-19T12:56:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f653800c3f0 con 0x7f6538009110 2024-08-19T12:56:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7712 (unknown 0 0 1880668767) 0x7f654c008650 con 0x7f6538009110 2024-08-19T12:56:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f653ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f653ffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.26:6793/0 conn(0x7f6538009110 legacy=0x7f6538009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f653ffff640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.26:6793/0 conn(0x7f6538009110 legacy=0x7f6538009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.642 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 294082, 2024-08-19T12:56:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:08.642 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:08.643 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:08.644 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.645 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:08.646 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.647 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.649 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.650 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:08.651 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.652 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:08.655 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.656 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.658 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:56:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.662 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:08.663 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:08.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f655dcbc640 10 librados: watch_flush enter 2024-08-19T12:56:08.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f655dcbc640 10 librados: watch_flush exit 2024-08-19T12:56:08.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f655dcbc640 20 client.194415.objecter shutdown clearing up homeless session... 2024-08-19T12:56:08.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f655dcbc640 10 client.194415.objecter successfully canceled tick 2024-08-19T12:56:08.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f655dcbc640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.104:6800/899887506 conn(0x7f6544041aa0 legacy=0x7f6544043f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.665 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f655dcbc640 10 monclient: shutdown 2024-08-19T12:56:08.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f655dcbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:08.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f655dcbc640 1 -- 172.21.15.26:0/2472651109 >> v1:172.21.15.26:6792/0 conn(0x7f655806e7b0 legacy=0x7f65581ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f6557fff640 1 -- 172.21.15.26:0/2472651109 reap_dead start 2024-08-19T12:56:08.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f6557fff640 1 -- 172.21.15.26:0/2472651109 reap_dead start 2024-08-19T12:56:08.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f655dcbc640 1 -- 172.21.15.26:0/2472651109 shutdown_connections 2024-08-19T12:56:08.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f655dcbc640 1 -- 172.21.15.26:0/2472651109 >> 172.21.15.26:0/2472651109 conn(0x7f65580a8740 msgr2=0x7f655806dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:08.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f655dcbc640 1 -- 172.21.15.26:0/2472651109 shutdown_connections 2024-08-19T12:56:08.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.616+0000 7f655dcbc640 1 -- 172.21.15.26:0/2472651109 wait complete. 2024-08-19T12:56:08.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.617+0000 7f655dcbc640 1 librados: shutdown 2024-08-19T12:56:08.666 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:08.667 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-08-19T12:56:08.667 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:56:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589161398}) 2024-08-19T12:56:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2365076546}) 2024-08-19T12:56:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.702 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589161398}) 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191053529}) 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589161398}) 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1772718988}) 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589161398}) 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2587823674}) 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589161398}) 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3305070535}) 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589161398}) 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1043099132}) 2024-08-19T12:56:08.703 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589161398}) 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1046577884}) 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589161398}) 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.690+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2899663594}) 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736122712}) 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125747702}) 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736122712}) 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3889848759}) 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736122712}) 2024-08-19T12:56:08.704 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3207597523}) 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736122712}) 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3382245813}) 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736122712}) 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4280364212}) 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736122712}) 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3785694360}) 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736122712}) 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3839229352}) 2024-08-19T12:56:08.705 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736122712}) 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.691+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2228138702}) 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607986600}) 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3002192670}) 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607986600}) 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=391459024}) 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607986600}) 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=425433291}) 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607986600}) 2024-08-19T12:56:08.706 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3353045269}) 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607986600}) 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=501302412}) 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607986600}) 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=534032914}) 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607986600}) 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=770018932}) 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607986600}) 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.693+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2130163032}) 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.707 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876806628}) 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1987657834}) 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876806628}) 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=947595575}) 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876806628}) 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241703323}) 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876806628}) 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2951806317}) 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876806628}) 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1268746680}) 2024-08-19T12:56:08.708 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876806628}) 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1506812110}) 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876806628}) 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2350037862}) 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876806628}) 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.694+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2267627027}) 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.695+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.695+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372958286}) 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.695+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1360917904}) 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.695+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.709 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.695+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372958286}) 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.695+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2396571547}) 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.695+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.695+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372958286}) 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.695+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=976157904}) 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372958286}) 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3520753737}) 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372958286}) 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1591968360}) 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372958286}) 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3249114025}) 2024-08-19T12:56:08.710 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372958286}) 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2986023636}) 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372958286}) 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.696+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3194552173}) 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788911010}) 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=478730593}) 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788911010}) 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=442647985}) 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.711 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788911010}) 2024-08-19T12:56:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1714516314}) 2024-08-19T12:56:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788911010}) 2024-08-19T12:56:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3573684606}) 2024-08-19T12:56:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788911010}) 2024-08-19T12:56:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2883786138}) 2024-08-19T12:56:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788911010}) 2024-08-19T12:56:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3197413603}) 2024-08-19T12:56:08.712 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788911010}) 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4049725544}) 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788911010}) 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.697+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1596293515}) 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.698+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.698+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538230986}) 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.698+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720081895}) 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.698+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.698+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538230986}) 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.698+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2727523473}) 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.698+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.713 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.698+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538230986}) 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1491008698}) 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538230986}) 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2409388631}) 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538230986}) 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2039399711}) 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538230986}) 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1993740080}) 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538230986}) 2024-08-19T12:56:08.714 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1063544236}) 2024-08-19T12:56:08.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538230986}) 2024-08-19T12:56:08.715 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.699+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3477060127}) 2024-08-19T12:56:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.769+0000 7fb64ad28640 10 monclient: get_monmap_and_config 2024-08-19T12:56:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.769+0000 7fb64ad28640 10 monclient: build_initial_monmap 2024-08-19T12:56:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.769+0000 7fb64ad28640 10 monclient: monmap: 2024-08-19T12:56:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.770 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:08.770431+0000 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:08.770431+0000 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:08.771 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.770+0000 7fb64ad28640 1 Processor -- start 2024-08-19T12:56:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.770+0000 7fb64ad28640 1 -- start start 2024-08-19T12:56:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.770+0000 7fb64ad28640 10 monclient: init 2024-08-19T12:56:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.770+0000 7fb64ad28640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.770+0000 7fb64ad28640 10 monclient: _add_conns ranks=[2,5,4,1,0,8,6,3,7] 2024-08-19T12:56:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.770+0000 7fb64ad28640 10 monclient(hunting): picked mon.noname-b con 0x7fb64406e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:56:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.770+0000 7fb64ad28640 10 monclient(hunting): picked mon.noname-h con 0x7fb6440a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.770+0000 7fb64ad28640 10 monclient(hunting): picked mon.noname-c con 0x7fb6440a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.770+0000 7fb64ad28640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb644136790 con 0x7fb64406e7b0 2024-08-19T12:56:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.770+0000 7fb64ad28640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb64409ddf0 con 0x7fb6440a5d50 2024-08-19T12:56:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.771+0000 7fb64ad28640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb64409e0d0 con 0x7fb6440a5920 2024-08-19T12:56:08.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.771+0000 7fb64ad28640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.771+0000 7fb64ad28640 10 monclient(hunting): authenticate will time out at 1762.858032s 2024-08-19T12:56:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.771+0000 7fb648ac4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb64406e7b0 0x7fb644136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39046/0 (socket says 172.21.15.26:39046) 2024-08-19T12:56:08.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.771+0000 7fb648ac4640 1 -- 172.21.15.26:0/789301736 learned_addr learned my addr 172.21.15.26:0/789301736 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb648ac4640 1 -- 172.21.15.26:0/789301736 >> v1:172.21.15.26:6790/0 conn(0x7fb64406e7b0 legacy=0x7fb644136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6492c5640 1 -- 172.21.15.26:0/789301736 >> v1:172.21.15.26:6791/0 conn(0x7fb6440a5d50 legacy=0x7fb644141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2871073126 0 0) 0x7fb644136790 con 0x7fb64406e7b0 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient(hunting): my global_id is 27416 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb62c003520 con 0x7fb64406e7b0 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 343340905 0 0) 0x7fb63c0014a0 con 0x7fb6440a5d50 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient(hunting): my global_id is 184765 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb644136790 con 0x7fb6440a5d50 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb643fff640 1 -- 172.21.15.26:0/789301736 >> v1:172.21.15.104:6791/0 conn(0x7fb6440a5920 legacy=0x7fb64413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 352943651 0 0) 0x7fb64409e0d0 con 0x7fb6440a5920 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient(hunting): my global_id is 184667 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb63c0014a0 con 0x7fb6440a5920 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1297650584 0 0) 0x7fb62c003520 con 0x7fb64406e7b0 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 >> v1:172.21.15.104:6791/0 conn(0x7fb6440a5920 legacy=0x7fb64413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 >> v1:172.21.15.26:6791/0 conn(0x7fb6440a5d50 legacy=0x7fb644141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient: _finish_hunting 0 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 monclient: found mon.noname-b 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64409e3b0 con 0x7fb64406e7b0 2024-08-19T12:56:08.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient: _finish_auth 0 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient: _check_auth_tickets 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb634001e60 con 0x7fb64406e7b0 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb64ad28640 5 monclient: authenticate success, global_id 27416 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb64ad28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient: dump: 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.780 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb64ad28640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb634004c20 con 0x7fb64406e7b0 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:08.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 1 -- 172.21.15.26:0/789301736 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb634005190 con 0x7fb64406e7b0 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb6437fe640 10 monclient: dump: 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.782 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb64ad28640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.773+0000 7fb64ad28640 4 set_mon_vals no callback set 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.774+0000 7fb64ad28640 10 monclient: shutdown 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.774+0000 7fb64ad28640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.774+0000 7fb64ad28640 1 -- 172.21.15.26:0/789301736 >> v1:172.21.15.26:6790/0 conn(0x7fb64406e7b0 legacy=0x7fb644136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.774+0000 7fb64ad28640 1 -- 172.21.15.26:0/789301736 shutdown_connections 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.774+0000 7fb64ad28640 1 -- 172.21.15.26:0/789301736 >> 172.21.15.26:0/789301736 conn(0x7fb6440a8740 msgr2=0x7fb6440a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.774+0000 7fb64ad28640 1 -- 172.21.15.26:0/789301736 shutdown_connections 2024-08-19T12:56:08.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.774+0000 7fb64ad28640 1 -- 172.21.15.26:0/789301736 wait complete. 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.774+0000 7fb64ad28640 10 monclient: build_initial_monmap 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.774+0000 7fb64ad28640 10 monclient: monmap: 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:08.784 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 1 librados: starting msgr at 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 1 librados: starting objecter 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 1 Processor -- start 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 1 -- start start 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 1 librados: setting wanted keys 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 1 librados: calling monclient init 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 10 monclient: init 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 10 monclient: _add_conns ranks=[2,0,8,6,7,5,3,4,1] 2024-08-19T12:56:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 10 monclient(hunting): picked mon.noname-b con 0x7fb64406e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:56:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 10 monclient(hunting): picked mon.noname-a con 0x7fb6440a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 10 monclient(hunting): picked mon.noname-e con 0x7fb6440a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.775+0000 7fb64ad28640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb64413aaf0 con 0x7fb6440a5920 2024-08-19T12:56:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.776+0000 7fb64ad28640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb644142e30 con 0x7fb64406e7b0 2024-08-19T12:56:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.776+0000 7fb64ad28640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb644139030 con 0x7fb6440a5d50 2024-08-19T12:56:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.776+0000 7fb64ad28640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.776+0000 7fb64ad28640 10 monclient(hunting): authenticate will time out at 1762.863032s 2024-08-19T12:56:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.776+0000 7fb648ac4640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb64406e7b0 0x7fb644137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39060/0 (socket says 172.21.15.26:39060) 2024-08-19T12:56:08.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.776+0000 7fb648ac4640 1 -- 172.21.15.26:0/2159825504 learned_addr learned my addr 172.21.15.26:0/2159825504 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:08.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb648ac4640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.26:6790/0 conn(0x7fb64406e7b0 legacy=0x7fb644137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 client.?.objecter ms_handle_connect 0x7fb64406e7b0 2024-08-19T12:56:08.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb643fff640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.26:6789/0 conn(0x7fb6440a5920 legacy=0x7fb644137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6492c5640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.26:6793/0 conn(0x7fb6440a5d50 legacy=0x7fb6441d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 client.?.objecter ms_handle_connect 0x7fb6440a5920 2024-08-19T12:56:08.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 client.?.objecter ms_handle_connect 0x7fb6440a5d50 2024-08-19T12:56:08.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4291337305 0 0) 0x7fb644142e30 con 0x7fb64406e7b0 2024-08-19T12:56:08.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): my global_id is 27425 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb630004eb0 con 0x7fb64406e7b0 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 806336220 0 0) 0x7fb63c0014a0 con 0x7fb6440a5920 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): my global_id is 194454 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb644142e30 con 0x7fb6440a5920 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4272814257 0 0) 0x7fb644139030 con 0x7fb6440a5d50 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): my global_id is 184607 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb63c0014a0 con 0x7fb6440a5d50 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1705657744 0 0) 0x7fb630004eb0 con 0x7fb64406e7b0 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb644139030 con 0x7fb64406e7b0 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb634005110 con 0x7fb64406e7b0 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): dump: 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.794 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2908538272 0 0) 0x7fb644142e30 con 0x7fb6440a5920 2024-08-19T12:56:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb630004eb0 con 0x7fb6440a5920 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb638001d10 con 0x7fb6440a5920 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): dump: 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.796 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3671709245 0 0) 0x7fb63c0014a0 con 0x7fb6440a5d50 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb644142e30 con 0x7fb6440a5d50 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb63c003680 con 0x7fb6440a5d50 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient(hunting): dump: 2024-08-19T12:56:08.797 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.798 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.798 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.798 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.798 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.798 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.798 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.798 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.798 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.798 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.798 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.798 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.798 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 510089356 0 0) 0x7fb644139030 con 0x7fb64406e7b0 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.26:6793/0 conn(0x7fb6440a5d50 legacy=0x7fb6441d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.26:6789/0 conn(0x7fb6440a5920 legacy=0x7fb644137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient: _finish_hunting 0 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 monclient: found mon.b 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6441384a0 con 0x7fb64406e7b0 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient: _finish_auth 0 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 10 monclient: _check_auth_tickets 2024-08-19T12:56:08.799 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb6417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:08.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb64ad28640 5 monclient: authenticate success, global_id 27425 2024-08-19T12:56:08.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb64ad28640 10 monclient: _renew_subs 2024-08-19T12:56:08.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb64ad28640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:56:08.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.778+0000 7fb64ad28640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb64413b390 con 0x7fb64406e7b0 2024-08-19T12:56:08.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb64ad28640 10 client.27425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:08.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb64ad28640 10 monclient: _renew_subs 2024-08-19T12:56:08.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb64ad28640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:56:08.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb64ad28640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb644142a70 con 0x7fb64406e7b0 2024-08-19T12:56:08.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb64ad28640 1 librados: init done 2024-08-19T12:56:08.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6340020b0 con 0x7fb64406e7b0 2024-08-19T12:56:08.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb6417fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:08.800 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb634004c20 con 0x7fb64406e7b0 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb6417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb642ffd640 4 set_mon_vals no callback set 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb6417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb6417fa640 10 monclient: dump: 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.801 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.802 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.802 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.802 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.802 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.802 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.802 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb64ad28640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:08.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb64ad28640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb6240065c0 con 0x7fb624002860 2024-08-19T12:56:08.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.779+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fb634011d80 con 0x7fb64406e7b0 2024-08-19T12:56:08.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.780+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(657..657 src has 350..657) ==== 5939+0+0 (unknown 4211950173 0 0) 0x7fb63404e7d0 con 0x7fb64406e7b0 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.780+0000 7fb6417fa640 10 client.27425.objecter ms_dispatch 0x7fb644000b90 osd_map(657..657 src has 350..657) 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.780+0000 7fb6417fa640 3 client.27425.objecter handle_osd_map got epochs [657,657] > 0 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.780+0000 7fb6417fa640 3 client.27425.objecter handle_osd_map decoding full epoch 657 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.780+0000 7fb6417fa640 20 client.27425.objecter dump_active .. 0 homeless 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.780+0000 7fb6492c5640 10 monclient: get_auth_request con 0x7fb630041ac0 auth_method 0 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 10 client.27425.objecter ms_handle_connect 0x7fb624002860 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb643fff640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.104:6789/0 conn(0x7fb624002860 legacy=0x7fb624004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 10 client.27425.objecter resend_mon_ops 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4111503380 0 0) 0x7fb6240065c0 con 0x7fb624002860 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 10 monclient(hunting): my global_id is 204247 2024-08-19T12:56:08.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb644139030 con 0x7fb624002860 2024-08-19T12:56:08.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 10 client.27425.objecter ms_handle_connect 0x7fb630041ac0 2024-08-19T12:56:08.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 364140111 0 0) 0x7fb644139030 con 0x7fb624002860 2024-08-19T12:56:08.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6240065c0 con 0x7fb624002860 2024-08-19T12:56:08.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb638003d10 con 0x7fb624002860 2024-08-19T12:56:08.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:08.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 10 monclient: dump: 2024-08-19T12:56:08.804 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.804 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.804 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.804 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2080935926 0 0) 0x7fb6240065c0 con 0x7fb624002860 2024-08-19T12:56:08.805 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.782+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb62400a160 con 0x7fb624002860 2024-08-19T12:56:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.783+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb638003f80 con 0x7fb624002860 2024-08-19T12:56:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.783+0000 7fb6417fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.783+0000 7fb6417fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.783+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.104:6789/0 conn(0x7fb624002860 legacy=0x7fb624004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.783+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.104:6789/0 conn(0x7fb624002860 legacy=0x7fb624004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.789+0000 7fb64ad28640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:08.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.789+0000 7fb64ad28640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb624009e50 con 0x7fb624009110 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.791+0000 7fb643fff640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.104:6789/0 conn(0x7fb624009110 legacy=0x7fb6240095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.791+0000 7fb6417fa640 10 client.27425.objecter ms_handle_connect 0x7fb624009110 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.791+0000 7fb6417fa640 10 client.27425.objecter resend_mon_ops 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797781088 0 0) 0x7fb624009e50 con 0x7fb624009110 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 10 monclient(hunting): my global_id is 204256 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6240065c0 con 0x7fb624009110 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 305216982 0 0) 0x7fb6240065c0 con 0x7fb624009110 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb624009e50 con 0x7fb624009110 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb638005b00 con 0x7fb624009110 2024-08-19T12:56:08.808 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 10 monclient: dump: 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.810 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.810 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.810 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.810 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.810 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2380500135 0 0) 0x7fb624009e50 con 0x7fb624009110 2024-08-19T12:56:08.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb62400c3a0 con 0x7fb624009110 2024-08-19T12:56:08.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7544 (unknown 0 0 765777650) 0x7fb638007d80 con 0x7fb624009110 2024-08-19T12:56:08.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:08.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:08.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.104:6789/0 conn(0x7fb624009110 legacy=0x7fb6240095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.792+0000 7fb6417fa640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.104:6789/0 conn(0x7fb624009110 legacy=0x7fb6240095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.811 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:08.811 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:08.811 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:08.811 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:08.811 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:08.811 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 21266, 2024-08-19T12:56:08.811 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:08.811 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:08.811 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:08.811 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:08.811 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.812 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.813 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:08.814 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:08.815 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.816 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.817 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.819 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:08.820 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:08.823 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.826 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.827 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.829 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:08.830 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:08.830 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 10 librados: watch_flush enter 2024-08-19T12:56:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 10 librados: watch_flush exit 2024-08-19T12:56:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 20 client.27425.objecter shutdown clearing up homeless session... 2024-08-19T12:56:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 10 client.27425.objecter successfully canceled tick 2024-08-19T12:56:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.104:6800/899887506 conn(0x7fb630041ac0 legacy=0x7fb630043f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 10 monclient: shutdown 2024-08-19T12:56:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 1 -- 172.21.15.26:0/2159825504 >> v1:172.21.15.26:6790/0 conn(0x7fb64406e7b0 legacy=0x7fb644137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb6492c5640 1 -- 172.21.15.26:0/2159825504 reap_dead start 2024-08-19T12:56:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb6492c5640 1 -- 172.21.15.26:0/2159825504 reap_dead start 2024-08-19T12:56:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 1 -- 172.21.15.26:0/2159825504 shutdown_connections 2024-08-19T12:56:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 1 -- 172.21.15.26:0/2159825504 >> 172.21.15.26:0/2159825504 conn(0x7fb6440a8740 msgr2=0x7fb6440aa580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 1 -- 172.21.15.26:0/2159825504 shutdown_connections 2024-08-19T12:56:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 1 -- 172.21.15.26:0/2159825504 wait complete. 2024-08-19T12:56:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.793+0000 7fb64ad28640 1 librados: shutdown 2024-08-19T12:56:08.833 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:08.834 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-08-19T12:56:08.834 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-08-19T12:56:08.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859350659}) 2024-08-19T12:56:08.855 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1867143644}) 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859350659}) 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4175601938}) 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859350659}) 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2586216278}) 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859350659}) 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3755095872}) 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859350659}) 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=236094483}) 2024-08-19T12:56:08.856 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.857 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859350659}) 2024-08-19T12:56:08.857 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1417020278}) 2024-08-19T12:56:08.857 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.857 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859350659}) 2024-08-19T12:56:08.857 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2103208139}) 2024-08-19T12:56:08.857 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.857 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859350659}) 2024-08-19T12:56:08.857 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.853+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1904230587}) 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833542250}) 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246635730}) 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833542250}) 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2616413713}) 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833542250}) 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2834091394}) 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833542250}) 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=156490117}) 2024-08-19T12:56:08.860 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.861 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833542250}) 2024-08-19T12:56:08.861 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2061493929}) 2024-08-19T12:56:08.861 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.861 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833542250}) 2024-08-19T12:56:08.861 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3634480243}) 2024-08-19T12:56:08.861 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.861 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833542250}) 2024-08-19T12:56:08.861 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1198093744}) 2024-08-19T12:56:08.861 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.861 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833542250}) 2024-08-19T12:56:08.861 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.858+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=715734763}) 2024-08-19T12:56:08.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833680566}) 2024-08-19T12:56:08.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=763646003}) 2024-08-19T12:56:08.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833680566}) 2024-08-19T12:56:08.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760397648}) 2024-08-19T12:56:08.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.946 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833680566}) 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1457117632}) 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833680566}) 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1290628254}) 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833680566}) 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2702178531}) 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833680566}) 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=821695624}) 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833680566}) 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1539852213}) 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.947 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833680566}) 2024-08-19T12:56:08.948 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.943+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3594832806}) 2024-08-19T12:56:08.958 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.958 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198304897}) 2024-08-19T12:56:08.958 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2874204572}) 2024-08-19T12:56:08.958 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198304897}) 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1510943576}) 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198304897}) 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=113398939}) 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198304897}) 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2816307759}) 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198304897}) 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2050401864}) 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.959 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198304897}) 2024-08-19T12:56:08.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1711588196}) 2024-08-19T12:56:08.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198304897}) 2024-08-19T12:56:08.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2615327505}) 2024-08-19T12:56:08.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:08.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198304897}) 2024-08-19T12:56:08.960 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:08.957+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=902710786}) 2024-08-19T12:56:08.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.964+0000 7f7eb7733640 10 monclient: get_monmap_and_config 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.964+0000 7f7eb7733640 10 monclient: build_initial_monmap 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.964+0000 7f7eb7733640 10 monclient: monmap: 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:08.965152+0000 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:08.965152+0000 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:08.965 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:08.966 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:08.966 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:08.966 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:08.966 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:08.966 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.964+0000 7f7eb7733640 1 Processor -- start 2024-08-19T12:56:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb7733640 1 -- start start 2024-08-19T12:56:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb7733640 10 monclient: init 2024-08-19T12:56:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb7733640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb7733640 10 monclient: _add_conns ranks=[2,6,7,4,3,5,8,0,1] 2024-08-19T12:56:08.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb7733640 10 monclient(hunting): picked mon.noname-b con 0x7f7eb0073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:56:08.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb7733640 10 monclient(hunting): picked mon.noname-d con 0x7f7eb00a57a0 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:08.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb7733640 10 monclient(hunting): picked mon.noname-i con 0x7f7eb013f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:08.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb7733640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eb009dae0 con 0x7f7eb0073470 2024-08-19T12:56:08.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb7733640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eb0136910 con 0x7f7eb00a57a0 2024-08-19T12:56:08.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb7733640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eb009dd50 con 0x7f7eb013f2e0 2024-08-19T12:56:08.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb7733640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:08.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb7733640 10 monclient(hunting): authenticate will time out at 1763.052599s 2024-08-19T12:56:08.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb4cce640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f7eb00a57a0 0x7f7eb00a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:44398/0 (socket says 172.21.15.26:44398) 2024-08-19T12:56:08.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb54cf640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f7eb0073470 0x7f7eb0136200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39072/0 (socket says 172.21.15.26:39072) 2024-08-19T12:56:08.968 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.965+0000 7f7eb4cce640 1 -- 172.21.15.26:0/137995294 learned_addr learned my addr 172.21.15.26:0/137995294 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.967+0000 7f7eb4cce640 1 -- 172.21.15.26:0/137995294 >> v1:172.21.15.26:6792/0 conn(0x7f7eb00a57a0 legacy=0x7f7eb00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.967+0000 7f7eb54cf640 1 -- 172.21.15.26:0/137995294 >> v1:172.21.15.26:6790/0 conn(0x7f7eb0073470 legacy=0x7f7eb0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7eb5cd0640 1 -- 172.21.15.26:0/137995294 >> v1:172.21.15.104:6792/0 conn(0x7f7eb013f2e0 legacy=0x7f7eb01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 1 -- 172.21.15.26:0/137995294 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3551592792 0 0) 0x7f7eb009dae0 con 0x7f7eb0073470 2024-08-19T12:56:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient(hunting): my global_id is 27434 2024-08-19T12:56:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 1 -- 172.21.15.26:0/137995294 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e9c003520 con 0x7f7eb0073470 2024-08-19T12:56:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 1 -- 172.21.15.26:0/137995294 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3049519134 0 0) 0x7f7eb0136910 con 0x7f7eb00a57a0 2024-08-19T12:56:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient(hunting): my global_id is 194424 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 1 -- 172.21.15.26:0/137995294 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eb009dae0 con 0x7f7eb00a57a0 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 1 -- 172.21.15.26:0/137995294 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2040595776 0 0) 0x7f7e9c003520 con 0x7f7eb0073470 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 1 -- 172.21.15.26:0/137995294 >> v1:172.21.15.104:6792/0 conn(0x7f7eb013f2e0 legacy=0x7f7eb01416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 1 -- 172.21.15.26:0/137995294 >> v1:172.21.15.26:6792/0 conn(0x7f7eb00a57a0 legacy=0x7f7eb00a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient: _finish_hunting 0 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 1 monclient: found mon.noname-b 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 1 -- 172.21.15.26:0/137995294 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb009e030 con 0x7f7eb0073470 2024-08-19T12:56:08.973 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient: _finish_auth 0 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient: _check_auth_tickets 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 1 -- 172.21.15.26:0/137995294 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7eac001ae0 con 0x7f7eb0073470 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7eb7733640 5 monclient: authenticate success, global_id 27434 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient: dump: 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.974 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7eb7733640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 1 -- 172.21.15.26:0/137995294 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7eac0047e0 con 0x7f7eb0073470 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:08.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 1 -- 172.21.15.26:0/137995294 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7eac004d50 con 0x7f7eb0073470 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7ea7fff640 10 monclient: dump: 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.976 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7eb7733640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.968+0000 7f7eb7733640 4 set_mon_vals no callback set 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.969+0000 7f7eb7733640 10 monclient: shutdown 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.969+0000 7f7eb7733640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.969+0000 7f7eb7733640 1 -- 172.21.15.26:0/137995294 >> v1:172.21.15.26:6790/0 conn(0x7f7eb0073470 legacy=0x7f7eb0136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.969+0000 7f7eb7733640 1 -- 172.21.15.26:0/137995294 shutdown_connections 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.969+0000 7f7eb7733640 1 -- 172.21.15.26:0/137995294 >> 172.21.15.26:0/137995294 conn(0x7f7eb00abff0 msgr2=0x7f7eb00ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:08.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.969+0000 7f7eb7733640 1 -- 172.21.15.26:0/137995294 shutdown_connections 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.969+0000 7f7eb7733640 1 -- 172.21.15.26:0/137995294 wait complete. 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.969+0000 7f7eb7733640 10 monclient: build_initial_monmap 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.969+0000 7f7eb7733640 10 monclient: monmap: 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:08.978 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.969+0000 7f7eb7733640 1 librados: starting msgr at 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.969+0000 7f7eb7733640 1 librados: starting objecter 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 1 Processor -- start 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 1 -- start start 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 1 librados: setting wanted keys 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 1 librados: calling monclient init 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 10 monclient: init 2024-08-19T12:56:08.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 10 monclient: _add_conns ranks=[7,8,0,1,2,5,3,4,6] 2024-08-19T12:56:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 10 monclient(hunting): picked mon.noname-i con 0x7f7eb0073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 10 monclient(hunting): picked mon.noname-e con 0x7f7eb00a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 10 monclient(hunting): picked mon.noname-a con 0x7f7eb013f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eb0131f00 con 0x7f7eb013f2e0 2024-08-19T12:56:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eb01cef20 con 0x7f7eb0073470 2024-08-19T12:56:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eb0131800 con 0x7f7eb00a57a0 2024-08-19T12:56:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb7733640 10 monclient(hunting): authenticate will time out at 1763.057552s 2024-08-19T12:56:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb4cce640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f7eb00a57a0 0x7f7eb01ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:52848/0 (socket says 172.21.15.26:52848) 2024-08-19T12:56:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb4cce640 1 -- 172.21.15.26:0/1779540285 learned_addr learned my addr 172.21.15.26:0/1779540285 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.970+0000 7f7eb5cd0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7eb013f2e0 0x7f7eb01d6360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39208/0 (socket says 172.21.15.26:39208) 2024-08-19T12:56:08.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7eb5cd0640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.26:6789/0 conn(0x7f7eb013f2e0 legacy=0x7f7eb01d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7eb4cce640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.26:6793/0 conn(0x7f7eb00a57a0 legacy=0x7f7eb01ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 client.?.objecter ms_handle_connect 0x7f7eb013f2e0 2024-08-19T12:56:08.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 client.?.objecter ms_handle_connect 0x7f7eb00a57a0 2024-08-19T12:56:08.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7eb54cf640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.104:6792/0 conn(0x7f7eb0073470 legacy=0x7f7eb01ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 client.?.objecter ms_handle_connect 0x7f7eb0073470 2024-08-19T12:56:08.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:08.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2681458605 0 0) 0x7f7eb009dd50 con 0x7f7eb013f2e0 2024-08-19T12:56:08.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 monclient(hunting): my global_id is 194463 2024-08-19T12:56:08.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e94004eb0 con 0x7f7eb013f2e0 2024-08-19T12:56:08.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 286466635 0 0) 0x7f7eb0131800 con 0x7f7eb00a57a0 2024-08-19T12:56:08.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 10 monclient(hunting): my global_id is 184616 2024-08-19T12:56:08.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.972+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eb009dd50 con 0x7f7eb00a57a0 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3812136306 0 0) 0x7f7eb01cef20 con 0x7f7eb0073470 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient(hunting): my global_id is 136402 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eb0131800 con 0x7f7eb0073470 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2209850954 0 0) 0x7f7e94004eb0 con 0x7f7eb013f2e0 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7eb01cef20 con 0x7f7eb013f2e0 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7ea8002030 con 0x7f7eb013f2e0 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient(hunting): dump: 2024-08-19T12:56:08.988 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.989 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2423552535 0 0) 0x7f7eb009dd50 con 0x7f7eb00a57a0 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e94004eb0 con 0x7f7eb00a57a0 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7ea00047e0 con 0x7f7eb00a57a0 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient(hunting): dump: 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.0 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 731210067 0 0) 0x7f7eb01cef20 con 0x7f7eb013f2e0 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.26:6793/0 conn(0x7f7eb00a57a0 legacy=0x7f7eb01ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.104:6792/0 conn(0x7f7eb0073470 legacy=0x7f7eb01ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 monclient: found mon.a 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb0131ae0 con 0x7f7eb013f2e0 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient: _finish_auth 0 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7eb7733640 5 monclient: authenticate success, global_id 194463 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7eb7733640 10 monclient: _renew_subs 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7eb7733640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7eb7733640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7eb01331f0 con 0x7f7eb013f2e0 2024-08-19T12:56:08.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.0 v1:172.21.15.26:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7ea8003d50 con 0x7f7eb013f2e0 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.0 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7ea80042c0 con 0x7f7eb013f2e0 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea5ffb640 10 monclient: dump: 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7eb7733640 10 client.194463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7eb7733640 10 monclient: _renew_subs 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7ea77fe640 4 set_mon_vals no callback set 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7eb7733640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.26:6789/0 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7eb7733640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7eb01d7a80 con 0x7f7eb013f2e0 2024-08-19T12:56:08.994 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.973+0000 7f7eb7733640 1 librados: init done 2024-08-19T12:56:08.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.974+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.0 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f7ea8004460 con 0x7f7eb013f2e0 2024-08-19T12:56:08.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.974+0000 7f7eb7733640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:08.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.974+0000 7f7eb7733640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7eb0131600 con 0x7f7eb0142df0 2024-08-19T12:56:08.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.974+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.0 v1:172.21.15.26:6789/0 8 ==== osd_map(658..658 src has 350..658) ==== 5939+0+0 (unknown 1707548120 0 0) 0x7f7ea804e540 con 0x7f7eb013f2e0 2024-08-19T12:56:08.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.974+0000 7f7ea5ffb640 10 client.194463.objecter ms_dispatch 0x7f7eb0000b90 osd_map(658..658 src has 350..658) 2024-08-19T12:56:08.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.974+0000 7f7ea5ffb640 3 client.194463.objecter handle_osd_map got epochs [658,658] > 0 2024-08-19T12:56:08.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.974+0000 7f7ea5ffb640 3 client.194463.objecter handle_osd_map decoding full epoch 658 2024-08-19T12:56:08.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.974+0000 7f7ea5ffb640 20 client.194463.objecter dump_active .. 0 homeless 2024-08-19T12:56:08.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.974+0000 7f7eb4cce640 10 monclient: get_auth_request con 0x7f7e94041510 auth_method 0 2024-08-19T12:56:08.995 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.976+0000 7f7ea5ffb640 10 client.194463.objecter ms_handle_connect 0x7f7eb0142df0 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.976+0000 7f7eb54cf640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.104:6790/0 conn(0x7f7eb0142df0 legacy=0x7f7eb0133dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.976+0000 7f7ea5ffb640 10 client.194463.objecter resend_mon_ops 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.976+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2971724158 0 0) 0x7f7eb0131600 con 0x7f7eb0142df0 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.976+0000 7f7ea5ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.976+0000 7f7ea5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.976+0000 7f7ea5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.976+0000 7f7ea5ffb640 10 monclient(hunting): my global_id is 194412 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.976+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eb01cef20 con 0x7f7eb0142df0 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 10 client.194463.objecter ms_handle_connect 0x7f7e94041510 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3228504181 0 0) 0x7f7eb01cef20 con 0x7f7eb0142df0 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7eb0131600 con 0x7f7eb0142df0 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7eac001d30 con 0x7f7eb0142df0 2024-08-19T12:56:08.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:08.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:56:08.997 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 10 monclient: dump: 2024-08-19T12:56:08.997 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:08.997 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:08.997 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.997 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:08.997 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:08.997 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:08.997 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:08.997 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:08.997 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4030538878 0 0) 0x7f7eb0131600 con 0x7f7eb0142df0 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.104:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7eb0135270 con 0x7f7eb0142df0 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7eac0053d0 con 0x7f7eb0142df0 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:08.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.104:6790/0 conn(0x7f7eb0142df0 legacy=0x7f7eb0133dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:08.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.977+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.104:6790/0 conn(0x7f7eb0142df0 legacy=0x7f7eb0133dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:08.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.983+0000 7f7eb7733640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:08.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.983+0000 7f7eb7733640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7eb012d5c0 con 0x7f7eb002e1d0 2024-08-19T12:56:09.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7eb54cf640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.104:6790/0 conn(0x7f7eb002e1d0 legacy=0x7f7eb01344d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 10 client.194463.objecter ms_handle_connect 0x7f7eb002e1d0 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 10 client.194463.objecter resend_mon_ops 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1597198886 0 0) 0x7f7eb012d5c0 con 0x7f7eb002e1d0 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 10 monclient(hunting): my global_id is 194421 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7eb0131600 con 0x7f7eb002e1d0 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 308603215 0 0) 0x7f7eb0131600 con 0x7f7eb002e1d0 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7eb012d5c0 con 0x7f7eb002e1d0 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f7eac005bf0 con 0x7f7eb002e1d0 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-08-19T12:56:09.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 10 monclient: dump: 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.002 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.003 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.003 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.003 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.3 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3274353322 0 0) 0x7f7eb012d5c0 con 0x7f7eb002e1d0 2024-08-19T12:56:09.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.986+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 --> v1:172.21.15.104:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7eb0132730 con 0x7f7eb002e1d0 2024-08-19T12:56:09.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 <== mon.3 v1:172.21.15.104:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7545 (unknown 0 0 299801622) 0x7f7eac0081b0 con 0x7f7eb002e1d0 2024-08-19T12:56:09.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7ea5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:09.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7ea5ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:09.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.104:6790/0 conn(0x7f7eb002e1d0 legacy=0x7f7eb01344d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7ea5ffb640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.104:6790/0 conn(0x7f7eb002e1d0 legacy=0x7f7eb01344d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:09.003 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 153909, 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:09.004 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:09.005 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:09.006 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:09.007 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:09.008 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.009 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:09.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:09.013 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:09.017 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.018 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.021 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:09.024 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-08-19T12:56:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.025 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:09.025 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:09.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 10 librados: watch_flush enter 2024-08-19T12:56:09.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 10 librados: watch_flush exit 2024-08-19T12:56:09.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 20 client.194463.objecter shutdown clearing up homeless session... 2024-08-19T12:56:09.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 10 client.194463.objecter successfully canceled tick 2024-08-19T12:56:09.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.104:6800/899887506 conn(0x7f7e94041510 legacy=0x7f7e940439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 10 monclient: shutdown 2024-08-19T12:56:09.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:09.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 1 -- 172.21.15.26:0/1779540285 >> v1:172.21.15.26:6789/0 conn(0x7f7eb013f2e0 legacy=0x7f7eb01d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.027 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb5cd0640 1 -- 172.21.15.26:0/1779540285 reap_dead start 2024-08-19T12:56:09.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb5cd0640 1 -- 172.21.15.26:0/1779540285 reap_dead start 2024-08-19T12:56:09.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 1 -- 172.21.15.26:0/1779540285 shutdown_connections 2024-08-19T12:56:09.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 1 -- 172.21.15.26:0/1779540285 >> 172.21.15.26:0/1779540285 conn(0x7f7eb00abff0 msgr2=0x7f7eb012c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:09.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 1 -- 172.21.15.26:0/1779540285 shutdown_connections 2024-08-19T12:56:09.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 1 -- 172.21.15.26:0/1779540285 wait complete. 2024-08-19T12:56:09.028 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:08.987+0000 7f7eb7733640 1 librados: shutdown 2024-08-19T12:56:09.028 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:09.028 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-08-19T12:56:09.029 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:56:09.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136123404}) 2024-08-19T12:56:09.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3935757728}) 2024-08-19T12:56:09.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136123404}) 2024-08-19T12:56:09.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1662039724}) 2024-08-19T12:56:09.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.097 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136123404}) 2024-08-19T12:56:09.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3798263312}) 2024-08-19T12:56:09.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136123404}) 2024-08-19T12:56:09.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2953549423}) 2024-08-19T12:56:09.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136123404}) 2024-08-19T12:56:09.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=549781368}) 2024-08-19T12:56:09.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136123404}) 2024-08-19T12:56:09.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=767928977}) 2024-08-19T12:56:09.098 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136123404}) 2024-08-19T12:56:09.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=812163368}) 2024-08-19T12:56:09.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136123404}) 2024-08-19T12:56:09.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.087+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3763308506}) 2024-08-19T12:56:09.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54688204}) 2024-08-19T12:56:09.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209049438}) 2024-08-19T12:56:09.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54688204}) 2024-08-19T12:56:09.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1853875763}) 2024-08-19T12:56:09.099 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54688204}) 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1885627105}) 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54688204}) 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3025477452}) 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54688204}) 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=707506473}) 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54688204}) 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4058636464}) 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.100 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54688204}) 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1721134257}) 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54688204}) 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.088+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3221384263}) 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.089+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.089+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712776158}) 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.089+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=574717241}) 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.089+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.089+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712776158}) 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2537496068}) 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712776158}) 2024-08-19T12:56:09.101 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3510614929}) 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712776158}) 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=143787882}) 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712776158}) 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3938469112}) 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712776158}) 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1258037480}) 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712776158}) 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=360594682}) 2024-08-19T12:56:09.102 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712776158}) 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.090+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4281382790}) 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621738139}) 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2554138995}) 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621738139}) 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=938368959}) 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621738139}) 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2738837899}) 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.103 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621738139}) 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=862817707}) 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621738139}) 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1002374199}) 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621738139}) 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=160234146}) 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621738139}) 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1534571433}) 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621738139}) 2024-08-19T12:56:09.104 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.091+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=792271981}) 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019974512}) 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579975096}) 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019974512}) 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1654541627}) 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019974512}) 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1111765671}) 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019974512}) 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4283451284}) 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.105 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019974512}) 2024-08-19T12:56:09.106 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3581264640}) 2024-08-19T12:56:09.106 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.106 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019974512}) 2024-08-19T12:56:09.106 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4130707810}) 2024-08-19T12:56:09.106 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.106 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019974512}) 2024-08-19T12:56:09.106 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2639758944}) 2024-08-19T12:56:09.106 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.106 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019974512}) 2024-08-19T12:56:09.106 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.092+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1004655789}) 2024-08-19T12:56:09.106 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.106 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310658108}) 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=337177258}) 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310658108}) 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1421995063}) 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310658108}) 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=416379201}) 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310658108}) 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=533720311}) 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310658108}) 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3155806940}) 2024-08-19T12:56:09.107 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310658108}) 2024-08-19T12:56:09.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2634460905}) 2024-08-19T12:56:09.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310658108}) 2024-08-19T12:56:09.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2567442794}) 2024-08-19T12:56:09.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310658108}) 2024-08-19T12:56:09.108 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.094+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1532066848}) 2024-08-19T12:56:09.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.106+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039203097}) 2024-08-19T12:56:09.113 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3291641276}) 2024-08-19T12:56:09.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039203097}) 2024-08-19T12:56:09.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2501847883}) 2024-08-19T12:56:09.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039203097}) 2024-08-19T12:56:09.114 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2470630144}) 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039203097}) 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=855185694}) 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039203097}) 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2886863595}) 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039203097}) 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=713124587}) 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039203097}) 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1283741172}) 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.115 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039203097}) 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.107+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1656217144}) 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658218901}) 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1033288276}) 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658218901}) 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2654423127}) 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658218901}) 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3178196647}) 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658218901}) 2024-08-19T12:56:09.116 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4221701674}) 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658218901}) 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=883996777}) 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658218901}) 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1428227821}) 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658218901}) 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=978751758}) 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=658218901}) 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.108+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3924980212}) 2024-08-19T12:56:09.117 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.109+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.109+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668345498}) 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.109+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3827759885}) 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.109+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668345498}) 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3756546462}) 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668345498}) 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=787509225}) 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668345498}) 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3849735457}) 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668345498}) 2024-08-19T12:56:09.118 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=69006788}) 2024-08-19T12:56:09.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668345498}) 2024-08-19T12:56:09.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1612492623}) 2024-08-19T12:56:09.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668345498}) 2024-08-19T12:56:09.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2631825541}) 2024-08-19T12:56:09.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668345498}) 2024-08-19T12:56:09.119 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.110+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1748130731}) 2024-08-19T12:56:09.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222293182}) 2024-08-19T12:56:09.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1473349602}) 2024-08-19T12:56:09.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222293182}) 2024-08-19T12:56:09.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1196138506}) 2024-08-19T12:56:09.137 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222293182}) 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=147101321}) 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222293182}) 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3286081221}) 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222293182}) 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3418580536}) 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222293182}) 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2156727269}) 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.138 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222293182}) 2024-08-19T12:56:09.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1867768486}) 2024-08-19T12:56:09.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222293182}) 2024-08-19T12:56:09.139 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.135+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=563102262}) 2024-08-19T12:56:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.172+0000 7f2c6dba3640 10 monclient: get_monmap_and_config 2024-08-19T12:56:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.172+0000 7f2c6dba3640 10 monclient: build_initial_monmap 2024-08-19T12:56:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.172+0000 7f2c6dba3640 10 monclient: monmap: 2024-08-19T12:56:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:09.173171+0000 2024-08-19T12:56:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:09.173171+0000 2024-08-19T12:56:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.173 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:09.174 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:09.174 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 1 Processor -- start 2024-08-19T12:56:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 1 -- start start 2024-08-19T12:56:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 10 monclient: init 2024-08-19T12:56:09.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 10 monclient: _add_conns ranks=[7,6,2,8,4,3,0,1,5] 2024-08-19T12:56:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 10 monclient(hunting): picked mon.noname-i con 0x7f2c6806e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 10 monclient(hunting): picked mon.noname-d con 0x7f2c680a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 10 monclient(hunting): picked mon.noname-b con 0x7f2c680a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:56:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c68136790 con 0x7f2c680a5d50 2024-08-19T12:56:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6809ddf0 con 0x7f2c680a5920 2024-08-19T12:56:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6809e0d0 con 0x7f2c6806e7b0 2024-08-19T12:56:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c6dba3640 10 monclient(hunting): authenticate will time out at 1763.260585s 2024-08-19T12:56:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c66ffd640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f2c680a5920 0x7f2c6813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:44414/0 (socket says 172.21.15.26:44414) 2024-08-19T12:56:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c66ffd640 1 -- 172.21.15.26:0/2040534221 learned_addr learned my addr 172.21.15.26:0/2040534221 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:09.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.173+0000 7f2c67fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f2c680a5d50 0x7f2c68141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39076/0 (socket says 172.21.15.26:39076) 2024-08-19T12:56:09.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.175+0000 7f2c66ffd640 1 -- 172.21.15.26:0/2040534221 >> v1:172.21.15.26:6792/0 conn(0x7f2c680a5920 legacy=0x7f2c6813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.175+0000 7f2c67fff640 1 -- 172.21.15.26:0/2040534221 >> v1:172.21.15.26:6790/0 conn(0x7f2c680a5d50 legacy=0x7f2c68141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 1 -- 172.21.15.26:0/2040534221 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1371466077 0 0) 0x7f2c50001450 con 0x7f2c680a5920 2024-08-19T12:56:09.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient(hunting): my global_id is 194433 2024-08-19T12:56:09.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 1 -- 172.21.15.26:0/2040534221 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c54003520 con 0x7f2c680a5920 2024-08-19T12:56:09.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 1 -- 172.21.15.26:0/2040534221 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4273116878 0 0) 0x7f2c68136790 con 0x7f2c680a5d50 2024-08-19T12:56:09.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient(hunting): my global_id is 27443 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 1 -- 172.21.15.26:0/2040534221 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c50001450 con 0x7f2c680a5d50 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c677fe640 1 -- 172.21.15.26:0/2040534221 >> v1:172.21.15.104:6792/0 conn(0x7f2c6806e7b0 legacy=0x7f2c68136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 1 -- 172.21.15.26:0/2040534221 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1912758676 0 0) 0x7f2c54003520 con 0x7f2c680a5920 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 1 -- 172.21.15.26:0/2040534221 >> v1:172.21.15.104:6792/0 conn(0x7f2c6806e7b0 legacy=0x7f2c68136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 1 -- 172.21.15.26:0/2040534221 >> v1:172.21.15.26:6790/0 conn(0x7f2c680a5d50 legacy=0x7f2c68141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient: _finish_hunting 0 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 1 monclient: found mon.noname-d 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.26:6792/0 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 1 -- 172.21.15.26:0/2040534221 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6809e3b0 con 0x7f2c680a5920 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient: _finish_auth 0 2024-08-19T12:56:09.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient: _check_auth_tickets 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 1 -- 172.21.15.26:0/2040534221 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c50001cb0 con 0x7f2c680a5920 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c6dba3640 5 monclient: authenticate success, global_id 194433 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c6dba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient: dump: 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.182 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c6dba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 1 -- 172.21.15.26:0/2040534221 <== mon.6 v1:172.21.15.26:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c50004950 con 0x7f2c680a5920 2024-08-19T12:56:09.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 1 -- 172.21.15.26:0/2040534221 <== mon.6 v1:172.21.15.26:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c50004ec0 con 0x7f2c680a5920 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c667fc640 10 monclient: dump: 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.184 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c6dba3640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c6dba3640 4 set_mon_vals no callback set 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c6dba3640 10 monclient: shutdown 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c6dba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c6dba3640 1 -- 172.21.15.26:0/2040534221 >> v1:172.21.15.26:6792/0 conn(0x7f2c680a5920 legacy=0x7f2c6813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.176+0000 7f2c6dba3640 1 -- 172.21.15.26:0/2040534221 shutdown_connections 2024-08-19T12:56:09.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.177+0000 7f2c6dba3640 1 -- 172.21.15.26:0/2040534221 >> 172.21.15.26:0/2040534221 conn(0x7f2c680a8740 msgr2=0x7f2c680a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.177+0000 7f2c6dba3640 1 -- 172.21.15.26:0/2040534221 shutdown_connections 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.177+0000 7f2c6dba3640 1 -- 172.21.15.26:0/2040534221 wait complete. 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.177+0000 7f2c6dba3640 10 monclient: build_initial_monmap 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.177+0000 7f2c6dba3640 10 monclient: monmap: 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:09.186 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.177+0000 7f2c6dba3640 1 librados: starting msgr at 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.177+0000 7f2c6dba3640 1 librados: starting objecter 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.177+0000 7f2c6dba3640 1 Processor -- start 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 1 -- start start 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 1 librados: setting wanted keys 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 1 librados: calling monclient init 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 10 monclient: init 2024-08-19T12:56:09.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:09.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 10 monclient: _add_conns ranks=[5,0,8,3,7,6,1,4,2] 2024-08-19T12:56:09.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 10 monclient(hunting): picked mon.noname-h con 0x7f2c6806e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:09.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 10 monclient(hunting): picked mon.noname-a con 0x7f2c680a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:09.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 10 monclient(hunting): picked mon.noname-e con 0x7f2c680a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:09.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c68125bf0 con 0x7f2c680a5920 2024-08-19T12:56:09.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c68128b10 con 0x7f2c6806e7b0 2024-08-19T12:56:09.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c68125350 con 0x7f2c680a5d50 2024-08-19T12:56:09.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:09.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c6dba3640 10 monclient(hunting): authenticate will time out at 1763.265556s 2024-08-19T12:56:09.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c66ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f2c680a5920 0x7f2c68122c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39214/0 (socket says 172.21.15.26:39214) 2024-08-19T12:56:09.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c66ffd640 1 -- 172.21.15.26:0/4145119392 learned_addr learned my addr 172.21.15.26:0/4145119392 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:09.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.178+0000 7f2c67fff640 1 --1- 172.21.15.26:0/4145119392 >> v1:172.21.15.26:6793/0 conn(0x7f2c680a5d50 0x7f2c681d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:52858/0 (socket says 172.21.15.26:52858) 2024-08-19T12:56:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.180+0000 7f2c67fff640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.26:6793/0 conn(0x7f2c680a5d50 legacy=0x7f2c681d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.180+0000 7f2c5ffff640 10 client.?.objecter ms_handle_connect 0x7f2c680a5d50 2024-08-19T12:56:09.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.180+0000 7f2c66ffd640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.26:6789/0 conn(0x7f2c680a5920 legacy=0x7f2c68122c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.180+0000 7f2c5ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:09.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.180+0000 7f2c5ffff640 10 client.?.objecter ms_handle_connect 0x7f2c680a5920 2024-08-19T12:56:09.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.180+0000 7f2c5ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:09.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.180+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1758233036 0 0) 0x7f2c68125350 con 0x7f2c680a5d50 2024-08-19T12:56:09.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.180+0000 7f2c5ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.180+0000 7f2c5ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.180+0000 7f2c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.180+0000 7f2c5ffff640 10 monclient(hunting): my global_id is 184625 2024-08-19T12:56:09.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c48004eb0 con 0x7f2c680a5d50 2024-08-19T12:56:09.194 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 963637960 0 0) 0x7f2c68125bf0 con 0x7f2c680a5920 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): my global_id is 194472 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c68125350 con 0x7f2c680a5920 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c677fe640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.104:6791/0 conn(0x7f2c6806e7b0 legacy=0x7f2c68122500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 client.?.objecter ms_handle_connect 0x7f2c6806e7b0 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 260704915 0 0) 0x7f2c48004eb0 con 0x7f2c680a5d50 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c68125bf0 con 0x7f2c680a5d50 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c60001c40 con 0x7f2c680a5d50 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): dump: 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3297487422 0 0) 0x7f2c68125350 con 0x7f2c680a5920 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c48004eb0 con 0x7f2c680a5920 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2203719502 0 0) 0x7f2c68128b10 con 0x7f2c6806e7b0 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): my global_id is 184676 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c68125350 con 0x7f2c6806e7b0 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c50004e40 con 0x7f2c680a5920 2024-08-19T12:56:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient(hunting): dump: 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 816084903 0 0) 0x7f2c68125bf0 con 0x7f2c680a5d50 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.104:6791/0 conn(0x7f2c6806e7b0 legacy=0x7f2c68122500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.26:6789/0 conn(0x7f2c680a5920 legacy=0x7f2c68122c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient: _finish_hunting 0 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 monclient: found mon.e 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c68125020 con 0x7f2c680a5d50 2024-08-19T12:56:09.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient: _finish_auth 0 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient: _check_auth_tickets 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c6dba3640 5 monclient: authenticate success, global_id 184625 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c6dba3640 10 monclient: _renew_subs 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c6dba3640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c6dba3640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2c681243f0 con 0x7f2c680a5d50 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c6dba3640 10 client.184625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c6dba3640 10 monclient: _renew_subs 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c6dba3640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c6dba3640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f2c68123500 con 0x7f2c680a5d50 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c6dba3640 1 librados: init done 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c60004a90 con 0x7f2c680a5d50 2024-08-19T12:56:09.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c60004f80 con 0x7f2c680a5d50 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c65ffb640 4 set_mon_vals no callback set 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.181+0000 7f2c5ffff640 10 monclient: dump: 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.182+0000 7f2c6dba3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.182+0000 7f2c6dba3640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2c4c0065c0 con 0x7f2c4c002860 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.182+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f2c600113e0 con 0x7f2c680a5d50 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.182+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(658..658 src has 350..658) ==== 5939+0+0 (unknown 1707548120 0 0) 0x7f2c6004e190 con 0x7f2c680a5d50 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.182+0000 7f2c5ffff640 10 client.184625.objecter ms_dispatch 0x7f2c68000b90 osd_map(658..658 src has 350..658) 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.182+0000 7f2c5ffff640 3 client.184625.objecter handle_osd_map got epochs [658,658] > 0 2024-08-19T12:56:09.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.182+0000 7f2c5ffff640 3 client.184625.objecter handle_osd_map decoding full epoch 658 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.182+0000 7f2c5ffff640 20 client.184625.objecter dump_active .. 0 homeless 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.182+0000 7f2c66ffd640 10 monclient: get_auth_request con 0x7f2c48041790 auth_method 0 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.184+0000 7f2c677fe640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.104:6791/0 conn(0x7f2c4c002860 legacy=0x7f2c4c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.184+0000 7f2c5ffff640 10 client.184625.objecter ms_handle_connect 0x7f2c4c002860 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.184+0000 7f2c5ffff640 10 client.184625.objecter resend_mon_ops 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3443237265 0 0) 0x7f2c4c0065c0 con 0x7f2c4c002860 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 10 monclient(hunting): my global_id is 184685 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c68125bf0 con 0x7f2c4c002860 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 10 client.184625.objecter ms_handle_connect 0x7f2c48041790 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2329089504 0 0) 0x7f2c68125bf0 con 0x7f2c4c002860 2024-08-19T12:56:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c4c0065c0 con 0x7f2c4c002860 2024-08-19T12:56:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c58002030 con 0x7f2c4c002860 2024-08-19T12:56:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 10 monclient: dump: 2024-08-19T12:56:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3807421812 0 0) 0x7f2c4c0065c0 con 0x7f2c4c002860 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c4c00a160 con 0x7f2c4c002860 2024-08-19T12:56:09.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2c58006bd0 con 0x7f2c4c002860 2024-08-19T12:56:09.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:09.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.185+0000 7f2c5ffff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:09.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.186+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.104:6791/0 conn(0x7f2c4c002860 legacy=0x7f2c4c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.186+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.104:6791/0 conn(0x7f2c4c002860 legacy=0x7f2c4c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:09.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.192+0000 7f2c6dba3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:09.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.192+0000 7f2c6dba3640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2c4c009e70 con 0x7f2c4c009130 2024-08-19T12:56:09.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.194+0000 7f2c677fe640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.104:6791/0 conn(0x7f2c4c009130 legacy=0x7f2c4c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.194+0000 7f2c5ffff640 10 client.184625.objecter ms_handle_connect 0x7f2c4c009130 2024-08-19T12:56:09.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.194+0000 7f2c5ffff640 10 client.184625.objecter resend_mon_ops 2024-08-19T12:56:09.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1748310424 0 0) 0x7f2c4c009e70 con 0x7f2c4c009130 2024-08-19T12:56:09.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 10 monclient(hunting): my global_id is 184694 2024-08-19T12:56:09.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c4c0065c0 con 0x7f2c4c009130 2024-08-19T12:56:09.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1373748639 0 0) 0x7f2c4c0065c0 con 0x7f2c4c009130 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c4c009e70 con 0x7f2c4c009130 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f2c58004100 con 0x7f2c4c009130 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 10 monclient: dump: 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.209 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1887619116 0 0) 0x7f2c4c009e70 con 0x7f2c4c009130 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2c4c00c410 con 0x7f2c4c009130 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7879 (unknown 0 0 1410514595) 0x7f2c58004410 con 0x7f2c4c009130 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:09.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.104:6791/0 conn(0x7f2c4c009130 legacy=0x7f2c4c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.195+0000 7f2c5ffff640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.104:6791/0 conn(0x7f2c4c009130 legacy=0x7f2c4c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:09.211 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:09.211 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:09.211 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:09.211 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:09.211 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:09.211 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 294664, 2024-08-19T12:56:09.211 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:09.211 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:09.211 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:09.212 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:09.212 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:09.212 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:09.212 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:09.212 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:09.212 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:09.212 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:09.212 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.212 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:56:09.212 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:09.213 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:09.214 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:09.215 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:09.216 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:09.217 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.218 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:09.221 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:09.222 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:09.222 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.222 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.223 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:09.224 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.225 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.226 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.228 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.229 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.230 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.231 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:56:09.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.232 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:09.232 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:09.232 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:09.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.232 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:09.232 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:09.233 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 10 librados: watch_flush enter 2024-08-19T12:56:09.233 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108224405}) 2024-08-19T12:56:09.233 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3263368132}) 2024-08-19T12:56:09.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 10 librados: watch_flush exit 2024-08-19T12:56:09.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 20 client.184625.objecter shutdown clearing up homeless session... 2024-08-19T12:56:09.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 10 client.184625.objecter successfully canceled tick 2024-08-19T12:56:09.235 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.235 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108224405}) 2024-08-19T12:56:09.235 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=589763631}) 2024-08-19T12:56:09.235 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.235 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108224405}) 2024-08-19T12:56:09.235 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2491333633}) 2024-08-19T12:56:09.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.104:6800/899887506 conn(0x7f2c48041790 legacy=0x7f2c48043c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.236 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 10 monclient: shutdown 2024-08-19T12:56:09.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:09.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 1 -- 172.21.15.26:0/4145119392 >> v1:172.21.15.26:6793/0 conn(0x7f2c680a5d50 legacy=0x7f2c681d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108224405}) 2024-08-19T12:56:09.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c67fff640 1 -- 172.21.15.26:0/4145119392 reap_dead start 2024-08-19T12:56:09.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2310130064}) 2024-08-19T12:56:09.238 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c67fff640 1 -- 172.21.15.26:0/4145119392 reap_dead start 2024-08-19T12:56:09.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108224405}) 2024-08-19T12:56:09.239 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=742232197}) 2024-08-19T12:56:09.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 1 -- 172.21.15.26:0/4145119392 shutdown_connections 2024-08-19T12:56:09.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 1 -- 172.21.15.26:0/4145119392 >> 172.21.15.26:0/4145119392 conn(0x7f2c680a8740 msgr2=0x7f2c680aa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:09.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108224405}) 2024-08-19T12:56:09.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2865287519}) 2024-08-19T12:56:09.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108224405}) 2024-08-19T12:56:09.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2035906519}) 2024-08-19T12:56:09.240 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 1 -- 172.21.15.26:0/4145119392 shutdown_connections 2024-08-19T12:56:09.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108224405}) 2024-08-19T12:56:09.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 1 -- 172.21.15.26:0/4145119392 wait complete. 2024-08-19T12:56:09.241 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1389671398}) 2024-08-19T12:56:09.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.196+0000 7f2c6dba3640 1 librados: shutdown 2024-08-19T12:56:09.243 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:09.243 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-08-19T12:56:09.243 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:56:09.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612483997}) 2024-08-19T12:56:09.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1001838026}) 2024-08-19T12:56:09.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612483997}) 2024-08-19T12:56:09.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2905681287}) 2024-08-19T12:56:09.246 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612483997}) 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2123457616}) 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612483997}) 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1325263395}) 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612483997}) 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3613077274}) 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612483997}) 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2145855284}) 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.247 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612483997}) 2024-08-19T12:56:09.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1365944321}) 2024-08-19T12:56:09.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612483997}) 2024-08-19T12:56:09.248 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.244+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3782238296}) 2024-08-19T12:56:09.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402077307}) 2024-08-19T12:56:09.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3208934239}) 2024-08-19T12:56:09.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402077307}) 2024-08-19T12:56:09.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333487469}) 2024-08-19T12:56:09.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.258 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402077307}) 2024-08-19T12:56:09.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2558551825}) 2024-08-19T12:56:09.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402077307}) 2024-08-19T12:56:09.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=215962572}) 2024-08-19T12:56:09.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402077307}) 2024-08-19T12:56:09.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2053176480}) 2024-08-19T12:56:09.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402077307}) 2024-08-19T12:56:09.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2517044811}) 2024-08-19T12:56:09.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.259 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402077307}) 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1205591137}) 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402077307}) 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.246+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3688637340}) 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364495652}) 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=866345696}) 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364495652}) 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4094503438}) 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364495652}) 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2880079026}) 2024-08-19T12:56:09.260 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364495652}) 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1037932472}) 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364495652}) 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2660540790}) 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364495652}) 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4076724300}) 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364495652}) 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=672538241}) 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.261 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364495652}) 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.247+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3212193854}) 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490327239}) 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=264413584}) 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490327239}) 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.248+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4067108473}) 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490327239}) 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3619305513}) 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490327239}) 2024-08-19T12:56:09.262 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2589278505}) 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490327239}) 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3660261240}) 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490327239}) 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=212244902}) 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490327239}) 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4161410342}) 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490327239}) 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.249+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1078120790}) 2024-08-19T12:56:09.263 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484919321}) 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1304770536}) 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484919321}) 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3160051068}) 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484919321}) 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1212692253}) 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484919321}) 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3271346586}) 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.264 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484919321}) 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=169462560}) 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484919321}) 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3233912833}) 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484919321}) 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=982463727}) 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484919321}) 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.250+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2521286023}) 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731458406}) 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1679656983}) 2024-08-19T12:56:09.265 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731458406}) 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.251+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3493336098}) 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731458406}) 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1631679053}) 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731458406}) 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1379461027}) 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731458406}) 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3585570933}) 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731458406}) 2024-08-19T12:56:09.266 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3006600897}) 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731458406}) 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1790015883}) 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731458406}) 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.252+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3298368166}) 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630871032}) 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=853877889}) 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630871032}) 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=125234823}) 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.267 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630871032}) 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1228749671}) 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630871032}) 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3622248881}) 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630871032}) 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3645200688}) 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630871032}) 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1217717137}) 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630871032}) 2024-08-19T12:56:09.268 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=930685232}) 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630871032}) 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.253+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2934417295}) 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178239624}) 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1458507607}) 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178239624}) 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=219354338}) 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178239624}) 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=727082891}) 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.269 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178239624}) 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1459025419}) 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178239624}) 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1367494316}) 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178239624}) 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1502199795}) 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178239624}) 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.254+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1051203043}) 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178239624}) 2024-08-19T12:56:09.270 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.255+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1112467100}) 2024-08-19T12:56:09.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519495499}) 2024-08-19T12:56:09.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3265349665}) 2024-08-19T12:56:09.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519495499}) 2024-08-19T12:56:09.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3870974543}) 2024-08-19T12:56:09.274 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519495499}) 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=809040887}) 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519495499}) 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3440133758}) 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519495499}) 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3286275290}) 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519495499}) 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2098217632}) 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.275 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519495499}) 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.266+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=177241815}) 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519495499}) 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.267+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=865332629}) 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46644801}) 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372366293}) 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46644801}) 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1187031918}) 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46644801}) 2024-08-19T12:56:09.276 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3802404739}) 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46644801}) 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3298987303}) 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46644801}) 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3535897390}) 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46644801}) 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4184544065}) 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46644801}) 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3322370196}) 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.277 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46644801}) 2024-08-19T12:56:09.278 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.268+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=840457784}) 2024-08-19T12:56:09.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129958953}) 2024-08-19T12:56:09.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753071060}) 2024-08-19T12:56:09.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129958953}) 2024-08-19T12:56:09.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516026670}) 2024-08-19T12:56:09.285 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129958953}) 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=693576426}) 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129958953}) 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3788816043}) 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129958953}) 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1003229909}) 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129958953}) 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4238256031}) 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129958953}) 2024-08-19T12:56:09.286 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=522261961}) 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129958953}) 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.280+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2394497070}) 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660185466}) 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=810552989}) 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660185466}) 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4160753725}) 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660185466}) 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2779677148}) 2024-08-19T12:56:09.287 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660185466}) 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=248303470}) 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660185466}) 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=934454936}) 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660185466}) 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=901902874}) 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660185466}) 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1110080527}) 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660185466}) 2024-08-19T12:56:09.288 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.281+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1073846616}) 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840769139}) 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3957625240}) 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840769139}) 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2774831742}) 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840769139}) 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2220969350}) 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840769139}) 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3468228902}) 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.289 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840769139}) 2024-08-19T12:56:09.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3620119490}) 2024-08-19T12:56:09.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840769139}) 2024-08-19T12:56:09.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2890011231}) 2024-08-19T12:56:09.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840769139}) 2024-08-19T12:56:09.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=933525246}) 2024-08-19T12:56:09.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840769139}) 2024-08-19T12:56:09.290 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.283+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1777386573}) 2024-08-19T12:56:09.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1003545475}) 2024-08-19T12:56:09.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1672571894}) 2024-08-19T12:56:09.302 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1003545475}) 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2686645954}) 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1003545475}) 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=374106010}) 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1003545475}) 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=844174440}) 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1003545475}) 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3435573410}) 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.303 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1003545475}) 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3148286374}) 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1003545475}) 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=320555215}) 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1003545475}) 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.293+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=144043910}) 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667089137}) 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3154164466}) 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667089137}) 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1630753776}) 2024-08-19T12:56:09.304 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.294+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667089137}) 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2299000424}) 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667089137}) 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1653639822}) 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667089137}) 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4068368266}) 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667089137}) 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=846778899}) 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667089137}) 2024-08-19T12:56:09.305 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1070970101}) 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667089137}) 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.295+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2668905659}) 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026869333}) 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=656445607}) 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026869333}) 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3385067479}) 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026869333}) 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=395862588}) 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.306 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026869333}) 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2071254924}) 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026869333}) 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1705438463}) 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026869333}) 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2421582598}) 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.296+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026869333}) 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1283106969}) 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026869333}) 2024-08-19T12:56:09.307 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.297+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1441858048}) 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271354384}) 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2235913583}) 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271354384}) 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=696017525}) 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271354384}) 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1099594611}) 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271354384}) 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3648929465}) 2024-08-19T12:56:09.308 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271354384}) 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=448461323}) 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271354384}) 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=757869808}) 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271354384}) 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1536327781}) 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271354384}) 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.298+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2850653978}) 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.309 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463392291}) 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742172728}) 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463392291}) 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1163054067}) 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463392291}) 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2562637399}) 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463392291}) 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3522819777}) 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463392291}) 2024-08-19T12:56:09.310 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2667294310}) 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463392291}) 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3984727187}) 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463392291}) 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=48254191}) 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463392291}) 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.299+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1423535813}) 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703293624}) 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3177566528}) 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.311 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703293624}) 2024-08-19T12:56:09.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756198206}) 2024-08-19T12:56:09.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703293624}) 2024-08-19T12:56:09.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1947882872}) 2024-08-19T12:56:09.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703293624}) 2024-08-19T12:56:09.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2576044778}) 2024-08-19T12:56:09.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703293624}) 2024-08-19T12:56:09.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1223350076}) 2024-08-19T12:56:09.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.312 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703293624}) 2024-08-19T12:56:09.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3815819214}) 2024-08-19T12:56:09.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703293624}) 2024-08-19T12:56:09.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3575191672}) 2024-08-19T12:56:09.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703293624}) 2024-08-19T12:56:09.313 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.301+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=273560003}) 2024-08-19T12:56:09.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263231641}) 2024-08-19T12:56:09.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1811168836}) 2024-08-19T12:56:09.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263231641}) 2024-08-19T12:56:09.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1504929833}) 2024-08-19T12:56:09.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263231641}) 2024-08-19T12:56:09.335 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.302+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3276757917}) 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263231641}) 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3208044302}) 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263231641}) 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4045002970}) 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263231641}) 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4087542372}) 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263231641}) 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3518020269}) 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.336 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263231641}) 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.303+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3345192785}) 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599668183}) 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2259607192}) 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599668183}) 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1302856309}) 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599668183}) 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3103021674}) 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599668183}) 2024-08-19T12:56:09.337 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=175673813}) 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599668183}) 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4074378544}) 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599668183}) 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2767262188}) 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599668183}) 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1255405632}) 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599668183}) 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.304+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4085965771}) 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.338 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=444647945}) 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3019084836}) 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=444647945}) 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080232626}) 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=444647945}) 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4236521190}) 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=444647945}) 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=31657637}) 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=444647945}) 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2767886764}) 2024-08-19T12:56:09.339 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=444647945}) 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2375959369}) 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=444647945}) 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=419538153}) 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=444647945}) 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.305+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3832491805}) 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.306+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.306+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20644952}) 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.306+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3188865865}) 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.306+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.340 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.306+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20644952}) 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.306+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3475947182}) 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.306+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.306+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20644952}) 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.306+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2511634074}) 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.306+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20644952}) 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3076882730}) 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20644952}) 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2838953732}) 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20644952}) 2024-08-19T12:56:09.341 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=45249779}) 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20644952}) 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=48343647}) 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20644952}) 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.307+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2633956694}) 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345010409}) 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2242298529}) 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345010409}) 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1449528874}) 2024-08-19T12:56:09.342 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345010409}) 2024-08-19T12:56:09.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2035641131}) 2024-08-19T12:56:09.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345010409}) 2024-08-19T12:56:09.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2498920138}) 2024-08-19T12:56:09.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345010409}) 2024-08-19T12:56:09.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1283420534}) 2024-08-19T12:56:09.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345010409}) 2024-08-19T12:56:09.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=908736884}) 2024-08-19T12:56:09.343 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345010409}) 2024-08-19T12:56:09.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3889576177}) 2024-08-19T12:56:09.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345010409}) 2024-08-19T12:56:09.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.308+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3127990185}) 2024-08-19T12:56:09.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267047161}) 2024-08-19T12:56:09.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=91702194}) 2024-08-19T12:56:09.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.346 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267047161}) 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2387334866}) 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267047161}) 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4180937420}) 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267047161}) 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3320691477}) 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267047161}) 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2318498632}) 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267047161}) 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1047114516}) 2024-08-19T12:56:09.347 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267047161}) 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2669689601}) 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267047161}) 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.309+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2337713448}) 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615524557}) 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3520747467}) 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615524557}) 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=200172988}) 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.348 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615524557}) 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3340397720}) 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615524557}) 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3693379587}) 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615524557}) 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=949231663}) 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615524557}) 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2182105664}) 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615524557}) 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=192356744}) 2024-08-19T12:56:09.349 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615524557}) 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.311+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3604911335}) 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854882921}) 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2348710375}) 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854882921}) 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1796815317}) 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854882921}) 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=378832081}) 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.350 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854882921}) 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2011492354}) 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854882921}) 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=118579018}) 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854882921}) 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1014766725}) 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854882921}) 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2605245634}) 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854882921}) 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.320+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3543287194}) 2024-08-19T12:56:09.351 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738209393}) 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=129469757}) 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738209393}) 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3795918773}) 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738209393}) 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2044157510}) 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738209393}) 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=401032481}) 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738209393}) 2024-08-19T12:56:09.352 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2186957797}) 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738209393}) 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1140751795}) 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738209393}) 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4287048827}) 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738209393}) 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.322+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1661962577}) 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478387857}) 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3884737238}) 2024-08-19T12:56:09.353 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478387857}) 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3575269226}) 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478387857}) 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2427657239}) 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478387857}) 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2881408884}) 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478387857}) 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=953636861}) 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478387857}) 2024-08-19T12:56:09.354 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=541568704}) 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478387857}) 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3403600948}) 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2478387857}) 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.323+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1320113808}) 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299517759}) 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1986352741}) 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299517759}) 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3423397590}) 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.355 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299517759}) 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3192110907}) 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299517759}) 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4075498906}) 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299517759}) 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3187620158}) 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299517759}) 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1090458650}) 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.324+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299517759}) 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.325+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=155527137}) 2024-08-19T12:56:09.356 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.325+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.325+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299517759}) 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.325+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1877782893}) 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207877813}) 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1820508581}) 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207877813}) 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622715704}) 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207877813}) 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1293488601}) 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207877813}) 2024-08-19T12:56:09.357 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3709454590}) 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207877813}) 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3300697368}) 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207877813}) 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=759057617}) 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207877813}) 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2494679311}) 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207877813}) 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.326+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=542725101}) 2024-08-19T12:56:09.358 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434756842}) 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3802768964}) 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434756842}) 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219943276}) 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434756842}) 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2136148620}) 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434756842}) 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3540104555}) 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434756842}) 2024-08-19T12:56:09.359 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=397474406}) 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434756842}) 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=705559173}) 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434756842}) 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1548468400}) 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.327+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434756842}) 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.328+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3813839114}) 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428229462}) 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=98960240}) 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.360 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428229462}) 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2936726325}) 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428229462}) 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1905266930}) 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428229462}) 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3143490280}) 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428229462}) 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=976749504}) 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428229462}) 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3701005514}) 2024-08-19T12:56:09.361 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428229462}) 2024-08-19T12:56:09.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1901652986}) 2024-08-19T12:56:09.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-08-19T12:56:09.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428229462}) 2024-08-19T12:56:09.362 INFO:tasks.ceph.mon.a.smithi026.stderr:2024-08-19T12:56:09.329+0000 7f5424a1f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3435433355}) 2024-08-19T12:56:09.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.357+0000 7fd88ac9f640 10 monclient: get_monmap_and_config 2024-08-19T12:56:09.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.357+0000 7fd88ac9f640 10 monclient: build_initial_monmap 2024-08-19T12:56:09.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.357+0000 7fd88ac9f640 10 monclient: monmap: 2024-08-19T12:56:09.372 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:09.372 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.372 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:09.358620+0000 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:09.358620+0000 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:09.373 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.358+0000 7fd88ac9f640 1 Processor -- start 2024-08-19T12:56:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.358+0000 7fd88ac9f640 1 -- start start 2024-08-19T12:56:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.358+0000 7fd88ac9f640 10 monclient: init 2024-08-19T12:56:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.358+0000 7fd88ac9f640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.359+0000 7fd88ac9f640 10 monclient: _add_conns ranks=[7,2,0,6,5,8,3,4,1] 2024-08-19T12:56:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.359+0000 7fd88ac9f640 10 monclient(hunting): picked mon.noname-i con 0x7fd88406e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.359+0000 7fd88ac9f640 10 monclient(hunting): picked mon.noname-b con 0x7fd8840a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:56:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.359+0000 7fd88ac9f640 10 monclient(hunting): picked mon.noname-a con 0x7fd8840a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.359+0000 7fd88ac9f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd884136790 con 0x7fd8840a5d50 2024-08-19T12:56:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.359+0000 7fd88ac9f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd88409ddf0 con 0x7fd8840a5920 2024-08-19T12:56:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.359+0000 7fd88ac9f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd88409e0d0 con 0x7fd88406e7b0 2024-08-19T12:56:09.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.359+0000 7fd88ac9f640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.359+0000 7fd88ac9f640 10 monclient(hunting): authenticate will time out at 1763.446114s 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.359+0000 7fd883fff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd8840a5920 0x7fd88413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39090/0 (socket says 172.21.15.26:39090) 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.359+0000 7fd883fff640 1 -- 172.21.15.26:0/2502714105 learned_addr learned my addr 172.21.15.26:0/2502714105 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.359+0000 7fd88923c640 1 --1- 172.21.15.26:0/2502714105 >> v1:172.21.15.26:6789/0 conn(0x7fd8840a5d50 0x7fd884141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:39220/0 (socket says 172.21.15.26:39220) 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd88923c640 1 -- 172.21.15.26:0/2502714105 >> v1:172.21.15.26:6789/0 conn(0x7fd8840a5d50 legacy=0x7fd884141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd883fff640 1 -- 172.21.15.26:0/2502714105 >> v1:172.21.15.26:6790/0 conn(0x7fd8840a5920 legacy=0x7fd88413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd888a3b640 1 -- 172.21.15.26:0/2502714105 >> v1:172.21.15.104:6792/0 conn(0x7fd88406e7b0 legacy=0x7fd884136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2378025062 0 0) 0x7fd884136790 con 0x7fd8840a5d50 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient(hunting): my global_id is 194481 2024-08-19T12:56:09.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd86c003520 con 0x7fd8840a5d50 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2070696322 0 0) 0x7fd878001380 con 0x7fd8840a5920 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient(hunting): my global_id is 27452 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd884136790 con 0x7fd8840a5920 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3498979371 0 0) 0x7fd88409e0d0 con 0x7fd88406e7b0 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient(hunting): my global_id is 136420 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd878001380 con 0x7fd88406e7b0 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3146494350 0 0) 0x7fd86c003520 con 0x7fd8840a5d50 2024-08-19T12:56:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 >> v1:172.21.15.104:6792/0 conn(0x7fd88406e7b0 legacy=0x7fd884136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 >> v1:172.21.15.26:6790/0 conn(0x7fd8840a5920 legacy=0x7fd88413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient: _finish_hunting 0 2024-08-19T12:56:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 monclient: found mon.noname-a 2024-08-19T12:56:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.26:6789/0 2024-08-19T12:56:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88409e3b0 con 0x7fd8840a5d50 2024-08-19T12:56:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient: _finish_auth 0 2024-08-19T12:56:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient: _check_auth_tickets 2024-08-19T12:56:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd87c001e60 con 0x7fd8840a5d50 2024-08-19T12:56:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd88ac9f640 5 monclient: authenticate success, global_id 194481 2024-08-19T12:56:09.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd88ac9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient: dump: 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.378 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.379 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.379 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.379 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.379 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.379 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.379 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd88ac9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:09.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 <== mon.0 v1:172.21.15.26:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd87c004b60 con 0x7fd8840a5d50 2024-08-19T12:56:09.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:09.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 1 -- 172.21.15.26:0/2502714105 <== mon.0 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd87c0050d0 con 0x7fd8840a5d50 2024-08-19T12:56:09.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:09.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd8837fe640 10 monclient: dump: 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.380 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.361+0000 7fd88ac9f640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.362+0000 7fd88ac9f640 4 set_mon_vals no callback set 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.362+0000 7fd88ac9f640 10 monclient: shutdown 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.362+0000 7fd88ac9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.362+0000 7fd88ac9f640 1 -- 172.21.15.26:0/2502714105 >> v1:172.21.15.26:6789/0 conn(0x7fd8840a5d50 legacy=0x7fd884141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.362+0000 7fd88ac9f640 1 -- 172.21.15.26:0/2502714105 shutdown_connections 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.362+0000 7fd88ac9f640 1 -- 172.21.15.26:0/2502714105 >> 172.21.15.26:0/2502714105 conn(0x7fd8840a8740 msgr2=0x7fd8840a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.362+0000 7fd88ac9f640 1 -- 172.21.15.26:0/2502714105 shutdown_connections 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.362+0000 7fd88ac9f640 1 -- 172.21.15.26:0/2502714105 wait complete. 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.363+0000 7fd88ac9f640 10 monclient: build_initial_monmap 2024-08-19T12:56:09.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.363+0000 7fd88ac9f640 10 monclient: monmap: 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:09.382 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.363+0000 7fd88ac9f640 1 librados: starting msgr at 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.363+0000 7fd88ac9f640 1 librados: starting objecter 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.363+0000 7fd88ac9f640 1 Processor -- start 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.363+0000 7fd88ac9f640 1 -- start start 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.363+0000 7fd88ac9f640 1 librados: setting wanted keys 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.363+0000 7fd88ac9f640 1 librados: calling monclient init 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.363+0000 7fd88ac9f640 10 monclient: init 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd88ac9f640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd88ac9f640 10 monclient: _add_conns ranks=[8,3,4,6,2,0,1,5,7] 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd88ac9f640 10 monclient(hunting): picked mon.noname-e con 0x7fd88406e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:09.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd88ac9f640 10 monclient(hunting): picked mon.noname-g con 0x7fd8840a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd88ac9f640 10 monclient(hunting): picked mon.noname-c con 0x7fd8840a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd88ac9f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd884142e50 con 0x7fd8840a5920 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd88ac9f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd884138f60 con 0x7fd8840a5d50 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd88ac9f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd884138460 con 0x7fd88406e7b0 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd88ac9f640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd88ac9f640 10 monclient(hunting): authenticate will time out at 1763.451201s 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd888a3b640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fd88406e7b0 0x7fd884137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:52882/0 (socket says 172.21.15.26:52882) 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd88923c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fd8840a5d50 0x7fd8841d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:38094/0 (socket says 172.21.15.26:38094) 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd88923c640 1 -- 172.21.15.26:0/722626941 learned_addr learned my addr 172.21.15.26:0/722626941 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd883fff640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.104:6790/0 conn(0x7fd8840a5920 legacy=0x7fd884137d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd883fff640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.104:6790/0 conn(0x7fd8840a5920 legacy=0x7fd884137d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd883fff640 1 --1- 172.21.15.26:0/722626941 >> v1:172.21.15.104:6790/0 conn(0x7fd8840a5920 0x7fd884137d50 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-08-19T12:56:09.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd883fff640 1 --1- 172.21.15.26:0/722626941 >> v1:172.21.15.104:6790/0 conn(0x7fd8840a5920 0x7fd884137d50 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.364+0000 7fd8817fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.104:6790/0 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd888a3b640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.26:6793/0 conn(0x7fd88406e7b0 legacy=0x7fd884137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd88923c640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.26:6791/0 conn(0x7fd8840a5d50 legacy=0x7fd8841d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 client.?.objecter ms_handle_connect 0x7fd88406e7b0 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 client.?.objecter ms_handle_connect 0x7fd8840a5d50 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 906079554 0 0) 0x7fd884138460 con 0x7fd88406e7b0 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): my global_id is 184634 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd870004eb0 con 0x7fd88406e7b0 2024-08-19T12:56:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3370008862 0 0) 0x7fd884138f60 con 0x7fd8840a5d50 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): my global_id is 184774 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd884138460 con 0x7fd8840a5d50 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2786154494 0 0) 0x7fd870004eb0 con 0x7fd88406e7b0 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd884138f60 con 0x7fd88406e7b0 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd874003610 con 0x7fd88406e7b0 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): dump: 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.386 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.387 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 387128583 0 0) 0x7fd884138460 con 0x7fd8840a5d50 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd870004eb0 con 0x7fd8840a5d50 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd87c005050 con 0x7fd8840a5d50 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.366+0000 7fd8817fa640 10 monclient(hunting): dump: 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.388 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1205011219 0 0) 0x7fd884138f60 con 0x7fd88406e7b0 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.26:6791/0 conn(0x7fd8840a5d50 legacy=0x7fd8841d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.104:6790/0 conn(0x7fd8840a5920 legacy=0x7fd884137d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:09.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 10 monclient: _finish_hunting 0 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 1 monclient: found mon.e 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88413aa20 con 0x7fd88406e7b0 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 10 monclient: _finish_auth 0 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 10 monclient: _check_auth_tickets 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd88ac9f640 5 monclient: authenticate success, global_id 184634 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd88ac9f640 10 monclient: _renew_subs 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd88ac9f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd88ac9f640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8841429a0 con 0x7fd88406e7b0 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd88ac9f640 10 client.184634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:09.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd88ac9f640 10 monclient: _renew_subs 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd88ac9f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd88ac9f640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fd88413b6e0 con 0x7fd88406e7b0 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd88ac9f640 1 librados: init done 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd874003b90 con 0x7fd88406e7b0 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd874004100 con 0x7fd88406e7b0 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd882ffd640 4 set_mon_vals no callback set 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 10 monclient: dump: 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.392 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.367+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd8740042a0 con 0x7fd88406e7b0 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.368+0000 7fd88ac9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.368+0000 7fd88ac9f640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd8640065c0 con 0x7fd864002860 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.368+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(658..658 src has 350..658) ==== 5939+0+0 (unknown 1707548120 0 0) 0x7fd87404e450 con 0x7fd88406e7b0 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.368+0000 7fd8817fa640 10 client.184634.objecter ms_dispatch 0x7fd884000b90 osd_map(658..658 src has 350..658) 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.368+0000 7fd8817fa640 3 client.184634.objecter handle_osd_map got epochs [658,658] > 0 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.368+0000 7fd8817fa640 3 client.184634.objecter handle_osd_map decoding full epoch 658 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.368+0000 7fd8817fa640 20 client.184634.objecter dump_active .. 0 homeless 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.368+0000 7fd88923c640 10 monclient: get_auth_request con 0x7fd8700415d0 auth_method 0 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 10 client.184634.objecter ms_handle_connect 0x7fd864002860 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd883fff640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.104:6792/0 conn(0x7fd864002860 legacy=0x7fd864004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 10 client.184634.objecter resend_mon_ops 2024-08-19T12:56:09.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 205413227 0 0) 0x7fd8640065c0 con 0x7fd864002860 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 10 monclient(hunting): my global_id is 136429 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd884138f60 con 0x7fd864002860 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 10 client.184634.objecter ms_handle_connect 0x7fd8700415d0 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 175336818 0 0) 0x7fd884138f60 con 0x7fd864002860 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8640065c0 con 0x7fd864002860 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd8780047e0 con 0x7fd864002860 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.370+0000 7fd8817fa640 10 monclient: dump: 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.394 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.395 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.371+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 138539456 0 0) 0x7fd8640065c0 con 0x7fd864002860 2024-08-19T12:56:09.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.371+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd86400a160 con 0x7fd864002860 2024-08-19T12:56:09.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.371+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd878006a20 con 0x7fd864002860 2024-08-19T12:56:09.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.371+0000 7fd8817fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:09.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.371+0000 7fd8817fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:09.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.371+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.104:6792/0 conn(0x7fd864002860 legacy=0x7fd864004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.396 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.371+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.104:6792/0 conn(0x7fd864002860 legacy=0x7fd864004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:09.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.377+0000 7fd88ac9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:09.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.377+0000 7fd88ac9f640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd864009e50 con 0x7fd864009110 2024-08-19T12:56:09.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.379+0000 7fd883fff640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.104:6792/0 conn(0x7fd864009110 legacy=0x7fd8640095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:09.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.379+0000 7fd8817fa640 10 client.184634.objecter ms_handle_connect 0x7fd864009110 2024-08-19T12:56:09.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.379+0000 7fd8817fa640 10 client.184634.objecter resend_mon_ops 2024-08-19T12:56:09.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1574659389 0 0) 0x7fd864009e50 con 0x7fd864009110 2024-08-19T12:56:09.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:09.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:09.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:09.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 10 monclient(hunting): my global_id is 136438 2024-08-19T12:56:09.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8640065c0 con 0x7fd864009110 2024-08-19T12:56:09.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1618315319 0 0) 0x7fd8640065c0 con 0x7fd864009110 2024-08-19T12:56:09.398 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd864009e50 con 0x7fd864009110 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd8780062d0 con 0x7fd864009110 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 10 monclient: dump: 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:09.399 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1217298626 0 0) 0x7fd864009e50 con 0x7fd864009110 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd86400c3f0 con 0x7fd864009110 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7545 (unknown 0 0 1432018058) 0x7fd878006510 con 0x7fd864009110 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.104:6792/0 conn(0x7fd864009110 legacy=0x7fd8640095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd8817fa640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.104:6792/0 conn(0x7fd864009110 legacy=0x7fd8640095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:09.401 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:09.401 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:09.401 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:09.401 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:09.401 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 124, 2024-08-19T12:56:09.401 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 939737, 2024-08-19T12:56:09.401 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:09.401 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-08-19T12:56:09.401 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:09.401 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 6, 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:09.402 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:09.403 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:09.404 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:09.404 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:09.404 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:09.404 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:09.404 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:09.404 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:09.404 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:09.404 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:09.404 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:09.404 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.404 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:09.405 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:09.406 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:09.407 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.408 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.408 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.408 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:09.408 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:09.408 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.408 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.408 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.408 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:09.409 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:09.409 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.409 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.409 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:09.409 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.409 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.409 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.410 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:09.410 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:09.410 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.410 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.410 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.410 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:09.410 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:09.410 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.411 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:09.411 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.411 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.411 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.411 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:09.411 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:09.412 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.413 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.414 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.414 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:09.414 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.414 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.414 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.415 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.417 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:09.417 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:09.417 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.417 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:09.417 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.417 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:09.418 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:09.418 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.418 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.418 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.418 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:09.418 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:09.418 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.419 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:09.421 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:09.422 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:09.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.380+0000 7fd88ac9f640 10 librados: watch_flush enter 2024-08-19T12:56:09.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88ac9f640 10 librados: watch_flush exit 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88ac9f640 20 client.184634.objecter shutdown clearing up homeless session... 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88ac9f640 10 client.184634.objecter successfully canceled tick 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88ac9f640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.104:6800/899887506 conn(0x7fd8700415d0 legacy=0x7fd870043a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88ac9f640 10 monclient: shutdown 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88ac9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88ac9f640 1 -- 172.21.15.26:0/722626941 >> v1:172.21.15.26:6793/0 conn(0x7fd88406e7b0 legacy=0x7fd884137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88923c640 1 -- 172.21.15.26:0/722626941 reap_dead start 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88923c640 1 -- 172.21.15.26:0/722626941 reap_dead start 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88ac9f640 1 -- 172.21.15.26:0/722626941 shutdown_connections 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88ac9f640 1 -- 172.21.15.26:0/722626941 >> 172.21.15.26:0/722626941 conn(0x7fd8840a8740 msgr2=0x7fd8840aae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88ac9f640 1 -- 172.21.15.26:0/722626941 shutdown_connections 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88ac9f640 1 -- 172.21.15.26:0/722626941 wait complete. 2024-08-19T12:56:09.425 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:09.381+0000 7fd88ac9f640 1 librados: shutdown 2024-08-19T12:56:09.426 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-08-19T12:56:09.426 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a', 'g', 'd'] 2024-08-19T12:56:09.426 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b', 'h'] 2024-08-19T12:56:09.426 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-08-19T12:56:09.426 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-08-19T12:56:09.427 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-19T12:56:09.427 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:56:09.477 INFO:tasks.ceph.mon.a:Stopped 2024-08-19T12:56:09.477 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-08-19T12:56:09.478 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-08-19T12:56:09.478 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-08-19T12:56:09.478 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:56:09.491 INFO:tasks.ceph.mon.g:Stopped 2024-08-19T12:56:09.491 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-08-19T12:56:09.491 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-08-19T12:56:09.491 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-08-19T12:56:09.491 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:56:09.530 INFO:tasks.ceph.mon.d:Stopped 2024-08-19T12:56:09.530 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-08-19T12:56:09.530 INFO:tasks.ceph.mon.b:Sent signal 19 2024-08-19T12:56:09.530 INFO:tasks.mon_thrash:Sending STOP to mon h 2024-08-19T12:56:09.530 INFO:tasks.ceph.mon.h:Sent signal 19 2024-08-19T12:56:09.531 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:56:24.531 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-08-19T12:56:24.532 INFO:tasks.ceph.mon.b:Sent signal 18 2024-08-19T12:56:24.532 INFO:tasks.mon_thrash:Sending CONT to mon h 2024-08-19T12:56:24.532 INFO:tasks.ceph.mon.h:Sent signal 18 2024-08-19T12:56:24.532 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 6 2024-08-19T12:56:24.532 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:56:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.641+0000 7f4609598640 10 monclient: get_monmap_and_config 2024-08-19T12:56:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.641+0000 7f4609598640 10 monclient: build_initial_monmap 2024-08-19T12:56:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.641+0000 7f4609598640 10 monclient: monmap: 2024-08-19T12:56:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:24.643712+0000 2024-08-19T12:56:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:24.643712+0000 2024-08-19T12:56:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:24.645 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.642+0000 7f4609598640 1 Processor -- start 2024-08-19T12:56:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.642+0000 7f4609598640 1 -- start start 2024-08-19T12:56:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.642+0000 7f4609598640 10 monclient: init 2024-08-19T12:56:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f4609598640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f4609598640 10 monclient: _add_conns ranks=[5,0,8,3,6,7,1,4,2] 2024-08-19T12:56:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f4609598640 10 monclient(hunting): picked mon.noname-h con 0x7f460406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f4609598640 10 monclient(hunting): picked mon.noname-a con 0x7f46040a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f4609598640 10 monclient(hunting): picked mon.noname-e con 0x7f46040a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f4609598640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4604136790 con 0x7f46040a5920 2024-08-19T12:56:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f4609598640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f460409ddf0 con 0x7f460406e7b0 2024-08-19T12:56:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f4609598640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f460409e0d0 con 0x7f46040a5d50 2024-08-19T12:56:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f4609598640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f4609598640 10 monclient(hunting): authenticate will time out at 1778.731252s 2024-08-19T12:56:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f45fbfff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f46040a5920 legacy=0x7f460413e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:56:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f46037fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f46040a5d50 0x7f4604141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:53706/0 (socket says 172.21.15.26:53706) 2024-08-19T12:56:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.643+0000 7f46037fe640 1 -- 172.21.15.26:0/2999556780 learned_addr learned my addr 172.21.15.26:0/2999556780 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.645+0000 7f46037fe640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6793/0 conn(0x7f46040a5d50 legacy=0x7f4604141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.645+0000 7f4602ffd640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6791/0 conn(0x7f460406e7b0 legacy=0x7f4604136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:24.846 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:24.844+0000 7f45fbfff640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6789/0 conn(0x7f46040a5920 legacy=0x7f460413e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:56:25.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:25.246+0000 7f45fbfff640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6789/0 conn(0x7f46040a5920 legacy=0x7f460413e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:56:26.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:26.047+0000 7f45fbfff640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6789/0 conn(0x7f46040a5920 legacy=0x7f460413e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:56:27.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 10 monclient(hunting): tick 2024-08-19T12:56:27.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 1 monclient(hunting): continuing hunt 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6793/0 conn(0x7f46040a5d50 legacy=0x7f4604141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6791/0 conn(0x7f460406e7b0 legacy=0x7f4604136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6789/0 conn(0x7f46040a5920 legacy=0x7f460413e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 10 monclient: _add_conns ranks=[4,7,1,6,3,2] 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 10 monclient(hunting): picked mon.noname-c con 0x7f45e40038c0 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 10 monclient(hunting): picked mon.noname-i con 0x7f45e4007550 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 10 monclient(hunting): picked mon.noname-f con 0x7f45e400b2a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f460409e3b0 con 0x7f45e400b2a0 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4604136790 con 0x7f45e40038c0 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45e4011110 con 0x7f45e4007550 2024-08-19T12:56:27.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.644+0000 7f46017fa640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:27.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.645+0000 7f4602ffd640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6791/0 conn(0x7f45e40038c0 legacy=0x7f45e4005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:27.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.646+0000 7f46037fe640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6789/0 conn(0x7f45e400b2a0 legacy=0x7f45e400d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:27.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:27.646+0000 7f45fbfff640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6792/0 conn(0x7f45e4007550 legacy=0x7f45e40099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:29.570+0000 7f4602ffd640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6791/0 conn(0x7f45e40038c0 legacy=0x7f45e4005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-08-19T12:56:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:29.570+0000 7f4602ffd640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6791/0 conn(0x7f45e40038c0 legacy=0x7f45e4005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:56:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:29.570+0000 7f4602ffd640 1 --1- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6791/0 conn(0x7f45e40038c0 0x7f45e4005d10 :-1 s=OPENED pgs=3698 cs=1 l=1).handle_message read tag failed 2024-08-19T12:56:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:29.570+0000 7f4602ffd640 1 --1- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6791/0 conn(0x7f45e40038c0 0x7f45e4005d10 :-1 s=OPENED pgs=3698 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:56:29.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:29.570+0000 7f46027fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6791/0 2024-08-19T12:56:30.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.005+0000 7f45fbfff640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6792/0 conn(0x7f45e4007550 legacy=0x7f45e40099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-08-19T12:56:30.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.005+0000 7f45fbfff640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6792/0 conn(0x7f45e4007550 legacy=0x7f45e40099d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:56:30.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.005+0000 7f45fbfff640 1 --1- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6792/0 conn(0x7f45e4007550 0x7f45e40099d0 :-1 s=OPENED pgs=855 cs=1 l=1).handle_message read tag failed 2024-08-19T12:56:30.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.005+0000 7f45fbfff640 1 --1- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6792/0 conn(0x7f45e4007550 0x7f45e40099d0 :-1 s=OPENED pgs=855 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:56:30.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.005+0000 7f46037fe640 1 -- 172.21.15.26:0/2999556780 reap_dead start 2024-08-19T12:56:30.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.005+0000 7f46027fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.104:6792/0 2024-08-19T12:56:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.644+0000 7f46017fa640 10 monclient(hunting): tick 2024-08-19T12:56:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.644+0000 7f46017fa640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:56:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.644+0000 7f46017fa640 1 monclient(hunting): continuing hunt 2024-08-19T12:56:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.644+0000 7f46017fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:56:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.644+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6792/0 conn(0x7f45e4007550 legacy=0x7f45e40099d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.644+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6791/0 conn(0x7f45e40038c0 legacy=0x7f45e4005d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.644+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6789/0 conn(0x7f45e400b2a0 legacy=0x7f45e400d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.644+0000 7f46017fa640 10 monclient: _add_conns ranks=[6,3,2] 2024-08-19T12:56:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.644+0000 7f46017fa640 10 monclient(hunting): picked mon.noname-d con 0x7f45e40038c0 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.644+0000 7f46017fa640 10 monclient(hunting): picked mon.noname-g con 0x7f45e4007550 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:30.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.644+0000 7f46017fa640 10 monclient(hunting): picked mon.noname-b con 0x7f45e400f1d0 addr v1:172.21.15.26:6790/0 2024-08-19T12:56:30.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.644+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45e4012470 con 0x7f45e400f1d0 2024-08-19T12:56:30.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.645+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45e40147b0 con 0x7f45e4007550 2024-08-19T12:56:30.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.645+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45e4015990 con 0x7f45e40038c0 2024-08-19T12:56:30.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.645+0000 7f46017fa640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:30.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.645+0000 7f4602ffd640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6792/0 conn(0x7f45e40038c0 legacy=0x7f45e4005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:56:30.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.645+0000 7f45fbfff640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6790/0 conn(0x7f45e4007550 legacy=0x7f45e40099d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:56:30.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.647+0000 7f46037fe640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6790/0 conn(0x7f45e400f1d0 legacy=0x7f45e400f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.846+0000 7f4602ffd640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6792/0 conn(0x7f45e40038c0 legacy=0x7f45e4005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:56:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:30.846+0000 7f45fbfff640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6790/0 conn(0x7f45e4007550 legacy=0x7f45e40099d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:56:31.247 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:31.247+0000 7f4602ffd640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6792/0 conn(0x7f45e40038c0 legacy=0x7f45e4005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:56:31.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:31.247+0000 7f45fbfff640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6790/0 conn(0x7f45e4007550 legacy=0x7f45e40099d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:56:32.048 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:32.048+0000 7f4602ffd640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6792/0 conn(0x7f45e40038c0 legacy=0x7f45e4005d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:56:32.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:32.048+0000 7f45fbfff640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6790/0 conn(0x7f45e4007550 legacy=0x7f45e40099d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:56:33.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 10 monclient(hunting): tick 2024-08-19T12:56:33.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:56:33.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 1 monclient(hunting): continuing hunt 2024-08-19T12:56:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:56:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6792/0 conn(0x7f45e40038c0 legacy=0x7f45e4005d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:56:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6790/0 conn(0x7f45e4007550 legacy=0x7f45e40099d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:56:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6790/0 conn(0x7f45e400f1d0 legacy=0x7f45e400f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 10 monclient: _add_conns ranks=[6,4,7,1,0,2,5,3,8] 2024-08-19T12:56:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 10 monclient(hunting): picked mon.noname-d con 0x7f45e40038c0 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 10 monclient(hunting): picked mon.noname-c con 0x7f45e4015c10 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:33.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 10 monclient(hunting): picked mon.noname-i con 0x7f45e40064c0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45e4016c50 con 0x7f45e4015c10 2024-08-19T12:56:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45e40147b0 con 0x7f45e40038c0 2024-08-19T12:56:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f4602ffd640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6792/0 conn(0x7f45e40038c0 legacy=0x7f45e4005cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:56:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 1 -- 172.21.15.26:0/2999556780 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45e4015990 con 0x7f45e40064c0 2024-08-19T12:56:33.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.645+0000 7f46017fa640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:33.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.647+0000 7f45fbfff640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6791/0 conn(0x7f45e4015c10 legacy=0x7f45e4016070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:33.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.647+0000 7f46037fe640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6792/0 conn(0x7f45e40064c0 legacy=0x7f45e40068f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:33.847 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.846+0000 7f4602ffd640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6792/0 conn(0x7f45e40038c0 legacy=0x7f45e4005cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:56:33.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 1 -- 172.21.15.26:0/2999556780 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3984724266 0 0) 0x7f45e4016c50 con 0x7f45e4015c10 2024-08-19T12:56:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient(hunting): my global_id is 184783 2024-08-19T12:56:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 1 -- 172.21.15.26:0/2999556780 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45f0003520 con 0x7f45e4015c10 2024-08-19T12:56:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 1 -- 172.21.15.26:0/2999556780 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1673622390 0 0) 0x7f45e4015990 con 0x7f45e40064c0 2024-08-19T12:56:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:33.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient(hunting): my global_id is 136447 2024-08-19T12:56:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 1 -- 172.21.15.26:0/2999556780 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45e4016c50 con 0x7f45e40064c0 2024-08-19T12:56:33.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 1 -- 172.21.15.26:0/2999556780 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3137961640 0 0) 0x7f45e4016c50 con 0x7f45e40064c0 2024-08-19T12:56:33.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6792/0 conn(0x7f45e40038c0 legacy=0x7f45e4005cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:56:33.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.26:6791/0 conn(0x7f45e4015c10 legacy=0x7f45e4016070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:33.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient: _finish_hunting 0 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 1 monclient: found mon.noname-i 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 1 -- 172.21.15.26:0/2999556780 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e4007000 con 0x7f45e40064c0 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient: _finish_auth 0 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient: _check_auth_tickets 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 1 -- 172.21.15.26:0/2999556780 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f45fc001d70 con 0x7f45e40064c0 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46027fc640 10 monclient: dump: 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:33.972 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:33.973 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f4609598640 5 monclient: authenticate success, global_id 136447 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f46037fe640 1 -- 172.21.15.26:0/2999556780 reap_dead start 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.964+0000 7f4609598640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.965+0000 7f46027fc640 1 -- 172.21.15.26:0/2999556780 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f45fc001f10 con 0x7f45e40064c0 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.965+0000 7f46027fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.965+0000 7f46027fc640 1 -- 172.21.15.26:0/2999556780 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f45fc0020c0 con 0x7f45e40064c0 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.965+0000 7f46027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.965+0000 7f46027fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.965+0000 7f46027fc640 10 monclient: dump: 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.965+0000 7f4609598640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.965+0000 7f4609598640 4 set_mon_vals no callback set 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.966+0000 7f4609598640 10 monclient: shutdown 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.966+0000 7f4609598640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.966+0000 7f4609598640 1 -- 172.21.15.26:0/2999556780 >> v1:172.21.15.104:6792/0 conn(0x7f45e40064c0 legacy=0x7f45e40068f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.966+0000 7f4609598640 1 -- 172.21.15.26:0/2999556780 shutdown_connections 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.966+0000 7f4609598640 1 -- 172.21.15.26:0/2999556780 >> 172.21.15.26:0/2999556780 conn(0x7f46040a8740 msgr2=0x7f46040a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.966+0000 7f4609598640 1 -- 172.21.15.26:0/2999556780 shutdown_connections 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.966+0000 7f4609598640 1 -- 172.21.15.26:0/2999556780 wait complete. 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.967+0000 7f4609598640 10 monclient: build_initial_monmap 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.967+0000 7f4609598640 10 monclient: monmap: 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:33.976 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:33.977 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:33.977 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 1 librados: starting msgr at 2024-08-19T12:56:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 1 librados: starting objecter 2024-08-19T12:56:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 1 Processor -- start 2024-08-19T12:56:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 1 -- start start 2024-08-19T12:56:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 1 librados: setting wanted keys 2024-08-19T12:56:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 1 librados: calling monclient init 2024-08-19T12:56:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 10 monclient: init 2024-08-19T12:56:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:33.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 10 monclient: _add_conns ranks=[1,5,6,8,4,0,2,3,7] 2024-08-19T12:56:33.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 10 monclient(hunting): picked mon.noname-f con 0x7f45e40064c0 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:33.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 10 monclient(hunting): picked mon.noname-h con 0x7f46041bee70 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:33.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 10 monclient(hunting): picked mon.noname-d con 0x7f46041bd860 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:33.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46041be310 con 0x7f45e40064c0 2024-08-19T12:56:33.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46041c0670 con 0x7f46041bee70 2024-08-19T12:56:33.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46041bfdd0 con 0x7f46041bd860 2024-08-19T12:56:33.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:33.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.968+0000 7f4609598640 10 monclient(hunting): authenticate will time out at 1788.055988s 2024-08-19T12:56:33.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.969+0000 7f46037fe640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f46041bd860 legacy=0x7f46041d6790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:56:33.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.969+0000 7f45fbfff640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f46041bee70 0x7f46041d3060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:48486/0 (socket says 172.21.15.26:48486) 2024-08-19T12:56:33.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.969+0000 7f45fbfff640 1 -- 172.21.15.26:0/1868386303 learned_addr learned my addr 172.21.15.26:0/1868386303 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:33.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.971+0000 7f45fbfff640 1 -- 172.21.15.26:0/1868386303 >> v1:172.21.15.104:6791/0 conn(0x7f46041bee70 legacy=0x7f46041d3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:33.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.971+0000 7f45fb7fe640 10 client.?.objecter ms_handle_connect 0x7f46041bee70 2024-08-19T12:56:33.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.971+0000 7f45fb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:33.983 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.976+0000 7f4602ffd640 1 -- 172.21.15.26:0/1868386303 >> v1:172.21.15.104:6789/0 conn(0x7f45e40064c0 legacy=0x7f4604132ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.976+0000 7f45fb7fe640 10 client.?.objecter ms_handle_connect 0x7f45e40064c0 2024-08-19T12:56:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.976+0000 7f45fb7fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.978+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2702663081 0 0) 0x7f46041be310 con 0x7f45e40064c0 2024-08-19T12:56:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.978+0000 7f45fb7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:33.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.978+0000 7f45fb7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.978+0000 7f45fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.978+0000 7f45fb7fe640 10 monclient(hunting): my global_id is 204265 2024-08-19T12:56:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.978+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45e40071d0 con 0x7f45e40064c0 2024-08-19T12:56:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.979+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1264508009 0 0) 0x7f45e40071d0 con 0x7f45e40064c0 2024-08-19T12:56:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.979+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46041be310 con 0x7f45e40064c0 2024-08-19T12:56:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.979+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f45f4001ce0 con 0x7f45e40064c0 2024-08-19T12:56:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.979+0000 7f45fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.979+0000 7f45fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:56:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.979+0000 7f45fb7fe640 10 monclient(hunting): dump: 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:33.986 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.979+0000 7f45fb7fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3692988369 0 0) 0x7f46041be310 con 0x7f45e40064c0 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 >> v1:172.21.15.26:6792/0 conn(0x7f46041bd860 legacy=0x7f46041d6790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 >> v1:172.21.15.104:6791/0 conn(0x7f46041bee70 legacy=0x7f46041d3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 1 monclient: found mon.f 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46041331d0 con 0x7f45e40064c0 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 10 monclient: _finish_auth 0 2024-08-19T12:56:33.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f4609598640 5 monclient: authenticate success, global_id 204265 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f4609598640 10 monclient: _renew_subs 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f4609598640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f4609598640 1 -- 172.21.15.26:0/1868386303 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f46041d7eb0 con 0x7f45e40064c0 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f4609598640 10 client.204265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f4609598640 10 monclient: _renew_subs 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f4609598640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f4609598640 1 -- 172.21.15.26:0/1868386303 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f46041d8260 con 0x7f45e40064c0 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f4609598640 1 librados: init done 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f45f4003ac0 con 0x7f45e40064c0 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f45f4003fb0 con 0x7f45e40064c0 2024-08-19T12:56:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f4601ffb640 4 set_mon_vals no callback set 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.980+0000 7f45fb7fe640 10 monclient: dump: 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:33.989 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.981+0000 7f4609598640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.981+0000 7f4609598640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.981+0000 7f4609598640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.981+0000 7f4609598640 1 -- 172.21.15.26:0/1868386303 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45e8005080 con 0x7f45e40064c0 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.981+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f45f4004250 con 0x7f45e40064c0 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.982+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(658..658 src has 350..658) ==== 5939+0+0 (unknown 1707548120 0 0) 0x7f45f404e3e0 con 0x7f45e40064c0 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.982+0000 7f45fb7fe640 10 client.204265.objecter ms_dispatch 0x7f4604000b90 osd_map(658..658 src has 350..658) 2024-08-19T12:56:33.990 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.982+0000 7f45fb7fe640 3 client.204265.objecter handle_osd_map got epochs [658,658] > 0 2024-08-19T12:56:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.982+0000 7f45fb7fe640 3 client.204265.objecter handle_osd_map decoding full epoch 658 2024-08-19T12:56:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.982+0000 7f45fb7fe640 20 client.204265.objecter dump_active .. 0 homeless 2024-08-19T12:56:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.982+0000 7f45fbfff640 10 monclient: get_auth_request con 0x7f45e4042de0 auth_method 0 2024-08-19T12:56:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.984+0000 7f45fb7fe640 10 client.204265.objecter ms_handle_connect 0x7f45e4042de0 2024-08-19T12:56:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.987+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 <== mon.1 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f460409ddf0 con 0x7f45e40064c0 2024-08-19T12:56:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.987+0000 7f45fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:33.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:33.987+0000 7f45fb7fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:34.252 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.251+0000 7f4609598640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-08-19T12:56:34.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.251+0000 7f4609598640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-08-19T12:56:34.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.251+0000 7f4609598640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:34.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.251+0000 7f4609598640 1 -- 172.21.15.26:0/1868386303 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f45e8005370 con 0x7f45e40064c0 2024-08-19T12:56:34.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.251+0000 7f45fb7fe640 1 -- 172.21.15.26:0/1868386303 <== mon.1 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2712 (unknown 254746657 0 2453146505) 0x7f45f4017530 con 0x7f45e40064c0 2024-08-19T12:56:34.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.251+0000 7f45fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-08-19T12:56:34.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.251+0000 7f45fb7fe640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:34.254 INFO:teuthology.orchestra.run.smithi026.stdout:{"election_epoch":130,"quorum":[1,2,4,5,7,8],"quorum_names":["f","b","c","h","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","modified":"2024-08-19T12:37:41.103727Z","created":"2024-08-19T12:37:41.103727Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6790","nonce":0}]},"addr":"172.21.15.104:6790/0","public_addr":"172.21.15.104:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6791","nonce":0}]},"addr":"172.21.15.26:6791/0","public_addr":"172.21.15.26:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6791","nonce":0}]},"addr":"172.21.15.104:6791/0","public_addr":"172.21.15.104:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6792","nonce":0}]},"addr":"172.21.15.26:6792/0","public_addr":"172.21.15.26:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6792","nonce":0}]},"addr":"172.21.15.104:6792/0","public_addr":"172.21.15.104:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6793","nonce":0}]},"addr":"172.21.15.26:6793/0","public_addr":"172.21.15.26:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-08-19T12:56:34.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 10 librados: watch_flush enter 2024-08-19T12:56:34.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 10 librados: watch_flush exit 2024-08-19T12:56:34.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 20 client.204265.objecter shutdown clearing up homeless session... 2024-08-19T12:56:34.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 10 client.204265.objecter successfully canceled tick 2024-08-19T12:56:34.257 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 1 -- 172.21.15.26:0/1868386303 >> v1:172.21.15.104:6800/899887506 conn(0x7f45e4042de0 legacy=0x7f45e40452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 10 monclient: shutdown 2024-08-19T12:56:34.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:34.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 1 -- 172.21.15.26:0/1868386303 >> v1:172.21.15.104:6789/0 conn(0x7f45e40064c0 legacy=0x7f4604132ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 1 -- 172.21.15.26:0/1868386303 shutdown_connections 2024-08-19T12:56:34.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 1 -- 172.21.15.26:0/1868386303 >> 172.21.15.26:0/1868386303 conn(0x7f46040a8740 msgr2=0x7f4604133970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:34.258 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 1 -- 172.21.15.26:0/1868386303 shutdown_connections 2024-08-19T12:56:34.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 1 -- 172.21.15.26:0/1868386303 wait complete. 2024-08-19T12:56:34.259 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.253+0000 7f4609598640 1 librados: shutdown 2024-08-19T12:56:34.265 INFO:tasks.mon_thrash.ceph_manager:quorum is size 6 2024-08-19T12:56:34.265 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon b 2024-08-19T12:56:34.266 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-08-19T12:56:34.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.364+0000 7f5551fa3640 10 monclient: get_monmap_and_config 2024-08-19T12:56:34.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.364+0000 7f5551fa3640 10 monclient: build_initial_monmap 2024-08-19T12:56:34.365 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.364+0000 7f5551fa3640 10 monclient: monmap: 2024-08-19T12:56:34.365 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:34.365736+0000 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:34.365736+0000 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:34.366 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:34.367 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:34.367 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.365+0000 7f5551fa3640 1 Processor -- start 2024-08-19T12:56:34.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.365+0000 7f5551fa3640 1 -- start start 2024-08-19T12:56:34.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.365+0000 7f5551fa3640 10 monclient: init 2024-08-19T12:56:34.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f5551fa3640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:34.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f5551fa3640 10 monclient: _add_conns ranks=[0,2,8,5,4,1,6,3,7] 2024-08-19T12:56:34.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f5551fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f554c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:34.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f5551fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f554c0a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:56:34.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f5551fa3640 10 monclient(hunting): picked mon.noname-e con 0x7f554c0a5d50 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:34.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f5551fa3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f554c136790 con 0x7f554c06e7b0 2024-08-19T12:56:34.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f5551fa3640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f554c09ddf0 con 0x7f554c0a5920 2024-08-19T12:56:34.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f5551fa3640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f554c09e0d0 con 0x7f554c0a5d50 2024-08-19T12:56:34.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f5551fa3640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:34.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f554b7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f554c06e7b0 legacy=0x7f554c136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:56:34.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f5551fa3640 10 monclient(hunting): authenticate will time out at 1788.453188s 2024-08-19T12:56:34.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f554affd640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f554c0a5920 0x7f554c13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60302/0 (socket says 172.21.15.26:60302) 2024-08-19T12:56:34.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f554bfff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f554c0a5d50 0x7f554c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40126/0 (socket says 172.21.15.26:40126) 2024-08-19T12:56:34.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.366+0000 7f554affd640 1 -- 172.21.15.26:0/2449880414 learned_addr learned my addr 172.21.15.26:0/2449880414 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:34.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.368+0000 7f554bfff640 1 -- 172.21.15.26:0/2449880414 >> v1:172.21.15.26:6793/0 conn(0x7f554c0a5d50 legacy=0x7f554c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.368+0000 7f554affd640 1 -- 172.21.15.26:0/2449880414 >> v1:172.21.15.26:6790/0 conn(0x7f554c0a5920 legacy=0x7f554c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.541+0000 7f554a7fc640 1 -- 172.21.15.26:0/2449880414 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 84031943 0 0) 0x7f5540001450 con 0x7f554c0a5920 2024-08-19T12:56:34.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.541+0000 7f554a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.541+0000 7f554a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.541+0000 7f554a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.544 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.541+0000 7f554a7fc640 10 monclient(hunting): my global_id is 27461 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.541+0000 7f554a7fc640 1 -- 172.21.15.26:0/2449880414 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5534003520 con 0x7f554c0a5920 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.541+0000 7f554a7fc640 1 -- 172.21.15.26:0/2449880414 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3730892165 0 0) 0x7f554c09e0d0 con 0x7f554c0a5d50 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.541+0000 7f554a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.541+0000 7f554a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.541+0000 7f554a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.541+0000 7f554a7fc640 10 monclient(hunting): my global_id is 184643 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.541+0000 7f554a7fc640 1 -- 172.21.15.26:0/2449880414 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5540001450 con 0x7f554c0a5d50 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 1 -- 172.21.15.26:0/2449880414 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3589344314 0 0) 0x7f5534003520 con 0x7f554c0a5920 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 1 -- 172.21.15.26:0/2449880414 >> v1:172.21.15.26:6793/0 conn(0x7f554c0a5d50 legacy=0x7f554c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 1 -- 172.21.15.26:0/2449880414 >> v1:172.21.15.26:6789/0 conn(0x7f554c06e7b0 legacy=0x7f554c136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 1 monclient: found mon.noname-b 2024-08-19T12:56:34.545 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 1 -- 172.21.15.26:0/2449880414 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554c09e3b0 con 0x7f554c0a5920 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 1 -- 172.21.15.26:0/2449880414 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5540001cb0 con 0x7f554c0a5920 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 10 monclient: dump: 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.546 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f5551fa3640 5 monclient: authenticate success, global_id 27461 2024-08-19T12:56:34.547 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f5551fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 1 -- 172.21.15.26:0/2449880414 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5540004950 con 0x7f554c0a5920 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 1 -- 172.21.15.26:0/2449880414 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5540004ec0 con 0x7f554c0a5920 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f5551fa3640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f5551fa3640 4 set_mon_vals no callback set 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f554a7fc640 10 monclient: dump: 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.549 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.549 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.549 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.549 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.549 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.549 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.549 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.549 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.549 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.549 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f5551fa3640 10 monclient: shutdown 2024-08-19T12:56:34.549 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f5551fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.542+0000 7f5551fa3640 1 -- 172.21.15.26:0/2449880414 >> v1:172.21.15.26:6790/0 conn(0x7f554c0a5920 legacy=0x7f554c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.543+0000 7f5551fa3640 1 -- 172.21.15.26:0/2449880414 shutdown_connections 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.543+0000 7f5551fa3640 1 -- 172.21.15.26:0/2449880414 >> 172.21.15.26:0/2449880414 conn(0x7f554c0a8740 msgr2=0x7f554c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.543+0000 7f5551fa3640 1 -- 172.21.15.26:0/2449880414 shutdown_connections 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.543+0000 7f5551fa3640 1 -- 172.21.15.26:0/2449880414 wait complete. 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.543+0000 7f5551fa3640 10 monclient: build_initial_monmap 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.543+0000 7f5551fa3640 10 monclient: monmap: 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:34.550 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.543+0000 7f5551fa3640 1 librados: starting msgr at 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.543+0000 7f5551fa3640 1 librados: starting objecter 2024-08-19T12:56:34.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 1 Processor -- start 2024-08-19T12:56:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 1 -- start start 2024-08-19T12:56:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 1 librados: setting wanted keys 2024-08-19T12:56:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 1 librados: calling monclient init 2024-08-19T12:56:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 10 monclient: init 2024-08-19T12:56:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 10 monclient: _add_conns ranks=[1,3,4,8,5,7,0,2,6] 2024-08-19T12:56:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 10 monclient(hunting): picked mon.noname-f con 0x7f554c06e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 10 monclient(hunting): picked mon.noname-g con 0x7f554c0a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f554c0a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:34.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f554c138f70 con 0x7f554c06e7b0 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f554c138460 con 0x7f554c0a5920 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f554c13aa30 con 0x7f554c0a5d50 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f5551fa3640 10 monclient(hunting): authenticate will time out at 1788.631807s 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f554affd640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f554c0a5920 legacy=0x7f554c137d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f554bfff640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f554c0a5d50 0x7f554c1d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:40446/0 (socket says 172.21.15.26:40446) 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.544+0000 7f554bfff640 1 -- 172.21.15.26:0/2203617708 learned_addr learned my addr 172.21.15.26:0/2203617708 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f554bfff640 1 -- 172.21.15.26:0/2203617708 >> v1:172.21.15.26:6791/0 conn(0x7f554c0a5d50 legacy=0x7f554c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 client.?.objecter ms_handle_connect 0x7f554c0a5d50 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 client.?.objecter ms_handle_connect 0x7f554c06e7b0 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:34.557 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f554b7fe640 1 -- 172.21.15.26:0/2203617708 >> v1:172.21.15.104:6789/0 conn(0x7f554c06e7b0 legacy=0x7f554c137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 749439068 0 0) 0x7f554c138f70 con 0x7f554c06e7b0 2024-08-19T12:56:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 monclient(hunting): my global_id is 204274 2024-08-19T12:56:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5538004eb0 con 0x7f554c06e7b0 2024-08-19T12:56:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 565800969 0 0) 0x7f5538004eb0 con 0x7f554c06e7b0 2024-08-19T12:56:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f554c138f70 con 0x7f554c06e7b0 2024-08-19T12:56:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f553c0046a0 con 0x7f554c06e7b0 2024-08-19T12:56:34.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:34.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 monclient(hunting): dump: 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.562 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1648104586 0 0) 0x7f554c138f70 con 0x7f554c06e7b0 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 >> v1:172.21.15.26:6791/0 conn(0x7f554c0a5d50 legacy=0x7f554c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 >> v1:172.21.15.104:6790/0 conn(0x7f554c0a5920 legacy=0x7f554c137d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 monclient: _finish_hunting 0 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 1 monclient: found mon.f 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:34.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554c1429b0 con 0x7f554c06e7b0 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 monclient: _finish_auth 0 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 10 monclient: _check_auth_tickets 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.547+0000 7f5533fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5551fa3640 5 monclient: authenticate success, global_id 204274 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5551fa3640 10 monclient: _renew_subs 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5551fa3640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5551fa3640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f554c139d70 con 0x7f554c06e7b0 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5551fa3640 10 client.204274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5551fa3640 10 monclient: _renew_subs 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5551fa3640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5551fa3640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f554c1d79b0 con 0x7f554c06e7b0 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5551fa3640 1 librados: init done 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f553c0048c0 con 0x7f554c06e7b0 2024-08-19T12:56:34.564 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5533fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f553c004e30 con 0x7f554c06e7b0 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5533fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5549ffb640 4 set_mon_vals no callback set 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5533fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5533fff640 10 monclient: dump: 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.565 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5551fa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5551fa3640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f552c0065c0 con 0x7f552c002860 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.548+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f553c0050d0 con 0x7f554c06e7b0 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.549+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(658..658 src has 350..658) ==== 5939+0+0 (unknown 1707548120 0 0) 0x7f553c04e520 con 0x7f554c06e7b0 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.549+0000 7f5533fff640 10 client.204274.objecter ms_dispatch 0x7f554c000b90 osd_map(658..658 src has 350..658) 2024-08-19T12:56:34.566 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.549+0000 7f5533fff640 3 client.204274.objecter handle_osd_map got epochs [658,658] > 0 2024-08-19T12:56:34.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.549+0000 7f5533fff640 3 client.204274.objecter handle_osd_map decoding full epoch 658 2024-08-19T12:56:34.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.549+0000 7f5533fff640 20 client.204274.objecter dump_active .. 0 homeless 2024-08-19T12:56:34.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.549+0000 7f554bfff640 10 monclient: get_auth_request con 0x7f5538040e80 auth_method 0 2024-08-19T12:56:34.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.551+0000 7f5533fff640 10 client.204274.objecter ms_handle_connect 0x7f552c002860 2024-08-19T12:56:34.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.551+0000 7f554affd640 1 -- 172.21.15.26:0/2203617708 >> v1:172.21.15.26:6790/0 conn(0x7f552c002860 legacy=0x7f552c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.551+0000 7f5533fff640 10 client.204274.objecter resend_mon_ops 2024-08-19T12:56:34.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.551+0000 7f5533fff640 10 client.204274.objecter ms_handle_connect 0x7f5538040e80 2024-08-19T12:56:34.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.565+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3939661480 0 0) 0x7f552c0065c0 con 0x7f552c002860 2024-08-19T12:56:34.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.565+0000 7f5533fff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.565+0000 7f5533fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.565+0000 7f5533fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.565+0000 7f5533fff640 10 monclient(hunting): my global_id is 27470 2024-08-19T12:56:34.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.565+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f554c138f70 con 0x7f552c002860 2024-08-19T12:56:34.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.566+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3379776136 0 0) 0x7f554c138f70 con 0x7f552c002860 2024-08-19T12:56:34.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.566+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f552c0065c0 con 0x7f552c002860 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.566+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5540004b40 con 0x7f552c002860 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.566+0000 7f5533fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.566+0000 7f5533fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.566+0000 7f5533fff640 10 monclient: dump: 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.570 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.566+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2111735116 0 0) 0x7f552c0065c0 con 0x7f552c002860 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.566+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.26:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f552c00a160 con 0x7f552c002860 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.566+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5540004e90 con 0x7f552c002860 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.566+0000 7f5533fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.566+0000 7f5533fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.567+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 >> v1:172.21.15.26:6790/0 conn(0x7f552c002860 legacy=0x7f552c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.567+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 >> v1:172.21.15.26:6790/0 conn(0x7f552c002860 legacy=0x7f552c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.574+0000 7f5551fa3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:34.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.574+0000 7f5551fa3640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f552c009e50 con 0x7f552c009110 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f554affd640 1 -- 172.21.15.26:0/2203617708 >> v1:172.21.15.26:6790/0 conn(0x7f552c009110 legacy=0x7f552c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 10 client.204274.objecter ms_handle_connect 0x7f552c009110 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 10 client.204274.objecter resend_mon_ops 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1191227012 0 0) 0x7f552c009e50 con 0x7f552c009110 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 10 monclient(hunting): my global_id is 27479 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f552c0065c0 con 0x7f552c009110 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2888815599 0 0) 0x7f552c0065c0 con 0x7f552c009110 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f552c009e50 con 0x7f552c009110 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5540005200 con 0x7f552c009110 2024-08-19T12:56:34.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 10 monclient: dump: 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.579 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.580 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1822275516 0 0) 0x7f552c009e50 con 0x7f552c009110 2024-08-19T12:56:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.576+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 --> v1:172.21.15.26:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f552c00c3f0 con 0x7f552c009110 2024-08-19T12:56:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 <== mon.2 v1:172.21.15.26:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7680 (unknown 0 0 1436181619) 0x7f5540007e10 con 0x7f552c009110 2024-08-19T12:56:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5533fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5533fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 >> v1:172.21.15.26:6790/0 conn(0x7f552c009110 legacy=0x7f552c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5533fff640 1 -- 172.21.15.26:0/2203617708 >> v1:172.21.15.26:6790/0 conn(0x7f552c009110 legacy=0x7f552c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:34.581 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:34.581 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:34.581 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:34.581 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:34.581 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 130, 2024-08-19T12:56:34.581 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 1107632, 2024-08-19T12:56:34.581 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:34.581 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:34.581 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:34.581 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:34.582 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:34.582 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:34.582 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:34.582 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.582 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:56:34.582 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:34.582 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:34.582 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:34.582 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:34.582 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:34.583 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:34.583 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:34.583 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:34.583 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:34.583 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:34.583 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:34.583 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:34.583 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:34.583 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:34.583 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.584 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:34.584 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:34.584 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:34.584 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:34.584 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:34.584 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:34.584 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:34.584 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:34.584 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:34.584 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:34.584 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:34.585 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:34.586 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.587 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:34.589 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:34.590 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.591 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:34.593 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.594 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.595 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:34.596 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.598 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:34.599 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:34.599 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.599 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.599 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.599 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.599 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.599 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:34.599 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.600 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:34.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.601 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:34.601 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:34.601 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 4 2024-08-19T12:56:34.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.601 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.601 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:34.601 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 10 librados: watch_flush enter 2024-08-19T12:56:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 10 librados: watch_flush exit 2024-08-19T12:56:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 20 client.204274.objecter shutdown clearing up homeless session... 2024-08-19T12:56:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 10 client.204274.objecter successfully canceled tick 2024-08-19T12:56:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 1 -- 172.21.15.26:0/2203617708 >> v1:172.21.15.104:6800/899887506 conn(0x7f5538040e80 legacy=0x7f5538043340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 10 monclient: shutdown 2024-08-19T12:56:34.603 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 1 -- 172.21.15.26:0/2203617708 >> v1:172.21.15.104:6789/0 conn(0x7f554c06e7b0 legacy=0x7f554c137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f554bfff640 1 -- 172.21.15.26:0/2203617708 reap_dead start 2024-08-19T12:56:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f554bfff640 1 -- 172.21.15.26:0/2203617708 reap_dead start 2024-08-19T12:56:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 1 -- 172.21.15.26:0/2203617708 shutdown_connections 2024-08-19T12:56:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 1 -- 172.21.15.26:0/2203617708 >> 172.21.15.26:0/2203617708 conn(0x7f554c0a8740 msgr2=0x7f554c0aa450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 1 -- 172.21.15.26:0/2203617708 shutdown_connections 2024-08-19T12:56:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 1 -- 172.21.15.26:0/2203617708 wait complete. 2024-08-19T12:56:34.604 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.577+0000 7f5551fa3640 1 librados: shutdown 2024-08-19T12:56:34.604 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T12:56:34.604 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon c 2024-08-19T12:56:34.605 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-08-19T12:56:34.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.704+0000 7f39120de640 10 monclient: get_monmap_and_config 2024-08-19T12:56:34.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.704+0000 7f39120de640 10 monclient: build_initial_monmap 2024-08-19T12:56:34.706 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.704+0000 7f39120de640 10 monclient: monmap: 2024-08-19T12:56:34.706 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:34.706 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.706 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:34.705296+0000 2024-08-19T12:56:34.706 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:34.705296+0000 2024-08-19T12:56:34.706 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:34.706 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.706 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:34.706 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:34.706 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:34.706 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:34.706 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:34.707 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:34.707 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:34.707 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:34.707 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:34.707 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 1 Processor -- start 2024-08-19T12:56:34.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 1 -- start start 2024-08-19T12:56:34.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 10 monclient: init 2024-08-19T12:56:34.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:34.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 10 monclient: _add_conns ranks=[0,4,6,1,2,7,3,5,8] 2024-08-19T12:56:34.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 10 monclient(hunting): picked mon.noname-a con 0x7f390c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:34.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 10 monclient(hunting): picked mon.noname-c con 0x7f390c0a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:34.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 10 monclient(hunting): picked mon.noname-d con 0x7f390c0a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:34.712 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390c136790 con 0x7f390c06e7b0 2024-08-19T12:56:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390c09ddf0 con 0x7f390c0a5920 2024-08-19T12:56:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390c09e0d0 con 0x7f390c0a5d50 2024-08-19T12:56:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f39120de640 10 monclient(hunting): authenticate will time out at 1788.792762s 2024-08-19T12:56:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f390b7fe640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f390c06e7b0 legacy=0x7f390c136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:56:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f390bfff640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f390c0a5d50 legacy=0x7f390c141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:56:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f390affd640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f390c0a5920 0x7f390c13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:40452/0 (socket says 172.21.15.26:40452) 2024-08-19T12:56:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.705+0000 7f390affd640 1 -- 172.21.15.26:0/4158124312 learned_addr learned my addr 172.21.15.26:0/4158124312 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:34.713 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390affd640 1 -- 172.21.15.26:0/4158124312 >> v1:172.21.15.26:6791/0 conn(0x7f390c0a5920 legacy=0x7f390c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 1 -- 172.21.15.26:0/4158124312 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4144149863 0 0) 0x7f3900001450 con 0x7f390c0a5920 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient(hunting): my global_id is 184792 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 1 -- 172.21.15.26:0/4158124312 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38f4003520 con 0x7f390c0a5920 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 1 -- 172.21.15.26:0/4158124312 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 617837959 0 0) 0x7f38f4003520 con 0x7f390c0a5920 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 1 -- 172.21.15.26:0/4158124312 >> v1:172.21.15.26:6792/0 conn(0x7f390c0a5d50 legacy=0x7f390c141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 1 -- 172.21.15.26:0/4158124312 >> v1:172.21.15.26:6789/0 conn(0x7f390c06e7b0 legacy=0x7f390c136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 1 monclient: found mon.noname-c 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:34.714 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 1 -- 172.21.15.26:0/4158124312 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390c09e3b0 con 0x7f390c0a5920 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f39120de640 5 monclient: authenticate success, global_id 184792 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 1 -- 172.21.15.26:0/4158124312 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3900001cb0 con 0x7f390c0a5920 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient: dump: 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.715 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f39120de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 1 -- 172.21.15.26:0/4158124312 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3900004990 con 0x7f390c0a5920 2024-08-19T12:56:34.716 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f39120de640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 1 -- 172.21.15.26:0/4158124312 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3900004ec0 con 0x7f390c0a5920 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f39120de640 4 set_mon_vals no callback set 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f390a7fc640 10 monclient: dump: 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.717 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.718 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.718 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.718 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.718 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.718 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.718 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.718 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.718 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.718 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f39120de640 10 monclient: shutdown 2024-08-19T12:56:34.719 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f39120de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:34.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.708+0000 7f39120de640 1 -- 172.21.15.26:0/4158124312 >> v1:172.21.15.26:6791/0 conn(0x7f390c0a5920 legacy=0x7f390c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.709+0000 7f39120de640 1 -- 172.21.15.26:0/4158124312 shutdown_connections 2024-08-19T12:56:34.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.709+0000 7f39120de640 1 -- 172.21.15.26:0/4158124312 >> 172.21.15.26:0/4158124312 conn(0x7f390c0a8740 msgr2=0x7f390c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:34.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.709+0000 7f39120de640 1 -- 172.21.15.26:0/4158124312 shutdown_connections 2024-08-19T12:56:34.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.709+0000 7f39120de640 1 -- 172.21.15.26:0/4158124312 wait complete. 2024-08-19T12:56:34.720 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.709+0000 7f39120de640 10 monclient: build_initial_monmap 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.709+0000 7f39120de640 10 monclient: monmap: 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:34.721 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.709+0000 7f39120de640 1 librados: starting msgr at 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.709+0000 7f39120de640 1 librados: starting objecter 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.709+0000 7f39120de640 1 Processor -- start 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 1 -- start start 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 1 librados: setting wanted keys 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 1 librados: calling monclient init 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 10 monclient: init 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:34.722 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 10 monclient: _add_conns ranks=[8,6,3,4,1,2,7,5,0] 2024-08-19T12:56:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 10 monclient(hunting): picked mon.noname-e con 0x7f390c06e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 10 monclient(hunting): picked mon.noname-d con 0x7f390c0a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 10 monclient(hunting): picked mon.noname-g con 0x7f390c0a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:56:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390c126b00 con 0x7f390c0a5d50 2024-08-19T12:56:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390c134780 con 0x7f390c0a5920 2024-08-19T12:56:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390c142ec0 con 0x7f390c06e7b0 2024-08-19T12:56:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f39120de640 10 monclient(hunting): authenticate will time out at 1788.797576s 2024-08-19T12:56:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f390affd640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f390c0a5920 legacy=0x7f390c123f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:56:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f390b7fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f390c06e7b0 0x7f390c123850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40128/0 (socket says 172.21.15.26:40128) 2024-08-19T12:56:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f390b7fe640 1 -- 172.21.15.26:0/3067673956 learned_addr learned my addr 172.21.15.26:0/3067673956 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:34.723 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.710+0000 7f390bfff640 1 -- 172.21.15.26:0/3067673956 >> v1:172.21.15.104:6790/0 conn(0x7f390c0a5d50 legacy=0x7f390c1d6290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:56:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.712+0000 7f390b7fe640 1 -- 172.21.15.26:0/3067673956 >> v1:172.21.15.26:6793/0 conn(0x7f390c06e7b0 legacy=0x7f390c123850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.712+0000 7f38f3fff640 10 client.?.objecter ms_handle_connect 0x7f390c06e7b0 2024-08-19T12:56:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.712+0000 7f38f3fff640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.712+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2430370540 0 0) 0x7f390c142ec0 con 0x7f390c06e7b0 2024-08-19T12:56:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.712+0000 7f38f3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.712+0000 7f38f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.712+0000 7f38f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.712+0000 7f38f3fff640 10 monclient(hunting): my global_id is 184652 2024-08-19T12:56:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.712+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38f8004eb0 con 0x7f390c06e7b0 2024-08-19T12:56:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3525841660 0 0) 0x7f38f8004eb0 con 0x7f390c06e7b0 2024-08-19T12:56:34.729 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f390c142ec0 con 0x7f390c06e7b0 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f38fc004620 con 0x7f390c06e7b0 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 10 monclient(hunting): dump: 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.730 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2754227978 0 0) 0x7f390c142ec0 con 0x7f390c06e7b0 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 >> v1:172.21.15.26:6792/0 conn(0x7f390c0a5920 legacy=0x7f390c123f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 >> v1:172.21.15.104:6790/0 conn(0x7f390c0a5d50 legacy=0x7f390c1d6290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 10 monclient: _finish_hunting 0 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 1 monclient: found mon.e 2024-08-19T12:56:34.731 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390c126820 con 0x7f390c06e7b0 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 10 monclient: _finish_auth 0 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 10 monclient: _check_auth_tickets 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f39120de640 5 monclient: authenticate success, global_id 184652 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f39120de640 10 monclient: _renew_subs 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f39120de640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f39120de640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f390c126de0 con 0x7f390c06e7b0 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f39120de640 10 client.184652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f39120de640 10 monclient: _renew_subs 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f39120de640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:34.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f39120de640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f390c124850 con 0x7f390c06e7b0 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f39120de640 1 librados: init done 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f38fc004930 con 0x7f390c06e7b0 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f38fc004ea0 con 0x7f390c06e7b0 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f3909ffb640 4 set_mon_vals no callback set 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.713+0000 7f38f3fff640 10 monclient: dump: 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.733 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.714+0000 7f39120de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.714+0000 7f39120de640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f38ec0065c0 con 0x7f38ec002860 2024-08-19T12:56:34.734 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.714+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f38fc005040 con 0x7f390c06e7b0 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.714+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(659..659 src has 350..659) ==== 5997+0+0 (unknown 3616967900 0 0) 0x7f38fc04e4a0 con 0x7f390c06e7b0 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.714+0000 7f38f3fff640 10 client.184652.objecter ms_dispatch 0x7f390c000b90 osd_map(659..659 src has 350..659) 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.714+0000 7f38f3fff640 3 client.184652.objecter handle_osd_map got epochs [659,659] > 0 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.714+0000 7f38f3fff640 3 client.184652.objecter handle_osd_map decoding full epoch 659 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.714+0000 7f38f3fff640 20 client.184652.objecter dump_active .. 0 homeless 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.714+0000 7f390bfff640 10 monclient: get_auth_request con 0x7f38f8040e20 auth_method 0 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 10 client.184652.objecter ms_handle_connect 0x7f38ec002860 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f390affd640 1 -- 172.21.15.26:0/3067673956 >> v1:172.21.15.26:6791/0 conn(0x7f38ec002860 legacy=0x7f38ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 10 client.184652.objecter resend_mon_ops 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1264056095 0 0) 0x7f38ec0065c0 con 0x7f38ec002860 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 10 monclient(hunting): my global_id is 184801 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f390c142ec0 con 0x7f38ec002860 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3619326436 0 0) 0x7f390c142ec0 con 0x7f38ec002860 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38ec0065c0 con 0x7f38ec002860 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3900001ad0 con 0x7f38ec002860 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.716+0000 7f38f3fff640 10 monclient: dump: 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.736 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.717+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4009896964 0 0) 0x7f38ec0065c0 con 0x7f38ec002860 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.717+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38ec00a160 con 0x7f38ec002860 2024-08-19T12:56:34.737 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.717+0000 7f38f3fff640 10 client.184652.objecter ms_handle_connect 0x7f38f8040e20 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.717+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3900001ad0 con 0x7f38ec002860 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.717+0000 7f38f3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.717+0000 7f38f3fff640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.717+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 >> v1:172.21.15.26:6791/0 conn(0x7f38ec002860 legacy=0x7f38ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.717+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 >> v1:172.21.15.26:6791/0 conn(0x7f38ec002860 legacy=0x7f38ec004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.723+0000 7f39120de640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.723+0000 7f39120de640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f38ec009e30 con 0x7f38ec009110 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.725+0000 7f390affd640 1 -- 172.21.15.26:0/3067673956 >> v1:172.21.15.26:6791/0 conn(0x7f38ec009110 legacy=0x7f38ec009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.725+0000 7f38f3fff640 10 client.184652.objecter ms_handle_connect 0x7f38ec009110 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.725+0000 7f38f3fff640 10 client.184652.objecter resend_mon_ops 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2732103967 0 0) 0x7f38ec009e30 con 0x7f38ec009110 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.738 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 10 monclient(hunting): my global_id is 184810 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38ec0065c0 con 0x7f38ec009110 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3110567383 0 0) 0x7f38ec0065c0 con 0x7f38ec009110 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f38ec009e30 con 0x7f38ec009110 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3900004e90 con 0x7f38ec009110 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 10 monclient: dump: 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.739 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3271389950 0 0) 0x7f38ec009e30 con 0x7f38ec009110 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 --> v1:172.21.15.26:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f38ec00c3f0 con 0x7f38ec009110 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 <== mon.4 v1:172.21.15.26:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7679 (unknown 0 0 3194023482) 0x7f39000086b0 con 0x7f38ec009110 2024-08-19T12:56:34.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 >> v1:172.21.15.26:6791/0 conn(0x7f38ec009110 legacy=0x7f38ec009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f38f3fff640 1 -- 172.21.15.26:0/3067673956 >> v1:172.21.15.26:6791/0 conn(0x7f38ec009110 legacy=0x7f38ec009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:34.741 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:34.741 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 130, 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 320192, 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.742 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:34.743 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:34.744 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:34.745 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:34.746 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.748 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.749 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.750 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.751 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:34.752 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:34.753 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:34.754 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.755 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.756 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.757 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:34.758 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.760 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.761 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:34.762 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:34.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f39120de640 10 librados: watch_flush enter 2024-08-19T12:56:34.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f39120de640 10 librados: watch_flush exit 2024-08-19T12:56:34.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f39120de640 20 client.184652.objecter shutdown clearing up homeless session... 2024-08-19T12:56:34.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.726+0000 7f39120de640 10 client.184652.objecter successfully canceled tick 2024-08-19T12:56:34.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.727+0000 7f39120de640 1 -- 172.21.15.26:0/3067673956 >> v1:172.21.15.104:6800/899887506 conn(0x7f38f8040e20 legacy=0x7f38f80432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.727+0000 7f39120de640 10 monclient: shutdown 2024-08-19T12:56:34.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.727+0000 7f39120de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.727+0000 7f39120de640 1 -- 172.21.15.26:0/3067673956 >> v1:172.21.15.26:6793/0 conn(0x7f390c06e7b0 legacy=0x7f390c123850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.727+0000 7f390bfff640 1 -- 172.21.15.26:0/3067673956 reap_dead start 2024-08-19T12:56:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.727+0000 7f390bfff640 1 -- 172.21.15.26:0/3067673956 reap_dead start 2024-08-19T12:56:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.727+0000 7f39120de640 1 -- 172.21.15.26:0/3067673956 shutdown_connections 2024-08-19T12:56:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.727+0000 7f39120de640 1 -- 172.21.15.26:0/3067673956 >> 172.21.15.26:0/3067673956 conn(0x7f390c0a8740 msgr2=0x7f390c06d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.727+0000 7f39120de640 1 -- 172.21.15.26:0/3067673956 shutdown_connections 2024-08-19T12:56:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.727+0000 7f39120de640 1 -- 172.21.15.26:0/3067673956 wait complete. 2024-08-19T12:56:34.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.727+0000 7f39120de640 1 librados: shutdown 2024-08-19T12:56:34.767 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T12:56:34.767 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon e 2024-08-19T12:56:34.768 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-08-19T12:56:34.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.868+0000 7f3dd31f2640 10 monclient: get_monmap_and_config 2024-08-19T12:56:34.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.868+0000 7f3dd31f2640 10 monclient: build_initial_monmap 2024-08-19T12:56:34.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.868+0000 7f3dd31f2640 10 monclient: monmap: 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:34.869600+0000 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:34.869600+0000 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:34.870 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:34.871 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:34.871 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:34.871 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.869+0000 7f3dd31f2640 1 Processor -- start 2024-08-19T12:56:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.869+0000 7f3dd31f2640 1 -- start start 2024-08-19T12:56:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.869+0000 7f3dd31f2640 10 monclient: init 2024-08-19T12:56:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.869+0000 7f3dd31f2640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.869+0000 7f3dd31f2640 10 monclient: _add_conns ranks=[7,5,8,3,0,4,1,2,6] 2024-08-19T12:56:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.869+0000 7f3dd31f2640 10 monclient(hunting): picked mon.noname-i con 0x7f3dcc073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.869+0000 7f3dd31f2640 10 monclient(hunting): picked mon.noname-h con 0x7f3dcc0a57a0 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.869+0000 7f3dd31f2640 10 monclient(hunting): picked mon.noname-e con 0x7f3dcc13f2e0 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.870+0000 7f3dd31f2640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc09dae0 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.870+0000 7f3dd31f2640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc136910 con 0x7f3dcc073470 2024-08-19T12:56:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.870+0000 7f3dd31f2640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc09dd50 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.875 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.870+0000 7f3dd31f2640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.870+0000 7f3dd31f2640 10 monclient(hunting): authenticate will time out at 1788.957058s 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.870+0000 7f3dd178f640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f3dcc13f2e0 0x7f3dcc1416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40136/0 (socket says 172.21.15.26:40136) 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.870+0000 7f3dd178f640 1 -- 172.21.15.26:0/2719992932 learned_addr learned my addr 172.21.15.26:0/2719992932 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dd178f640 1 -- 172.21.15.26:0/2719992932 >> v1:172.21.15.26:6793/0 conn(0x7f3dcc13f2e0 legacy=0x7f3dcc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc3fff640 1 -- 172.21.15.26:0/2719992932 >> v1:172.21.15.104:6791/0 conn(0x7f3dcc0a57a0 legacy=0x7f3dcc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 1 -- 172.21.15.26:0/2719992932 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1517923280 0 0) 0x7f3dcc09dd50 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient(hunting): my global_id is 184661 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 1 -- 172.21.15.26:0/2719992932 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3db4003520 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dd0f8e640 1 -- 172.21.15.26:0/2719992932 >> v1:172.21.15.104:6792/0 conn(0x7f3dcc073470 legacy=0x7f3dcc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 1 -- 172.21.15.26:0/2719992932 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2454964796 0 0) 0x7f3db4003520 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 1 -- 172.21.15.26:0/2719992932 >> v1:172.21.15.104:6792/0 conn(0x7f3dcc073470 legacy=0x7f3dcc136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 1 -- 172.21.15.26:0/2719992932 >> v1:172.21.15.104:6791/0 conn(0x7f3dcc0a57a0 legacy=0x7f3dcc0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient: _finish_hunting 0 2024-08-19T12:56:34.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 1 monclient: found mon.noname-e 2024-08-19T12:56:34.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:56:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 1 -- 172.21.15.26:0/2719992932 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc09e030 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient: _finish_auth 0 2024-08-19T12:56:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient: _check_auth_tickets 2024-08-19T12:56:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 1 -- 172.21.15.26:0/2719992932 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dc4001e60 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dd31f2640 5 monclient: authenticate success, global_id 184661 2024-08-19T12:56:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dd31f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:56:34.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient: dump: 2024-08-19T12:56:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.879 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.880 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dd31f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 1 -- 172.21.15.26:0/2719992932 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3dc4004c20 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 1 -- 172.21.15.26:0/2719992932 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dc4005190 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dd31f2640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dd31f2640 4 set_mon_vals no callback set 2024-08-19T12:56:34.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.872+0000 7f3dc37fe640 10 monclient: dump: 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.881 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.873+0000 7f3dd31f2640 10 monclient: shutdown 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.873+0000 7f3dd31f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.873+0000 7f3dd31f2640 1 -- 172.21.15.26:0/2719992932 >> v1:172.21.15.26:6793/0 conn(0x7f3dcc13f2e0 legacy=0x7f3dcc1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.873+0000 7f3dd31f2640 1 -- 172.21.15.26:0/2719992932 shutdown_connections 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.873+0000 7f3dd31f2640 1 -- 172.21.15.26:0/2719992932 >> 172.21.15.26:0/2719992932 conn(0x7f3dcc0abff0 msgr2=0x7f3dcc0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.873+0000 7f3dd31f2640 1 -- 172.21.15.26:0/2719992932 shutdown_connections 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.873+0000 7f3dd31f2640 1 -- 172.21.15.26:0/2719992932 wait complete. 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.873+0000 7f3dd31f2640 10 monclient: build_initial_monmap 2024-08-19T12:56:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.873+0000 7f3dd31f2640 10 monclient: monmap: 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:34.884 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:34.884 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.874+0000 7f3dd31f2640 1 librados: starting msgr at 2024-08-19T12:56:34.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.874+0000 7f3dd31f2640 1 librados: starting objecter 2024-08-19T12:56:34.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.874+0000 7f3dd31f2640 1 Processor -- start 2024-08-19T12:56:34.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.874+0000 7f3dd31f2640 1 -- start start 2024-08-19T12:56:34.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.874+0000 7f3dd31f2640 1 librados: setting wanted keys 2024-08-19T12:56:34.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.874+0000 7f3dd31f2640 1 librados: calling monclient init 2024-08-19T12:56:34.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.874+0000 7f3dd31f2640 10 monclient: init 2024-08-19T12:56:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.874+0000 7f3dd31f2640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.874+0000 7f3dd31f2640 10 monclient: _add_conns ranks=[0,1,7,6,2,8,3,5,4] 2024-08-19T12:56:34.894 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.874+0000 7f3dd31f2640 10 monclient(hunting): picked mon.noname-a con 0x7f3dcc073470 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.875+0000 7f3dd31f2640 10 monclient(hunting): picked mon.noname-f con 0x7f3dcc0a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.875+0000 7f3dd31f2640 10 monclient(hunting): picked mon.noname-i con 0x7f3dcc13f2e0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.875+0000 7f3dd31f2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc131740 con 0x7f3dcc073470 2024-08-19T12:56:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.875+0000 7f3dd0f8e640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f3dcc073470 legacy=0x7f3dcc12de40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:56:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.875+0000 7f3dd31f2640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc0aa1b0 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.875+0000 7f3dd31f2640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc13f070 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.875+0000 7f3dd31f2640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.875+0000 7f3dd31f2640 10 monclient(hunting): authenticate will time out at 1788.962083s 2024-08-19T12:56:34.895 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.875+0000 7f3dd178f640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f3dcc13f2e0 0x7f3dcc1d6150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:35236/0 (socket says 172.21.15.26:35236) 2024-08-19T12:56:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.875+0000 7f3dd178f640 1 -- 172.21.15.26:0/3569950949 learned_addr learned my addr 172.21.15.26:0/3569950949 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc3fff640 1 -- 172.21.15.26:0/3569950949 >> v1:172.21.15.104:6789/0 conn(0x7f3dcc0a57a0 legacy=0x7f3dcc12e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 10 client.?.objecter ms_handle_connect 0x7f3dcc0a57a0 2024-08-19T12:56:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 10 client.?.objecter ms_handle_connect 0x7f3dcc13f2e0 2024-08-19T12:56:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dd178f640 1 -- 172.21.15.26:0/3569950949 >> v1:172.21.15.104:6792/0 conn(0x7f3dcc13f2e0 legacy=0x7f3dcc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2156461264 0 0) 0x7f3dcc0aa1b0 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.896 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 10 monclient(hunting): my global_id is 204283 2024-08-19T12:56:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3db8004eb0 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2672520960 0 0) 0x7f3dcc13f070 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 10 monclient(hunting): my global_id is 136465 2024-08-19T12:56:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.877+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dcc0aa1b0 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 589649686 0 0) 0x7f3db8004eb0 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3dcc13f070 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dbc003610 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:56:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient(hunting): dump: 2024-08-19T12:56:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.898 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.899 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.899 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 30579864 0 0) 0x7f3dcc0aa1b0 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3db8004eb0 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dc40020b0 con 0x7f3dcc13f2e0 2024-08-19T12:56:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient(hunting): dump: 2024-08-19T12:56:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.900 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.901 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.901 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3077610977 0 0) 0x7f3dcc13f070 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 >> v1:172.21.15.104:6792/0 conn(0x7f3dcc13f2e0 legacy=0x7f3dcc1d6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 >> v1:172.21.15.26:6789/0 conn(0x7f3dcc073470 legacy=0x7f3dcc12de40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:56:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient: _finish_hunting 0 2024-08-19T12:56:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 monclient: found mon.f 2024-08-19T12:56:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc142ba0 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient: _finish_auth 0 2024-08-19T12:56:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient: _check_auth_tickets 2024-08-19T12:56:34.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dd31f2640 5 monclient: authenticate success, global_id 204283 2024-08-19T12:56:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dd31f2640 10 monclient: _renew_subs 2024-08-19T12:56:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dd31f2640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dd31f2640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3dcc12ec60 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dd31f2640 10 client.204283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dd31f2640 10 monclient: _renew_subs 2024-08-19T12:56:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dd31f2640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:56:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dd31f2640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3dcc132f80 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dd31f2640 1 librados: init done 2024-08-19T12:56:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3dbc003b90 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.903 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dbc004140 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc2ffd640 4 set_mon_vals no callback set 2024-08-19T12:56:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.878+0000 7f3dc17fa640 10 monclient: dump: 2024-08-19T12:56:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.905 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.879+0000 7f3dd31f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:34.905 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.879+0000 7f3dd31f2640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3dac0065c0 con 0x7f3dac002860 2024-08-19T12:56:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.879+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f3dbc011a80 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.879+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(659..659 src has 350..659) ==== 5997+0+0 (unknown 3616967900 0 0) 0x7f3dbc04e500 con 0x7f3dcc0a57a0 2024-08-19T12:56:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.879+0000 7f3dc17fa640 10 client.204283.objecter ms_dispatch 0x7f3dcc000b90 osd_map(659..659 src has 350..659) 2024-08-19T12:56:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.879+0000 7f3dc17fa640 3 client.204283.objecter handle_osd_map got epochs [659,659] > 0 2024-08-19T12:56:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.879+0000 7f3dc17fa640 3 client.204283.objecter handle_osd_map decoding full epoch 659 2024-08-19T12:56:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.879+0000 7f3dc17fa640 20 client.204283.objecter dump_active .. 0 homeless 2024-08-19T12:56:34.906 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.879+0000 7f3dd178f640 10 monclient: get_auth_request con 0x7f3db80415f0 auth_method 0 2024-08-19T12:56:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dd0f8e640 1 -- 172.21.15.26:0/3569950949 >> v1:172.21.15.26:6793/0 conn(0x7f3dac002860 legacy=0x7f3dac004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 10 client.204283.objecter ms_handle_connect 0x7f3dac002860 2024-08-19T12:56:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 10 client.204283.objecter resend_mon_ops 2024-08-19T12:56:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4174141266 0 0) 0x7f3dac0065c0 con 0x7f3dac002860 2024-08-19T12:56:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 10 monclient(hunting): my global_id is 184670 2024-08-19T12:56:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dcc13f070 con 0x7f3dac002860 2024-08-19T12:56:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1575163596 0 0) 0x7f3dcc13f070 con 0x7f3dac002860 2024-08-19T12:56:34.907 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3dac0065c0 con 0x7f3dac002860 2024-08-19T12:56:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dc8001be0 con 0x7f3dac002860 2024-08-19T12:56:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 10 monclient: dump: 2024-08-19T12:56:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.908 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.909 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 773886601 0 0) 0x7f3dac0065c0 con 0x7f3dac002860 2024-08-19T12:56:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.881+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.26:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3dac00a160 con 0x7f3dac002860 2024-08-19T12:56:34.909 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.882+0000 7f3dc17fa640 10 client.204283.objecter ms_handle_connect 0x7f3db80415f0 2024-08-19T12:56:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.882+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3dc80057c0 con 0x7f3dac002860 2024-08-19T12:56:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.882+0000 7f3dc17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.882+0000 7f3dc17fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.882+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 >> v1:172.21.15.26:6793/0 conn(0x7f3dac002860 legacy=0x7f3dac004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.882+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 >> v1:172.21.15.26:6793/0 conn(0x7f3dac002860 legacy=0x7f3dac004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.888+0000 7f3dd31f2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.888+0000 7f3dd31f2640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3dac009e70 con 0x7f3dac009130 2024-08-19T12:56:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dd0f8e640 1 -- 172.21.15.26:0/3569950949 >> v1:172.21.15.26:6793/0 conn(0x7f3dac009130 legacy=0x7f3dac0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 10 client.204283.objecter ms_handle_connect 0x7f3dac009130 2024-08-19T12:56:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 10 client.204283.objecter resend_mon_ops 2024-08-19T12:56:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1502393929 0 0) 0x7f3dac009e70 con 0x7f3dac009130 2024-08-19T12:56:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 10 monclient(hunting): my global_id is 184679 2024-08-19T12:56:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dac0065c0 con 0x7f3dac009130 2024-08-19T12:56:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3048627297 0 0) 0x7f3dac0065c0 con 0x7f3dac009130 2024-08-19T12:56:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3dac009e70 con 0x7f3dac009130 2024-08-19T12:56:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3dc8003d40 con 0x7f3dac009130 2024-08-19T12:56:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:34.911 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.890+0000 7f3dc17fa640 10 monclient: dump: 2024-08-19T12:56:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:34.912 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:34.913 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:34.913 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:34.913 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:34.913 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:34.913 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:34.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2204860013 0 0) 0x7f3dac009e70 con 0x7f3dac009130 2024-08-19T12:56:34.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 --> v1:172.21.15.26:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3dac00c410 con 0x7f3dac009130 2024-08-19T12:56:34.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 <== mon.8 v1:172.21.15.26:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7512 (unknown 0 0 710726878) 0x7f3dc8005970 con 0x7f3dac009130 2024-08-19T12:56:34.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dc17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:34.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dc17fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:34.913 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 >> v1:172.21.15.26:6793/0 conn(0x7f3dac009130 legacy=0x7f3dac0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.914 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dc17fa640 1 -- 172.21.15.26:0/3569950949 >> v1:172.21.15.26:6793/0 conn(0x7f3dac009130 legacy=0x7f3dac0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:34.914 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:34.914 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 130, 2024-08-19T12:56:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 320358, 2024-08-19T12:56:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 0, 2024-08-19T12:56:34.915 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:34.916 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:34.917 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:34.918 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:34.919 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:34.920 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:34.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.923 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.924 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.925 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:34.926 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:34.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.928 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:34.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:34.934 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.934 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.934 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.934 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:34.934 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:34.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:34.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:34.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:34.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:34.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:34.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:34.935 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:34.935 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:34.935 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:34.935 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:34.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.936 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:34.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:34.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.936 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:34.936 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:34.936 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:34.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.936 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:34.937 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:34.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:34.937 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:34.937 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:34.937 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:56:34.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:34.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:34.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:34.937 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:34.937 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:34.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 10 librados: watch_flush enter 2024-08-19T12:56:34.939 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 10 librados: watch_flush exit 2024-08-19T12:56:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 20 client.204283.objecter shutdown clearing up homeless session... 2024-08-19T12:56:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 10 client.204283.objecter successfully canceled tick 2024-08-19T12:56:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 1 -- 172.21.15.26:0/3569950949 >> v1:172.21.15.104:6800/899887506 conn(0x7f3db80415f0 legacy=0x7f3db8043ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 10 monclient: shutdown 2024-08-19T12:56:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 1 -- 172.21.15.26:0/3569950949 >> v1:172.21.15.104:6789/0 conn(0x7f3dcc0a57a0 legacy=0x7f3dcc12e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd178f640 1 -- 172.21.15.26:0/3569950949 reap_dead start 2024-08-19T12:56:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd178f640 1 -- 172.21.15.26:0/3569950949 reap_dead start 2024-08-19T12:56:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 1 -- 172.21.15.26:0/3569950949 shutdown_connections 2024-08-19T12:56:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 1 -- 172.21.15.26:0/3569950949 >> 172.21.15.26:0/3569950949 conn(0x7f3dcc0abff0 msgr2=0x7f3dcc0a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:34.940 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 1 -- 172.21.15.26:0/3569950949 shutdown_connections 2024-08-19T12:56:34.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 1 -- 172.21.15.26:0/3569950949 wait complete. 2024-08-19T12:56:34.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:34.891+0000 7f3dd31f2640 1 librados: shutdown 2024-08-19T12:56:34.941 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T12:56:34.941 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon f 2024-08-19T12:56:34.941 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-08-19T12:56:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.040+0000 7fd5822c7640 10 monclient: get_monmap_and_config 2024-08-19T12:56:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.040+0000 7fd5822c7640 10 monclient: build_initial_monmap 2024-08-19T12:56:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.040+0000 7fd5822c7640 10 monclient: monmap: 2024-08-19T12:56:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:35.041524+0000 2024-08-19T12:56:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:35.041524+0000 2024-08-19T12:56:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.043 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 1 Processor -- start 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 1 -- start start 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 10 monclient: init 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 10 monclient: _add_conns ranks=[7,8,1,0,4,3,5,6,2] 2024-08-19T12:56:35.044 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 10 monclient(hunting): picked mon.noname-i con 0x7fd57c073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 10 monclient(hunting): picked mon.noname-e con 0x7fd57c0a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 10 monclient(hunting): picked mon.noname-f con 0x7fd57c13f2e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd57c09dae0 con 0x7fd57c13f2e0 2024-08-19T12:56:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd57c136910 con 0x7fd57c073470 2024-08-19T12:56:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd57c09dd50 con 0x7fd57c0a57a0 2024-08-19T12:56:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.041+0000 7fd5822c7640 10 monclient(hunting): authenticate will time out at 1789.128977s 2024-08-19T12:56:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.042+0000 7fd57b7fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fd57c0a57a0 0x7fd57c0a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40162/0 (socket says 172.21.15.26:40162) 2024-08-19T12:56:35.045 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.042+0000 7fd57b7fe640 1 -- 172.21.15.26:0/3941534106 learned_addr learned my addr 172.21.15.26:0/3941534106 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57b7fe640 1 -- 172.21.15.26:0/3941534106 >> v1:172.21.15.26:6793/0 conn(0x7fd57c0a57a0 legacy=0x7fd57c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd580864640 1 -- 172.21.15.26:0/3941534106 >> v1:172.21.15.104:6789/0 conn(0x7fd57c13f2e0 legacy=0x7fd57c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57bfff640 1 -- 172.21.15.26:0/3941534106 >> v1:172.21.15.104:6792/0 conn(0x7fd57c073470 legacy=0x7fd57c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3612023182 0 0) 0x7fd57c09dd50 con 0x7fd57c0a57a0 2024-08-19T12:56:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient(hunting): my global_id is 184688 2024-08-19T12:56:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd564003520 con 0x7fd57c0a57a0 2024-08-19T12:56:35.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3929991143 0 0) 0x7fd57c09dae0 con 0x7fd57c13f2e0 2024-08-19T12:56:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient(hunting): my global_id is 204292 2024-08-19T12:56:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd57c09dd50 con 0x7fd57c13f2e0 2024-08-19T12:56:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3302864902 0 0) 0x7fd57c136910 con 0x7fd57c073470 2024-08-19T12:56:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient(hunting): my global_id is 136474 2024-08-19T12:56:35.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd57c09dae0 con 0x7fd57c073470 2024-08-19T12:56:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 907934379 0 0) 0x7fd564003520 con 0x7fd57c0a57a0 2024-08-19T12:56:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 >> v1:172.21.15.104:6792/0 conn(0x7fd57c073470 legacy=0x7fd57c136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 >> v1:172.21.15.104:6789/0 conn(0x7fd57c13f2e0 legacy=0x7fd57c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient: _finish_hunting 0 2024-08-19T12:56:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 1 monclient: found mon.noname-e 2024-08-19T12:56:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:56:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd57c09e030 con 0x7fd57c0a57a0 2024-08-19T12:56:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient: _finish_auth 0 2024-08-19T12:56:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient: _check_auth_tickets 2024-08-19T12:56:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:35.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd570001ae0 con 0x7fd57c0a57a0 2024-08-19T12:56:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd5822c7640 5 monclient: authenticate success, global_id 184688 2024-08-19T12:56:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd5822c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:56:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.044+0000 7fd57affd640 10 monclient: dump: 2024-08-19T12:56:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.056 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd5822c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd5700048a0 con 0x7fd57c0a57a0 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd57affd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:35.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd57affd640 1 -- 172.21.15.26:0/3941534106 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd570004e10 con 0x7fd57c0a57a0 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd5822c7640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd5822c7640 4 set_mon_vals no callback set 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd57affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd57affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd57affd640 10 monclient: dump: 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.058 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.059 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd5822c7640 10 monclient: shutdown 2024-08-19T12:56:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd5822c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:35.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd5822c7640 1 -- 172.21.15.26:0/3941534106 >> v1:172.21.15.26:6793/0 conn(0x7fd57c0a57a0 legacy=0x7fd57c0a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd5822c7640 1 -- 172.21.15.26:0/3941534106 shutdown_connections 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd5822c7640 1 -- 172.21.15.26:0/3941534106 >> 172.21.15.26:0/3941534106 conn(0x7fd57c0abff0 msgr2=0x7fd57c0ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd5822c7640 1 -- 172.21.15.26:0/3941534106 shutdown_connections 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.045+0000 7fd5822c7640 1 -- 172.21.15.26:0/3941534106 wait complete. 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.046+0000 7fd5822c7640 10 monclient: build_initial_monmap 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.046+0000 7fd5822c7640 10 monclient: monmap: 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:35.060 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.046+0000 7fd5822c7640 1 librados: starting msgr at 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.046+0000 7fd5822c7640 1 librados: starting objecter 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.046+0000 7fd5822c7640 1 Processor -- start 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.046+0000 7fd5822c7640 1 -- start start 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.046+0000 7fd5822c7640 1 librados: setting wanted keys 2024-08-19T12:56:35.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.046+0000 7fd5822c7640 1 librados: calling monclient init 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.046+0000 7fd5822c7640 10 monclient: init 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd5822c7640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd5822c7640 10 monclient: _add_conns ranks=[7,4,6,5,2,0,3,8,1] 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd5822c7640 10 monclient(hunting): picked mon.noname-i con 0x7fd57c073470 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd5822c7640 10 monclient(hunting): picked mon.noname-c con 0x7fd57c0a57a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd5822c7640 10 monclient(hunting): picked mon.noname-d con 0x7fd57c13f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd5822c7640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd57c131d10 con 0x7fd57c0a57a0 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd5822c7640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd57c1cef20 con 0x7fd57c13f2e0 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd5822c7640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd57c13f0b0 con 0x7fd57c073470 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd5822c7640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd5822c7640 10 monclient(hunting): authenticate will time out at 1789.134224s 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd580864640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fd57c13f2e0 legacy=0x7fd57c1d6360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd57b7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fd57c0a57a0 0x7fd57c1ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:40480/0 (socket says 172.21.15.26:40480) 2024-08-19T12:56:35.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.047+0000 7fd57b7fe640 1 -- 172.21.15.26:0/3806978301 learned_addr learned my addr 172.21.15.26:0/3806978301 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd57b7fe640 1 -- 172.21.15.26:0/3806978301 >> v1:172.21.15.26:6791/0 conn(0x7fd57c0a57a0 legacy=0x7fd57c1ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 client.?.objecter ms_handle_connect 0x7fd57c0a57a0 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 107345406 0 0) 0x7fd57c131d10 con 0x7fd57c0a57a0 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd57bfff640 1 -- 172.21.15.26:0/3806978301 >> v1:172.21.15.104:6792/0 conn(0x7fd57c073470 legacy=0x7fd57c1ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 monclient(hunting): my global_id is 184819 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd568004eb0 con 0x7fd57c0a57a0 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 client.?.objecter ms_handle_connect 0x7fd57c073470 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 776232286 0 0) 0x7fd57c13f0b0 con 0x7fd57c073470 2024-08-19T12:56:35.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 monclient(hunting): my global_id is 136483 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd57c131d10 con 0x7fd57c073470 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1997379995 0 0) 0x7fd568004eb0 con 0x7fd57c0a57a0 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd57c13f0b0 con 0x7fd57c0a57a0 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd570004d90 con 0x7fd57c0a57a0 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 monclient(hunting): dump: 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.064 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.049+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1731870632 0 0) 0x7fd57c131d10 con 0x7fd57c073470 2024-08-19T12:56:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd568004eb0 con 0x7fd57c073470 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd56c003610 con 0x7fd57c073470 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 10 monclient(hunting): dump: 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.066 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2906415912 0 0) 0x7fd57c13f0b0 con 0x7fd57c0a57a0 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 >> v1:172.21.15.104:6792/0 conn(0x7fd57c073470 legacy=0x7fd57c1ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 >> v1:172.21.15.26:6792/0 conn(0x7fd57c13f2e0 legacy=0x7fd57c1d6360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 1 monclient: found mon.c 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:35.067 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd57c12f9b0 con 0x7fd57c0a57a0 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 10 monclient: _finish_auth 0 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd5822c7640 5 monclient: authenticate success, global_id 184819 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd5822c7640 10 monclient: _renew_subs 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd5822c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd5822c7640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd57c131a30 con 0x7fd57c0a57a0 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd5822c7640 10 client.184819.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd5822c7640 10 monclient: _renew_subs 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd5822c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd5822c7640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fd57c1324a0 con 0x7fd57c0a57a0 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd5822c7640 1 librados: init done 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd570001d30 con 0x7fd57c0a57a0 2024-08-19T12:56:35.068 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd5700048a0 con 0x7fd57c0a57a0 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd57a7fc640 4 set_mon_vals no callback set 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 10 monclient: dump: 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.069 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd5822c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd5822c7640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd55c0065c0 con 0x7fd55c002860 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.050+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd570011a00 con 0x7fd57c0a57a0 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.051+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(659..659 src has 350..659) ==== 5997+0+0 (unknown 3616967900 0 0) 0x7fd57004e5a0 con 0x7fd57c0a57a0 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.051+0000 7fd578ff9640 10 client.184819.objecter ms_dispatch 0x7fd57c000b90 osd_map(659..659 src has 350..659) 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.051+0000 7fd578ff9640 3 client.184819.objecter handle_osd_map got epochs [659,659] > 0 2024-08-19T12:56:35.070 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.051+0000 7fd578ff9640 3 client.184819.objecter handle_osd_map decoding full epoch 659 2024-08-19T12:56:35.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.051+0000 7fd578ff9640 20 client.184819.objecter dump_active .. 0 homeless 2024-08-19T12:56:35.071 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.051+0000 7fd580864640 10 monclient: get_auth_request con 0x7fd5680415f0 auth_method 0 2024-08-19T12:56:35.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 10 client.184819.objecter ms_handle_connect 0x7fd55c002860 2024-08-19T12:56:35.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd57bfff640 1 -- 172.21.15.26:0/3806978301 >> v1:172.21.15.104:6789/0 conn(0x7fd55c002860 legacy=0x7fd55c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 10 client.184819.objecter resend_mon_ops 2024-08-19T12:56:35.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2399862351 0 0) 0x7fd55c0065c0 con 0x7fd55c002860 2024-08-19T12:56:35.074 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 10 monclient(hunting): my global_id is 204301 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd57c13f0b0 con 0x7fd55c002860 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 10 client.184819.objecter ms_handle_connect 0x7fd5680415f0 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2372296347 0 0) 0x7fd57c13f0b0 con 0x7fd55c002860 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd55c0065c0 con 0x7fd55c002860 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd56c003de0 con 0x7fd55c002860 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.053+0000 7fd578ff9640 10 monclient: dump: 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.075 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.076 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.054+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2777817972 0 0) 0x7fd55c0065c0 con 0x7fd55c002860 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.054+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.104:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd55c00a160 con 0x7fd55c002860 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.054+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd56c0040f0 con 0x7fd55c002860 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.054+0000 7fd578ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.054+0000 7fd578ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.054+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 >> v1:172.21.15.104:6789/0 conn(0x7fd55c002860 legacy=0x7fd55c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.054+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 >> v1:172.21.15.104:6789/0 conn(0x7fd55c002860 legacy=0x7fd55c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.060+0000 7fd5822c7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.060+0000 7fd5822c7640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd55c009e50 con 0x7fd55c009110 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 10 client.184819.objecter ms_handle_connect 0x7fd55c009110 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 10 client.184819.objecter resend_mon_ops 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd57bfff640 1 -- 172.21.15.26:0/3806978301 >> v1:172.21.15.104:6789/0 conn(0x7fd55c009110 legacy=0x7fd55c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 779804764 0 0) 0x7fd55c009e50 con 0x7fd55c009110 2024-08-19T12:56:35.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 10 monclient(hunting): my global_id is 204310 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd55c0065c0 con 0x7fd55c009110 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2466541340 0 0) 0x7fd55c0065c0 con 0x7fd55c009110 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd55c009e50 con 0x7fd55c009110 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd56c005b00 con 0x7fd55c009110 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 10 monclient: dump: 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2241395975 0 0) 0x7fd55c009e50 con 0x7fd55c009110 2024-08-19T12:56:35.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.063+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 --> v1:172.21.15.104:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd55c00c3a0 con 0x7fd55c009110 2024-08-19T12:56:35.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 <== mon.1 v1:172.21.15.104:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7513 (unknown 0 0 3343729235) 0x7fd56c007d70 con 0x7fd55c009110 2024-08-19T12:56:35.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd578ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:35.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd578ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:35.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 >> v1:172.21.15.104:6789/0 conn(0x7fd55c009110 legacy=0x7fd55c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd578ff9640 1 -- 172.21.15.26:0/3806978301 >> v1:172.21.15.104:6789/0 conn(0x7fd55c009110 legacy=0x7fd55c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:35.080 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:35.080 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:35.080 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:35.080 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "leader", 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 130, 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 47538, 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:35.081 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:35.082 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:35.082 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:35.082 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:35.082 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:35.082 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:35.082 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:35.082 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:35.082 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:35.082 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:35.082 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:35.082 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:35.083 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.083 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:35.083 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:35.083 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:35.083 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:35.083 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:35.083 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:35.083 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:35.083 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:35.083 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:35.083 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:35.083 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:35.084 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:35.084 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:35.084 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.084 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:35.084 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:35.084 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:35.084 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:35.084 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:35.084 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:35.085 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:35.086 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:35.086 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:35.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:35.086 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:35.086 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:35.086 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:35.086 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:35.086 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:35.086 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:35.086 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.086 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:35.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:35.089 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:35.091 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:35.092 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.092 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.092 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.092 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:35.092 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:35.092 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:35.093 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:35.096 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:35.097 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:35.099 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:35.100 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:35.100 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.100 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.100 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:35.100 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:35.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 10 librados: watch_flush enter 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 10 librados: watch_flush exit 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 20 client.184819.objecter shutdown clearing up homeless session... 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 10 client.184819.objecter successfully canceled tick 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 1 -- 172.21.15.26:0/3806978301 >> v1:172.21.15.104:6800/899887506 conn(0x7fd5680415f0 legacy=0x7fd568043ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 10 monclient: shutdown 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 1 -- 172.21.15.26:0/3806978301 >> v1:172.21.15.26:6791/0 conn(0x7fd57c0a57a0 legacy=0x7fd57c1ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd580864640 1 -- 172.21.15.26:0/3806978301 reap_dead start 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd580864640 1 -- 172.21.15.26:0/3806978301 reap_dead start 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 1 -- 172.21.15.26:0/3806978301 shutdown_connections 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 1 -- 172.21.15.26:0/3806978301 >> 172.21.15.26:0/3806978301 conn(0x7fd57c0abff0 msgr2=0x7fd57c076880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 1 -- 172.21.15.26:0/3806978301 shutdown_connections 2024-08-19T12:56:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 1 -- 172.21.15.26:0/3806978301 wait complete. 2024-08-19T12:56:35.103 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.064+0000 7fd5822c7640 1 librados: shutdown 2024-08-19T12:56:35.103 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T12:56:35.103 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon h 2024-08-19T12:56:35.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-08-19T12:56:35.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.202+0000 7fbc96b7b640 10 monclient: get_monmap_and_config 2024-08-19T12:56:35.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.202+0000 7fbc96b7b640 10 monclient: build_initial_monmap 2024-08-19T12:56:35.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.202+0000 7fbc96b7b640 10 monclient: monmap: 2024-08-19T12:56:35.204 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:35.204 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.204 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:35.203783+0000 2024-08-19T12:56:35.204 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:35.203783+0000 2024-08-19T12:56:35.204 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:35.204 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.204 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:35.204 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:35.204 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:35.204 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:35.204 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:35.204 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:35.205 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:35.205 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:35.205 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:35.205 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.203+0000 7fbc96b7b640 1 Processor -- start 2024-08-19T12:56:35.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.203+0000 7fbc96b7b640 1 -- start start 2024-08-19T12:56:35.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.203+0000 7fbc96b7b640 10 monclient: init 2024-08-19T12:56:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc96b7b640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc96b7b640 10 monclient: _add_conns ranks=[2,6,1,0,8,5,3,4,7] 2024-08-19T12:56:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc96b7b640 10 monclient(hunting): picked mon.noname-b con 0x7fbc9006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:56:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc96b7b640 10 monclient(hunting): picked mon.noname-d con 0x7fbc900a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc96b7b640 10 monclient(hunting): picked mon.noname-f con 0x7fbc900a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc96b7b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc90136790 con 0x7fbc900a5d50 2024-08-19T12:56:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc96b7b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc9009ddf0 con 0x7fbc9006e7b0 2024-08-19T12:56:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc96b7b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc9009e0d0 con 0x7fbc900a5920 2024-08-19T12:56:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc96b7b640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc96b7b640 10 monclient(hunting): authenticate will time out at 1789.291261s 2024-08-19T12:56:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc8ffff640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fbc900a5920 legacy=0x7fbc9013e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:56:35.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc94917640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbc9006e7b0 0x7fbc90136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:60328/0 (socket says 172.21.15.26:60328) 2024-08-19T12:56:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.204+0000 7fbc94917640 1 -- 172.21.15.26:0/695605644 learned_addr learned my addr 172.21.15.26:0/695605644 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:35.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc94917640 1 -- 172.21.15.26:0/695605644 >> v1:172.21.15.26:6790/0 conn(0x7fbc9006e7b0 legacy=0x7fbc90136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc95118640 1 -- 172.21.15.26:0/695605644 >> v1:172.21.15.104:6789/0 conn(0x7fbc900a5d50 legacy=0x7fbc90141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc8f7fe640 1 -- 172.21.15.26:0/695605644 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 610942527 0 0) 0x7fbc880017d0 con 0x7fbc9006e7b0 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc8f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc8f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc8f7fe640 10 monclient(hunting): my global_id is 27488 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc8f7fe640 1 -- 172.21.15.26:0/695605644 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc7c003520 con 0x7fbc9006e7b0 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc8f7fe640 1 -- 172.21.15.26:0/695605644 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4153698190 0 0) 0x7fbc90136790 con 0x7fbc900a5d50 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc8f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc8f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc8f7fe640 10 monclient(hunting): my global_id is 204319 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.206+0000 7fbc8f7fe640 1 -- 172.21.15.26:0/695605644 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc880017d0 con 0x7fbc900a5d50 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 1 -- 172.21.15.26:0/695605644 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 640102428 0 0) 0x7fbc7c003520 con 0x7fbc9006e7b0 2024-08-19T12:56:35.218 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 1 -- 172.21.15.26:0/695605644 >> v1:172.21.15.26:6792/0 conn(0x7fbc900a5920 legacy=0x7fbc9013e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 1 -- 172.21.15.26:0/695605644 >> v1:172.21.15.104:6789/0 conn(0x7fbc900a5d50 legacy=0x7fbc90141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 1 monclient: found mon.noname-b 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 1 -- 172.21.15.26:0/695605644 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9009e3b0 con 0x7fbc9006e7b0 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 1 -- 172.21.15.26:0/695605644 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbc88002030 con 0x7fbc9006e7b0 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc96b7b640 5 monclient: authenticate success, global_id 27488 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc96b7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 10 monclient: dump: 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.220 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc96b7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 1 -- 172.21.15.26:0/695605644 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc88004cd0 con 0x7fbc9006e7b0 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 1 -- 172.21.15.26:0/695605644 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbc88005240 con 0x7fbc9006e7b0 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc96b7b640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc96b7b640 4 set_mon_vals no callback set 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:56:35.221 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc8f7fe640 10 monclient: dump: 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc96b7b640 10 monclient: shutdown 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc96b7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc96b7b640 1 -- 172.21.15.26:0/695605644 >> v1:172.21.15.26:6790/0 conn(0x7fbc9006e7b0 legacy=0x7fbc90136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc96b7b640 1 -- 172.21.15.26:0/695605644 shutdown_connections 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc96b7b640 1 -- 172.21.15.26:0/695605644 >> 172.21.15.26:0/695605644 conn(0x7fbc900a8740 msgr2=0x7fbc900a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc96b7b640 1 -- 172.21.15.26:0/695605644 shutdown_connections 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.207+0000 7fbc96b7b640 1 -- 172.21.15.26:0/695605644 wait complete. 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.208+0000 7fbc96b7b640 10 monclient: build_initial_monmap 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.208+0000 7fbc96b7b640 10 monclient: monmap: 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:35.223 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:35.224 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:35.225 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.208+0000 7fbc96b7b640 1 librados: starting msgr at 2024-08-19T12:56:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.208+0000 7fbc96b7b640 1 librados: starting objecter 2024-08-19T12:56:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.208+0000 7fbc96b7b640 1 Processor -- start 2024-08-19T12:56:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.208+0000 7fbc96b7b640 1 -- start start 2024-08-19T12:56:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.208+0000 7fbc96b7b640 1 librados: setting wanted keys 2024-08-19T12:56:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.208+0000 7fbc96b7b640 1 librados: calling monclient init 2024-08-19T12:56:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.208+0000 7fbc96b7b640 10 monclient: init 2024-08-19T12:56:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc96b7b640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc96b7b640 10 monclient: _add_conns ranks=[8,6,1,4,7,0,3,2,5] 2024-08-19T12:56:35.225 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc96b7b640 10 monclient(hunting): picked mon.noname-e con 0x7fbc9006e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:56:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc96b7b640 10 monclient(hunting): picked mon.noname-d con 0x7fbc900a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:56:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc96b7b640 10 monclient(hunting): picked mon.noname-f con 0x7fbc900a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc96b7b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc90142060 con 0x7fbc900a5d50 2024-08-19T12:56:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc96b7b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc90134770 con 0x7fbc900a5920 2024-08-19T12:56:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc8ffff640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fbc900a5920 legacy=0x7fbc90137cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:56:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc96b7b640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc90138f70 con 0x7fbc9006e7b0 2024-08-19T12:56:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc96b7b640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc96b7b640 10 monclient(hunting): authenticate will time out at 1789.296300s 2024-08-19T12:56:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc94917640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fbc9006e7b0 0x7fbc901375a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:40174/0 (socket says 172.21.15.26:40174) 2024-08-19T12:56:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.209+0000 7fbc94917640 1 -- 172.21.15.26:0/3196502823 learned_addr learned my addr 172.21.15.26:0/3196502823 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc94917640 1 -- 172.21.15.26:0/3196502823 >> v1:172.21.15.26:6793/0 conn(0x7fbc9006e7b0 legacy=0x7fbc901375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.226 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 client.?.objecter ms_handle_connect 0x7fbc9006e7b0 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc95118640 1 -- 172.21.15.26:0/3196502823 >> v1:172.21.15.104:6789/0 conn(0x7fbc900a5d50 legacy=0x7fbc901d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4048222596 0 0) 0x7fbc90138f70 con 0x7fbc9006e7b0 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 monclient(hunting): my global_id is 184697 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc70004eb0 con 0x7fbc9006e7b0 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 client.?.objecter ms_handle_connect 0x7fbc900a5d50 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 11900414 0 0) 0x7fbc90142060 con 0x7fbc900a5d50 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.227 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 monclient(hunting): my global_id is 204328 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc90138f70 con 0x7fbc900a5d50 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1043947253 0 0) 0x7fbc70004eb0 con 0x7fbc9006e7b0 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc90142060 con 0x7fbc9006e7b0 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbc88004e80 con 0x7fbc9006e7b0 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 monclient(hunting): dump: 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.228 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.211+0000 7fbc8d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:35.229 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4059162463 0 0) 0x7fbc90138f70 con 0x7fbc900a5d50 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc70004eb0 con 0x7fbc900a5d50 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbc78003610 con 0x7fbc900a5d50 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 10 monclient(hunting): dump: 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.230 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4203620531 0 0) 0x7fbc90142060 con 0x7fbc9006e7b0 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 >> v1:172.21.15.26:6792/0 conn(0x7fbc900a5920 legacy=0x7fbc90137cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 >> v1:172.21.15.104:6789/0 conn(0x7fbc900a5d50 legacy=0x7fbc901d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 10 monclient: _finish_hunting 0 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 1 monclient: found mon.e 2024-08-19T12:56:35.231 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc901383c0 con 0x7fbc9006e7b0 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 10 monclient: _finish_auth 0 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 10 monclient: _check_auth_tickets 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc96b7b640 5 monclient: authenticate success, global_id 184697 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc96b7b640 10 monclient: _renew_subs 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc96b7b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc96b7b640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc9013aa30 con 0x7fbc9006e7b0 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc96b7b640 10 client.184697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc96b7b640 10 monclient: _renew_subs 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc96b7b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc96b7b640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc901429b0 con 0x7fbc9006e7b0 2024-08-19T12:56:35.232 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc96b7b640 1 librados: init done 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc88005580 con 0x7fbc9006e7b0 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbc88005a70 con 0x7fbc9006e7b0 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8effd640 4 set_mon_vals no callback set 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.212+0000 7fbc8d7fa640 10 monclient: dump: 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.233 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.213+0000 7fbc96b7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.213+0000 7fbc96b7b640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbc740065c0 con 0x7fbc74002860 2024-08-19T12:56:35.234 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.213+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fbc88011ed0 con 0x7fbc9006e7b0 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.213+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(659..659 src has 350..659) ==== 5997+0+0 (unknown 3616967900 0 0) 0x7fbc8804e870 con 0x7fbc9006e7b0 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.213+0000 7fbc8d7fa640 10 client.184697.objecter ms_dispatch 0x7fbc90000b90 osd_map(659..659 src has 350..659) 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.213+0000 7fbc8d7fa640 3 client.184697.objecter handle_osd_map got epochs [659,659] > 0 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.213+0000 7fbc8d7fa640 3 client.184697.objecter handle_osd_map decoding full epoch 659 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.213+0000 7fbc8d7fa640 20 client.184697.objecter dump_active .. 0 homeless 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.213+0000 7fbc95118640 10 monclient: get_auth_request con 0x7fbc700414e0 auth_method 0 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8ffff640 1 -- 172.21.15.26:0/3196502823 >> v1:172.21.15.104:6791/0 conn(0x7fbc74002860 legacy=0x7fbc74004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 10 client.184697.objecter ms_handle_connect 0x7fbc74002860 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 10 client.184697.objecter resend_mon_ops 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3779547424 0 0) 0x7fbc740065c0 con 0x7fbc74002860 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.235 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 10 monclient(hunting): my global_id is 184712 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc90142060 con 0x7fbc74002860 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 746615120 0 0) 0x7fbc90142060 con 0x7fbc74002860 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc740065c0 con 0x7fbc74002860 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 10 client.184697.objecter ms_handle_connect 0x7fbc700414e0 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbc80004440 con 0x7fbc74002860 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.215+0000 7fbc8d7fa640 10 monclient: dump: 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.236 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.216+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3577011358 0 0) 0x7fbc740065c0 con 0x7fbc74002860 2024-08-19T12:56:35.237 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.216+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.104:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc7400a160 con 0x7fbc74002860 2024-08-19T12:56:35.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.216+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbc800067f0 con 0x7fbc74002860 2024-08-19T12:56:35.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.216+0000 7fbc8d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:35.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.216+0000 7fbc8d7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:35.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.216+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 >> v1:172.21.15.104:6791/0 conn(0x7fbc74002860 legacy=0x7fbc74004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.216+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 >> v1:172.21.15.104:6791/0 conn(0x7fbc74002860 legacy=0x7fbc74004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:35.239 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.222+0000 7fbc96b7b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.222+0000 7fbc96b7b640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbc74009e70 con 0x7fbc74009130 2024-08-19T12:56:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8ffff640 1 -- 172.21.15.26:0/3196502823 >> v1:172.21.15.104:6791/0 conn(0x7fbc74009130 legacy=0x7fbc740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 10 client.184697.objecter ms_handle_connect 0x7fbc74009130 2024-08-19T12:56:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 10 client.184697.objecter resend_mon_ops 2024-08-19T12:56:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2562418165 0 0) 0x7fbc74009e70 con 0x7fbc74009130 2024-08-19T12:56:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 10 monclient(hunting): my global_id is 184721 2024-08-19T12:56:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc740065c0 con 0x7fbc74009130 2024-08-19T12:56:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1022692287 0 0) 0x7fbc740065c0 con 0x7fbc74009130 2024-08-19T12:56:35.240 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc74009e70 con 0x7fbc74009130 2024-08-19T12:56:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbc80004f30 con 0x7fbc74009130 2024-08-19T12:56:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 10 monclient: dump: 2024-08-19T12:56:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.241 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 723598857 0 0) 0x7fbc74009e70 con 0x7fbc74009130 2024-08-19T12:56:35.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.225+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 --> v1:172.21.15.104:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbc7400c410 con 0x7fbc74009130 2024-08-19T12:56:35.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 <== mon.5 v1:172.21.15.104:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7846 (unknown 0 0 2635831822) 0x7fbc800063c0 con 0x7fbc74009130 2024-08-19T12:56:35.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc8d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:35.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc8d7fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:35.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 >> v1:172.21.15.104:6791/0 conn(0x7fbc74009130 legacy=0x7fbc740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc8d7fa640 1 -- 172.21.15.26:0/3196502823 >> v1:172.21.15.104:6791/0 conn(0x7fbc74009130 legacy=0x7fbc740095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:35.243 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:35.243 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 130, 2024-08-19T12:56:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 320694, 2024-08-19T12:56:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:35.244 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:56:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:35.245 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:35.246 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:35.247 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:35.248 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:35.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:35.250 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:35.254 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.257 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:35.258 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:35.262 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 2 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": [ 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:35.263 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.264 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:35.264 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:35.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc96b7b640 10 librados: watch_flush enter 2024-08-19T12:56:35.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc96b7b640 10 librados: watch_flush exit 2024-08-19T12:56:35.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc96b7b640 20 client.184697.objecter shutdown clearing up homeless session... 2024-08-19T12:56:35.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc96b7b640 10 client.184697.objecter successfully canceled tick 2024-08-19T12:56:35.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc96b7b640 1 -- 172.21.15.26:0/3196502823 >> v1:172.21.15.104:6800/899887506 conn(0x7fbc700414e0 legacy=0x7fbc700439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc96b7b640 10 monclient: shutdown 2024-08-19T12:56:35.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc96b7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:35.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc96b7b640 1 -- 172.21.15.26:0/3196502823 >> v1:172.21.15.26:6793/0 conn(0x7fbc9006e7b0 legacy=0x7fbc901375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc95118640 1 -- 172.21.15.26:0/3196502823 reap_dead start 2024-08-19T12:56:35.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc95118640 1 -- 172.21.15.26:0/3196502823 reap_dead start 2024-08-19T12:56:35.266 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc96b7b640 1 -- 172.21.15.26:0/3196502823 shutdown_connections 2024-08-19T12:56:35.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.226+0000 7fbc96b7b640 1 -- 172.21.15.26:0/3196502823 >> 172.21.15.26:0/3196502823 conn(0x7fbc900a8740 msgr2=0x7fbc900ab060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:35.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.227+0000 7fbc96b7b640 1 -- 172.21.15.26:0/3196502823 shutdown_connections 2024-08-19T12:56:35.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.227+0000 7fbc96b7b640 1 -- 172.21.15.26:0/3196502823 wait complete. 2024-08-19T12:56:35.267 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.227+0000 7fbc96b7b640 1 librados: shutdown 2024-08-19T12:56:35.267 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T12:56:35.267 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon i 2024-08-19T12:56:35.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-08-19T12:56:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.366+0000 7f499eb5a640 10 monclient: get_monmap_and_config 2024-08-19T12:56:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.366+0000 7f499eb5a640 10 monclient: build_initial_monmap 2024-08-19T12:56:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.366+0000 7f499eb5a640 10 monclient: monmap: 2024-08-19T12:56:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:35.367 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:56:35.367648+0000 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:56:35.367648+0000 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:35.368 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:35.369 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.367+0000 7f499eb5a640 1 Processor -- start 2024-08-19T12:56:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.367+0000 7f499eb5a640 1 -- start start 2024-08-19T12:56:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.367+0000 7f499eb5a640 10 monclient: init 2024-08-19T12:56:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f499eb5a640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f499eb5a640 10 monclient: _add_conns ranks=[7,1,5,8,6,2,0,3,4] 2024-08-19T12:56:35.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f499eb5a640 10 monclient(hunting): picked mon.noname-i con 0x7f499806e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f499eb5a640 10 monclient(hunting): picked mon.noname-f con 0x7f49980a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:56:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f499eb5a640 10 monclient(hunting): picked mon.noname-h con 0x7f49980a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f499eb5a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4998136790 con 0x7f49980a5920 2024-08-19T12:56:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f499eb5a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499809ddf0 con 0x7f49980a5d50 2024-08-19T12:56:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f499eb5a640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499809e0d0 con 0x7f499806e7b0 2024-08-19T12:56:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f499eb5a640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f499eb5a640 10 monclient(hunting): authenticate will time out at 1789.455189s 2024-08-19T12:56:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f4997fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f49980a5920 0x7f499813e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:49838/0 (socket says 172.21.15.26:49838) 2024-08-19T12:56:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f499c8f6640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f499806e7b0 0x7f4998136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:35274/0 (socket says 172.21.15.26:35274) 2024-08-19T12:56:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f4997fff640 1 -- 172.21.15.26:0/1635357335 learned_addr learned my addr 172.21.15.26:0/1635357335 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:35.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.368+0000 7f499d0f7640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f49980a5d50 0x7f4998141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:48526/0 (socket says 172.21.15.26:48526) 2024-08-19T12:56:35.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.370+0000 7f4997fff640 1 -- 172.21.15.26:0/1635357335 >> v1:172.21.15.104:6789/0 conn(0x7f49980a5920 legacy=0x7f499813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.370+0000 7f499c8f6640 1 -- 172.21.15.26:0/1635357335 >> v1:172.21.15.104:6792/0 conn(0x7f499806e7b0 legacy=0x7f4998136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.370+0000 7f499d0f7640 1 -- 172.21.15.26:0/1635357335 >> v1:172.21.15.104:6791/0 conn(0x7f49980a5d50 legacy=0x7f4998141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1528327159 0 0) 0x7f499809e0d0 con 0x7f499806e7b0 2024-08-19T12:56:35.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient(hunting): my global_id is 136492 2024-08-19T12:56:35.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4988003520 con 0x7f499806e7b0 2024-08-19T12:56:35.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1144830030 0 0) 0x7f4990001700 con 0x7f49980a5d50 2024-08-19T12:56:35.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient(hunting): my global_id is 184730 2024-08-19T12:56:35.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f499809e0d0 con 0x7f49980a5d50 2024-08-19T12:56:35.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3447432148 0 0) 0x7f4998136790 con 0x7f49980a5920 2024-08-19T12:56:35.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient(hunting): my global_id is 204337 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4990001700 con 0x7f49980a5920 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2715391138 0 0) 0x7f4988003520 con 0x7f499806e7b0 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 >> v1:172.21.15.104:6791/0 conn(0x7f49980a5d50 legacy=0x7f4998141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 >> v1:172.21.15.104:6789/0 conn(0x7f49980a5920 legacy=0x7f499813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient: _finish_hunting 0 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 monclient: found mon.noname-i 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499809e3b0 con 0x7f499806e7b0 2024-08-19T12:56:35.386 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient: _finish_auth 0 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient: _check_auth_tickets 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f499eb5a640 5 monclient: authenticate success, global_id 136492 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f499eb5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f498c001ae0 con 0x7f499806e7b0 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient: dump: 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.387 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f499eb5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f498c0047e0 con 0x7f499806e7b0 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f499eb5a640 10 monclient: get_monmap_and_config success 2024-08-19T12:56:35.388 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f499eb5a640 4 set_mon_vals no callback set 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 1 -- 172.21.15.26:0/1635357335 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f498c004d90 con 0x7f499806e7b0 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.371+0000 7f49977fe640 10 monclient: dump: 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.389 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.372+0000 7f499eb5a640 10 monclient: shutdown 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.372+0000 7f499eb5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.372+0000 7f499eb5a640 1 -- 172.21.15.26:0/1635357335 >> v1:172.21.15.104:6792/0 conn(0x7f499806e7b0 legacy=0x7f4998136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.372+0000 7f499eb5a640 1 -- 172.21.15.26:0/1635357335 shutdown_connections 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.372+0000 7f499eb5a640 1 -- 172.21.15.26:0/1635357335 >> 172.21.15.26:0/1635357335 conn(0x7f49980a8740 msgr2=0x7f49980a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.372+0000 7f499eb5a640 1 -- 172.21.15.26:0/1635357335 shutdown_connections 2024-08-19T12:56:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.372+0000 7f499eb5a640 1 -- 172.21.15.26:0/1635357335 wait complete. 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.372+0000 7f499eb5a640 10 monclient: build_initial_monmap 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.372+0000 7f499eb5a640 10 monclient: monmap: 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:56:35.391 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.372+0000 7f499eb5a640 1 librados: starting msgr at 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.372+0000 7f499eb5a640 1 librados: starting objecter 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 1 Processor -- start 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 1 -- start start 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 1 librados: setting wanted keys 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 1 librados: calling monclient init 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 10 monclient: init 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 10 monclient: _reopen_session rank -1 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 10 monclient: _add_conns ranks=[5,7,0,2,1,3,6,8,4] 2024-08-19T12:56:35.392 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 10 monclient(hunting): picked mon.noname-h con 0x7f499806e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 10 monclient(hunting): picked mon.noname-i con 0x7f49980a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 10 monclient(hunting): picked mon.noname-a con 0x7f49980a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499813aa00 con 0x7f49980a5d50 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49981346c0 con 0x7f499806e7b0 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499d0f7640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f49980a5d50 legacy=0x7f49981d6c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4998139200 con 0x7f49980a5920 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 10 monclient(hunting): _renew_subs 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499eb5a640 10 monclient(hunting): authenticate will time out at 1789.460802s 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499c8f6640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f499806e7b0 0x7f49981ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:48536/0 (socket says 172.21.15.26:48536) 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f499c8f6640 1 -- 172.21.15.26:0/3136684777 learned_addr learned my addr 172.21.15.26:0/3136684777 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.373+0000 7f4997fff640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f49980a5920 0x7f49981d34f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:35288/0 (socket says 172.21.15.26:35288) 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f499c8f6640 1 -- 172.21.15.26:0/3136684777 >> v1:172.21.15.104:6791/0 conn(0x7f499806e7b0 legacy=0x7f49981ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 client.?.objecter ms_handle_connect 0x7f499806e7b0 2024-08-19T12:56:35.393 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 client.?.objecter ms_handle_connect 0x7f49980a5920 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f4997fff640 1 -- 172.21.15.26:0/3136684777 >> v1:172.21.15.104:6792/0 conn(0x7f49980a5920 legacy=0x7f49981d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 991859008 0 0) 0x7f49981346c0 con 0x7f499806e7b0 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): my global_id is 184739 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f497c004eb0 con 0x7f499806e7b0 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 933569084 0 0) 0x7f4998139200 con 0x7f49980a5920 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.394 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): my global_id is 136501 2024-08-19T12:56:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49981346c0 con 0x7f49980a5920 2024-08-19T12:56:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1074825832 0 0) 0x7f497c004eb0 con 0x7f499806e7b0 2024-08-19T12:56:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4998139200 con 0x7f499806e7b0 2024-08-19T12:56:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f498c001a60 con 0x7f499806e7b0 2024-08-19T12:56:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:56:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): dump: 2024-08-19T12:56:35.395 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.396 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3976329151 0 0) 0x7f49981346c0 con 0x7f49980a5920 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f497c004eb0 con 0x7f49980a5920 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4984001cc0 con 0x7f49980a5920 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient(hunting): dump: 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1768293151 0 0) 0x7f4998139200 con 0x7f499806e7b0 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 >> v1:172.21.15.104:6792/0 conn(0x7f49980a5920 legacy=0x7f49981d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 >> v1:172.21.15.26:6789/0 conn(0x7f49980a5d50 legacy=0x7f49981d6c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient: _finish_hunting 0 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 monclient: found mon.h 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499813b560 con 0x7f499806e7b0 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient: _finish_auth 0 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient: _check_auth_tickets 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:56:35.399 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f499eb5a640 5 monclient: authenticate success, global_id 184739 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f499eb5a640 10 monclient: _renew_subs 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f499eb5a640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f499eb5a640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f499813acc0 con 0x7f499806e7b0 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f499eb5a640 10 client.184739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f499eb5a640 10 monclient: _renew_subs 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f499eb5a640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f499eb5a640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f499813bf90 con 0x7f499806e7b0 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f499eb5a640 1 librados: init done 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f498c0047e0 con 0x7f499806e7b0 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.376+0000 7f49957fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.377+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f498c004d30 con 0x7f499806e7b0 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.377+0000 7f49957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.377+0000 7f49967fc640 4 set_mon_vals no callback set 2024-08-19T12:56:35.400 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.377+0000 7f49957fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.377+0000 7f49957fa640 10 monclient: dump: 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.401 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.377+0000 7f499eb5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.377+0000 7f499eb5a640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f49780065c0 con 0x7f4978002860 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.377+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f498c011880 con 0x7f499806e7b0 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.378+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(659..659 src has 350..659) ==== 5997+0+0 (unknown 3616967900 0 0) 0x7f498c04e5a0 con 0x7f499806e7b0 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.378+0000 7f49957fa640 10 client.184739.objecter ms_dispatch 0x7f4998000b90 osd_map(659..659 src has 350..659) 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.378+0000 7f49957fa640 3 client.184739.objecter handle_osd_map got epochs [659,659] > 0 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.378+0000 7f49957fa640 3 client.184739.objecter handle_osd_map decoding full epoch 659 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.378+0000 7f49957fa640 20 client.184739.objecter dump_active .. 0 homeless 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.378+0000 7f499d0f7640 10 monclient: get_auth_request con 0x7f497c041450 auth_method 0 2024-08-19T12:56:35.402 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.379+0000 7f4997fff640 1 -- 172.21.15.26:0/3136684777 >> v1:172.21.15.104:6792/0 conn(0x7f4978002860 legacy=0x7f4978004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.379+0000 7f49957fa640 10 client.184739.objecter ms_handle_connect 0x7f4978002860 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.379+0000 7f49957fa640 10 client.184739.objecter resend_mon_ops 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 831453106 0 0) 0x7f49780065c0 con 0x7f4978002860 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 10 monclient(hunting): my global_id is 136510 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4998139200 con 0x7f4978002860 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 641238832 0 0) 0x7f4998139200 con 0x7f4978002860 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49780065c0 con 0x7f4978002860 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4984004f20 con 0x7f4978002860 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 10 monclient: dump: 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.404 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.405 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3764432809 0 0) 0x7f49780065c0 con 0x7f4978002860 2024-08-19T12:56:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f497800a160 con 0x7f4978002860 2024-08-19T12:56:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 10 client.184739.objecter ms_handle_connect 0x7f497c041450 2024-08-19T12:56:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4984005210 con 0x7f4978002860 2024-08-19T12:56:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:56:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:56:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 >> v1:172.21.15.104:6792/0 conn(0x7f4978002860 legacy=0x7f4978004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.405 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.380+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 >> v1:172.21.15.104:6792/0 conn(0x7f4978002860 legacy=0x7f4978004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.387+0000 7f499eb5a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-08-19T12:56:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.387+0000 7f499eb5a640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4978009e50 con 0x7f4978009110 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.389+0000 7f4997fff640 1 -- 172.21.15.26:0/3136684777 >> v1:172.21.15.104:6792/0 conn(0x7f4978009110 legacy=0x7f49780095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.389+0000 7f49957fa640 10 client.184739.objecter ms_handle_connect 0x7f4978009110 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.389+0000 7f49957fa640 10 client.184739.objecter resend_mon_ops 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.389+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1051778044 0 0) 0x7f4978009e50 con 0x7f4978009110 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.389+0000 7f49957fa640 10 monclient(hunting): _negotiate 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.389+0000 7f49957fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.389+0000 7f49957fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.389+0000 7f49957fa640 10 monclient(hunting): my global_id is 136519 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.389+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49780065c0 con 0x7f4978009110 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2874619466 0 0) 0x7f49780065c0 con 0x7f4978009110 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4978009e50 con 0x7f4978009110 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4984005c10 con 0x7f4978009110 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:56:35.408 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 10 monclient: dump: 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:56:35.409 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:56:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:56:35.410 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:56:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 611294197 0 0) 0x7f4978009e50 con 0x7f4978009110 2024-08-19T12:56:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 --> v1:172.21.15.104:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f497800c3f0 con 0x7f4978009110 2024-08-19T12:56:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 <== mon.7 v1:172.21.15.104:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7679 (unknown 0 0 1260064010) 0x7f4984007e10 con 0x7f4978009110 2024-08-19T12:56:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-08-19T12:56:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:56:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 >> v1:172.21.15.104:6792/0 conn(0x7f4978009110 legacy=0x7f49780095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f49957fa640 1 -- 172.21.15.26:0/3136684777 >> v1:172.21.15.104:6792/0 conn(0x7f4978009110 legacy=0x7f49780095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:56:35.410 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: "state": "peon", 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: "election_epoch": 130, 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: "uptime": 965747, 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum": [ 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-08-19T12:56:35.411 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_age": 1, 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "required_con": "2450521708007866372", 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "required_mon": [ 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:35.412 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_con": "4540701547738038271", 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: "quorum_mon": [ 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:35.413 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: "outside_quorum": [], 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: "extra_probe_peers": [], 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: "sync_provider": [], 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": { 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1, 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "10d02fe0-13d8-499a-99b3-d3a725cba366", 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:35.414 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-08-19T12:37:41.103727Z", 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release": 19, 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "min_mon_release_name": "squid", 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "election_strategy": 1, 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "disallowed_leaders: ": "", 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false, 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "tiebreaker_mon": "", 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_ranks: ": "", 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "features": { 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "persistent": [ 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "kraken", 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "luminous", 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "mimic", 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "osdmap-prune", 2024-08-19T12:56:35.415 INFO:teuthology.orchestra.run.smithi026.stdout: "nautilus", 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "octopus", 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "pacific", 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "elector-pinging", 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "quincy", 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "reef", 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "squid" 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "optional": [] 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "mons": [ 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 0, 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "a", 2024-08-19T12:56:35.416 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789", 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6789/0", 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6789/0", 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.417 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 1, 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "f", 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789", 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.418 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6789/0", 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6789/0", 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 2, 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "b", 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790", 2024-08-19T12:56:35.419 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6790/0", 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6790/0", 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 3, 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "g", 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.420 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790", 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6790/0", 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6790/0", 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 4, 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "c", 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791", 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6791/0", 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6791/0", 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 5, 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "h", 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791", 2024-08-19T12:56:35.423 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6791/0", 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6791/0", 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 6, 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "d", 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792", 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6792/0", 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6792/0", 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 7, 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "i", 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792", 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.104:6792/0", 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.104:6792/0", 2024-08-19T12:56:35.426 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: "rank": 8, 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "e", 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793", 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 0 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.427 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6793/0", 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6793/0", 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: "priority": 0, 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "{}" 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: "feature_map": { 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: "mon": [ 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.428 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": [ 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: "client": [ 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-08-19T12:56:35.429 INFO:teuthology.orchestra.run.smithi026.stdout: "features": "0x3f03cffffffdffff", 2024-08-19T12:56:35.430 INFO:teuthology.orchestra.run.smithi026.stdout: "release": "squid", 2024-08-19T12:56:35.430 INFO:teuthology.orchestra.run.smithi026.stdout: "num": 1 2024-08-19T12:56:35.430 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-08-19T12:56:35.430 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-08-19T12:56:35.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-08-19T12:56:35.430 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": false 2024-08-19T12:56:35.430 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:56:35.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.390+0000 7f499eb5a640 10 librados: watch_flush enter 2024-08-19T12:56:35.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499eb5a640 10 librados: watch_flush exit 2024-08-19T12:56:35.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499eb5a640 20 client.184739.objecter shutdown clearing up homeless session... 2024-08-19T12:56:35.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499eb5a640 10 client.184739.objecter successfully canceled tick 2024-08-19T12:56:35.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499eb5a640 1 -- 172.21.15.26:0/3136684777 >> v1:172.21.15.104:6800/899887506 conn(0x7f497c041450 legacy=0x7f497c043910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499eb5a640 10 monclient: shutdown 2024-08-19T12:56:35.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499eb5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:56:35.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499eb5a640 1 -- 172.21.15.26:0/3136684777 >> v1:172.21.15.104:6791/0 conn(0x7f499806e7b0 legacy=0x7f49981ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:56:35.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499d0f7640 1 -- 172.21.15.26:0/3136684777 reap_dead start 2024-08-19T12:56:35.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499d0f7640 1 -- 172.21.15.26:0/3136684777 reap_dead start 2024-08-19T12:56:35.432 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499eb5a640 1 -- 172.21.15.26:0/3136684777 shutdown_connections 2024-08-19T12:56:35.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499eb5a640 1 -- 172.21.15.26:0/3136684777 >> 172.21.15.26:0/3136684777 conn(0x7f49980a8740 msgr2=0x7f4998135200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:56:35.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499eb5a640 1 -- 172.21.15.26:0/3136684777 shutdown_connections 2024-08-19T12:56:35.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499eb5a640 1 -- 172.21.15.26:0/3136684777 wait complete. 2024-08-19T12:56:35.433 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:56:35.391+0000 7f499eb5a640 1 librados: shutdown 2024-08-19T12:56:35.433 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-08-19T12:56:35.433 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-08-19T12:57:37.427 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7686 ms) 2024-08-19T12:57:37.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-08-19T12:57:37.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13075 ms) 2024-08-19T12:57:37.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-08-19T12:57:37.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9030 ms) 2024-08-19T12:57:37.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-08-19T12:57:37.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (14951 ms) 2024-08-19T12:57:37.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-08-19T12:57:37.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8588 ms) 2024-08-19T12:57:37.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-08-19T12:57:37.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8242 ms) 2024-08-19T12:57:37.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-08-19T12:57:37.428 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: hmm, no HitSet yet 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: ok, hit_set contains 332:602f83fe:::foo:head 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8524 ms) 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072137,0 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: first is 1724072137 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072137,0 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072137,0 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072137,1724072139,1724072140,0 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072137,1724072139,1724072140,0 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072137,1724072139,1724072140,0 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072137,1724072139,1724072140,1724072142,1724072143,0 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072137,1724072139,1724072140,1724072142,1724072143,0 2024-08-19T12:57:37.429 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072137,1724072139,1724072140,1724072142,1724072143,0 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072137,1724072139,1724072140,1724072142,1724072143,1724072145,1724072146,0 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072137,1724072139,1724072140,1724072142,1724072143,1724072145,1724072146,0 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072137,1724072139,1724072140,1724072142,1724072143,1724072145,1724072146,0 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: got ls 1724072140,1724072142,1724072143,1724072145,1724072146,1724072148,1724072149,0 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: first now 1724072140, trimmed 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (26630 ms) 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: foo0 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (41431 ms) 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18018 ms) 2024-08-19T12:57:37.430 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (23794 ms) 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4178 ms) 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4917 ms) 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3002 ms) 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3084 ms) 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (306061 ms total) 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-08-19T12:57:37.431 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1081783 ms total) 2024-08-19T12:57:37.432 INFO:tasks.workunit.client.0.smithi104.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-08-19T12:57:37.437 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.437 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24472 2024-08-19T12:57:37.437 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24472 2024-08-19T12:57:37.437 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.437 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24656 2024-08-19T12:57:37.437 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24656 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25184 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25184 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24987 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24987 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24819 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24819 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24549 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24549 2024-08-19T12:57:37.438 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24904 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24904 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25366 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25366 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25408 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25408 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24575 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24575 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25209 2024-08-19T12:57:37.439 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25209 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24446 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24446 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24512 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24512 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25124 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25124 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24437 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24437 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.440 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24458 2024-08-19T12:57:37.441 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24458 2024-08-19T12:57:37.441 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.441 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25637 2024-08-19T12:57:37.441 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25637 2024-08-19T12:57:37.441 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.441 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24725 2024-08-19T12:57:37.441 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24725 2024-08-19T12:57:37.441 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.442 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25672 2024-08-19T12:57:37.442 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25672 2024-08-19T12:57:37.442 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.442 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24866 2024-08-19T12:57:37.442 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24866 2024-08-19T12:57:37.442 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.442 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25319 2024-08-19T12:57:37.442 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25319 2024-08-19T12:57:37.442 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.442 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25736 2024-08-19T12:57:37.443 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25736 2024-08-19T12:57:37.443 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.443 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24776 2024-08-19T12:57:37.443 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24776 2024-08-19T12:57:37.443 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.443 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25142 2024-08-19T12:57:37.443 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25142 2024-08-19T12:57:37.443 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.443 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25571 2024-08-19T12:57:37.443 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25571 2024-08-19T12:57:37.443 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.443 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25251 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25251 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25663 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25663 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25093 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25093 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24601 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24601 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25708 2024-08-19T12:57:37.444 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25708 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25034 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25034 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24430 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24430 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24488 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24488 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24963 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24963 2024-08-19T12:57:37.445 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=24528 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 24528 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25052 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25052 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25067 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25067 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25531 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25531 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ for t in "${!pids[@]}" 2024-08-19T12:57:37.446 INFO:tasks.workunit.client.0.smithi104.stderr:+ pid=25764 2024-08-19T12:57:37.447 INFO:tasks.workunit.client.0.smithi104.stderr:+ wait 25764 2024-08-19T12:57:37.447 INFO:tasks.workunit.client.0.smithi104.stderr:+ exit 0 2024-08-19T12:57:37.447 INFO:tasks.workunit.client.0.smithi104.stderr:+ cleanup 2024-08-19T12:57:37.447 INFO:tasks.workunit.client.0.smithi104.stderr:+ pkill -P 24424 2024-08-19T12:57:37.514 INFO:tasks.workunit.client.0.smithi104.stderr:+ true 2024-08-19T12:57:37.514 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-19T12:57:37.515 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-19T12:57:37.544 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2024-08-19T12:57:37.544 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-19T12:57:37.871 DEBUG:teuthology.parallel:result is None 2024-08-19T12:57:37.872 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-08-19T12:57:37.897 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-08-19T12:57:37.897 DEBUG:teuthology.orchestra.run.smithi104:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-08-19T12:57:37.957 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-08-19T12:57:37.957 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-08-19T12:57:37.967 INFO:tasks.mon_thrash:joining mon_thrasher 2024-08-19T12:57:37.968 ERROR:teuthology.run_tasks:Manager failed: mon_thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 466, in task thrash_proc.stop_and_join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 178, in stop_and_join return self.join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 170, in join self.stopping.set() AttributeError: 'bool' object has no attribute 'set' 2024-08-19T12:57:37.969 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-19T12:57:37.994 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-19T12:57:37.994 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T12:57:38.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.105+0000 7fd53518f640 10 monclient: get_monmap_and_config 2024-08-19T12:57:38.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.105+0000 7fd53518f640 10 monclient: build_initial_monmap 2024-08-19T12:57:38.106 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.105+0000 7fd53518f640 10 monclient: monmap: 2024-08-19T12:57:38.106 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:38.106 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.106 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:38.106144+0000 2024-08-19T12:57:38.106 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:38.106144+0000 2024-08-19T12:57:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:38.107 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:38.107 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.105+0000 7fd53518f640 1 Processor -- start 2024-08-19T12:57:38.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.105+0000 7fd53518f640 1 -- start start 2024-08-19T12:57:38.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.105+0000 7fd53518f640 10 monclient: init 2024-08-19T12:57:38.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd53518f640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:38.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd53518f640 10 monclient: _add_conns ranks=[7,2,3,4,0,5,8,6,1] 2024-08-19T12:57:38.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd53518f640 10 monclient(hunting): picked mon.noname-i con 0x7fd530139690 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:38.109 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd53518f640 10 monclient(hunting): picked mon.noname-b con 0x7fd53006eb10 addr v1:172.21.15.26:6790/0 2024-08-19T12:57:38.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd53518f640 10 monclient(hunting): picked mon.noname-g con 0x7fd53013fc10 addr v1:172.21.15.104:6790/0 2024-08-19T12:57:38.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd53518f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5300a5c10 con 0x7fd53006eb10 2024-08-19T12:57:38.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd53518f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5300a6090 con 0x7fd53013fc10 2024-08-19T12:57:38.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd53518f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5300a5e10 con 0x7fd530139690 2024-08-19T12:57:38.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd53518f640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:38.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd53518f640 10 monclient(hunting): authenticate will time out at 1852.193545s 2024-08-19T12:57:38.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd52f59e640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7fd53013fc10 legacy=0x7fd530142000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:38.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd52e59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fd53006eb10 0x7fd53013e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38586/0 (socket says 172.21.15.26:38586) 2024-08-19T12:57:38.110 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.106+0000 7fd52e59c640 1 -- 172.21.15.26:0/160040983 learned_addr learned my addr 172.21.15.26:0/160040983 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:38.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.108+0000 7fd52e59c640 1 -- 172.21.15.26:0/160040983 >> v1:172.21.15.26:6790/0 conn(0x7fd53006eb10 legacy=0x7fd53013e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:38.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52ed9d640 1 -- 172.21.15.26:0/160040983 >> v1:172.21.15.104:6792/0 conn(0x7fd530139690 legacy=0x7fd530139a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:38.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 1 -- 172.21.15.26:0/160040983 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2228870324 0 0) 0x7fd5300a5c10 con 0x7fd53006eb10 2024-08-19T12:57:38.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:57:38.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:38.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:38.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient(hunting): my global_id is 27497 2024-08-19T12:57:38.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 1 -- 172.21.15.26:0/160040983 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd518003520 con 0x7fd53006eb10 2024-08-19T12:57:38.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 1 -- 172.21.15.26:0/160040983 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2677033796 0 0) 0x7fd518003520 con 0x7fd53006eb10 2024-08-19T12:57:38.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 1 -- 172.21.15.26:0/160040983 >> v1:172.21.15.104:6792/0 conn(0x7fd530139690 legacy=0x7fd530139a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:38.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 1 -- 172.21.15.26:0/160040983 >> v1:172.21.15.104:6790/0 conn(0x7fd53013fc10 legacy=0x7fd530142000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:38.119 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient: _finish_hunting 0 2024-08-19T12:57:38.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 1 monclient: found mon.noname-b 2024-08-19T12:57:38.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:38.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:57:38.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 1 -- 172.21.15.26:0/160040983 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53006f590 con 0x7fd53006eb10 2024-08-19T12:57:38.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient: _finish_auth 0 2024-08-19T12:57:38.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient: _check_auth_tickets 2024-08-19T12:57:38.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:38.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 1 -- 172.21.15.26:0/160040983 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd524001ba0 con 0x7fd53006eb10 2024-08-19T12:57:38.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd53518f640 5 monclient: authenticate success, global_id 27497 2024-08-19T12:57:38.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd53518f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:38.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:38.120 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:57:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient: dump: 2024-08-19T12:57:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.121 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.122 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.122 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.122 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.122 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.122 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd53518f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:38.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 1 -- 172.21.15.26:0/160040983 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd5240048a0 con 0x7fd53006eb10 2024-08-19T12:57:38.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:38.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 1 -- 172.21.15.26:0/160040983 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd524004e10 con 0x7fd53006eb10 2024-08-19T12:57:38.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd53518f640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:38.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd53518f640 4 set_mon_vals no callback set 2024-08-19T12:57:38.122 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd52dd9b640 10 monclient: dump: 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.123 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.124 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.124 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.124 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.124 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.124 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd53518f640 10 monclient: shutdown 2024-08-19T12:57:38.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd53518f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:38.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd53518f640 1 -- 172.21.15.26:0/160040983 >> v1:172.21.15.26:6790/0 conn(0x7fd53006eb10 legacy=0x7fd53013e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:38.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd53518f640 1 -- 172.21.15.26:0/160040983 shutdown_connections 2024-08-19T12:57:38.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd53518f640 1 -- 172.21.15.26:0/160040983 >> 172.21.15.26:0/160040983 conn(0x7fd5300a65d0 msgr2=0x7fd5300a69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:38.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd53518f640 1 -- 172.21.15.26:0/160040983 shutdown_connections 2024-08-19T12:57:38.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.109+0000 7fd53518f640 1 -- 172.21.15.26:0/160040983 wait complete. 2024-08-19T12:57:38.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.110+0000 7fd53518f640 10 monclient: build_initial_monmap 2024-08-19T12:57:38.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.110+0000 7fd53518f640 10 monclient: monmap: 2024-08-19T12:57:38.125 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:38.125 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:38.126 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:38.126 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:38.126 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:38.126 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.126 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.110+0000 7fd53518f640 1 librados: starting msgr at 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.110+0000 7fd53518f640 1 librados: starting objecter 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 1 Processor -- start 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 1 -- start start 2024-08-19T12:57:38.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 1 librados: setting wanted keys 2024-08-19T12:57:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 1 librados: calling monclient init 2024-08-19T12:57:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 10 monclient: init 2024-08-19T12:57:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 10 monclient: _add_conns ranks=[1,6,7,2,3,4,8,0,5] 2024-08-19T12:57:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 10 monclient(hunting): picked mon.noname-f con 0x7fd53006eb10 addr v1:172.21.15.104:6789/0 2024-08-19T12:57:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 10 monclient(hunting): picked mon.noname-d con 0x7fd530139690 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 10 monclient(hunting): picked mon.noname-i con 0x7fd53013fc10 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5301382a0 con 0x7fd53006eb10 2024-08-19T12:57:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd530137060 con 0x7fd530139690 2024-08-19T12:57:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd530139b10 con 0x7fd53013fc10 2024-08-19T12:57:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:38.128 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd53518f640 10 monclient(hunting): authenticate will time out at 1852.198720s 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd52e59c640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fd530139690 legacy=0x7fd5301d2f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd52ed9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fd53006eb10 0x7fd53013f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:37572/0 (socket says 172.21.15.26:37572) 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd52f59e640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7fd53013fc10 0x7fd5301d6690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:45304/0 (socket says 172.21.15.26:45304) 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.111+0000 7fd52ed9d640 1 -- 172.21.15.26:0/2175216667 learned_addr learned my addr 172.21.15.26:0/2175216667 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd52f59e640 1 -- 172.21.15.26:0/2175216667 >> v1:172.21.15.104:6792/0 conn(0x7fd53013fc10 legacy=0x7fd5301d6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd52ed9d640 1 -- 172.21.15.26:0/2175216667 >> v1:172.21.15.104:6789/0 conn(0x7fd53006eb10 legacy=0x7fd53013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 client.?.objecter ms_handle_connect 0x7fd53013fc10 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 client.?.objecter ms_handle_connect 0x7fd53006eb10 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1001799331 0 0) 0x7fd530139b10 con 0x7fd53013fc10 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:38.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): my global_id is 136537 2024-08-19T12:57:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd51c004eb0 con 0x7fd53013fc10 2024-08-19T12:57:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2012111171 0 0) 0x7fd5301382a0 con 0x7fd53006eb10 2024-08-19T12:57:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): my global_id is 204346 2024-08-19T12:57:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd530139b10 con 0x7fd53006eb10 2024-08-19T12:57:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2165231864 0 0) 0x7fd530139b10 con 0x7fd53006eb10 2024-08-19T12:57:38.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5301382a0 con 0x7fd53006eb10 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2911771378 0 0) 0x7fd51c004eb0 con 0x7fd53013fc10 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd530139b10 con 0x7fd53013fc10 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd520003610 con 0x7fd53006eb10 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): dump: 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd528004840 con 0x7fd53013fc10 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient(hunting): dump: 2024-08-19T12:57:38.132 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.133 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.134 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mon.1 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2104886053 0 0) 0x7fd5301382a0 con 0x7fd53006eb10 2024-08-19T12:57:38.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 >> v1:172.21.15.104:6792/0 conn(0x7fd53013fc10 legacy=0x7fd5301d6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:38.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 >> v1:172.21.15.26:6792/0 conn(0x7fd530139690 legacy=0x7fd5301d2f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:38.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient: _finish_hunting 0 2024-08-19T12:57:38.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 monclient: found mon.f 2024-08-19T12:57:38.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:38.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:57:38.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd530137bc0 con 0x7fd53006eb10 2024-08-19T12:57:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient: _finish_auth 0 2024-08-19T12:57:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient: _check_auth_tickets 2024-08-19T12:57:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd53518f640 5 monclient: authenticate success, global_id 204346 2024-08-19T12:57:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd53518f640 10 monclient: _renew_subs 2024-08-19T12:57:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd53518f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:57:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd53518f640 1 -- 172.21.15.26:0/2175216667 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd53013f790 con 0x7fd53006eb10 2024-08-19T12:57:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd53518f640 10 client.204346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd53518f640 10 monclient: _renew_subs 2024-08-19T12:57:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd53518f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:57:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd53518f640 1 -- 172.21.15.26:0/2175216667 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd5301ce030 con 0x7fd53006eb10 2024-08-19T12:57:38.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd53518f640 1 librados: init done 2024-08-19T12:57:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mon.1 v1:172.21.15.104:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd520003bc0 con 0x7fd53006eb10 2024-08-19T12:57:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mon.1 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fd5200040b0 con 0x7fd53006eb10 2024-08-19T12:57:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:57:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd52d59a640 4 set_mon_vals no callback set 2024-08-19T12:57:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.114+0000 7fd5177fe640 10 monclient: dump: 2024-08-19T12:57:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.136 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.137 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.138 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.115+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mon.1 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fd520004310 con 0x7fd53006eb10 2024-08-19T12:57:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.115+0000 7fd53518f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.115+0000 7fd53518f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.115+0000 7fd53518f640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.104:6789/0 2024-08-19T12:57:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.115+0000 7fd53518f640 1 -- 172.21.15.26:0/2175216667 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd510005080 con 0x7fd53006eb10 2024-08-19T12:57:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.116+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mon.1 v1:172.21.15.104:6789/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7fd52004dfa0 con 0x7fd53006eb10 2024-08-19T12:57:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.116+0000 7fd5177fe640 10 client.204346.objecter ms_dispatch 0x7fd530000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.116+0000 7fd5177fe640 3 client.204346.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.116+0000 7fd5177fe640 3 client.204346.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.116+0000 7fd5177fe640 20 client.204346.objecter dump_active .. 0 homeless 2024-08-19T12:57:38.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.116+0000 7fd52e59c640 10 monclient: get_auth_request con 0x7fd51c0413f0 auth_method 0 2024-08-19T12:57:38.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.118+0000 7fd5177fe640 10 client.204346.objecter ms_handle_connect 0x7fd51c0413f0 2024-08-19T12:57:38.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.120+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mon.1 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fd520017640 con 0x7fd53006eb10 2024-08-19T12:57:38.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.120+0000 7fd5177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:38.139 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.120+0000 7fd5177fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:57:38.302 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.302+0000 7fd53518f640 1 -- 172.21.15.26:0/2175216667 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd510000b90 con 0x7fd51c0413f0 2024-08-19T12:57:38.305 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.304+0000 7fd5177fe640 1 -- 172.21.15.26:0/2175216667 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114255 (unknown 2965378022 0 1799848966) 0x7fd510000b90 con 0x7fd51c0413f0 2024-08-19T12:57:38.306 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:57:38.306 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-08-19T12:57:38.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 10 librados: watch_flush enter 2024-08-19T12:57:38.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 10 librados: watch_flush exit 2024-08-19T12:57:38.309 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 20 client.204346.objecter shutdown clearing up homeless session... 2024-08-19T12:57:38.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 10 client.204346.objecter successfully canceled tick 2024-08-19T12:57:38.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 1 -- 172.21.15.26:0/2175216667 >> v1:172.21.15.104:6800/899887506 conn(0x7fd51c0413f0 legacy=0x7fd51c0438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:38.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 10 monclient: shutdown 2024-08-19T12:57:38.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:38.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 1 -- 172.21.15.26:0/2175216667 >> v1:172.21.15.104:6789/0 conn(0x7fd53006eb10 legacy=0x7fd53013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:38.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 1 -- 172.21.15.26:0/2175216667 shutdown_connections 2024-08-19T12:57:38.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 1 -- 172.21.15.26:0/2175216667 >> 172.21.15.26:0/2175216667 conn(0x7fd5300a65d0 msgr2=0x7fd530072b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:38.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 1 -- 172.21.15.26:0/2175216667 shutdown_connections 2024-08-19T12:57:38.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 1 -- 172.21.15.26:0/2175216667 wait complete. 2024-08-19T12:57:38.310 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.306+0000 7fd53518f640 1 librados: shutdown 2024-08-19T12:57:38.318 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1266,"stamp":"2024-08-19T12:57:37.417739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":99,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":441372,"kb_used_data":33484,"kb_used_omap":9,"kb_used_meta":407734,"kb_avail":565789668,"statfs":{"total":579820584960,"available":579368620032,"internally_reserved":0,"allocated":34287616,"data_stored":15227072,"data_compressed":10271,"data_compressed_allocated":69632,"data_compressed_original":1089536,"omap_allocated":9566,"internal_metadata":417520290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032576"},"pg_stats":[{"pgid":"7.17","version":"0'0","reported_seq":1357,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T12:57:32.558282+0000","last_change":"2024-08-19T12:39:38.156896+0000","last_active":"2024-08-19T12:57:32.558282+0000","last_peered":"2024-08-19T12:57:32.558282+0000","last_clean":"2024-08-19T12:57:32.558282+0000","last_became_active":"2024-08-19T12:39:38.143851+0000","last_became_peered":"2024-08-19T12:39:38.143851+0000","last_unstale":"2024-08-19T12:57:32.558282+0000","last_undegraded":"2024-08-19T12:57:32.558282+0000","last_fullsized":"2024-08-19T12:57:32.558282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:32:37.888575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1367,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.581141+0000","last_change":"2024-08-19T12:39:38.273079+0000","last_active":"2024-08-19T12:57:35.581141+0000","last_peered":"2024-08-19T12:57:35.581141+0000","last_clean":"2024-08-19T12:57:35.581141+0000","last_became_active":"2024-08-19T12:39:38.269168+0000","last_became_peered":"2024-08-19T12:39:38.269168+0000","last_unstale":"2024-08-19T12:57:35.581141+0000","last_undegraded":"2024-08-19T12:57:35.581141+0000","last_fullsized":"2024-08-19T12:57:35.581141+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:27:10.963118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578862+0000","last_change":"2024-08-19T12:39:38.205974+0000","last_active":"2024-08-19T12:57:35.578862+0000","last_peered":"2024-08-19T12:57:35.578862+0000","last_clean":"2024-08-19T12:57:35.578862+0000","last_became_active":"2024-08-19T12:39:38.201163+0000","last_became_peered":"2024-08-19T12:39:38.201163+0000","last_unstale":"2024-08-19T12:57:35.578862+0000","last_undegraded":"2024-08-19T12:57:35.578862+0000","last_fullsized":"2024-08-19T12:57:35.578862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:33:44.282679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1357,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861615+0000","last_change":"2024-08-19T12:39:38.230576+0000","last_active":"2024-08-19T12:57:35.861615+0000","last_peered":"2024-08-19T12:57:35.861615+0000","last_clean":"2024-08-19T12:57:35.861615+0000","last_became_active":"2024-08-19T12:39:38.183511+0000","last_became_peered":"2024-08-19T12:39:38.183511+0000","last_unstale":"2024-08-19T12:57:35.861615+0000","last_undegraded":"2024-08-19T12:57:35.861615+0000","last_fullsized":"2024-08-19T12:57:35.861615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:13:04.215179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1358,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861477+0000","last_change":"2024-08-19T12:39:38.095466+0000","last_active":"2024-08-19T12:57:35.861477+0000","last_peered":"2024-08-19T12:57:35.861477+0000","last_clean":"2024-08-19T12:57:35.861477+0000","last_became_active":"2024-08-19T12:39:38.094806+0000","last_became_peered":"2024-08-19T12:39:38.094806+0000","last_unstale":"2024-08-19T12:57:35.861477+0000","last_undegraded":"2024-08-19T12:57:35.861477+0000","last_fullsized":"2024-08-19T12:57:35.861477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:03:19.792598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1369,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579208+0000","last_change":"2024-08-19T12:39:38.222591+0000","last_active":"2024-08-19T12:57:35.579208+0000","last_peered":"2024-08-19T12:57:35.579208+0000","last_clean":"2024-08-19T12:57:35.579208+0000","last_became_active":"2024-08-19T12:39:38.219371+0000","last_became_peered":"2024-08-19T12:39:38.219371+0000","last_unstale":"2024-08-19T12:57:35.579208+0000","last_undegraded":"2024-08-19T12:57:35.579208+0000","last_fullsized":"2024-08-19T12:57:35.579208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:17:01.926550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740597+0000","last_change":"2024-08-19T12:39:38.237985+0000","last_active":"2024-08-19T12:57:34.740597+0000","last_peered":"2024-08-19T12:57:34.740597+0000","last_clean":"2024-08-19T12:57:34.740597+0000","last_became_active":"2024-08-19T12:39:38.228951+0000","last_became_peered":"2024-08-19T12:39:38.228951+0000","last_unstale":"2024-08-19T12:57:34.740597+0000","last_undegraded":"2024-08-19T12:57:34.740597+0000","last_fullsized":"2024-08-19T12:57:34.740597+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:47:35.671835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1369,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579208+0000","last_change":"2024-08-19T12:39:38.302683+0000","last_active":"2024-08-19T12:57:35.579208+0000","last_peered":"2024-08-19T12:57:35.579208+0000","last_clean":"2024-08-19T12:57:35.579208+0000","last_became_active":"2024-08-19T12:39:38.301100+0000","last_became_peered":"2024-08-19T12:39:38.301100+0000","last_unstale":"2024-08-19T12:57:35.579208+0000","last_undegraded":"2024-08-19T12:57:35.579208+0000","last_fullsized":"2024-08-19T12:57:35.579208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:02:38.662623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740971+0000","last_change":"2024-08-19T12:39:38.152043+0000","last_active":"2024-08-19T12:57:34.740971+0000","last_peered":"2024-08-19T12:57:34.740971+0000","last_clean":"2024-08-19T12:57:34.740971+0000","last_became_active":"2024-08-19T12:39:38.149472+0000","last_became_peered":"2024-08-19T12:39:38.149472+0000","last_unstale":"2024-08-19T12:57:34.740971+0000","last_undegraded":"2024-08-19T12:57:34.740971+0000","last_fullsized":"2024-08-19T12:57:34.740971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:03:37.142791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1369,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579403+0000","last_change":"2024-08-19T12:39:38.184326+0000","last_active":"2024-08-19T12:57:35.579403+0000","last_peered":"2024-08-19T12:57:35.579403+0000","last_clean":"2024-08-19T12:57:35.579403+0000","last_became_active":"2024-08-19T12:39:38.181280+0000","last_became_peered":"2024-08-19T12:39:38.181280+0000","last_unstale":"2024-08-19T12:57:35.579403+0000","last_undegraded":"2024-08-19T12:57:35.579403+0000","last_fullsized":"2024-08-19T12:57:35.579403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:00:41.274939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1368,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.581234+0000","last_change":"2024-08-19T12:39:38.154946+0000","last_active":"2024-08-19T12:57:35.581234+0000","last_peered":"2024-08-19T12:57:35.581234+0000","last_clean":"2024-08-19T12:57:35.581234+0000","last_became_active":"2024-08-19T12:39:38.147647+0000","last_became_peered":"2024-08-19T12:39:38.147647+0000","last_unstale":"2024-08-19T12:57:35.581234+0000","last_undegraded":"2024-08-19T12:57:35.581234+0000","last_fullsized":"2024-08-19T12:57:35.581234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:24:26.064970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1371,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578991+0000","last_change":"2024-08-19T12:39:38.137671+0000","last_active":"2024-08-19T12:57:35.578991+0000","last_peered":"2024-08-19T12:57:35.578991+0000","last_clean":"2024-08-19T12:57:35.578991+0000","last_became_active":"2024-08-19T12:39:38.135334+0000","last_became_peered":"2024-08-19T12:39:38.135334+0000","last_unstale":"2024-08-19T12:57:35.578991+0000","last_undegraded":"2024-08-19T12:57:35.578991+0000","last_fullsized":"2024-08-19T12:57:35.578991+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:37:29.348405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1478,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578723+0000","last_change":"2024-08-19T12:38:22.728884+0000","last_active":"2024-08-19T12:57:35.578723+0000","last_peered":"2024-08-19T12:57:35.578723+0000","last_clean":"2024-08-19T12:57:35.578723+0000","last_became_active":"2024-08-19T12:38:22.728595+0000","last_became_peered":"2024-08-19T12:38:22.728595+0000","last_unstale":"2024-08-19T12:57:35.578723+0000","last_undegraded":"2024-08-19T12:57:35.578723+0000","last_fullsized":"2024-08-19T12:57:35.578723+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_clean_scrub_stamp":"2024-08-19T12:38:21.711006+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:42:50.800603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1375,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T12:57:32.558167+0000","last_change":"2024-08-19T12:38:31.959938+0000","last_active":"2024-08-19T12:57:32.558167+0000","last_peered":"2024-08-19T12:57:32.558167+0000","last_clean":"2024-08-19T12:57:32.558167+0000","last_became_active":"2024-08-19T12:38:29.952471+0000","last_became_peered":"2024-08-19T12:38:29.952471+0000","last_unstale":"2024-08-19T12:57:32.558167+0000","last_undegraded":"2024-08-19T12:57:32.558167+0000","last_fullsized":"2024-08-19T12:57:32.558167+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:48:18.927151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057881800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740882+0000","last_change":"2024-08-19T12:39:38.261181+0000","last_active":"2024-08-19T12:57:34.740882+0000","last_peered":"2024-08-19T12:57:34.740882+0000","last_clean":"2024-08-19T12:57:34.740882+0000","last_became_active":"2024-08-19T12:39:38.260911+0000","last_became_peered":"2024-08-19T12:39:38.260911+0000","last_unstale":"2024-08-19T12:57:34.740882+0000","last_undegraded":"2024-08-19T12:57:34.740882+0000","last_fullsized":"2024-08-19T12:57:34.740882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:21:51.938672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1387,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578717+0000","last_change":"2024-08-19T12:38:31.960408+0000","last_active":"2024-08-19T12:57:35.578717+0000","last_peered":"2024-08-19T12:57:35.578717+0000","last_clean":"2024-08-19T12:57:35.578717+0000","last_became_active":"2024-08-19T12:38:29.951331+0000","last_became_peered":"2024-08-19T12:38:29.951331+0000","last_unstale":"2024-08-19T12:57:35.578717+0000","last_undegraded":"2024-08-19T12:57:35.578717+0000","last_fullsized":"2024-08-19T12:57:35.578717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:55:28.246342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018789830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1368,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.581284+0000","last_change":"2024-08-19T12:39:38.296122+0000","last_active":"2024-08-19T12:57:35.581284+0000","last_peered":"2024-08-19T12:57:35.581284+0000","last_clean":"2024-08-19T12:57:35.581284+0000","last_became_active":"2024-08-19T12:39:38.295614+0000","last_became_peered":"2024-08-19T12:39:38.295614+0000","last_unstale":"2024-08-19T12:57:35.581284+0000","last_undegraded":"2024-08-19T12:57:35.581284+0000","last_fullsized":"2024-08-19T12:57:35.581284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:57:09.397165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1385,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579218+0000","last_change":"2024-08-19T12:38:31.957412+0000","last_active":"2024-08-19T12:57:35.579218+0000","last_peered":"2024-08-19T12:57:35.579218+0000","last_clean":"2024-08-19T12:57:35.579218+0000","last_became_active":"2024-08-19T12:38:29.952459+0000","last_became_peered":"2024-08-19T12:38:29.952459+0000","last_unstale":"2024-08-19T12:57:35.579218+0000","last_undegraded":"2024-08-19T12:57:35.579218+0000","last_fullsized":"2024-08-19T12:57:35.579218+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:47:27.910005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039863799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740798+0000","last_change":"2024-08-19T12:39:38.224517+0000","last_active":"2024-08-19T12:57:34.740798+0000","last_peered":"2024-08-19T12:57:34.740798+0000","last_clean":"2024-08-19T12:57:34.740798+0000","last_became_active":"2024-08-19T12:39:38.218230+0000","last_became_peered":"2024-08-19T12:39:38.218230+0000","last_unstale":"2024-08-19T12:57:34.740798+0000","last_undegraded":"2024-08-19T12:57:34.740798+0000","last_fullsized":"2024-08-19T12:57:34.740798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:20:03.811299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1376,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T12:57:32.557730+0000","last_change":"2024-08-19T12:38:31.959929+0000","last_active":"2024-08-19T12:57:32.557730+0000","last_peered":"2024-08-19T12:57:32.557730+0000","last_clean":"2024-08-19T12:57:32.557730+0000","last_became_active":"2024-08-19T12:38:29.951392+0000","last_became_peered":"2024-08-19T12:38:29.951392+0000","last_unstale":"2024-08-19T12:57:32.557730+0000","last_undegraded":"2024-08-19T12:57:32.557730+0000","last_fullsized":"2024-08-19T12:57:32.557730+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:47:44.756214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056602099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1371,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578919+0000","last_change":"2024-08-19T12:39:38.220796+0000","last_active":"2024-08-19T12:57:35.578919+0000","last_peered":"2024-08-19T12:57:35.578919+0000","last_clean":"2024-08-19T12:57:35.578919+0000","last_became_active":"2024-08-19T12:39:38.217569+0000","last_became_peered":"2024-08-19T12:39:38.217569+0000","last_unstale":"2024-08-19T12:57:35.578919+0000","last_undegraded":"2024-08-19T12:57:35.578919+0000","last_fullsized":"2024-08-19T12:57:35.578919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:14:24.659508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1374,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861595+0000","last_change":"2024-08-19T12:38:31.963352+0000","last_active":"2024-08-19T12:57:35.861595+0000","last_peered":"2024-08-19T12:57:35.861595+0000","last_clean":"2024-08-19T12:57:35.861595+0000","last_became_active":"2024-08-19T12:38:29.951233+0000","last_became_peered":"2024-08-19T12:38:29.951233+0000","last_unstale":"2024-08-19T12:57:35.861595+0000","last_undegraded":"2024-08-19T12:57:35.861595+0000","last_fullsized":"2024-08-19T12:57:35.861595+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:45:51.806389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048735899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"22'7","reported_seq":1381,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.581124+0000","last_change":"2024-08-19T12:39:38.270101+0000","last_active":"2024-08-19T12:57:35.581124+0000","last_peered":"2024-08-19T12:57:35.581124+0000","last_clean":"2024-08-19T12:57:35.581124+0000","last_became_active":"2024-08-19T12:39:38.269442+0000","last_became_peered":"2024-08-19T12:39:38.269442+0000","last_unstale":"2024-08-19T12:57:35.581124+0000","last_undegraded":"2024-08-19T12:57:35.581124+0000","last_fullsized":"2024-08-19T12:57:35.581124+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:02:35.119493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1387,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578970+0000","last_change":"2024-08-19T12:38:31.960297+0000","last_active":"2024-08-19T12:57:35.578970+0000","last_peered":"2024-08-19T12:57:35.578970+0000","last_clean":"2024-08-19T12:57:35.578970+0000","last_became_active":"2024-08-19T12:38:29.949846+0000","last_became_peered":"2024-08-19T12:38:29.949846+0000","last_unstale":"2024-08-19T12:57:35.578970+0000","last_undegraded":"2024-08-19T12:57:35.578970+0000","last_fullsized":"2024-08-19T12:57:35.578970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:34:31.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001822192,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1358,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T12:57:32.557275+0000","last_change":"2024-08-19T12:39:38.181922+0000","last_active":"2024-08-19T12:57:32.557275+0000","last_peered":"2024-08-19T12:57:32.557275+0000","last_clean":"2024-08-19T12:57:32.557275+0000","last_became_active":"2024-08-19T12:39:38.166658+0000","last_became_peered":"2024-08-19T12:39:38.166658+0000","last_unstale":"2024-08-19T12:57:32.557275+0000","last_undegraded":"2024-08-19T12:57:32.557275+0000","last_fullsized":"2024-08-19T12:57:32.557275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:21:22.453156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1373,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861373+0000","last_change":"2024-08-19T12:38:31.963379+0000","last_active":"2024-08-19T12:57:35.861373+0000","last_peered":"2024-08-19T12:57:35.861373+0000","last_clean":"2024-08-19T12:57:35.861373+0000","last_became_active":"2024-08-19T12:38:29.950467+0000","last_became_peered":"2024-08-19T12:38:29.950467+0000","last_unstale":"2024-08-19T12:57:35.861373+0000","last_undegraded":"2024-08-19T12:57:35.861373+0000","last_fullsized":"2024-08-19T12:57:35.861373+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:40:07.574639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049350399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740778+0000","last_change":"2024-08-19T12:39:38.196037+0000","last_active":"2024-08-19T12:57:34.740778+0000","last_peered":"2024-08-19T12:57:34.740778+0000","last_clean":"2024-08-19T12:57:34.740778+0000","last_became_active":"2024-08-19T12:39:38.190786+0000","last_became_peered":"2024-08-19T12:39:38.190786+0000","last_unstale":"2024-08-19T12:57:34.740778+0000","last_undegraded":"2024-08-19T12:57:34.740778+0000","last_fullsized":"2024-08-19T12:57:34.740778+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:07:56.534454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1386,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578815+0000","last_change":"2024-08-19T12:38:31.959032+0000","last_active":"2024-08-19T12:57:35.578815+0000","last_peered":"2024-08-19T12:57:35.578815+0000","last_clean":"2024-08-19T12:57:35.578815+0000","last_became_active":"2024-08-19T12:38:29.948183+0000","last_became_peered":"2024-08-19T12:38:29.948183+0000","last_unstale":"2024-08-19T12:57:35.578815+0000","last_undegraded":"2024-08-19T12:57:35.578815+0000","last_fullsized":"2024-08-19T12:57:35.578815+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:51:55.545957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052740100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1369,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579281+0000","last_change":"2024-08-19T12:39:38.221167+0000","last_active":"2024-08-19T12:57:35.579281+0000","last_peered":"2024-08-19T12:57:35.579281+0000","last_clean":"2024-08-19T12:57:35.579281+0000","last_became_active":"2024-08-19T12:39:38.191158+0000","last_became_peered":"2024-08-19T12:39:38.191158+0000","last_unstale":"2024-08-19T12:57:35.579281+0000","last_undegraded":"2024-08-19T12:57:35.579281+0000","last_fullsized":"2024-08-19T12:57:35.579281+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T22:59:47.700950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1357,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861035+0000","last_change":"2024-08-19T12:39:38.194777+0000","last_active":"2024-08-19T12:57:35.861035+0000","last_peered":"2024-08-19T12:57:35.861035+0000","last_clean":"2024-08-19T12:57:35.861035+0000","last_became_active":"2024-08-19T12:39:38.161352+0000","last_became_peered":"2024-08-19T12:39:38.161352+0000","last_unstale":"2024-08-19T12:57:35.861035+0000","last_undegraded":"2024-08-19T12:57:35.861035+0000","last_fullsized":"2024-08-19T12:57:35.861035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:31:21.012106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578750+0000","last_change":"2024-08-19T12:39:38.210090+0000","last_active":"2024-08-19T12:57:35.578750+0000","last_peered":"2024-08-19T12:57:35.578750+0000","last_clean":"2024-08-19T12:57:35.578750+0000","last_became_active":"2024-08-19T12:39:38.202668+0000","last_became_peered":"2024-08-19T12:39:38.202668+0000","last_unstale":"2024-08-19T12:57:35.578750+0000","last_undegraded":"2024-08-19T12:57:35.578750+0000","last_fullsized":"2024-08-19T12:57:35.578750+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:58:28.292312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1357,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T12:57:32.557636+0000","last_change":"2024-08-19T12:39:38.224507+0000","last_active":"2024-08-19T12:57:32.557636+0000","last_peered":"2024-08-19T12:57:32.557636+0000","last_clean":"2024-08-19T12:57:32.557636+0000","last_became_active":"2024-08-19T12:39:38.222877+0000","last_became_peered":"2024-08-19T12:39:38.222877+0000","last_unstale":"2024-08-19T12:57:32.557636+0000","last_undegraded":"2024-08-19T12:57:32.557636+0000","last_fullsized":"2024-08-19T12:57:32.557636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:55:08.088675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740614+0000","last_change":"2024-08-19T12:39:38.211834+0000","last_active":"2024-08-19T12:57:34.740614+0000","last_peered":"2024-08-19T12:57:34.740614+0000","last_clean":"2024-08-19T12:57:34.740614+0000","last_became_active":"2024-08-19T12:39:38.210985+0000","last_became_peered":"2024-08-19T12:39:38.210985+0000","last_unstale":"2024-08-19T12:57:34.740614+0000","last_undegraded":"2024-08-19T12:57:34.740614+0000","last_fullsized":"2024-08-19T12:57:34.740614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:39:57.635965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578850+0000","last_change":"2024-08-19T12:39:38.268396+0000","last_active":"2024-08-19T12:57:35.578850+0000","last_peered":"2024-08-19T12:57:35.578850+0000","last_clean":"2024-08-19T12:57:35.578850+0000","last_became_active":"2024-08-19T12:39:38.267362+0000","last_became_peered":"2024-08-19T12:39:38.267362+0000","last_unstale":"2024-08-19T12:57:35.578850+0000","last_undegraded":"2024-08-19T12:57:35.578850+0000","last_fullsized":"2024-08-19T12:57:35.578850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:26:11.466849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1357,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861686+0000","last_change":"2024-08-19T12:39:38.195841+0000","last_active":"2024-08-19T12:57:35.861686+0000","last_peered":"2024-08-19T12:57:35.861686+0000","last_clean":"2024-08-19T12:57:35.861686+0000","last_became_active":"2024-08-19T12:39:38.173967+0000","last_became_peered":"2024-08-19T12:39:38.173967+0000","last_unstale":"2024-08-19T12:57:35.861686+0000","last_undegraded":"2024-08-19T12:57:35.861686+0000","last_fullsized":"2024-08-19T12:57:35.861686+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:29:39.717188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1359,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861315+0000","last_change":"2024-08-19T12:39:38.110553+0000","last_active":"2024-08-19T12:57:35.861315+0000","last_peered":"2024-08-19T12:57:35.861315+0000","last_clean":"2024-08-19T12:57:35.861315+0000","last_became_active":"2024-08-19T12:39:38.108670+0000","last_became_peered":"2024-08-19T12:39:38.108670+0000","last_unstale":"2024-08-19T12:57:35.861315+0000","last_undegraded":"2024-08-19T12:57:35.861315+0000","last_fullsized":"2024-08-19T12:57:35.861315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:32:52.104320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1358,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T12:57:32.558370+0000","last_change":"2024-08-19T12:39:38.200400+0000","last_active":"2024-08-19T12:57:32.558370+0000","last_peered":"2024-08-19T12:57:32.558370+0000","last_clean":"2024-08-19T12:57:32.558370+0000","last_became_active":"2024-08-19T12:39:38.197445+0000","last_became_peered":"2024-08-19T12:39:38.197445+0000","last_unstale":"2024-08-19T12:57:32.558370+0000","last_undegraded":"2024-08-19T12:57:32.558370+0000","last_fullsized":"2024-08-19T12:57:32.558370+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:21:39.167660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740451+0000","last_change":"2024-08-19T12:39:38.274353+0000","last_active":"2024-08-19T12:57:34.740451+0000","last_peered":"2024-08-19T12:57:34.740451+0000","last_clean":"2024-08-19T12:57:34.740451+0000","last_became_active":"2024-08-19T12:39:38.267711+0000","last_became_peered":"2024-08-19T12:39:38.267711+0000","last_unstale":"2024-08-19T12:57:34.740451+0000","last_undegraded":"2024-08-19T12:57:34.740451+0000","last_fullsized":"2024-08-19T12:57:34.740451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:56:56.848415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578730+0000","last_change":"2024-08-19T12:39:38.188765+0000","last_active":"2024-08-19T12:57:35.578730+0000","last_peered":"2024-08-19T12:57:35.578730+0000","last_clean":"2024-08-19T12:57:35.578730+0000","last_became_active":"2024-08-19T12:39:38.177735+0000","last_became_peered":"2024-08-19T12:39:38.177735+0000","last_unstale":"2024-08-19T12:57:35.578730+0000","last_undegraded":"2024-08-19T12:57:35.578730+0000","last_fullsized":"2024-08-19T12:57:35.578730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:38:45.637343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578720+0000","last_change":"2024-08-19T12:39:38.200875+0000","last_active":"2024-08-19T12:57:35.578720+0000","last_peered":"2024-08-19T12:57:35.578720+0000","last_clean":"2024-08-19T12:57:35.578720+0000","last_became_active":"2024-08-19T12:39:38.199715+0000","last_became_peered":"2024-08-19T12:39:38.199715+0000","last_unstale":"2024-08-19T12:57:35.578720+0000","last_undegraded":"2024-08-19T12:57:35.578720+0000","last_fullsized":"2024-08-19T12:57:35.578720+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T22:27:29.457630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740661+0000","last_change":"2024-08-19T12:39:38.196277+0000","last_active":"2024-08-19T12:57:34.740661+0000","last_peered":"2024-08-19T12:57:34.740661+0000","last_clean":"2024-08-19T12:57:34.740661+0000","last_became_active":"2024-08-19T12:39:38.190871+0000","last_became_peered":"2024-08-19T12:39:38.190871+0000","last_unstale":"2024-08-19T12:57:34.740661+0000","last_undegraded":"2024-08-19T12:57:34.740661+0000","last_fullsized":"2024-08-19T12:57:34.740661+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:45:27.127112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68660,"kb_used_data":5536,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303180,"statfs":{"total":96636764160,"available":96566456320,"internally_reserved":0,"allocated":5668864,"data_stored":2406798,"data_compressed":43,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607786,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68704,"kb_used_data":5572,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303136,"statfs":{"total":96636764160,"available":96566411264,"internally_reserved":0,"allocated":5705728,"data_stored":2348545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607789,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68752,"kb_used_data":5620,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303088,"statfs":{"total":96636764160,"available":96566362112,"internally_reserved":0,"allocated":5754880,"data_stored":2877468,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1592,"internal_metadata":64616904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607785,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77020,"kb_used_data":5568,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294820,"statfs":{"total":96636764160,"available":96557895680,"internally_reserved":0,"allocated":5701632,"data_stored":2348540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607786,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77016,"kb_used_data":5572,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294824,"statfs":{"total":96636764160,"available":96557899776,"internally_reserved":0,"allocated":5705728,"data_stored":2348559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640491,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81220,"kb_used_data":5616,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290620,"statfs":{"total":96636764160,"available":96553594880,"internally_reserved":0,"allocated":5750784,"data_stored":2897162,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T12:57:38.320 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T12:57:38.418 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.418+0000 7fa7a9b08640 10 monclient: get_monmap_and_config 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.418+0000 7fa7a9b08640 10 monclient: build_initial_monmap 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.418+0000 7fa7a9b08640 10 monclient: monmap: 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:38.419033+0000 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:38.419033+0000 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:38.419 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:38.420 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:38.420 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:38.420 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:38.420 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:38.420 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 1 Processor -- start 2024-08-19T12:57:38.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 1 -- start start 2024-08-19T12:57:38.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 10 monclient: init 2024-08-19T12:57:38.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:38.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 10 monclient: _add_conns ranks=[5,8,6,7,4,0,3,1,2] 2024-08-19T12:57:38.421 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 10 monclient(hunting): picked mon.noname-h con 0x7fa7a4073470 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:38.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 10 monclient(hunting): picked mon.noname-e con 0x7fa7a40a57a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:57:38.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 10 monclient(hunting): picked mon.noname-d con 0x7fa7a413f2e0 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:38.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a409dae0 con 0x7fa7a4073470 2024-08-19T12:57:38.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a4136910 con 0x7fa7a413f2e0 2024-08-19T12:57:38.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a409dd50 con 0x7fa7a40a57a0 2024-08-19T12:57:38.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:38.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a9b08640 10 monclient(hunting): authenticate will time out at 1852.506715s 2024-08-19T12:57:38.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa7a3fff640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fa7a413f2e0 legacy=0x7fa7a41416d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:38.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa79bfff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fa7a40a57a0 0x7fa7a40a5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:54722/0 (socket says 172.21.15.26:54722) 2024-08-19T12:57:38.422 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.419+0000 7fa79bfff640 1 -- 172.21.15.26:0/1673954543 learned_addr learned my addr 172.21.15.26:0/1673954543 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:38.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.421+0000 7fa79bfff640 1 -- 172.21.15.26:0/1673954543 >> v1:172.21.15.26:6793/0 conn(0x7fa7a40a57a0 legacy=0x7fa7a40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:38.423 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.422+0000 7fa7a37fe640 1 -- 172.21.15.26:0/1673954543 >> v1:172.21.15.104:6791/0 conn(0x7fa7a4073470 legacy=0x7fa7a4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:38.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 1 -- 172.21.15.26:0/1673954543 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2445150425 0 0) 0x7fa7a409dd50 con 0x7fa7a40a57a0 2024-08-19T12:57:38.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:57:38.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:38.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:38.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient(hunting): my global_id is 184706 2024-08-19T12:57:38.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 1 -- 172.21.15.26:0/1673954543 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa790003520 con 0x7fa7a40a57a0 2024-08-19T12:57:38.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 1 -- 172.21.15.26:0/1673954543 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1043595803 0 0) 0x7fa7a409dae0 con 0x7fa7a4073470 2024-08-19T12:57:38.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:57:38.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:38.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:38.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient(hunting): my global_id is 184748 2024-08-19T12:57:38.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 1 -- 172.21.15.26:0/1673954543 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7a409dd50 con 0x7fa7a4073470 2024-08-19T12:57:38.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 1 -- 172.21.15.26:0/1673954543 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1911654501 0 0) 0x7fa790003520 con 0x7fa7a40a57a0 2024-08-19T12:57:38.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 1 -- 172.21.15.26:0/1673954543 >> v1:172.21.15.26:6792/0 conn(0x7fa7a413f2e0 legacy=0x7fa7a41416d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:38.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 1 -- 172.21.15.26:0/1673954543 >> v1:172.21.15.104:6791/0 conn(0x7fa7a4073470 legacy=0x7fa7a4136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:38.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:57:38.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 1 monclient: found mon.noname-e 2024-08-19T12:57:38.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:38.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:57:38.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 1 -- 172.21.15.26:0/1673954543 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a409e030 con 0x7fa7a40a57a0 2024-08-19T12:57:38.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient: _finish_auth 0 2024-08-19T12:57:38.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:57:38.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:38.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 1 -- 172.21.15.26:0/1673954543 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa78c001ae0 con 0x7fa7a40a57a0 2024-08-19T12:57:38.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:38.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:57:38.556 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient: dump: 2024-08-19T12:57:38.556 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.556 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.557 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a9b08640 5 monclient: authenticate success, global_id 184706 2024-08-19T12:57:38.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a9b08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:38.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 1 -- 172.21.15.26:0/1673954543 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa78c0048a0 con 0x7fa7a40a57a0 2024-08-19T12:57:38.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:38.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 1 -- 172.21.15.26:0/1673954543 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa78c004e10 con 0x7fa7a40a57a0 2024-08-19T12:57:38.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:38.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:57:38.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a2ffd640 10 monclient: dump: 2024-08-19T12:57:38.558 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.558 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.558 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.558 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a9b08640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.550+0000 7fa7a9b08640 4 set_mon_vals no callback set 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.551+0000 7fa7a9b08640 10 monclient: shutdown 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.551+0000 7fa7a9b08640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.551+0000 7fa7a9b08640 1 -- 172.21.15.26:0/1673954543 >> v1:172.21.15.26:6793/0 conn(0x7fa7a40a57a0 legacy=0x7fa7a40a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.551+0000 7fa7a9b08640 1 -- 172.21.15.26:0/1673954543 shutdown_connections 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.551+0000 7fa7a9b08640 1 -- 172.21.15.26:0/1673954543 >> 172.21.15.26:0/1673954543 conn(0x7fa7a40abff0 msgr2=0x7fa7a40ac400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.551+0000 7fa7a9b08640 1 -- 172.21.15.26:0/1673954543 shutdown_connections 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.551+0000 7fa7a9b08640 1 -- 172.21.15.26:0/1673954543 wait complete. 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.551+0000 7fa7a9b08640 10 monclient: build_initial_monmap 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.551+0000 7fa7a9b08640 10 monclient: monmap: 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 1 librados: starting msgr at 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 1 librados: starting objecter 2024-08-19T12:57:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 1 Processor -- start 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 1 -- start start 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 1 librados: setting wanted keys 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 1 librados: calling monclient init 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 10 monclient: init 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 10 monclient: _add_conns ranks=[2,1,0,6,7,8,5,4,3] 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a4073470 addr v1:172.21.15.26:6790/0 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 10 monclient(hunting): picked mon.noname-f con 0x7fa7a40a57a0 addr v1:172.21.15.104:6789/0 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a413f2e0 addr v1:172.21.15.26:6789/0 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a4131f90 con 0x7fa7a413f2e0 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a4142ba0 con 0x7fa7a40a57a0 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a412ece0 con 0x7fa7a4073470 2024-08-19T12:57:38.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:38.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a3fff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa7a413f2e0 legacy=0x7fa7a41d6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:57:38.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a9b08640 10 monclient(hunting): authenticate will time out at 1852.639802s 2024-08-19T12:57:38.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a37fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fa7a4073470 0x7fa7a412dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:38602/0 (socket says 172.21.15.26:38602) 2024-08-19T12:57:38.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.552+0000 7fa7a37fe640 1 -- 172.21.15.26:0/936628588 learned_addr learned my addr 172.21.15.26:0/936628588 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a37fe640 1 -- 172.21.15.26:0/936628588 >> v1:172.21.15.26:6790/0 conn(0x7fa7a4073470 legacy=0x7fa7a412dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 client.?.objecter ms_handle_connect 0x7fa7a4073470 2024-08-19T12:57:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa79bfff640 1 -- 172.21.15.26:0/936628588 >> v1:172.21.15.104:6789/0 conn(0x7fa7a40a57a0 legacy=0x7fa7a412e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 client.?.objecter ms_handle_connect 0x7fa7a40a57a0 2024-08-19T12:57:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3773972746 0 0) 0x7fa7a412ece0 con 0x7fa7a4073470 2024-08-19T12:57:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:57:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:38.567 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): my global_id is 27506 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa784004ee0 con 0x7fa7a4073470 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3334247456 0 0) 0x7fa7a409e030 con 0x7fa7a40a57a0 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): my global_id is 204355 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7a412ece0 con 0x7fa7a40a57a0 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2542148367 0 0) 0x7fa784004ee0 con 0x7fa7a4073470 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7a409e030 con 0x7fa7a4073470 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa794003950 con 0x7fa7a4073470 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): dump: 2024-08-19T12:57:38.568 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.569 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3312265546 0 0) 0x7fa7a412ece0 con 0x7fa7a40a57a0 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa784004ee0 con 0x7fa7a40a57a0 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa78c004d90 con 0x7fa7a40a57a0 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient(hunting): dump: 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.570 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2736302617 0 0) 0x7fa7a409e030 con 0x7fa7a4073470 2024-08-19T12:57:38.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 >> v1:172.21.15.104:6789/0 conn(0x7fa7a40a57a0 legacy=0x7fa7a412e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 >> v1:172.21.15.26:6789/0 conn(0x7fa7a413f2e0 legacy=0x7fa7a41d6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 monclient: found mon.b 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a4131890 con 0x7fa7a4073470 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient: _finish_auth 0 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a9b08640 5 monclient: authenticate success, global_id 27506 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a9b08640 10 monclient: _renew_subs 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a9b08640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a9b08640 1 -- 172.21.15.26:0/936628588 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a4131b70 con 0x7fa7a4073470 2024-08-19T12:57:38.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a9b08640 10 client.27506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a9b08640 10 monclient: _renew_subs 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa794001cf0 con 0x7fa7a4073470 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a9b08640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a9b08640 1 -- 172.21.15.26:0/936628588 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa7a41305e0 con 0x7fa7a4073470 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a9b08640 1 librados: init done 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa794004160 con 0x7fa7a4073470 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a0ff9640 10 monclient: dump: 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.573 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.555+0000 7fa7a27fc640 4 set_mon_vals no callback set 2024-08-19T12:57:38.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.556+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa794011a80 con 0x7fa7a4073470 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.556+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7fa79404e0a0 con 0x7fa7a4073470 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.556+0000 7fa7a0ff9640 10 client.27506.objecter ms_dispatch 0x7fa7a4000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.556+0000 7fa7a0ff9640 3 client.27506.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.556+0000 7fa7a0ff9640 3 client.27506.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.556+0000 7fa7a0ff9640 20 client.27506.objecter dump_active .. 0 homeless 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.556+0000 7fa79bfff640 10 monclient: get_auth_request con 0x7fa7840414d0 auth_method 0 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.556+0000 7fa7a9b08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.556+0000 7fa7a9b08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.556+0000 7fa7a9b08640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.556+0000 7fa7a9b08640 1 -- 172.21.15.26:0/936628588 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a413f020 con 0x7fa7a4073470 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.559+0000 7fa7a0ff9640 10 client.27506.objecter ms_handle_connect 0x7fa7840414d0 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.561+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fa794017680 con 0x7fa7a4073470 2024-08-19T12:57:38.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.561+0000 7fa7a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:38.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.561+0000 7fa7a0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:57:38.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.741+0000 7fa7a9b08640 1 -- 172.21.15.26:0/936628588 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa7a406c840 con 0x7fa7840414d0 2024-08-19T12:57:38.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.743+0000 7fa7a0ff9640 1 -- 172.21.15.26:0/936628588 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114255 (unknown 2965378022 0 1799848966) 0x7fa78c024e60 con 0x7fa7840414d0 2024-08-19T12:57:38.745 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:57:38.746 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-08-19T12:57:38.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 10 librados: watch_flush enter 2024-08-19T12:57:38.749 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 10 librados: watch_flush exit 2024-08-19T12:57:38.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 20 client.27506.objecter shutdown clearing up homeless session... 2024-08-19T12:57:38.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 10 client.27506.objecter successfully canceled tick 2024-08-19T12:57:38.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 1 -- 172.21.15.26:0/936628588 >> v1:172.21.15.104:6800/899887506 conn(0x7fa7840414d0 legacy=0x7fa784043990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:38.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 10 monclient: shutdown 2024-08-19T12:57:38.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:38.750 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 1 -- 172.21.15.26:0/936628588 >> v1:172.21.15.26:6790/0 conn(0x7fa7a4073470 legacy=0x7fa7a412dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:38.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 1 -- 172.21.15.26:0/936628588 shutdown_connections 2024-08-19T12:57:38.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 1 -- 172.21.15.26:0/936628588 >> 172.21.15.26:0/936628588 conn(0x7fa7a40abff0 msgr2=0x7fa7a412c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:38.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 1 -- 172.21.15.26:0/936628588 shutdown_connections 2024-08-19T12:57:38.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 1 -- 172.21.15.26:0/936628588 wait complete. 2024-08-19T12:57:38.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.745+0000 7fa7a9b08640 1 librados: shutdown 2024-08-19T12:57:38.756 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1266,"stamp":"2024-08-19T12:57:37.417739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":99,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":441372,"kb_used_data":33484,"kb_used_omap":9,"kb_used_meta":407734,"kb_avail":565789668,"statfs":{"total":579820584960,"available":579368620032,"internally_reserved":0,"allocated":34287616,"data_stored":15227072,"data_compressed":10271,"data_compressed_allocated":69632,"data_compressed_original":1089536,"omap_allocated":9566,"internal_metadata":417520290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032576"},"pg_stats":[{"pgid":"7.17","version":"0'0","reported_seq":1357,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T12:57:32.558282+0000","last_change":"2024-08-19T12:39:38.156896+0000","last_active":"2024-08-19T12:57:32.558282+0000","last_peered":"2024-08-19T12:57:32.558282+0000","last_clean":"2024-08-19T12:57:32.558282+0000","last_became_active":"2024-08-19T12:39:38.143851+0000","last_became_peered":"2024-08-19T12:39:38.143851+0000","last_unstale":"2024-08-19T12:57:32.558282+0000","last_undegraded":"2024-08-19T12:57:32.558282+0000","last_fullsized":"2024-08-19T12:57:32.558282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:32:37.888575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1367,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.581141+0000","last_change":"2024-08-19T12:39:38.273079+0000","last_active":"2024-08-19T12:57:35.581141+0000","last_peered":"2024-08-19T12:57:35.581141+0000","last_clean":"2024-08-19T12:57:35.581141+0000","last_became_active":"2024-08-19T12:39:38.269168+0000","last_became_peered":"2024-08-19T12:39:38.269168+0000","last_unstale":"2024-08-19T12:57:35.581141+0000","last_undegraded":"2024-08-19T12:57:35.581141+0000","last_fullsized":"2024-08-19T12:57:35.581141+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:27:10.963118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578862+0000","last_change":"2024-08-19T12:39:38.205974+0000","last_active":"2024-08-19T12:57:35.578862+0000","last_peered":"2024-08-19T12:57:35.578862+0000","last_clean":"2024-08-19T12:57:35.578862+0000","last_became_active":"2024-08-19T12:39:38.201163+0000","last_became_peered":"2024-08-19T12:39:38.201163+0000","last_unstale":"2024-08-19T12:57:35.578862+0000","last_undegraded":"2024-08-19T12:57:35.578862+0000","last_fullsized":"2024-08-19T12:57:35.578862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:33:44.282679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1357,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861615+0000","last_change":"2024-08-19T12:39:38.230576+0000","last_active":"2024-08-19T12:57:35.861615+0000","last_peered":"2024-08-19T12:57:35.861615+0000","last_clean":"2024-08-19T12:57:35.861615+0000","last_became_active":"2024-08-19T12:39:38.183511+0000","last_became_peered":"2024-08-19T12:39:38.183511+0000","last_unstale":"2024-08-19T12:57:35.861615+0000","last_undegraded":"2024-08-19T12:57:35.861615+0000","last_fullsized":"2024-08-19T12:57:35.861615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:13:04.215179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1358,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861477+0000","last_change":"2024-08-19T12:39:38.095466+0000","last_active":"2024-08-19T12:57:35.861477+0000","last_peered":"2024-08-19T12:57:35.861477+0000","last_clean":"2024-08-19T12:57:35.861477+0000","last_became_active":"2024-08-19T12:39:38.094806+0000","last_became_peered":"2024-08-19T12:39:38.094806+0000","last_unstale":"2024-08-19T12:57:35.861477+0000","last_undegraded":"2024-08-19T12:57:35.861477+0000","last_fullsized":"2024-08-19T12:57:35.861477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:03:19.792598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1369,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579208+0000","last_change":"2024-08-19T12:39:38.222591+0000","last_active":"2024-08-19T12:57:35.579208+0000","last_peered":"2024-08-19T12:57:35.579208+0000","last_clean":"2024-08-19T12:57:35.579208+0000","last_became_active":"2024-08-19T12:39:38.219371+0000","last_became_peered":"2024-08-19T12:39:38.219371+0000","last_unstale":"2024-08-19T12:57:35.579208+0000","last_undegraded":"2024-08-19T12:57:35.579208+0000","last_fullsized":"2024-08-19T12:57:35.579208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:17:01.926550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740597+0000","last_change":"2024-08-19T12:39:38.237985+0000","last_active":"2024-08-19T12:57:34.740597+0000","last_peered":"2024-08-19T12:57:34.740597+0000","last_clean":"2024-08-19T12:57:34.740597+0000","last_became_active":"2024-08-19T12:39:38.228951+0000","last_became_peered":"2024-08-19T12:39:38.228951+0000","last_unstale":"2024-08-19T12:57:34.740597+0000","last_undegraded":"2024-08-19T12:57:34.740597+0000","last_fullsized":"2024-08-19T12:57:34.740597+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:47:35.671835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1369,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579208+0000","last_change":"2024-08-19T12:39:38.302683+0000","last_active":"2024-08-19T12:57:35.579208+0000","last_peered":"2024-08-19T12:57:35.579208+0000","last_clean":"2024-08-19T12:57:35.579208+0000","last_became_active":"2024-08-19T12:39:38.301100+0000","last_became_peered":"2024-08-19T12:39:38.301100+0000","last_unstale":"2024-08-19T12:57:35.579208+0000","last_undegraded":"2024-08-19T12:57:35.579208+0000","last_fullsized":"2024-08-19T12:57:35.579208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:02:38.662623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740971+0000","last_change":"2024-08-19T12:39:38.152043+0000","last_active":"2024-08-19T12:57:34.740971+0000","last_peered":"2024-08-19T12:57:34.740971+0000","last_clean":"2024-08-19T12:57:34.740971+0000","last_became_active":"2024-08-19T12:39:38.149472+0000","last_became_peered":"2024-08-19T12:39:38.149472+0000","last_unstale":"2024-08-19T12:57:34.740971+0000","last_undegraded":"2024-08-19T12:57:34.740971+0000","last_fullsized":"2024-08-19T12:57:34.740971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:03:37.142791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1369,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579403+0000","last_change":"2024-08-19T12:39:38.184326+0000","last_active":"2024-08-19T12:57:35.579403+0000","last_peered":"2024-08-19T12:57:35.579403+0000","last_clean":"2024-08-19T12:57:35.579403+0000","last_became_active":"2024-08-19T12:39:38.181280+0000","last_became_peered":"2024-08-19T12:39:38.181280+0000","last_unstale":"2024-08-19T12:57:35.579403+0000","last_undegraded":"2024-08-19T12:57:35.579403+0000","last_fullsized":"2024-08-19T12:57:35.579403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:00:41.274939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1368,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.581234+0000","last_change":"2024-08-19T12:39:38.154946+0000","last_active":"2024-08-19T12:57:35.581234+0000","last_peered":"2024-08-19T12:57:35.581234+0000","last_clean":"2024-08-19T12:57:35.581234+0000","last_became_active":"2024-08-19T12:39:38.147647+0000","last_became_peered":"2024-08-19T12:39:38.147647+0000","last_unstale":"2024-08-19T12:57:35.581234+0000","last_undegraded":"2024-08-19T12:57:35.581234+0000","last_fullsized":"2024-08-19T12:57:35.581234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:24:26.064970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1371,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578991+0000","last_change":"2024-08-19T12:39:38.137671+0000","last_active":"2024-08-19T12:57:35.578991+0000","last_peered":"2024-08-19T12:57:35.578991+0000","last_clean":"2024-08-19T12:57:35.578991+0000","last_became_active":"2024-08-19T12:39:38.135334+0000","last_became_peered":"2024-08-19T12:39:38.135334+0000","last_unstale":"2024-08-19T12:57:35.578991+0000","last_undegraded":"2024-08-19T12:57:35.578991+0000","last_fullsized":"2024-08-19T12:57:35.578991+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:37:29.348405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1478,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578723+0000","last_change":"2024-08-19T12:38:22.728884+0000","last_active":"2024-08-19T12:57:35.578723+0000","last_peered":"2024-08-19T12:57:35.578723+0000","last_clean":"2024-08-19T12:57:35.578723+0000","last_became_active":"2024-08-19T12:38:22.728595+0000","last_became_peered":"2024-08-19T12:38:22.728595+0000","last_unstale":"2024-08-19T12:57:35.578723+0000","last_undegraded":"2024-08-19T12:57:35.578723+0000","last_fullsized":"2024-08-19T12:57:35.578723+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_clean_scrub_stamp":"2024-08-19T12:38:21.711006+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:42:50.800603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1375,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T12:57:32.558167+0000","last_change":"2024-08-19T12:38:31.959938+0000","last_active":"2024-08-19T12:57:32.558167+0000","last_peered":"2024-08-19T12:57:32.558167+0000","last_clean":"2024-08-19T12:57:32.558167+0000","last_became_active":"2024-08-19T12:38:29.952471+0000","last_became_peered":"2024-08-19T12:38:29.952471+0000","last_unstale":"2024-08-19T12:57:32.558167+0000","last_undegraded":"2024-08-19T12:57:32.558167+0000","last_fullsized":"2024-08-19T12:57:32.558167+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:48:18.927151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057881800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740882+0000","last_change":"2024-08-19T12:39:38.261181+0000","last_active":"2024-08-19T12:57:34.740882+0000","last_peered":"2024-08-19T12:57:34.740882+0000","last_clean":"2024-08-19T12:57:34.740882+0000","last_became_active":"2024-08-19T12:39:38.260911+0000","last_became_peered":"2024-08-19T12:39:38.260911+0000","last_unstale":"2024-08-19T12:57:34.740882+0000","last_undegraded":"2024-08-19T12:57:34.740882+0000","last_fullsized":"2024-08-19T12:57:34.740882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:21:51.938672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1387,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578717+0000","last_change":"2024-08-19T12:38:31.960408+0000","last_active":"2024-08-19T12:57:35.578717+0000","last_peered":"2024-08-19T12:57:35.578717+0000","last_clean":"2024-08-19T12:57:35.578717+0000","last_became_active":"2024-08-19T12:38:29.951331+0000","last_became_peered":"2024-08-19T12:38:29.951331+0000","last_unstale":"2024-08-19T12:57:35.578717+0000","last_undegraded":"2024-08-19T12:57:35.578717+0000","last_fullsized":"2024-08-19T12:57:35.578717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:55:28.246342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018789830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1368,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.581284+0000","last_change":"2024-08-19T12:39:38.296122+0000","last_active":"2024-08-19T12:57:35.581284+0000","last_peered":"2024-08-19T12:57:35.581284+0000","last_clean":"2024-08-19T12:57:35.581284+0000","last_became_active":"2024-08-19T12:39:38.295614+0000","last_became_peered":"2024-08-19T12:39:38.295614+0000","last_unstale":"2024-08-19T12:57:35.581284+0000","last_undegraded":"2024-08-19T12:57:35.581284+0000","last_fullsized":"2024-08-19T12:57:35.581284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:57:09.397165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1385,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579218+0000","last_change":"2024-08-19T12:38:31.957412+0000","last_active":"2024-08-19T12:57:35.579218+0000","last_peered":"2024-08-19T12:57:35.579218+0000","last_clean":"2024-08-19T12:57:35.579218+0000","last_became_active":"2024-08-19T12:38:29.952459+0000","last_became_peered":"2024-08-19T12:38:29.952459+0000","last_unstale":"2024-08-19T12:57:35.579218+0000","last_undegraded":"2024-08-19T12:57:35.579218+0000","last_fullsized":"2024-08-19T12:57:35.579218+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:47:27.910005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039863799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740798+0000","last_change":"2024-08-19T12:39:38.224517+0000","last_active":"2024-08-19T12:57:34.740798+0000","last_peered":"2024-08-19T12:57:34.740798+0000","last_clean":"2024-08-19T12:57:34.740798+0000","last_became_active":"2024-08-19T12:39:38.218230+0000","last_became_peered":"2024-08-19T12:39:38.218230+0000","last_unstale":"2024-08-19T12:57:34.740798+0000","last_undegraded":"2024-08-19T12:57:34.740798+0000","last_fullsized":"2024-08-19T12:57:34.740798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:20:03.811299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1376,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T12:57:32.557730+0000","last_change":"2024-08-19T12:38:31.959929+0000","last_active":"2024-08-19T12:57:32.557730+0000","last_peered":"2024-08-19T12:57:32.557730+0000","last_clean":"2024-08-19T12:57:32.557730+0000","last_became_active":"2024-08-19T12:38:29.951392+0000","last_became_peered":"2024-08-19T12:38:29.951392+0000","last_unstale":"2024-08-19T12:57:32.557730+0000","last_undegraded":"2024-08-19T12:57:32.557730+0000","last_fullsized":"2024-08-19T12:57:32.557730+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:47:44.756214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056602099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1371,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578919+0000","last_change":"2024-08-19T12:39:38.220796+0000","last_active":"2024-08-19T12:57:35.578919+0000","last_peered":"2024-08-19T12:57:35.578919+0000","last_clean":"2024-08-19T12:57:35.578919+0000","last_became_active":"2024-08-19T12:39:38.217569+0000","last_became_peered":"2024-08-19T12:39:38.217569+0000","last_unstale":"2024-08-19T12:57:35.578919+0000","last_undegraded":"2024-08-19T12:57:35.578919+0000","last_fullsized":"2024-08-19T12:57:35.578919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:14:24.659508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1374,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861595+0000","last_change":"2024-08-19T12:38:31.963352+0000","last_active":"2024-08-19T12:57:35.861595+0000","last_peered":"2024-08-19T12:57:35.861595+0000","last_clean":"2024-08-19T12:57:35.861595+0000","last_became_active":"2024-08-19T12:38:29.951233+0000","last_became_peered":"2024-08-19T12:38:29.951233+0000","last_unstale":"2024-08-19T12:57:35.861595+0000","last_undegraded":"2024-08-19T12:57:35.861595+0000","last_fullsized":"2024-08-19T12:57:35.861595+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:45:51.806389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048735899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"22'7","reported_seq":1381,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.581124+0000","last_change":"2024-08-19T12:39:38.270101+0000","last_active":"2024-08-19T12:57:35.581124+0000","last_peered":"2024-08-19T12:57:35.581124+0000","last_clean":"2024-08-19T12:57:35.581124+0000","last_became_active":"2024-08-19T12:39:38.269442+0000","last_became_peered":"2024-08-19T12:39:38.269442+0000","last_unstale":"2024-08-19T12:57:35.581124+0000","last_undegraded":"2024-08-19T12:57:35.581124+0000","last_fullsized":"2024-08-19T12:57:35.581124+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:02:35.119493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1387,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578970+0000","last_change":"2024-08-19T12:38:31.960297+0000","last_active":"2024-08-19T12:57:35.578970+0000","last_peered":"2024-08-19T12:57:35.578970+0000","last_clean":"2024-08-19T12:57:35.578970+0000","last_became_active":"2024-08-19T12:38:29.949846+0000","last_became_peered":"2024-08-19T12:38:29.949846+0000","last_unstale":"2024-08-19T12:57:35.578970+0000","last_undegraded":"2024-08-19T12:57:35.578970+0000","last_fullsized":"2024-08-19T12:57:35.578970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:34:31.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001822192,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1358,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T12:57:32.557275+0000","last_change":"2024-08-19T12:39:38.181922+0000","last_active":"2024-08-19T12:57:32.557275+0000","last_peered":"2024-08-19T12:57:32.557275+0000","last_clean":"2024-08-19T12:57:32.557275+0000","last_became_active":"2024-08-19T12:39:38.166658+0000","last_became_peered":"2024-08-19T12:39:38.166658+0000","last_unstale":"2024-08-19T12:57:32.557275+0000","last_undegraded":"2024-08-19T12:57:32.557275+0000","last_fullsized":"2024-08-19T12:57:32.557275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:21:22.453156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1373,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861373+0000","last_change":"2024-08-19T12:38:31.963379+0000","last_active":"2024-08-19T12:57:35.861373+0000","last_peered":"2024-08-19T12:57:35.861373+0000","last_clean":"2024-08-19T12:57:35.861373+0000","last_became_active":"2024-08-19T12:38:29.950467+0000","last_became_peered":"2024-08-19T12:38:29.950467+0000","last_unstale":"2024-08-19T12:57:35.861373+0000","last_undegraded":"2024-08-19T12:57:35.861373+0000","last_fullsized":"2024-08-19T12:57:35.861373+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:40:07.574639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049350399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740778+0000","last_change":"2024-08-19T12:39:38.196037+0000","last_active":"2024-08-19T12:57:34.740778+0000","last_peered":"2024-08-19T12:57:34.740778+0000","last_clean":"2024-08-19T12:57:34.740778+0000","last_became_active":"2024-08-19T12:39:38.190786+0000","last_became_peered":"2024-08-19T12:39:38.190786+0000","last_unstale":"2024-08-19T12:57:34.740778+0000","last_undegraded":"2024-08-19T12:57:34.740778+0000","last_fullsized":"2024-08-19T12:57:34.740778+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:07:56.534454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1386,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578815+0000","last_change":"2024-08-19T12:38:31.959032+0000","last_active":"2024-08-19T12:57:35.578815+0000","last_peered":"2024-08-19T12:57:35.578815+0000","last_clean":"2024-08-19T12:57:35.578815+0000","last_became_active":"2024-08-19T12:38:29.948183+0000","last_became_peered":"2024-08-19T12:38:29.948183+0000","last_unstale":"2024-08-19T12:57:35.578815+0000","last_undegraded":"2024-08-19T12:57:35.578815+0000","last_fullsized":"2024-08-19T12:57:35.578815+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:51:55.545957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052740100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1369,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579281+0000","last_change":"2024-08-19T12:39:38.221167+0000","last_active":"2024-08-19T12:57:35.579281+0000","last_peered":"2024-08-19T12:57:35.579281+0000","last_clean":"2024-08-19T12:57:35.579281+0000","last_became_active":"2024-08-19T12:39:38.191158+0000","last_became_peered":"2024-08-19T12:39:38.191158+0000","last_unstale":"2024-08-19T12:57:35.579281+0000","last_undegraded":"2024-08-19T12:57:35.579281+0000","last_fullsized":"2024-08-19T12:57:35.579281+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T22:59:47.700950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1357,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861035+0000","last_change":"2024-08-19T12:39:38.194777+0000","last_active":"2024-08-19T12:57:35.861035+0000","last_peered":"2024-08-19T12:57:35.861035+0000","last_clean":"2024-08-19T12:57:35.861035+0000","last_became_active":"2024-08-19T12:39:38.161352+0000","last_became_peered":"2024-08-19T12:39:38.161352+0000","last_unstale":"2024-08-19T12:57:35.861035+0000","last_undegraded":"2024-08-19T12:57:35.861035+0000","last_fullsized":"2024-08-19T12:57:35.861035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:31:21.012106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578750+0000","last_change":"2024-08-19T12:39:38.210090+0000","last_active":"2024-08-19T12:57:35.578750+0000","last_peered":"2024-08-19T12:57:35.578750+0000","last_clean":"2024-08-19T12:57:35.578750+0000","last_became_active":"2024-08-19T12:39:38.202668+0000","last_became_peered":"2024-08-19T12:39:38.202668+0000","last_unstale":"2024-08-19T12:57:35.578750+0000","last_undegraded":"2024-08-19T12:57:35.578750+0000","last_fullsized":"2024-08-19T12:57:35.578750+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:58:28.292312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1357,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T12:57:32.557636+0000","last_change":"2024-08-19T12:39:38.224507+0000","last_active":"2024-08-19T12:57:32.557636+0000","last_peered":"2024-08-19T12:57:32.557636+0000","last_clean":"2024-08-19T12:57:32.557636+0000","last_became_active":"2024-08-19T12:39:38.222877+0000","last_became_peered":"2024-08-19T12:39:38.222877+0000","last_unstale":"2024-08-19T12:57:32.557636+0000","last_undegraded":"2024-08-19T12:57:32.557636+0000","last_fullsized":"2024-08-19T12:57:32.557636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:55:08.088675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740614+0000","last_change":"2024-08-19T12:39:38.211834+0000","last_active":"2024-08-19T12:57:34.740614+0000","last_peered":"2024-08-19T12:57:34.740614+0000","last_clean":"2024-08-19T12:57:34.740614+0000","last_became_active":"2024-08-19T12:39:38.210985+0000","last_became_peered":"2024-08-19T12:39:38.210985+0000","last_unstale":"2024-08-19T12:57:34.740614+0000","last_undegraded":"2024-08-19T12:57:34.740614+0000","last_fullsized":"2024-08-19T12:57:34.740614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:39:57.635965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578850+0000","last_change":"2024-08-19T12:39:38.268396+0000","last_active":"2024-08-19T12:57:35.578850+0000","last_peered":"2024-08-19T12:57:35.578850+0000","last_clean":"2024-08-19T12:57:35.578850+0000","last_became_active":"2024-08-19T12:39:38.267362+0000","last_became_peered":"2024-08-19T12:39:38.267362+0000","last_unstale":"2024-08-19T12:57:35.578850+0000","last_undegraded":"2024-08-19T12:57:35.578850+0000","last_fullsized":"2024-08-19T12:57:35.578850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:26:11.466849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1357,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861686+0000","last_change":"2024-08-19T12:39:38.195841+0000","last_active":"2024-08-19T12:57:35.861686+0000","last_peered":"2024-08-19T12:57:35.861686+0000","last_clean":"2024-08-19T12:57:35.861686+0000","last_became_active":"2024-08-19T12:39:38.173967+0000","last_became_peered":"2024-08-19T12:39:38.173967+0000","last_unstale":"2024-08-19T12:57:35.861686+0000","last_undegraded":"2024-08-19T12:57:35.861686+0000","last_fullsized":"2024-08-19T12:57:35.861686+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:29:39.717188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1359,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861315+0000","last_change":"2024-08-19T12:39:38.110553+0000","last_active":"2024-08-19T12:57:35.861315+0000","last_peered":"2024-08-19T12:57:35.861315+0000","last_clean":"2024-08-19T12:57:35.861315+0000","last_became_active":"2024-08-19T12:39:38.108670+0000","last_became_peered":"2024-08-19T12:39:38.108670+0000","last_unstale":"2024-08-19T12:57:35.861315+0000","last_undegraded":"2024-08-19T12:57:35.861315+0000","last_fullsized":"2024-08-19T12:57:35.861315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:32:52.104320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1358,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-08-19T12:57:32.558370+0000","last_change":"2024-08-19T12:39:38.200400+0000","last_active":"2024-08-19T12:57:32.558370+0000","last_peered":"2024-08-19T12:57:32.558370+0000","last_clean":"2024-08-19T12:57:32.558370+0000","last_became_active":"2024-08-19T12:39:38.197445+0000","last_became_peered":"2024-08-19T12:39:38.197445+0000","last_unstale":"2024-08-19T12:57:32.558370+0000","last_undegraded":"2024-08-19T12:57:32.558370+0000","last_fullsized":"2024-08-19T12:57:32.558370+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:21:39.167660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740451+0000","last_change":"2024-08-19T12:39:38.274353+0000","last_active":"2024-08-19T12:57:34.740451+0000","last_peered":"2024-08-19T12:57:34.740451+0000","last_clean":"2024-08-19T12:57:34.740451+0000","last_became_active":"2024-08-19T12:39:38.267711+0000","last_became_peered":"2024-08-19T12:39:38.267711+0000","last_unstale":"2024-08-19T12:57:34.740451+0000","last_undegraded":"2024-08-19T12:57:34.740451+0000","last_fullsized":"2024-08-19T12:57:34.740451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:56:56.848415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578730+0000","last_change":"2024-08-19T12:39:38.188765+0000","last_active":"2024-08-19T12:57:35.578730+0000","last_peered":"2024-08-19T12:57:35.578730+0000","last_clean":"2024-08-19T12:57:35.578730+0000","last_became_active":"2024-08-19T12:39:38.177735+0000","last_became_peered":"2024-08-19T12:39:38.177735+0000","last_unstale":"2024-08-19T12:57:35.578730+0000","last_undegraded":"2024-08-19T12:57:35.578730+0000","last_fullsized":"2024-08-19T12:57:35.578730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:38:45.637343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578720+0000","last_change":"2024-08-19T12:39:38.200875+0000","last_active":"2024-08-19T12:57:35.578720+0000","last_peered":"2024-08-19T12:57:35.578720+0000","last_clean":"2024-08-19T12:57:35.578720+0000","last_became_active":"2024-08-19T12:39:38.199715+0000","last_became_peered":"2024-08-19T12:39:38.199715+0000","last_unstale":"2024-08-19T12:57:35.578720+0000","last_undegraded":"2024-08-19T12:57:35.578720+0000","last_fullsized":"2024-08-19T12:57:35.578720+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T22:27:29.457630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740661+0000","last_change":"2024-08-19T12:39:38.196277+0000","last_active":"2024-08-19T12:57:34.740661+0000","last_peered":"2024-08-19T12:57:34.740661+0000","last_clean":"2024-08-19T12:57:34.740661+0000","last_became_active":"2024-08-19T12:39:38.190871+0000","last_became_peered":"2024-08-19T12:39:38.190871+0000","last_unstale":"2024-08-19T12:57:34.740661+0000","last_undegraded":"2024-08-19T12:57:34.740661+0000","last_fullsized":"2024-08-19T12:57:34.740661+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:45:27.127112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68660,"kb_used_data":5536,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303180,"statfs":{"total":96636764160,"available":96566456320,"internally_reserved":0,"allocated":5668864,"data_stored":2406798,"data_compressed":43,"data_compressed_allocated":4096,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607786,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68704,"kb_used_data":5572,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303136,"statfs":{"total":96636764160,"available":96566411264,"internally_reserved":0,"allocated":5705728,"data_stored":2348545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607789,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68752,"kb_used_data":5620,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303088,"statfs":{"total":96636764160,"available":96566362112,"internally_reserved":0,"allocated":5754880,"data_stored":2877468,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1592,"internal_metadata":64616904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607785,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77020,"kb_used_data":5568,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294820,"statfs":{"total":96636764160,"available":96557895680,"internally_reserved":0,"allocated":5701632,"data_stored":2348540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607786,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77016,"kb_used_data":5572,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294824,"statfs":{"total":96636764160,"available":96557899776,"internally_reserved":0,"allocated":5705728,"data_stored":2348559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640491,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81220,"kb_used_data":5616,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290620,"statfs":{"total":96636764160,"available":96553594880,"internally_reserved":0,"allocated":5750784,"data_stored":2897162,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T12:57:38.758 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-19T12:57:38.758 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T12:57:38.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.856+0000 7f4188516640 10 monclient: get_monmap_and_config 2024-08-19T12:57:38.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.856+0000 7f4188516640 10 monclient: build_initial_monmap 2024-08-19T12:57:38.857 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.856+0000 7f4188516640 10 monclient: monmap: 2024-08-19T12:57:38.857 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:38.857 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.857 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:38.857242+0000 2024-08-19T12:57:38.857 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:38.857242+0000 2024-08-19T12:57:38.857 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:38.857 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.857 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:38.858 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:38.858 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:38.858 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:38.858 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:38.858 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:38.858 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:38.858 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:38.858 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:38.858 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.859 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.856+0000 7f4188516640 1 Processor -- start 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4188516640 1 -- start start 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4188516640 10 monclient: init 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4188516640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4188516640 10 monclient: _add_conns ranks=[0,3,1,8,5,7,4,6,2] 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4188516640 10 monclient(hunting): picked mon.noname-a con 0x7f418006e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4188516640 10 monclient(hunting): picked mon.noname-g con 0x7f41800a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4188516640 10 monclient(hunting): picked mon.noname-f con 0x7f41800a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4188516640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4180136790 con 0x7f418006e7b0 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4188516640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418009ddf0 con 0x7f41800a5d50 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4188516640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418009e0d0 con 0x7f41800a5920 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4188516640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4188516640 10 monclient(hunting): authenticate will time out at 1852.944635s 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f41862b2640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f418006e7b0 legacy=0x7f4180136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:57:38.860 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4185ab1640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f41800a5920 legacy=0x7f418013e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:38.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4186ab3640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f41800a5d50 0x7f4180141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:58322/0 (socket says 172.21.15.26:58322) 2024-08-19T12:57:38.861 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.857+0000 7f4186ab3640 1 -- 172.21.15.26:0/2288065919 learned_addr learned my addr 172.21.15.26:0/2288065919 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:38.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f4186ab3640 1 -- 172.21.15.26:0/2288065919 >> v1:172.21.15.104:6789/0 conn(0x7f41800a5d50 legacy=0x7f4180141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:38.864 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 1 -- 172.21.15.26:0/2288065919 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2367623430 0 0) 0x7f41700017d0 con 0x7f41800a5d50 2024-08-19T12:57:38.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient(hunting): _negotiate 2024-08-19T12:57:38.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:38.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:38.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient(hunting): my global_id is 204364 2024-08-19T12:57:38.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 1 -- 172.21.15.26:0/2288065919 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4168003520 con 0x7f41800a5d50 2024-08-19T12:57:38.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 1 -- 172.21.15.26:0/2288065919 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 806148701 0 0) 0x7f4168003520 con 0x7f41800a5d50 2024-08-19T12:57:38.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 1 -- 172.21.15.26:0/2288065919 >> v1:172.21.15.104:6790/0 conn(0x7f41800a5920 legacy=0x7f418013e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:38.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 1 -- 172.21.15.26:0/2288065919 >> v1:172.21.15.26:6789/0 conn(0x7f418006e7b0 legacy=0x7f4180136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:38.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient: _finish_hunting 0 2024-08-19T12:57:38.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 1 monclient: found mon.noname-f 2024-08-19T12:57:38.865 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:38.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.104:6789/0 2024-08-19T12:57:38.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 1 -- 172.21.15.26:0/2288065919 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418009e3b0 con 0x7f41800a5d50 2024-08-19T12:57:38.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient: _finish_auth 0 2024-08-19T12:57:38.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient: _check_auth_tickets 2024-08-19T12:57:38.866 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:38.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 1 -- 172.21.15.26:0/2288065919 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4170004c20 con 0x7f41800a5d50 2024-08-19T12:57:38.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f4188516640 5 monclient: authenticate success, global_id 204364 2024-08-19T12:57:38.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f4188516640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:38.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:38.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-08-19T12:57:38.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient: dump: 2024-08-19T12:57:38.867 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.867 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.867 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.868 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f4188516640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 1 -- 172.21.15.26:0/2288065919 <== mon.1 v1:172.21.15.104:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4170004ec0 con 0x7f41800a5d50 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 1 -- 172.21.15.26:0/2288065919 <== mon.1 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f41700053f0 con 0x7f41800a5d50 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f4188516640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f4188516640 4 set_mon_vals no callback set 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.860+0000 7f41852b0640 10 monclient: dump: 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.869 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.861+0000 7f4188516640 10 monclient: shutdown 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.861+0000 7f4188516640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.861+0000 7f4188516640 1 -- 172.21.15.26:0/2288065919 >> v1:172.21.15.104:6789/0 conn(0x7f41800a5d50 legacy=0x7f4180141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.861+0000 7f4188516640 1 -- 172.21.15.26:0/2288065919 shutdown_connections 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.861+0000 7f4188516640 1 -- 172.21.15.26:0/2288065919 >> 172.21.15.26:0/2288065919 conn(0x7f41800a8740 msgr2=0x7f41800a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.861+0000 7f4188516640 1 -- 172.21.15.26:0/2288065919 shutdown_connections 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.861+0000 7f4188516640 1 -- 172.21.15.26:0/2288065919 wait complete. 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.861+0000 7f4188516640 10 monclient: build_initial_monmap 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.861+0000 7f4188516640 10 monclient: monmap: 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.871 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.861+0000 7f4188516640 1 librados: starting msgr at 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.861+0000 7f4188516640 1 librados: starting objecter 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 1 Processor -- start 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 1 -- start start 2024-08-19T12:57:38.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 1 librados: setting wanted keys 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 1 librados: calling monclient init 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 10 monclient: init 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 10 monclient: _add_conns ranks=[3,6,7,8,1,2,5,4,0] 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 10 monclient(hunting): picked mon.noname-g con 0x7f418006e7b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 10 monclient(hunting): picked mon.noname-d con 0x7f41800a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 10 monclient(hunting): picked mon.noname-i con 0x7f41800a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41800a0e60 con 0x7f418006e7b0 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4180142e30 con 0x7f41800a5920 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4185ab1640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f41800a5920 legacy=0x7f4180132960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418009f660 con 0x7f41800a5d50 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4188516640 10 monclient(hunting): authenticate will time out at 1852.949691s 2024-08-19T12:57:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f41862b2640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f418006e7b0 legacy=0x7f4180132250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:38.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4186ab3640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f41800a5d50 0x7f41801d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:42042/0 (socket says 172.21.15.26:42042) 2024-08-19T12:57:38.874 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.862+0000 7f4186ab3640 1 -- 172.21.15.26:0/4047206145 learned_addr learned my addr 172.21.15.26:0/4047206145 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:38.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f4186ab3640 1 -- 172.21.15.26:0/4047206145 >> v1:172.21.15.104:6792/0 conn(0x7f41800a5d50 legacy=0x7f41801d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:38.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 client.?.objecter ms_handle_connect 0x7f41800a5d50 2024-08-19T12:57:38.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:38.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3042621624 0 0) 0x7f418009f660 con 0x7f41800a5d50 2024-08-19T12:57:38.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient(hunting): _negotiate 2024-08-19T12:57:38.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient(hunting): my global_id is 136546 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f416c004eb0 con 0x7f41800a5d50 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4223566397 0 0) 0x7f416c004eb0 con 0x7f41800a5d50 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f418009f660 con 0x7f41800a5d50 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f41700020b0 con 0x7f41800a5d50 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient(hunting): dump: 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.878 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.879 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 725450906 0 0) 0x7f418009f660 con 0x7f41800a5d50 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 >> v1:172.21.15.26:6792/0 conn(0x7f41800a5920 legacy=0x7f4180132960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 >> v1:172.21.15.104:6790/0 conn(0x7f418006e7b0 legacy=0x7f4180132250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient: _finish_hunting 0 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 1 monclient: found mon.i 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418009eb00 con 0x7f41800a5d50 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient: _finish_auth 0 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient: _check_auth_tickets 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f4188516640 5 monclient: authenticate success, global_id 136546 2024-08-19T12:57:38.880 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f4188516640 10 monclient: _renew_subs 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f4188516640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f4188516640 1 -- 172.21.15.26:0/4047206145 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f41800a0b80 con 0x7f41800a5d50 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f4188516640 10 client.136546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f4188516640 10 monclient: _renew_subs 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f4188516640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f4188516640 1 -- 172.21.15.26:0/4047206145 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f4180133070 con 0x7f41800a5d50 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f4188516640 1 librados: init done 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4170005320 con 0x7f41800a5d50 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f4170005810 con 0x7f41800a5d50 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f4184aaf640 4 set_mon_vals no callback set 2024-08-19T12:57:38.881 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.865+0000 7f417effd640 10 monclient: dump: 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:38.882 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.866+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f4170011c70 con 0x7f41800a5d50 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.866+0000 7f4188516640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.866+0000 7f4188516640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.866+0000 7f4188516640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.866+0000 7f4188516640 1 -- 172.21.15.26:0/4047206145 --> v1:172.21.15.104:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4160005080 con 0x7f41800a5d50 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.867+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7f417004e2c0 con 0x7f41800a5d50 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.867+0000 7f417effd640 10 client.136546.objecter ms_dispatch 0x7f4180000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.867+0000 7f417effd640 3 client.136546.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.867+0000 7f417effd640 3 client.136546.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:38.883 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.867+0000 7f417effd640 20 client.136546.objecter dump_active .. 0 homeless 2024-08-19T12:57:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.867+0000 7f41862b2640 10 monclient: get_auth_request con 0x7f416c040e10 auth_method 0 2024-08-19T12:57:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.869+0000 7f417effd640 10 client.136546.objecter ms_handle_connect 0x7f416c040e10 2024-08-19T12:57:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.872+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 <== mon.7 v1:172.21.15.104:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f418009ddf0 con 0x7f41800a5d50 2024-08-19T12:57:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.872+0000 7f417effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:38.884 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:38.872+0000 7f417effd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:57:39.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.052+0000 7f4188516640 1 -- 172.21.15.26:0/4047206145 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4160000b90 con 0x7f416c040e10 2024-08-19T12:57:39.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.052+0000 7f41862b2640 1 -- 172.21.15.26:0/4047206145 >> v1:172.21.15.104:6800/899887506 conn(0x7f416c040e10 legacy=0x7f416c0432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-08-19T12:57:39.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.052+0000 7f41862b2640 1 -- 172.21.15.26:0/4047206145 >> v1:172.21.15.104:6800/899887506 conn(0x7f416c040e10 legacy=0x7f416c0432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:57:39.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.052+0000 7f41862b2640 1 --1- 172.21.15.26:0/4047206145 >> v1:172.21.15.104:6800/899887506 conn(0x7f416c040e10 0x7f416c0432d0 :-1 s=OPENED pgs=4192 cs=1 l=1).handle_message read tag failed 2024-08-19T12:57:39.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.052+0000 7f41862b2640 1 --1- 172.21.15.26:0/4047206145 >> v1:172.21.15.104:6800/899887506 conn(0x7f416c040e10 0x7f416c0432d0 :-1 s=OPENED pgs=4192 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:57:39.053 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.052+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 >> v1:172.21.15.104:6800/899887506 conn(0x7f416c040e10 legacy=0x7f416c0432d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:57:39.867 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.867+0000 7f417d7fa640 1 -- 172.21.15.26:0/4047206145 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4164006c70 con 0x7f416c040e10 2024-08-19T12:57:39.868 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.867+0000 7f41862b2640 10 monclient: get_auth_request con 0x7f416c040e10 auth_method 0 2024-08-19T12:57:39.870 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.870+0000 7f417effd640 10 client.136546.objecter ms_handle_connect 0x7f416c040e10 2024-08-19T12:57:39.872 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.872+0000 7f417effd640 1 -- 172.21.15.26:0/4047206145 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114243 (unknown 2965378022 0 4010058935) 0x7f4164006c70 con 0x7f416c040e10 2024-08-19T12:57:39.873 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:57:39.874 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-08-19T12:57:39.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 10 librados: watch_flush enter 2024-08-19T12:57:39.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 10 librados: watch_flush exit 2024-08-19T12:57:39.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 20 client.136546.objecter shutdown clearing up homeless session... 2024-08-19T12:57:39.876 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 10 client.136546.objecter successfully canceled tick 2024-08-19T12:57:39.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 1 -- 172.21.15.26:0/4047206145 >> v1:172.21.15.104:6800/899887506 conn(0x7f416c040e10 legacy=0x7f4164005530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:39.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 10 monclient: shutdown 2024-08-19T12:57:39.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:39.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 1 -- 172.21.15.26:0/4047206145 >> v1:172.21.15.104:6792/0 conn(0x7f41800a5d50 legacy=0x7f41801d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:39.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 1 -- 172.21.15.26:0/4047206145 shutdown_connections 2024-08-19T12:57:39.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 1 -- 172.21.15.26:0/4047206145 >> 172.21.15.26:0/4047206145 conn(0x7f41800a8740 msgr2=0x7f418006d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:39.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 1 -- 172.21.15.26:0/4047206145 shutdown_connections 2024-08-19T12:57:39.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 1 -- 172.21.15.26:0/4047206145 wait complete. 2024-08-19T12:57:39.877 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.875+0000 7f4188516640 1 librados: shutdown 2024-08-19T12:57:39.888 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1267,"stamp":"2024-08-19T12:57:39.386833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":441408,"kb_used_data":33520,"kb_used_omap":9,"kb_used_meta":407734,"kb_avail":565789632,"statfs":{"total":579820584960,"available":579368583168,"internally_reserved":0,"allocated":34324480,"data_stored":15168833,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9566,"internal_metadata":417520290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.822854"},"pg_stats":[{"pgid":"7.17","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.418390+0000","last_change":"2024-08-19T12:39:38.156896+0000","last_active":"2024-08-19T12:57:37.418390+0000","last_peered":"2024-08-19T12:57:37.418390+0000","last_clean":"2024-08-19T12:57:37.418390+0000","last_became_active":"2024-08-19T12:39:38.143851+0000","last_became_peered":"2024-08-19T12:39:38.143851+0000","last_unstale":"2024-08-19T12:57:37.418390+0000","last_undegraded":"2024-08-19T12:57:37.418390+0000","last_fullsized":"2024-08-19T12:57:37.418390+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:32:37.888575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1367,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.581141+0000","last_change":"2024-08-19T12:39:38.273079+0000","last_active":"2024-08-19T12:57:35.581141+0000","last_peered":"2024-08-19T12:57:35.581141+0000","last_clean":"2024-08-19T12:57:35.581141+0000","last_became_active":"2024-08-19T12:39:38.269168+0000","last_became_peered":"2024-08-19T12:39:38.269168+0000","last_unstale":"2024-08-19T12:57:35.581141+0000","last_undegraded":"2024-08-19T12:57:35.581141+0000","last_fullsized":"2024-08-19T12:57:35.581141+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:27:10.963118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578862+0000","last_change":"2024-08-19T12:39:38.205974+0000","last_active":"2024-08-19T12:57:35.578862+0000","last_peered":"2024-08-19T12:57:35.578862+0000","last_clean":"2024-08-19T12:57:35.578862+0000","last_became_active":"2024-08-19T12:39:38.201163+0000","last_became_peered":"2024-08-19T12:39:38.201163+0000","last_unstale":"2024-08-19T12:57:35.578862+0000","last_undegraded":"2024-08-19T12:57:35.578862+0000","last_fullsized":"2024-08-19T12:57:35.578862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:33:44.282679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1357,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861615+0000","last_change":"2024-08-19T12:39:38.230576+0000","last_active":"2024-08-19T12:57:35.861615+0000","last_peered":"2024-08-19T12:57:35.861615+0000","last_clean":"2024-08-19T12:57:35.861615+0000","last_became_active":"2024-08-19T12:39:38.183511+0000","last_became_peered":"2024-08-19T12:39:38.183511+0000","last_unstale":"2024-08-19T12:57:35.861615+0000","last_undegraded":"2024-08-19T12:57:35.861615+0000","last_fullsized":"2024-08-19T12:57:35.861615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:13:04.215179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1358,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861477+0000","last_change":"2024-08-19T12:39:38.095466+0000","last_active":"2024-08-19T12:57:35.861477+0000","last_peered":"2024-08-19T12:57:35.861477+0000","last_clean":"2024-08-19T12:57:35.861477+0000","last_became_active":"2024-08-19T12:39:38.094806+0000","last_became_peered":"2024-08-19T12:39:38.094806+0000","last_unstale":"2024-08-19T12:57:35.861477+0000","last_undegraded":"2024-08-19T12:57:35.861477+0000","last_fullsized":"2024-08-19T12:57:35.861477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:03:19.792598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1369,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579208+0000","last_change":"2024-08-19T12:39:38.222591+0000","last_active":"2024-08-19T12:57:35.579208+0000","last_peered":"2024-08-19T12:57:35.579208+0000","last_clean":"2024-08-19T12:57:35.579208+0000","last_became_active":"2024-08-19T12:39:38.219371+0000","last_became_peered":"2024-08-19T12:39:38.219371+0000","last_unstale":"2024-08-19T12:57:35.579208+0000","last_undegraded":"2024-08-19T12:57:35.579208+0000","last_fullsized":"2024-08-19T12:57:35.579208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:17:01.926550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740597+0000","last_change":"2024-08-19T12:39:38.237985+0000","last_active":"2024-08-19T12:57:34.740597+0000","last_peered":"2024-08-19T12:57:34.740597+0000","last_clean":"2024-08-19T12:57:34.740597+0000","last_became_active":"2024-08-19T12:39:38.228951+0000","last_became_peered":"2024-08-19T12:39:38.228951+0000","last_unstale":"2024-08-19T12:57:34.740597+0000","last_undegraded":"2024-08-19T12:57:34.740597+0000","last_fullsized":"2024-08-19T12:57:34.740597+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:47:35.671835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1369,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579208+0000","last_change":"2024-08-19T12:39:38.302683+0000","last_active":"2024-08-19T12:57:35.579208+0000","last_peered":"2024-08-19T12:57:35.579208+0000","last_clean":"2024-08-19T12:57:35.579208+0000","last_became_active":"2024-08-19T12:39:38.301100+0000","last_became_peered":"2024-08-19T12:39:38.301100+0000","last_unstale":"2024-08-19T12:57:35.579208+0000","last_undegraded":"2024-08-19T12:57:35.579208+0000","last_fullsized":"2024-08-19T12:57:35.579208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:02:38.662623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740971+0000","last_change":"2024-08-19T12:39:38.152043+0000","last_active":"2024-08-19T12:57:34.740971+0000","last_peered":"2024-08-19T12:57:34.740971+0000","last_clean":"2024-08-19T12:57:34.740971+0000","last_became_active":"2024-08-19T12:39:38.149472+0000","last_became_peered":"2024-08-19T12:39:38.149472+0000","last_unstale":"2024-08-19T12:57:34.740971+0000","last_undegraded":"2024-08-19T12:57:34.740971+0000","last_fullsized":"2024-08-19T12:57:34.740971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:03:37.142791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1369,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579403+0000","last_change":"2024-08-19T12:39:38.184326+0000","last_active":"2024-08-19T12:57:35.579403+0000","last_peered":"2024-08-19T12:57:35.579403+0000","last_clean":"2024-08-19T12:57:35.579403+0000","last_became_active":"2024-08-19T12:39:38.181280+0000","last_became_peered":"2024-08-19T12:39:38.181280+0000","last_unstale":"2024-08-19T12:57:35.579403+0000","last_undegraded":"2024-08-19T12:57:35.579403+0000","last_fullsized":"2024-08-19T12:57:35.579403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:00:41.274939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1368,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.581234+0000","last_change":"2024-08-19T12:39:38.154946+0000","last_active":"2024-08-19T12:57:35.581234+0000","last_peered":"2024-08-19T12:57:35.581234+0000","last_clean":"2024-08-19T12:57:35.581234+0000","last_became_active":"2024-08-19T12:39:38.147647+0000","last_became_peered":"2024-08-19T12:39:38.147647+0000","last_unstale":"2024-08-19T12:57:35.581234+0000","last_undegraded":"2024-08-19T12:57:35.581234+0000","last_fullsized":"2024-08-19T12:57:35.581234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:24:26.064970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1371,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578991+0000","last_change":"2024-08-19T12:39:38.137671+0000","last_active":"2024-08-19T12:57:35.578991+0000","last_peered":"2024-08-19T12:57:35.578991+0000","last_clean":"2024-08-19T12:57:35.578991+0000","last_became_active":"2024-08-19T12:39:38.135334+0000","last_became_peered":"2024-08-19T12:39:38.135334+0000","last_unstale":"2024-08-19T12:57:35.578991+0000","last_undegraded":"2024-08-19T12:57:35.578991+0000","last_fullsized":"2024-08-19T12:57:35.578991+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:37:29.348405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1478,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578723+0000","last_change":"2024-08-19T12:38:22.728884+0000","last_active":"2024-08-19T12:57:35.578723+0000","last_peered":"2024-08-19T12:57:35.578723+0000","last_clean":"2024-08-19T12:57:35.578723+0000","last_became_active":"2024-08-19T12:38:22.728595+0000","last_became_peered":"2024-08-19T12:38:22.728595+0000","last_unstale":"2024-08-19T12:57:35.578723+0000","last_undegraded":"2024-08-19T12:57:35.578723+0000","last_fullsized":"2024-08-19T12:57:35.578723+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_clean_scrub_stamp":"2024-08-19T12:38:21.711006+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:42:50.800603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1383,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.418232+0000","last_change":"2024-08-19T12:38:31.959938+0000","last_active":"2024-08-19T12:57:37.418232+0000","last_peered":"2024-08-19T12:57:37.418232+0000","last_clean":"2024-08-19T12:57:37.418232+0000","last_became_active":"2024-08-19T12:38:29.952471+0000","last_became_peered":"2024-08-19T12:38:29.952471+0000","last_unstale":"2024-08-19T12:57:37.418232+0000","last_undegraded":"2024-08-19T12:57:37.418232+0000","last_fullsized":"2024-08-19T12:57:37.418232+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:48:18.927151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057881800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740882+0000","last_change":"2024-08-19T12:39:38.261181+0000","last_active":"2024-08-19T12:57:34.740882+0000","last_peered":"2024-08-19T12:57:34.740882+0000","last_clean":"2024-08-19T12:57:34.740882+0000","last_became_active":"2024-08-19T12:39:38.260911+0000","last_became_peered":"2024-08-19T12:39:38.260911+0000","last_unstale":"2024-08-19T12:57:34.740882+0000","last_undegraded":"2024-08-19T12:57:34.740882+0000","last_fullsized":"2024-08-19T12:57:34.740882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:21:51.938672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1387,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578717+0000","last_change":"2024-08-19T12:38:31.960408+0000","last_active":"2024-08-19T12:57:35.578717+0000","last_peered":"2024-08-19T12:57:35.578717+0000","last_clean":"2024-08-19T12:57:35.578717+0000","last_became_active":"2024-08-19T12:38:29.951331+0000","last_became_peered":"2024-08-19T12:38:29.951331+0000","last_unstale":"2024-08-19T12:57:35.578717+0000","last_undegraded":"2024-08-19T12:57:35.578717+0000","last_fullsized":"2024-08-19T12:57:35.578717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:55:28.246342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018789830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1368,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.581284+0000","last_change":"2024-08-19T12:39:38.296122+0000","last_active":"2024-08-19T12:57:35.581284+0000","last_peered":"2024-08-19T12:57:35.581284+0000","last_clean":"2024-08-19T12:57:35.581284+0000","last_became_active":"2024-08-19T12:39:38.295614+0000","last_became_peered":"2024-08-19T12:39:38.295614+0000","last_unstale":"2024-08-19T12:57:35.581284+0000","last_undegraded":"2024-08-19T12:57:35.581284+0000","last_fullsized":"2024-08-19T12:57:35.581284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:57:09.397165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1385,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579218+0000","last_change":"2024-08-19T12:38:31.957412+0000","last_active":"2024-08-19T12:57:35.579218+0000","last_peered":"2024-08-19T12:57:35.579218+0000","last_clean":"2024-08-19T12:57:35.579218+0000","last_became_active":"2024-08-19T12:38:29.952459+0000","last_became_peered":"2024-08-19T12:38:29.952459+0000","last_unstale":"2024-08-19T12:57:35.579218+0000","last_undegraded":"2024-08-19T12:57:35.579218+0000","last_fullsized":"2024-08-19T12:57:35.579218+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:47:27.910005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039863799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740798+0000","last_change":"2024-08-19T12:39:38.224517+0000","last_active":"2024-08-19T12:57:34.740798+0000","last_peered":"2024-08-19T12:57:34.740798+0000","last_clean":"2024-08-19T12:57:34.740798+0000","last_became_active":"2024-08-19T12:39:38.218230+0000","last_became_peered":"2024-08-19T12:39:38.218230+0000","last_unstale":"2024-08-19T12:57:34.740798+0000","last_undegraded":"2024-08-19T12:57:34.740798+0000","last_fullsized":"2024-08-19T12:57:34.740798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:20:03.811299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1384,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.418296+0000","last_change":"2024-08-19T12:38:31.959929+0000","last_active":"2024-08-19T12:57:37.418296+0000","last_peered":"2024-08-19T12:57:37.418296+0000","last_clean":"2024-08-19T12:57:37.418296+0000","last_became_active":"2024-08-19T12:38:29.951392+0000","last_became_peered":"2024-08-19T12:38:29.951392+0000","last_unstale":"2024-08-19T12:57:37.418296+0000","last_undegraded":"2024-08-19T12:57:37.418296+0000","last_fullsized":"2024-08-19T12:57:37.418296+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:47:44.756214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056602099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1371,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578919+0000","last_change":"2024-08-19T12:39:38.220796+0000","last_active":"2024-08-19T12:57:35.578919+0000","last_peered":"2024-08-19T12:57:35.578919+0000","last_clean":"2024-08-19T12:57:35.578919+0000","last_became_active":"2024-08-19T12:39:38.217569+0000","last_became_peered":"2024-08-19T12:39:38.217569+0000","last_unstale":"2024-08-19T12:57:35.578919+0000","last_undegraded":"2024-08-19T12:57:35.578919+0000","last_fullsized":"2024-08-19T12:57:35.578919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:14:24.659508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1374,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861595+0000","last_change":"2024-08-19T12:38:31.963352+0000","last_active":"2024-08-19T12:57:35.861595+0000","last_peered":"2024-08-19T12:57:35.861595+0000","last_clean":"2024-08-19T12:57:35.861595+0000","last_became_active":"2024-08-19T12:38:29.951233+0000","last_became_peered":"2024-08-19T12:38:29.951233+0000","last_unstale":"2024-08-19T12:57:35.861595+0000","last_undegraded":"2024-08-19T12:57:35.861595+0000","last_fullsized":"2024-08-19T12:57:35.861595+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:45:51.806389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048735899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"22'7","reported_seq":1381,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.581124+0000","last_change":"2024-08-19T12:39:38.270101+0000","last_active":"2024-08-19T12:57:35.581124+0000","last_peered":"2024-08-19T12:57:35.581124+0000","last_clean":"2024-08-19T12:57:35.581124+0000","last_became_active":"2024-08-19T12:39:38.269442+0000","last_became_peered":"2024-08-19T12:39:38.269442+0000","last_unstale":"2024-08-19T12:57:35.581124+0000","last_undegraded":"2024-08-19T12:57:35.581124+0000","last_fullsized":"2024-08-19T12:57:35.581124+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:02:35.119493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1387,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578970+0000","last_change":"2024-08-19T12:38:31.960297+0000","last_active":"2024-08-19T12:57:35.578970+0000","last_peered":"2024-08-19T12:57:35.578970+0000","last_clean":"2024-08-19T12:57:35.578970+0000","last_became_active":"2024-08-19T12:38:29.949846+0000","last_became_peered":"2024-08-19T12:38:29.949846+0000","last_unstale":"2024-08-19T12:57:35.578970+0000","last_undegraded":"2024-08-19T12:57:35.578970+0000","last_fullsized":"2024-08-19T12:57:35.578970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:34:31.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001822192,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1367,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.418257+0000","last_change":"2024-08-19T12:39:38.181922+0000","last_active":"2024-08-19T12:57:37.418257+0000","last_peered":"2024-08-19T12:57:37.418257+0000","last_clean":"2024-08-19T12:57:37.418257+0000","last_became_active":"2024-08-19T12:39:38.166658+0000","last_became_peered":"2024-08-19T12:39:38.166658+0000","last_unstale":"2024-08-19T12:57:37.418257+0000","last_undegraded":"2024-08-19T12:57:37.418257+0000","last_fullsized":"2024-08-19T12:57:37.418257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:21:22.453156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1373,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861373+0000","last_change":"2024-08-19T12:38:31.963379+0000","last_active":"2024-08-19T12:57:35.861373+0000","last_peered":"2024-08-19T12:57:35.861373+0000","last_clean":"2024-08-19T12:57:35.861373+0000","last_became_active":"2024-08-19T12:38:29.950467+0000","last_became_peered":"2024-08-19T12:38:29.950467+0000","last_unstale":"2024-08-19T12:57:35.861373+0000","last_undegraded":"2024-08-19T12:57:35.861373+0000","last_fullsized":"2024-08-19T12:57:35.861373+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:40:07.574639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049350399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740778+0000","last_change":"2024-08-19T12:39:38.196037+0000","last_active":"2024-08-19T12:57:34.740778+0000","last_peered":"2024-08-19T12:57:34.740778+0000","last_clean":"2024-08-19T12:57:34.740778+0000","last_became_active":"2024-08-19T12:39:38.190786+0000","last_became_peered":"2024-08-19T12:39:38.190786+0000","last_unstale":"2024-08-19T12:57:34.740778+0000","last_undegraded":"2024-08-19T12:57:34.740778+0000","last_fullsized":"2024-08-19T12:57:34.740778+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:07:56.534454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1386,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578815+0000","last_change":"2024-08-19T12:38:31.959032+0000","last_active":"2024-08-19T12:57:35.578815+0000","last_peered":"2024-08-19T12:57:35.578815+0000","last_clean":"2024-08-19T12:57:35.578815+0000","last_became_active":"2024-08-19T12:38:29.948183+0000","last_became_peered":"2024-08-19T12:38:29.948183+0000","last_unstale":"2024-08-19T12:57:35.578815+0000","last_undegraded":"2024-08-19T12:57:35.578815+0000","last_fullsized":"2024-08-19T12:57:35.578815+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:51:55.545957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052740100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1369,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.579281+0000","last_change":"2024-08-19T12:39:38.221167+0000","last_active":"2024-08-19T12:57:35.579281+0000","last_peered":"2024-08-19T12:57:35.579281+0000","last_clean":"2024-08-19T12:57:35.579281+0000","last_became_active":"2024-08-19T12:39:38.191158+0000","last_became_peered":"2024-08-19T12:39:38.191158+0000","last_unstale":"2024-08-19T12:57:35.579281+0000","last_undegraded":"2024-08-19T12:57:35.579281+0000","last_fullsized":"2024-08-19T12:57:35.579281+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T22:59:47.700950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1357,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861035+0000","last_change":"2024-08-19T12:39:38.194777+0000","last_active":"2024-08-19T12:57:35.861035+0000","last_peered":"2024-08-19T12:57:35.861035+0000","last_clean":"2024-08-19T12:57:35.861035+0000","last_became_active":"2024-08-19T12:39:38.161352+0000","last_became_peered":"2024-08-19T12:39:38.161352+0000","last_unstale":"2024-08-19T12:57:35.861035+0000","last_undegraded":"2024-08-19T12:57:35.861035+0000","last_fullsized":"2024-08-19T12:57:35.861035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:31:21.012106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578750+0000","last_change":"2024-08-19T12:39:38.210090+0000","last_active":"2024-08-19T12:57:35.578750+0000","last_peered":"2024-08-19T12:57:35.578750+0000","last_clean":"2024-08-19T12:57:35.578750+0000","last_became_active":"2024-08-19T12:39:38.202668+0000","last_became_peered":"2024-08-19T12:39:38.202668+0000","last_unstale":"2024-08-19T12:57:35.578750+0000","last_undegraded":"2024-08-19T12:57:35.578750+0000","last_fullsized":"2024-08-19T12:57:35.578750+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:58:28.292312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.418365+0000","last_change":"2024-08-19T12:39:38.224507+0000","last_active":"2024-08-19T12:57:37.418365+0000","last_peered":"2024-08-19T12:57:37.418365+0000","last_clean":"2024-08-19T12:57:37.418365+0000","last_became_active":"2024-08-19T12:39:38.222877+0000","last_became_peered":"2024-08-19T12:39:38.222877+0000","last_unstale":"2024-08-19T12:57:37.418365+0000","last_undegraded":"2024-08-19T12:57:37.418365+0000","last_fullsized":"2024-08-19T12:57:37.418365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:55:08.088675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740614+0000","last_change":"2024-08-19T12:39:38.211834+0000","last_active":"2024-08-19T12:57:34.740614+0000","last_peered":"2024-08-19T12:57:34.740614+0000","last_clean":"2024-08-19T12:57:34.740614+0000","last_became_active":"2024-08-19T12:39:38.210985+0000","last_became_peered":"2024-08-19T12:39:38.210985+0000","last_unstale":"2024-08-19T12:57:34.740614+0000","last_undegraded":"2024-08-19T12:57:34.740614+0000","last_fullsized":"2024-08-19T12:57:34.740614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:39:57.635965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578850+0000","last_change":"2024-08-19T12:39:38.268396+0000","last_active":"2024-08-19T12:57:35.578850+0000","last_peered":"2024-08-19T12:57:35.578850+0000","last_clean":"2024-08-19T12:57:35.578850+0000","last_became_active":"2024-08-19T12:39:38.267362+0000","last_became_peered":"2024-08-19T12:39:38.267362+0000","last_unstale":"2024-08-19T12:57:35.578850+0000","last_undegraded":"2024-08-19T12:57:35.578850+0000","last_fullsized":"2024-08-19T12:57:35.578850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:26:11.466849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1357,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861686+0000","last_change":"2024-08-19T12:39:38.195841+0000","last_active":"2024-08-19T12:57:35.861686+0000","last_peered":"2024-08-19T12:57:35.861686+0000","last_clean":"2024-08-19T12:57:35.861686+0000","last_became_active":"2024-08-19T12:39:38.173967+0000","last_became_peered":"2024-08-19T12:39:38.173967+0000","last_unstale":"2024-08-19T12:57:35.861686+0000","last_undegraded":"2024-08-19T12:57:35.861686+0000","last_fullsized":"2024-08-19T12:57:35.861686+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:29:39.717188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1359,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.861315+0000","last_change":"2024-08-19T12:39:38.110553+0000","last_active":"2024-08-19T12:57:35.861315+0000","last_peered":"2024-08-19T12:57:35.861315+0000","last_clean":"2024-08-19T12:57:35.861315+0000","last_became_active":"2024-08-19T12:39:38.108670+0000","last_became_peered":"2024-08-19T12:39:38.108670+0000","last_unstale":"2024-08-19T12:57:35.861315+0000","last_undegraded":"2024-08-19T12:57:35.861315+0000","last_fullsized":"2024-08-19T12:57:35.861315+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:32:52.104320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1366,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.418463+0000","last_change":"2024-08-19T12:39:38.200400+0000","last_active":"2024-08-19T12:57:37.418463+0000","last_peered":"2024-08-19T12:57:37.418463+0000","last_clean":"2024-08-19T12:57:37.418463+0000","last_became_active":"2024-08-19T12:39:38.197445+0000","last_became_peered":"2024-08-19T12:39:38.197445+0000","last_unstale":"2024-08-19T12:57:37.418463+0000","last_undegraded":"2024-08-19T12:57:37.418463+0000","last_fullsized":"2024-08-19T12:57:37.418463+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:21:39.167660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740451+0000","last_change":"2024-08-19T12:39:38.274353+0000","last_active":"2024-08-19T12:57:34.740451+0000","last_peered":"2024-08-19T12:57:34.740451+0000","last_clean":"2024-08-19T12:57:34.740451+0000","last_became_active":"2024-08-19T12:39:38.267711+0000","last_became_peered":"2024-08-19T12:39:38.267711+0000","last_unstale":"2024-08-19T12:57:34.740451+0000","last_undegraded":"2024-08-19T12:57:34.740451+0000","last_fullsized":"2024-08-19T12:57:34.740451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:56:56.848415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578730+0000","last_change":"2024-08-19T12:39:38.188765+0000","last_active":"2024-08-19T12:57:35.578730+0000","last_peered":"2024-08-19T12:57:35.578730+0000","last_clean":"2024-08-19T12:57:35.578730+0000","last_became_active":"2024-08-19T12:39:38.177735+0000","last_became_peered":"2024-08-19T12:39:38.177735+0000","last_unstale":"2024-08-19T12:57:35.578730+0000","last_undegraded":"2024-08-19T12:57:35.578730+0000","last_fullsized":"2024-08-19T12:57:35.578730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:38:45.637343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1370,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-08-19T12:57:35.578720+0000","last_change":"2024-08-19T12:39:38.200875+0000","last_active":"2024-08-19T12:57:35.578720+0000","last_peered":"2024-08-19T12:57:35.578720+0000","last_clean":"2024-08-19T12:57:35.578720+0000","last_became_active":"2024-08-19T12:39:38.199715+0000","last_became_peered":"2024-08-19T12:39:38.199715+0000","last_unstale":"2024-08-19T12:57:35.578720+0000","last_undegraded":"2024-08-19T12:57:35.578720+0000","last_fullsized":"2024-08-19T12:57:35.578720+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T22:27:29.457630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-08-19T12:57:34.740661+0000","last_change":"2024-08-19T12:39:38.196277+0000","last_active":"2024-08-19T12:57:34.740661+0000","last_peered":"2024-08-19T12:57:34.740661+0000","last_clean":"2024-08-19T12:57:34.740661+0000","last_became_active":"2024-08-19T12:39:38.190871+0000","last_became_peered":"2024-08-19T12:39:38.190871+0000","last_unstale":"2024-08-19T12:57:34.740661+0000","last_undegraded":"2024-08-19T12:57:34.740661+0000","last_fullsized":"2024-08-19T12:57:34.740661+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:45:27.127112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607786,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68696,"kb_used_data":5572,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303144,"statfs":{"total":96636764160,"available":96566419456,"internally_reserved":0,"allocated":5705728,"data_stored":2348559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607786,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68704,"kb_used_data":5572,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303136,"statfs":{"total":96636764160,"available":96566411264,"internally_reserved":0,"allocated":5705728,"data_stored":2348545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607789,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68752,"kb_used_data":5620,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303088,"statfs":{"total":96636764160,"available":96566362112,"internally_reserved":0,"allocated":5754880,"data_stored":2877468,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1592,"internal_metadata":64616904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607785,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77020,"kb_used_data":5568,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294820,"statfs":{"total":96636764160,"available":96557895680,"internally_reserved":0,"allocated":5701632,"data_stored":2348540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607786,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77016,"kb_used_data":5572,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294824,"statfs":{"total":96636764160,"available":96557899776,"internally_reserved":0,"allocated":5705728,"data_stored":2348559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640491,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81220,"kb_used_data":5616,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290620,"statfs":{"total":96636764160,"available":96553594880,"internally_reserved":0,"allocated":5750784,"data_stored":2897162,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T12:57:39.891 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-19T12:57:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.995+0000 7fa5ab30b640 10 monclient: get_monmap_and_config 2024-08-19T12:57:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.995+0000 7fa5ab30b640 10 monclient: build_initial_monmap 2024-08-19T12:57:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.995+0000 7fa5ab30b640 10 monclient: monmap: 2024-08-19T12:57:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:39.996586+0000 2024-08-19T12:57:39.996 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:39.996586+0000 2024-08-19T12:57:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:39.997 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:39.997 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:39.998 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.996+0000 7fa5ab30b640 1 Processor -- start 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.996+0000 7fa5ab30b640 1 -- start start 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.996+0000 7fa5ab30b640 10 monclient: init 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.996+0000 7fa5ab30b640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.997+0000 7fa5ab30b640 10 monclient: _add_conns ranks=[7,5,1,0,2,8,6,3,4] 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.997+0000 7fa5ab30b640 10 monclient(hunting): picked mon.noname-i con 0x7fa5a406e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.997+0000 7fa5ab30b640 10 monclient(hunting): picked mon.noname-h con 0x7fa5a40a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.997+0000 7fa5ab30b640 10 monclient(hunting): picked mon.noname-f con 0x7fa5a40a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.997+0000 7fa5ab30b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5a4136790 con 0x7fa5a40a5d50 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.997+0000 7fa5ab30b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5a409ddf0 con 0x7fa5a40a5920 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.997+0000 7fa5ab30b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5a409e0d0 con 0x7fa5a406e7b0 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.997+0000 7fa5ab30b640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.997+0000 7fa5ab30b640 10 monclient(hunting): authenticate will time out at 1854.084099s 2024-08-19T12:57:39.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.997+0000 7fa5a88a6640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7fa5a40a5920 0x7fa5a413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:49980/0 (socket says 172.21.15.26:49980) 2024-08-19T12:57:40.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.997+0000 7fa5a88a6640 1 -- 172.21.15.26:0/3815129716 learned_addr learned my addr 172.21.15.26:0/3815129716 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:40.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa5a88a6640 1 -- 172.21.15.26:0/3815129716 >> v1:172.21.15.104:6791/0 conn(0x7fa5a40a5920 legacy=0x7fa5a413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:40.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa5a90a7640 1 -- 172.21.15.26:0/3815129716 >> v1:172.21.15.104:6792/0 conn(0x7fa5a406e7b0 legacy=0x7fa5a4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:40.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa5a98a8640 1 -- 172.21.15.26:0/3815129716 >> v1:172.21.15.104:6789/0 conn(0x7fa5a40a5d50 legacy=0x7fa5a4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa59bfff640 1 -- 172.21.15.26:0/3815129716 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1574619227 0 0) 0x7fa594001450 con 0x7fa5a40a5920 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa59bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa59bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa59bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa59bfff640 10 monclient(hunting): my global_id is 184757 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa59bfff640 1 -- 172.21.15.26:0/3815129716 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa590003520 con 0x7fa5a40a5920 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa59bfff640 1 -- 172.21.15.26:0/3815129716 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1683162424 0 0) 0x7fa5a4136790 con 0x7fa5a40a5d50 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa59bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa59bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa59bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa59bfff640 10 monclient(hunting): my global_id is 204373 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:39.999+0000 7fa59bfff640 1 -- 172.21.15.26:0/3815129716 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa594001450 con 0x7fa5a40a5d50 2024-08-19T12:57:40.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 1 -- 172.21.15.26:0/3815129716 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1426265520 0 0) 0x7fa590003520 con 0x7fa5a40a5920 2024-08-19T12:57:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 1 -- 172.21.15.26:0/3815129716 >> v1:172.21.15.104:6792/0 conn(0x7fa5a406e7b0 legacy=0x7fa5a4136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 1 -- 172.21.15.26:0/3815129716 >> v1:172.21.15.104:6789/0 conn(0x7fa5a40a5d50 legacy=0x7fa5a4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:57:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 1 monclient: found mon.noname-h 2024-08-19T12:57:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:57:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 1 -- 172.21.15.26:0/3815129716 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a409e3b0 con 0x7fa5a40a5920 2024-08-19T12:57:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 10 monclient: _finish_auth 0 2024-08-19T12:57:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:57:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 1 -- 172.21.15.26:0/3815129716 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa5940048a0 con 0x7fa5a40a5920 2024-08-19T12:57:40.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 10 monclient: dump: 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:40.007 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa5ab30b640 5 monclient: authenticate success, global_id 184757 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa5ab30b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 1 -- 172.21.15.26:0/3815129716 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa594004ac0 con 0x7fa5a40a5920 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 1 -- 172.21.15.26:0/3815129716 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa594005070 con 0x7fa5a40a5920 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa59bfff640 10 monclient: dump: 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:40.008 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:40.009 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa5ab30b640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa5ab30b640 4 set_mon_vals no callback set 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa5ab30b640 10 monclient: shutdown 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa5ab30b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3815129716 >> v1:172.21.15.104:6791/0 conn(0x7fa5a40a5920 legacy=0x7fa5a413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3815129716 shutdown_connections 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3815129716 >> 172.21.15.26:0/3815129716 conn(0x7fa5a40a8740 msgr2=0x7fa5a40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3815129716 shutdown_connections 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.000+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3815129716 wait complete. 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.001+0000 7fa5ab30b640 10 monclient: build_initial_monmap 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.001+0000 7fa5ab30b640 10 monclient: monmap: 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:40.010 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:40.011 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.001+0000 7fa5ab30b640 1 librados: starting msgr at 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.001+0000 7fa5ab30b640 1 librados: starting objecter 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.001+0000 7fa5ab30b640 1 Processor -- start 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.001+0000 7fa5ab30b640 1 -- start start 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.001+0000 7fa5ab30b640 1 librados: setting wanted keys 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.001+0000 7fa5ab30b640 1 librados: calling monclient init 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.001+0000 7fa5ab30b640 10 monclient: init 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5ab30b640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5ab30b640 10 monclient: _add_conns ranks=[5,0,3,1,7,8,2,6,4] 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5ab30b640 10 monclient(hunting): picked mon.noname-h con 0x7fa5a406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5ab30b640 10 monclient(hunting): picked mon.noname-a con 0x7fa5a40a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5ab30b640 10 monclient(hunting): picked mon.noname-g con 0x7fa5a40a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5ab30b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5a4128bb0 con 0x7fa5a40a5920 2024-08-19T12:57:40.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5ab30b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5a4125c90 con 0x7fa5a40a5d50 2024-08-19T12:57:40.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5ab30b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5a41346a0 con 0x7fa5a406e7b0 2024-08-19T12:57:40.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5ab30b640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:40.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5a88a6640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fa5a40a5920 legacy=0x7fa5a41d3480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:57:40.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5ab30b640 10 monclient(hunting): authenticate will time out at 1854.089383s 2024-08-19T12:57:40.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5a98a8640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7fa5a40a5d50 legacy=0x7fa5a41d6bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:40.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5a90a7640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7fa5a406e7b0 0x7fa5a41ce4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:49988/0 (socket says 172.21.15.26:49988) 2024-08-19T12:57:40.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.002+0000 7fa5a90a7640 1 -- 172.21.15.26:0/3737466978 learned_addr learned my addr 172.21.15.26:0/3737466978 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:40.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.004+0000 7fa5a90a7640 1 -- 172.21.15.26:0/3737466978 >> v1:172.21.15.104:6791/0 conn(0x7fa5a406e7b0 legacy=0x7fa5a41ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:40.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.004+0000 7fa599ffb640 10 client.?.objecter ms_handle_connect 0x7fa5a406e7b0 2024-08-19T12:57:40.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.004+0000 7fa599ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:40.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.004+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1229272134 0 0) 0x7fa5a41346a0 con 0x7fa5a406e7b0 2024-08-19T12:57:40.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.004+0000 7fa599ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:57:40.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.004+0000 7fa599ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:40.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.004+0000 7fa599ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:40.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.004+0000 7fa599ffb640 10 monclient(hunting): my global_id is 184766 2024-08-19T12:57:40.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.004+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa588004eb0 con 0x7fa5a406e7b0 2024-08-19T12:57:40.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1193725686 0 0) 0x7fa588004eb0 con 0x7fa5a406e7b0 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5a41346a0 con 0x7fa5a406e7b0 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa5a0003610 con 0x7fa5a406e7b0 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 10 monclient(hunting): dump: 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:40.017 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:40.018 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:40.018 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:40.018 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:40.018 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:40.018 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:40.018 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:40.018 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:40.018 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:40.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:40.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4078872242 0 0) 0x7fa5a41346a0 con 0x7fa5a406e7b0 2024-08-19T12:57:40.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 >> v1:172.21.15.104:6790/0 conn(0x7fa5a40a5d50 legacy=0x7fa5a41d6bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:40.018 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 >> v1:172.21.15.26:6789/0 conn(0x7fa5a40a5920 legacy=0x7fa5a41d3480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 1 monclient: found mon.h 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a4123930 con 0x7fa5a406e7b0 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 10 monclient: _finish_auth 0 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa5ab30b640 5 monclient: authenticate success, global_id 184766 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa5ab30b640 10 monclient: _renew_subs 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa5ab30b640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3737466978 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a4126570 con 0x7fa5a406e7b0 2024-08-19T12:57:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa5a0003bd0 con 0x7fa5a406e7b0 2024-08-19T12:57:40.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:40.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa5a0004140 con 0x7fa5a406e7b0 2024-08-19T12:57:40.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:40.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:57:40.020 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa599ffb640 10 monclient: dump: 2024-08-19T12:57:40.020 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:40.020 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:40.020 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:40.020 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:40.020 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:40.020 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:40.020 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa5ab30b640 10 client.184766.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa5ab30b640 10 monclient: _renew_subs 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa59affd640 4 set_mon_vals no callback set 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa5ab30b640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:40.021 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3737466978 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fa5a4124490 con 0x7fa5a406e7b0 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.005+0000 7fa5ab30b640 1 librados: init done 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.006+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa5a0011a80 con 0x7fa5a406e7b0 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.006+0000 7fa5ab30b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.006+0000 7fa5ab30b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.006+0000 7fa5ab30b640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.006+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3737466978 --> v1:172.21.15.104:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5a41344c0 con 0x7fa5a406e7b0 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.006+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7fa5a004dfd0 con 0x7fa5a406e7b0 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.006+0000 7fa599ffb640 10 client.184766.objecter ms_dispatch 0x7fa5a4000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.006+0000 7fa599ffb640 3 client.184766.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.006+0000 7fa599ffb640 3 client.184766.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.006+0000 7fa599ffb640 20 client.184766.objecter dump_active .. 0 homeless 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.006+0000 7fa5a88a6640 10 monclient: get_auth_request con 0x7fa588040de0 auth_method 0 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.009+0000 7fa599ffb640 10 client.184766.objecter ms_handle_connect 0x7fa588040de0 2024-08-19T12:57:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.011+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 <== mon.5 v1:172.21.15.104:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fa5a41344c0 con 0x7fa5a406e7b0 2024-08-19T12:57:40.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.011+0000 7fa599ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:40.023 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.011+0000 7fa599ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:57:40.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.195+0000 7fa5ab30b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:57:40.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.195+0000 7fa5ab30b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:57:40.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.195+0000 7fa5ab30b640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:40.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.195+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3737466978 --> v1:172.21.15.104:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5a40a1fd0 con 0x7fa5a406e7b0 2024-08-19T12:57:40.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.197+0000 7fa599ffb640 1 -- 172.21.15.26:0/3737466978 <== mon.5 v1:172.21.15.104:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v701) ==== 74+0+13949 (unknown 3666114881 0 755609505) 0x7fa5a00176d0 con 0x7fa5a406e7b0 2024-08-19T12:57:40.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.197+0000 7fa599ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-19T12:57:40.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.197+0000 7fa599ffb640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:57:40.198 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:57:40.198 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":701,"fsid":"10d02fe0-13d8-499a-99b3-d3a725cba366","created":"2024-08-19T12:38:08.215243+0000","modified":"2024-08-19T12:57:37.410303+0000","last_up_change":"2024-08-19T12:38:20.704140+0000","last_in_change":"2024-08-19T12:38:12.064517+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":102,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":340,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-19T12:38:21.116424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-19T12:38:27.276571+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-08-19T12:39:35.592943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae69b71a-73f9-47e2-b81e-92e9cdd7df0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":696,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6800","nonce":3531533167}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6801","nonce":3531533167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6803","nonce":3531533167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6802","nonce":3531533167}]},"public_addr":"172.21.15.26:6800/3531533167","cluster_addr":"172.21.15.26:6801/3531533167","heartbeat_back_addr":"172.21.15.26:6803/3531533167","heartbeat_front_addr":"172.21.15.26:6802/3531533167","state":["exists","up"]},{"osd":1,"uuid":"fc6dcabb-211c-484a-b215-2ef5fe1cbd21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":696,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6804","nonce":2594673920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6805","nonce":2594673920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6807","nonce":2594673920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6806","nonce":2594673920}]},"public_addr":"172.21.15.26:6804/2594673920","cluster_addr":"172.21.15.26:6805/2594673920","heartbeat_back_addr":"172.21.15.26:6807/2594673920","heartbeat_front_addr":"172.21.15.26:6806/2594673920","state":["exists","up"]},{"osd":2,"uuid":"88a4da36-5efe-4f64-84c3-28413eba4011","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":696,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6808","nonce":1443966639}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6809","nonce":1443966639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6811","nonce":1443966639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.26:6810","nonce":1443966639}]},"public_addr":"172.21.15.26:6808/1443966639","cluster_addr":"172.21.15.26:6809/1443966639","heartbeat_back_addr":"172.21.15.26:6811/1443966639","heartbeat_front_addr":"172.21.15.26:6810/1443966639","state":["exists","up"]},{"osd":3,"uuid":"98c08a65-4fb2-4e5f-a9bc-e8899aed1103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":696,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6809","nonce":847838248}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6810","nonce":847838248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6812","nonce":847838248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6811","nonce":847838248}]},"public_addr":"172.21.15.104:6809/847838248","cluster_addr":"172.21.15.104:6810/847838248","heartbeat_back_addr":"172.21.15.104:6812/847838248","heartbeat_front_addr":"172.21.15.104:6811/847838248","state":["exists","up"]},{"osd":4,"uuid":"a4c308b6-5e55-4328-b7b0-1db11fd51d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":696,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6805","nonce":1352976736}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6806","nonce":1352976736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6808","nonce":1352976736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6807","nonce":1352976736}]},"public_addr":"172.21.15.104:6805/1352976736","cluster_addr":"172.21.15.104:6806/1352976736","heartbeat_back_addr":"172.21.15.104:6808/1352976736","heartbeat_front_addr":"172.21.15.104:6807/1352976736","state":["exists","up"]},{"osd":5,"uuid":"99117114-07ce-4b9b-8df6-39f5fb1b610b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":696,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6801","nonce":3713183117}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6802","nonce":3713183117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6804","nonce":3713183117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.104:6803","nonce":3713183117}]},"public_addr":"172.21.15.104:6801/3713183117","cluster_addr":"172.21.15.104:6802/3713183117","heartbeat_back_addr":"172.21.15.104:6804/3713183117","heartbeat_front_addr":"172.21.15.104:6803/3713183117","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.063585+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.566627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.758447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.327856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.657329+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-19T12:38:18.299879+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":2,"length":3}]},{"pool":13,"snaps":[{"begin":2,"length":5}]},{"pool":35,"snaps":[{"begin":1,"length":1}]},{"pool":65,"snaps":[{"begin":1,"length":1}]},{"pool":93,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2}]},{"pool":103,"snaps":[{"begin":1,"length":1}]},{"pool":122,"snaps":[{"begin":1,"length":2}]},{"pool":137,"snaps":[{"begin":3,"length":1}]},{"pool":142,"snaps":[{"begin":1,"length":4}]},{"pool":148,"snaps":[{"begin":2,"length":1}]},{"pool":149,"snaps":[{"begin":2,"length":2}]},{"pool":157,"snaps":[{"begin":2,"length":3}]},{"pool":162,"snaps":[{"begin":3,"length":4}]},{"pool":164,"snaps":[{"begin":2,"length":4}]},{"pool":166,"snaps":[{"begin":2,"length":2}]},{"pool":173,"snaps":[{"begin":2,"length":1}]},{"pool":178,"snaps":[{"begin":1,"length":4}]},{"pool":185,"snaps":[{"begin":7,"length":1}]},{"pool":196,"snaps":[{"begin":4,"length":4}]},{"pool":218,"snaps":[{"begin":8,"length":1}]},{"pool":230,"snaps":[{"begin":10,"length":1}]},{"pool":235,"snaps":[{"begin":11,"length":2}]},{"pool":250,"snaps":[{"begin":14,"length":1}]},{"pool":321,"snaps":[{"begin":2,"length":1}]},{"pool":322,"snaps":[{"begin":3,"length":1}]},{"pool":325,"snaps":[{"begin":4,"length":1}]},{"pool":329,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-19T12:57:40.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 10 librados: watch_flush enter 2024-08-19T12:57:40.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 10 librados: watch_flush exit 2024-08-19T12:57:40.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 20 client.184766.objecter shutdown clearing up homeless session... 2024-08-19T12:57:40.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 10 client.184766.objecter successfully canceled tick 2024-08-19T12:57:40.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3737466978 >> v1:172.21.15.104:6800/899887506 conn(0x7fa588040de0 legacy=0x7fa5880432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:40.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 10 monclient: shutdown 2024-08-19T12:57:40.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:40.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3737466978 >> v1:172.21.15.104:6791/0 conn(0x7fa5a406e7b0 legacy=0x7fa5a41ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:40.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3737466978 shutdown_connections 2024-08-19T12:57:40.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3737466978 >> 172.21.15.26:0/3737466978 conn(0x7fa5a40a8740 msgr2=0x7fa5a4135150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:40.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3737466978 shutdown_connections 2024-08-19T12:57:40.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 1 -- 172.21.15.26:0/3737466978 wait complete. 2024-08-19T12:57:40.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:40.199+0000 7fa5ab30b640 1 librados: shutdown 2024-08-19T12:57:41.212 INFO:tasks.ceph:Scrubbing osd.0 2024-08-19T12:57:41.212 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-08-19T12:57:41.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.326+0000 7fef65181640 10 monclient: get_monmap_and_config 2024-08-19T12:57:41.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.326+0000 7fef65181640 10 monclient: build_initial_monmap 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.326+0000 7fef65181640 10 monclient: monmap: 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:41.326935+0000 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:41.326935+0000 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:41.327 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:41.328 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:41.328 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:41.328 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:41.328 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.326+0000 7fef65181640 1 Processor -- start 2024-08-19T12:57:41.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.326+0000 7fef65181640 1 -- start start 2024-08-19T12:57:41.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.326+0000 7fef65181640 10 monclient: init 2024-08-19T12:57:41.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.327+0000 7fef65181640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:41.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.327+0000 7fef65181640 10 monclient: _add_conns ranks=[8,5,7,4,3,2,6,1,0] 2024-08-19T12:57:41.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.327+0000 7fef65181640 10 monclient(hunting): picked mon.noname-e con 0x7fef600a6640 addr v1:172.21.15.26:6793/0 2024-08-19T12:57:41.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.327+0000 7fef65181640 10 monclient(hunting): picked mon.noname-h con 0x7fef60074110 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.327+0000 7fef65181640 10 monclient(hunting): picked mon.noname-i con 0x7fef6013f8d0 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.327+0000 7fef65181640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef6009ea70 con 0x7fef60074110 2024-08-19T12:57:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.327+0000 7fef65181640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef6009dd20 con 0x7fef6013f8d0 2024-08-19T12:57:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.327+0000 7fef65181640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef6009e000 con 0x7fef600a6640 2024-08-19T12:57:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.327+0000 7fef65181640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.327+0000 7fef65181640 10 monclient(hunting): authenticate will time out at 1855.414270s 2024-08-19T12:57:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.327+0000 7fef5ed9d640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fef600a6640 0x7fef600a6a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:57076/0 (socket says 172.21.15.26:57076) 2024-08-19T12:57:41.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.327+0000 7fef5ed9d640 1 -- 172.21.15.26:0/2739096796 learned_addr learned my addr 172.21.15.26:0/2739096796 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:41.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.329+0000 7fef5ed9d640 1 -- 172.21.15.26:0/2739096796 >> v1:172.21.15.26:6793/0 conn(0x7fef600a6640 legacy=0x7fef600a6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.329+0000 7fef5e59c640 1 -- 172.21.15.26:0/2739096796 >> v1:172.21.15.104:6791/0 conn(0x7fef60074110 legacy=0x7fef600a7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.329+0000 7fef5f59e640 1 -- 172.21.15.26:0/2739096796 >> v1:172.21.15.104:6792/0 conn(0x7fef6013f8d0 legacy=0x7fef60141cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1594322327 0 0) 0x7fef6009ea70 con 0x7fef60074110 2024-08-19T12:57:41.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient(hunting): my global_id is 184775 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef4c003520 con 0x7fef60074110 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3889975287 0 0) 0x7fef6009e000 con 0x7fef600a6640 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient(hunting): my global_id is 184715 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef6009ea70 con 0x7fef600a6640 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3753769958 0 0) 0x7fef6009dd20 con 0x7fef6013f8d0 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient(hunting): my global_id is 136555 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef6009e000 con 0x7fef6013f8d0 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3498149586 0 0) 0x7fef6009ea70 con 0x7fef600a6640 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 >> v1:172.21.15.104:6792/0 conn(0x7fef6013f8d0 legacy=0x7fef60141cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 >> v1:172.21.15.104:6791/0 conn(0x7fef60074110 legacy=0x7fef600a7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient: _finish_hunting 0 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 monclient: found mon.noname-e 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef600a7de0 con 0x7fef600a6640 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient: _finish_auth 0 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient: _check_auth_tickets 2024-08-19T12:57:41.336 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:41.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fef50001e40 con 0x7fef600a6640 2024-08-19T12:57:41.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef65181640 5 monclient: authenticate success, global_id 184715 2024-08-19T12:57:41.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef65181640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:41.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:41.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:57:41.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient: dump: 2024-08-19T12:57:41.337 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.337 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.337 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.337 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.337 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef65181640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:41.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef50004c20 con 0x7fef600a6640 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef65181640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 1 -- 172.21.15.26:0/2739096796 <== mon.8 v1:172.21.15.26:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fef50005190 con 0x7fef600a6640 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef65181640 4 set_mon_vals no callback set 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef5dd9b640 10 monclient: dump: 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.339 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef65181640 10 monclient: shutdown 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef65181640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.330+0000 7fef65181640 1 -- 172.21.15.26:0/2739096796 >> v1:172.21.15.26:6793/0 conn(0x7fef600a6640 legacy=0x7fef600a6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.331+0000 7fef65181640 1 -- 172.21.15.26:0/2739096796 shutdown_connections 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.331+0000 7fef65181640 1 -- 172.21.15.26:0/2739096796 >> 172.21.15.26:0/2739096796 conn(0x7fef600a8740 msgr2=0x7fef600a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.331+0000 7fef65181640 1 -- 172.21.15.26:0/2739096796 shutdown_connections 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.331+0000 7fef65181640 1 -- 172.21.15.26:0/2739096796 wait complete. 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.331+0000 7fef65181640 10 monclient: build_initial_monmap 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.331+0000 7fef65181640 10 monclient: monmap: 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:41.341 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 1 librados: starting msgr at 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 1 librados: starting objecter 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 1 Processor -- start 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 1 -- start start 2024-08-19T12:57:41.342 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 1 librados: setting wanted keys 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 1 librados: calling monclient init 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 10 monclient: init 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 10 monclient: _add_conns ranks=[7,2,0,4,6,8,5,1,3] 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 10 monclient(hunting): picked mon.noname-i con 0x7fef60074110 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 10 monclient(hunting): picked mon.noname-b con 0x7fef600a6640 addr v1:172.21.15.26:6790/0 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 10 monclient(hunting): picked mon.noname-a con 0x7fef6013f8d0 addr v1:172.21.15.26:6789/0 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef600a1a10 con 0x7fef6013f8d0 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef600aabe0 con 0x7fef600a6640 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef6013e970 con 0x7fef60074110 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef65181640 10 monclient(hunting): authenticate will time out at 1855.419861s 2024-08-19T12:57:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef5f59e640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fef6013f8d0 legacy=0x7fef601d6530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:57:41.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef5e59c640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fef600a6640 0x7fef600a0780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39106/0 (socket says 172.21.15.26:39106) 2024-08-19T12:57:41.344 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.332+0000 7fef5e59c640 1 -- 172.21.15.26:0/1617709435 learned_addr learned my addr 172.21.15.26:0/1617709435 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:41.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef5e59c640 1 -- 172.21.15.26:0/1617709435 >> v1:172.21.15.26:6790/0 conn(0x7fef600a6640 legacy=0x7fef600a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.348 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 client.?.objecter ms_handle_connect 0x7fef600a6640 2024-08-19T12:57:41.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:41.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1870509575 0 0) 0x7fef600aabe0 con 0x7fef600a6640 2024-08-19T12:57:41.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): my global_id is 27515 2024-08-19T12:57:41.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef5ed9d640 1 -- 172.21.15.26:0/1617709435 >> v1:172.21.15.104:6792/0 conn(0x7fef60074110 legacy=0x7fef600a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef40004eb0 con 0x7fef600a6640 2024-08-19T12:57:41.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 client.?.objecter ms_handle_connect 0x7fef60074110 2024-08-19T12:57:41.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:41.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1680820196 0 0) 0x7fef6013e970 con 0x7fef60074110 2024-08-19T12:57:41.349 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): my global_id is 136564 2024-08-19T12:57:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef600aabe0 con 0x7fef60074110 2024-08-19T12:57:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2933100387 0 0) 0x7fef40004eb0 con 0x7fef600a6640 2024-08-19T12:57:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef6013e970 con 0x7fef600a6640 2024-08-19T12:57:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fef48003610 con 0x7fef600a6640 2024-08-19T12:57:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:57:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): dump: 2024-08-19T12:57:41.350 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.351 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.352 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3193082533 0 0) 0x7fef600aabe0 con 0x7fef60074110 2024-08-19T12:57:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef40004eb0 con 0x7fef60074110 2024-08-19T12:57:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fef50005110 con 0x7fef60074110 2024-08-19T12:57:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:57:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient(hunting): dump: 2024-08-19T12:57:41.352 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.353 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.354 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.354 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.354 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2027561670 0 0) 0x7fef6013e970 con 0x7fef600a6640 2024-08-19T12:57:41.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 >> v1:172.21.15.104:6792/0 conn(0x7fef60074110 legacy=0x7fef600a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 >> v1:172.21.15.26:6789/0 conn(0x7fef6013f8d0 legacy=0x7fef601d6530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:41.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient: _finish_hunting 0 2024-08-19T12:57:41.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 monclient: found mon.b 2024-08-19T12:57:41.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:41.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:41.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef600ab110 con 0x7fef600a6640 2024-08-19T12:57:41.354 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient: _finish_auth 0 2024-08-19T12:57:41.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient: _check_auth_tickets 2024-08-19T12:57:41.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:41.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef65181640 5 monclient: authenticate success, global_id 27515 2024-08-19T12:57:41.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef65181640 10 monclient: _renew_subs 2024-08-19T12:57:41.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef65181640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:41.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef65181640 1 -- 172.21.15.26:0/1617709435 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fef600a0e90 con 0x7fef600a6640 2024-08-19T12:57:41.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef65181640 10 client.27515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:41.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef65181640 10 monclient: _renew_subs 2024-08-19T12:57:41.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef65181640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:41.355 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef65181640 1 -- 172.21.15.26:0/1617709435 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fef600a4520 con 0x7fef600a6640 2024-08-19T12:57:41.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef65181640 1 librados: init done 2024-08-19T12:57:41.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef48003bc0 con 0x7fef600a6640 2024-08-19T12:57:41.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:41.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fef480040f0 con 0x7fef600a6640 2024-08-19T12:57:41.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:41.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef5d59a640 4 set_mon_vals no callback set 2024-08-19T12:57:41.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:57:41.358 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.335+0000 7fef577fe640 10 monclient: dump: 2024-08-19T12:57:41.358 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.358 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.359 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.360 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.336+0000 7fef65181640 10 client.27515.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:41.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.336+0000 7fef65181640 15 client.27515.objecter _session_command_op_assign -1 1 2024-08-19T12:57:41.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.336+0000 7fef65181640 20 client.27515.objecter _get_session osd=-1 returning homeless 2024-08-19T12:57:41.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.336+0000 7fef65181640 10 client.27515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:41.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.336+0000 7fef65181640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:41.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.336+0000 7fef65181640 1 -- 172.21.15.26:0/1617709435 --> v1:172.21.15.26:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fef44002910 con 0x7fef600a6640 2024-08-19T12:57:41.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.336+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fef48004290 con 0x7fef600a6640 2024-08-19T12:57:41.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7fef4804dfa0 con 0x7fef600a6640 2024-08-19T12:57:41.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 10 client.27515.objecter ms_dispatch 0x7fef60000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:41.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 3 client.27515.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:41.360 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 3 client.27515.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 10 client.27515.objecter checking command 1 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 20 client.27515.objecter _get_session s=0x7fef400493b0 osd=0 3 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 20 client.27515.objecter put_session s=0x7fef400493b0 osd=0 3 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 15 client.27515.objecter _session_command_op_remove -1 1 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 20 client.27515.objecter _get_session s=0x7fef400493b0 osd=0 3 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 20 client.27515.objecter get_session s=0x7fef400493b0 osd=0 3 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 15 client.27515.objecter _session_command_op_assign 0 1 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 20 client.27515.objecter put_session s=0x7fef400493b0 osd=0 4 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 10 client.27515.objecter _send_command 1 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 --> v1:172.21.15.26:6800/3531533167 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fef4004d770 con 0x7fef40049a60 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 20 client.27515.objecter dump_active .. 0 homeless 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_get_version_reply(handle=1 version=701) ==== 24+0+0 (unknown 2985145993 0 0) 0x7fef4804e410 con 0x7fef600a6640 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef577fe640 10 monclient: handle_get_version_reply finishing 1 version 701 2024-08-19T12:57:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef5ed9d640 10 monclient: get_auth_request con 0x7fef400414e0 auth_method 0 2024-08-19T12:57:41.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.337+0000 7fef5f59e640 10 monclient: get_auth_request con 0x7fef40049a60 auth_method 0 2024-08-19T12:57:41.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.339+0000 7fef577fe640 10 client.27515.objecter ms_handle_connect 0x7fef400414e0 2024-08-19T12:57:41.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.341+0000 7fef577fe640 10 client.27515.objecter ms_handle_connect 0x7fef40049a60 2024-08-19T12:57:41.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.342+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== osd.0 v1:172.21.15.26:6800/3531533167 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fef4004d770 con 0x7fef40049a60 2024-08-19T12:57:41.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.342+0000 7fef577fe640 10 client.27515.objecter ms_dispatch 0x7fef60000b90 command_reply(tid 1: 0 ) 2024-08-19T12:57:41.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.342+0000 7fef577fe640 10 client.27515.objecter _finish_command 1 = system:0 2024-08-19T12:57:41.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.342+0000 7fef577fe640 20 client.27515.objecter put_session s=0x7fef400493b0 osd=0 4 2024-08-19T12:57:41.362 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.342+0000 7fef577fe640 15 client.27515.objecter _session_command_op_remove 0 1 2024-08-19T12:57:41.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 10 client.27515.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-08-19T12:57:41.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 15 client.27515.objecter _session_command_op_assign -1 2 2024-08-19T12:57:41.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 20 client.27515.objecter _get_session s=0x7fef400493b0 osd=0 3 2024-08-19T12:57:41.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 20 client.27515.objecter put_session s=0x7fef400493b0 osd=0 3 2024-08-19T12:57:41.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 20 client.27515.objecter _get_session s=0x7fef400493b0 osd=0 3 2024-08-19T12:57:41.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 15 client.27515.objecter _session_command_op_remove -1 2 2024-08-19T12:57:41.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 20 client.27515.objecter get_session s=0x7fef400493b0 osd=0 3 2024-08-19T12:57:41.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 15 client.27515.objecter _session_command_op_assign 0 2 2024-08-19T12:57:41.363 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 20 client.27515.objecter put_session s=0x7fef400493b0 osd=0 4 2024-08-19T12:57:41.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 10 client.27515.objecter _send_command 2 2024-08-19T12:57:41.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 1 -- 172.21.15.26:0/1617709435 --> v1:172.21.15.26:6800/3531533167 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fef440012b0 con 0x7fef40049a60 2024-08-19T12:57:41.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef577fe640 1 -- 172.21.15.26:0/1617709435 <== osd.0 v1:172.21.15.26:6800/3531533167 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fef440012b0 con 0x7fef40049a60 2024-08-19T12:57:41.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef577fe640 10 client.27515.objecter ms_dispatch 0x7fef60000b90 command_reply(tid 2: 0 ) 2024-08-19T12:57:41.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef577fe640 10 client.27515.objecter _finish_command 2 = system:0 2024-08-19T12:57:41.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef577fe640 20 client.27515.objecter put_session s=0x7fef400493b0 osd=0 4 2024-08-19T12:57:41.364 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef577fe640 15 client.27515.objecter _session_command_op_remove 0 2 2024-08-19T12:57:41.364 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:57:41.365 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-19T12:57:41.365 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:57:41.366 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 10 librados: watch_flush enter 2024-08-19T12:57:41.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 10 librados: watch_flush exit 2024-08-19T12:57:41.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 10 client.27515.objecter close_session for osd.0 2024-08-19T12:57:41.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 1 -- 172.21.15.26:0/1617709435 >> v1:172.21.15.26:6800/3531533167 conn(0x7fef40049a60 legacy=0x7fef4004bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 20 client.27515.objecter put_session s=0x7fef400493b0 osd=0 1 2024-08-19T12:57:41.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 20 client.27515.objecter shutdown clearing up homeless session... 2024-08-19T12:57:41.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 10 client.27515.objecter successfully canceled tick 2024-08-19T12:57:41.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 1 -- 172.21.15.26:0/1617709435 >> v1:172.21.15.104:6800/899887506 conn(0x7fef400414e0 legacy=0x7fef400439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 10 monclient: shutdown 2024-08-19T12:57:41.367 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:41.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 1 -- 172.21.15.26:0/1617709435 >> v1:172.21.15.26:6790/0 conn(0x7fef600a6640 legacy=0x7fef600a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef5f59e640 1 -- 172.21.15.26:0/1617709435 reap_dead start 2024-08-19T12:57:41.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 1 -- 172.21.15.26:0/1617709435 shutdown_connections 2024-08-19T12:57:41.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.355+0000 7fef65181640 1 -- 172.21.15.26:0/1617709435 >> 172.21.15.26:0/1617709435 conn(0x7fef600a8740 msgr2=0x7fef600a9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:41.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.356+0000 7fef65181640 1 -- 172.21.15.26:0/1617709435 shutdown_connections 2024-08-19T12:57:41.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.356+0000 7fef65181640 1 -- 172.21.15.26:0/1617709435 wait complete. 2024-08-19T12:57:41.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.356+0000 7fef65181640 1 librados: shutdown 2024-08-19T12:57:41.368 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-08-19T12:57:41.468 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.468+0000 7f56241cf640 10 monclient: get_monmap_and_config 2024-08-19T12:57:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.468+0000 7f56241cf640 10 monclient: build_initial_monmap 2024-08-19T12:57:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.468+0000 7f56241cf640 10 monclient: monmap: 2024-08-19T12:57:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:41.469067+0000 2024-08-19T12:57:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:41.469067+0000 2024-08-19T12:57:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:41.469 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:41.470 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:41.470 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.468+0000 7f56241cf640 1 Processor -- start 2024-08-19T12:57:41.471 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.468+0000 7f56241cf640 1 -- start start 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.468+0000 7f56241cf640 10 monclient: init 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f56241cf640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f56241cf640 10 monclient: _add_conns ranks=[4,6,0,2,8,3,5,7,1] 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f56241cf640 10 monclient(hunting): picked mon.noname-c con 0x7f561c06e7b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f56241cf640 10 monclient(hunting): picked mon.noname-d con 0x7f561c0a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f56241cf640 10 monclient(hunting): picked mon.noname-a con 0x7f561c0a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f56241cf640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f561c136790 con 0x7f561c0a5d50 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f56241cf640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f561c09ddf0 con 0x7f561c06e7b0 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f56241cf640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f561c09e0d0 con 0x7f561c0a5920 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f56241cf640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f56241cf640 10 monclient(hunting): authenticate will time out at 1855.556505s 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f562176a640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f561c0a5920 legacy=0x7f561c13e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:41.472 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f562276c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f561c0a5d50 legacy=0x7f561c141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:57:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f5621f6b640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f561c06e7b0 0x7f561c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36744/0 (socket says 172.21.15.26:36744) 2024-08-19T12:57:41.473 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.469+0000 7f5621f6b640 1 -- 172.21.15.26:0/3192470043 learned_addr learned my addr 172.21.15.26:0/3192470043 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.471+0000 7f5621f6b640 1 -- 172.21.15.26:0/3192470043 >> v1:172.21.15.26:6791/0 conn(0x7f561c06e7b0 legacy=0x7f561c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.471+0000 7f5620f69640 1 -- 172.21.15.26:0/3192470043 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3512764866 0 0) 0x7f560c0017d0 con 0x7f561c06e7b0 2024-08-19T12:57:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.471+0000 7f5620f69640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.471+0000 7f5620f69640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.471+0000 7f5620f69640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.476 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.471+0000 7f5620f69640 10 monclient(hunting): my global_id is 184828 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.471+0000 7f5620f69640 1 -- 172.21.15.26:0/3192470043 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5604003520 con 0x7f561c06e7b0 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 1 -- 172.21.15.26:0/3192470043 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 45032307 0 0) 0x7f5604003520 con 0x7f561c06e7b0 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 1 -- 172.21.15.26:0/3192470043 >> v1:172.21.15.26:6792/0 conn(0x7f561c0a5920 legacy=0x7f561c13e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 1 -- 172.21.15.26:0/3192470043 >> v1:172.21.15.26:6789/0 conn(0x7f561c0a5d50 legacy=0x7f561c141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 10 monclient: _finish_hunting 0 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 1 monclient: found mon.noname-c 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 1 -- 172.21.15.26:0/3192470043 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c09e3b0 con 0x7f561c06e7b0 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 10 monclient: _finish_auth 0 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 10 monclient: _check_auth_tickets 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 1 -- 172.21.15.26:0/3192470043 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f560c002030 con 0x7f561c06e7b0 2024-08-19T12:57:41.477 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f56241cf640 5 monclient: authenticate success, global_id 184828 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f56241cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 10 monclient: dump: 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.478 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f56241cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 1 -- 172.21.15.26:0/3192470043 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f560c004d10 con 0x7f561c06e7b0 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f56241cf640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 1 -- 172.21.15.26:0/3192470043 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f560c005240 con 0x7f561c06e7b0 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f56241cf640 4 set_mon_vals no callback set 2024-08-19T12:57:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f5620f69640 10 monclient: dump: 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.480 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f56241cf640 10 monclient: shutdown 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f56241cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f56241cf640 1 -- 172.21.15.26:0/3192470043 >> v1:172.21.15.26:6791/0 conn(0x7f561c06e7b0 legacy=0x7f561c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f56241cf640 1 -- 172.21.15.26:0/3192470043 shutdown_connections 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f56241cf640 1 -- 172.21.15.26:0/3192470043 >> 172.21.15.26:0/3192470043 conn(0x7f561c0a8740 msgr2=0x7f561c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f56241cf640 1 -- 172.21.15.26:0/3192470043 shutdown_connections 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.472+0000 7f56241cf640 1 -- 172.21.15.26:0/3192470043 wait complete. 2024-08-19T12:57:41.481 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.473+0000 7f56241cf640 10 monclient: build_initial_monmap 2024-08-19T12:57:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.473+0000 7f56241cf640 10 monclient: monmap: 2024-08-19T12:57:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:41.482 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.473+0000 7f56241cf640 1 librados: starting msgr at 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.473+0000 7f56241cf640 1 librados: starting objecter 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.473+0000 7f56241cf640 1 Processor -- start 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.473+0000 7f56241cf640 1 -- start start 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.473+0000 7f56241cf640 1 librados: setting wanted keys 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.473+0000 7f56241cf640 1 librados: calling monclient init 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.473+0000 7f56241cf640 10 monclient: init 2024-08-19T12:57:41.483 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f56241cf640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f56241cf640 10 monclient: _add_conns ranks=[1,2,3,7,5,6,0,8,4] 2024-08-19T12:57:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f56241cf640 10 monclient(hunting): picked mon.noname-f con 0x7f561c06e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:57:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f56241cf640 10 monclient(hunting): picked mon.noname-b con 0x7f561c0a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:57:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f56241cf640 10 monclient(hunting): picked mon.noname-g con 0x7f561c0a5d50 addr v1:172.21.15.104:6790/0 2024-08-19T12:57:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f56241cf640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f561c142ec0 con 0x7f561c06e7b0 2024-08-19T12:57:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f56241cf640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f561c132ba0 con 0x7f561c0a5920 2024-08-19T12:57:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f56241cf640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f561c0a0bc0 con 0x7f561c0a5d50 2024-08-19T12:57:41.484 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f56241cf640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f56241cf640 10 monclient(hunting): authenticate will time out at 1855.561278s 2024-08-19T12:57:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f562276c640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f561c0a5d50 legacy=0x7f561c1d6290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f562176a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f561c0a5920 0x7f561c132490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39118/0 (socket says 172.21.15.26:39118) 2024-08-19T12:57:41.485 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.474+0000 7f562176a640 1 -- 172.21.15.26:0/2517260182 learned_addr learned my addr 172.21.15.26:0/2517260182 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f562176a640 1 -- 172.21.15.26:0/2517260182 >> v1:172.21.15.26:6790/0 conn(0x7f561c0a5920 legacy=0x7f561c132490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.488 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 client.?.objecter ms_handle_connect 0x7f561c0a5920 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 415354776 0 0) 0x7f561c132ba0 con 0x7f561c0a5920 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 monclient(hunting): my global_id is 27524 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5608004eb0 con 0x7f561c0a5920 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 client.?.objecter ms_handle_connect 0x7f561c06e7b0 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f5621f6b640 1 -- 172.21.15.26:0/2517260182 >> v1:172.21.15.104:6789/0 conn(0x7f561c06e7b0 legacy=0x7f561c131d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1177626631 0 0) 0x7f5608004eb0 con 0x7f561c0a5920 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f561c132ba0 con 0x7f561c0a5920 2024-08-19T12:57:41.489 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5610004760 con 0x7f561c0a5920 2024-08-19T12:57:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:57:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 monclient(hunting): dump: 2024-08-19T12:57:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.490 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 657954620 0 0) 0x7f561c09e3b0 con 0x7f561c06e7b0 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.476+0000 7f561a7fc640 10 monclient(hunting): my global_id is 204382 2024-08-19T12:57:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5608004eb0 con 0x7f561c06e7b0 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2290465428 0 0) 0x7f561c132ba0 con 0x7f561c0a5920 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 >> v1:172.21.15.104:6790/0 conn(0x7f561c0a5d50 legacy=0x7f561c1d6290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 >> v1:172.21.15.104:6789/0 conn(0x7f561c06e7b0 legacy=0x7f561c131d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 1 monclient: found mon.b 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c0a0ea0 con 0x7f561c0a5920 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 10 monclient: _finish_auth 0 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f56241cf640 5 monclient: authenticate success, global_id 27524 2024-08-19T12:57:41.492 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f56241cf640 10 monclient: _renew_subs 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f56241cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f56241cf640 1 -- 172.21.15.26:0/2517260182 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f561c09f910 con 0x7f561c0a5920 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f56241cf640 10 client.27524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f56241cf640 10 monclient: _renew_subs 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f56241cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f56241cf640 1 -- 172.21.15.26:0/2517260182 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f561c1d79b0 con 0x7f561c0a5920 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f56241cf640 1 librados: init done 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5610004980 con 0x7f561c0a5920 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f5610004ef0 con 0x7f561c0a5920 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561bfff640 4 set_mon_vals no callback set 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:57:41.493 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.477+0000 7f561a7fc640 10 monclient: dump: 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.478+0000 7f56241cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.478+0000 7f56241cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.478+0000 7f56241cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.478+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f5610005140 con 0x7f561c0a5920 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.478+0000 7f56241cf640 1 -- 172.21.15.26:0/2517260182 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55fc005080 con 0x7f561c0a5920 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.478+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7f561004e0a0 con 0x7f561c0a5920 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.478+0000 7f561a7fc640 10 client.27524.objecter ms_dispatch 0x7f561c000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.478+0000 7f561a7fc640 3 client.27524.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.478+0000 7f561a7fc640 3 client.27524.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.478+0000 7f561a7fc640 20 client.27524.objecter dump_active .. 0 homeless 2024-08-19T12:57:41.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.478+0000 7f5621f6b640 10 monclient: get_auth_request con 0x7f5608040ed0 auth_method 0 2024-08-19T12:57:41.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.480+0000 7f561a7fc640 10 client.27524.objecter ms_handle_connect 0x7f5608040ed0 2024-08-19T12:57:41.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.482+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f5610017680 con 0x7f561c0a5920 2024-08-19T12:57:41.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.482+0000 7f561a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:41.496 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.482+0000 7f561a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:57:41.666 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.665+0000 7f56241cf640 1 -- 172.21.15.26:0/2517260182 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f55fc000b90 con 0x7f5608040ed0 2024-08-19T12:57:41.667 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.666+0000 7f561a7fc640 1 -- 172.21.15.26:0/2517260182 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7f55fc000b90 con 0x7f5608040ed0 2024-08-19T12:57:41.667 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 0 to deep-scrub 2024-08-19T12:57:41.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 10 librados: watch_flush enter 2024-08-19T12:57:41.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 10 librados: watch_flush exit 2024-08-19T12:57:41.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 20 client.27524.objecter shutdown clearing up homeless session... 2024-08-19T12:57:41.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 10 client.27524.objecter successfully canceled tick 2024-08-19T12:57:41.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 1 -- 172.21.15.26:0/2517260182 >> v1:172.21.15.104:6800/899887506 conn(0x7f5608040ed0 legacy=0x7f5608043390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 10 monclient: shutdown 2024-08-19T12:57:41.669 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:41.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 1 -- 172.21.15.26:0/2517260182 >> v1:172.21.15.26:6790/0 conn(0x7f561c0a5920 legacy=0x7f561c132490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 1 -- 172.21.15.26:0/2517260182 shutdown_connections 2024-08-19T12:57:41.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 1 -- 172.21.15.26:0/2517260182 >> 172.21.15.26:0/2517260182 conn(0x7f561c0a8740 msgr2=0x7f561c06d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:41.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 1 -- 172.21.15.26:0/2517260182 shutdown_connections 2024-08-19T12:57:41.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 1 -- 172.21.15.26:0/2517260182 wait complete. 2024-08-19T12:57:41.670 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.668+0000 7f56241cf640 1 librados: shutdown 2024-08-19T12:57:41.679 INFO:tasks.ceph:Scrubbing osd.1 2024-08-19T12:57:41.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-08-19T12:57:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.781+0000 7f928ec48640 10 monclient: get_monmap_and_config 2024-08-19T12:57:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.781+0000 7f928ec48640 10 monclient: build_initial_monmap 2024-08-19T12:57:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.781+0000 7f928ec48640 10 monclient: monmap: 2024-08-19T12:57:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:41.782463+0000 2024-08-19T12:57:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:41.782463+0000 2024-08-19T12:57:41.782 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:41.783 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:41.783 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 1 Processor -- start 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 1 -- start start 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 10 monclient: init 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 10 monclient: _add_conns ranks=[5,6,3,1,8,0,7,4,2] 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 10 monclient(hunting): picked mon.noname-h con 0x7f928806e700 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 10 monclient(hunting): picked mon.noname-d con 0x7f9288138fe0 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 10 monclient(hunting): picked mon.noname-g con 0x7f9288139b80 addr v1:172.21.15.104:6790/0 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92880a80f0 con 0x7f9288139b80 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f928809de80 con 0x7f928806e700 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f928809e160 con 0x7f9288138fe0 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:41.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928ec48640 10 monclient(hunting): authenticate will time out at 1855.869888s 2024-08-19T12:57:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f927ffff640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f9288138fe0 legacy=0x7f928813e360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.782+0000 7f928d1e5640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f9288139b80 legacy=0x7f9288141a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.783+0000 7f928c9e4640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f928806e700 0x7f9288139470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:49996/0 (socket says 172.21.15.26:49996) 2024-08-19T12:57:41.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.783+0000 7f928c9e4640 1 -- 172.21.15.26:0/4014336303 learned_addr learned my addr 172.21.15.26:0/4014336303 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f928c9e4640 1 -- 172.21.15.26:0/4014336303 >> v1:172.21.15.104:6791/0 conn(0x7f928806e700 legacy=0x7f9288139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 1 -- 172.21.15.26:0/4014336303 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3005576388 0 0) 0x7f928809de80 con 0x7f928806e700 2024-08-19T12:57:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 10 monclient(hunting): my global_id is 184784 2024-08-19T12:57:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 1 -- 172.21.15.26:0/4014336303 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9270003520 con 0x7f928806e700 2024-08-19T12:57:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 1 -- 172.21.15.26:0/4014336303 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3044019441 0 0) 0x7f9270003520 con 0x7f928806e700 2024-08-19T12:57:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 1 -- 172.21.15.26:0/4014336303 >> v1:172.21.15.26:6792/0 conn(0x7f9288138fe0 legacy=0x7f928813e360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 1 -- 172.21.15.26:0/4014336303 >> v1:172.21.15.104:6790/0 conn(0x7f9288139b80 legacy=0x7f9288141a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:57:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 1 monclient: found mon.noname-h 2024-08-19T12:57:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.104:6791/0 2024-08-19T12:57:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 1 -- 172.21.15.26:0/4014336303 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928809e440 con 0x7f928806e700 2024-08-19T12:57:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 10 monclient: _finish_auth 0 2024-08-19T12:57:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:57:41.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 1 -- 172.21.15.26:0/4014336303 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9284001ae0 con 0x7f928806e700 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f927f7fe640 10 monclient: dump: 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.791 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f928ec48640 5 monclient: authenticate success, global_id 184784 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.785+0000 7f928ec48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f927f7fe640 1 -- 172.21.15.26:0/4014336303 <== mon.5 v1:172.21.15.104:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92840048a0 con 0x7f928806e700 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f927f7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f928ec48640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f928ec48640 4 set_mon_vals no callback set 2024-08-19T12:57:41.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f927f7fe640 1 -- 172.21.15.26:0/4014336303 <== mon.5 v1:172.21.15.104:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9284004e50 con 0x7f928806e700 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f927f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f927f7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f927f7fe640 10 monclient: dump: 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.793 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f928ec48640 10 monclient: shutdown 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f928ec48640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f928ec48640 1 -- 172.21.15.26:0/4014336303 >> v1:172.21.15.104:6791/0 conn(0x7f928806e700 legacy=0x7f9288139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f928ec48640 1 -- 172.21.15.26:0/4014336303 shutdown_connections 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f928ec48640 1 -- 172.21.15.26:0/4014336303 >> 172.21.15.26:0/4014336303 conn(0x7f92880a86d0 msgr2=0x7f92880a8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f928ec48640 1 -- 172.21.15.26:0/4014336303 shutdown_connections 2024-08-19T12:57:41.794 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.786+0000 7f928ec48640 1 -- 172.21.15.26:0/4014336303 wait complete. 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.787+0000 7f928ec48640 10 monclient: build_initial_monmap 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.787+0000 7f928ec48640 10 monclient: monmap: 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:41.795 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.787+0000 7f928ec48640 1 librados: starting msgr at 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.787+0000 7f928ec48640 1 librados: starting objecter 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.787+0000 7f928ec48640 1 Processor -- start 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.787+0000 7f928ec48640 1 -- start start 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.787+0000 7f928ec48640 1 librados: setting wanted keys 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.787+0000 7f928ec48640 1 librados: calling monclient init 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.787+0000 7f928ec48640 10 monclient: init 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f928ec48640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f928ec48640 10 monclient: _add_conns ranks=[5,6,7,0,8,4,2,1,3] 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f928ec48640 10 monclient(hunting): picked mon.noname-h con 0x7f928806e700 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:41.796 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f928ec48640 10 monclient(hunting): picked mon.noname-d con 0x7f9288139b80 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f928ec48640 10 monclient(hunting): picked mon.noname-i con 0x7f9288131010 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f928ec48640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92881304b0 con 0x7f928806e700 2024-08-19T12:57:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f928ec48640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9288132fa0 con 0x7f9288139b80 2024-08-19T12:57:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f927ffff640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f9288139b80 legacy=0x7f92881d2aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f928ec48640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9288133170 con 0x7f9288131010 2024-08-19T12:57:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f928ec48640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f928ec48640 10 monclient(hunting): authenticate will time out at 1855.875140s 2024-08-19T12:57:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f928c9e4640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f928806e700 0x7f928812f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:50004/0 (socket says 172.21.15.26:50004) 2024-08-19T12:57:41.797 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.788+0000 7f928c9e4640 1 -- 172.21.15.26:0/2279815304 learned_addr learned my addr 172.21.15.26:0/2279815304 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:41.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f928c9e4640 1 -- 172.21.15.26:0/2279815304 >> v1:172.21.15.104:6791/0 conn(0x7f928806e700 legacy=0x7f928812f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f928d1e5640 1 -- 172.21.15.26:0/2279815304 >> v1:172.21.15.104:6792/0 conn(0x7f9288131010 legacy=0x7f92881d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 client.?.objecter ms_handle_connect 0x7f928806e700 2024-08-19T12:57:41.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:41.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 client.?.objecter ms_handle_connect 0x7f9288131010 2024-08-19T12:57:41.801 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443890209 0 0) 0x7f92881304b0 con 0x7f928806e700 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 monclient(hunting): my global_id is 184793 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9274004eb0 con 0x7f928806e700 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4268424342 0 0) 0x7f9288133170 con 0x7f9288131010 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 monclient(hunting): my global_id is 136573 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92881304b0 con 0x7f9288131010 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4000868260 0 0) 0x7f9274004eb0 con 0x7f928806e700 2024-08-19T12:57:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9288133170 con 0x7f928806e700 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9284001d30 con 0x7f928806e700 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 monclient(hunting): dump: 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.803 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.790+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1718532648 0 0) 0x7f92881304b0 con 0x7f9288131010 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9274004eb0 con 0x7f9288131010 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9280004840 con 0x7f9288131010 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 10 monclient(hunting): dump: 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.804 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.805 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 995567020 0 0) 0x7f9288133170 con 0x7f928806e700 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 >> v1:172.21.15.104:6792/0 conn(0x7f9288131010 legacy=0x7f92881d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 >> v1:172.21.15.26:6792/0 conn(0x7f9288139b80 legacy=0x7f92881d2aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 10 monclient: _finish_hunting 0 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 1 monclient: found mon.h 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9288133340 con 0x7f928806e700 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 10 monclient: _finish_auth 0 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 10 monclient: _check_auth_tickets 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 5 monclient: authenticate success, global_id 184793 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 10 monclient: _renew_subs 2024-08-19T12:57:41.806 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 1 -- 172.21.15.26:0/2279815304 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f928812fbe0 con 0x7f928806e700 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 10 client.184793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 10 monclient: _renew_subs 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 1 -- 172.21.15.26:0/2279815304 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f92881d78f0 con 0x7f928806e700 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 1 librados: init done 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92840048a0 con 0x7f928806e700 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9284005380 con 0x7f928806e700 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927effd640 4 set_mon_vals no callback set 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f927d7fa640 10 monclient: dump: 2024-08-19T12:57:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 10 client.184793.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 15 client.184793.objecter _session_command_op_assign -1 1 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 20 client.184793.objecter _get_session osd=-1 returning homeless 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.791+0000 7f928ec48640 10 client.184793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f928ec48640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f928ec48640 1 -- 172.21.15.26:0/2279815304 --> v1:172.21.15.104:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f9268002910 con 0x7f928806e700 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f92840117e0 con 0x7f928806e700 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7f928404e0d0 con 0x7f928806e700 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 10 client.184793.objecter ms_dispatch 0x7f9288000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 3 client.184793.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 3 client.184793.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 10 client.184793.objecter checking command 1 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 20 client.184793.objecter _get_session s=0x7f9274049510 osd=1 3 2024-08-19T12:57:41.809 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 20 client.184793.objecter put_session s=0x7f9274049510 osd=1 3 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 15 client.184793.objecter _session_command_op_remove -1 1 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 20 client.184793.objecter _get_session s=0x7f9274049510 osd=1 3 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 20 client.184793.objecter get_session s=0x7f9274049510 osd=1 3 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 15 client.184793.objecter _session_command_op_assign 1 1 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 20 client.184793.objecter put_session s=0x7f9274049510 osd=1 4 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 10 client.184793.objecter _send_command 1 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 --> v1:172.21.15.26:6804/2594673920 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9284001d30 con 0x7f9274049bc0 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 20 client.184793.objecter dump_active .. 0 homeless 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== mon.5 v1:172.21.15.104:6791/0 9 ==== mon_get_version_reply(handle=1 version=701) ==== 24+0+0 (unknown 2985145993 0 0) 0x7f928404e540 con 0x7f928806e700 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927d7fa640 10 monclient: handle_get_version_reply finishing 1 version 701 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f927ffff640 10 monclient: get_auth_request con 0x7f9274041640 auth_method 0 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.792+0000 7f928d1e5640 10 monclient: get_auth_request con 0x7f9274049bc0 auth_method 0 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.795+0000 7f927d7fa640 10 client.184793.objecter ms_handle_connect 0x7f9274041640 2024-08-19T12:57:41.810 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.797+0000 7f927d7fa640 10 client.184793.objecter ms_handle_connect 0x7f9274049bc0 2024-08-19T12:57:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.797+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== osd.1 v1:172.21.15.26:6804/2594673920 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9284001d30 con 0x7f9274049bc0 2024-08-19T12:57:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.797+0000 7f927d7fa640 10 client.184793.objecter ms_dispatch 0x7f9288000b90 command_reply(tid 1: 0 ) 2024-08-19T12:57:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.797+0000 7f927d7fa640 10 client.184793.objecter _finish_command 1 = system:0 2024-08-19T12:57:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.797+0000 7f927d7fa640 20 client.184793.objecter put_session s=0x7f9274049510 osd=1 4 2024-08-19T12:57:41.811 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.797+0000 7f927d7fa640 15 client.184793.objecter _session_command_op_remove 1 1 2024-08-19T12:57:41.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 10 client.184793.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-08-19T12:57:41.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 15 client.184793.objecter _session_command_op_assign -1 2 2024-08-19T12:57:41.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 20 client.184793.objecter _get_session s=0x7f9274049510 osd=1 3 2024-08-19T12:57:41.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 20 client.184793.objecter put_session s=0x7f9274049510 osd=1 3 2024-08-19T12:57:41.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 20 client.184793.objecter _get_session s=0x7f9274049510 osd=1 3 2024-08-19T12:57:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 15 client.184793.objecter _session_command_op_remove -1 2 2024-08-19T12:57:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 20 client.184793.objecter get_session s=0x7f9274049510 osd=1 3 2024-08-19T12:57:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 15 client.184793.objecter _session_command_op_assign 1 2 2024-08-19T12:57:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 20 client.184793.objecter put_session s=0x7f9274049510 osd=1 4 2024-08-19T12:57:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 10 client.184793.objecter _send_command 2 2024-08-19T12:57:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 1 -- 172.21.15.26:0/2279815304 --> v1:172.21.15.26:6804/2594673920 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f92680012b0 con 0x7f9274049bc0 2024-08-19T12:57:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f927d7fa640 1 -- 172.21.15.26:0/2279815304 <== osd.1 v1:172.21.15.26:6804/2594673920 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f92680012b0 con 0x7f9274049bc0 2024-08-19T12:57:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f927d7fa640 10 client.184793.objecter ms_dispatch 0x7f9288000b90 command_reply(tid 2: 0 ) 2024-08-19T12:57:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f927d7fa640 10 client.184793.objecter _finish_command 2 = system:0 2024-08-19T12:57:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f927d7fa640 20 client.184793.objecter put_session s=0x7f9274049510 osd=1 4 2024-08-19T12:57:41.813 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f927d7fa640 15 client.184793.objecter _session_command_op_remove 1 2 2024-08-19T12:57:41.814 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:57:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-19T12:57:41.814 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:57:41.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 10 librados: watch_flush enter 2024-08-19T12:57:41.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 10 librados: watch_flush exit 2024-08-19T12:57:41.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 10 client.184793.objecter close_session for osd.1 2024-08-19T12:57:41.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 1 -- 172.21.15.26:0/2279815304 >> v1:172.21.15.26:6804/2594673920 conn(0x7f9274049bc0 legacy=0x7f927404c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 20 client.184793.objecter put_session s=0x7f9274049510 osd=1 1 2024-08-19T12:57:41.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 20 client.184793.objecter shutdown clearing up homeless session... 2024-08-19T12:57:41.815 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.810+0000 7f928ec48640 10 client.184793.objecter successfully canceled tick 2024-08-19T12:57:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.811+0000 7f928ec48640 1 -- 172.21.15.26:0/2279815304 >> v1:172.21.15.104:6800/899887506 conn(0x7f9274041640 legacy=0x7f9274043b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.811+0000 7f928ec48640 10 monclient: shutdown 2024-08-19T12:57:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.811+0000 7f928ec48640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.811+0000 7f928ec48640 1 -- 172.21.15.26:0/2279815304 >> v1:172.21.15.104:6791/0 conn(0x7f928806e700 legacy=0x7f928812f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.811+0000 7f928d1e5640 1 -- 172.21.15.26:0/2279815304 reap_dead start 2024-08-19T12:57:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.811+0000 7f928ec48640 1 -- 172.21.15.26:0/2279815304 shutdown_connections 2024-08-19T12:57:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.811+0000 7f928ec48640 1 -- 172.21.15.26:0/2279815304 >> 172.21.15.26:0/2279815304 conn(0x7f92880a86d0 msgr2=0x7f9288137670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.811+0000 7f928ec48640 1 -- 172.21.15.26:0/2279815304 shutdown_connections 2024-08-19T12:57:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.811+0000 7f928ec48640 1 -- 172.21.15.26:0/2279815304 wait complete. 2024-08-19T12:57:41.816 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.811+0000 7f928ec48640 1 librados: shutdown 2024-08-19T12:57:41.821 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-08-19T12:57:41.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.920+0000 7f601b6fd640 10 monclient: get_monmap_and_config 2024-08-19T12:57:41.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.920+0000 7f601b6fd640 10 monclient: build_initial_monmap 2024-08-19T12:57:41.921 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.920+0000 7f601b6fd640 10 monclient: monmap: 2024-08-19T12:57:41.921 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:41.921 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.921 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:41.921528+0000 2024-08-19T12:57:41.921 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:41.921528+0000 2024-08-19T12:57:41.921 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:41.922 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.922 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:41.922 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:41.922 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:41.922 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:41.922 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:41.922 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:41.922 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:41.922 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:41.922 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:41.922 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.923 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 1 Processor -- start 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 1 -- start start 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 10 monclient: init 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 10 monclient: _add_conns ranks=[1,5,2,7,8,0,4,3,6] 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 10 monclient(hunting): picked mon.noname-f con 0x7f601406e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 10 monclient(hunting): picked mon.noname-h con 0x7f60140a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 10 monclient(hunting): picked mon.noname-b con 0x7f60140a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6014136790 con 0x7f601406e7b0 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601409ddf0 con 0x7f60140a5d50 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601409e0d0 con 0x7f60140a5920 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.921+0000 7f601b6fd640 10 monclient(hunting): authenticate will time out at 1856.008987s 2024-08-19T12:57:41.924 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.922+0000 7f6019c9a640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f60140a5d50 0x7f6014141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39126/0 (socket says 172.21.15.26:39126) 2024-08-19T12:57:41.925 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.922+0000 7f6019c9a640 1 -- 172.21.15.26:0/2318013745 learned_addr learned my addr 172.21.15.26:0/2318013745 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:41.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f6019c9a640 1 -- 172.21.15.26:0/2318013745 >> v1:172.21.15.26:6790/0 conn(0x7f60140a5d50 legacy=0x7f6014141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 1 -- 172.21.15.26:0/2318013745 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2622728541 0 0) 0x7f600c0017d0 con 0x7f60140a5d50 2024-08-19T12:57:41.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.928 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient(hunting): my global_id is 27533 2024-08-19T12:57:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 1 -- 172.21.15.26:0/2318013745 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6000003520 con 0x7f60140a5d50 2024-08-19T12:57:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f6018c98640 1 -- 172.21.15.26:0/2318013745 >> v1:172.21.15.104:6791/0 conn(0x7f60140a5920 legacy=0x7f601413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f6019499640 1 -- 172.21.15.26:0/2318013745 >> v1:172.21.15.104:6789/0 conn(0x7f601406e7b0 legacy=0x7f6014136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 1 -- 172.21.15.26:0/2318013745 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2432441338 0 0) 0x7f6000003520 con 0x7f60140a5d50 2024-08-19T12:57:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 1 -- 172.21.15.26:0/2318013745 >> v1:172.21.15.104:6791/0 conn(0x7f60140a5920 legacy=0x7f601413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 1 -- 172.21.15.26:0/2318013745 >> v1:172.21.15.104:6789/0 conn(0x7f601406e7b0 legacy=0x7f6014136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient: _finish_hunting 0 2024-08-19T12:57:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 1 monclient: found mon.noname-b 2024-08-19T12:57:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:57:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 1 -- 172.21.15.26:0/2318013745 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601409e3b0 con 0x7f60140a5d50 2024-08-19T12:57:41.929 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient: _finish_auth 0 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient: _check_auth_tickets 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 1 -- 172.21.15.26:0/2318013745 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f600c002030 con 0x7f60140a5d50 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f601b6fd640 5 monclient: authenticate success, global_id 27533 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f601b6fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient: dump: 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.930 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f601b6fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 1 -- 172.21.15.26:0/2318013745 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f600c004d10 con 0x7f60140a5d50 2024-08-19T12:57:41.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 1 -- 172.21.15.26:0/2318013745 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f600c005200 con 0x7f60140a5d50 2024-08-19T12:57:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:57:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f600bfff640 10 monclient: dump: 2024-08-19T12:57:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.932 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f601b6fd640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.924+0000 7f601b6fd640 4 set_mon_vals no callback set 2024-08-19T12:57:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.925+0000 7f601b6fd640 10 monclient: shutdown 2024-08-19T12:57:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.925+0000 7f601b6fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.925+0000 7f601b6fd640 1 -- 172.21.15.26:0/2318013745 >> v1:172.21.15.26:6790/0 conn(0x7f60140a5d50 legacy=0x7f6014141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.925+0000 7f601b6fd640 1 -- 172.21.15.26:0/2318013745 shutdown_connections 2024-08-19T12:57:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.925+0000 7f601b6fd640 1 -- 172.21.15.26:0/2318013745 >> 172.21.15.26:0/2318013745 conn(0x7f60140a8740 msgr2=0x7f60140a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.925+0000 7f601b6fd640 1 -- 172.21.15.26:0/2318013745 shutdown_connections 2024-08-19T12:57:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.925+0000 7f601b6fd640 1 -- 172.21.15.26:0/2318013745 wait complete. 2024-08-19T12:57:41.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.925+0000 7f601b6fd640 10 monclient: build_initial_monmap 2024-08-19T12:57:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.925+0000 7f601b6fd640 10 monclient: monmap: 2024-08-19T12:57:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:41.935 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:41.936 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 1 librados: starting msgr at 2024-08-19T12:57:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 1 librados: starting objecter 2024-08-19T12:57:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 1 Processor -- start 2024-08-19T12:57:41.936 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 1 -- start start 2024-08-19T12:57:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 1 librados: setting wanted keys 2024-08-19T12:57:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 1 librados: calling monclient init 2024-08-19T12:57:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 10 monclient: init 2024-08-19T12:57:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 10 monclient: _add_conns ranks=[1,6,5,7,3,4,2,0,8] 2024-08-19T12:57:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 10 monclient(hunting): picked mon.noname-f con 0x7f601406e7b0 addr v1:172.21.15.104:6789/0 2024-08-19T12:57:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 10 monclient(hunting): picked mon.noname-d con 0x7f60140a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 10 monclient(hunting): picked mon.noname-h con 0x7f60140a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60140a34d0 con 0x7f601406e7b0 2024-08-19T12:57:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60141349c0 con 0x7f60140a5d50 2024-08-19T12:57:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60140a1170 con 0x7f60140a5920 2024-08-19T12:57:41.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f601b6fd640 10 monclient(hunting): authenticate will time out at 1856.013733s 2024-08-19T12:57:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f6018c98640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f60140a5920 legacy=0x7f60141d36d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f6019c9a640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f60140a5d50 0x7f60141d6e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:50020/0 (socket says 172.21.15.26:50020) 2024-08-19T12:57:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f6019499640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f601406e7b0 0x7f60141ce690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:58374/0 (socket says 172.21.15.26:58374) 2024-08-19T12:57:41.938 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.926+0000 7f6019c9a640 1 -- 172.21.15.26:0/3173370576 learned_addr learned my addr 172.21.15.26:0/3173370576 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6019499640 1 -- 172.21.15.26:0/3173370576 >> v1:172.21.15.104:6789/0 conn(0x7f601406e7b0 legacy=0x7f60141ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 client.?.objecter ms_handle_connect 0x7f601406e7b0 2024-08-19T12:57:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 client.?.objecter ms_handle_connect 0x7f60140a5d50 2024-08-19T12:57:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6019c9a640 1 -- 172.21.15.26:0/3173370576 >> v1:172.21.15.104:6791/0 conn(0x7f60140a5d50 legacy=0x7f60141d6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1469596062 0 0) 0x7f60140a34d0 con 0x7f601406e7b0 2024-08-19T12:57:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.942 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): my global_id is 204400 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ff8004eb0 con 0x7f601406e7b0 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 184451781 0 0) 0x7f60141349c0 con 0x7f60140a5d50 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): my global_id is 184811 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60140a34d0 con 0x7f60140a5d50 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2713715336 0 0) 0x7f60140a34d0 con 0x7f60140a5d50 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60141349c0 con 0x7f60140a5d50 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f600c002030 con 0x7f60140a5d50 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:57:41.943 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): dump: 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.944 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 41224045 0 0) 0x7f5ff8004eb0 con 0x7f601406e7b0 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60140a34d0 con 0x7f601406e7b0 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6010003610 con 0x7f601406e7b0 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.929+0000 7f6009ffb640 10 monclient(hunting): dump: 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.945 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.946 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1612040903 0 0) 0x7f60141349c0 con 0x7f60140a5d50 2024-08-19T12:57:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 >> v1:172.21.15.26:6792/0 conn(0x7f60140a5920 legacy=0x7f60141d36d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 >> v1:172.21.15.104:6789/0 conn(0x7f601406e7b0 legacy=0x7f60141ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:57:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 1 monclient: found mon.h 2024-08-19T12:57:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:41.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60140a31f0 con 0x7f60140a5d50 2024-08-19T12:57:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 10 monclient: _finish_auth 0 2024-08-19T12:57:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:57:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f601b6fd640 5 monclient: authenticate success, global_id 184811 2024-08-19T12:57:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f601b6fd640 10 monclient: _renew_subs 2024-08-19T12:57:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f601b6fd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f601b6fd640 1 -- 172.21.15.26:0/3173370576 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f60140a5300 con 0x7f60140a5d50 2024-08-19T12:57:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f600c0051e0 con 0x7f60140a5d50 2024-08-19T12:57:41.948 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f600c005710 con 0x7f60140a5d50 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f6009ffb640 10 monclient: dump: 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:41.949 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:41.950 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f600b7fe640 4 set_mon_vals no callback set 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f601b6fd640 10 client.184811.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f601b6fd640 10 monclient: _renew_subs 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f601b6fd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f601b6fd640 1 -- 172.21.15.26:0/3173370576 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f60140a0910 con 0x7f60140a5d50 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.930+0000 7f601b6fd640 1 librados: init done 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.931+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f600c0058b0 con 0x7f60140a5d50 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.931+0000 7f601b6fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.931+0000 7f601b6fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.931+0000 7f601b6fd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:41.951 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.931+0000 7f601b6fd640 1 -- 172.21.15.26:0/3173370576 --> v1:172.21.15.104:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f601406bfa0 con 0x7f60140a5d50 2024-08-19T12:57:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.931+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7f600c04dfc0 con 0x7f60140a5d50 2024-08-19T12:57:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.931+0000 7f6009ffb640 10 client.184811.objecter ms_dispatch 0x7f6014000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.931+0000 7f6009ffb640 3 client.184811.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.931+0000 7f6009ffb640 3 client.184811.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.931+0000 7f6009ffb640 20 client.184811.objecter dump_active .. 0 homeless 2024-08-19T12:57:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.931+0000 7f6019499640 10 monclient: get_auth_request con 0x7f5ff8041490 auth_method 0 2024-08-19T12:57:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.933+0000 7f6009ffb640 10 client.184811.objecter ms_handle_connect 0x7f5ff8041490 2024-08-19T12:57:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.936+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mon.5 v1:172.21.15.104:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f601406bfa0 con 0x7f60140a5d50 2024-08-19T12:57:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.936+0000 7f6009ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:41.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:41.936+0000 7f6009ffb640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:57:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.120+0000 7f601b6fd640 1 -- 172.21.15.26:0/3173370576 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f60140a3eb0 con 0x7f5ff8041490 2024-08-19T12:57:42.121 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.121+0000 7f6009ffb640 1 -- 172.21.15.26:0/3173370576 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7f6010003b20 con 0x7f5ff8041490 2024-08-19T12:57:42.122 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 1 to deep-scrub 2024-08-19T12:57:42.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 10 librados: watch_flush enter 2024-08-19T12:57:42.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 10 librados: watch_flush exit 2024-08-19T12:57:42.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 20 client.184811.objecter shutdown clearing up homeless session... 2024-08-19T12:57:42.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 10 client.184811.objecter successfully canceled tick 2024-08-19T12:57:42.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 1 -- 172.21.15.26:0/3173370576 >> v1:172.21.15.104:6800/899887506 conn(0x7f5ff8041490 legacy=0x7f5ff8043950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 10 monclient: shutdown 2024-08-19T12:57:42.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:42.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 1 -- 172.21.15.26:0/3173370576 >> v1:172.21.15.104:6791/0 conn(0x7f60140a5d50 legacy=0x7f60141d6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.124 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 1 -- 172.21.15.26:0/3173370576 shutdown_connections 2024-08-19T12:57:42.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 1 -- 172.21.15.26:0/3173370576 >> 172.21.15.26:0/3173370576 conn(0x7f60140a8740 msgr2=0x7f60140aa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:42.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 1 -- 172.21.15.26:0/3173370576 shutdown_connections 2024-08-19T12:57:42.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 1 -- 172.21.15.26:0/3173370576 wait complete. 2024-08-19T12:57:42.125 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.123+0000 7f601b6fd640 1 librados: shutdown 2024-08-19T12:57:42.134 INFO:tasks.ceph:Scrubbing osd.2 2024-08-19T12:57:42.135 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-08-19T12:57:42.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.237+0000 7fc314ff2640 10 monclient: get_monmap_and_config 2024-08-19T12:57:42.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.237+0000 7fc314ff2640 10 monclient: build_initial_monmap 2024-08-19T12:57:42.238 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.237+0000 7fc314ff2640 10 monclient: monmap: 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:42.238774+0000 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:42.238774+0000 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:42.240 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:42.240 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.238+0000 7fc314ff2640 1 Processor -- start 2024-08-19T12:57:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.238+0000 7fc314ff2640 1 -- start start 2024-08-19T12:57:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.238+0000 7fc314ff2640 10 monclient: init 2024-08-19T12:57:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc314ff2640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc314ff2640 10 monclient: _add_conns ranks=[5,3,7,2,0,8,1,6,4] 2024-08-19T12:57:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc314ff2640 10 monclient(hunting): picked mon.noname-h con 0x7fc31006e700 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc314ff2640 10 monclient(hunting): picked mon.noname-g con 0x7fc310138fe0 addr v1:172.21.15.104:6790/0 2024-08-19T12:57:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc314ff2640 10 monclient(hunting): picked mon.noname-i con 0x7fc310139b80 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc314ff2640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3100a80f0 con 0x7fc310138fe0 2024-08-19T12:57:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc314ff2640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc31009de80 con 0x7fc31006e700 2024-08-19T12:57:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc314ff2640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc31009e160 con 0x7fc310139b80 2024-08-19T12:57:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc314ff2640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc314ff2640 10 monclient(hunting): authenticate will time out at 1856.326227s 2024-08-19T12:57:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc30dd9b640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7fc310138fe0 legacy=0x7fc31013e360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc30e59c640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7fc31006e700 0x7fc310139470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:50026/0 (socket says 172.21.15.26:50026) 2024-08-19T12:57:42.242 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc30e59c640 1 -- 172.21.15.26:0/4232527620 learned_addr learned my addr 172.21.15.26:0/4232527620 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.239+0000 7fc30ed9d640 1 --1- 172.21.15.26:0/4232527620 >> v1:172.21.15.104:6792/0 conn(0x7fc310139b80 0x7fc310141a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:42084/0 (socket says 172.21.15.26:42084) 2024-08-19T12:57:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.241+0000 7fc30ed9d640 1 -- 172.21.15.26:0/4232527620 >> v1:172.21.15.104:6792/0 conn(0x7fc310139b80 legacy=0x7fc310141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:42.243 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.241+0000 7fc30e59c640 1 -- 172.21.15.26:0/4232527620 >> v1:172.21.15.104:6791/0 conn(0x7fc31006e700 legacy=0x7fc310139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:42.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.440+0000 7fc30dd9b640 1 -- 172.21.15.26:0/4232527620 >> v1:172.21.15.104:6790/0 conn(0x7fc310138fe0 legacy=0x7fc31013e360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.575+0000 7fc30d59a640 1 -- 172.21.15.26:0/4232527620 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085452436 0 0) 0x7fc31009e160 con 0x7fc310139b80 2024-08-19T12:57:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.575+0000 7fc30d59a640 10 monclient(hunting): _negotiate 2024-08-19T12:57:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.575+0000 7fc30d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.575+0000 7fc30d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.575+0000 7fc30d59a640 10 monclient(hunting): my global_id is 136582 2024-08-19T12:57:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.575+0000 7fc30d59a640 1 -- 172.21.15.26:0/4232527620 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2fc003520 con 0x7fc310139b80 2024-08-19T12:57:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.575+0000 7fc30d59a640 1 -- 172.21.15.26:0/4232527620 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2995133666 0 0) 0x7fc31009de80 con 0x7fc31006e700 2024-08-19T12:57:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.575+0000 7fc30d59a640 10 monclient(hunting): _negotiate 2024-08-19T12:57:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.575+0000 7fc30d59a640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:42.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.575+0000 7fc30d59a640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:42.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.575+0000 7fc30d59a640 10 monclient(hunting): my global_id is 184820 2024-08-19T12:57:42.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.575+0000 7fc30d59a640 1 -- 172.21.15.26:0/4232527620 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc31009e160 con 0x7fc31006e700 2024-08-19T12:57:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 1 -- 172.21.15.26:0/4232527620 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1461536232 0 0) 0x7fc2fc003520 con 0x7fc310139b80 2024-08-19T12:57:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 1 -- 172.21.15.26:0/4232527620 >> v1:172.21.15.104:6791/0 conn(0x7fc31006e700 legacy=0x7fc310139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 1 -- 172.21.15.26:0/4232527620 >> v1:172.21.15.104:6790/0 conn(0x7fc310138fe0 legacy=0x7fc31013e360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 10 monclient: _finish_hunting 0 2024-08-19T12:57:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 1 monclient: found mon.noname-i 2024-08-19T12:57:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:57:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 1 -- 172.21.15.26:0/4232527620 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31009e440 con 0x7fc310139b80 2024-08-19T12:57:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 10 monclient: _finish_auth 0 2024-08-19T12:57:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 10 monclient: _check_auth_tickets 2024-08-19T12:57:42.593 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 1 -- 172.21.15.26:0/4232527620 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc300001ae0 con 0x7fc310139b80 2024-08-19T12:57:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:57:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc30d59a640 10 monclient: dump: 2024-08-19T12:57:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:42.594 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:42.595 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:42.596 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc314ff2640 5 monclient: authenticate success, global_id 136582 2024-08-19T12:57:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.591+0000 7fc314ff2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.592+0000 7fc30d59a640 1 -- 172.21.15.26:0/4232527620 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3000047e0 con 0x7fc310139b80 2024-08-19T12:57:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.592+0000 7fc30d59a640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.592+0000 7fc30d59a640 1 -- 172.21.15.26:0/4232527620 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc300004d50 con 0x7fc310139b80 2024-08-19T12:57:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.592+0000 7fc30d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.592+0000 7fc30d59a640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:57:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.592+0000 7fc30d59a640 10 monclient: dump: 2024-08-19T12:57:42.596 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:42.597 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:42.598 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.592+0000 7fc314ff2640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.592+0000 7fc314ff2640 4 set_mon_vals no callback set 2024-08-19T12:57:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.593+0000 7fc314ff2640 10 monclient: shutdown 2024-08-19T12:57:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.593+0000 7fc314ff2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.593+0000 7fc314ff2640 1 -- 172.21.15.26:0/4232527620 >> v1:172.21.15.104:6792/0 conn(0x7fc310139b80 legacy=0x7fc310141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.593+0000 7fc314ff2640 1 -- 172.21.15.26:0/4232527620 shutdown_connections 2024-08-19T12:57:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.593+0000 7fc314ff2640 1 -- 172.21.15.26:0/4232527620 >> 172.21.15.26:0/4232527620 conn(0x7fc3100a86d0 msgr2=0x7fc3100a8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.593+0000 7fc314ff2640 1 -- 172.21.15.26:0/4232527620 shutdown_connections 2024-08-19T12:57:42.605 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.593+0000 7fc314ff2640 1 -- 172.21.15.26:0/4232527620 wait complete. 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.594+0000 7fc314ff2640 10 monclient: build_initial_monmap 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.594+0000 7fc314ff2640 10 monclient: monmap: 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:42.606 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:42.607 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.595+0000 7fc314ff2640 1 librados: starting msgr at 2024-08-19T12:57:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.595+0000 7fc314ff2640 1 librados: starting objecter 2024-08-19T12:57:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.595+0000 7fc314ff2640 1 Processor -- start 2024-08-19T12:57:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.595+0000 7fc314ff2640 1 -- start start 2024-08-19T12:57:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.595+0000 7fc314ff2640 1 librados: setting wanted keys 2024-08-19T12:57:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.595+0000 7fc314ff2640 1 librados: calling monclient init 2024-08-19T12:57:42.607 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.595+0000 7fc314ff2640 10 monclient: init 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.596+0000 7fc314ff2640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.596+0000 7fc314ff2640 10 monclient: _add_conns ranks=[4,1,7,3,6,5,8,2,0] 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.596+0000 7fc314ff2640 10 monclient(hunting): picked mon.noname-c con 0x7fc31006e700 addr v1:172.21.15.26:6791/0 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.596+0000 7fc314ff2640 10 monclient(hunting): picked mon.noname-f con 0x7fc310139b80 addr v1:172.21.15.104:6789/0 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.596+0000 7fc314ff2640 10 monclient(hunting): picked mon.noname-i con 0x7fc3101d3e30 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.596+0000 7fc314ff2640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc310132490 con 0x7fc310139b80 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.596+0000 7fc314ff2640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc310132770 con 0x7fc31006e700 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.596+0000 7fc314ff2640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3101ce640 con 0x7fc3101d3e30 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.596+0000 7fc314ff2640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.596+0000 7fc314ff2640 10 monclient(hunting): authenticate will time out at 1856.683309s 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.596+0000 7fc30e59c640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fc31006e700 0x7fc3101cd820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36750/0 (socket says 172.21.15.26:36750) 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.596+0000 7fc30e59c640 1 -- 172.21.15.26:0/2119686355 learned_addr learned my addr 172.21.15.26:0/2119686355 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc30e59c640 1 -- 172.21.15.26:0/2119686355 >> v1:172.21.15.26:6791/0 conn(0x7fc31006e700 legacy=0x7fc3101cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:42.608 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 client.?.objecter ms_handle_connect 0x7fc31006e700 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc30ed9d640 1 -- 172.21.15.26:0/2119686355 >> v1:172.21.15.104:6792/0 conn(0x7fc3101d3e30 legacy=0x7fc3101d6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 client.?.objecter ms_handle_connect 0x7fc3101d3e30 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 client.?.objecter ms_handle_connect 0x7fc310139b80 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc30dd9b640 1 -- 172.21.15.26:0/2119686355 >> v1:172.21.15.104:6789/0 conn(0x7fc310139b80 legacy=0x7fc3101cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3630111088 0 0) 0x7fc310132770 con 0x7fc31006e700 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 monclient(hunting): my global_id is 184837 2024-08-19T12:57:42.609 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2f0004eb0 con 0x7fc31006e700 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2343959787 0 0) 0x7fc3101ce640 con 0x7fc3101d3e30 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 monclient(hunting): my global_id is 136591 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc310132770 con 0x7fc3101d3e30 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2831333184 0 0) 0x7fc310132490 con 0x7fc310139b80 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 10 monclient(hunting): my global_id is 204409 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.598+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3101ce640 con 0x7fc310139b80 2024-08-19T12:57:42.610 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 77615949 0 0) 0x7fc2f0004eb0 con 0x7fc31006e700 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc310132490 con 0x7fc31006e700 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc308002010 con 0x7fc31006e700 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient(hunting): dump: 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:42.612 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3179682891 0 0) 0x7fc310132770 con 0x7fc3101d3e30 2024-08-19T12:57:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2f0004eb0 con 0x7fc3101d3e30 2024-08-19T12:57:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc300004cd0 con 0x7fc3101d3e30 2024-08-19T12:57:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-08-19T12:57:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient(hunting): dump: 2024-08-19T12:57:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:42.613 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1317347915 0 0) 0x7fc3101ce640 con 0x7fc310139b80 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc310132770 con 0x7fc310139b80 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc2f8004660 con 0x7fc310139b80 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:57:42.614 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient(hunting): dump: 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:42.615 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2638866470 0 0) 0x7fc310132490 con 0x7fc31006e700 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 >> v1:172.21.15.104:6792/0 conn(0x7fc3101d3e30 legacy=0x7fc3101d6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 >> v1:172.21.15.104:6789/0 conn(0x7fc310139b80 legacy=0x7fc3101cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 monclient: found mon.c 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31006e040 con 0x7fc31006e700 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient: _finish_auth 0 2024-08-19T12:57:42.616 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:57:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc314ff2640 5 monclient: authenticate success, global_id 184837 2024-08-19T12:57:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc314ff2640 10 monclient: _renew_subs 2024-08-19T12:57:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc314ff2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:57:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc314ff2640 1 -- 172.21.15.26:0/2119686355 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc310138b60 con 0x7fc31006e700 2024-08-19T12:57:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc314ff2640 10 client.184837.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc314ff2640 10 monclient: _renew_subs 2024-08-19T12:57:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc314ff2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:57:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc314ff2640 1 -- 172.21.15.26:0/2119686355 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3101321b0 con 0x7fc31006e700 2024-08-19T12:57:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc314ff2640 1 librados: init done 2024-08-19T12:57:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc308004e90 con 0x7fc31006e700 2024-08-19T12:57:42.617 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fc3080053c0 con 0x7fc31006e700 2024-08-19T12:57:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:57:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc30cd99640 4 set_mon_vals no callback set 2024-08-19T12:57:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.599+0000 7fc306ffd640 10 monclient: dump: 2024-08-19T12:57:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:42.618 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:42.619 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.600+0000 7fc314ff2640 10 client.184837.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.600+0000 7fc314ff2640 15 client.184837.objecter _session_command_op_assign -1 1 2024-08-19T12:57:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.600+0000 7fc314ff2640 20 client.184837.objecter _get_session osd=-1 returning homeless 2024-08-19T12:57:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.600+0000 7fc314ff2640 10 client.184837.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.600+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fc308011820 con 0x7fc31006e700 2024-08-19T12:57:42.620 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.600+0000 7fc314ff2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.600+0000 7fc314ff2640 1 -- 172.21.15.26:0/2119686355 --> v1:172.21.15.26:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fc2f4002910 con 0x7fc31006e700 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7fc30804e1c0 con 0x7fc31006e700 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 10 client.184837.objecter ms_dispatch 0x7fc310000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 3 client.184837.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 3 client.184837.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc30dd9b640 10 monclient: get_auth_request con 0x7fc2f0041970 auth_method 0 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 10 client.184837.objecter checking command 1 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 20 client.184837.objecter _get_session s=0x7fc2f0049780 osd=2 3 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 20 client.184837.objecter put_session s=0x7fc2f0049780 osd=2 3 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 15 client.184837.objecter _session_command_op_remove -1 1 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 20 client.184837.objecter _get_session s=0x7fc2f0049780 osd=2 3 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 20 client.184837.objecter get_session s=0x7fc2f0049780 osd=2 3 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 15 client.184837.objecter _session_command_op_assign 2 1 2024-08-19T12:57:42.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 20 client.184837.objecter put_session s=0x7fc2f0049780 osd=2 4 2024-08-19T12:57:42.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 10 client.184837.objecter _send_command 1 2024-08-19T12:57:42.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 --> v1:172.21.15.26:6808/1443966639 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2f004db40 con 0x7fc2f0049e30 2024-08-19T12:57:42.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 20 client.184837.objecter dump_active .. 0 homeless 2024-08-19T12:57:42.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== mon.4 v1:172.21.15.26:6791/0 9 ==== mon_get_version_reply(handle=1 version=701) ==== 24+0+0 (unknown 2985145993 0 0) 0x7fc30804e680 con 0x7fc31006e700 2024-08-19T12:57:42.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc306ffd640 10 monclient: handle_get_version_reply finishing 1 version 701 2024-08-19T12:57:42.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.601+0000 7fc30ed9d640 10 monclient: get_auth_request con 0x7fc2f0049e30 auth_method 0 2024-08-19T12:57:42.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.604+0000 7fc306ffd640 10 client.184837.objecter ms_handle_connect 0x7fc2f0041970 2024-08-19T12:57:42.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.606+0000 7fc306ffd640 10 client.184837.objecter ms_handle_connect 0x7fc2f0049e30 2024-08-19T12:57:42.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.606+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== osd.2 v1:172.21.15.26:6808/1443966639 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc2f004db40 con 0x7fc2f0049e30 2024-08-19T12:57:42.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.606+0000 7fc306ffd640 10 client.184837.objecter ms_dispatch 0x7fc310000b90 command_reply(tid 1: 0 ) 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.606+0000 7fc306ffd640 10 client.184837.objecter _finish_command 1 = system:0 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.606+0000 7fc306ffd640 20 client.184837.objecter put_session s=0x7fc2f0049780 osd=2 4 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.606+0000 7fc306ffd640 15 client.184837.objecter _session_command_op_remove 2 1 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc314ff2640 10 client.184837.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc314ff2640 15 client.184837.objecter _session_command_op_assign -1 2 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc314ff2640 20 client.184837.objecter _get_session s=0x7fc2f0049780 osd=2 3 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc314ff2640 20 client.184837.objecter put_session s=0x7fc2f0049780 osd=2 3 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc314ff2640 20 client.184837.objecter _get_session s=0x7fc2f0049780 osd=2 3 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc314ff2640 15 client.184837.objecter _session_command_op_remove -1 2 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc314ff2640 20 client.184837.objecter get_session s=0x7fc2f0049780 osd=2 3 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc314ff2640 15 client.184837.objecter _session_command_op_assign 2 2 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc314ff2640 20 client.184837.objecter put_session s=0x7fc2f0049780 osd=2 4 2024-08-19T12:57:42.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc314ff2640 10 client.184837.objecter _send_command 2 2024-08-19T12:57:42.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc314ff2640 1 -- 172.21.15.26:0/2119686355 --> v1:172.21.15.26:6808/1443966639 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc2f40012b0 con 0x7fc2f0049e30 2024-08-19T12:57:42.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc306ffd640 1 -- 172.21.15.26:0/2119686355 <== osd.2 v1:172.21.15.26:6808/1443966639 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fc2f40012b0 con 0x7fc2f0049e30 2024-08-19T12:57:42.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc306ffd640 10 client.184837.objecter ms_dispatch 0x7fc310000b90 command_reply(tid 2: 0 ) 2024-08-19T12:57:42.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc306ffd640 10 client.184837.objecter _finish_command 2 = system:0 2024-08-19T12:57:42.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc306ffd640 20 client.184837.objecter put_session s=0x7fc2f0049780 osd=2 4 2024-08-19T12:57:42.625 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.620+0000 7fc306ffd640 15 client.184837.objecter _session_command_op_remove 2 2 2024-08-19T12:57:42.625 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:57:42.625 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-19T12:57:42.625 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:57:42.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 10 librados: watch_flush enter 2024-08-19T12:57:42.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 10 librados: watch_flush exit 2024-08-19T12:57:42.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 10 client.184837.objecter close_session for osd.2 2024-08-19T12:57:42.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 1 -- 172.21.15.26:0/2119686355 >> v1:172.21.15.26:6808/1443966639 conn(0x7fc2f0049e30 legacy=0x7fc2f004c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 20 client.184837.objecter put_session s=0x7fc2f0049780 osd=2 1 2024-08-19T12:57:42.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 20 client.184837.objecter shutdown clearing up homeless session... 2024-08-19T12:57:42.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 10 client.184837.objecter successfully canceled tick 2024-08-19T12:57:42.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 1 -- 172.21.15.26:0/2119686355 >> v1:172.21.15.104:6800/899887506 conn(0x7fc2f0041970 legacy=0x7fc2f0043e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.627 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 10 monclient: shutdown 2024-08-19T12:57:42.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:42.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 1 -- 172.21.15.26:0/2119686355 >> v1:172.21.15.26:6791/0 conn(0x7fc31006e700 legacy=0x7fc3101cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc30ed9d640 1 -- 172.21.15.26:0/2119686355 reap_dead start 2024-08-19T12:57:42.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 1 -- 172.21.15.26:0/2119686355 shutdown_connections 2024-08-19T12:57:42.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 1 -- 172.21.15.26:0/2119686355 >> 172.21.15.26:0/2119686355 conn(0x7fc3100a86d0 msgr2=0x7fc3100a9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:42.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 1 -- 172.21.15.26:0/2119686355 shutdown_connections 2024-08-19T12:57:42.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 1 -- 172.21.15.26:0/2119686355 wait complete. 2024-08-19T12:57:42.628 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.621+0000 7fc314ff2640 1 librados: shutdown 2024-08-19T12:57:42.632 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-08-19T12:57:42.732 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.732+0000 7fb4dc3d9640 10 monclient: get_monmap_and_config 2024-08-19T12:57:42.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.732+0000 7fb4dc3d9640 10 monclient: build_initial_monmap 2024-08-19T12:57:42.733 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.732+0000 7fb4dc3d9640 10 monclient: monmap: 2024-08-19T12:57:42.733 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:42.733 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:42.733 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:42.733065+0000 2024-08-19T12:57:42.733 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:42.733065+0000 2024-08-19T12:57:42.733 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:42.733 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.733 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:42.733 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:42.733 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:42.734 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:42.734 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:42.734 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:42.734 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:42.734 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:42.734 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:42.734 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.732+0000 7fb4dc3d9640 1 Processor -- start 2024-08-19T12:57:42.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.732+0000 7fb4dc3d9640 1 -- start start 2024-08-19T12:57:42.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.732+0000 7fb4dc3d9640 10 monclient: init 2024-08-19T12:57:42.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4dc3d9640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4dc3d9640 10 monclient: _add_conns ranks=[6,0,2,4,5,7,1,8,3] 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4dc3d9640 10 monclient(hunting): picked mon.noname-d con 0x7fb4d406e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4dc3d9640 10 monclient(hunting): picked mon.noname-a con 0x7fb4d40a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4dc3d9640 10 monclient(hunting): picked mon.noname-b con 0x7fb4d40a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4dc3d9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4d4136790 con 0x7fb4d40a5920 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4dc3d9640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4d409ddf0 con 0x7fb4d40a5d50 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4dc3d9640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4d409e0d0 con 0x7fb4d406e7b0 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4dc3d9640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4dc3d9640 10 monclient(hunting): authenticate will time out at 1856.820499s 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4da175640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fb4d406e7b0 legacy=0x7fb4d4136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4d9974640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fb4d40a5920 legacy=0x7fb4d413e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4da976640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fb4d40a5d50 0x7fb4d4141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39138/0 (socket says 172.21.15.26:39138) 2024-08-19T12:57:42.736 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.733+0000 7fb4da976640 1 -- 172.21.15.26:0/84511848 learned_addr learned my addr 172.21.15.26:0/84511848 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.735+0000 7fb4da976640 1 -- 172.21.15.26:0/84511848 >> v1:172.21.15.26:6790/0 conn(0x7fb4d40a5d50 legacy=0x7fb4d4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.735+0000 7fb4d9173640 1 -- 172.21.15.26:0/84511848 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2204756120 0 0) 0x7fb4cc001450 con 0x7fb4d40a5d50 2024-08-19T12:57:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.735+0000 7fb4d9173640 10 monclient(hunting): _negotiate 2024-08-19T12:57:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.735+0000 7fb4d9173640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.735+0000 7fb4d9173640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.735+0000 7fb4d9173640 10 monclient(hunting): my global_id is 27542 2024-08-19T12:57:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.735+0000 7fb4d9173640 1 -- 172.21.15.26:0/84511848 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4bc003520 con 0x7fb4d40a5d50 2024-08-19T12:57:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 1 -- 172.21.15.26:0/84511848 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4140936296 0 0) 0x7fb4bc003520 con 0x7fb4d40a5d50 2024-08-19T12:57:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 1 -- 172.21.15.26:0/84511848 >> v1:172.21.15.26:6792/0 conn(0x7fb4d406e7b0 legacy=0x7fb4d4136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:42.740 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 1 -- 172.21.15.26:0/84511848 >> v1:172.21.15.26:6789/0 conn(0x7fb4d40a5920 legacy=0x7fb4d413e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 10 monclient: _finish_hunting 0 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 1 monclient: found mon.noname-b 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 1 -- 172.21.15.26:0/84511848 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d409e3b0 con 0x7fb4d40a5d50 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 10 monclient: _finish_auth 0 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 10 monclient: _check_auth_tickets 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 1 -- 172.21.15.26:0/84511848 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb4cc001cb0 con 0x7fb4d40a5d50 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 10 monclient: dump: 2024-08-19T12:57:42.741 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:42.742 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4dc3d9640 5 monclient: authenticate success, global_id 27542 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4dc3d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 1 -- 172.21.15.26:0/84511848 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb4cc004990 con 0x7fb4d40a5d50 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 1 -- 172.21.15.26:0/84511848 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb4cc004ec0 con 0x7fb4d40a5d50 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4d9173640 10 monclient: dump: 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:42.743 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.744 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4dc3d9640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4dc3d9640 4 set_mon_vals no callback set 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4dc3d9640 10 monclient: shutdown 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4dc3d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/84511848 >> v1:172.21.15.26:6790/0 conn(0x7fb4d40a5d50 legacy=0x7fb4d4141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/84511848 shutdown_connections 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/84511848 >> 172.21.15.26:0/84511848 conn(0x7fb4d40a8740 msgr2=0x7fb4d40a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/84511848 shutdown_connections 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.736+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/84511848 wait complete. 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.737+0000 7fb4dc3d9640 10 monclient: build_initial_monmap 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.737+0000 7fb4dc3d9640 10 monclient: monmap: 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:42.745 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:42.746 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.737+0000 7fb4dc3d9640 1 librados: starting msgr at 2024-08-19T12:57:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.737+0000 7fb4dc3d9640 1 librados: starting objecter 2024-08-19T12:57:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.737+0000 7fb4dc3d9640 1 Processor -- start 2024-08-19T12:57:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.737+0000 7fb4dc3d9640 1 -- start start 2024-08-19T12:57:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.737+0000 7fb4dc3d9640 1 librados: setting wanted keys 2024-08-19T12:57:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.737+0000 7fb4dc3d9640 1 librados: calling monclient init 2024-08-19T12:57:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.737+0000 7fb4dc3d9640 10 monclient: init 2024-08-19T12:57:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4dc3d9640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4dc3d9640 10 monclient: _add_conns ranks=[5,1,0,6,8,4,3,7,2] 2024-08-19T12:57:42.747 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4dc3d9640 10 monclient(hunting): picked mon.noname-h con 0x7fb4d406e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4dc3d9640 10 monclient(hunting): picked mon.noname-f con 0x7fb4d40a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:57:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4dc3d9640 10 monclient(hunting): picked mon.noname-a con 0x7fb4d40a5d50 addr v1:172.21.15.26:6789/0 2024-08-19T12:57:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4dc3d9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4d413b290 con 0x7fb4d40a5d50 2024-08-19T12:57:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4dc3d9640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4d4134550 con 0x7fb4d40a5920 2024-08-19T12:57:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4dc3d9640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4d4138f30 con 0x7fb4d406e7b0 2024-08-19T12:57:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4dc3d9640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4dc3d9640 10 monclient(hunting): authenticate will time out at 1856.825468s 2024-08-19T12:57:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4da976640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fb4d40a5d50 legacy=0x7fb4d41383f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:57:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4da175640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7fb4d406e7b0 0x7fb4d41375d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:50030/0 (socket says 172.21.15.26:50030) 2024-08-19T12:57:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4d9974640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb4d40a5920 0x7fb4d4137ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:58392/0 (socket says 172.21.15.26:58392) 2024-08-19T12:57:42.748 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.738+0000 7fb4da175640 1 -- 172.21.15.26:0/987595064 learned_addr learned my addr 172.21.15.26:0/987595064 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.740+0000 7fb4da175640 1 -- 172.21.15.26:0/987595064 >> v1:172.21.15.104:6791/0 conn(0x7fb4d406e7b0 legacy=0x7fb4d41375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.740+0000 7fb4d2ffd640 10 client.?.objecter ms_handle_connect 0x7fb4d406e7b0 2024-08-19T12:57:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.740+0000 7fb4d2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:42.751 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.740+0000 7fb4d9974640 1 -- 172.21.15.26:0/987595064 >> v1:172.21.15.104:6789/0 conn(0x7fb4d40a5920 legacy=0x7fb4d4137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.740+0000 7fb4d2ffd640 10 client.?.objecter ms_handle_connect 0x7fb4d40a5920 2024-08-19T12:57:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.740+0000 7fb4d2ffd640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3785851373 0 0) 0x7fb4d4138f30 con 0x7fb4d406e7b0 2024-08-19T12:57:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:57:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d9974640 1 --1- 172.21.15.26:0/987595064 >> v1:172.21.15.104:6789/0 conn(0x7fb4d40a5920 0x7fb4d4137ce0 :-1 s=OPENED pgs=3871 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fb4d4134550 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:57:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient(hunting): my global_id is 184829 2024-08-19T12:57:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4c0004ee0 con 0x7fb4d406e7b0 2024-08-19T12:57:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 532015155 0 0) 0x7fb4c0004ee0 con 0x7fb4d406e7b0 2024-08-19T12:57:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4d4138f30 con 0x7fb4d406e7b0 2024-08-19T12:57:42.752 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb4c4004a60 con 0x7fb4d406e7b0 2024-08-19T12:57:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:57:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient(hunting): dump: 2024-08-19T12:57:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.753 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:42.754 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:42.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3286818641 0 0) 0x7fb4d4138f30 con 0x7fb4d406e7b0 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 >> v1:172.21.15.104:6789/0 conn(0x7fb4d40a5920 legacy=0x7fb4d4137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 >> v1:172.21.15.26:6789/0 conn(0x7fb4d40a5d50 legacy=0x7fb4d41383f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 monclient: found mon.h 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d413a9f0 con 0x7fb4d406e7b0 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient: _finish_auth 0 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4dc3d9640 5 monclient: authenticate success, global_id 184829 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4dc3d9640 10 monclient: _renew_subs 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4dc3d9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:42.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 214803270 0 0) 0x7fb4d4134550 con 0x7fb4d40a5920 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/987595064 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb4d4139d00 con 0x7fb4d406e7b0 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4dc3d9640 10 client.184829.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4dc3d9640 10 monclient: _renew_subs 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4dc3d9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/987595064 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fb4d41d7480 con 0x7fb4d406e7b0 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4dc3d9640 1 librados: init done 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb4c4004c80 con 0x7fb4d406e7b0 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fb4c4005230 con 0x7fb4d406e7b0 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d8972640 4 set_mon_vals no callback set 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:57:42.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.741+0000 7fb4d2ffd640 10 monclient: dump: 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:42.757 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.742+0000 7fb4dc3d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.742+0000 7fb4dc3d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.742+0000 7fb4dc3d9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.742+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/987595064 --> v1:172.21.15.104:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4b4005080 con 0x7fb4d406e7b0 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.742+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fb4c4005450 con 0x7fb4d406e7b0 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.743+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7fb4c404e4b0 con 0x7fb4d406e7b0 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.743+0000 7fb4d2ffd640 10 client.184829.objecter ms_dispatch 0x7fb4d4000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.743+0000 7fb4d2ffd640 3 client.184829.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.743+0000 7fb4d2ffd640 3 client.184829.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.743+0000 7fb4d2ffd640 20 client.184829.objecter dump_active .. 0 homeless 2024-08-19T12:57:42.758 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.743+0000 7fb4d9974640 10 monclient: get_auth_request con 0x7fb4c0040dd0 auth_method 0 2024-08-19T12:57:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.745+0000 7fb4d2ffd640 10 client.184829.objecter ms_handle_connect 0x7fb4c0040dd0 2024-08-19T12:57:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.747+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 <== mon.5 v1:172.21.15.104:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fb4c4017af0 con 0x7fb4d406e7b0 2024-08-19T12:57:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.747+0000 7fb4d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.747+0000 7fb4d2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:57:42.930 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.930+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/987595064 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fb4b4000b90 con 0x7fb4c0040dd0 2024-08-19T12:57:42.931 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.930+0000 7fb4d2ffd640 1 -- 172.21.15.26:0/987595064 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7fb4b4000b90 con 0x7fb4c0040dd0 2024-08-19T12:57:42.931 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 2 to deep-scrub 2024-08-19T12:57:42.933 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 10 librados: watch_flush enter 2024-08-19T12:57:42.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 10 librados: watch_flush exit 2024-08-19T12:57:42.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 20 client.184829.objecter shutdown clearing up homeless session... 2024-08-19T12:57:42.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 10 client.184829.objecter successfully canceled tick 2024-08-19T12:57:42.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/987595064 >> v1:172.21.15.104:6800/899887506 conn(0x7fb4c0040dd0 legacy=0x7fb4c0043290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 10 monclient: shutdown 2024-08-19T12:57:42.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:42.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/987595064 >> v1:172.21.15.104:6791/0 conn(0x7fb4d406e7b0 legacy=0x7fb4d41375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:42.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/987595064 shutdown_connections 2024-08-19T12:57:42.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/987595064 >> 172.21.15.26:0/987595064 conn(0x7fb4d40a8740 msgr2=0x7fb4d40ab000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:42.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/987595064 shutdown_connections 2024-08-19T12:57:42.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 1 -- 172.21.15.26:0/987595064 wait complete. 2024-08-19T12:57:42.934 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:42.932+0000 7fb4dc3d9640 1 librados: shutdown 2024-08-19T12:57:42.944 INFO:tasks.ceph:Scrubbing osd.3 2024-08-19T12:57:42.944 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-08-19T12:57:43.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.045+0000 7f32dff7f640 10 monclient: get_monmap_and_config 2024-08-19T12:57:43.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.045+0000 7f32dff7f640 10 monclient: build_initial_monmap 2024-08-19T12:57:43.046 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.045+0000 7f32dff7f640 10 monclient: monmap: 2024-08-19T12:57:43.046 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:43.046 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:43.046599+0000 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:43.046599+0000 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:43.047 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:43.048 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:43.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.046+0000 7f32dff7f640 1 Processor -- start 2024-08-19T12:57:43.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.046+0000 7f32dff7f640 1 -- start start 2024-08-19T12:57:43.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.046+0000 7f32dff7f640 10 monclient: init 2024-08-19T12:57:43.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.046+0000 7f32dff7f640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:43.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.046+0000 7f32dff7f640 10 monclient: _add_conns ranks=[4,3,1,5,6,7,8,0,2] 2024-08-19T12:57:43.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.046+0000 7f32dff7f640 10 monclient(hunting): picked mon.noname-c con 0x7f32d806e700 addr v1:172.21.15.26:6791/0 2024-08-19T12:57:43.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.047+0000 7f32dff7f640 10 monclient(hunting): picked mon.noname-g con 0x7f32d8138fe0 addr v1:172.21.15.104:6790/0 2024-08-19T12:57:43.049 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.047+0000 7f32dff7f640 10 monclient(hunting): picked mon.noname-f con 0x7f32d8139b80 addr v1:172.21.15.104:6789/0 2024-08-19T12:57:43.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.047+0000 7f32dff7f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d80a80f0 con 0x7f32d8139b80 2024-08-19T12:57:43.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.047+0000 7f32dff7f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d809de80 con 0x7f32d8138fe0 2024-08-19T12:57:43.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.047+0000 7f32dff7f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d809e160 con 0x7f32d806e700 2024-08-19T12:57:43.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.047+0000 7f32dff7f640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:43.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.047+0000 7f32dff7f640 10 monclient(hunting): authenticate will time out at 1857.134054s 2024-08-19T12:57:43.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.047+0000 7f32ddd1b640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f32d806e700 0x7f32d8139470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36756/0 (socket says 172.21.15.26:36756) 2024-08-19T12:57:43.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.047+0000 7f32ddd1b640 1 -- 172.21.15.26:0/960121856 learned_addr learned my addr 172.21.15.26:0/960121856 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:43.050 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.047+0000 7f32dd51a640 1 -- 172.21.15.26:0/960121856 >> v1:172.21.15.104:6790/0 conn(0x7f32d8138fe0 legacy=0x7f32d813e360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32ddd1b640 1 -- 172.21.15.26:0/960121856 >> v1:172.21.15.26:6791/0 conn(0x7f32d806e700 legacy=0x7f32d8139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 1 -- 172.21.15.26:0/960121856 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3922018990 0 0) 0x7f32d809e160 con 0x7f32d806e700 2024-08-19T12:57:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient(hunting): _negotiate 2024-08-19T12:57:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient(hunting): my global_id is 184846 2024-08-19T12:57:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 1 -- 172.21.15.26:0/960121856 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32c4003520 con 0x7f32d806e700 2024-08-19T12:57:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32de51c640 1 -- 172.21.15.26:0/960121856 >> v1:172.21.15.104:6789/0 conn(0x7f32d8139b80 legacy=0x7f32d8141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 1 -- 172.21.15.26:0/960121856 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2865447762 0 0) 0x7f32c4003520 con 0x7f32d806e700 2024-08-19T12:57:43.054 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 1 -- 172.21.15.26:0/960121856 >> v1:172.21.15.104:6790/0 conn(0x7f32d8138fe0 legacy=0x7f32d813e360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 1 -- 172.21.15.26:0/960121856 >> v1:172.21.15.104:6789/0 conn(0x7f32d8139b80 legacy=0x7f32d8141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient: _finish_hunting 0 2024-08-19T12:57:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 1 monclient: found mon.noname-c 2024-08-19T12:57:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:57:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 1 -- 172.21.15.26:0/960121856 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d809e440 con 0x7f32d806e700 2024-08-19T12:57:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient: _finish_auth 0 2024-08-19T12:57:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient: _check_auth_tickets 2024-08-19T12:57:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 1 -- 172.21.15.26:0/960121856 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f32d4001ae0 con 0x7f32d806e700 2024-08-19T12:57:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dff7f640 5 monclient: authenticate success, global_id 184846 2024-08-19T12:57:43.055 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dff7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient: dump: 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:43.056 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dff7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 1 -- 172.21.15.26:0/960121856 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f32d40048a0 con 0x7f32d806e700 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 1 -- 172.21.15.26:0/960121856 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f32d4004e10 con 0x7f32d806e700 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dff7f640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dff7f640 4 set_mon_vals no callback set 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:43.057 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.049+0000 7f32dcd19640 10 monclient: dump: 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:43.058 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.050+0000 7f32dff7f640 10 monclient: shutdown 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.050+0000 7f32dff7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.050+0000 7f32dff7f640 1 -- 172.21.15.26:0/960121856 >> v1:172.21.15.26:6791/0 conn(0x7f32d806e700 legacy=0x7f32d8139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.050+0000 7f32dff7f640 1 -- 172.21.15.26:0/960121856 shutdown_connections 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.050+0000 7f32dff7f640 1 -- 172.21.15.26:0/960121856 >> 172.21.15.26:0/960121856 conn(0x7f32d80a86d0 msgr2=0x7f32d80a8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.050+0000 7f32dff7f640 1 -- 172.21.15.26:0/960121856 shutdown_connections 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.050+0000 7f32dff7f640 1 -- 172.21.15.26:0/960121856 wait complete. 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 10 monclient: build_initial_monmap 2024-08-19T12:57:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 10 monclient: monmap: 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:43.060 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 1 librados: starting msgr at 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 1 librados: starting objecter 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 1 Processor -- start 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 1 -- start start 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 1 librados: setting wanted keys 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 1 librados: calling monclient init 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 10 monclient: init 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 10 monclient: _add_conns ranks=[3,6,5,1,8,4,0,7,2] 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 10 monclient(hunting): picked mon.noname-g con 0x7f32d806e700 addr v1:172.21.15.104:6790/0 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 10 monclient(hunting): picked mon.noname-d con 0x7f32d8139b80 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:43.061 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 10 monclient(hunting): picked mon.noname-h con 0x7f32d8135c40 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:43.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d8138c20 con 0x7f32d806e700 2024-08-19T12:57:43.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d806f7a0 con 0x7f32d8135c40 2024-08-19T12:57:43.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dd51a640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f32d8139b80 legacy=0x7f32d8135530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:43.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d8123b20 con 0x7f32d8139b80 2024-08-19T12:57:43.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:43.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.051+0000 7f32dff7f640 10 monclient(hunting): authenticate will time out at 1857.138964s 2024-08-19T12:57:43.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.052+0000 7f32ddd1b640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f32d806e700 legacy=0x7f32d8134e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:43.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.052+0000 7f32de51c640 1 --1- >> v1:172.21.15.104:6791/0 conn(0x7f32d8135c40 0x7f32d81d6300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6791/0 says I am v1:172.21.15.26:50038/0 (socket says 172.21.15.26:50038) 2024-08-19T12:57:43.062 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.052+0000 7f32de51c640 1 -- 172.21.15.26:0/3899274895 learned_addr learned my addr 172.21.15.26:0/3899274895 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:43.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.054+0000 7f32de51c640 1 -- 172.21.15.26:0/3899274895 >> v1:172.21.15.104:6791/0 conn(0x7f32d8135c40 legacy=0x7f32d81d6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:43.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.054+0000 7f32ce7fc640 10 client.?.objecter ms_handle_connect 0x7f32d8135c40 2024-08-19T12:57:43.063 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.054+0000 7f32ce7fc640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.054+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4159369032 0 0) 0x7f32d80a80f0 con 0x7f32d8135c40 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.054+0000 7f32ce7fc640 10 monclient(hunting): _negotiate 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.054+0000 7f32ce7fc640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.054+0000 7f32ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.054+0000 7f32ce7fc640 10 monclient(hunting): my global_id is 184838 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.054+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32bc004eb0 con 0x7f32d8135c40 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.054+0000 7f32de51c640 1 --1- 172.21.15.26:0/3899274895 >> v1:172.21.15.104:6791/0 conn(0x7f32d8135c40 0x7f32d81d6300 :-1 s=OPENED pgs=3752 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f32bc004eb0 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.054+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3597489306 0 0) 0x7f32bc004eb0 con 0x7f32d8135c40 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.054+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f32d80a80f0 con 0x7f32d8135c40 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.055+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f32d0003990 con 0x7f32d8135c40 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.055+0000 7f32ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.055+0000 7f32ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.055+0000 7f32ce7fc640 10 monclient(hunting): dump: 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:43.064 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:43.065 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.055+0000 7f32ce7fc640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:43.253 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.252+0000 7f32dd51a640 1 -- 172.21.15.26:0/3899274895 >> v1:172.21.15.26:6792/0 conn(0x7f32d8139b80 legacy=0x7f32d8135530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:43.254 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.253+0000 7f32ddd1b640 1 -- 172.21.15.26:0/3899274895 >> v1:172.21.15.104:6790/0 conn(0x7f32d806e700 legacy=0x7f32d8134e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:43.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.653+0000 7f32dd51a640 1 -- 172.21.15.26:0/3899274895 >> v1:172.21.15.26:6792/0 conn(0x7f32d8139b80 legacy=0x7f32d8135530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:43.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.654+0000 7f32ddd1b640 1 -- 172.21.15.26:0/3899274895 >> v1:172.21.15.104:6790/0 conn(0x7f32d806e700 legacy=0x7f32d8134e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:43.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 <== mon.5 v1:172.21.15.104:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3134595060 0 0) 0x7f32d80a80f0 con 0x7f32d8135c40 2024-08-19T12:57:43.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 >> v1:172.21.15.26:6792/0 conn(0x7f32d8139b80 legacy=0x7f32d8135530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:43.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 >> v1:172.21.15.104:6790/0 conn(0x7f32d806e700 legacy=0x7f32d8134e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:43.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 10 monclient: _finish_hunting 0 2024-08-19T12:57:43.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 1 monclient: found mon.h 2024-08-19T12:57:43.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:43.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:43.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 --> v1:172.21.15.104:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d8123e00 con 0x7f32d8135c40 2024-08-19T12:57:43.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 10 monclient: _finish_auth 0 2024-08-19T12:57:43.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 10 monclient: _check_auth_tickets 2024-08-19T12:57:43.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:43.680 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32dff7f640 5 monclient: authenticate success, global_id 184838 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32dff7f640 10 monclient: _renew_subs 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32dff7f640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32dff7f640 1 -- 172.21.15.26:0/3899274895 --> v1:172.21.15.104:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f32d81228e0 con 0x7f32d8135c40 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32dff7f640 10 client.184838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32dff7f640 10 monclient: _renew_subs 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32dff7f640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32dff7f640 1 -- 172.21.15.26:0/3899274895 --> v1:172.21.15.104:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f32d81d7a20 con 0x7f32d8135c40 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32dff7f640 1 librados: init done 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 <== mon.5 v1:172.21.15.104:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f32d0003fb0 con 0x7f32d8135c40 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 <== mon.5 v1:172.21.15.104:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f32d00044a0 con 0x7f32d8135c40 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-08-19T12:57:43.681 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32cffff640 4 set_mon_vals no callback set 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.678+0000 7f32ce7fc640 10 monclient: dump: 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:43.682 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:43.683 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.679+0000 7f32dff7f640 10 client.184838.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.679+0000 7f32dff7f640 15 client.184838.objecter _session_command_op_assign -1 1 2024-08-19T12:57:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.679+0000 7f32dff7f640 20 client.184838.objecter _get_session osd=-1 returning homeless 2024-08-19T12:57:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.679+0000 7f32dff7f640 10 client.184838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.679+0000 7f32dff7f640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.104:6791/0 2024-08-19T12:57:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.679+0000 7f32dff7f640 1 -- 172.21.15.26:0/3899274895 --> v1:172.21.15.104:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f32c0002910 con 0x7f32d8135c40 2024-08-19T12:57:43.683 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.679+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 <== mon.5 v1:172.21.15.104:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f32d0004640 con 0x7f32d8135c40 2024-08-19T12:57:43.684 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.679+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 <== mon.5 v1:172.21.15.104:6791/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7f32d004e340 con 0x7f32d8135c40 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.679+0000 7f32ce7fc640 10 client.184838.objecter ms_dispatch 0x7f32d8000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.679+0000 7f32ce7fc640 3 client.184838.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.679+0000 7f32ce7fc640 3 client.184838.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 10 client.184838.objecter checking command 1 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 20 client.184838.objecter _get_session s=0x7f32bc048c20 osd=3 3 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 20 client.184838.objecter put_session s=0x7f32bc048c20 osd=3 3 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 15 client.184838.objecter _session_command_op_remove -1 1 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 20 client.184838.objecter _get_session s=0x7f32bc048c20 osd=3 3 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 20 client.184838.objecter get_session s=0x7f32bc048c20 osd=3 3 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 15 client.184838.objecter _session_command_op_assign 3 1 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 20 client.184838.objecter put_session s=0x7f32bc048c20 osd=3 4 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 10 client.184838.objecter _send_command 1 2024-08-19T12:57:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 --> v1:172.21.15.104:6809/847838248 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32bc04cfe0 con 0x7f32bc0492d0 2024-08-19T12:57:43.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 20 client.184838.objecter dump_active .. 0 homeless 2024-08-19T12:57:43.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 <== mon.5 v1:172.21.15.104:6791/0 9 ==== mon_get_version_reply(handle=1 version=701) ==== 24+0+0 (unknown 2985145993 0 0) 0x7f32d00170d0 con 0x7f32d8135c40 2024-08-19T12:57:43.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ce7fc640 10 monclient: handle_get_version_reply finishing 1 version 701 2024-08-19T12:57:43.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32ddd1b640 10 monclient: get_auth_request con 0x7f32bc040d50 auth_method 0 2024-08-19T12:57:43.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.680+0000 7f32dd51a640 10 monclient: get_auth_request con 0x7f32bc0492d0 auth_method 0 2024-08-19T12:57:43.686 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.682+0000 7f32ce7fc640 10 client.184838.objecter ms_handle_connect 0x7f32bc040d50 2024-08-19T12:57:43.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.684+0000 7f32ce7fc640 10 client.184838.objecter ms_handle_connect 0x7f32bc0492d0 2024-08-19T12:57:43.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.685+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 <== osd.3 v1:172.21.15.104:6809/847838248 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f32bc04cfe0 con 0x7f32bc0492d0 2024-08-19T12:57:43.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.685+0000 7f32ce7fc640 10 client.184838.objecter ms_dispatch 0x7f32d8000b90 command_reply(tid 1: 0 ) 2024-08-19T12:57:43.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.685+0000 7f32ce7fc640 10 client.184838.objecter _finish_command 1 = system:0 2024-08-19T12:57:43.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.685+0000 7f32ce7fc640 20 client.184838.objecter put_session s=0x7f32bc048c20 osd=3 4 2024-08-19T12:57:43.687 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.685+0000 7f32ce7fc640 15 client.184838.objecter _session_command_op_remove 3 1 2024-08-19T12:57:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.698+0000 7f32dff7f640 10 client.184838.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-08-19T12:57:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.698+0000 7f32dff7f640 15 client.184838.objecter _session_command_op_assign -1 2 2024-08-19T12:57:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.698+0000 7f32dff7f640 20 client.184838.objecter _get_session s=0x7f32bc048c20 osd=3 3 2024-08-19T12:57:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.698+0000 7f32dff7f640 20 client.184838.objecter put_session s=0x7f32bc048c20 osd=3 3 2024-08-19T12:57:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.698+0000 7f32dff7f640 20 client.184838.objecter _get_session s=0x7f32bc048c20 osd=3 3 2024-08-19T12:57:43.699 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.698+0000 7f32dff7f640 15 client.184838.objecter _session_command_op_remove -1 2 2024-08-19T12:57:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.698+0000 7f32dff7f640 20 client.184838.objecter get_session s=0x7f32bc048c20 osd=3 3 2024-08-19T12:57:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.698+0000 7f32dff7f640 15 client.184838.objecter _session_command_op_assign 3 2 2024-08-19T12:57:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.698+0000 7f32dff7f640 20 client.184838.objecter put_session s=0x7f32bc048c20 osd=3 4 2024-08-19T12:57:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.698+0000 7f32dff7f640 10 client.184838.objecter _send_command 2 2024-08-19T12:57:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.698+0000 7f32dff7f640 1 -- 172.21.15.26:0/3899274895 --> v1:172.21.15.104:6809/847838248 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f32c00012b0 con 0x7f32bc0492d0 2024-08-19T12:57:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32ce7fc640 1 -- 172.21.15.26:0/3899274895 <== osd.3 v1:172.21.15.104:6809/847838248 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f32c00012b0 con 0x7f32bc0492d0 2024-08-19T12:57:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32ce7fc640 10 client.184838.objecter ms_dispatch 0x7f32d8000b90 command_reply(tid 2: 0 ) 2024-08-19T12:57:43.700 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32ce7fc640 10 client.184838.objecter _finish_command 2 = system:0 2024-08-19T12:57:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32ce7fc640 20 client.184838.objecter put_session s=0x7f32bc048c20 osd=3 4 2024-08-19T12:57:43.701 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32ce7fc640 15 client.184838.objecter _session_command_op_remove 3 2 2024-08-19T12:57:43.701 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:57:43.701 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-19T12:57:43.701 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:57:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32dff7f640 10 librados: watch_flush enter 2024-08-19T12:57:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32dff7f640 10 librados: watch_flush exit 2024-08-19T12:57:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32dff7f640 10 client.184838.objecter close_session for osd.3 2024-08-19T12:57:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32dff7f640 1 -- 172.21.15.26:0/3899274895 >> v1:172.21.15.104:6809/847838248 conn(0x7f32bc0492d0 legacy=0x7f32bc04b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32dff7f640 20 client.184838.objecter put_session s=0x7f32bc048c20 osd=3 1 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32dff7f640 20 client.184838.objecter shutdown clearing up homeless session... 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32dff7f640 10 client.184838.objecter successfully canceled tick 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.699+0000 7f32dff7f640 1 -- 172.21.15.26:0/3899274895 >> v1:172.21.15.104:6800/899887506 conn(0x7f32bc040d50 legacy=0x7f32bc043210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.700+0000 7f32dff7f640 10 monclient: shutdown 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.700+0000 7f32dff7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.700+0000 7f32dff7f640 1 -- 172.21.15.26:0/3899274895 >> v1:172.21.15.104:6791/0 conn(0x7f32d8135c40 legacy=0x7f32d81d6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.700+0000 7f32de51c640 1 -- 172.21.15.26:0/3899274895 reap_dead start 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.700+0000 7f32dff7f640 1 -- 172.21.15.26:0/3899274895 shutdown_connections 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.700+0000 7f32dff7f640 1 -- 172.21.15.26:0/3899274895 >> 172.21.15.26:0/3899274895 conn(0x7f32d80a86d0 msgr2=0x7f32d812ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.700+0000 7f32dff7f640 1 -- 172.21.15.26:0/3899274895 shutdown_connections 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.700+0000 7f32dff7f640 1 -- 172.21.15.26:0/3899274895 wait complete. 2024-08-19T12:57:43.703 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.700+0000 7f32dff7f640 1 librados: shutdown 2024-08-19T12:57:43.711 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-08-19T12:57:43.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.811+0000 7f3734e9f640 10 monclient: get_monmap_and_config 2024-08-19T12:57:43.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.811+0000 7f3734e9f640 10 monclient: build_initial_monmap 2024-08-19T12:57:43.812 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.811+0000 7f3734e9f640 10 monclient: monmap: 2024-08-19T12:57:43.812 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:43.812 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:43.812 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:43.812314+0000 2024-08-19T12:57:43.812 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:43.812314+0000 2024-08-19T12:57:43.812 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:43.812 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:43.813 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:43.813 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:43.813 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:43.813 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:43.813 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:43.813 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:43.813 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:43.813 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:43.813 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:43.813 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:43.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 1 Processor -- start 2024-08-19T12:57:43.824 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 1 -- start start 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 10 monclient: init 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 10 monclient: _add_conns ranks=[6,4,2,8,7,1,3,5,0] 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 10 monclient(hunting): picked mon.noname-d con 0x7f373006e7b0 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 10 monclient(hunting): picked mon.noname-c con 0x7f37300a5920 addr v1:172.21.15.26:6791/0 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 10 monclient(hunting): picked mon.noname-b con 0x7f37300a5d50 addr v1:172.21.15.26:6790/0 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3730136790 con 0x7f37300a5d50 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f373009ddf0 con 0x7f37300a5920 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f373009e0d0 con 0x7f373006e7b0 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3734e9f640 10 monclient(hunting): authenticate will time out at 1857.899612s 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f372e59c640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f373006e7b0 legacy=0x7f3730136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:43.825 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3725d9b640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7f37300a5920 0x7f373013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36766/0 (socket says 172.21.15.26:36766) 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f372ed9d640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f37300a5d50 0x7f3730141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39142/0 (socket says 172.21.15.26:39142) 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.812+0000 7f3725d9b640 1 -- 172.21.15.26:0/1894212802 learned_addr learned my addr 172.21.15.26:0/1894212802 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.814+0000 7f372ed9d640 1 -- 172.21.15.26:0/1894212802 >> v1:172.21.15.26:6790/0 conn(0x7f37300a5d50 legacy=0x7f3730141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.814+0000 7f3725d9b640 1 -- 172.21.15.26:0/1894212802 >> v1:172.21.15.26:6791/0 conn(0x7f37300a5920 legacy=0x7f373013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 1 -- 172.21.15.26:0/1894212802 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4265539836 0 0) 0x7f3718001450 con 0x7f37300a5920 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient(hunting): my global_id is 184855 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 1 -- 172.21.15.26:0/1894212802 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f371c003520 con 0x7f37300a5920 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 1 -- 172.21.15.26:0/1894212802 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2394925630 0 0) 0x7f3730136790 con 0x7f37300a5d50 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient(hunting): _negotiate 2024-08-19T12:57:43.826 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient(hunting): my global_id is 27551 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 1 -- 172.21.15.26:0/1894212802 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3718001450 con 0x7f37300a5d50 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 1 -- 172.21.15.26:0/1894212802 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2140271853 0 0) 0x7f371c003520 con 0x7f37300a5920 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 1 -- 172.21.15.26:0/1894212802 >> v1:172.21.15.26:6792/0 conn(0x7f373006e7b0 legacy=0x7f3730136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 1 -- 172.21.15.26:0/1894212802 >> v1:172.21.15.26:6790/0 conn(0x7f37300a5d50 legacy=0x7f3730141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient: _finish_hunting 0 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 1 monclient: found mon.noname-c 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 1 -- 172.21.15.26:0/1894212802 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373009e3b0 con 0x7f37300a5920 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient: _finish_auth 0 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient: _check_auth_tickets 2024-08-19T12:57:43.827 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 1 -- 172.21.15.26:0/1894212802 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3718001cb0 con 0x7f37300a5920 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f3734e9f640 5 monclient: authenticate success, global_id 184855 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f3734e9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient: dump: 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:43.828 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f3734e9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 1 -- 172.21.15.26:0/1894212802 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3718004950 con 0x7f37300a5920 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 1 -- 172.21.15.26:0/1894212802 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3718004ec0 con 0x7f37300a5920 2024-08-19T12:57:43.829 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f3734e9f640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f3734e9f640 4 set_mon_vals no callback set 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f372dd9b640 10 monclient: dump: 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:43.830 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:43.831 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f3734e9f640 10 monclient: shutdown 2024-08-19T12:57:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f3734e9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f3734e9f640 1 -- 172.21.15.26:0/1894212802 >> v1:172.21.15.26:6791/0 conn(0x7f37300a5920 legacy=0x7f373013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:43.831 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.815+0000 7f3734e9f640 1 -- 172.21.15.26:0/1894212802 shutdown_connections 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.816+0000 7f3734e9f640 1 -- 172.21.15.26:0/1894212802 >> 172.21.15.26:0/1894212802 conn(0x7f37300a8740 msgr2=0x7f37300a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.816+0000 7f3734e9f640 1 -- 172.21.15.26:0/1894212802 shutdown_connections 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.816+0000 7f3734e9f640 1 -- 172.21.15.26:0/1894212802 wait complete. 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.816+0000 7f3734e9f640 10 monclient: build_initial_monmap 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.816+0000 7f3734e9f640 10 monclient: monmap: 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:43.832 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:43.833 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.816+0000 7f3734e9f640 1 librados: starting msgr at 2024-08-19T12:57:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.816+0000 7f3734e9f640 1 librados: starting objecter 2024-08-19T12:57:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 1 Processor -- start 2024-08-19T12:57:43.833 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 1 -- start start 2024-08-19T12:57:43.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 1 librados: setting wanted keys 2024-08-19T12:57:43.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 1 librados: calling monclient init 2024-08-19T12:57:43.834 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 10 monclient: init 2024-08-19T12:57:43.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:43.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 10 monclient: _add_conns ranks=[8,6,7,5,2,4,3,1,0] 2024-08-19T12:57:43.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 10 monclient(hunting): picked mon.noname-e con 0x7f373006e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:57:43.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 10 monclient(hunting): picked mon.noname-d con 0x7f37300a5920 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:43.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 10 monclient(hunting): picked mon.noname-i con 0x7f37300a5d50 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:43.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3730132ad0 con 0x7f37300a5920 2024-08-19T12:57:43.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37300a0430 con 0x7f37300a5d50 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37300a09f0 con 0x7f373006e7b0 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3725d9b640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f37300a5920 legacy=0x7f37301323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f3734e9f640 10 monclient(hunting): authenticate will time out at 1857.904712s 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f372e59c640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f373006e7b0 0x7f3730131cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:57088/0 (socket says 172.21.15.26:57088) 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.817+0000 7f372e59c640 1 -- 172.21.15.26:0/10981068 learned_addr learned my addr 172.21.15.26:0/10981068 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.819+0000 7f372e59c640 1 -- 172.21.15.26:0/10981068 >> v1:172.21.15.26:6793/0 conn(0x7f373006e7b0 legacy=0x7f3730131cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.819+0000 7f37277fe640 10 client.?.objecter ms_handle_connect 0x7f373006e7b0 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4039619118 0 0) 0x7f37300a09f0 con 0x7f373006e7b0 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:43.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient(hunting): my global_id is 184724 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3710004eb0 con 0x7f373006e7b0 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 client.?.objecter ms_handle_connect 0x7f37300a5d50 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f372ed9d640 1 -- 172.21.15.26:0/10981068 >> v1:172.21.15.104:6792/0 conn(0x7f37300a5d50 legacy=0x7f37301d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2414846131 0 0) 0x7f3710004eb0 con 0x7f373006e7b0 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37300a09f0 con 0x7f373006e7b0 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3720004760 con 0x7f373006e7b0 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f372ed9d640 1 --1- 172.21.15.26:0/10981068 >> v1:172.21.15.104:6792/0 conn(0x7f37300a5d50 0x7f37301d6290 :-1 s=OPENED pgs=918 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7f37300a0430 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient(hunting): dump: 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:43.837 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:43.838 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 342704180 0 0) 0x7f37300a09f0 con 0x7f373006e7b0 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 >> v1:172.21.15.104:6792/0 conn(0x7f37300a5d50 legacy=0x7f37301d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 >> v1:172.21.15.26:6792/0 conn(0x7f37300a5920 legacy=0x7f37301323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient: _finish_hunting 0 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 monclient: found mon.e 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37300a0cd0 con 0x7f373006e7b0 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient: _finish_auth 0 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient: _check_auth_tickets 2024-08-19T12:57:43.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f3734e9f640 5 monclient: authenticate success, global_id 184724 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f3734e9f640 10 monclient: _renew_subs 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4281217359 0 0) 0x7f37300a0430 con 0x7f37300a5d50 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f3734e9f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f3734e9f640 1 -- 172.21.15.26:0/10981068 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3730132f10 con 0x7f373006e7b0 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f3734e9f640 10 client.184724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f3734e9f640 10 monclient: _renew_subs 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f3734e9f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f3734e9f640 1 -- 172.21.15.26:0/10981068 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f373009f6b0 con 0x7f373006e7b0 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f3734e9f640 1 librados: init done 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3720004980 con 0x7f373006e7b0 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:43.840 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f3720004ef0 con 0x7f373006e7b0 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f372d59a640 4 set_mon_vals no callback set 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.820+0000 7f37277fe640 10 monclient: dump: 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:43.841 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.821+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f37200051c0 con 0x7f373006e7b0 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.821+0000 7f3734e9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.821+0000 7f3734e9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.821+0000 7f3734e9f640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.821+0000 7f3734e9f640 1 -- 172.21.15.26:0/10981068 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3714005080 con 0x7f373006e7b0 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.821+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7f372004e090 con 0x7f373006e7b0 2024-08-19T12:57:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.821+0000 7f37277fe640 10 client.184724.objecter ms_dispatch 0x7f3730000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:43.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.821+0000 7f37277fe640 3 client.184724.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:43.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.821+0000 7f37277fe640 3 client.184724.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:43.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.821+0000 7f37277fe640 20 client.184724.objecter dump_active .. 0 homeless 2024-08-19T12:57:43.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.822+0000 7f3725d9b640 10 monclient: get_auth_request con 0x7f3710040d10 auth_method 0 2024-08-19T12:57:43.843 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.824+0000 7f37277fe640 10 client.184724.objecter ms_handle_connect 0x7f3710040d10 2024-08-19T12:57:43.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.826+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f37200176d0 con 0x7f373006e7b0 2024-08-19T12:57:43.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.826+0000 7f37277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:43.844 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:43.826+0000 7f37277fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:57:44.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.007+0000 7f3734e9f640 1 -- 172.21.15.26:0/10981068 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f3714000b90 con 0x7f3710040d10 2024-08-19T12:57:44.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.008+0000 7f37277fe640 1 -- 172.21.15.26:0/10981068 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7f3714000b90 con 0x7f3710040d10 2024-08-19T12:57:44.009 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 3 to deep-scrub 2024-08-19T12:57:44.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 10 librados: watch_flush enter 2024-08-19T12:57:44.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 10 librados: watch_flush exit 2024-08-19T12:57:44.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 20 client.184724.objecter shutdown clearing up homeless session... 2024-08-19T12:57:44.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 10 client.184724.objecter successfully canceled tick 2024-08-19T12:57:44.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 1 -- 172.21.15.26:0/10981068 >> v1:172.21.15.104:6800/899887506 conn(0x7f3710040d10 legacy=0x7f37100431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:44.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 10 monclient: shutdown 2024-08-19T12:57:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 1 -- 172.21.15.26:0/10981068 >> v1:172.21.15.26:6793/0 conn(0x7f373006e7b0 legacy=0x7f3730131cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 1 -- 172.21.15.26:0/10981068 shutdown_connections 2024-08-19T12:57:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 1 -- 172.21.15.26:0/10981068 >> 172.21.15.26:0/10981068 conn(0x7f37300a8740 msgr2=0x7f37300aa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 1 -- 172.21.15.26:0/10981068 shutdown_connections 2024-08-19T12:57:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 1 -- 172.21.15.26:0/10981068 wait complete. 2024-08-19T12:57:44.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.010+0000 7f3734e9f640 1 librados: shutdown 2024-08-19T12:57:44.023 INFO:tasks.ceph:Scrubbing osd.4 2024-08-19T12:57:44.023 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-08-19T12:57:44.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.126+0000 7fa8af172640 10 monclient: get_monmap_and_config 2024-08-19T12:57:44.126 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.126+0000 7fa8af172640 10 monclient: build_initial_monmap 2024-08-19T12:57:44.127 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.126+0000 7fa8af172640 10 monclient: monmap: 2024-08-19T12:57:44.127 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:44.127 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:44.127 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:44.126943+0000 2024-08-19T12:57:44.127 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:44.126943+0000 2024-08-19T12:57:44.127 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:44.127 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:44.127 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:44.127 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:44.127 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:44.128 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:44.128 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:44.128 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:44.128 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:44.128 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:44.128 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:44.128 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:44.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.126+0000 7fa8af172640 1 Processor -- start 2024-08-19T12:57:44.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.126+0000 7fa8af172640 1 -- start start 2024-08-19T12:57:44.129 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.126+0000 7fa8af172640 10 monclient: init 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8af172640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8af172640 10 monclient: _add_conns ranks=[4,5,6,1,3,2,8,0,7] 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8af172640 10 monclient(hunting): picked mon.noname-c con 0x7fa8a80a6640 addr v1:172.21.15.26:6791/0 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8af172640 10 monclient(hunting): picked mon.noname-h con 0x7fa8a8074110 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8af172640 10 monclient(hunting): picked mon.noname-d con 0x7fa8a813f8d0 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8af172640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8a809ea70 con 0x7fa8a80a6640 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8af172640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8a809dd20 con 0x7fa8a8074110 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8af172640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8a809e000 con 0x7fa8a813f8d0 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8af172640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8af172640 10 monclient(hunting): authenticate will time out at 1858.214498s 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8ad70f640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fa8a813f8d0 legacy=0x7fa8a8141cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8acf0e640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fa8a80a6640 0x7fa8a80a6a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36782/0 (socket says 172.21.15.26:36782) 2024-08-19T12:57:44.130 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.127+0000 7fa8acf0e640 1 -- 172.21.15.26:0/2576224660 learned_addr learned my addr 172.21.15.26:0/2576224660 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:44.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.129+0000 7fa8acf0e640 1 -- 172.21.15.26:0/2576224660 >> v1:172.21.15.26:6791/0 conn(0x7fa8a80a6640 legacy=0x7fa8a80a6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:44.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.129+0000 7fa8a77fe640 1 -- 172.21.15.26:0/2576224660 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 111579412 0 0) 0x7fa8a809ea70 con 0x7fa8a80a6640 2024-08-19T12:57:44.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.129+0000 7fa8a77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:44.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.129+0000 7fa8a77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:44.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.129+0000 7fa8a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:44.134 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.129+0000 7fa8a77fe640 10 monclient(hunting): my global_id is 184864 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.129+0000 7fa8a77fe640 1 -- 172.21.15.26:0/2576224660 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa894003520 con 0x7fa8a80a6640 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a7fff640 1 -- 172.21.15.26:0/2576224660 >> v1:172.21.15.104:6791/0 conn(0x7fa8a8074110 legacy=0x7fa8a80a7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 1 -- 172.21.15.26:0/2576224660 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1056491926 0 0) 0x7fa894003520 con 0x7fa8a80a6640 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 1 -- 172.21.15.26:0/2576224660 >> v1:172.21.15.26:6792/0 conn(0x7fa8a813f8d0 legacy=0x7fa8a8141cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 1 -- 172.21.15.26:0/2576224660 >> v1:172.21.15.104:6791/0 conn(0x7fa8a8074110 legacy=0x7fa8a80a7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 10 monclient: _finish_hunting 0 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 1 monclient: found mon.noname-c 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 1 -- 172.21.15.26:0/2576224660 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8a80a7de0 con 0x7fa8a80a6640 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 10 monclient: _finish_auth 0 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 10 monclient: _check_auth_tickets 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:44.135 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8af172640 5 monclient: authenticate success, global_id 184864 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 1 -- 172.21.15.26:0/2576224660 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa8a0001ad0 con 0x7fa8a80a6640 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8af172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 10 monclient: dump: 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:44.136 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:44.137 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:44.137 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:44.137 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:44.137 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:44.137 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:44.137 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:44.137 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8af172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 1 -- 172.21.15.26:0/2576224660 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8a00048a0 con 0x7fa8a80a6640 2024-08-19T12:57:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 1 -- 172.21.15.26:0/2576224660 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa8a0004e10 con 0x7fa8a80a6640 2024-08-19T12:57:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8af172640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8af172640 4 set_mon_vals no callback set 2024-08-19T12:57:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:57:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8a77fe640 10 monclient: dump: 2024-08-19T12:57:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:44.138 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:44.139 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8af172640 10 monclient: shutdown 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8af172640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8af172640 1 -- 172.21.15.26:0/2576224660 >> v1:172.21.15.26:6791/0 conn(0x7fa8a80a6640 legacy=0x7fa8a80a6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8af172640 1 -- 172.21.15.26:0/2576224660 shutdown_connections 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8af172640 1 -- 172.21.15.26:0/2576224660 >> 172.21.15.26:0/2576224660 conn(0x7fa8a80a8740 msgr2=0x7fa8a80a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8af172640 1 -- 172.21.15.26:0/2576224660 shutdown_connections 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.130+0000 7fa8af172640 1 -- 172.21.15.26:0/2576224660 wait complete. 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.131+0000 7fa8af172640 10 monclient: build_initial_monmap 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.131+0000 7fa8af172640 10 monclient: monmap: 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:44.140 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:44.141 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.131+0000 7fa8af172640 1 librados: starting msgr at 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.131+0000 7fa8af172640 1 librados: starting objecter 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.131+0000 7fa8af172640 1 Processor -- start 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.131+0000 7fa8af172640 1 -- start start 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.131+0000 7fa8af172640 1 librados: setting wanted keys 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.131+0000 7fa8af172640 1 librados: calling monclient init 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.131+0000 7fa8af172640 10 monclient: init 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8af172640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8af172640 10 monclient: _add_conns ranks=[6,7,3,8,0,4,5,1,2] 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8af172640 10 monclient(hunting): picked mon.noname-d con 0x7fa8a8074110 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8af172640 10 monclient(hunting): picked mon.noname-i con 0x7fa8a80a6640 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:44.142 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8af172640 10 monclient(hunting): picked mon.noname-g con 0x7fa8a813f8d0 addr v1:172.21.15.104:6790/0 2024-08-19T12:57:44.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8af172640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8a8134cd0 con 0x7fa8a813f8d0 2024-08-19T12:57:44.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8af172640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8a8134fd0 con 0x7fa8a8074110 2024-08-19T12:57:44.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8af172640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8a80aac20 con 0x7fa8a80a6640 2024-08-19T12:57:44.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8af172640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:44.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8acf0e640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7fa8a8074110 legacy=0x7fa8a80a1660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:44.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8af172640 10 monclient(hunting): authenticate will time out at 1858.219370s 2024-08-19T12:57:44.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8ad70f640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7fa8a813f8d0 legacy=0x7fa8a81d6530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:44.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8a7fff640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7fa8a80a6640 0x7fa8a80a1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:42106/0 (socket says 172.21.15.26:42106) 2024-08-19T12:57:44.143 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.132+0000 7fa8a7fff640 1 -- 172.21.15.26:0/4293408152 learned_addr learned my addr 172.21.15.26:0/4293408152 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:44.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.134+0000 7fa8a7fff640 1 -- 172.21.15.26:0/4293408152 >> v1:172.21.15.104:6792/0 conn(0x7fa8a80a6640 legacy=0x7fa8a80a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:44.144 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.134+0000 7fa8a57fa640 10 client.?.objecter ms_handle_connect 0x7fa8a80a6640 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.134+0000 7fa8a57fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1755365649 0 0) 0x7fa8a80aac20 con 0x7fa8a80a6640 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 10 monclient(hunting): _negotiate 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 10 monclient(hunting): my global_id is 136609 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa888004eb0 con 0x7fa8a80a6640 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a7fff640 1 --1- 172.21.15.26:0/4293408152 >> v1:172.21.15.104:6792/0 conn(0x7fa8a80a6640 0x7fa8a80a1d70 :-1 s=OPENED pgs=919 cs=1 l=1).queue_received will delay after 1.72407e+09 on 0x7fa888004eb0 auth_reply(proto 2 0 (0) Success) 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 515707756 0 0) 0x7fa888004eb0 con 0x7fa8a80a6640 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 --> v1:172.21.15.104:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8a80aac20 con 0x7fa8a80a6640 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa898003990 con 0x7fa8a80a6640 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:44.145 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 10 monclient(hunting): dump: 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:44.147 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:44.147 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:44.147 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:44.147 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.135+0000 7fa8a57fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:44.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.333+0000 7fa8acf0e640 1 -- 172.21.15.26:0/4293408152 >> v1:172.21.15.26:6792/0 conn(0x7fa8a8074110 legacy=0x7fa8a80a1660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:44.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.333+0000 7fa8ad70f640 1 -- 172.21.15.26:0/4293408152 >> v1:172.21.15.104:6790/0 conn(0x7fa8a813f8d0 legacy=0x7fa8a81d6530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:44.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.734+0000 7fa8acf0e640 1 -- 172.21.15.26:0/4293408152 >> v1:172.21.15.26:6792/0 conn(0x7fa8a8074110 legacy=0x7fa8a80a1660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:44.735 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.734+0000 7fa8ad70f640 1 -- 172.21.15.26:0/4293408152 >> v1:172.21.15.104:6790/0 conn(0x7fa8a813f8d0 legacy=0x7fa8a81d6530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:44.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 <== mon.7 v1:172.21.15.104:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1011453933 0 0) 0x7fa8a80aac20 con 0x7fa8a80a6640 2024-08-19T12:57:44.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 >> v1:172.21.15.26:6792/0 conn(0x7fa8a8074110 legacy=0x7fa8a80a1660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:44.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 >> v1:172.21.15.104:6790/0 conn(0x7fa8a813f8d0 legacy=0x7fa8a81d6530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:44.759 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 10 monclient: _finish_hunting 0 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 1 monclient: found mon.i 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8a80a43e0 con 0x7fa8a80a6640 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 10 monclient: _finish_auth 0 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 10 monclient: _check_auth_tickets 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8af172640 5 monclient: authenticate success, global_id 136609 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8af172640 10 monclient: _renew_subs 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8af172640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8af172640 1 -- 172.21.15.26:0/4293408152 --> v1:172.21.15.104:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa8a80a2c90 con 0x7fa8a80a6640 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8af172640 10 client.136609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:44.760 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8af172640 10 monclient: _renew_subs 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8af172640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8af172640 1 -- 172.21.15.26:0/4293408152 --> v1:172.21.15.104:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fa8a80a4a30 con 0x7fa8a80a6640 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8af172640 1 librados: init done 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 <== mon.7 v1:172.21.15.104:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa898004030 con 0x7fa8a80a6640 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 <== mon.7 v1:172.21.15.104:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fa898004520 con 0x7fa8a80a6640 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a6ffd640 4 set_mon_vals no callback set 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.758+0000 7fa8a57fa640 10 monclient: dump: 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:44.762 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:44.762 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:44.762 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:44.762 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:44.762 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:44.762 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:44.762 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:44.762 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:44.762 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:44.762 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:44.762 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.759+0000 7fa8af172640 10 client.136609.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.759+0000 7fa8af172640 15 client.136609.objecter _session_command_op_assign -1 1 2024-08-19T12:57:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.759+0000 7fa8af172640 20 client.136609.objecter _get_session osd=-1 returning homeless 2024-08-19T12:57:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.759+0000 7fa8af172640 10 client.136609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.759+0000 7fa8af172640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.104:6792/0 2024-08-19T12:57:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.759+0000 7fa8af172640 1 -- 172.21.15.26:0/4293408152 --> v1:172.21.15.104:6792/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fa88c002910 con 0x7fa8a80a6640 2024-08-19T12:57:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.759+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 <== mon.7 v1:172.21.15.104:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fa898011e00 con 0x7fa8a80a6640 2024-08-19T12:57:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 <== mon.7 v1:172.21.15.104:6792/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7fa89804e390 con 0x7fa8a80a6640 2024-08-19T12:57:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 10 client.136609.objecter ms_dispatch 0x7fa8a8000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 3 client.136609.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 3 client.136609.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:44.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 10 client.136609.objecter checking command 1 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 20 client.136609.objecter _get_session s=0x7fa888048c30 osd=4 3 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 20 client.136609.objecter put_session s=0x7fa888048c30 osd=4 3 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 15 client.136609.objecter _session_command_op_remove -1 1 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 20 client.136609.objecter _get_session s=0x7fa888048c30 osd=4 3 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 20 client.136609.objecter get_session s=0x7fa888048c30 osd=4 3 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 15 client.136609.objecter _session_command_op_assign 4 1 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 20 client.136609.objecter put_session s=0x7fa888048c30 osd=4 4 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 10 client.136609.objecter _send_command 1 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 --> v1:172.21.15.104:6805/1352976736 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa88804cff0 con 0x7fa8880492e0 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8a57fa640 20 client.136609.objecter dump_active .. 0 homeless 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8acf0e640 10 monclient: get_auth_request con 0x7fa888040d60 auth_method 0 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.760+0000 7fa8ad70f640 10 monclient: get_auth_request con 0x7fa8880492e0 auth_method 0 2024-08-19T12:57:44.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.762+0000 7fa8a57fa640 10 client.136609.objecter ms_handle_connect 0x7fa888040d60 2024-08-19T12:57:44.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.765+0000 7fa8a57fa640 10 client.136609.objecter ms_handle_connect 0x7fa8880492e0 2024-08-19T12:57:44.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.766+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 <== osd.4 v1:172.21.15.104:6805/1352976736 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa88804cff0 con 0x7fa8880492e0 2024-08-19T12:57:44.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.766+0000 7fa8a57fa640 10 client.136609.objecter ms_dispatch 0x7fa8a8000b90 command_reply(tid 1: 0 ) 2024-08-19T12:57:44.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.766+0000 7fa8a57fa640 10 client.136609.objecter _finish_command 1 = system:0 2024-08-19T12:57:44.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.766+0000 7fa8a57fa640 20 client.136609.objecter put_session s=0x7fa888048c30 osd=4 4 2024-08-19T12:57:44.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.766+0000 7fa8a57fa640 15 client.136609.objecter _session_command_op_remove 4 1 2024-08-19T12:57:44.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.779+0000 7fa8af172640 10 client.136609.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-08-19T12:57:44.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.779+0000 7fa8af172640 15 client.136609.objecter _session_command_op_assign -1 2 2024-08-19T12:57:44.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.779+0000 7fa8af172640 20 client.136609.objecter _get_session s=0x7fa888048c30 osd=4 3 2024-08-19T12:57:44.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.779+0000 7fa8af172640 20 client.136609.objecter put_session s=0x7fa888048c30 osd=4 3 2024-08-19T12:57:44.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.779+0000 7fa8af172640 20 client.136609.objecter _get_session s=0x7fa888048c30 osd=4 3 2024-08-19T12:57:44.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.779+0000 7fa8af172640 15 client.136609.objecter _session_command_op_remove -1 2 2024-08-19T12:57:44.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.779+0000 7fa8af172640 20 client.136609.objecter get_session s=0x7fa888048c30 osd=4 3 2024-08-19T12:57:44.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.779+0000 7fa8af172640 15 client.136609.objecter _session_command_op_assign 4 2 2024-08-19T12:57:44.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.779+0000 7fa8af172640 20 client.136609.objecter put_session s=0x7fa888048c30 osd=4 4 2024-08-19T12:57:44.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.779+0000 7fa8af172640 10 client.136609.objecter _send_command 2 2024-08-19T12:57:44.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.779+0000 7fa8af172640 1 -- 172.21.15.26:0/4293408152 --> v1:172.21.15.104:6805/1352976736 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa88c0012b0 con 0x7fa8880492e0 2024-08-19T12:57:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8a57fa640 1 -- 172.21.15.26:0/4293408152 <== osd.4 v1:172.21.15.104:6805/1352976736 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fa88c0012b0 con 0x7fa8880492e0 2024-08-19T12:57:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8a57fa640 10 client.136609.objecter ms_dispatch 0x7fa8a8000b90 command_reply(tid 2: 0 ) 2024-08-19T12:57:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8a57fa640 10 client.136609.objecter _finish_command 2 = system:0 2024-08-19T12:57:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8a57fa640 20 client.136609.objecter put_session s=0x7fa888048c30 osd=4 4 2024-08-19T12:57:44.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8a57fa640 15 client.136609.objecter _session_command_op_remove 4 2 2024-08-19T12:57:44.781 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:57:44.782 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-19T12:57:44.782 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:57:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 10 librados: watch_flush enter 2024-08-19T12:57:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 10 librados: watch_flush exit 2024-08-19T12:57:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 10 client.136609.objecter close_session for osd.4 2024-08-19T12:57:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 1 -- 172.21.15.26:0/4293408152 >> v1:172.21.15.104:6805/1352976736 conn(0x7fa8880492e0 legacy=0x7fa88804b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 20 client.136609.objecter put_session s=0x7fa888048c30 osd=4 1 2024-08-19T12:57:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 20 client.136609.objecter shutdown clearing up homeless session... 2024-08-19T12:57:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 10 client.136609.objecter successfully canceled tick 2024-08-19T12:57:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 1 -- 172.21.15.26:0/4293408152 >> v1:172.21.15.104:6800/899887506 conn(0x7fa888040d60 legacy=0x7fa888043220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 10 monclient: shutdown 2024-08-19T12:57:44.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 20 monclient: shutdown canceling and discarding version request 1 2024-08-19T12:57:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 1 -- 172.21.15.26:0/4293408152 >> v1:172.21.15.104:6792/0 conn(0x7fa8a80a6640 legacy=0x7fa8a80a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8ad70f640 1 -- 172.21.15.26:0/4293408152 reap_dead start 2024-08-19T12:57:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 1 -- 172.21.15.26:0/4293408152 shutdown_connections 2024-08-19T12:57:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 1 -- 172.21.15.26:0/4293408152 >> 172.21.15.26:0/4293408152 conn(0x7fa8a80a8740 msgr2=0x7fa8a80a9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 1 -- 172.21.15.26:0/4293408152 shutdown_connections 2024-08-19T12:57:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 1 -- 172.21.15.26:0/4293408152 wait complete. 2024-08-19T12:57:44.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.780+0000 7fa8af172640 1 librados: shutdown 2024-08-19T12:57:44.794 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-08-19T12:57:44.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.897+0000 7f97c631b640 10 monclient: get_monmap_and_config 2024-08-19T12:57:44.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.897+0000 7f97c631b640 10 monclient: build_initial_monmap 2024-08-19T12:57:44.898 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.897+0000 7f97c631b640 10 monclient: monmap: 2024-08-19T12:57:44.898 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:44.898 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:44.898 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:44.898411+0000 2024-08-19T12:57:44.898 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:44.898411+0000 2024-08-19T12:57:44.898 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:44.899 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:44.899 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:44.899 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:44.899 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:44.899 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:44.899 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:44.899 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:44.899 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:44.899 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:44.899 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:44.899 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:44.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 1 Processor -- start 2024-08-19T12:57:44.900 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 1 -- start start 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 10 monclient: init 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 10 monclient: _add_conns ranks=[2,7,6,8,0,1,4,5,3] 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 10 monclient(hunting): picked mon.noname-b con 0x7f97c006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 10 monclient(hunting): picked mon.noname-i con 0x7f97c00a5920 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 10 monclient(hunting): picked mon.noname-d con 0x7f97c00a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c0136790 con 0x7f97c006e7b0 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c009ddf0 con 0x7f97c00a5d50 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c009e0d0 con 0x7f97c00a5920 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c631b640 10 monclient(hunting): authenticate will time out at 1858.985876s 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97c48b8640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f97c00a5d50 legacy=0x7f97c0141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97bffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f97c006e7b0 0x7f97c0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39148/0 (socket says 172.21.15.26:39148) 2024-08-19T12:57:44.901 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.898+0000 7f97bffff640 1 -- 172.21.15.26:0/2900713292 learned_addr learned my addr 172.21.15.26:0/2900713292 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:44.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.901+0000 7f97bffff640 1 -- 172.21.15.26:0/2900713292 >> v1:172.21.15.26:6790/0 conn(0x7f97c006e7b0 legacy=0x7f97c0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:44.902 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:44.901+0000 7f97bf7fe640 1 -- 172.21.15.26:0/2900713292 >> v1:172.21.15.104:6792/0 conn(0x7f97c00a5920 legacy=0x7f97c013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.099+0000 7f97c48b8640 1 -- 172.21.15.26:0/2900713292 >> v1:172.21.15.26:6792/0 conn(0x7f97c00a5d50 legacy=0x7f97c0141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:45.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.308+0000 7f97beffd640 1 -- 172.21.15.26:0/2900713292 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3428148321 0 0) 0x7f97c0136790 con 0x7f97c006e7b0 2024-08-19T12:57:45.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.308+0000 7f97beffd640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.311 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.308+0000 7f97beffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.308+0000 7f97beffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.308+0000 7f97beffd640 10 monclient(hunting): my global_id is 27560 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.308+0000 7f97beffd640 1 -- 172.21.15.26:0/2900713292 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97a8003520 con 0x7f97c006e7b0 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.308+0000 7f97beffd640 1 -- 172.21.15.26:0/2900713292 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1445329177 0 0) 0x7f97c009e0d0 con 0x7f97c00a5920 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.308+0000 7f97beffd640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.308+0000 7f97beffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.308+0000 7f97beffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.308+0000 7f97beffd640 10 monclient(hunting): my global_id is 136618 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 1 -- 172.21.15.26:0/2900713292 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97c0136790 con 0x7f97c00a5920 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 1 -- 172.21.15.26:0/2900713292 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 979049918 0 0) 0x7f97a8003520 con 0x7f97c006e7b0 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 1 -- 172.21.15.26:0/2900713292 >> v1:172.21.15.104:6792/0 conn(0x7f97c00a5920 legacy=0x7f97c013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 1 -- 172.21.15.26:0/2900713292 >> v1:172.21.15.26:6792/0 conn(0x7f97c00a5d50 legacy=0x7f97c0141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 10 monclient: _finish_hunting 0 2024-08-19T12:57:45.312 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 1 monclient: found mon.noname-b 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 1 -- 172.21.15.26:0/2900713292 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c009e3b0 con 0x7f97c006e7b0 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 10 monclient: _finish_auth 0 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 10 monclient: _check_auth_tickets 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 1 -- 172.21.15.26:0/2900713292 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f97b0001ae0 con 0x7f97c006e7b0 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 10 monclient: dump: 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.313 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97c631b640 5 monclient: authenticate success, global_id 27560 2024-08-19T12:57:45.314 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97c631b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 1 -- 172.21.15.26:0/2900713292 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97b00048a0 con 0x7f97c006e7b0 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 1 -- 172.21.15.26:0/2900713292 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f97b0004e50 con 0x7f97c006e7b0 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97c631b640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97c631b640 4 set_mon_vals no callback set 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.309+0000 7f97beffd640 10 monclient: dump: 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.315 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.316 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.316 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.316 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.316 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.316 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.316 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.316 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.316 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.316 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.316 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.316 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.316 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.310+0000 7f97c631b640 10 monclient: shutdown 2024-08-19T12:57:45.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.310+0000 7f97c631b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:45.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.310+0000 7f97c631b640 1 -- 172.21.15.26:0/2900713292 >> v1:172.21.15.26:6790/0 conn(0x7f97c006e7b0 legacy=0x7f97c0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.317 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.310+0000 7f97c631b640 1 -- 172.21.15.26:0/2900713292 shutdown_connections 2024-08-19T12:57:45.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.310+0000 7f97c631b640 1 -- 172.21.15.26:0/2900713292 >> 172.21.15.26:0/2900713292 conn(0x7f97c00a8740 msgr2=0x7f97c00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:45.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.310+0000 7f97c631b640 1 -- 172.21.15.26:0/2900713292 shutdown_connections 2024-08-19T12:57:45.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.310+0000 7f97c631b640 1 -- 172.21.15.26:0/2900713292 wait complete. 2024-08-19T12:57:45.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.310+0000 7f97c631b640 10 monclient: build_initial_monmap 2024-08-19T12:57:45.318 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.310+0000 7f97c631b640 10 monclient: monmap: 2024-08-19T12:57:45.318 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:45.318 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:45.318 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:45.318 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:45.318 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:45.318 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.318 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:45.319 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:45.319 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:45.319 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:45.319 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:45.319 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:45.319 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:45.319 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:45.319 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:45.319 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.311+0000 7f97c631b640 1 librados: starting msgr at 2024-08-19T12:57:45.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.311+0000 7f97c631b640 1 librados: starting objecter 2024-08-19T12:57:45.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.311+0000 7f97c631b640 1 Processor -- start 2024-08-19T12:57:45.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.311+0000 7f97c631b640 1 -- start start 2024-08-19T12:57:45.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.311+0000 7f97c631b640 1 librados: setting wanted keys 2024-08-19T12:57:45.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.311+0000 7f97c631b640 1 librados: calling monclient init 2024-08-19T12:57:45.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.311+0000 7f97c631b640 10 monclient: init 2024-08-19T12:57:45.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.311+0000 7f97c631b640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:45.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.311+0000 7f97c631b640 10 monclient: _add_conns ranks=[2,5,1,8,6,4,7,0,3] 2024-08-19T12:57:45.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.311+0000 7f97c631b640 10 monclient(hunting): picked mon.noname-b con 0x7f97c006e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:57:45.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.311+0000 7f97c631b640 10 monclient(hunting): picked mon.noname-h con 0x7f97c00a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:45.325 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.312+0000 7f97c631b640 10 monclient(hunting): picked mon.noname-f con 0x7f97c00a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.312+0000 7f97c631b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c0133350 con 0x7f97c00a5d50 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.312+0000 7f97c631b640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c009f6d0 con 0x7f97c006e7b0 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.312+0000 7f97c631b640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c009f950 con 0x7f97c00a5920 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.312+0000 7f97c631b640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.312+0000 7f97c631b640 10 monclient(hunting): authenticate will time out at 1859.399064s 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.312+0000 7f97bffff640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7f97c006e7b0 0x7f97c009e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39158/0 (socket says 172.21.15.26:39158) 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.312+0000 7f97bffff640 1 -- 172.21.15.26:0/2808561254 learned_addr learned my addr 172.21.15.26:0/2808561254 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bffff640 1 -- 172.21.15.26:0/2808561254 >> v1:172.21.15.26:6790/0 conn(0x7f97c006e7b0 legacy=0x7f97c009e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 client.?.objecter ms_handle_connect 0x7f97c006e7b0 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bf7fe640 1 -- 172.21.15.26:0/2808561254 >> v1:172.21.15.104:6791/0 conn(0x7f97c00a5920 legacy=0x7f97c009efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 client.?.objecter ms_handle_connect 0x7f97c00a5920 2024-08-19T12:57:45.326 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 client.?.objecter ms_handle_connect 0x7f97c00a5d50 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97c48b8640 1 -- 172.21.15.26:0/2808561254 >> v1:172.21.15.104:6789/0 conn(0x7f97c00a5d50 legacy=0x7f97c01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4030981428 0 0) 0x7f97c009f6d0 con 0x7f97c006e7b0 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): my global_id is 27569 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97ac004eb0 con 0x7f97c006e7b0 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4044545467 0 0) 0x7f97c009f950 con 0x7f97c00a5920 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.327 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:45.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): my global_id is 184856 2024-08-19T12:57:45.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97c009f6d0 con 0x7f97c00a5920 2024-08-19T12:57:45.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 474539182 0 0) 0x7f97c0133350 con 0x7f97c00a5d50 2024-08-19T12:57:45.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:45.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): my global_id is 204436 2024-08-19T12:57:45.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97c009f950 con 0x7f97c00a5d50 2024-08-19T12:57:45.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 873220669 0 0) 0x7f97ac004eb0 con 0x7f97c006e7b0 2024-08-19T12:57:45.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 --> v1:172.21.15.26:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97c0133350 con 0x7f97c006e7b0 2024-08-19T12:57:45.328 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f97b0001a60 con 0x7f97c006e7b0 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): dump: 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.329 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.5 v1:172.21.15.104:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3943946510 0 0) 0x7f97c009f6d0 con 0x7f97c00a5920 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 --> v1:172.21.15.104:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97ac004eb0 con 0x7f97c00a5920 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.5 v1:172.21.15.104:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f97b4003610 con 0x7f97c00a5920 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): dump: 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.330 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.331 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.1 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4146475578 0 0) 0x7f97c009f950 con 0x7f97c00a5d50 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97c009f6d0 con 0x7f97c00a5d50 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.1 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f97b8004810 con 0x7f97c00a5d50 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 10 monclient(hunting): dump: 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.333 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.333 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.333 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.333 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.333 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.333 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.333 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.333 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.314+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.2 v1:172.21.15.26:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1392725881 0 0) 0x7f97c0133350 con 0x7f97c006e7b0 2024-08-19T12:57:45.333 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 >> v1:172.21.15.104:6791/0 conn(0x7f97c00a5920 legacy=0x7f97c009efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 >> v1:172.21.15.104:6789/0 conn(0x7f97c00a5d50 legacy=0x7f97c01d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 10 monclient: _finish_hunting 0 2024-08-19T12:57:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 1 monclient: found mon.b 2024-08-19T12:57:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c00a0570 con 0x7f97c006e7b0 2024-08-19T12:57:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 10 monclient: _finish_auth 0 2024-08-19T12:57:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 10 monclient: _check_auth_tickets 2024-08-19T12:57:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97c631b640 5 monclient: authenticate success, global_id 27569 2024-08-19T12:57:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97c631b640 10 monclient: _renew_subs 2024-08-19T12:57:45.334 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97c631b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97c631b640 1 -- 172.21.15.26:0/2808561254 --> v1:172.21.15.26:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f97c00a0850 con 0x7f97c006e7b0 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97c631b640 10 client.27569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97c631b640 10 monclient: _renew_subs 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97c631b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97c631b640 1 -- 172.21.15.26:0/2808561254 --> v1:172.21.15.26:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f97c0132c40 con 0x7f97c006e7b0 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97c631b640 1 librados: init done 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.2 v1:172.21.15.26:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97b00048a0 con 0x7f97c006e7b0 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.2 v1:172.21.15.26:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f97b0004db0 con 0x7f97c006e7b0 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97be7fc640 4 set_mon_vals no callback set 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 10 monclient: dump: 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.336 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.315+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.2 v1:172.21.15.26:6790/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f97b0011940 con 0x7f97c006e7b0 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.316+0000 7f97c631b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.316+0000 7f97c631b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.316+0000 7f97c631b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6790/0 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.316+0000 7f97c631b640 1 -- 172.21.15.26:0/2808561254 --> v1:172.21.15.26:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a0005080 con 0x7f97c006e7b0 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.316+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.2 v1:172.21.15.26:6790/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7f97b004df00 con 0x7f97c006e7b0 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.316+0000 7f97bcff9640 10 client.27569.objecter ms_dispatch 0x7f97c0000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.316+0000 7f97bcff9640 3 client.27569.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.316+0000 7f97bcff9640 3 client.27569.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.316+0000 7f97bcff9640 20 client.27569.objecter dump_active .. 0 homeless 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.316+0000 7f97bf7fe640 10 monclient: get_auth_request con 0x7f97ac0418e0 auth_method 0 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.318+0000 7f97bcff9640 10 client.27569.objecter ms_handle_connect 0x7f97ac0418e0 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.320+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mon.2 v1:172.21.15.26:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f97b001fe90 con 0x7f97c006e7b0 2024-08-19T12:57:45.337 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.320+0000 7f97bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:45.338 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.320+0000 7f97bcff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:57:45.505 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.504+0000 7f97c631b640 1 -- 172.21.15.26:0/2808561254 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f97a0000b90 con 0x7f97ac0418e0 2024-08-19T12:57:45.506 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.505+0000 7f97bcff9640 1 -- 172.21.15.26:0/2808561254 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (unknown 3782719169 0 0) 0x7f97a0000b90 con 0x7f97ac0418e0 2024-08-19T12:57:45.506 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 4 to deep-scrub 2024-08-19T12:57:45.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 10 librados: watch_flush enter 2024-08-19T12:57:45.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 10 librados: watch_flush exit 2024-08-19T12:57:45.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 20 client.27569.objecter shutdown clearing up homeless session... 2024-08-19T12:57:45.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 10 client.27569.objecter successfully canceled tick 2024-08-19T12:57:45.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 1 -- 172.21.15.26:0/2808561254 >> v1:172.21.15.104:6800/899887506 conn(0x7f97ac0418e0 legacy=0x7f97ac043da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 10 monclient: shutdown 2024-08-19T12:57:45.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:45.508 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 1 -- 172.21.15.26:0/2808561254 >> v1:172.21.15.26:6790/0 conn(0x7f97c006e7b0 legacy=0x7f97c009e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 1 -- 172.21.15.26:0/2808561254 shutdown_connections 2024-08-19T12:57:45.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 1 -- 172.21.15.26:0/2808561254 >> 172.21.15.26:0/2808561254 conn(0x7f97c00a8740 msgr2=0x7f97c0133970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:45.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 1 -- 172.21.15.26:0/2808561254 shutdown_connections 2024-08-19T12:57:45.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 1 -- 172.21.15.26:0/2808561254 wait complete. 2024-08-19T12:57:45.509 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.507+0000 7f97c631b640 1 librados: shutdown 2024-08-19T12:57:45.518 INFO:tasks.ceph:Scrubbing osd.5 2024-08-19T12:57:45.518 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-08-19T12:57:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.621+0000 7fbfd54ec640 10 monclient: get_monmap_and_config 2024-08-19T12:57:45.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.621+0000 7fbfd54ec640 10 monclient: build_initial_monmap 2024-08-19T12:57:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.621+0000 7fbfd54ec640 10 monclient: monmap: 2024-08-19T12:57:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:45.622537+0000 2024-08-19T12:57:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:45.622537+0000 2024-08-19T12:57:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:45.623 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:45.624 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 1 Processor -- start 2024-08-19T12:57:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 1 -- start start 2024-08-19T12:57:45.624 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 10 monclient: init 2024-08-19T12:57:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 10 monclient: _add_conns ranks=[5,7,2,6,4,1,0,3,8] 2024-08-19T12:57:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 10 monclient(hunting): picked mon.noname-h con 0x7fbfd00a6640 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 10 monclient(hunting): picked mon.noname-i con 0x7fbfd0074110 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 10 monclient(hunting): picked mon.noname-b con 0x7fbfd013f8d0 addr v1:172.21.15.26:6790/0 2024-08-19T12:57:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd009ea70 con 0x7fbfd013f8d0 2024-08-19T12:57:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd009dd20 con 0x7fbfd00a6640 2024-08-19T12:57:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd009e000 con 0x7fbfd0074110 2024-08-19T12:57:45.629 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfd54ec640 10 monclient(hunting): authenticate will time out at 1859.709855s 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfcf7fe640 1 --1- >> v1:172.21.15.26:6790/0 conn(0x7fbfd013f8d0 0x7fbfd0141cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39172/0 (socket says 172.21.15.26:39172) 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.622+0000 7fbfcf7fe640 1 -- 172.21.15.26:0/4034289085 learned_addr learned my addr 172.21.15.26:0/4034289085 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcf7fe640 1 -- 172.21.15.26:0/4034289085 >> v1:172.21.15.26:6790/0 conn(0x7fbfd013f8d0 legacy=0x7fbfd0141cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfce7fc640 1 -- 172.21.15.26:0/4034289085 >> v1:172.21.15.104:6792/0 conn(0x7fbfd0074110 legacy=0x7fbfd00a7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 725903819 0 0) 0x7fbfd009ea70 con 0x7fbfd013f8d0 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfceffd640 1 -- 172.21.15.26:0/4034289085 >> v1:172.21.15.104:6791/0 conn(0x7fbfd00a6640 legacy=0x7fbfd00a6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient(hunting): my global_id is 27578 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfb8003520 con 0x7fbfd013f8d0 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182708937 0 0) 0x7fbfd009e000 con 0x7fbfd0074110 2024-08-19T12:57:45.630 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient(hunting): my global_id is 136627 2024-08-19T12:57:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfd009ea70 con 0x7fbfd0074110 2024-08-19T12:57:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1923947226 0 0) 0x7fbfd009dd20 con 0x7fbfd00a6640 2024-08-19T12:57:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient(hunting): my global_id is 184865 2024-08-19T12:57:45.631 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfd009e000 con 0x7fbfd00a6640 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 <== mon.2 v1:172.21.15.26:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2196430432 0 0) 0x7fbfb8003520 con 0x7fbfd013f8d0 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 >> v1:172.21.15.104:6792/0 conn(0x7fbfd0074110 legacy=0x7fbfd00a7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 >> v1:172.21.15.104:6791/0 conn(0x7fbfd00a6640 legacy=0x7fbfd00a6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient: _finish_hunting 0 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 monclient: found mon.noname-b 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.26:6790/0 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 --> v1:172.21.15.26:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd00a7de0 con 0x7fbfd013f8d0 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient: _finish_auth 0 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient: _check_auth_tickets 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfd54ec640 5 monclient: authenticate success, global_id 27578 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfd54ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 <== mon.2 v1:172.21.15.26:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbfc8001e50 con 0x7fbfd013f8d0 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient: dump: 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.633 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfd54ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 <== mon.2 v1:172.21.15.26:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbfc8004c20 con 0x7fbfd013f8d0 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 1 -- 172.21.15.26:0/4034289085 <== mon.2 v1:172.21.15.26:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbfc8005190 con 0x7fbfd013f8d0 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfcdffb640 10 monclient: dump: 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.634 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.635 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfd54ec640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.625+0000 7fbfd54ec640 4 set_mon_vals no callback set 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.626+0000 7fbfd54ec640 10 monclient: shutdown 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.626+0000 7fbfd54ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.626+0000 7fbfd54ec640 1 -- 172.21.15.26:0/4034289085 >> v1:172.21.15.26:6790/0 conn(0x7fbfd013f8d0 legacy=0x7fbfd0141cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.626+0000 7fbfd54ec640 1 -- 172.21.15.26:0/4034289085 shutdown_connections 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.626+0000 7fbfd54ec640 1 -- 172.21.15.26:0/4034289085 >> 172.21.15.26:0/4034289085 conn(0x7fbfd00a8740 msgr2=0x7fbfd00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.626+0000 7fbfd54ec640 1 -- 172.21.15.26:0/4034289085 shutdown_connections 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.626+0000 7fbfd54ec640 1 -- 172.21.15.26:0/4034289085 wait complete. 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.626+0000 7fbfd54ec640 10 monclient: build_initial_monmap 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.626+0000 7fbfd54ec640 10 monclient: monmap: 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:45.636 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 1 librados: starting msgr at 2024-08-19T12:57:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 1 librados: starting objecter 2024-08-19T12:57:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 1 Processor -- start 2024-08-19T12:57:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 1 -- start start 2024-08-19T12:57:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 1 librados: setting wanted keys 2024-08-19T12:57:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 1 librados: calling monclient init 2024-08-19T12:57:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 10 monclient: init 2024-08-19T12:57:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 10 monclient: _add_conns ranks=[3,0,8,6,5,1,7,4,2] 2024-08-19T12:57:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 10 monclient(hunting): picked mon.noname-g con 0x7fbfd0074110 addr v1:172.21.15.104:6790/0 2024-08-19T12:57:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 10 monclient(hunting): picked mon.noname-a con 0x7fbfd00a6640 addr v1:172.21.15.26:6789/0 2024-08-19T12:57:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 10 monclient(hunting): picked mon.noname-e con 0x7fbfd013f8d0 addr v1:172.21.15.26:6793/0 2024-08-19T12:57:45.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd01230b0 con 0x7fbfd00a6640 2024-08-19T12:57:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd00aae50 con 0x7fbfd0074110 2024-08-19T12:57:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfce7fc640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fbfd00a6640 legacy=0x7fbfd01229a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:57:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd00ab360 con 0x7fbfd013f8d0 2024-08-19T12:57:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfd54ec640 10 monclient(hunting): authenticate will time out at 1859.714869s 2024-08-19T12:57:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfceffd640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7fbfd0074110 legacy=0x7fbfd0122290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfb77fe640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:57:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfcf7fe640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7fbfd013f8d0 0x7fbfd01d6530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:57090/0 (socket says 172.21.15.26:57090) 2024-08-19T12:57:45.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.627+0000 7fbfcf7fe640 1 -- 172.21.15.26:0/766498569 learned_addr learned my addr 172.21.15.26:0/766498569 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfcf7fe640 1 -- 172.21.15.26:0/766498569 >> v1:172.21.15.26:6793/0 conn(0x7fbfd013f8d0 legacy=0x7fbfd01d6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 client.?.objecter ms_handle_connect 0x7fbfd013f8d0 2024-08-19T12:57:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458074940 0 0) 0x7fbfd00ab360 con 0x7fbfd013f8d0 2024-08-19T12:57:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient(hunting): my global_id is 184733 2024-08-19T12:57:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfbc004ee0 con 0x7fbfd013f8d0 2024-08-19T12:57:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2801151500 0 0) 0x7fbfbc004ee0 con 0x7fbfd013f8d0 2024-08-19T12:57:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbfd00ab360 con 0x7fbfd013f8d0 2024-08-19T12:57:45.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbfc80020a0 con 0x7fbfd013f8d0 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient(hunting): dump: 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.645 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2253806183 0 0) 0x7fbfd00ab360 con 0x7fbfd013f8d0 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 >> v1:172.21.15.104:6790/0 conn(0x7fbfd0074110 legacy=0x7fbfd0122290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 >> v1:172.21.15.26:6789/0 conn(0x7fbfd00a6640 legacy=0x7fbfd01229a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient: _finish_hunting 0 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 1 monclient: found mon.e 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:45.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd0124de0 con 0x7fbfd013f8d0 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient: _finish_auth 0 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient: _check_auth_tickets 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfd54ec640 5 monclient: authenticate success, global_id 184733 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfd54ec640 10 monclient: _renew_subs 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfd54ec640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfd54ec640 1 -- 172.21.15.26:0/766498569 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd0125c50 con 0x7fbfd013f8d0 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfd54ec640 10 client.184733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfd54ec640 10 monclient: _renew_subs 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfd54ec640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfd54ec640 1 -- 172.21.15.26:0/766498569 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fbfd0124450 con 0x7fbfd013f8d0 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfd54ec640 1 librados: init done 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbfc80053c0 con 0x7fbfd013f8d0 2024-08-19T12:57:45.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:45.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fbfc8004c20 con 0x7fbfd013f8d0 2024-08-19T12:57:45.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:45.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfcd7fa640 4 set_mon_vals no callback set 2024-08-19T12:57:45.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:57:45.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.630+0000 7fbfb77fe640 10 monclient: dump: 2024-08-19T12:57:45.648 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.648 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.648 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.648 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.648 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfd54ec640 10 client.184733.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:45.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfd54ec640 15 client.184733.objecter _session_command_op_assign -1 1 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfd54ec640 20 client.184733.objecter _get_session osd=-1 returning homeless 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfd54ec640 10 client.184733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfd54ec640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfd54ec640 1 -- 172.21.15.26:0/766498569 --> v1:172.21.15.26:6793/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbfb0002910 con 0x7fbfd013f8d0 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fbfc8011b70 con 0x7fbfd013f8d0 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7fbfc804e1f0 con 0x7fbfd013f8d0 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 10 client.184733.objecter ms_dispatch 0x7fbfd0000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 3 client.184733.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 3 client.184733.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 10 client.184733.objecter checking command 1 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 20 client.184733.objecter _get_session s=0x7fbfbc048e70 osd=5 3 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 20 client.184733.objecter put_session s=0x7fbfbc048e70 osd=5 3 2024-08-19T12:57:45.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 15 client.184733.objecter _session_command_op_remove -1 1 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 20 client.184733.objecter _get_session s=0x7fbfbc048e70 osd=5 3 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 20 client.184733.objecter get_session s=0x7fbfbc048e70 osd=5 3 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 15 client.184733.objecter _session_command_op_assign 5 1 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 20 client.184733.objecter put_session s=0x7fbfbc048e70 osd=5 4 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 10 client.184733.objecter _send_command 1 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 --> v1:172.21.15.104:6801/3713183117 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfc8004c20 con 0x7fbfbc049520 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 20 client.184733.objecter dump_active .. 0 homeless 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_get_version_reply(handle=1 version=701) ==== 24+0+0 (unknown 2985145993 0 0) 0x7fbfc804e660 con 0x7fbfd013f8d0 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.631+0000 7fbfb77fe640 10 monclient: handle_get_version_reply finishing 1 version 701 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.632+0000 7fbfceffd640 10 monclient: get_auth_request con 0x7fbfbc040f90 auth_method 0 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.632+0000 7fbfce7fc640 10 monclient: get_auth_request con 0x7fbfbc049520 auth_method 0 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.634+0000 7fbfb77fe640 10 client.184733.objecter ms_handle_connect 0x7fbfbc040f90 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.636+0000 7fbfb77fe640 10 client.184733.objecter ms_handle_connect 0x7fbfbc049520 2024-08-19T12:57:45.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.637+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 <== osd.5 v1:172.21.15.104:6801/3713183117 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbfc8004c20 con 0x7fbfbc049520 2024-08-19T12:57:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.637+0000 7fbfb77fe640 10 client.184733.objecter ms_dispatch 0x7fbfd0000b90 command_reply(tid 1: 0 ) 2024-08-19T12:57:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.637+0000 7fbfb77fe640 10 client.184733.objecter _finish_command 1 = system:0 2024-08-19T12:57:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.637+0000 7fbfb77fe640 20 client.184733.objecter put_session s=0x7fbfbc048e70 osd=5 4 2024-08-19T12:57:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.637+0000 7fbfb77fe640 15 client.184733.objecter _session_command_op_remove 5 1 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfd54ec640 10 client.184733.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfd54ec640 15 client.184733.objecter _session_command_op_assign -1 2 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfd54ec640 20 client.184733.objecter _get_session s=0x7fbfbc048e70 osd=5 3 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfd54ec640 20 client.184733.objecter put_session s=0x7fbfbc048e70 osd=5 3 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfd54ec640 20 client.184733.objecter _get_session s=0x7fbfbc048e70 osd=5 3 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfd54ec640 15 client.184733.objecter _session_command_op_remove -1 2 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfd54ec640 20 client.184733.objecter get_session s=0x7fbfbc048e70 osd=5 3 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfd54ec640 15 client.184733.objecter _session_command_op_assign 5 2 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfd54ec640 20 client.184733.objecter put_session s=0x7fbfbc048e70 osd=5 4 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfd54ec640 10 client.184733.objecter _send_command 2 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfd54ec640 1 -- 172.21.15.26:0/766498569 --> v1:172.21.15.104:6801/3713183117 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbfb00012b0 con 0x7fbfbc049520 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfb77fe640 1 -- 172.21.15.26:0/766498569 <== osd.5 v1:172.21.15.104:6801/3713183117 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fbfb00012b0 con 0x7fbfbc049520 2024-08-19T12:57:45.653 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfb77fe640 10 client.184733.objecter ms_dispatch 0x7fbfd0000b90 command_reply(tid 2: 0 ) 2024-08-19T12:57:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfb77fe640 10 client.184733.objecter _finish_command 2 = system:0 2024-08-19T12:57:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfb77fe640 20 client.184733.objecter put_session s=0x7fbfbc048e70 osd=5 4 2024-08-19T12:57:45.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.650+0000 7fbfb77fe640 15 client.184733.objecter _session_command_op_remove 5 2 2024-08-19T12:57:45.654 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-08-19T12:57:45.654 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-08-19T12:57:45.654 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-08-19T12:57:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 10 librados: watch_flush enter 2024-08-19T12:57:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 10 librados: watch_flush exit 2024-08-19T12:57:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 10 client.184733.objecter close_session for osd.5 2024-08-19T12:57:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 1 -- 172.21.15.26:0/766498569 >> v1:172.21.15.104:6801/3713183117 conn(0x7fbfbc049520 legacy=0x7fbfbc04b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 20 client.184733.objecter put_session s=0x7fbfbc048e70 osd=5 1 2024-08-19T12:57:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 20 client.184733.objecter shutdown clearing up homeless session... 2024-08-19T12:57:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 10 client.184733.objecter successfully canceled tick 2024-08-19T12:57:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 1 -- 172.21.15.26:0/766498569 >> v1:172.21.15.104:6800/899887506 conn(0x7fbfbc040f90 legacy=0x7fbfbc043450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 10 monclient: shutdown 2024-08-19T12:57:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 1 -- 172.21.15.26:0/766498569 >> v1:172.21.15.26:6793/0 conn(0x7fbfd013f8d0 legacy=0x7fbfd01d6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfcf7fe640 1 -- 172.21.15.26:0/766498569 reap_dead start 2024-08-19T12:57:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 1 -- 172.21.15.26:0/766498569 shutdown_connections 2024-08-19T12:57:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 1 -- 172.21.15.26:0/766498569 >> 172.21.15.26:0/766498569 conn(0x7fbfd00a8740 msgr2=0x7fbfd0142b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 1 -- 172.21.15.26:0/766498569 shutdown_connections 2024-08-19T12:57:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 1 -- 172.21.15.26:0/766498569 wait complete. 2024-08-19T12:57:45.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.651+0000 7fbfd54ec640 1 librados: shutdown 2024-08-19T12:57:45.663 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-08-19T12:57:45.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.762+0000 7f79d6d4c640 10 monclient: get_monmap_and_config 2024-08-19T12:57:45.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.762+0000 7f79d6d4c640 10 monclient: build_initial_monmap 2024-08-19T12:57:45.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.762+0000 7f79d6d4c640 10 monclient: monmap: 2024-08-19T12:57:45.763 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:45.763 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.763 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:45.763531+0000 2024-08-19T12:57:45.763 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:45.763531+0000 2024-08-19T12:57:45.763 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:45.764 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:45.764 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 1 Processor -- start 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 1 -- start start 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 10 monclient: init 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 10 monclient: _add_conns ranks=[8,2,5,3,4,1,0,7,6] 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 10 monclient(hunting): picked mon.noname-e con 0x7f79d006e7b0 addr v1:172.21.15.26:6793/0 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 10 monclient(hunting): picked mon.noname-b con 0x7f79d00a5920 addr v1:172.21.15.26:6790/0 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 10 monclient(hunting): picked mon.noname-h con 0x7f79d00a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79d0136790 con 0x7f79d00a5920 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79d009ddf0 con 0x7f79d00a5d50 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79d009e0d0 con 0x7f79d006e7b0 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.763+0000 7f79d6d4c640 10 monclient(hunting): authenticate will time out at 1859.850954s 2024-08-19T12:57:45.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.764+0000 7f79d4ae8640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f79d006e7b0 0x7f79d0136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:57102/0 (socket says 172.21.15.26:57102) 2024-08-19T12:57:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.764+0000 7f79d4ae8640 1 -- 172.21.15.26:0/2643020926 learned_addr learned my addr 172.21.15.26:0/2643020926 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:45.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.764+0000 7f79cffff640 1 --1- 172.21.15.26:0/2643020926 >> v1:172.21.15.26:6790/0 conn(0x7f79d00a5920 0x7f79d013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6790/0 says I am v1:172.21.15.26:39176/0 (socket says 172.21.15.26:39176) 2024-08-19T12:57:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79d4ae8640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.26:6793/0 conn(0x7f79d006e7b0 legacy=0x7f79d0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cffff640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.26:6790/0 conn(0x7f79d00a5920 legacy=0x7f79d013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79d52e9640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.104:6791/0 conn(0x7f79d00a5d50 legacy=0x7f79d0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006249414 0 0) 0x7f79d009e0d0 con 0x7f79d006e7b0 2024-08-19T12:57:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient(hunting): my global_id is 184742 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79c0003520 con 0x7f79d006e7b0 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 <== mon.2 v1:172.21.15.26:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 952536521 0 0) 0x7f79d0136790 con 0x7f79d00a5920 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient(hunting): my global_id is 27587 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 --> v1:172.21.15.26:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79d009e0d0 con 0x7f79d00a5920 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3494637332 0 0) 0x7f79c0003520 con 0x7f79d006e7b0 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.104:6791/0 conn(0x7f79d00a5d50 legacy=0x7f79d0141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.26:6790/0 conn(0x7f79d00a5920 legacy=0x7f79d013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:57:45.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 1 monclient: found mon.noname-e 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.26:6793/0 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d009e3b0 con 0x7f79d006e7b0 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient: _finish_auth 0 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79c4001ae0 con 0x7f79d006e7b0 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79d6d4c640 5 monclient: authenticate success, global_id 184742 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79d6d4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.766+0000 7f79cf7fe640 10 monclient: dump: 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.771 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.772 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79d6d4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79d4ae8640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.26:6793/0 conn(0x7f79d006e7b0 legacy=0x7f79d0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79d4ae8640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.26:6793/0 conn(0x7f79d006e7b0 legacy=0x7f79d0136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79d4ae8640 1 --1- 172.21.15.26:0/2643020926 >> v1:172.21.15.26:6793/0 conn(0x7f79d006e7b0 0x7f79d0136080 :-1 s=OPENED pgs=3718 cs=1 l=1).handle_message read tag failed 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79d4ae8640 1 --1- 172.21.15.26:0/2643020926 >> v1:172.21.15.26:6793/0 conn(0x7f79d006e7b0 0x7f79d0136080 :-1 s=OPENED pgs=3718 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.26:6793/0 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.26:6793/0 conn(0x7f79d006e7b0 legacy=0x7f79d0136080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 10 monclient: _add_conns ranks=[7,3,4,2,1,0,6,5,8] 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 10 monclient(hunting): picked mon.i con 0x7f79c00037a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 10 monclient(hunting): picked mon.g con 0x7f79c0004290 addr v1:172.21.15.104:6790/0 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 10 monclient(hunting): picked mon.c con 0x7f79c000c190 addr v1:172.21.15.26:6791/0 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f79c0003520 con 0x7f79c0004290 2024-08-19T12:57:45.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f79d009e0d0 con 0x7f79c000c190 2024-08-19T12:57:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f79c8001820 con 0x7f79c00037a0 2024-08-19T12:57:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 10 monclient(hunting): ms_handle_reset stray mon v1:172.21.15.26:6793/0 2024-08-19T12:57:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 <== mon.8 v1:172.21.15.26:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79c40048a0 con 0x7f79d006e7b0 2024-08-19T12:57:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cf7fe640 10 monclient(hunting): discarding stray monitor message config(0 keys) 2024-08-19T12:57:45.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.767+0000 7f79cffff640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.104:6790/0 conn(0x7f79c0004290 legacy=0x7f79c000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:57:45.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79d52e9640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.26:6791/0 conn(0x7f79c000c190 legacy=0x7f79c000e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906852984 0 0) 0x7f79d009e0d0 con 0x7f79c000c190 2024-08-19T12:57:45.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79d4ae8640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.104:6792/0 conn(0x7f79c00037a0 legacy=0x7f79c0003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 --> v1:172.21.15.26:6791/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f79c00110c0 con 0x7f79c000c190 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 2684192915 0 0) 0x7f79c00110c0 con 0x7f79c000c190 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.104:6792/0 conn(0x7f79c00037a0 legacy=0x7f79c0003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.104:6790/0 conn(0x7f79c0004290 legacy=0x7f79c000a9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 10 monclient: _finish_hunting 0 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 1 monclient: found mon.c 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79d52e9640 1 -- 172.21.15.26:0/2643020926 reap_dead start 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=2+}) -- 0x7f79d0136790 con 0x7f79c000c190 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 10 monclient: _finish_auth 0 2024-08-19T12:57:45.781 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 10 monclient: _check_auth_tickets 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79c8003990 con 0x7f79c000c190 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 10 monclient: dump: 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.782 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79d6d4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 1 -- 172.21.15.26:0/2643020926 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79c8003c60 con 0x7f79c000c190 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79cf7fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79d6d4c640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.769+0000 7f79d6d4c640 4 set_mon_vals no callback set 2024-08-19T12:57:45.783 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.770+0000 7f79d6d4c640 10 monclient: shutdown 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.770+0000 7f79d6d4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.770+0000 7f79d6d4c640 1 -- 172.21.15.26:0/2643020926 >> v1:172.21.15.26:6791/0 conn(0x7f79c000c190 legacy=0x7f79c000e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.770+0000 7f79d6d4c640 1 -- 172.21.15.26:0/2643020926 shutdown_connections 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.770+0000 7f79d6d4c640 1 -- 172.21.15.26:0/2643020926 >> 172.21.15.26:0/2643020926 conn(0x7f79d00a8740 msgr2=0x7f79d00a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.770+0000 7f79d6d4c640 1 -- 172.21.15.26:0/2643020926 shutdown_connections 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.770+0000 7f79d6d4c640 1 -- 172.21.15.26:0/2643020926 wait complete. 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.771+0000 7f79d6d4c640 10 monclient: build_initial_monmap 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.771+0000 7f79d6d4c640 10 monclient: monmap: 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.784 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.771+0000 7f79d6d4c640 1 librados: starting msgr at 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.771+0000 7f79d6d4c640 1 librados: starting objecter 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.771+0000 7f79d6d4c640 1 Processor -- start 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.771+0000 7f79d6d4c640 1 -- start start 2024-08-19T12:57:45.785 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.771+0000 7f79d6d4c640 1 librados: setting wanted keys 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.771+0000 7f79d6d4c640 1 librados: calling monclient init 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.771+0000 7f79d6d4c640 10 monclient: init 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79d6d4c640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79d6d4c640 10 monclient: _add_conns ranks=[6,8,0,4,3,7,5,2,1] 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79d6d4c640 10 monclient(hunting): picked mon.noname-d con 0x7f79c000c190 addr v1:172.21.15.26:6792/0 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79d6d4c640 10 monclient(hunting): picked mon.noname-e con 0x7f79d0132aa0 addr v1:172.21.15.26:6793/0 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79d6d4c640 10 monclient(hunting): picked mon.noname-a con 0x7f79d0132f20 addr v1:172.21.15.26:6789/0 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79d4ae8640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f79c000c190 legacy=0x7f79d01ce100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79d6d4c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79d01cedf0 con 0x7f79d0132f20 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79d6d4c640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79d01272a0 con 0x7f79c000c190 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79d52e9640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f79d0132f20 legacy=0x7f79d01d6940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79d6d4c640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79d0134f70 con 0x7f79d0132aa0 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79d6d4c640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:45.786 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79d6d4c640 10 monclient(hunting): authenticate will time out at 1859.859346s 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79cffff640 1 --1- >> v1:172.21.15.26:6793/0 conn(0x7f79d0132aa0 0x7f79d01d3210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6793/0 says I am v1:172.21.15.26:57118/0 (socket says 172.21.15.26:57118) 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.772+0000 7f79cffff640 1 -- 172.21.15.26:0/100473311 learned_addr learned my addr 172.21.15.26:0/100473311 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cffff640 1 -- 172.21.15.26:0/100473311 >> v1:172.21.15.26:6793/0 conn(0x7f79d0132aa0 legacy=0x7f79d01d3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 10 client.?.objecter ms_handle_connect 0x7f79d0132aa0 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 <== mon.8 v1:172.21.15.26:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1406228226 0 0) 0x7f79d0134f70 con 0x7f79d0132aa0 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 10 monclient(hunting): _negotiate 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 10 monclient(hunting): my global_id is 184751 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 --> v1:172.21.15.26:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79b4004ee0 con 0x7f79d0132aa0 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 <== mon.8 v1:172.21.15.26:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2168268708 0 0) 0x7f79b4004ee0 con 0x7f79d0132aa0 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 --> v1:172.21.15.26:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79d0134f70 con 0x7f79d0132aa0 2024-08-19T12:57:45.787 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 <== mon.8 v1:172.21.15.26:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79bc004b20 con 0x7f79d0132aa0 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 10 monclient(hunting): dump: 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.788 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.774+0000 7f79cd7fa640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 <== mon.8 v1:172.21.15.26:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 593652188 0 0) 0x7f79d0134f70 con 0x7f79d0132aa0 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 >> v1:172.21.15.26:6792/0 conn(0x7f79c000c190 legacy=0x7f79d01ce100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 >> v1:172.21.15.26:6789/0 conn(0x7f79d0132f20 legacy=0x7f79d01d6940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 10 monclient: _finish_hunting 0 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 1 monclient: found mon.e 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 --> v1:172.21.15.26:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d0124f40 con 0x7f79d0132aa0 2024-08-19T12:57:45.789 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 10 monclient: _finish_auth 0 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 10 monclient: _check_auth_tickets 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79d6d4c640 5 monclient: authenticate success, global_id 184751 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79d6d4c640 10 monclient: _renew_subs 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79d6d4c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79d6d4c640 1 -- 172.21.15.26:0/100473311 --> v1:172.21.15.26:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f79d0126a00 con 0x7f79d0132aa0 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79d6d4c640 10 client.184751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79d6d4c640 10 monclient: _renew_subs 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79d6d4c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79d6d4c640 1 -- 172.21.15.26:0/100473311 --> v1:172.21.15.26:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f79d01d8060 con 0x7f79d0132aa0 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79d6d4c640 1 librados: init done 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 <== mon.8 v1:172.21.15.26:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79bc004f70 con 0x7f79d0132aa0 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:45.790 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 <== mon.8 v1:172.21.15.26:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f79bc005460 con 0x7f79d0132aa0 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79ceffd640 4 set_mon_vals no callback set 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.775+0000 7f79cd7fa640 10 monclient: dump: 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.776+0000 7f79d6d4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.776+0000 7f79d6d4c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.776+0000 7f79d6d4c640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.26:6793/0 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.776+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 <== mon.8 v1:172.21.15.26:6793/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f79bc0118c0 con 0x7f79d0132aa0 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.776+0000 7f79d6d4c640 1 -- 172.21.15.26:0/100473311 --> v1:172.21.15.26:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b0005080 con 0x7f79d0132aa0 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.776+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 <== mon.8 v1:172.21.15.26:6793/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7f79bc04e100 con 0x7f79d0132aa0 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.776+0000 7f79cd7fa640 10 client.184751.objecter ms_dispatch 0x7f79d0000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:45.792 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.776+0000 7f79cd7fa640 3 client.184751.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:45.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.776+0000 7f79cd7fa640 3 client.184751.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:45.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.776+0000 7f79cd7fa640 20 client.184751.objecter dump_active .. 0 homeless 2024-08-19T12:57:45.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.776+0000 7f79d4ae8640 10 monclient: get_auth_request con 0x7f79b4040dd0 auth_method 0 2024-08-19T12:57:45.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.779+0000 7f79cd7fa640 10 client.184751.objecter ms_handle_connect 0x7f79b4040dd0 2024-08-19T12:57:45.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.780+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 <== mon.8 v1:172.21.15.26:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f79bc017740 con 0x7f79d0132aa0 2024-08-19T12:57:45.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.780+0000 7f79cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:45.793 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.780+0000 7f79cd7fa640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:57:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.963+0000 7f79d6d4c640 1 -- 172.21.15.26:0/100473311 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f79b0000b90 con 0x7f79b4040dd0 2024-08-19T12:57:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.963+0000 7f79cd7fa640 1 -- 172.21.15.26:0/100473311 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (unknown 2264484800 0 0) 0x7f79b0000b90 con 0x7f79b4040dd0 2024-08-19T12:57:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:instructed osd(s) 5 to deep-scrub 2024-08-19T12:57:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.965+0000 7f79d6d4c640 10 librados: watch_flush enter 2024-08-19T12:57:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.965+0000 7f79d6d4c640 10 librados: watch_flush exit 2024-08-19T12:57:45.966 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.965+0000 7f79d6d4c640 20 client.184751.objecter shutdown clearing up homeless session... 2024-08-19T12:57:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.965+0000 7f79d6d4c640 10 client.184751.objecter successfully canceled tick 2024-08-19T12:57:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.965+0000 7f79d6d4c640 1 -- 172.21.15.26:0/100473311 >> v1:172.21.15.104:6800/899887506 conn(0x7f79b4040dd0 legacy=0x7f79b4043290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.965+0000 7f79d6d4c640 10 monclient: shutdown 2024-08-19T12:57:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.965+0000 7f79d6d4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.965+0000 7f79d6d4c640 1 -- 172.21.15.26:0/100473311 >> v1:172.21.15.26:6793/0 conn(0x7f79d0132aa0 legacy=0x7f79d01d3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.965+0000 7f79d6d4c640 1 -- 172.21.15.26:0/100473311 shutdown_connections 2024-08-19T12:57:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.965+0000 7f79d6d4c640 1 -- 172.21.15.26:0/100473311 >> 172.21.15.26:0/100473311 conn(0x7f79d00a8740 msgr2=0x7f79d0133970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.965+0000 7f79d6d4c640 1 -- 172.21.15.26:0/100473311 shutdown_connections 2024-08-19T12:57:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.965+0000 7f79d6d4c640 1 -- 172.21.15.26:0/100473311 wait complete. 2024-08-19T12:57:45.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:45.966+0000 7f79d6d4c640 1 librados: shutdown 2024-08-19T12:57:45.977 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T12:57:46.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.075+0000 7fec6167e640 10 monclient: get_monmap_and_config 2024-08-19T12:57:46.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.075+0000 7fec6167e640 10 monclient: build_initial_monmap 2024-08-19T12:57:46.076 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.075+0000 7fec6167e640 10 monclient: monmap: 2024-08-19T12:57:46.076 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:46.076 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:46.076 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:57:46.076673+0000 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:57:46.076673+0000 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:46.077 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:46.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.076+0000 7fec6167e640 1 Processor -- start 2024-08-19T12:57:46.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.076+0000 7fec6167e640 1 -- start start 2024-08-19T12:57:46.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.076+0000 7fec6167e640 10 monclient: init 2024-08-19T12:57:46.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.076+0000 7fec6167e640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:46.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.076+0000 7fec6167e640 10 monclient: _add_conns ranks=[0,5,4,7,3,2,8,1,6] 2024-08-19T12:57:46.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.076+0000 7fec6167e640 10 monclient(hunting): picked mon.noname-a con 0x7fec5c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:57:46.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.077+0000 7fec6167e640 10 monclient(hunting): picked mon.noname-h con 0x7fec5c0a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:46.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.077+0000 7fec6167e640 10 monclient(hunting): picked mon.noname-c con 0x7fec5c0a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:57:46.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.077+0000 7fec6167e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec5c136790 con 0x7fec5c06e7b0 2024-08-19T12:57:46.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.077+0000 7fec5affd640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fec5c06e7b0 legacy=0x7fec5c136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:57:46.079 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.077+0000 7fec6167e640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec5c09ddf0 con 0x7fec5c0a5d50 2024-08-19T12:57:46.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.077+0000 7fec6167e640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec5c09e0d0 con 0x7fec5c0a5920 2024-08-19T12:57:46.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.077+0000 7fec6167e640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:46.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.077+0000 7fec6167e640 10 monclient(hunting): authenticate will time out at 1860.164102s 2024-08-19T12:57:46.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.077+0000 7fec5b7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fec5c0a5d50 0x7fec5c141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36808/0 (socket says 172.21.15.26:36808) 2024-08-19T12:57:46.080 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.077+0000 7fec5b7fe640 1 -- 172.21.15.26:0/1193712782 learned_addr learned my addr 172.21.15.26:0/1193712782 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:46.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec5b7fe640 1 -- 172.21.15.26:0/1193712782 >> v1:172.21.15.26:6791/0 conn(0x7fec5c0a5d50 legacy=0x7fec5c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:46.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 1 -- 172.21.15.26:0/1193712782 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737797687 0 0) 0x7fec4c001450 con 0x7fec5c0a5d50 2024-08-19T12:57:46.083 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient(hunting): _negotiate 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient(hunting): my global_id is 184873 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 1 -- 172.21.15.26:0/1193712782 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec48003520 con 0x7fec5c0a5d50 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec5a7fc640 1 -- 172.21.15.26:0/1193712782 >> v1:172.21.15.104:6791/0 conn(0x7fec5c0a5920 legacy=0x7fec5c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 1 -- 172.21.15.26:0/1193712782 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1612542100 0 0) 0x7fec48003520 con 0x7fec5c0a5d50 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 1 -- 172.21.15.26:0/1193712782 >> v1:172.21.15.104:6791/0 conn(0x7fec5c0a5920 legacy=0x7fec5c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 1 -- 172.21.15.26:0/1193712782 >> v1:172.21.15.26:6789/0 conn(0x7fec5c06e7b0 legacy=0x7fec5c136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient: _finish_hunting 0 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 1 monclient: found mon.noname-c 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:57:46.084 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 1 -- 172.21.15.26:0/1193712782 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec5c09e3b0 con 0x7fec5c0a5d50 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient: _finish_auth 0 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient: _check_auth_tickets 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec6167e640 5 monclient: authenticate success, global_id 184873 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec6167e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 1 -- 172.21.15.26:0/1193712782 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fec4c001cb0 con 0x7fec5c0a5d50 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient: dump: 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:46.085 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec6167e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 1 -- 172.21.15.26:0/1193712782 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec4c004990 con 0x7fec5c0a5d50 2024-08-19T12:57:46.086 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 1 -- 172.21.15.26:0/1193712782 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fec4c004f00 con 0x7fec5c0a5d50 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec6167e640 10 monclient: get_monmap_and_config success 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec6167e640 4 set_mon_vals no callback set 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.079+0000 7fec59ffb640 10 monclient: dump: 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:46.087 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.080+0000 7fec6167e640 10 monclient: shutdown 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.080+0000 7fec6167e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.080+0000 7fec6167e640 1 -- 172.21.15.26:0/1193712782 >> v1:172.21.15.26:6791/0 conn(0x7fec5c0a5d50 legacy=0x7fec5c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.080+0000 7fec6167e640 1 -- 172.21.15.26:0/1193712782 shutdown_connections 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.080+0000 7fec6167e640 1 -- 172.21.15.26:0/1193712782 >> 172.21.15.26:0/1193712782 conn(0x7fec5c0a8740 msgr2=0x7fec5c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.080+0000 7fec6167e640 1 -- 172.21.15.26:0/1193712782 shutdown_connections 2024-08-19T12:57:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.080+0000 7fec6167e640 1 -- 172.21.15.26:0/1193712782 wait complete. 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.080+0000 7fec6167e640 10 monclient: build_initial_monmap 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.080+0000 7fec6167e640 10 monclient: monmap: 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:57:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 1 librados: starting msgr at 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 1 librados: starting objecter 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 1 Processor -- start 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 1 -- start start 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 1 librados: setting wanted keys 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 1 librados: calling monclient init 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 10 monclient: init 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 10 monclient: _reopen_session rank -1 2024-08-19T12:57:46.090 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 10 monclient: _add_conns ranks=[5,0,4,1,7,6,2,8,3] 2024-08-19T12:57:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 10 monclient(hunting): picked mon.noname-h con 0x7fec5c06e7b0 addr v1:172.21.15.104:6791/0 2024-08-19T12:57:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 10 monclient(hunting): picked mon.noname-a con 0x7fec5c0a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:57:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 10 monclient(hunting): picked mon.noname-c con 0x7fec5c0a5d50 addr v1:172.21.15.26:6791/0 2024-08-19T12:57:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec5c13b3c0 con 0x7fec5c0a5920 2024-08-19T12:57:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec6167e640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec5c139060 con 0x7fec5c0a5d50 2024-08-19T12:57:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.081+0000 7fec5a7fc640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7fec5c0a5920 legacy=0x7fec5c1380f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:57:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.082+0000 7fec6167e640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec5c13ab20 con 0x7fec5c06e7b0 2024-08-19T12:57:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.082+0000 7fec6167e640 10 monclient(hunting): _renew_subs 2024-08-19T12:57:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.082+0000 7fec6167e640 10 monclient(hunting): authenticate will time out at 1860.169021s 2024-08-19T12:57:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.082+0000 7fec5b7fe640 1 --1- >> v1:172.21.15.26:6791/0 conn(0x7fec5c0a5d50 0x7fec5c1d6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6791/0 says I am v1:172.21.15.26:36820/0 (socket says 172.21.15.26:36820) 2024-08-19T12:57:46.091 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.082+0000 7fec5b7fe640 1 -- 172.21.15.26:0/3212279365 learned_addr learned my addr 172.21.15.26:0/3212279365 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:57:46.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec5b7fe640 1 -- 172.21.15.26:0/3212279365 >> v1:172.21.15.26:6791/0 conn(0x7fec5c0a5d50 legacy=0x7fec5c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:46.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 client.?.objecter ms_handle_connect 0x7fec5c0a5d50 2024-08-19T12:57:46.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:46.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 223294711 0 0) 0x7fec5c139060 con 0x7fec5c0a5d50 2024-08-19T12:57:46.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:46.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:46.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:46.095 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient(hunting): my global_id is 184882 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec3c004eb0 con 0x7fec5c0a5d50 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 client.?.objecter ms_handle_connect 0x7fec5c06e7b0 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec5affd640 1 -- 172.21.15.26:0/3212279365 >> v1:172.21.15.104:6791/0 conn(0x7fec5c06e7b0 legacy=0x7fec5c1379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3918244563 0 0) 0x7fec3c004eb0 con 0x7fec5c0a5d50 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 --> v1:172.21.15.26:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec5c139060 con 0x7fec5c0a5d50 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fec4c001cd0 con 0x7fec5c0a5d50 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient(hunting): dump: 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:46.096 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:46.097 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 <== mon.5 v1:172.21.15.104:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 86933011 0 0) 0x7fec5c13ab20 con 0x7fec5c06e7b0 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient(hunting): _negotiate 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient(hunting): my global_id is 184892 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 --> v1:172.21.15.104:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec3c004eb0 con 0x7fec5c06e7b0 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 <== mon.4 v1:172.21.15.26:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 512289615 0 0) 0x7fec5c139060 con 0x7fec5c0a5d50 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 >> v1:172.21.15.104:6791/0 conn(0x7fec5c06e7b0 legacy=0x7fec5c1379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 >> v1:172.21.15.26:6789/0 conn(0x7fec5c0a5920 legacy=0x7fec5c1380f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient: _finish_hunting 0 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 monclient: found mon.c 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:57:46.098 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec5c13ae00 con 0x7fec5c0a5d50 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient: _finish_auth 0 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient: _check_auth_tickets 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec6167e640 5 monclient: authenticate success, global_id 184882 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec6167e640 10 monclient: _renew_subs 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec6167e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec6167e640 1 -- 172.21.15.26:0/3212279365 --> v1:172.21.15.26:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fec5c139bc0 con 0x7fec5c0a5d50 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec6167e640 10 client.184882.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec6167e640 10 monclient: _renew_subs 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec6167e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec6167e640 1 -- 172.21.15.26:0/3212279365 --> v1:172.21.15.26:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fec5c138800 con 0x7fec5c0a5d50 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec6167e640 1 librados: init done 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 <== mon.4 v1:172.21.15.26:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec4c005130 con 0x7fec5c0a5d50 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:57:46.099 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 <== mon.4 v1:172.21.15.26:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7fec4c005620 con 0x7fec5c0a5d50 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec597fa640 4 set_mon_vals no callback set 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.084+0000 7fec537fe640 10 monclient: dump: 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:57:46.100 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.085+0000 7fec6167e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.085+0000 7fec6167e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.085+0000 7fec6167e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.26:6791/0 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.085+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 <== mon.4 v1:172.21.15.26:6791/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7fec4c011a80 con 0x7fec5c0a5d50 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.085+0000 7fec6167e640 1 -- 172.21.15.26:0/3212279365 --> v1:172.21.15.26:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec40005080 con 0x7fec5c0a5d50 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.086+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 <== mon.4 v1:172.21.15.26:6791/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7fec4c04deb0 con 0x7fec5c0a5d50 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.086+0000 7fec537fe640 10 client.184882.objecter ms_dispatch 0x7fec5c000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:57:46.101 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.086+0000 7fec537fe640 3 client.184882.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:57:46.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.086+0000 7fec537fe640 3 client.184882.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:57:46.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.086+0000 7fec537fe640 20 client.184882.objecter dump_active .. 0 homeless 2024-08-19T12:57:46.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.086+0000 7fec5affd640 10 monclient: get_auth_request con 0x7fec3c040f20 auth_method 0 2024-08-19T12:57:46.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.088+0000 7fec537fe640 10 client.184882.objecter ms_handle_connect 0x7fec3c040f20 2024-08-19T12:57:46.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.090+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 <== mon.4 v1:172.21.15.26:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7fec5c09ddf0 con 0x7fec5c0a5d50 2024-08-19T12:57:46.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.090+0000 7fec537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:57:46.102 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.090+0000 7fec537fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:57:46.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.274+0000 7fec6167e640 1 -- 172.21.15.26:0/3212279365 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fec40000b90 con 0x7fec3c040f20 2024-08-19T12:57:46.277 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.276+0000 7fec537fe640 1 -- 172.21.15.26:0/3212279365 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114021 (unknown 2965378022 0 3967502364) 0x7fec40000b90 con 0x7fec3c040f20 2024-08-19T12:57:46.278 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:57:46.279 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-08-19T12:57:46.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 10 librados: watch_flush enter 2024-08-19T12:57:46.280 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 10 librados: watch_flush exit 2024-08-19T12:57:46.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 20 client.184882.objecter shutdown clearing up homeless session... 2024-08-19T12:57:46.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 10 client.184882.objecter successfully canceled tick 2024-08-19T12:57:46.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 1 -- 172.21.15.26:0/3212279365 >> v1:172.21.15.104:6800/899887506 conn(0x7fec3c040f20 legacy=0x7fec3c0433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:46.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 10 monclient: shutdown 2024-08-19T12:57:46.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:57:46.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 1 -- 172.21.15.26:0/3212279365 >> v1:172.21.15.26:6791/0 conn(0x7fec5c0a5d50 legacy=0x7fec5c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:57:46.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 1 -- 172.21.15.26:0/3212279365 shutdown_connections 2024-08-19T12:57:46.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 1 -- 172.21.15.26:0/3212279365 >> 172.21.15.26:0/3212279365 conn(0x7fec5c0a8740 msgr2=0x7fec5c06dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:57:46.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 1 -- 172.21.15.26:0/3212279365 shutdown_connections 2024-08-19T12:57:46.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 1 -- 172.21.15.26:0/3212279365 wait complete. 2024-08-19T12:57:46.281 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:57:46.278+0000 7fec6167e640 1 librados: shutdown 2024-08-19T12:57:46.289 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1270,"stamp":"2024-08-19T12:57:45.388165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":441468,"kb_used_data":33600,"kb_used_omap":9,"kb_used_meta":407734,"kb_avail":565789572,"statfs":{"total":579820584960,"available":579368521728,"internally_reserved":0,"allocated":34406400,"data_stored":15183806,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9566,"internal_metadata":417520290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.809987"},"pg_stats":[{"pgid":"7.17","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.418390+0000","last_change":"2024-08-19T12:39:38.156896+0000","last_active":"2024-08-19T12:57:37.418390+0000","last_peered":"2024-08-19T12:57:37.418390+0000","last_clean":"2024-08-19T12:57:37.418390+0000","last_became_active":"2024-08-19T12:39:38.143851+0000","last_became_peered":"2024-08-19T12:39:38.143851+0000","last_unstale":"2024-08-19T12:57:37.418390+0000","last_undegraded":"2024-08-19T12:57:37.418390+0000","last_fullsized":"2024-08-19T12:57:37.418390+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:32:37.888575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1369,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.419726+0000","last_change":"2024-08-19T12:39:38.273079+0000","last_active":"2024-08-19T12:57:37.419726+0000","last_peered":"2024-08-19T12:57:37.419726+0000","last_clean":"2024-08-19T12:57:37.419726+0000","last_became_active":"2024-08-19T12:39:38.269168+0000","last_became_peered":"2024-08-19T12:39:38.269168+0000","last_unstale":"2024-08-19T12:57:37.419726+0000","last_undegraded":"2024-08-19T12:57:37.419726+0000","last_fullsized":"2024-08-19T12:57:37.419726+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:27:10.963118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1372,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438861+0000","last_change":"2024-08-19T12:39:38.205974+0000","last_active":"2024-08-19T12:57:37.438861+0000","last_peered":"2024-08-19T12:57:37.438861+0000","last_clean":"2024-08-19T12:57:37.438861+0000","last_became_active":"2024-08-19T12:39:38.201163+0000","last_became_peered":"2024-08-19T12:39:38.201163+0000","last_unstale":"2024-08-19T12:57:37.438861+0000","last_undegraded":"2024-08-19T12:57:37.438861+0000","last_fullsized":"2024-08-19T12:57:37.438861+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:33:44.282679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1359,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.417713+0000","last_change":"2024-08-19T12:39:38.230576+0000","last_active":"2024-08-19T12:57:37.417713+0000","last_peered":"2024-08-19T12:57:37.417713+0000","last_clean":"2024-08-19T12:57:37.417713+0000","last_became_active":"2024-08-19T12:39:38.183511+0000","last_became_peered":"2024-08-19T12:39:38.183511+0000","last_unstale":"2024-08-19T12:57:37.417713+0000","last_undegraded":"2024-08-19T12:57:37.417713+0000","last_fullsized":"2024-08-19T12:57:37.417713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:13:04.215179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1360,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.417640+0000","last_change":"2024-08-19T12:39:38.095466+0000","last_active":"2024-08-19T12:57:37.417640+0000","last_peered":"2024-08-19T12:57:37.417640+0000","last_clean":"2024-08-19T12:57:37.417640+0000","last_became_active":"2024-08-19T12:39:38.094806+0000","last_became_peered":"2024-08-19T12:39:38.094806+0000","last_unstale":"2024-08-19T12:57:37.417640+0000","last_undegraded":"2024-08-19T12:57:37.417640+0000","last_fullsized":"2024-08-19T12:57:37.417640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:03:19.792598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1371,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.419250+0000","last_change":"2024-08-19T12:39:38.222591+0000","last_active":"2024-08-19T12:57:37.419250+0000","last_peered":"2024-08-19T12:57:37.419250+0000","last_clean":"2024-08-19T12:57:37.419250+0000","last_became_active":"2024-08-19T12:39:38.219371+0000","last_became_peered":"2024-08-19T12:39:38.219371+0000","last_unstale":"2024-08-19T12:57:37.419250+0000","last_undegraded":"2024-08-19T12:57:37.419250+0000","last_fullsized":"2024-08-19T12:57:37.419250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:17:01.926550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:43.835092+0000","last_change":"2024-08-19T12:57:43.835137+0000","last_active":"2024-08-19T12:57:43.835092+0000","last_peered":"2024-08-19T12:57:43.835092+0000","last_clean":"2024-08-19T12:57:43.835092+0000","last_became_active":"2024-08-19T12:39:38.228951+0000","last_became_peered":"2024-08-19T12:39:38.228951+0000","last_unstale":"2024-08-19T12:57:43.835092+0000","last_undegraded":"2024-08-19T12:57:43.835092+0000","last_fullsized":"2024-08-19T12:57:43.835092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:43.835002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:43.835002+0000","last_clean_scrub_stamp":"2024-08-19T12:57:43.835002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:09:15.625703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1371,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.419265+0000","last_change":"2024-08-19T12:39:38.302683+0000","last_active":"2024-08-19T12:57:37.419265+0000","last_peered":"2024-08-19T12:57:37.419265+0000","last_clean":"2024-08-19T12:57:37.419265+0000","last_became_active":"2024-08-19T12:39:38.301100+0000","last_became_peered":"2024-08-19T12:39:38.301100+0000","last_unstale":"2024-08-19T12:57:37.419265+0000","last_undegraded":"2024-08-19T12:57:37.419265+0000","last_fullsized":"2024-08-19T12:57:37.419265+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:02:38.662623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1357,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:39.849159+0000","last_change":"2024-08-19T12:39:38.152043+0000","last_active":"2024-08-19T12:57:39.849159+0000","last_peered":"2024-08-19T12:57:39.849159+0000","last_clean":"2024-08-19T12:57:39.849159+0000","last_became_active":"2024-08-19T12:39:38.149472+0000","last_became_peered":"2024-08-19T12:39:38.149472+0000","last_unstale":"2024-08-19T12:57:39.849159+0000","last_undegraded":"2024-08-19T12:57:39.849159+0000","last_fullsized":"2024-08-19T12:57:39.849159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1371,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.419351+0000","last_change":"2024-08-19T12:39:38.184326+0000","last_active":"2024-08-19T12:57:37.419351+0000","last_peered":"2024-08-19T12:57:37.419351+0000","last_clean":"2024-08-19T12:57:37.419351+0000","last_became_active":"2024-08-19T12:39:38.181280+0000","last_became_peered":"2024-08-19T12:39:38.181280+0000","last_unstale":"2024-08-19T12:57:37.419351+0000","last_undegraded":"2024-08-19T12:57:37.419351+0000","last_fullsized":"2024-08-19T12:57:37.419351+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:00:41.274939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1370,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.419824+0000","last_change":"2024-08-19T12:39:38.154946+0000","last_active":"2024-08-19T12:57:37.419824+0000","last_peered":"2024-08-19T12:57:37.419824+0000","last_clean":"2024-08-19T12:57:37.419824+0000","last_became_active":"2024-08-19T12:39:38.147647+0000","last_became_peered":"2024-08-19T12:39:38.147647+0000","last_unstale":"2024-08-19T12:57:37.419824+0000","last_undegraded":"2024-08-19T12:57:37.419824+0000","last_fullsized":"2024-08-19T12:57:37.419824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:24:26.064970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1373,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438882+0000","last_change":"2024-08-19T12:39:38.137671+0000","last_active":"2024-08-19T12:57:37.438882+0000","last_peered":"2024-08-19T12:57:37.438882+0000","last_clean":"2024-08-19T12:57:37.438882+0000","last_became_active":"2024-08-19T12:39:38.135334+0000","last_became_peered":"2024-08-19T12:39:38.135334+0000","last_unstale":"2024-08-19T12:57:37.438882+0000","last_undegraded":"2024-08-19T12:57:37.438882+0000","last_fullsized":"2024-08-19T12:57:37.438882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:37:29.348405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1480,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438761+0000","last_change":"2024-08-19T12:38:22.728884+0000","last_active":"2024-08-19T12:57:37.438761+0000","last_peered":"2024-08-19T12:57:37.438761+0000","last_clean":"2024-08-19T12:57:37.438761+0000","last_became_active":"2024-08-19T12:38:22.728595+0000","last_became_peered":"2024-08-19T12:38:22.728595+0000","last_unstale":"2024-08-19T12:57:37.438761+0000","last_undegraded":"2024-08-19T12:57:37.438761+0000","last_fullsized":"2024-08-19T12:57:37.438761+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:21.711006+0000","last_clean_scrub_stamp":"2024-08-19T12:38:21.711006+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:42:50.800603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1383,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.418232+0000","last_change":"2024-08-19T12:38:31.959938+0000","last_active":"2024-08-19T12:57:37.418232+0000","last_peered":"2024-08-19T12:57:37.418232+0000","last_clean":"2024-08-19T12:57:37.418232+0000","last_became_active":"2024-08-19T12:38:29.952471+0000","last_became_peered":"2024-08-19T12:38:29.952471+0000","last_unstale":"2024-08-19T12:57:37.418232+0000","last_undegraded":"2024-08-19T12:57:37.418232+0000","last_fullsized":"2024-08-19T12:57:37.418232+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:48:18.927151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057881800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:42.856838+0000","last_change":"2024-08-19T12:57:42.856860+0000","last_active":"2024-08-19T12:57:42.856838+0000","last_peered":"2024-08-19T12:57:42.856838+0000","last_clean":"2024-08-19T12:57:42.856838+0000","last_became_active":"2024-08-19T12:39:38.260911+0000","last_became_peered":"2024-08-19T12:39:38.260911+0000","last_unstale":"2024-08-19T12:57:42.856838+0000","last_undegraded":"2024-08-19T12:57:42.856838+0000","last_fullsized":"2024-08-19T12:57:42.856838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:42.856756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:42.856756+0000","last_clean_scrub_stamp":"2024-08-19T12:57:42.856756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:24:09.740358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1389,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438770+0000","last_change":"2024-08-19T12:38:31.960408+0000","last_active":"2024-08-19T12:57:37.438770+0000","last_peered":"2024-08-19T12:57:37.438770+0000","last_clean":"2024-08-19T12:57:37.438770+0000","last_became_active":"2024-08-19T12:38:29.951331+0000","last_became_peered":"2024-08-19T12:38:29.951331+0000","last_unstale":"2024-08-19T12:57:37.438770+0000","last_undegraded":"2024-08-19T12:57:37.438770+0000","last_fullsized":"2024-08-19T12:57:37.438770+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:55:28.246342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018789830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1370,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.419903+0000","last_change":"2024-08-19T12:39:38.296122+0000","last_active":"2024-08-19T12:57:37.419903+0000","last_peered":"2024-08-19T12:57:37.419903+0000","last_clean":"2024-08-19T12:57:37.419903+0000","last_became_active":"2024-08-19T12:39:38.295614+0000","last_became_peered":"2024-08-19T12:39:38.295614+0000","last_unstale":"2024-08-19T12:57:37.419903+0000","last_undegraded":"2024-08-19T12:57:37.419903+0000","last_fullsized":"2024-08-19T12:57:37.419903+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:57:09.397165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1387,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.419246+0000","last_change":"2024-08-19T12:38:31.957412+0000","last_active":"2024-08-19T12:57:37.419246+0000","last_peered":"2024-08-19T12:57:37.419246+0000","last_clean":"2024-08-19T12:57:37.419246+0000","last_became_active":"2024-08-19T12:38:29.952459+0000","last_became_peered":"2024-08-19T12:38:29.952459+0000","last_unstale":"2024-08-19T12:57:37.419246+0000","last_undegraded":"2024-08-19T12:57:37.419246+0000","last_fullsized":"2024-08-19T12:57:37.419246+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:47:27.910005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039863799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:44.787946+0000","last_change":"2024-08-19T12:57:44.787971+0000","last_active":"2024-08-19T12:57:44.787946+0000","last_peered":"2024-08-19T12:57:44.787946+0000","last_clean":"2024-08-19T12:57:44.787946+0000","last_became_active":"2024-08-19T12:39:38.218230+0000","last_became_peered":"2024-08-19T12:39:38.218230+0000","last_unstale":"2024-08-19T12:57:44.787946+0000","last_undegraded":"2024-08-19T12:57:44.787946+0000","last_fullsized":"2024-08-19T12:57:44.787946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:44.787860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:44.787860+0000","last_clean_scrub_stamp":"2024-08-19T12:57:44.787860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:42:10.169968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1384,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.418296+0000","last_change":"2024-08-19T12:38:31.959929+0000","last_active":"2024-08-19T12:57:37.418296+0000","last_peered":"2024-08-19T12:57:37.418296+0000","last_clean":"2024-08-19T12:57:37.418296+0000","last_became_active":"2024-08-19T12:38:29.951392+0000","last_became_peered":"2024-08-19T12:38:29.951392+0000","last_unstale":"2024-08-19T12:57:37.418296+0000","last_undegraded":"2024-08-19T12:57:37.418296+0000","last_fullsized":"2024-08-19T12:57:37.418296+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:47:44.756214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056602099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1373,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438931+0000","last_change":"2024-08-19T12:39:38.220796+0000","last_active":"2024-08-19T12:57:37.438931+0000","last_peered":"2024-08-19T12:57:37.438931+0000","last_clean":"2024-08-19T12:57:37.438931+0000","last_became_active":"2024-08-19T12:39:38.217569+0000","last_became_peered":"2024-08-19T12:39:38.217569+0000","last_unstale":"2024-08-19T12:57:37.438931+0000","last_undegraded":"2024-08-19T12:57:37.438931+0000","last_fullsized":"2024-08-19T12:57:37.438931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:14:24.659508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1376,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.417689+0000","last_change":"2024-08-19T12:38:31.963352+0000","last_active":"2024-08-19T12:57:37.417689+0000","last_peered":"2024-08-19T12:57:37.417689+0000","last_clean":"2024-08-19T12:57:37.417689+0000","last_became_active":"2024-08-19T12:38:29.951233+0000","last_became_peered":"2024-08-19T12:38:29.951233+0000","last_unstale":"2024-08-19T12:57:37.417689+0000","last_undegraded":"2024-08-19T12:57:37.417689+0000","last_fullsized":"2024-08-19T12:57:37.417689+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:45:51.806389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048735899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"22'7","reported_seq":1383,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.419666+0000","last_change":"2024-08-19T12:39:38.270101+0000","last_active":"2024-08-19T12:57:37.419666+0000","last_peered":"2024-08-19T12:57:37.419666+0000","last_clean":"2024-08-19T12:57:37.419666+0000","last_became_active":"2024-08-19T12:39:38.269442+0000","last_became_peered":"2024-08-19T12:39:38.269442+0000","last_unstale":"2024-08-19T12:57:37.419666+0000","last_undegraded":"2024-08-19T12:57:37.419666+0000","last_fullsized":"2024-08-19T12:57:37.419666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:02:35.119493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1389,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438986+0000","last_change":"2024-08-19T12:38:31.960297+0000","last_active":"2024-08-19T12:57:37.438986+0000","last_peered":"2024-08-19T12:57:37.438986+0000","last_clean":"2024-08-19T12:57:37.438986+0000","last_became_active":"2024-08-19T12:38:29.949846+0000","last_became_peered":"2024-08-19T12:38:29.949846+0000","last_unstale":"2024-08-19T12:57:37.438986+0000","last_undegraded":"2024-08-19T12:57:37.438986+0000","last_fullsized":"2024-08-19T12:57:37.438986+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:34:31.447838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001822192,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1367,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.418257+0000","last_change":"2024-08-19T12:39:38.181922+0000","last_active":"2024-08-19T12:57:37.418257+0000","last_peered":"2024-08-19T12:57:37.418257+0000","last_clean":"2024-08-19T12:57:37.418257+0000","last_became_active":"2024-08-19T12:39:38.166658+0000","last_became_peered":"2024-08-19T12:39:38.166658+0000","last_unstale":"2024-08-19T12:57:37.418257+0000","last_undegraded":"2024-08-19T12:57:37.418257+0000","last_fullsized":"2024-08-19T12:57:37.418257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:21:22.453156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1375,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.417942+0000","last_change":"2024-08-19T12:38:31.963379+0000","last_active":"2024-08-19T12:57:37.417942+0000","last_peered":"2024-08-19T12:57:37.417942+0000","last_clean":"2024-08-19T12:57:37.417942+0000","last_became_active":"2024-08-19T12:38:29.950467+0000","last_became_peered":"2024-08-19T12:38:29.950467+0000","last_unstale":"2024-08-19T12:57:37.417942+0000","last_undegraded":"2024-08-19T12:57:37.417942+0000","last_fullsized":"2024-08-19T12:57:37.417942+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:40:07.574639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049350399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:41.906039+0000","last_change":"2024-08-19T12:57:41.906063+0000","last_active":"2024-08-19T12:57:41.906039+0000","last_peered":"2024-08-19T12:57:41.906039+0000","last_clean":"2024-08-19T12:57:41.906039+0000","last_became_active":"2024-08-19T12:39:38.190786+0000","last_became_peered":"2024-08-19T12:39:38.190786+0000","last_unstale":"2024-08-19T12:57:41.906039+0000","last_undegraded":"2024-08-19T12:57:41.906039+0000","last_fullsized":"2024-08-19T12:57:41.906039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:41.905967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:41.905967+0000","last_clean_scrub_stamp":"2024-08-19T12:57:41.905967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:43:22.005981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1388,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438945+0000","last_change":"2024-08-19T12:38:31.959032+0000","last_active":"2024-08-19T12:57:37.438945+0000","last_peered":"2024-08-19T12:57:37.438945+0000","last_clean":"2024-08-19T12:57:37.438945+0000","last_became_active":"2024-08-19T12:38:29.948183+0000","last_became_peered":"2024-08-19T12:38:29.948183+0000","last_unstale":"2024-08-19T12:57:37.438945+0000","last_undegraded":"2024-08-19T12:57:37.438945+0000","last_fullsized":"2024-08-19T12:57:37.438945+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:38:28.013302+0000","last_clean_scrub_stamp":"2024-08-19T12:38:28.013302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T12:51:55.545957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052740100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1371,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.419270+0000","last_change":"2024-08-19T12:39:38.221167+0000","last_active":"2024-08-19T12:57:37.419270+0000","last_peered":"2024-08-19T12:57:37.419270+0000","last_clean":"2024-08-19T12:57:37.419270+0000","last_became_active":"2024-08-19T12:39:38.191158+0000","last_became_peered":"2024-08-19T12:39:38.191158+0000","last_unstale":"2024-08-19T12:57:37.419270+0000","last_undegraded":"2024-08-19T12:57:37.419270+0000","last_fullsized":"2024-08-19T12:57:37.419270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T22:59:47.700950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1359,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.417494+0000","last_change":"2024-08-19T12:39:38.194777+0000","last_active":"2024-08-19T12:57:37.417494+0000","last_peered":"2024-08-19T12:57:37.417494+0000","last_clean":"2024-08-19T12:57:37.417494+0000","last_became_active":"2024-08-19T12:39:38.161352+0000","last_became_peered":"2024-08-19T12:39:38.161352+0000","last_unstale":"2024-08-19T12:57:37.417494+0000","last_undegraded":"2024-08-19T12:57:37.417494+0000","last_fullsized":"2024-08-19T12:57:37.417494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:31:21.012106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1372,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438759+0000","last_change":"2024-08-19T12:39:38.210090+0000","last_active":"2024-08-19T12:57:37.438759+0000","last_peered":"2024-08-19T12:57:37.438759+0000","last_clean":"2024-08-19T12:57:37.438759+0000","last_became_active":"2024-08-19T12:39:38.202668+0000","last_became_peered":"2024-08-19T12:39:38.202668+0000","last_unstale":"2024-08-19T12:57:37.438759+0000","last_undegraded":"2024-08-19T12:57:37.438759+0000","last_fullsized":"2024-08-19T12:57:37.438759+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:58:28.292312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.418365+0000","last_change":"2024-08-19T12:39:38.224507+0000","last_active":"2024-08-19T12:57:37.418365+0000","last_peered":"2024-08-19T12:57:37.418365+0000","last_clean":"2024-08-19T12:57:37.418365+0000","last_became_active":"2024-08-19T12:39:38.222877+0000","last_became_peered":"2024-08-19T12:39:38.222877+0000","last_unstale":"2024-08-19T12:57:37.418365+0000","last_undegraded":"2024-08-19T12:57:37.418365+0000","last_fullsized":"2024-08-19T12:57:37.418365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:55:08.088675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1356,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438874+0000","last_change":"2024-08-19T12:39:38.211834+0000","last_active":"2024-08-19T12:57:37.438874+0000","last_peered":"2024-08-19T12:57:37.438874+0000","last_clean":"2024-08-19T12:57:37.438874+0000","last_became_active":"2024-08-19T12:39:38.210985+0000","last_became_peered":"2024-08-19T12:39:38.210985+0000","last_unstale":"2024-08-19T12:57:37.438874+0000","last_undegraded":"2024-08-19T12:57:37.438874+0000","last_fullsized":"2024-08-19T12:57:37.438874+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1372,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438823+0000","last_change":"2024-08-19T12:39:38.268396+0000","last_active":"2024-08-19T12:57:37.438823+0000","last_peered":"2024-08-19T12:57:37.438823+0000","last_clean":"2024-08-19T12:57:37.438823+0000","last_became_active":"2024-08-19T12:39:38.267362+0000","last_became_peered":"2024-08-19T12:39:38.267362+0000","last_unstale":"2024-08-19T12:57:37.438823+0000","last_undegraded":"2024-08-19T12:57:37.438823+0000","last_fullsized":"2024-08-19T12:57:37.438823+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T13:26:11.466849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1359,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.417859+0000","last_change":"2024-08-19T12:39:38.195841+0000","last_active":"2024-08-19T12:57:37.417859+0000","last_peered":"2024-08-19T12:57:37.417859+0000","last_clean":"2024-08-19T12:57:37.417859+0000","last_became_active":"2024-08-19T12:39:38.173967+0000","last_became_peered":"2024-08-19T12:39:38.173967+0000","last_unstale":"2024-08-19T12:57:37.417859+0000","last_undegraded":"2024-08-19T12:57:37.417859+0000","last_fullsized":"2024-08-19T12:57:37.417859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:29:39.717188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1361,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.417782+0000","last_change":"2024-08-19T12:39:38.110553+0000","last_active":"2024-08-19T12:57:37.417782+0000","last_peered":"2024-08-19T12:57:37.417782+0000","last_clean":"2024-08-19T12:57:37.417782+0000","last_became_active":"2024-08-19T12:39:38.108670+0000","last_became_peered":"2024-08-19T12:39:38.108670+0000","last_unstale":"2024-08-19T12:57:37.417782+0000","last_undegraded":"2024-08-19T12:57:37.417782+0000","last_fullsized":"2024-08-19T12:57:37.417782+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:32:52.104320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1366,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.418463+0000","last_change":"2024-08-19T12:39:38.200400+0000","last_active":"2024-08-19T12:57:37.418463+0000","last_peered":"2024-08-19T12:57:37.418463+0000","last_clean":"2024-08-19T12:57:37.418463+0000","last_became_active":"2024-08-19T12:39:38.197445+0000","last_became_peered":"2024-08-19T12:39:38.197445+0000","last_unstale":"2024-08-19T12:57:37.418463+0000","last_undegraded":"2024-08-19T12:57:37.418463+0000","last_fullsized":"2024-08-19T12:57:37.418463+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:21:39.167660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1356,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438945+0000","last_change":"2024-08-19T12:39:38.274353+0000","last_active":"2024-08-19T12:57:37.438945+0000","last_peered":"2024-08-19T12:57:37.438945+0000","last_clean":"2024-08-19T12:57:37.438945+0000","last_became_active":"2024-08-19T12:39:38.267711+0000","last_became_peered":"2024-08-19T12:39:38.267711+0000","last_unstale":"2024-08-19T12:57:37.438945+0000","last_undegraded":"2024-08-19T12:57:37.438945+0000","last_fullsized":"2024-08-19T12:57:37.438945+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1372,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438784+0000","last_change":"2024-08-19T12:39:38.188765+0000","last_active":"2024-08-19T12:57:37.438784+0000","last_peered":"2024-08-19T12:57:37.438784+0000","last_clean":"2024-08-19T12:57:37.438784+0000","last_became_active":"2024-08-19T12:39:38.177735+0000","last_became_peered":"2024-08-19T12:39:38.177735+0000","last_unstale":"2024-08-19T12:57:37.438784+0000","last_undegraded":"2024-08-19T12:57:37.438784+0000","last_fullsized":"2024-08-19T12:57:37.438784+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:38:45.637343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1372,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438759+0000","last_change":"2024-08-19T12:39:38.200875+0000","last_active":"2024-08-19T12:57:37.438759+0000","last_peered":"2024-08-19T12:57:37.438759+0000","last_clean":"2024-08-19T12:57:37.438759+0000","last_became_active":"2024-08-19T12:39:38.199715+0000","last_became_peered":"2024-08-19T12:39:38.199715+0000","last_unstale":"2024-08-19T12:57:37.438759+0000","last_undegraded":"2024-08-19T12:57:37.438759+0000","last_fullsized":"2024-08-19T12:57:37.438759+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T22:27:29.457630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1356,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:37.438880+0000","last_change":"2024-08-19T12:39:38.196277+0000","last_active":"2024-08-19T12:57:37.438880+0000","last_peered":"2024-08-19T12:57:37.438880+0000","last_clean":"2024-08-19T12:57:37.438880+0000","last_became_active":"2024-08-19T12:39:38.190871+0000","last_became_peered":"2024-08-19T12:39:38.190871+0000","last_unstale":"2024-08-19T12:57:37.438880+0000","last_undegraded":"2024-08-19T12:57:37.438880+0000","last_fullsized":"2024-08-19T12:57:37.438880+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:39:36.248575+0000","last_clean_scrub_stamp":"2024-08-19T12:39:36.248575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607787,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68712,"kb_used_data":5584,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303128,"statfs":{"total":96636764160,"available":96566403072,"internally_reserved":0,"allocated":5718016,"data_stored":2354336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607787,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68708,"kb_used_data":5584,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303132,"statfs":{"total":96636764160,"available":96566407168,"internally_reserved":0,"allocated":5718016,"data_stored":2354322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607790,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68756,"kb_used_data":5632,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303084,"statfs":{"total":96636764160,"available":96566358016,"internally_reserved":0,"allocated":5767168,"data_stored":2883245,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1592,"internal_metadata":64616904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607786,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77024,"kb_used_data":5580,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294816,"statfs":{"total":96636764160,"available":96557891584,"internally_reserved":0,"allocated":5713920,"data_stored":2354317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607787,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77032,"kb_used_data":5584,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294808,"statfs":{"total":96636764160,"available":96557883392,"internally_reserved":0,"allocated":5718016,"data_stored":2354336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640493,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81236,"kb_used_data":5636,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290604,"statfs":{"total":96636764160,"available":96553578496,"internally_reserved":0,"allocated":5771264,"data_stored":2883250,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T12:57:46.295 INFO:tasks.ceph:pgid 7.17 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.295 INFO:tasks.ceph:pgid 7.16 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.295 INFO:tasks.ceph:pgid 7.15 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.295 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.295 INFO:tasks.ceph:pgid 7.b last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.295 INFO:tasks.ceph:pgid 7.a last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.295 INFO:tasks.ceph:pgid 7.8 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.295 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.296 INFO:tasks.ceph:pgid 7.e last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.296 INFO:tasks.ceph:pgid 7.d last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.296 INFO:tasks.ceph:pgid 7.c last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.296 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-08-19T12:38:21.711006+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=38, tm_sec=21, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.296 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-08-19T12:38:28.013302+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=38, tm_sec=28, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.296 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-08-19T12:38:28.013302+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=38, tm_sec=28, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.296 INFO:tasks.ceph:pgid 7.5 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.296 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-08-19T12:38:28.013302+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=38, tm_sec=28, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.296 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-08-19T12:38:28.013302+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=38, tm_sec=28, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.297 INFO:tasks.ceph:pgid 7.7 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.297 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-08-19T12:38:28.013302+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=38, tm_sec=28, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.297 INFO:tasks.ceph:pgid 7.1 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.297 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-08-19T12:38:28.013302+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=38, tm_sec=28, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.297 INFO:tasks.ceph:pgid 7.0 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.297 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-08-19T12:38:28.013302+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=38, tm_sec=28, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.297 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-08-19T12:38:28.013302+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=38, tm_sec=28, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.297 INFO:tasks.ceph:pgid 7.2 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.297 INFO:tasks.ceph:pgid 7.10 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.298 INFO:tasks.ceph:pgid 7.11 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.298 INFO:tasks.ceph:pgid 7.12 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.298 INFO:tasks.ceph:pgid 7.13 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.298 INFO:tasks.ceph:pgid 7.1c last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.298 INFO:tasks.ceph:pgid 7.1d last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.298 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.298 INFO:tasks.ceph:pgid 7.1f last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.298 INFO:tasks.ceph:pgid 7.18 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.298 INFO:tasks.ceph:pgid 7.19 last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.299 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.299 INFO:tasks.ceph:pgid 7.1b last_scrub_stamp 2024-08-19T12:39:36.248575+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=39, tm_sec=36, tm_wday=0, tm_yday=232, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=19, tm_hour=12, tm_min=57, tm_sec=40, tm_wday=0, tm_yday=232, tm_isdst=0) 2024-08-19T12:57:46.299 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-19T12:58:05.434 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-08-19T12:58:05.434 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-19T12:58:05.434 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-19T12:58:05.437 INFO:tasks.ceph.mon.a:Started 2024-08-19T12:58:05.437 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-08-19T12:58:05.437 INFO:tasks.ceph.mon.g:Restarting daemon 2024-08-19T12:58:05.437 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-08-19T12:58:05.440 INFO:tasks.ceph.mon.g:Started 2024-08-19T12:58:05.440 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-08-19T12:58:05.440 INFO:tasks.ceph.mon.d:Restarting daemon 2024-08-19T12:58:05.440 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-08-19T12:58:05.443 INFO:tasks.ceph.mon.d:Started 2024-08-19T12:58:05.443 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-08-19T12:58:05.443 INFO:tasks.ceph.mon.b:Sent signal 19 2024-08-19T12:58:05.443 INFO:tasks.mon_thrash:Sending STOP to mon h 2024-08-19T12:58:05.443 INFO:tasks.ceph.mon.h:Sent signal 19 2024-08-19T12:58:05.443 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-08-19T12:58:06.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-19T12:58:06.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.406+0000 7f91a10de640 10 monclient: get_monmap_and_config 2024-08-19T12:58:06.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.406+0000 7f91a10de640 10 monclient: build_initial_monmap 2024-08-19T12:58:06.407 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.406+0000 7f91a10de640 10 monclient: monmap: 2024-08-19T12:58:06.407 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:58:06.408 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:06.408 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:58:06.407764+0000 2024-08-19T12:58:06.408 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:58:06.407764+0000 2024-08-19T12:58:06.408 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:58:06.408 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:06.408 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:58:06.408 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:58:06.408 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:58:06.408 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:58:06.408 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:58:06.409 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:58:06.409 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:58:06.409 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:58:06.409 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:58:06.409 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:06.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.407+0000 7f91a10de640 1 Processor -- start 2024-08-19T12:58:06.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.407+0000 7f91a10de640 1 -- start start 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.407+0000 7f91a10de640 10 monclient: init 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.407+0000 7f91a10de640 10 monclient: _reopen_session rank -1 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.408+0000 7f91a10de640 10 monclient: _add_conns ranks=[0,1,5,2,8,6,3,4,7] 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.408+0000 7f91a10de640 10 monclient(hunting): picked mon.noname-a con 0x7f919c06e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.408+0000 7f91a10de640 10 monclient(hunting): picked mon.noname-f con 0x7f919c0a5920 addr v1:172.21.15.104:6789/0 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.408+0000 7f91a10de640 10 monclient(hunting): picked mon.noname-h con 0x7f919c0a5d50 addr v1:172.21.15.104:6791/0 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.408+0000 7f91a10de640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919c136790 con 0x7f919c06e7b0 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.408+0000 7f91a10de640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919c09ddf0 con 0x7f919c0a5920 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.408+0000 7f91a10de640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919c09e0d0 con 0x7f919c0a5d50 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.408+0000 7f91a10de640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.408+0000 7f91a10de640 10 monclient(hunting): authenticate will time out at 1880.495099s 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.408+0000 7f919ad9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f919c06e7b0 0x7f919c136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:34712/0 (socket says 172.21.15.26:34712) 2024-08-19T12:58:06.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.408+0000 7f919ad9d640 1 -- 172.21.15.26:0/571466071 learned_addr learned my addr 172.21.15.26:0/571466071 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:58:06.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.410+0000 7f919ad9d640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6789/0 conn(0x7f919c06e7b0 legacy=0x7f919c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:06.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:06.410+0000 7f919a59c640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.104:6789/0 conn(0x7f919c0a5920 legacy=0x7f919c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:09.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 10 monclient(hunting): tick 2024-08-19T12:58:09.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:58:09.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 1 monclient(hunting): continuing hunt 2024-08-19T12:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.104:6791/0 conn(0x7f919c0a5d50 legacy=0x7f919c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T12:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.104:6789/0 conn(0x7f919c0a5920 legacy=0x7f919c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6789/0 conn(0x7f919c06e7b0 legacy=0x7f919c136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 10 monclient: _add_conns ranks=[6,4,2,8,7,3] 2024-08-19T12:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 10 monclient(hunting): picked mon.noname-d con 0x7f917c0038c0 addr v1:172.21.15.26:6792/0 2024-08-19T12:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 10 monclient(hunting): picked mon.noname-c con 0x7f917c007550 addr v1:172.21.15.26:6791/0 2024-08-19T12:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 10 monclient(hunting): picked mon.noname-b con 0x7f917c00b2c0 addr v1:172.21.15.26:6790/0 2024-08-19T12:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919c09e3b0 con 0x7f917c00b2c0 2024-08-19T12:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919c09e0d0 con 0x7f917c007550 2024-08-19T12:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917c011110 con 0x7f917c0038c0 2024-08-19T12:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.408+0000 7f9198d99640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:09.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.410+0000 7f919ad9d640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6792/0 conn(0x7f917c0038c0 legacy=0x7f917c005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:09.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:09.410+0000 7f919a59c640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6791/0 conn(0x7f917c007550 legacy=0x7f917c0099f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:12.409 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 10 monclient(hunting): tick 2024-08-19T12:58:12.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:58:12.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 1 monclient(hunting): continuing hunt 2024-08-19T12:58:12.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:58:12.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6792/0 conn(0x7f917c0038c0 legacy=0x7f917c005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:12.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6791/0 conn(0x7f917c007550 legacy=0x7f917c0099f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:12.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6790/0 conn(0x7f917c00b2c0 legacy=0x7f917c00d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T12:58:12.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 10 monclient: _add_conns ranks=[8,3,7] 2024-08-19T12:58:12.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f919b59e640 1 -- 172.21.15.26:0/571466071 reap_dead start 2024-08-19T12:58:12.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 10 monclient(hunting): picked mon.noname-e con 0x7f917c0103a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:58:12.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f919b59e640 1 -- 172.21.15.26:0/571466071 reap_dead start 2024-08-19T12:58:12.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 10 monclient(hunting): picked mon.noname-g con 0x7f917c011340 addr v1:172.21.15.104:6790/0 2024-08-19T12:58:12.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 10 monclient(hunting): picked mon.noname-i con 0x7f917c0117c0 addr v1:172.21.15.104:6792/0 2024-08-19T12:58:12.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917c012470 con 0x7f917c011340 2024-08-19T12:58:12.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919c09e3b0 con 0x7f917c0117c0 2024-08-19T12:58:12.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917c011d30 con 0x7f917c0103a0 2024-08-19T12:58:12.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.409+0000 7f9198d99640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:12.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.411+0000 7f919ad9d640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6793/0 conn(0x7f917c0103a0 legacy=0x7f917c0107e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:12.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.411+0000 7f9199d9b640 1 -- 172.21.15.26:0/571466071 DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:58:12.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.411+0000 7f919b59e640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.104:6792/0 conn(0x7f917c0117c0 legacy=0x7f917c0099f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:12.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:12.411+0000 7f919a59c640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.104:6790/0 conn(0x7f917c011340 legacy=0x7f917c005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:15.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 10 monclient(hunting): tick 2024-08-19T12:58:15.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:58:15.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 1 monclient(hunting): continuing hunt 2024-08-19T12:58:15.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:58:15.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6793/0 conn(0x7f917c0103a0 legacy=0x7f917c0107e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.104:6792/0 conn(0x7f917c0117c0 legacy=0x7f917c0099f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.410 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.104:6790/0 conn(0x7f917c011340 legacy=0x7f917c005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 10 monclient: _add_conns ranks=[3,4,6,7,2,0,8,5,1] 2024-08-19T12:58:15.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 10 monclient(hunting): picked mon.noname-g con 0x7f917c00e290 addr v1:172.21.15.104:6790/0 2024-08-19T12:58:15.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 10 monclient(hunting): picked mon.noname-c con 0x7f917c00e6b0 addr v1:172.21.15.26:6791/0 2024-08-19T12:58:15.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 10 monclient(hunting): picked mon.noname-d con 0x7f917c005d10 addr v1:172.21.15.26:6792/0 2024-08-19T12:58:15.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917c011fc0 con 0x7f917c00e290 2024-08-19T12:58:15.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917c006120 con 0x7f917c00e6b0 2024-08-19T12:58:15.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 1 -- 172.21.15.26:0/571466071 --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917c00eb90 con 0x7f917c005d10 2024-08-19T12:58:15.411 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.409+0000 7f9198d99640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:15.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.411+0000 7f919b59e640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6792/0 conn(0x7f917c005d10 legacy=0x7f917c006bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:15.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.411+0000 7f919a59c640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6791/0 conn(0x7f917c00e6b0 legacy=0x7f917c0064e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:15.412 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.411+0000 7f919ad9d640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.104:6790/0 conn(0x7f917c00e290 legacy=0x7f917c00d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:15.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.540+0000 7f919b59e640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6792/0 conn(0x7f917c005d10 legacy=0x7f917c006bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-08-19T12:58:15.540 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.540+0000 7f919b59e640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6792/0 conn(0x7f917c005d10 legacy=0x7f917c006bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-19T12:58:15.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.540+0000 7f919b59e640 1 --1- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6792/0 conn(0x7f917c005d10 0x7f917c006bf0 :-1 s=OPENED pgs=3582 cs=1 l=1).handle_message read tag failed 2024-08-19T12:58:15.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.540+0000 7f919b59e640 1 --1- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6792/0 conn(0x7f917c005d10 0x7f917c006bf0 :-1 s=OPENED pgs=3582 cs=1 l=1).fault on lossy channel, failing 2024-08-19T12:58:15.541 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.540+0000 7f9199d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.26:6792/0 2024-08-19T12:58:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.632+0000 7f9199d9b640 1 -- 172.21.15.26:0/571466071 <== mon.4 v1:172.21.15.26:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3696485862 0 0) 0x7f917c006120 con 0x7f917c00e6b0 2024-08-19T12:58:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.632+0000 7f9199d9b640 10 monclient(hunting): _negotiate 2024-08-19T12:58:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.632+0000 7f9199d9b640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:58:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.632+0000 7f9199d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:58:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.632+0000 7f9199d9b640 10 monclient(hunting): my global_id is 184891 2024-08-19T12:58:15.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.632+0000 7f9199d9b640 1 -- 172.21.15.26:0/571466071 --> v1:172.21.15.26:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9188003520 con 0x7f917c00e6b0 2024-08-19T12:58:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 1 -- 172.21.15.26:0/571466071 <== mon.4 v1:172.21.15.26:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2384381213 0 0) 0x7f9188003520 con 0x7f917c00e6b0 2024-08-19T12:58:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6792/0 conn(0x7f917c005d10 legacy=0x7f917c006bf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-19T12:58:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.104:6790/0 conn(0x7f917c00e290 legacy=0x7f917c00d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 10 monclient: _finish_hunting 0 2024-08-19T12:58:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 1 monclient: found mon.noname-c 2024-08-19T12:58:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:58:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6791/0 2024-08-19T12:58:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 1 -- 172.21.15.26:0/571466071 --> v1:172.21.15.26:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c00ed60 con 0x7f917c00e6b0 2024-08-19T12:58:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 10 monclient: _finish_auth 0 2024-08-19T12:58:15.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 10 monclient: _check_auth_tickets 2024-08-19T12:58:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:58:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 1 -- 172.21.15.26:0/571466071 <== mon.4 v1:172.21.15.26:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9184001a40 con 0x7f917c00e6b0 2024-08-19T12:58:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:58:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-19T12:58:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 10 monclient: dump: 2024-08-19T12:58:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:58:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:58:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.636 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:58:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:58:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:58:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:58:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:58:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:58:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:58:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:58:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:58:15.637 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f91a10de640 5 monclient: authenticate success, global_id 184891 2024-08-19T12:58:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f91a10de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:58:15.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 1 -- 172.21.15.26:0/571466071 <== mon.4 v1:172.21.15.26:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9184001ca0 con 0x7f917c00e6b0 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 10 monclient: handle_config config(0 keys) 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 1 -- 172.21.15.26:0/571466071 <== mon.4 v1:172.21.15.26:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9184003950 con 0x7f917c00e6b0 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f91a10de640 10 monclient: get_monmap_and_config success 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f91a10de640 4 set_mon_vals no callback set 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.633+0000 7f9199d9b640 10 monclient: dump: 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:58:15.638 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:58:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:58:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:58:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:58:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:58:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:58:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:58:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:58:15.639 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:58:15.639 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.634+0000 7f91a10de640 10 monclient: shutdown 2024-08-19T12:58:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.634+0000 7f91a10de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:58:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.634+0000 7f91a10de640 1 -- 172.21.15.26:0/571466071 >> v1:172.21.15.26:6791/0 conn(0x7f917c00e6b0 legacy=0x7f917c0064e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.634+0000 7f919b59e640 1 -- 172.21.15.26:0/571466071 reap_dead start 2024-08-19T12:58:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.634+0000 7f91a10de640 1 -- 172.21.15.26:0/571466071 shutdown_connections 2024-08-19T12:58:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.634+0000 7f91a10de640 1 -- 172.21.15.26:0/571466071 >> 172.21.15.26:0/571466071 conn(0x7f919c0a8740 msgr2=0x7f919c0a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:58:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.634+0000 7f91a10de640 1 -- 172.21.15.26:0/571466071 shutdown_connections 2024-08-19T12:58:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.634+0000 7f91a10de640 1 -- 172.21.15.26:0/571466071 wait complete. 2024-08-19T12:58:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.635+0000 7f91a10de640 10 monclient: build_initial_monmap 2024-08-19T12:58:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.635+0000 7f91a10de640 10 monclient: monmap: 2024-08-19T12:58:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:58:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:58:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:58:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:58:15.643 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:58:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:58:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:58:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:58:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:58:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:58:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:58:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:58:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:58:15.644 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:58:15.644 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.635+0000 7f91a10de640 1 librados: starting msgr at 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.635+0000 7f91a10de640 1 librados: starting objecter 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.636+0000 7f91a10de640 1 Processor -- start 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.636+0000 7f91a10de640 1 -- start start 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.636+0000 7f91a10de640 1 librados: setting wanted keys 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.636+0000 7f91a10de640 1 librados: calling monclient init 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.636+0000 7f91a10de640 10 monclient: init 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.636+0000 7f91a10de640 10 monclient: _reopen_session rank -1 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.636+0000 7f91a10de640 10 monclient: _add_conns ranks=[2,6,1,0,3,8,7,4,5] 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.636+0000 7f91a10de640 10 monclient(hunting): picked mon.noname-b con 0x7f917c00e6b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.636+0000 7f91a10de640 10 monclient(hunting): picked mon.noname-d con 0x7f919c1c2520 addr v1:172.21.15.26:6792/0 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.636+0000 7f91a10de640 10 monclient(hunting): picked mon.noname-f con 0x7f919c1ce000 addr v1:172.21.15.104:6789/0 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.637+0000 7f91a10de640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919c1ce420 con 0x7f919c1ce000 2024-08-19T12:58:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.637+0000 7f91a10de640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919c0aaea0 con 0x7f917c00e6b0 2024-08-19T12:58:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.637+0000 7f91a10de640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919c135090 con 0x7f919c1c2520 2024-08-19T12:58:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.637+0000 7f91a10de640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.637+0000 7f91a10de640 10 monclient(hunting): authenticate will time out at 1889.724189s 2024-08-19T12:58:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.637+0000 7f919a59c640 1 --1- >> v1:172.21.15.26:6792/0 conn(0x7f919c1c2520 0x7f919c1d3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6792/0 says I am v1:172.21.15.26:42730/0 (socket says 172.21.15.26:42730) 2024-08-19T12:58:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.637+0000 7f919b59e640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f919c1ce000 0x7f919c1d6a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:56228/0 (socket says 172.21.15.26:56228) 2024-08-19T12:58:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.637+0000 7f919a59c640 1 -- 172.21.15.26:0/610156022 learned_addr learned my addr 172.21.15.26:0/610156022 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:58:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.639+0000 7f919b59e640 1 -- 172.21.15.26:0/610156022 >> v1:172.21.15.104:6789/0 conn(0x7f919c1ce000 legacy=0x7f919c1d6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.639+0000 7f919a59c640 1 -- 172.21.15.26:0/610156022 >> v1:172.21.15.26:6792/0 conn(0x7f919c1c2520 legacy=0x7f919c1d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.639+0000 7f91937fe640 10 client.?.objecter ms_handle_connect 0x7f919c1ce000 2024-08-19T12:58:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.639+0000 7f91937fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:58:15.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.639+0000 7f91937fe640 10 client.?.objecter ms_handle_connect 0x7f919c1c2520 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.639+0000 7f91937fe640 10 client.?.objecter resend_mon_ops 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.639+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 705008608 0 0) 0x7f919c135090 con 0x7f919c1c2520 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.639+0000 7f91937fe640 10 monclient(hunting): _negotiate 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.639+0000 7f91937fe640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.639+0000 7f91937fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.639+0000 7f91937fe640 10 monclient(hunting): my global_id is 214107 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.639+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f917c00e290 con 0x7f919c1c2520 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.640+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2679167826 0 0) 0x7f917c00e290 con 0x7f919c1c2520 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.640+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f919c135090 con 0x7f919c1c2520 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.640+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9184003950 con 0x7f919c1c2520 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.640+0000 7f91937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.640+0000 7f91937fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.640+0000 7f91937fe640 10 monclient(hunting): dump: 2024-08-19T12:58:15.647 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:58:15.648 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.640+0000 7f91937fe640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:58:15.650 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1295135422 0 0) 0x7f919c135090 con 0x7f919c1c2520 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 >> v1:172.21.15.26:6790/0 conn(0x7f917c00e6b0 legacy=0x7f919c1c1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 >> v1:172.21.15.104:6789/0 conn(0x7f919c1ce000 legacy=0x7f919c1d6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 10 monclient: _finish_hunting 0 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 1 monclient: found mon.d 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919c1ce6b0 con 0x7f919c1c2520 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 10 monclient: _finish_auth 0 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 10 monclient: _check_auth_tickets 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91a10de640 5 monclient: authenticate success, global_id 214107 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91a10de640 10 monclient: _renew_subs 2024-08-19T12:58:15.651 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91a10de640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91a10de640 1 -- 172.21.15.26:0/610156022 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f919c1be580 con 0x7f919c1c2520 2024-08-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91a10de640 10 client.214107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91a10de640 10 monclient: _renew_subs 2024-08-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91a10de640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91a10de640 1 -- 172.21.15.26:0/610156022 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f919c1c0040 con 0x7f919c1c2520 2024-08-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91a10de640 1 librados: init done 2024-08-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9184003ee0 con 0x7f919c1c2520 2024-08-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 10 monclient: handle_config config(0 keys) 2024-08-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f9184004450 con 0x7f919c1c2520 2024-08-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:58:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f91937fe640 10 monclient: dump: 2024-08-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:58:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:58:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:58:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:58:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:58:15.654 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.644+0000 7f919959a640 4 set_mon_vals no callback set 2024-08-19T12:58:15.654 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.645+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f9184011a80 con 0x7f919c1c2520 2024-08-19T12:58:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.645+0000 7f91a10de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:58:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.645+0000 7f91a10de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:58:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.645+0000 7f91a10de640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:58:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.645+0000 7f91a10de640 1 -- 172.21.15.26:0/610156022 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f919c0a6ee0 con 0x7f919c1c2520 2024-08-19T12:58:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.646+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7f918404de00 con 0x7f919c1c2520 2024-08-19T12:58:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.646+0000 7f91937fe640 10 client.214107.objecter ms_dispatch 0x7f919c000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:58:15.655 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.646+0000 7f91937fe640 3 client.214107.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:58:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.646+0000 7f91937fe640 3 client.214107.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:58:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.646+0000 7f91937fe640 20 client.214107.objecter dump_active .. 0 homeless 2024-08-19T12:58:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.646+0000 7f919ad9d640 10 monclient: get_auth_request con 0x7f917c044010 auth_method 0 2024-08-19T12:58:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.648+0000 7f91937fe640 10 client.214107.objecter ms_handle_connect 0x7f917c044010 2024-08-19T12:58:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.651+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f917c00ed60 con 0x7f919c1c2520 2024-08-19T12:58:15.656 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.651+0000 7f91937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:58:15.657 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.651+0000 7f91937fe640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:58:15.835 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.834+0000 7f91a10de640 1 -- 172.21.15.26:0/610156022 --> v1:172.21.15.104:6800/899887506 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f919c068df0 con 0x7f917c044010 2024-08-19T12:58:15.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.837+0000 7f91937fe640 1 -- 172.21.15.26:0/610156022 <== mgr.4115 v1:172.21.15.104:6800/899887506 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114156 (unknown 2965378022 0 276104064) 0x7f919c068df0 con 0x7f917c044010 2024-08-19T12:58:15.838 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-08-19T12:58:15.839 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-08-19T12:58:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 10 librados: watch_flush enter 2024-08-19T12:58:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 10 librados: watch_flush exit 2024-08-19T12:58:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 20 client.214107.objecter shutdown clearing up homeless session... 2024-08-19T12:58:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 10 client.214107.objecter successfully canceled tick 2024-08-19T12:58:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 1 -- 172.21.15.26:0/610156022 >> v1:172.21.15.104:6800/899887506 conn(0x7f917c044010 legacy=0x7f917c0464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 10 monclient: shutdown 2024-08-19T12:58:15.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:58:15.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 1 -- 172.21.15.26:0/610156022 >> v1:172.21.15.26:6792/0 conn(0x7f919c1c2520 legacy=0x7f919c1d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 1 -- 172.21.15.26:0/610156022 shutdown_connections 2024-08-19T12:58:15.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 1 -- 172.21.15.26:0/610156022 >> 172.21.15.26:0/610156022 conn(0x7f919c0a8740 msgr2=0x7f919c133970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:58:15.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 1 -- 172.21.15.26:0/610156022 shutdown_connections 2024-08-19T12:58:15.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 1 -- 172.21.15.26:0/610156022 wait complete. 2024-08-19T12:58:15.842 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.839+0000 7f91a10de640 1 librados: shutdown 2024-08-19T12:58:15.850 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1285,"stamp":"2024-08-19T12:58:15.395588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":441440,"kb_used_data":33600,"kb_used_omap":9,"kb_used_meta":407734,"kb_avail":565789600,"statfs":{"total":579820584960,"available":579368550400,"internally_reserved":0,"allocated":34406400,"data_stored":15183806,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9566,"internal_metadata":417520290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002779"},"pg_stats":[{"pgid":"7.17","version":"0'0","reported_seq":1374,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:49.861725+0000","last_change":"2024-08-19T12:57:49.861768+0000","last_active":"2024-08-19T12:57:49.861725+0000","last_peered":"2024-08-19T12:57:49.861725+0000","last_clean":"2024-08-19T12:57:49.861725+0000","last_became_active":"2024-08-19T12:39:38.143851+0000","last_became_peered":"2024-08-19T12:39:38.143851+0000","last_unstale":"2024-08-19T12:57:49.861725+0000","last_undegraded":"2024-08-19T12:57:49.861725+0000","last_fullsized":"2024-08-19T12:57:49.861725+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:49.861630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:49.861630+0000","last_clean_scrub_stamp":"2024-08-19T12:57:49.861630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:42:43.807796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1378,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:48.580600+0000","last_change":"2024-08-19T12:57:48.580629+0000","last_active":"2024-08-19T12:57:48.580600+0000","last_peered":"2024-08-19T12:57:48.580600+0000","last_clean":"2024-08-19T12:57:48.580600+0000","last_became_active":"2024-08-19T12:39:38.269168+0000","last_became_peered":"2024-08-19T12:39:38.269168+0000","last_unstale":"2024-08-19T12:57:48.580600+0000","last_undegraded":"2024-08-19T12:57:48.580600+0000","last_fullsized":"2024-08-19T12:57:48.580600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:48.580506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:48.580506+0000","last_clean_scrub_stamp":"2024-08-19T12:57:48.580506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T22:08:23.519081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1381,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:54.492872+0000","last_change":"2024-08-19T12:57:54.492900+0000","last_active":"2024-08-19T12:57:54.492872+0000","last_peered":"2024-08-19T12:57:54.492872+0000","last_clean":"2024-08-19T12:57:54.492872+0000","last_became_active":"2024-08-19T12:39:38.201163+0000","last_became_peered":"2024-08-19T12:39:38.201163+0000","last_unstale":"2024-08-19T12:57:54.492872+0000","last_undegraded":"2024-08-19T12:57:54.492872+0000","last_fullsized":"2024-08-19T12:57:54.492872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:54.492747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:54.492747+0000","last_clean_scrub_stamp":"2024-08-19T12:57:54.492747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:08:54.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1368,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:46.706328+0000","last_change":"2024-08-19T12:57:46.706351+0000","last_active":"2024-08-19T12:57:46.706328+0000","last_peered":"2024-08-19T12:57:46.706328+0000","last_clean":"2024-08-19T12:57:46.706328+0000","last_became_active":"2024-08-19T12:39:38.183511+0000","last_became_peered":"2024-08-19T12:39:38.183511+0000","last_unstale":"2024-08-19T12:57:46.706328+0000","last_undegraded":"2024-08-19T12:57:46.706328+0000","last_fullsized":"2024-08-19T12:57:46.706328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:46.706258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:46.706258+0000","last_clean_scrub_stamp":"2024-08-19T12:57:46.706258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:07:52.860510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1369,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:42.731332+0000","last_change":"2024-08-19T12:57:42.731353+0000","last_active":"2024-08-19T12:57:42.731332+0000","last_peered":"2024-08-19T12:57:42.731332+0000","last_clean":"2024-08-19T12:57:42.731332+0000","last_became_active":"2024-08-19T12:39:38.094806+0000","last_became_peered":"2024-08-19T12:39:38.094806+0000","last_unstale":"2024-08-19T12:57:42.731332+0000","last_undegraded":"2024-08-19T12:57:42.731332+0000","last_fullsized":"2024-08-19T12:57:42.731332+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:42.731264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:42.731264+0000","last_clean_scrub_stamp":"2024-08-19T12:57:42.731264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:11:14.746343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1380,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:46.200612+0000","last_change":"2024-08-19T12:57:46.200661+0000","last_active":"2024-08-19T12:57:46.200612+0000","last_peered":"2024-08-19T12:57:46.200612+0000","last_clean":"2024-08-19T12:57:46.200612+0000","last_became_active":"2024-08-19T12:39:38.219371+0000","last_became_peered":"2024-08-19T12:39:38.219371+0000","last_unstale":"2024-08-19T12:57:46.200612+0000","last_undegraded":"2024-08-19T12:57:46.200612+0000","last_fullsized":"2024-08-19T12:57:46.200612+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:46.200528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:46.200528+0000","last_clean_scrub_stamp":"2024-08-19T12:57:46.200528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:38:11.689874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:43.835092+0000","last_change":"2024-08-19T12:57:43.835137+0000","last_active":"2024-08-19T12:57:43.835092+0000","last_peered":"2024-08-19T12:57:43.835092+0000","last_clean":"2024-08-19T12:57:43.835092+0000","last_became_active":"2024-08-19T12:39:38.228951+0000","last_became_peered":"2024-08-19T12:39:38.228951+0000","last_unstale":"2024-08-19T12:57:43.835092+0000","last_undegraded":"2024-08-19T12:57:43.835092+0000","last_fullsized":"2024-08-19T12:57:43.835092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:43.835002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:43.835002+0000","last_clean_scrub_stamp":"2024-08-19T12:57:43.835002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:09:15.625703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1380,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:45.178649+0000","last_change":"2024-08-19T12:57:45.178672+0000","last_active":"2024-08-19T12:57:45.178649+0000","last_peered":"2024-08-19T12:57:45.178649+0000","last_clean":"2024-08-19T12:57:45.178649+0000","last_became_active":"2024-08-19T12:39:38.301100+0000","last_became_peered":"2024-08-19T12:39:38.301100+0000","last_unstale":"2024-08-19T12:57:45.178649+0000","last_undegraded":"2024-08-19T12:57:45.178649+0000","last_fullsized":"2024-08-19T12:57:45.178649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:45.178577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:45.178577+0000","last_clean_scrub_stamp":"2024-08-19T12:57:45.178577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:40:53.330246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:45.772398+0000","last_change":"2024-08-19T12:57:45.772418+0000","last_active":"2024-08-19T12:57:45.772398+0000","last_peered":"2024-08-19T12:57:45.772398+0000","last_clean":"2024-08-19T12:57:45.772398+0000","last_became_active":"2024-08-19T12:39:38.149472+0000","last_became_peered":"2024-08-19T12:39:38.149472+0000","last_unstale":"2024-08-19T12:57:45.772398+0000","last_undegraded":"2024-08-19T12:57:45.772398+0000","last_fullsized":"2024-08-19T12:57:45.772398+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:45.772330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:45.772330+0000","last_clean_scrub_stamp":"2024-08-19T12:57:45.772330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:56:29.623191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1380,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:47.168713+0000","last_change":"2024-08-19T12:57:47.168736+0000","last_active":"2024-08-19T12:57:47.168713+0000","last_peered":"2024-08-19T12:57:47.168713+0000","last_clean":"2024-08-19T12:57:47.168713+0000","last_became_active":"2024-08-19T12:39:38.181280+0000","last_became_peered":"2024-08-19T12:39:38.181280+0000","last_unstale":"2024-08-19T12:57:47.168713+0000","last_undegraded":"2024-08-19T12:57:47.168713+0000","last_fullsized":"2024-08-19T12:57:47.168713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:47.168632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:47.168632+0000","last_clean_scrub_stamp":"2024-08-19T12:57:47.168632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:19:43.998213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1379,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:45.567329+0000","last_change":"2024-08-19T12:57:45.567352+0000","last_active":"2024-08-19T12:57:45.567329+0000","last_peered":"2024-08-19T12:57:45.567329+0000","last_clean":"2024-08-19T12:57:45.567329+0000","last_became_active":"2024-08-19T12:39:38.147647+0000","last_became_peered":"2024-08-19T12:39:38.147647+0000","last_unstale":"2024-08-19T12:57:45.567329+0000","last_undegraded":"2024-08-19T12:57:45.567329+0000","last_fullsized":"2024-08-19T12:57:45.567329+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:45.567212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:45.567212+0000","last_clean_scrub_stamp":"2024-08-19T12:57:45.567212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:20:06.327578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1382,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:49.550060+0000","last_change":"2024-08-19T12:57:49.550091+0000","last_active":"2024-08-19T12:57:49.550060+0000","last_peered":"2024-08-19T12:57:49.550060+0000","last_clean":"2024-08-19T12:57:49.550060+0000","last_became_active":"2024-08-19T12:39:38.135334+0000","last_became_peered":"2024-08-19T12:39:38.135334+0000","last_unstale":"2024-08-19T12:57:49.550060+0000","last_undegraded":"2024-08-19T12:57:49.550060+0000","last_fullsized":"2024-08-19T12:57:49.550060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:49.549946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:49.549946+0000","last_clean_scrub_stamp":"2024-08-19T12:57:49.549946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:04:46.164070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"701'74","reported_seq":1491,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:48.592420+0000","last_change":"2024-08-19T12:57:48.592462+0000","last_active":"2024-08-19T12:57:48.592420+0000","last_peered":"2024-08-19T12:57:48.592420+0000","last_clean":"2024-08-19T12:57:48.592420+0000","last_became_active":"2024-08-19T12:38:22.728595+0000","last_became_peered":"2024-08-19T12:38:22.728595+0000","last_unstale":"2024-08-19T12:57:48.592420+0000","last_undegraded":"2024-08-19T12:57:48.592420+0000","last_fullsized":"2024-08-19T12:57:48.592420+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"701'74","last_scrub_stamp":"2024-08-19T12:57:48.592342+0000","last_deep_scrub":"701'74","last_deep_scrub_stamp":"2024-08-19T12:57:48.592342+0000","last_clean_scrub_stamp":"2024-08-19T12:57:48.592342+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:01:11.955730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"701'2","reported_seq":1393,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:45.975197+0000","last_change":"2024-08-19T12:57:45.975225+0000","last_active":"2024-08-19T12:57:45.975197+0000","last_peered":"2024-08-19T12:57:45.975197+0000","last_clean":"2024-08-19T12:57:45.975197+0000","last_became_active":"2024-08-19T12:38:29.952471+0000","last_became_peered":"2024-08-19T12:38:29.952471+0000","last_unstale":"2024-08-19T12:57:45.975197+0000","last_undegraded":"2024-08-19T12:57:45.975197+0000","last_fullsized":"2024-08-19T12:57:45.975197+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"701'2","last_scrub_stamp":"2024-08-19T12:57:45.975120+0000","last_deep_scrub":"701'2","last_deep_scrub_stamp":"2024-08-19T12:57:45.975120+0000","last_clean_scrub_stamp":"2024-08-19T12:57:45.975120+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:43:14.342739+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00057881800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:42.856838+0000","last_change":"2024-08-19T12:57:42.856860+0000","last_active":"2024-08-19T12:57:42.856838+0000","last_peered":"2024-08-19T12:57:42.856838+0000","last_clean":"2024-08-19T12:57:42.856838+0000","last_became_active":"2024-08-19T12:39:38.260911+0000","last_became_peered":"2024-08-19T12:39:38.260911+0000","last_unstale":"2024-08-19T12:57:42.856838+0000","last_undegraded":"2024-08-19T12:57:42.856838+0000","last_fullsized":"2024-08-19T12:57:42.856838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:42.856756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:42.856756+0000","last_clean_scrub_stamp":"2024-08-19T12:57:42.856756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:24:09.740358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1398,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:44.611466+0000","last_change":"2024-08-19T12:57:44.611523+0000","last_active":"2024-08-19T12:57:44.611466+0000","last_peered":"2024-08-19T12:57:44.611466+0000","last_clean":"2024-08-19T12:57:44.611466+0000","last_became_active":"2024-08-19T12:38:29.951331+0000","last_became_peered":"2024-08-19T12:38:29.951331+0000","last_unstale":"2024-08-19T12:57:44.611466+0000","last_undegraded":"2024-08-19T12:57:44.611466+0000","last_fullsized":"2024-08-19T12:57:44.611466+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:44.611369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:44.611369+0000","last_clean_scrub_stamp":"2024-08-19T12:57:44.611369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:01:35.605008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018789830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1379,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:47.599976+0000","last_change":"2024-08-19T12:57:47.599998+0000","last_active":"2024-08-19T12:57:47.599976+0000","last_peered":"2024-08-19T12:57:47.599976+0000","last_clean":"2024-08-19T12:57:47.599976+0000","last_became_active":"2024-08-19T12:39:38.295614+0000","last_became_peered":"2024-08-19T12:39:38.295614+0000","last_unstale":"2024-08-19T12:57:47.599976+0000","last_undegraded":"2024-08-19T12:57:47.599976+0000","last_fullsized":"2024-08-19T12:57:47.599976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:47.599900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:47.599900+0000","last_clean_scrub_stamp":"2024-08-19T12:57:47.599900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:20:22.230789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1396,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:43.217237+0000","last_change":"2024-08-19T12:57:43.217258+0000","last_active":"2024-08-19T12:57:43.217237+0000","last_peered":"2024-08-19T12:57:43.217237+0000","last_clean":"2024-08-19T12:57:43.217237+0000","last_became_active":"2024-08-19T12:38:29.952459+0000","last_became_peered":"2024-08-19T12:38:29.952459+0000","last_unstale":"2024-08-19T12:57:43.217237+0000","last_undegraded":"2024-08-19T12:57:43.217237+0000","last_fullsized":"2024-08-19T12:57:43.217237+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:43.217168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:43.217168+0000","last_clean_scrub_stamp":"2024-08-19T12:57:43.217168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T18:20:57.812482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039863799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:44.787946+0000","last_change":"2024-08-19T12:57:44.787971+0000","last_active":"2024-08-19T12:57:44.787946+0000","last_peered":"2024-08-19T12:57:44.787946+0000","last_clean":"2024-08-19T12:57:44.787946+0000","last_became_active":"2024-08-19T12:39:38.218230+0000","last_became_peered":"2024-08-19T12:39:38.218230+0000","last_unstale":"2024-08-19T12:57:44.787946+0000","last_undegraded":"2024-08-19T12:57:44.787946+0000","last_fullsized":"2024-08-19T12:57:44.787946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:44.787860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:44.787860+0000","last_clean_scrub_stamp":"2024-08-19T12:57:44.787860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:42:10.169968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"701'3","reported_seq":1394,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:46.937789+0000","last_change":"2024-08-19T12:57:46.937820+0000","last_active":"2024-08-19T12:57:46.937789+0000","last_peered":"2024-08-19T12:57:46.937789+0000","last_clean":"2024-08-19T12:57:46.937789+0000","last_became_active":"2024-08-19T12:38:29.951392+0000","last_became_peered":"2024-08-19T12:38:29.951392+0000","last_unstale":"2024-08-19T12:57:46.937789+0000","last_undegraded":"2024-08-19T12:57:46.937789+0000","last_fullsized":"2024-08-19T12:57:46.937789+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"701'3","last_scrub_stamp":"2024-08-19T12:57:46.937702+0000","last_deep_scrub":"701'3","last_deep_scrub_stamp":"2024-08-19T12:57:46.937702+0000","last_clean_scrub_stamp":"2024-08-19T12:57:46.937702+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:54:12.795026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056602099999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1382,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:47.595587+0000","last_change":"2024-08-19T12:57:47.595622+0000","last_active":"2024-08-19T12:57:47.595587+0000","last_peered":"2024-08-19T12:57:47.595587+0000","last_clean":"2024-08-19T12:57:47.595587+0000","last_became_active":"2024-08-19T12:39:38.217569+0000","last_became_peered":"2024-08-19T12:39:38.217569+0000","last_unstale":"2024-08-19T12:57:47.595587+0000","last_undegraded":"2024-08-19T12:57:47.595587+0000","last_fullsized":"2024-08-19T12:57:47.595587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:47.595486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:47.595486+0000","last_clean_scrub_stamp":"2024-08-19T12:57:47.595486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T16:36:10.806033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1385,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:44.679220+0000","last_change":"2024-08-19T12:57:44.679245+0000","last_active":"2024-08-19T12:57:44.679220+0000","last_peered":"2024-08-19T12:57:44.679220+0000","last_clean":"2024-08-19T12:57:44.679220+0000","last_became_active":"2024-08-19T12:38:29.951233+0000","last_became_peered":"2024-08-19T12:38:29.951233+0000","last_unstale":"2024-08-19T12:57:44.679220+0000","last_undegraded":"2024-08-19T12:57:44.679220+0000","last_fullsized":"2024-08-19T12:57:44.679220+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:44.679136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:44.679136+0000","last_clean_scrub_stamp":"2024-08-19T12:57:44.679136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:14:41.205478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048735899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"701'8","reported_seq":1393,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:46.599791+0000","last_change":"2024-08-19T12:57:46.599828+0000","last_active":"2024-08-19T12:57:46.599791+0000","last_peered":"2024-08-19T12:57:46.599791+0000","last_clean":"2024-08-19T12:57:46.599791+0000","last_became_active":"2024-08-19T12:39:38.269442+0000","last_became_peered":"2024-08-19T12:39:38.269442+0000","last_unstale":"2024-08-19T12:57:46.599791+0000","last_undegraded":"2024-08-19T12:57:46.599791+0000","last_fullsized":"2024-08-19T12:57:46.599791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"701'8","last_scrub_stamp":"2024-08-19T12:57:46.599705+0000","last_deep_scrub":"701'8","last_deep_scrub_stamp":"2024-08-19T12:57:46.599705+0000","last_clean_scrub_stamp":"2024-08-19T12:57:46.599705+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:11:27.212524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1398,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:46.610297+0000","last_change":"2024-08-19T12:57:46.610323+0000","last_active":"2024-08-19T12:57:46.610297+0000","last_peered":"2024-08-19T12:57:46.610297+0000","last_clean":"2024-08-19T12:57:46.610297+0000","last_became_active":"2024-08-19T12:38:29.949846+0000","last_became_peered":"2024-08-19T12:38:29.949846+0000","last_unstale":"2024-08-19T12:57:46.610297+0000","last_undegraded":"2024-08-19T12:57:46.610297+0000","last_fullsized":"2024-08-19T12:57:46.610297+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:46.610181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:46.610181+0000","last_clean_scrub_stamp":"2024-08-19T12:57:46.610181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:16:13.763708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001822192,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1376,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:47.906741+0000","last_change":"2024-08-19T12:57:47.906786+0000","last_active":"2024-08-19T12:57:47.906741+0000","last_peered":"2024-08-19T12:57:47.906741+0000","last_clean":"2024-08-19T12:57:47.906741+0000","last_became_active":"2024-08-19T12:39:38.166658+0000","last_became_peered":"2024-08-19T12:39:38.166658+0000","last_unstale":"2024-08-19T12:57:47.906741+0000","last_undegraded":"2024-08-19T12:57:47.906741+0000","last_fullsized":"2024-08-19T12:57:47.906741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:47.906660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:47.906660+0000","last_clean_scrub_stamp":"2024-08-19T12:57:47.906660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T19:50:40.929232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1385,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:43.690322+0000","last_change":"2024-08-19T12:57:43.690322+0000","last_active":"2024-08-19T12:57:43.690322+0000","last_peered":"2024-08-19T12:57:43.690322+0000","last_clean":"2024-08-19T12:57:43.690322+0000","last_became_active":"2024-08-19T12:38:29.950467+0000","last_became_peered":"2024-08-19T12:38:29.950467+0000","last_unstale":"2024-08-19T12:57:43.690322+0000","last_undegraded":"2024-08-19T12:57:43.690322+0000","last_fullsized":"2024-08-19T12:57:43.690322+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:43.690211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:43.690211+0000","last_clean_scrub_stamp":"2024-08-19T12:57:43.690211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:21:48.889201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049350399999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:41.906039+0000","last_change":"2024-08-19T12:57:41.906063+0000","last_active":"2024-08-19T12:57:41.906039+0000","last_peered":"2024-08-19T12:57:41.906039+0000","last_clean":"2024-08-19T12:57:41.906039+0000","last_became_active":"2024-08-19T12:39:38.190786+0000","last_became_peered":"2024-08-19T12:39:38.190786+0000","last_unstale":"2024-08-19T12:57:41.906039+0000","last_undegraded":"2024-08-19T12:57:41.906039+0000","last_fullsized":"2024-08-19T12:57:41.906039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:41.905967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:41.905967+0000","last_clean_scrub_stamp":"2024-08-19T12:57:41.905967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T14:43:22.005981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1398,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:45.611033+0000","last_change":"2024-08-19T12:57:45.611033+0000","last_active":"2024-08-19T12:57:45.611033+0000","last_peered":"2024-08-19T12:57:45.611033+0000","last_clean":"2024-08-19T12:57:45.611033+0000","last_became_active":"2024-08-19T12:38:29.948183+0000","last_became_peered":"2024-08-19T12:38:29.948183+0000","last_unstale":"2024-08-19T12:57:45.611033+0000","last_undegraded":"2024-08-19T12:57:45.611033+0000","last_fullsized":"2024-08-19T12:57:45.611033+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:45.610911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:45.610911+0000","last_clean_scrub_stamp":"2024-08-19T12:57:45.610911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:09:33.589737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052740100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1380,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:44.222769+0000","last_change":"2024-08-19T12:57:44.222790+0000","last_active":"2024-08-19T12:57:44.222769+0000","last_peered":"2024-08-19T12:57:44.222769+0000","last_clean":"2024-08-19T12:57:44.222769+0000","last_became_active":"2024-08-19T12:39:38.191158+0000","last_became_peered":"2024-08-19T12:39:38.191158+0000","last_unstale":"2024-08-19T12:57:44.222769+0000","last_undegraded":"2024-08-19T12:57:44.222769+0000","last_fullsized":"2024-08-19T12:57:44.222769+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:44.222700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:44.222700+0000","last_clean_scrub_stamp":"2024-08-19T12:57:44.222700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T21:15:30.862909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1368,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:45.669170+0000","last_change":"2024-08-19T12:57:45.669209+0000","last_active":"2024-08-19T12:57:45.669170+0000","last_peered":"2024-08-19T12:57:45.669170+0000","last_clean":"2024-08-19T12:57:45.669170+0000","last_became_active":"2024-08-19T12:39:38.161352+0000","last_became_peered":"2024-08-19T12:39:38.161352+0000","last_unstale":"2024-08-19T12:57:45.669170+0000","last_undegraded":"2024-08-19T12:57:45.669170+0000","last_fullsized":"2024-08-19T12:57:45.669170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:45.669067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:45.669067+0000","last_clean_scrub_stamp":"2024-08-19T12:57:45.669067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:19:14.031299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1381,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:50.548211+0000","last_change":"2024-08-19T12:57:50.548240+0000","last_active":"2024-08-19T12:57:50.548211+0000","last_peered":"2024-08-19T12:57:50.548211+0000","last_clean":"2024-08-19T12:57:50.548211+0000","last_became_active":"2024-08-19T12:39:38.202668+0000","last_became_peered":"2024-08-19T12:39:38.202668+0000","last_unstale":"2024-08-19T12:57:50.548211+0000","last_undegraded":"2024-08-19T12:57:50.548211+0000","last_fullsized":"2024-08-19T12:57:50.548211+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:50.548111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:50.548111+0000","last_clean_scrub_stamp":"2024-08-19T12:57:50.548111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:07:23.118892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1374,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:48.896534+0000","last_change":"2024-08-19T12:57:48.896576+0000","last_active":"2024-08-19T12:57:48.896534+0000","last_peered":"2024-08-19T12:57:48.896534+0000","last_clean":"2024-08-19T12:57:48.896534+0000","last_became_active":"2024-08-19T12:39:38.222877+0000","last_became_peered":"2024-08-19T12:39:38.222877+0000","last_unstale":"2024-08-19T12:57:48.896534+0000","last_undegraded":"2024-08-19T12:57:48.896534+0000","last_fullsized":"2024-08-19T12:57:48.896534+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:48.896401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:48.896401+0000","last_clean_scrub_stamp":"2024-08-19T12:57:48.896401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T17:52:58.430627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1364,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:47.794058+0000","last_change":"2024-08-19T12:57:47.794081+0000","last_active":"2024-08-19T12:57:47.794058+0000","last_peered":"2024-08-19T12:57:47.794058+0000","last_clean":"2024-08-19T12:57:47.794058+0000","last_became_active":"2024-08-19T12:39:38.210985+0000","last_became_peered":"2024-08-19T12:39:38.210985+0000","last_unstale":"2024-08-19T12:57:47.794058+0000","last_undegraded":"2024-08-19T12:57:47.794058+0000","last_fullsized":"2024-08-19T12:57:47.794058+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:47.793980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:47.793980+0000","last_clean_scrub_stamp":"2024-08-19T12:57:47.793980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:47:57.083242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1381,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:53.487776+0000","last_change":"2024-08-19T12:57:53.487801+0000","last_active":"2024-08-19T12:57:53.487776+0000","last_peered":"2024-08-19T12:57:53.487776+0000","last_clean":"2024-08-19T12:57:53.487776+0000","last_became_active":"2024-08-19T12:39:38.267362+0000","last_became_peered":"2024-08-19T12:39:38.267362+0000","last_unstale":"2024-08-19T12:57:53.487776+0000","last_undegraded":"2024-08-19T12:57:53.487776+0000","last_fullsized":"2024-08-19T12:57:53.487776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:53.487674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:53.487674+0000","last_clean_scrub_stamp":"2024-08-19T12:57:53.487674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T15:15:51.408681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1368,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:47.728189+0000","last_change":"2024-08-19T12:57:47.728222+0000","last_active":"2024-08-19T12:57:47.728189+0000","last_peered":"2024-08-19T12:57:47.728189+0000","last_clean":"2024-08-19T12:57:47.728189+0000","last_became_active":"2024-08-19T12:39:38.173967+0000","last_became_peered":"2024-08-19T12:39:38.173967+0000","last_unstale":"2024-08-19T12:57:47.728189+0000","last_undegraded":"2024-08-19T12:57:47.728189+0000","last_fullsized":"2024-08-19T12:57:47.728189+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:47.728102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:47.728102+0000","last_clean_scrub_stamp":"2024-08-19T12:57:47.728102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:40:45.465330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1370,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:48.698683+0000","last_change":"2024-08-19T12:57:48.698713+0000","last_active":"2024-08-19T12:57:48.698683+0000","last_peered":"2024-08-19T12:57:48.698683+0000","last_clean":"2024-08-19T12:57:48.698683+0000","last_became_active":"2024-08-19T12:39:38.108670+0000","last_became_peered":"2024-08-19T12:39:38.108670+0000","last_unstale":"2024-08-19T12:57:48.698683+0000","last_undegraded":"2024-08-19T12:57:48.698683+0000","last_fullsized":"2024-08-19T12:57:48.698683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:48.698583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:48.698583+0000","last_clean_scrub_stamp":"2024-08-19T12:57:48.698583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-21T00:46:49.571540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1375,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:50.876505+0000","last_change":"2024-08-19T12:57:50.876531+0000","last_active":"2024-08-19T12:57:50.876505+0000","last_peered":"2024-08-19T12:57:50.876505+0000","last_clean":"2024-08-19T12:57:50.876505+0000","last_became_active":"2024-08-19T12:39:38.197445+0000","last_became_peered":"2024-08-19T12:39:38.197445+0000","last_unstale":"2024-08-19T12:57:50.876505+0000","last_undegraded":"2024-08-19T12:57:50.876505+0000","last_fullsized":"2024-08-19T12:57:50.876505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:50.876407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:50.876407+0000","last_clean_scrub_stamp":"2024-08-19T12:57:50.876407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T22:57:59.498592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1364,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:46.804689+0000","last_change":"2024-08-19T12:57:46.804709+0000","last_active":"2024-08-19T12:57:46.804689+0000","last_peered":"2024-08-19T12:57:46.804689+0000","last_clean":"2024-08-19T12:57:46.804689+0000","last_became_active":"2024-08-19T12:39:38.267711+0000","last_became_peered":"2024-08-19T12:39:38.267711+0000","last_unstale":"2024-08-19T12:57:46.804689+0000","last_undegraded":"2024-08-19T12:57:46.804689+0000","last_fullsized":"2024-08-19T12:57:46.804689+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:46.804625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:46.804625+0000","last_clean_scrub_stamp":"2024-08-19T12:57:46.804625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:21:21.124794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1381,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:51.512125+0000","last_change":"2024-08-19T12:57:51.512174+0000","last_active":"2024-08-19T12:57:51.512125+0000","last_peered":"2024-08-19T12:57:51.512125+0000","last_clean":"2024-08-19T12:57:51.512125+0000","last_became_active":"2024-08-19T12:39:38.177735+0000","last_became_peered":"2024-08-19T12:39:38.177735+0000","last_unstale":"2024-08-19T12:57:51.512125+0000","last_undegraded":"2024-08-19T12:57:51.512125+0000","last_fullsized":"2024-08-19T12:57:51.512125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:51.512019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:51.512019+0000","last_clean_scrub_stamp":"2024-08-19T12:57:51.512019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T22:49:02.301072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1381,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:52.489843+0000","last_change":"2024-08-19T12:57:52.489871+0000","last_active":"2024-08-19T12:57:52.489843+0000","last_peered":"2024-08-19T12:57:52.489843+0000","last_clean":"2024-08-19T12:57:52.489843+0000","last_became_active":"2024-08-19T12:39:38.199715+0000","last_became_peered":"2024-08-19T12:39:38.199715+0000","last_unstale":"2024-08-19T12:57:52.489843+0000","last_undegraded":"2024-08-19T12:57:52.489843+0000","last_fullsized":"2024-08-19T12:57:52.489843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:52.489704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:52.489704+0000","last_clean_scrub_stamp":"2024-08-19T12:57:52.489704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T23:38:05.591647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1364,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-08-19T12:57:48.760238+0000","last_change":"2024-08-19T12:57:48.760274+0000","last_active":"2024-08-19T12:57:48.760238+0000","last_peered":"2024-08-19T12:57:48.760238+0000","last_clean":"2024-08-19T12:57:48.760238+0000","last_became_active":"2024-08-19T12:39:38.190871+0000","last_became_peered":"2024-08-19T12:39:38.190871+0000","last_unstale":"2024-08-19T12:57:48.760238+0000","last_undegraded":"2024-08-19T12:57:48.760238+0000","last_fullsized":"2024-08-19T12:57:48.760238+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-19T12:57:48.760142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-19T12:57:48.760142+0000","last_clean_scrub_stamp":"2024-08-19T12:57:48.760142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-20T20:45:55.454100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10228,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607793,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68704,"kb_used_data":5584,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303136,"statfs":{"total":96636764160,"available":96566411264,"internally_reserved":0,"allocated":5718016,"data_stored":2354336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607793,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68704,"kb_used_data":5584,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303136,"statfs":{"total":96636764160,"available":96566411264,"internally_reserved":0,"allocated":5718016,"data_stored":2354322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607796,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68752,"kb_used_data":5632,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303088,"statfs":{"total":96636764160,"available":96566362112,"internally_reserved":0,"allocated":5767168,"data_stored":2883245,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1592,"internal_metadata":64616904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607792,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77020,"kb_used_data":5580,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294820,"statfs":{"total":96636764160,"available":96557895680,"internally_reserved":0,"allocated":5713920,"data_stored":2354317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607793,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77024,"kb_used_data":5584,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94294816,"statfs":{"total":96636764160,"available":96557891584,"internally_reserved":0,"allocated":5718016,"data_stored":2354336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640499,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81236,"kb_used_data":5636,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94290604,"statfs":{"total":96636764160,"available":96553578496,"internally_reserved":0,"allocated":5771264,"data_stored":2883250,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5114,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-19T12:58:15.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-08-19T12:58:15.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.951+0000 7f6d4e74c640 10 monclient: get_monmap_and_config 2024-08-19T12:58:15.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.951+0000 7f6d4e74c640 10 monclient: build_initial_monmap 2024-08-19T12:58:15.952 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.951+0000 7f6d4e74c640 10 monclient: monmap: 2024-08-19T12:58:15.952 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:58:15.952562+0000 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:58:15.952562+0000 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:58:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:58:15.954 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:58:15.954 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 1 Processor -- start 2024-08-19T12:58:15.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 1 -- start start 2024-08-19T12:58:15.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 10 monclient: init 2024-08-19T12:58:15.955 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 10 monclient: _reopen_session rank -1 2024-08-19T12:58:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 10 monclient: _add_conns ranks=[7,5,1,2,8,0,4,3,6] 2024-08-19T12:58:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 10 monclient(hunting): picked mon.noname-i con 0x7f6d4806e7b0 addr v1:172.21.15.104:6792/0 2024-08-19T12:58:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 10 monclient(hunting): picked mon.noname-h con 0x7f6d480a5920 addr v1:172.21.15.104:6791/0 2024-08-19T12:58:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 10 monclient(hunting): picked mon.noname-f con 0x7f6d480a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:58:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d48136790 con 0x7f6d480a5d50 2024-08-19T12:58:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d4809ddf0 con 0x7f6d480a5920 2024-08-19T12:58:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d4809e0d0 con 0x7f6d4806e7b0 2024-08-19T12:58:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:15.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.952+0000 7f6d4e74c640 10 monclient(hunting): authenticate will time out at 1890.039982s 2024-08-19T12:58:15.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.953+0000 7f6d47fff640 1 --1- >> v1:172.21.15.104:6792/0 conn(0x7f6d4806e7b0 0x7f6d48136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6792/0 says I am v1:172.21.15.26:49414/0 (socket says 172.21.15.26:49414) 2024-08-19T12:58:15.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.953+0000 7f6d47fff640 1 -- 172.21.15.26:0/3817129406 learned_addr learned my addr 172.21.15.26:0/3817129406 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:58:15.957 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.953+0000 7f6d4cce9640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6d480a5d50 0x7f6d48141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.26:56242/0 (socket says 172.21.15.26:56242) 2024-08-19T12:58:15.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d4cce9640 1 -- 172.21.15.26:0/3817129406 >> v1:172.21.15.104:6789/0 conn(0x7f6d480a5d50 legacy=0x7f6d48141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:15.960 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d47fff640 1 -- 172.21.15.26:0/3817129406 >> v1:172.21.15.104:6792/0 conn(0x7f6d4806e7b0 legacy=0x7f6d48136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 1 -- 172.21.15.26:0/3817129406 <== mon.7 v1:172.21.15.104:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2377774157 0 0) 0x7f6d4809e0d0 con 0x7f6d4806e7b0 2024-08-19T12:58:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:58:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:58:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:58:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 10 monclient(hunting): my global_id is 136636 2024-08-19T12:58:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 1 -- 172.21.15.26:0/3817129406 --> v1:172.21.15.104:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d30003520 con 0x7f6d4806e7b0 2024-08-19T12:58:15.961 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 1 -- 172.21.15.26:0/3817129406 <== mon.1 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2546270069 0 0) 0x7f6d48136790 con 0x7f6d480a5d50 2024-08-19T12:58:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 10 monclient(hunting): _negotiate 2024-08-19T12:58:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:58:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:58:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 10 monclient(hunting): my global_id is 204445 2024-08-19T12:58:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 1 -- 172.21.15.26:0/3817129406 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d4809e0d0 con 0x7f6d480a5d50 2024-08-19T12:58:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 1 -- 172.21.15.26:0/3817129406 <== mon.7 v1:172.21.15.104:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4073184178 0 0) 0x7f6d30003520 con 0x7f6d4806e7b0 2024-08-19T12:58:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.955+0000 7f6d46ffd640 1 -- 172.21.15.26:0/3817129406 >> v1:172.21.15.104:6791/0 conn(0x7f6d480a5920 legacy=0x7f6d4813e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-19T12:58:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 1 -- 172.21.15.26:0/3817129406 >> v1:172.21.15.104:6789/0 conn(0x7f6d480a5d50 legacy=0x7f6d48141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.962 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 10 monclient: _finish_hunting 0 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 1 monclient: found mon.noname-i 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.104:6792/0 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 1 -- 172.21.15.26:0/3817129406 --> v1:172.21.15.104:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4809e3b0 con 0x7f6d4806e7b0 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 10 monclient: _finish_auth 0 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 10 monclient: _check_auth_tickets 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 1 -- 172.21.15.26:0/3817129406 <== mon.7 v1:172.21.15.104:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6d38001a10 con 0x7f6d4806e7b0 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 10 monclient: dump: 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:58:15.963 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:58:15.964 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d4e74c640 5 monclient: authenticate success, global_id 136636 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d4e74c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 1 -- 172.21.15.26:0/3817129406 <== mon.7 v1:172.21.15.104:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6d38001ce0 con 0x7f6d4806e7b0 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 10 monclient: handle_config config(0 keys) 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d4e74c640 10 monclient: get_monmap_and_config success 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d4e74c640 4 set_mon_vals no callback set 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 1 -- 172.21.15.26:0/3817129406 <== mon.7 v1:172.21.15.104:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6d38004c20 con 0x7f6d4806e7b0 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d46ffd640 10 monclient: dump: 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:58:15.965 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:58:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d4e74c640 10 monclient: shutdown 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d4e74c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d4e74c640 1 -- 172.21.15.26:0/3817129406 >> v1:172.21.15.104:6792/0 conn(0x7f6d4806e7b0 legacy=0x7f6d48136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d4e74c640 1 -- 172.21.15.26:0/3817129406 shutdown_connections 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d4e74c640 1 -- 172.21.15.26:0/3817129406 >> 172.21.15.26:0/3817129406 conn(0x7f6d480a8740 msgr2=0x7f6d480a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d4e74c640 1 -- 172.21.15.26:0/3817129406 shutdown_connections 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.956+0000 7f6d4e74c640 1 -- 172.21.15.26:0/3817129406 wait complete. 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.957+0000 7f6d4e74c640 10 monclient: build_initial_monmap 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.957+0000 7f6d4e74c640 10 monclient: monmap: 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-19T12:58:15.967 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-08-19T12:58:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-08-19T12:58:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:58:15.968 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:58:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:58:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:58:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:58:15.969 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:58:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:58:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:58:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:58:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:58:15.970 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.957+0000 7f6d4e74c640 1 librados: starting msgr at 2024-08-19T12:58:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.957+0000 7f6d4e74c640 1 librados: starting objecter 2024-08-19T12:58:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.957+0000 7f6d4e74c640 1 Processor -- start 2024-08-19T12:58:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.957+0000 7f6d4e74c640 1 -- start start 2024-08-19T12:58:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.957+0000 7f6d4e74c640 1 librados: setting wanted keys 2024-08-19T12:58:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.957+0000 7f6d4e74c640 1 librados: calling monclient init 2024-08-19T12:58:15.970 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.957+0000 7f6d4e74c640 10 monclient: init 2024-08-19T12:58:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.958+0000 7f6d4e74c640 10 monclient: _reopen_session rank -1 2024-08-19T12:58:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.958+0000 7f6d4e74c640 10 monclient: _add_conns ranks=[0,3,6,2,1,8,4,7,5] 2024-08-19T12:58:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.958+0000 7f6d4e74c640 10 monclient(hunting): picked mon.noname-a con 0x7f6d4806e7b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:58:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.958+0000 7f6d4e74c640 10 monclient(hunting): picked mon.noname-g con 0x7f6d480a5920 addr v1:172.21.15.104:6790/0 2024-08-19T12:58:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.958+0000 7f6d4e74c640 10 monclient(hunting): picked mon.noname-d con 0x7f6d480a5d50 addr v1:172.21.15.26:6792/0 2024-08-19T12:58:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.958+0000 7f6d4e74c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d480a0d40 con 0x7f6d4806e7b0 2024-08-19T12:58:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.958+0000 7f6d4e74c640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d48134390 con 0x7f6d480a5920 2024-08-19T12:58:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.958+0000 7f6d4e74c640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d480ab8c0 con 0x7f6d480a5d50 2024-08-19T12:58:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.958+0000 7f6d4e74c640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.958+0000 7f6d4e74c640 10 monclient(hunting): authenticate will time out at 1890.045222s 2024-08-19T12:58:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.958+0000 7f6d47fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6d4806e7b0 0x7f6d48132190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:47382/0 (socket says 172.21.15.26:47382) 2024-08-19T12:58:15.971 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.958+0000 7f6d47fff640 1 -- 172.21.15.26:0/2940135447 learned_addr learned my addr 172.21.15.26:0/2940135447 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-08-19T12:58:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d47fff640 1 -- 172.21.15.26:0/2940135447 >> v1:172.21.15.26:6789/0 conn(0x7f6d4806e7b0 legacy=0x7f6d48132190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:15.976 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 client.?.objecter ms_handle_connect 0x7f6d4806e7b0 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d4cce9640 1 -- 172.21.15.26:0/2940135447 >> v1:172.21.15.26:6792/0 conn(0x7f6d480a5d50 legacy=0x7f6d481d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 client.?.objecter ms_handle_connect 0x7f6d480a5d50 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 client.?.objecter ms_handle_connect 0x7f6d480a5920 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 client.?.objecter resend_mon_ops 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d477fe640 1 -- 172.21.15.26:0/2940135447 >> v1:172.21.15.104:6790/0 conn(0x7f6d480a5920 legacy=0x7f6d481328a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.6 v1:172.21.15.26:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 597346700 0 0) 0x7f6d480ab8c0 con 0x7f6d480a5d50 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 monclient(hunting): my global_id is 214116 2024-08-19T12:58:15.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 --> v1:172.21.15.26:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d34004eb0 con 0x7f6d480a5d50 2024-08-19T12:58:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.0 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1912133478 0 0) 0x7f6d480a0d40 con 0x7f6d4806e7b0 2024-08-19T12:58:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:58:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:58:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:58:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 monclient(hunting): my global_id is 214101 2024-08-19T12:58:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d480ab8c0 con 0x7f6d4806e7b0 2024-08-19T12:58:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.3 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145671650 0 0) 0x7f6d48134390 con 0x7f6d480a5920 2024-08-19T12:58:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 monclient(hunting): _negotiate 2024-08-19T12:58:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 monclient(hunting): _init_auth method 2 2024-08-19T12:58:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-08-19T12:58:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 10 monclient(hunting): my global_id is 214104 2024-08-19T12:58:15.978 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.960+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d480a0d40 con 0x7f6d480a5920 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.0 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 816518657 0 0) 0x7f6d480ab8c0 con 0x7f6d4806e7b0 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d48134390 con 0x7f6d4806e7b0 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.6 v1:172.21.15.26:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3957984819 0 0) 0x7f6d34004eb0 con 0x7f6d480a5d50 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 --> v1:172.21.15.26:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d480ab8c0 con 0x7f6d480a5d50 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.0 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6d38004b20 con 0x7f6d4806e7b0 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient(hunting): dump: 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.979 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient(hunting): _finish_auth 0 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.6 v1:172.21.15.26:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6d40002020 con 0x7f6d480a5d50 2024-08-19T12:58:15.980 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient(hunting): dump: 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:58:15.981 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:58:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:58:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:58:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:58:15.982 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.3 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2522336413 0 0) 0x7f6d480a0d40 con 0x7f6d480a5920 2024-08-19T12:58:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6d34004eb0 con 0x7f6d480a5920 2024-08-19T12:58:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.3 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6d3c001c30 con 0x7f6d480a5920 2024-08-19T12:58:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-08-19T12:58:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-08-19T12:58:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient(hunting): dump: 2024-08-19T12:58:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:58:15.982 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:58:15.983 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.6 v1:172.21.15.26:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 222226354 0 0) 0x7f6d480ab8c0 con 0x7f6d480a5d50 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 >> v1:172.21.15.104:6790/0 conn(0x7f6d480a5920 legacy=0x7f6d481328a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 >> v1:172.21.15.26:6789/0 conn(0x7f6d4806e7b0 legacy=0x7f6d48132190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient: _finish_hunting 0 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 monclient: found mon.d 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 --> v1:172.21.15.26:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d480a0a60 con 0x7f6d480a5d50 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient: _finish_auth 0 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient: _check_auth_tickets 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d4e74c640 5 monclient: authenticate success, global_id 214116 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d4e74c640 10 monclient: _renew_subs 2024-08-19T12:58:15.984 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d4e74c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:58:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d4e74c640 1 -- 172.21.15.26:0/2940135447 --> v1:172.21.15.26:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4809f540 con 0x7f6d480a5d50 2024-08-19T12:58:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d4e74c640 10 client.214116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-19T12:58:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d4e74c640 10 monclient: _renew_subs 2024-08-19T12:58:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d4e74c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:58:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d4e74c640 1 -- 172.21.15.26:0/2940135447 --> v1:172.21.15.26:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f6d48133190 con 0x7f6d480a5d50 2024-08-19T12:58:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d4e74c640 1 librados: init done 2024-08-19T12:58:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.6 v1:172.21.15.26:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6d40004ec0 con 0x7f6d480a5d50 2024-08-19T12:58:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient: handle_config config(0 keys) 2024-08-19T12:58:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.6 v1:172.21.15.26:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 251107226 0 0) 0x7f6d400053f0 con 0x7f6d480a5d50 2024-08-19T12:58:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-19T12:58:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-08-19T12:58:15.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d467fc640 4 set_mon_vals no callback set 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.961+0000 7f6d44ff9640 10 monclient: dump: 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:37:41.103727+0000 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.a 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.f 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.b 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.g 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.c 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.h 2024-08-19T12:58:15.986 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.d 2024-08-19T12:58:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.i 2024-08-19T12:58:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.e 2024-08-19T12:58:15.987 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.962+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.6 v1:172.21.15.26:6792/0 7 ==== mgrmap(e 5) ==== 49845+0+0 (unknown 604826443 0 0) 0x7f6d40005590 con 0x7f6d480a5d50 2024-08-19T12:58:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.962+0000 7f6d4e74c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-19T12:58:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.962+0000 7f6d4e74c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:58:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.962+0000 7f6d4e74c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:58:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.962+0000 7f6d4e74c640 1 -- 172.21.15.26:0/2940135447 --> v1:172.21.15.26:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d28005080 con 0x7f6d480a5d50 2024-08-19T12:58:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.962+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.6 v1:172.21.15.26:6792/0 8 ==== osd_map(701..701 src has 658..701) ==== 4673+0+0 (unknown 2465364956 0 0) 0x7f6d4004e030 con 0x7f6d480a5d50 2024-08-19T12:58:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.962+0000 7f6d44ff9640 10 client.214116.objecter ms_dispatch 0x7f6d48000b90 osd_map(701..701 src has 658..701) 2024-08-19T12:58:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.962+0000 7f6d44ff9640 3 client.214116.objecter handle_osd_map got epochs [701,701] > 0 2024-08-19T12:58:15.987 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.962+0000 7f6d44ff9640 3 client.214116.objecter handle_osd_map decoding full epoch 701 2024-08-19T12:58:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.962+0000 7f6d44ff9640 20 client.214116.objecter dump_active .. 0 homeless 2024-08-19T12:58:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.962+0000 7f6d47fff640 10 monclient: get_auth_request con 0x7f6d340418e0 auth_method 0 2024-08-19T12:58:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.965+0000 7f6d44ff9640 10 client.214116.objecter ms_handle_connect 0x7f6d340418e0 2024-08-19T12:58:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.967+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.6 v1:172.21.15.26:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (unknown 1092875540 0 565763019) 0x7f6d40017020 con 0x7f6d480a5d50 2024-08-19T12:58:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.967+0000 7f6d44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-19T12:58:15.988 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:15.967+0000 7f6d44ff9640 10 monclient: _finish_command 1 = system:0 2024-08-19T12:58:16.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.150+0000 7f6d4e74c640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-08-19T12:58:16.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.150+0000 7f6d4e74c640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-08-19T12:58:16.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.150+0000 7f6d4e74c640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.26:6792/0 2024-08-19T12:58:16.151 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.150+0000 7f6d4e74c640 1 -- 172.21.15.26:0/2940135447 --> v1:172.21.15.26:6792/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f6d28005370 con 0x7f6d480a5d50 2024-08-19T12:58:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.157+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.6 v1:172.21.15.26:6792/0 10 ==== config(1 keys) ==== 35+0+0 (unknown 2555425225 0 0) 0x7f6d40016b90 con 0x7f6d480a5d50 2024-08-19T12:58:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.157+0000 7f6d44ff9640 10 monclient: handle_config config(1 keys) 2024-08-19T12:58:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.157+0000 7f6d467fc640 4 set_mon_vals no callback set 2024-08-19T12:58:16.158 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.157+0000 7f6d467fc640 10 set_mon_vals mon_health_to_clog = false 2024-08-19T12:58:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.158+0000 7f6d44ff9640 1 -- 172.21.15.26:0/2940135447 <== mon.6 v1:172.21.15.26:6792/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v7)=0 v7) ==== 123+0+0 (unknown 3414686427 0 0) 0x7f6d4001b510 con 0x7f6d480a5d50 2024-08-19T12:58:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.158+0000 7f6d44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-08-19T12:58:16.159 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.158+0000 7f6d44ff9640 10 monclient: _finish_command 2 = system:0 2024-08-19T12:58:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 10 librados: watch_flush enter 2024-08-19T12:58:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 10 librados: watch_flush exit 2024-08-19T12:58:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 20 client.214116.objecter shutdown clearing up homeless session... 2024-08-19T12:58:16.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 10 client.214116.objecter successfully canceled tick 2024-08-19T12:58:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 1 -- 172.21.15.26:0/2940135447 >> v1:172.21.15.104:6800/899887506 conn(0x7f6d340418e0 legacy=0x7f6d34043da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 10 monclient: shutdown 2024-08-19T12:58:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-19T12:58:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 1 -- 172.21.15.26:0/2940135447 >> v1:172.21.15.26:6792/0 conn(0x7f6d480a5d50 legacy=0x7f6d481d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-19T12:58:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 1 -- 172.21.15.26:0/2940135447 shutdown_connections 2024-08-19T12:58:16.163 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 1 -- 172.21.15.26:0/2940135447 >> 172.21.15.26:0/2940135447 conn(0x7f6d480a8740 msgr2=0x7f6d4806d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-19T12:58:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 1 -- 172.21.15.26:0/2940135447 shutdown_connections 2024-08-19T12:58:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 1 -- 172.21.15.26:0/2940135447 wait complete. 2024-08-19T12:58:16.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:16.160+0000 7f6d4e74c640 1 librados: shutdown 2024-08-19T12:58:16.171 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/ceph.py", line 1980, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 466, in task thrash_proc.stop_and_join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 178, in stop_and_join return self.join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 170, in join self.stopping.set() AttributeError: 'bool' object has no attribute 'set' 2024-08-19T12:58:16.173 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-19T12:58:16.173 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-19T12:58:16.174 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-08-19T12:58:16.174 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.208 INFO:tasks.ceph.osd.0:Stopped 2024-08-19T12:58:16.208 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-08-19T12:58:16.208 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.289 INFO:tasks.ceph.osd.1:Stopped 2024-08-19T12:58:16.289 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-08-19T12:58:16.289 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.374 INFO:tasks.ceph.osd.2:Stopped 2024-08-19T12:58:16.374 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-08-19T12:58:16.374 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.409 INFO:tasks.ceph.osd.3:Stopped 2024-08-19T12:58:16.410 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-08-19T12:58:16.410 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.490 INFO:tasks.ceph.osd.4:Stopped 2024-08-19T12:58:16.490 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-08-19T12:58:16.490 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.560 INFO:tasks.ceph.osd.5:Stopped 2024-08-19T12:58:16.561 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-19T12:58:16.561 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-08-19T12:58:16.561 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.628 INFO:tasks.ceph.mgr.x:Stopped 2024-08-19T12:58:16.629 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-19T12:58:16.629 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-19T12:58:16.629 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.639 INFO:tasks.ceph.mon.a:Stopped 2024-08-19T12:58:16.639 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-19T12:58:16.640 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.697 INFO:tasks.ceph.mon.b:Stopped 2024-08-19T12:58:16.698 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-19T12:58:16.698 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.751 INFO:tasks.ceph.mon.c:Stopped 2024-08-19T12:58:16.752 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-08-19T12:58:16.752 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.802 INFO:tasks.ceph.mon.d:Stopped 2024-08-19T12:58:16.803 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-08-19T12:58:16.803 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.856 INFO:tasks.ceph.mon.e:Stopped 2024-08-19T12:58:16.856 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-08-19T12:58:16.856 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.871 INFO:tasks.ceph.mon.f:Stopped 2024-08-19T12:58:16.871 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-08-19T12:58:16.871 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.924 INFO:tasks.ceph.mon.g:Stopped 2024-08-19T12:58:16.924 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-08-19T12:58:16.925 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:16.981 INFO:tasks.ceph.mon.h:Stopped 2024-08-19T12:58:16.981 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-08-19T12:58:16.981 INFO:teuthology.orchestra.run:waiting for 300 2024-08-19T12:58:17.041 INFO:tasks.ceph.mon.i:Stopped 2024-08-19T12:58:17.041 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-19T12:58:17.041 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-08-19T12:58:17.094 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi026.front.sepia.ceph.com 2024-08-19T12:58:17.094 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-08-19T12:58:17.272 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi026.front.sepia.ceph.com 2024-08-19T12:58:17.272 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-08-19T12:58:17.346 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi026.front.sepia.ceph.com 2024-08-19T12:58:17.346 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-08-19T12:58:17.412 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi104.front.sepia.ceph.com 2024-08-19T12:58:17.412 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-08-19T12:58:17.526 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi104.front.sepia.ceph.com 2024-08-19T12:58:17.526 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-08-19T12:58:17.639 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi104.front.sepia.ceph.com 2024-08-19T12:58:17.639 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-08-19T12:58:17.696 INFO:tasks.ceph:Archiving mon data... 2024-08-19T12:58:17.698 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/data/mon.a.tgz 2024-08-19T12:58:17.698 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-08-19T12:58:17.713 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.U8su9CAf1D 2024-08-19T12:58:17.714 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.U8su9CAf1D 2024-08-19T12:58:17.964 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.U8su9CAf1D 2024-08-19T12:58:18.047 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.U8su9CAf1D is 1MB 2024-08-19T12:58:18.156 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.U8su9CAf1D 2024-08-19T12:58:18.171 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/data/mon.b.tgz 2024-08-19T12:58:18.171 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-08-19T12:58:18.225 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.jC8iqKJfsK 2024-08-19T12:58:18.226 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.jC8iqKJfsK 2024-08-19T12:58:18.371 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.jC8iqKJfsK 2024-08-19T12:58:18.454 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.jC8iqKJfsK is 597KB 2024-08-19T12:58:18.553 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.jC8iqKJfsK 2024-08-19T12:58:18.568 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/data/mon.c.tgz 2024-08-19T12:58:18.568 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-08-19T12:58:18.622 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.zSLFVanCbU 2024-08-19T12:58:18.623 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.zSLFVanCbU 2024-08-19T12:58:18.767 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.zSLFVanCbU 2024-08-19T12:58:18.851 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.zSLFVanCbU is 590KB 2024-08-19T12:58:18.925 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.zSLFVanCbU 2024-08-19T12:58:18.939 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-d to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/data/mon.d.tgz 2024-08-19T12:58:18.940 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-08-19T12:58:18.994 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.CQTNBanSnD 2024-08-19T12:58:18.994 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-d -- . > /tmp/tmp.CQTNBanSnD 2024-08-19T12:58:19.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.CQTNBanSnD 2024-08-19T12:58:19.321 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.CQTNBanSnD is 1MB 2024-08-19T12:58:19.429 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.CQTNBanSnD 2024-08-19T12:58:19.444 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-e to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/data/mon.e.tgz 2024-08-19T12:58:19.445 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-08-19T12:58:19.498 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.8uTXHGMnuG 2024-08-19T12:58:19.499 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-e -- . > /tmp/tmp.8uTXHGMnuG 2024-08-19T12:58:19.643 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.8uTXHGMnuG 2024-08-19T12:58:19.727 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.8uTXHGMnuG is 589KB 2024-08-19T12:58:19.827 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.8uTXHGMnuG 2024-08-19T12:58:19.842 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/mon/ceph-f to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/data/mon.f.tgz 2024-08-19T12:58:19.842 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-08-19T12:58:19.861 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.qu4CvULGXz 2024-08-19T12:58:19.861 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-f -- . > /tmp/tmp.qu4CvULGXz 2024-08-19T12:58:20.096 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.qu4CvULGXz 2024-08-19T12:58:20.181 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.qu4CvULGXz is 584KB 2024-08-19T12:58:20.280 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.qu4CvULGXz 2024-08-19T12:58:20.295 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/mon/ceph-g to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/data/mon.g.tgz 2024-08-19T12:58:20.295 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-08-19T12:58:20.350 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.iwrzfhy3DQ 2024-08-19T12:58:20.350 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-g -- . > /tmp/tmp.iwrzfhy3DQ 2024-08-19T12:58:20.444 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-08-19T12:58:20.444 ERROR:tasks.ceph.mon.b:No such daemon running 2024-08-19T12:58:20.445 INFO:tasks.mon_thrash:Sending CONT to mon h 2024-08-19T12:58:20.445 ERROR:tasks.ceph.mon.h:No such daemon running 2024-08-19T12:58:20.445 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-08-19T12:58:20.445 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-08-19T12:58:20.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.550+0000 7f2fdcfb8640 10 monclient: get_monmap_and_config 2024-08-19T12:58:20.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.550+0000 7f2fdcfb8640 10 monclient: build_initial_monmap 2024-08-19T12:58:20.551 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.550+0000 7f2fdcfb8640 10 monclient: monmap: 2024-08-19T12:58:20.551 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-08-19T12:58:20.551 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 10d02fe0-13d8-499a-99b3-d3a725cba366 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-08-19T12:58:20.551737+0000 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-08-19T12:58:20.551737+0000 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.26:6789/0 mon.noname-a 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.104:6789/0 mon.noname-f 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.26:6790/0 mon.noname-b 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:3: v1:172.21.15.104:6790/0 mon.noname-g 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:4: v1:172.21.15.26:6791/0 mon.noname-c 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:5: v1:172.21.15.104:6791/0 mon.noname-h 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:6: v1:172.21.15.26:6792/0 mon.noname-d 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:7: v1:172.21.15.104:6792/0 mon.noname-i 2024-08-19T12:58:20.552 INFO:teuthology.orchestra.run.smithi026.stderr:8: v1:172.21.15.26:6793/0 mon.noname-e 2024-08-19T12:58:20.553 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:20.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.551+0000 7f2fdcfb8640 1 Processor -- start 2024-08-19T12:58:20.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.551+0000 7f2fdcfb8640 1 -- start start 2024-08-19T12:58:20.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.551+0000 7f2fdcfb8640 10 monclient: init 2024-08-19T12:58:20.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fdcfb8640 10 monclient: _reopen_session rank -1 2024-08-19T12:58:20.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fdcfb8640 10 monclient: _add_conns ranks=[2,0,1,6,8,4,3,5,7] 2024-08-19T12:58:20.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fdcfb8640 10 monclient(hunting): picked mon.noname-b con 0x7f2fd806e7b0 addr v1:172.21.15.26:6790/0 2024-08-19T12:58:20.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fdcfb8640 10 monclient(hunting): picked mon.noname-a con 0x7f2fd80a5920 addr v1:172.21.15.26:6789/0 2024-08-19T12:58:20.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fdcfb8640 10 monclient(hunting): picked mon.noname-f con 0x7f2fd80a5d50 addr v1:172.21.15.104:6789/0 2024-08-19T12:58:20.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fdcfb8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd8136790 con 0x7f2fd80a5920 2024-08-19T12:58:20.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fdcfb8640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd809ddf0 con 0x7f2fd80a5d50 2024-08-19T12:58:20.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fdcfb8640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd809e0d0 con 0x7f2fd806e7b0 2024-08-19T12:58:20.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fdcfb8640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:20.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fdcfb8640 10 monclient(hunting): authenticate will time out at 1894.639260s 2024-08-19T12:58:20.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2fd806e7b0 legacy=0x7f2fd8136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-08-19T12:58:20.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fd80a5920 legacy=0x7f2fd813e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:58:20.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.552+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2fd80a5d50 legacy=0x7f2fd8141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:58:20.585 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.iwrzfhy3DQ 2024-08-19T12:58:20.669 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.iwrzfhy3DQ is 1MB 2024-08-19T12:58:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.753+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fd80a5920 legacy=0x7f2fd813e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:58:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.753+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2fd806e7b0 legacy=0x7f2fd8136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-08-19T12:58:20.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:20.753+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2fd80a5d50 legacy=0x7f2fd8141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:58:20.789 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.iwrzfhy3DQ 2024-08-19T12:58:20.804 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/mon/ceph-h to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/data/mon.h.tgz 2024-08-19T12:58:20.804 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-08-19T12:58:20.859 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.gRLQa697sz 2024-08-19T12:58:20.859 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-h -- . > /tmp/tmp.gRLQa697sz 2024-08-19T12:58:21.004 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.gRLQa697sz 2024-08-19T12:58:21.089 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.gRLQa697sz is 584KB 2024-08-19T12:58:21.154 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:21.154+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fd80a5920 legacy=0x7f2fd813e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:58:21.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:21.154+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2fd806e7b0 legacy=0x7f2fd8136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-08-19T12:58:21.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:21.154+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2fd80a5d50 legacy=0x7f2fd8141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:58:21.187 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.gRLQa697sz 2024-08-19T12:58:21.203 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/mon/ceph-i to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/data/mon.i.tgz 2024-08-19T12:58:21.203 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-08-19T12:58:21.257 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.IeRnX20FlY 2024-08-19T12:58:21.258 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-i -- . > /tmp/tmp.IeRnX20FlY 2024-08-19T12:58:21.399 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.IeRnX20FlY 2024-08-19T12:58:21.483 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.IeRnX20FlY is 601KB 2024-08-19T12:58:21.583 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.IeRnX20FlY 2024-08-19T12:58:21.599 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-19T12:58:21.599 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-19T12:58:21.602 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-19T12:58:21.687 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-19T12:58:21.718 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-19T12:58:21.718 INFO:tasks.ceph:Archiving crash dumps... 2024-08-19T12:58:21.722 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/remote/smithi026/crash 2024-08-19T12:58:21.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-19T12:58:21.755 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/remote/smithi104/crash 2024-08-19T12:58:21.756 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-19T12:58:21.782 INFO:tasks.ceph:Compressing logs... 2024-08-19T12:58:21.782 DEBUG:teuthology.orchestra.run.smithi026:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-19T12:58:21.800 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-19T12:58:21.824 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37433.log 2024-08-19T12:58:21.824 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27161.log 2024-08-19T12:58:21.829 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34521.log 2024-08-19T12:58:21.829 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37433.log: /var/log/ceph/ceph-client.admin.27161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28326.log 2024-08-19T12:58:21.829 INFO:teuthology.orchestra.run.smithi026.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37433.log.gz 2024-08-19T12:58:21.829 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34143.log 2024-08-19T12:58:21.829 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27161.log.gz 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr:gzip/var/log/ceph/ceph-client.admin.28326.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31388.log 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34521.log.gz 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36467.log 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28326.log.gz 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36929.log 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34143.log.gz 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28536.log 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31388.log.gz 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32360.log 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36467.log.gz 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28536.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36929.log.gz 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29168.log 2024-08-19T12:58:21.830 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28956.log 2024-08-19T12:58:21.831 INFO:teuthology.orchestra.run.smithi026.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28536.log.gz 2024-08-19T12:58:21.831 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24265.log 2024-08-19T12:58:21.831 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32360.log.gz 2024-08-19T12:58:21.831 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37969.log 2024-08-19T12:58:21.831 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29168.log.gz 2024-08-19T12:58:21.831 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31262.log 2024-08-19T12:58:21.831 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28956.log.gz 2024-08-19T12:58:21.831 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34725.log 2024-08-19T12:58:21.831 INFO:teuthology.orchestra.run.smithi026.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.24265.log.gz 2024-08-19T12:58:21.831 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34017.log 2024-08-19T12:58:21.831 INFO:teuthology.orchestra.run.smithi026.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37969.log.gz 2024-08-19T12:58:21.894 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26547.log 2024-08-19T12:58:21.895 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.h.log 2024-08-19T12:58:21.895 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25642.log 2024-08-19T12:58:21.895 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26547.log: /var/log/ceph/ceph-mon.h.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24748.log 2024-08-19T12:58:21.895 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25087.log 2024-08-19T12:58:21.895 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24748.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.i.log 2024-08-19T12:58:21.895 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21214.log 2024-08-19T12:58:21.895 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mon.i.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24537.log 2024-08-19T12:58:21.895 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25354.log 2024-08-19T12:58:21.895 INFO:teuthology.orchestra.run.smithi104.stderr: 90.5%/var/log/ceph/ceph-client.admin.24537.log: -- replaced with /var/log/ceph/ceph-client.admin.26547.log.gz 2024-08-19T12:58:21.895 INFO:teuthology.orchestra.run.smithi104.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.21214.log.gz 2024-08-19T12:58:21.895 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26396.log 2024-08-19T12:58:21.895 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25178.log 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.25354.log.gz 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26396.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.26396.log.gz 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26067.log 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26116.log 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.24537.log.gz 91.6% -- replaced with /var/log/ceph/ceph-client.admin.25087.log.gz 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26119.log 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25873.log 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26119.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.25642.log.gz 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.26119.log.gz 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.26067.log.gz 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr: 90.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.26116.log.gz -5 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr: --verbose -- /var/log/ceph/ceph-client.admin.26121.log 2024-08-19T12:58:21.896 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24494.log 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26121.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24660.log 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.26121.log.gz 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.3.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.25873.log.gz 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.25178.log.gz 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25103.log 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24660.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.g.log 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26117.log 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mon.g.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.24660.log.gz 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24631.log 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26117.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.26117.log.gz 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24631.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.25103.log.gz 2024-08-19T12:58:21.897 INFO:teuthology.orchestra.run.smithi104.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.24631.log.gz 2024-08-19T12:58:21.898 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25648.log 2024-08-19T12:58:21.898 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26397.log 2024-08-19T12:58:21.898 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25648.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.25648.log.gz 2024-08-19T12:58:21.898 INFO:teuthology.orchestra.run.smithi104.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.24494.log.gz 2024-08-19T12:58:21.901 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-19T12:58:21.901 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25378.log 2024-08-19T12:58:21.901 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.audit.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.26397.log.gz 2024-08-19T12:58:21.914 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26448.log 2024-08-19T12:58:21.914 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25378.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.25378.log.gz 2024-08-19T12:58:21.927 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24561.log 2024-08-19T12:58:21.930 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26448.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.26448.log.gz 2024-08-19T12:58:21.932 INFO:teuthology.orchestra.run.smithi104.stderr: 96.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-19T12:58:21.932 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26548.log 2024-08-19T12:58:21.942 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25381.log 2024-08-19T12:58:21.942 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24561.log: /var/log/ceph/ceph-client.admin.26548.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.26548.log.gz 2024-08-19T12:58:21.942 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24517.log 2024-08-19T12:58:21.942 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25381.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.25381.log.gz 2024-08-19T12:58:21.942 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25826.log 2024-08-19T12:58:21.954 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24517.log: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.24561.log.gz 2024-08-19T12:58:21.954 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25850.log 2024-08-19T12:58:21.954 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25826.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.25826.log.gz 2024-08-19T12:58:21.954 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25363.log 2024-08-19T12:58:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32950.log 2024-08-19T12:58:21.961 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31262.log.gz 2024-08-19T12:58:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32572.log 2024-08-19T12:58:21.961 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34725.log.gz 2024-08-19T12:58:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27071.log 2024-08-19T12:58:21.961 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34017.log.gz 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35355.log 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32950.log.gz 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28662.log 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.32572.log.gz 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32528.log 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27071.log.gz 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32150.log 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35355.log.gz 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27581.log 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28662.log.gz 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24267.log 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32528.log.gz 2024-08-19T12:58:21.962 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35733.log 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32150.log.gz 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30606.log 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27581.log.gz 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29892.log 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.24267.log.gz 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30144.log 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35733.log.gz 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33597.log 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30606.log.gz 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33555.log 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29892.log.gz 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25388.log 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30144.log.gz 2024-08-19T12:58:21.963 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27833.log 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33597.log.gz 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23926.log 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33555.log.gz 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27833.log: gzip 88.1% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.25388.log.gz 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.30839.log 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.23926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30923.log 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27833.log.gz 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24519.log 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.23926.log.gz 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37055.log 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30839.log.gz 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32234.log 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30923.log.gz 2024-08-19T12:58:21.964 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37055.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.24519.log.gz 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30438.log 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33723.log 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32234.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37055.log.gz 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26102.log 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32234.log.gz 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25178.log 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30438.log.gz 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31049.log 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33723.log.gz 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30522.log 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.26102.log.gz 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30965.log 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.25178.log.gz 2024-08-19T12:58:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35607.log 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31049.log.gz 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36887.log 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30522.log.gz 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24128.log 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30965.log.gz 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31472.log 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35607.log.gz 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26735.log 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36887.log.gz 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25598.log 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.24128.log.gz 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33118.log 2024-08-19T12:58:21.966 INFO:teuthology.orchestra.run.smithi026.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31472.log.gz 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25086.log 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.26735.log.gz 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36719.log 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.25598.log.gz 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36257.log 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33118.log.gz 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35313.log 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.25086.log.gz 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26903.log 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36719.log.gz 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25682.log 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36257.log.gz 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29084.log 2024-08-19T12:58:21.967 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35313.log.gz 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33807.log 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.26903.log.gz 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26018.log 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.25682.log.gz 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30396.log 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29084.log.gz 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26018.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33807.log.gz 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28578.log 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31219.log 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26018.log.gz 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36593.log 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30396.log.gz 2024-08-19T12:58:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24520.log 2024-08-19T12:58:21.969 INFO:teuthology.orchestra.run.smithi026.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28578.log.gz 2024-08-19T12:58:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36593.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.31219.log.gz 2024-08-19T12:58:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29934.log 2024-08-19T12:58:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27329.log 2024-08-19T12:58:21.969 INFO:teuthology.orchestra.run.smithi026.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36593.log.gz 2024-08-19T12:58:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29934.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.24520.log.gz -5 2024-08-19T12:58:21.969 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32108.log 2024-08-19T12:58:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36425.log 2024-08-19T12:58:21.969 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29934.log.gz 2024-08-19T12:58:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33204.log 2024-08-19T12:58:21.969 INFO:teuthology.orchestra.run.smithi026.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27329.log.gz 2024-08-19T12:58:21.969 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35439.log 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32108.log.gz 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24694.log 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36425.log.gz 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35439.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33204.log.gz 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30312.log 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35439.log.gz 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28070.log 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.24694.log.gz 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28154.log 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28070.log: gzip -5 88.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.30312.log.gz 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr: /var/log/ceph/ceph-client.admin.33471.log 2024-08-19T12:58:21.970 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28914.log 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28070.log.gz 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36005.log 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28154.log.gz 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28914.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33471.log.gz 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37559.log 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36971.log 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28914.log.gz 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32402.log 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36005.log.gz 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36971.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37559.log.gz 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29766.log 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38662.log 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36971.log.gz 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37013.log 2024-08-19T12:58:21.971 INFO:teuthology.orchestra.run.smithi026.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32402.log.gz 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29766.log.gz 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.38662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30186.log 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr: 82.5% -- replaced with /var/log/ceph/ceph-client.admin.38662.log.gz 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29042.log 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34767.log 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37013.log.gz 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28200.log 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30186.log.gz 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36089.log 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29042.log.gz 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28452.log 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34767.log.gz 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33387.log 2024-08-19T12:58:21.972 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28200.log.gz 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- 89.4%/var/log/ceph/ceph-client.admin.28452.log: /var/log/ceph/ceph-client.admin.34479.log 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36089.log.gz 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34851.log 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28452.log.gz 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34479.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33387.log.gz 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34269.log 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32908.log 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34851.log: /var/log/ceph/ceph-client.admin.34269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33975.log 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34479.log.gz 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32908.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34851.log.gz 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29210.log 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34269.log.gz 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26987.log 2024-08-19T12:58:21.973 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33975.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32908.log.gz 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33034.log 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33975.log.gz 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37349.log 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29210.log.gz 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25892.log 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.26987.log.gz 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37349.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33034.log.gz 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25640.log 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31091.log 86.3% 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37349.log.gz 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25430.log 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25892.log.gz 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24693.log 2024-08-19T12:58:21.974 INFO:teuthology.orchestra.run.smithi026.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.25640.log.gz 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26691.log 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25430.log: /var/log/ceph/ceph-client.admin.24693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29294.log 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31091.log.gz 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.25430.log.gz 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26691.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.24693.log.gz 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29252.log 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34437.log 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.26691.log.gz 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25080.log 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29294.log.gz 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24266.log 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29252.log.gz 2024-08-19T12:58:21.975 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37307.log 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25080.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34437.log.gz 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27875.log 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.25080.log.gz 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20804.log 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.24266.log.gz 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37769.log 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37307.log.gz 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.20804.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27875.log.gz 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36677.log 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37769.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.20804.log.gz 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32866.log 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31688.log 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37769.log.gz 2024-08-19T12:58:21.976 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37517.log 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36677.log.gz 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30060.log 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31688.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32866.log.gz 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24926.log 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31688.log.gz 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36131.log 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.37517.log.gz 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24926.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30060.log.gz 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37265.log 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36131.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.24926.log.gz 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30354.log 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36131.log.gz 2024-08-19T12:58:21.977 INFO:teuthology.orchestra.run.smithi026.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37265.log.gz 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27245.log 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27623.log 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31940.log 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30354.log.gz 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27623.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27245.log.gz 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33765.log 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27791.log 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27623.log.gz 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28830.log 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31940.log.gz 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31600.log 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33765.log.gz 2024-08-19T12:58:21.978 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26861.log 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28830.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27791.log.gz 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27455.log 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3%gzip/var/log/ceph/ceph-client.admin.26861.log: -- replaced with /var/log/ceph/ceph-client.admin.28830.log.gz 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29682.log 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31600.log.gz 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34311.log 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29682.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.26861.log.gz 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37685.log 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27455.log.gz 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29640.log 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34311.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29682.log.gz 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37139.log 2024-08-19T12:58:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29640.log: 89.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.34311.log.gz -5 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-mon.a.log 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37685.log.gz 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36047.log 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29640.log.gz 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24002.log 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37139.log.gz 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34809.log 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24714.log 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36047.log.gz 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34809.log: gzip -5 87.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.24002.log.gz /var/log/ceph/ceph-client.admin.34977.log 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28788.log 2024-08-19T12:58:21.980 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34977.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34809.log.gz 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35691.log 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.24714.log.gz 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26186.log 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34977.log.gz 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29472.log 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28788.log.gz 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30228.log 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35691.log.gz 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32992.log 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.26186.log.gz 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29000.log 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29472.log.gz 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25472.log 2024-08-19T12:58:21.981 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30228.log.gz 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26228.log 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32992.log.gz 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27497.log 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29000.log.gz 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26228.log: gzip 87.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.25472.log.gz --verbose 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/ceph-client.admin.32444.log 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.27497.log: -- /var/log/ceph/ceph-client.admin.27749.log 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26228.log.gz 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37097.log 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27497.log.gz 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25850.log 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32444.log.gz 2024-08-19T12:58:21.982 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36383.log 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27749.log.gz 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36341.log 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37097.log.gz 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37727.log 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.25850.log.gz 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35187.log 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36383.log.gz 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29976.log 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36341.log.gz 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28620.log 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37727.log.gz 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33076.log 2024-08-19T12:58:21.983 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35187.log.gz 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28494.log 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29976.log.gz 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28284.log 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28620.log.gz 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29388.log 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28494.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33076.log.gz 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28872.log 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29388.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28494.log.gz 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37181.log 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28284.log.gz 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31346.log 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.29388.log.gz 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30102.log 2024-08-19T12:58:21.984 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28872.log.gz 2024-08-19T12:58:21.985 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25027.log 2024-08-19T12:58:21.985 INFO:teuthology.orchestra.run.smithi026.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37181.log.gz 2024-08-19T12:58:21.985 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29850.log 2024-08-19T12:58:21.985 INFO:teuthology.orchestra.run.smithi026.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31346.log.gz 2024-08-19T12:58:21.985 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31516.log 2024-08-19T12:58:21.985 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30102.log.gz 2024-08-19T12:58:21.985 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37643.log 2024-08-19T12:58:21.985 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29850.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.25027.log.gz 2024-08-19T12:58:21.985 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28410.log 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37643.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29850.log.gz 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35775.log 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31516.log.gz 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24518.log 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37643.log.gz 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35775.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28410.log.gz 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25976.log 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mon.b.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35775.log.gz 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32698.log 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.24518.log.gz 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34227.log 2024-08-19T12:58:21.986 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24044.log 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25976.log.gz 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36761.log 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32698.log.gz 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26777.log 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24044.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34227.log.gz 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26819.log 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.24044.log.gz 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26945.log 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26777.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36761.log.gz 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28746.log 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26945.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26777.log.gz 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25934.log 2024-08-19T12:58:21.987 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.26819.log.gz 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28746.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26945.log.gz 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28746.log.gz 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32192.log 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35481.log 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34683.log 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.25934.log.gz 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24263.log 88.5% 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32192.log.gz 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25514.log 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34683.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35481.log.gz 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35523.log 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34683.log.gz 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27203.log 2024-08-19T12:58:21.988 INFO:teuthology.orchestra.run.smithi026.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.24263.log.gz 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25514.log: /var/log/ceph/ceph-client.admin.35523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24513.log 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.25514.log.gz 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27203.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.d.log 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35523.log.gz 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24808.log 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27203.log.gz 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32486.log 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mon.d.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.24513.log.gz 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr:gzip/var/log/ceph/ceph-client.admin.24808.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36551.log 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34059.log 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.24808.log.gz 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32486.log: /var/log/ceph/ceph-client.admin.36551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34185.log 2024-08-19T12:58:21.989 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34059.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32486.log.gz 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36551.log.gz 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35103.log 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34059.log.gz 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33933.log 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34185.log: /var/log/ceph/ceph-client.admin.35103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26144.log 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34185.log.gz 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33933.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35103.log.gz 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26312.log 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29724.log 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33933.log.gz 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26312.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26144.log.gz 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.26312.log.gz 2024-08-19T12:58:21.990 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35565.log 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32276.log 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36173.log 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29724.log.gz 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr: 97.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35565.log.gz 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25808.log 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25724.log 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32276.log.gz 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29430.log 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36173.log.gz 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25724.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.25808.log.gz 2024-08-19T12:58:21.991 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33162.log 2024-08-19T12:58:21.992 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21154.log 2024-08-19T12:58:21.992 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25724.log.gz 2024-08-19T12:58:21.992 INFO:teuthology.orchestra.run.smithi026.stderr: 88.4%/var/log/ceph/ceph-osd.0.log: -- replaced with /var/log/ceph/ceph-client.admin.29430.log.gz 2024-08-19T12:58:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26270.log 2024-08-19T12:58:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.21154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36299.log 2024-08-19T12:58:21.992 INFO:teuthology.orchestra.run.smithi026.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33162.log.gz 2024-08-19T12:58:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.26270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37391.log 88.0% 2024-08-19T12:58:21.992 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.21154.log.gz 2024-08-19T12:58:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27287.log 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26270.log.gz 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37391.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36299.log.gz 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34395.log 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37391.log.gz 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24522.log 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34395.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28026.log 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27287.log.gz 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34395.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28368.log 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24522.log: /var/log/ceph/ceph-client.admin.28026.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.24522.log.gz 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28026.log.gz 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27665.log 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32656.log 2024-08-19T12:58:21.993 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25220.log 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28368.log.gz 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32656.log: gzip -5 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27665.log.gz 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33681.log 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25304.log 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr: 89.4% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.25220.log.gz 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33681.log: -- replaced with /var/log/ceph/ceph-client.admin.32656.log.gz 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31007.log 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25304.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33681.log.gz 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31898.log 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31007.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.25304.log.gz 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31730.log 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31814.log 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31007.log.gz 2024-08-19T12:58:21.994 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31898.log: /var/log/ceph/ceph-client.admin.31730.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31898.log.gz 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31730.log.gz 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33849.log 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37475.log 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30270.log 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31814.log.gz 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33849.log.gz 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27029.log 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr: 88.1%/var/log/ceph/ceph-client.admin.30270.log: -- replaced with /var/log/ceph/ceph-client.admin.37475.log.gz 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31558.log 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35649.log 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30270.log.gz/var/log/ceph/ceph-client.admin.27029.log: 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31558.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27029.log.gz 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31558.log.gz 2024-08-19T12:58:21.995 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32318.log 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29808.log 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35145.log 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35649.log.gz 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29808.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32318.log.gz 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35019.log 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29808.log.gz/var/log/ceph/ceph-client.admin.35145.log: 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37601.log 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25556.log 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35145.log.gz 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37601.log: /var/log/ceph/ceph-client.admin.35019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29126.log 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25556.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.37601.log.gz 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35019.log.gz 2024-08-19T12:58:21.996 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose 89.3% -- replaced with /var/log/ceph/ceph-client.admin.25556.log.gz 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr: -- /var/log/ceph/ceph-client.admin.29556.log 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25262.log 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29556.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36845.log 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29126.log.gz 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr:gzip 89.3% -5 --verbose -- /var/log/ceph/ceph-client.admin.31644.log 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29556.log.gz 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25262.log: /var/log/ceph/ceph-client.admin.36845.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.25262.log.gz 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36845.log.gz 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34935.log 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30704.log 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31644.log: /var/log/ceph/ceph-client.admin.34935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29514.log 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30704.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31644.log.gz 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34935.log.gz 2024-08-19T12:58:21.997 INFO:teuthology.orchestra.run.smithi026.stderr:gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.30704.log.gz 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30018.log 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.1.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35271.log 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29514.log.gz 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30018.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30018.log.gz 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26060.log 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.16122.log 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35271.log: /var/log/ceph/ceph-client.admin.26060.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35271.log.gz 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23870.log 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26060.log.gz 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.tmp-client.admin.16122.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.16122.log.gz 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25346.log 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35963.log 2024-08-19T12:58:21.998 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.23870.log: /var/log/ceph/ceph-client.admin.25346.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.23870.log.gz 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.25346.log.gz 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24269.log 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35963.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35963.log.gz 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33429.log 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33639.log 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24269.log: /var/log/ceph/ceph-client.admin.33429.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.24269.log.gz 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33429.log.gz 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30797.log 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33639.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33639.log.gz 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32614.log 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24946.log 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30797.log: /var/log/ceph/ceph-client.admin.32614.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30797.log.gz 2024-08-19T12:58:21.999 INFO:teuthology.orchestra.run.smithi026.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32614.log.gz 2024-08-19T12:58:22.000 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-19T12:58:22.000 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24946.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.24946.log.gz 2024-08-19T12:58:22.000 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31430.log 2024-08-19T12:58:22.000 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36509.log 2024-08-19T12:58:22.000 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.31430.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31430.log.gz 2024-08-19T12:58:22.000 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37223.log 2024-08-19T12:58:22.000 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36509.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36509.log.gz 2024-08-19T12:58:22.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:21.955+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2fd806e7b0 legacy=0x7f2fd8136080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-08-19T12:58:22.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:21.955+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fd80a5920 legacy=0x7f2fd813e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:58:22.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:21.955+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2fd80a5d50 legacy=0x7f2fd8141a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:58:22.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:21.955+0000 7f2fd559a640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-08-19T12:58:22.018 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24859.log 2024-08-19T12:58:22.018 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25363.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.25363.log.gz 2024-08-19T12:58:22.018 INFO:teuthology.orchestra.run.smithi104.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.25850.log.gz 2024-08-19T12:58:22.018 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26069.log 2024-08-19T12:58:22.018 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24859.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.24859.log.gz 2024-08-19T12:58:22.018 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24815.log 2024-08-19T12:58:22.018 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26069.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.26069.log.gz 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24815.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.24815.log.gz 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26451.log 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24954.log 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26451.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.26451.log.gz 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.24517.log.gz 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25380.log 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24954.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.24954.log.gz 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24570.log 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25380.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.25380.log.gz 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21174.log 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24570.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.24570.log.gz 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24760.log 2024-08-19T12:58:22.019 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21174.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.21174.log.gz 2024-08-19T12:58:22.020 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21254.log 2024-08-19T12:58:22.020 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25779.log 2024-08-19T12:58:22.024 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35859.log 2024-08-19T12:58:22.024 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37223.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.37223.log.gz 2024-08-19T12:58:22.024 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27371.log 2024-08-19T12:58:22.024 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35859.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35859.log.gz 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31135.log 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27371.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.27371.log.gz 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24086.log 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31135.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31135.log.gz 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32782.log 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24086.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.24086.log.gz 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31856.log 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32782.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32782.log.gz 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28704.log 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31856.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31856.log.gz 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33891.log 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28704.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28704.log.gz 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27917.log 2024-08-19T12:58:22.025 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33891.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33891.log.gz 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34353.log 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27917.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27917.log.gz 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38009.log 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34353.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34353.log.gz 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30881.log 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.38009.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38009.log.gz 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30564.log 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30881.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30881.log.gz 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24516.log 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30564.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30564.log.gz 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24807.log 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24516.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.24516.log.gz 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24811.log 2024-08-19T12:58:22.026 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24807.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.24807.log.gz 2024-08-19T12:58:22.027 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-08-19T12:58:22.027 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24811.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.24811.log.gz 2024-08-19T12:58:22.027 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31772.log 2024-08-19T12:58:22.027 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36803.log 2024-08-19T12:58:22.027 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31772.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31772.log.gz 2024-08-19T12:58:22.027 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25766.log 2024-08-19T12:58:22.027 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36803.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36803.log.gz 2024-08-19T12:58:22.028 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21254.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.21254.log.gz 2024-08-19T12:58:22.028 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25238.log 2024-08-19T12:58:22.031 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24270.log 2024-08-19T12:58:22.042 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25766.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25766.log.gz 2024-08-19T12:58:22.047 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24727.log 2024-08-19T12:58:22.048 INFO:teuthology.orchestra.run.smithi026.stderr: 92.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-19T12:58:22.049 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28112.log 2024-08-19T12:58:22.049 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.24270.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.24270.log.gz 2024-08-19T12:58:22.050 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23721.log 2024-08-19T12:58:22.051 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28112.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28112.log.gz 2024-08-19T12:58:22.052 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31982.log 2024-08-19T12:58:22.052 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.23721.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.23721.log.gz 2024-08-19T12:58:22.053 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25238.log: 93.4% -- replaced with /var/log/ceph/ceph-client.admin.25238.log.gz 2024-08-19T12:58:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31304.log 2024-08-19T12:58:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31982.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31982.log.gz 2024-08-19T12:58:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36635.log 2024-08-19T12:58:22.055 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31304.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31304.log.gz 2024-08-19T12:58:22.056 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26068.log 2024-08-19T12:58:22.057 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35229.log 2024-08-19T12:58:22.058 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36635.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36635.log.gz 2024-08-19T12:58:22.058 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35817.log 2024-08-19T12:58:22.058 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35229.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35229.log.gz 2024-08-19T12:58:22.059 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29598.log 2024-08-19T12:58:22.060 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35817.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35817.log.gz 2024-08-19T12:58:22.061 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25006.log 2024-08-19T12:58:22.062 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.29598.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29598.log.gz 2024-08-19T12:58:22.063 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33513.log 2024-08-19T12:58:22.063 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.25006.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.25006.log.gz 2024-08-19T12:58:22.064 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35901.log 2024-08-19T12:58:22.065 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.33513.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33513.log.gz 2024-08-19T12:58:22.066 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32824.log 2024-08-19T12:58:22.067 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35901.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35901.log.gz 2024-08-19T12:58:22.067 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36215.log 2024-08-19T12:58:22.068 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32824.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32824.log.gz 2024-08-19T12:58:22.069 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28242.log 2024-08-19T12:58:22.070 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.36215.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36215.log.gz 2024-08-19T12:58:22.070 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31177.log 2024-08-19T12:58:22.071 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.28242.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28242.log.gz 2024-08-19T12:58:22.072 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35397.log 2024-08-19T12:58:22.072 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.31177.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.31177.log.gz 2024-08-19T12:58:22.073 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32740.log 2024-08-19T12:58:22.074 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35397.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35397.log.gz 2024-08-19T12:58:22.075 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27539.log 2024-08-19T12:58:22.076 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.32740.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32740.log.gz 2024-08-19T12:58:22.076 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23951.log 2024-08-19T12:58:22.077 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24727.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.25779.log.gz 2024-08-19T12:58:22.078 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27539.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27539.log.gz 2024-08-19T12:58:22.078 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.e.log 2024-08-19T12:58:22.080 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.23951.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.23951.log.gz 2024-08-19T12:58:22.080 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21228.log 2024-08-19T12:58:22.081 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25143.log 2024-08-19T12:58:22.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26068.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.26068.log.gz 2024-08-19T12:58:22.090 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mon.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27113.log 2024-08-19T12:58:22.091 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.21228.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.21228.log.gz 2024-08-19T12:58:22.094 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26450.log 2024-08-19T12:58:22.106 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27413.log 2024-08-19T12:58:22.106 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27113.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.27113.log.gz 2024-08-19T12:58:22.107 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34101.log 2024-08-19T12:58:22.108 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27413.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27413.log.gz 2024-08-19T12:58:22.113 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25143.log: 91.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26447.log 2024-08-19T12:58:22.113 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26450.log: -- replaced with /var/log/ceph/ceph-client.admin.25143.log.gz 2024-08-19T12:58:22.114 INFO:teuthology.orchestra.run.smithi104.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.26450.log.gz 2024-08-19T12:58:22.118 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35061.log 2024-08-19T12:58:22.119 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34101.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34101.log.gz 2024-08-19T12:58:22.119 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26120.log 2024-08-19T12:58:22.129 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26449.log 2024-08-19T12:58:22.135 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26120.log: 90.6% 90.6% -- replaced with /var/log/ceph/ceph-client.admin.26447.log.gz 2024-08-19T12:58:22.136 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34893.log 2024-08-19T12:58:22.136 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.35061.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35061.log.gz 2024-08-19T12:58:22.142 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30480.log 2024-08-19T12:58:22.142 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34893.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34893.log.gz 2024-08-19T12:58:22.145 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26120.log.gz 2024-08-19T12:58:22.147 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25434.log 2024-08-19T12:58:22.151 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27707.log 2024-08-19T12:58:22.151 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.30480.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30480.log.gz 2024-08-19T12:58:22.152 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26449.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.26449.log.gz 2024-08-19T12:58:22.168 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37811.log 2024-08-19T12:58:22.169 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.27707.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27707.log.gz 2024-08-19T12:58:22.169 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21338.log 2024-08-19T12:58:22.179 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.37811.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37811.log.gz 2024-08-19T12:58:22.182 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25434.log: 91.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21298.log 2024-08-19T12:58:22.182 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21338.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.21338.log.gz 2024-08-19T12:58:22.186 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25434.log.gz 2024-08-19T12:58:22.187 INFO:teuthology.orchestra.run.smithi104.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.admin.24727.log.gz 2024-08-19T12:58:22.202 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25015.log 2024-08-19T12:58:22.202 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21298.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.21298.log.gz 2024-08-19T12:58:22.209 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25212.log 2024-08-19T12:58:22.219 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25015.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-08-19T12:58:22.227 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25212.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.25015.log.gz 2024-08-19T12:58:22.233 INFO:teuthology.orchestra.run.smithi104.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.25212.log.gz 2024-08-19T12:58:22.239 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24597.log 2024-08-19T12:58:22.258 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25859.log 2024-08-19T12:58:22.267 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26398.log 2024-08-19T12:58:22.275 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25349.log 2024-08-19T12:58:22.284 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26398.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-19T12:58:22.290 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25349.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.26398.log.gz 2024-08-19T12:58:22.295 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25353.log 2024-08-19T12:58:22.297 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.25349.log.gz 2024-08-19T12:58:22.306 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24757.log 2024-08-19T12:58:22.310 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25353.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.25353.log.gz 2024-08-19T12:58:22.320 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25216.log 2024-08-19T12:58:22.333 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24757.log: 90.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21134.log 2024-08-19T12:58:22.337 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25216.log: -- replaced with /var/log/ceph/ceph-client.admin.24757.log.gz 2024-08-19T12:58:22.349 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25753.log 2024-08-19T12:58:22.350 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.21134.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.21134.log.gz 2024-08-19T12:58:22.356 INFO:teuthology.orchestra.run.smithi104.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.25859.log.gz 2024-08-19T12:58:22.363 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25377.log 2024-08-19T12:58:22.367 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25753.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.24597.log.gz 2024-08-19T12:58:22.380 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25830.log 2024-08-19T12:58:22.383 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25377.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.25377.log.gz 2024-08-19T12:58:22.393 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24837.log 2024-08-19T12:58:22.409 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25830.log: 92.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26549.log 2024-08-19T12:58:22.414 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.24837.log: -- replaced with /var/log/ceph/ceph-client.admin.25830.log.gz 2024-08-19T12:58:22.418 INFO:teuthology.orchestra.run.smithi104.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.25753.log.gz 2024-08-19T12:58:22.418 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25862.log 2024-08-19T12:58:22.431 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.26549.log: 92.3% 91.2% -- replaced with /var/log/ceph/ceph-client.admin.24837.log.gz -- replaced with /var/log/ceph/ceph-client.admin.25216.log.gz 2024-08-19T12:58:22.431 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:58:22.433 INFO:teuthology.orchestra.run.smithi104.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.26549.log.gz 2024-08-19T12:58:22.437 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.f.log 2024-08-19T12:58:22.448 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25862.log: 92.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25549.log 2024-08-19T12:58:22.448 INFO:teuthology.orchestra.run.smithi104.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.25862.log.gz 2024-08-19T12:58:22.453 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mon.f.log: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.24748.log.gz 2024-08-19T12:58:22.458 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-08-19T12:58:22.467 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.25549.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.25549.log.gz 2024-08-19T12:58:22.505 INFO:teuthology.orchestra.run.smithi104.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-08-19T12:58:22.928 INFO:teuthology.orchestra.run.smithi104.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.i.log.gz 2024-08-19T12:58:23.151 INFO:teuthology.orchestra.run.smithi026.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.d.log.gz 2024-08-19T12:58:23.172 INFO:teuthology.orchestra.run.smithi104.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.g.log.gz 2024-08-19T12:58:23.310 INFO:teuthology.orchestra.run.smithi104.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.24760.log.gz 2024-08-19T12:58:23.351 INFO:teuthology.orchestra.run.smithi104.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.h.log.gz 2024-08-19T12:58:23.374 INFO:teuthology.orchestra.run.smithi026.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-08-19T12:58:23.418 INFO:teuthology.orchestra.run.smithi026.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-08-19T12:58:23.499 INFO:teuthology.orchestra.run.smithi026.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.e.log.gz 2024-08-19T12:58:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 10 monclient(hunting): tick 2024-08-19T12:58:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:58:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 1 monclient(hunting): continuing hunt 2024-08-19T12:58:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:58:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2fd806e7b0 legacy=0x7f2fd8136080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2fd80a5d50 legacy=0x7f2fd8141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fd80a5920 legacy=0x7f2fd813e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:23.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 10 monclient: _add_conns ranks=[6,8,4,3,7,5] 2024-08-19T12:58:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-d con 0x7f2fb8003780 addr v1:172.21.15.26:6792/0 2024-08-19T12:58:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-e con 0x7f2fb8007410 addr v1:172.21.15.26:6793/0 2024-08-19T12:58:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2fb800b1a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:58:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f2fb8003780 legacy=0x7f2fb8005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:58:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb800b1a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:58:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd809e3b0 con 0x7f2fb800b1a0 2024-08-19T12:58:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb8007410 legacy=0x7f2fb80098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:58:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd8136790 con 0x7f2fb8003780 2024-08-19T12:58:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd809e0d0 con 0x7f2fb8007410 2024-08-19T12:58:23.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.552+0000 7f2fcffff640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:23.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.753+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2fb800b1a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:58:23.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.753+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb8007410 legacy=0x7f2fb80098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:58:23.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:23.753+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f2fb8003780 legacy=0x7f2fb8005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:58:24.075 INFO:teuthology.orchestra.run.smithi104.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.f.log.gz 2024-08-19T12:58:24.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:24.154+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f2fb8003780 legacy=0x7f2fb8005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:58:24.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:24.154+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2fb800b1a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:58:24.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:24.155+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb8007410 legacy=0x7f2fb80098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:58:24.725 INFO:teuthology.orchestra.run.smithi026.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-08-19T12:58:24.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:24.955+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f2fb8003780 legacy=0x7f2fb8005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:58:24.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:24.955+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2fb800b1a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:58:24.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:24.956+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb8007410 legacy=0x7f2fb80098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:58:26.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 10 monclient(hunting): tick 2024-08-19T12:58:26.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:58:26.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 1 monclient(hunting): continuing hunt 2024-08-19T12:58:26.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:58:26.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb8007410 legacy=0x7f2fb80098d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:26.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f2fb8003780 legacy=0x7f2fb8005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:26.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fd6d9d640 1 -- reap_dead start 2024-08-19T12:58:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2fb800b1a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 10 monclient: _add_conns ranks=[3,7,5] 2024-08-19T12:58:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-g con 0x7f2fb8011030 addr v1:172.21.15.104:6790/0 2024-08-19T12:58:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-i con 0x7f2fb800a0a0 addr v1:172.21.15.104:6792/0 2024-08-19T12:58:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-h con 0x7f2fb800a4e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:58:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb8011fa0 con 0x7f2fb8011030 2024-08-19T12:58:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd809e3b0 con 0x7f2fb800a4e0 2024-08-19T12:58:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd8136790 con 0x7f2fb800a0a0 2024-08-19T12:58:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fcffff640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fd659c640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb8011030 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:58:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:58:26.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.552+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb8005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:58:26.753 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.753+0000 7f2fd659c640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb8011030 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:58:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.753+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:58:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:26.753+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb8005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:58:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:27.154+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:58:27.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:27.154+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb8005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:58:27.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:27.154+0000 7f2fd659c640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb8011030 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:58:27.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:27.955+0000 7f2fd659c640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb8011030 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:58:27.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:27.955+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb8005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:58:27.956 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:27.955+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:58:29.353 INFO:teuthology.orchestra.run.smithi104.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-08-19T12:58:29.489 INFO:teuthology.orchestra.run.smithi104.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-08-19T12:58:29.537 INFO:teuthology.orchestra.run.smithi026.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-08-19T12:58:29.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.552+0000 7f2fcffff640 10 monclient(hunting): tick 2024-08-19T12:58:29.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.552+0000 7f2fcffff640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:58:29.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.552+0000 7f2fcffff640 1 monclient(hunting): continuing hunt 2024-08-19T12:58:29.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.552+0000 7f2fcffff640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:58:29.554 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.552+0000 7f2fcffff640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb8005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.552+0000 7f2fcffff640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.553+0000 7f2fcffff640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb8011030 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.553+0000 7f2fcffff640 10 monclient: _add_conns ranks=[0,4,5,2,6,8,7,3,1] 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.553+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb80099b0 addr v1:172.21.15.26:6789/0 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.553+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb800b1a0 addr v1:172.21.15.26:6791/0 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.553+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-h con 0x7f2fb800a4e0 addr v1:172.21.15.104:6791/0 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.553+0000 7f2fcffff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd809e0d0 con 0x7f2fb80099b0 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.553+0000 7f2fcffff640 1 -- --> v1:172.21.15.26:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb8011fa0 con 0x7f2fb800b1a0 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.553+0000 7f2fcffff640 1 -- --> v1:172.21.15.104:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd809e3b0 con 0x7f2fb800a4e0 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.553+0000 7f2fcffff640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.553+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fb80099b0 legacy=0x7f2fb80162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.553+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2fb800b1a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:58:29.555 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.553+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:58:29.735 INFO:teuthology.orchestra.run.smithi104.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-08-19T12:58:29.737 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-08-19T12:58:29.737 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m7.898s 2024-08-19T12:58:29.737 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m29.501s 2024-08-19T12:58:29.737 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m0.970s 2024-08-19T12:58:29.746 INFO:teuthology.orchestra.run.smithi026.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-08-19T12:58:29.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.754+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2fb800b1a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:58:29.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.754+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fb80099b0 legacy=0x7f2fb80162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:58:29.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:29.754+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:58:30.155 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:30.155+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:58:30.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:30.156+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fb80099b0 legacy=0x7f2fb80162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:58:30.156 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:30.156+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2fb800b1a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:58:30.228 INFO:teuthology.orchestra.run.smithi026.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-08-19T12:58:30.230 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-08-19T12:58:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:real 0m8.416s 2024-08-19T12:58:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:user 0m31.633s 2024-08-19T12:58:30.230 INFO:teuthology.orchestra.run.smithi026.stderr:sys 0m1.130s 2024-08-19T12:58:30.230 INFO:tasks.ceph:Archiving logs... 2024-08-19T12:58:30.231 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/remote/smithi026/log 2024-08-19T12:58:30.232 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-19T12:58:30.974 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:30.956+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800a990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6791/0 2024-08-19T12:58:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:30.958+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2fb800b1a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6791/0 2024-08-19T12:58:30.977 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:30.958+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fb80099b0 legacy=0x7f2fb80162d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:58:32.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 10 monclient(hunting): tick 2024-08-19T12:58:32.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:58:32.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 1 monclient(hunting): continuing hunt 2024-08-19T12:58:32.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:58:32.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 1 -- >> v1:172.21.15.104:6791/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:32.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 1 -- >> v1:172.21.15.26:6791/0 conn(0x7f2fb800b1a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:32.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fb80099b0 legacy=0x7f2fb80162d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:32.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fd6d9d640 1 -- reap_dead start 2024-08-19T12:58:32.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 10 monclient: _add_conns ranks=[3,8,1,7,6,2] 2024-08-19T12:58:32.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-g con 0x7f2fb80099b0 addr v1:172.21.15.104:6790/0 2024-08-19T12:58:32.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-e con 0x7f2fb800a0a0 addr v1:172.21.15.26:6793/0 2024-08-19T12:58:32.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-f con 0x7f2fb800a4e0 addr v1:172.21.15.104:6789/0 2024-08-19T12:58:32.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd8136790 con 0x7f2fb800a4e0 2024-08-19T12:58:32.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd809e0d0 con 0x7f2fb80099b0 2024-08-19T12:58:32.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb8011fa0 con 0x7f2fb800a0a0 2024-08-19T12:58:32.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fcffff640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:32.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:58:32.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:58:32.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.552+0000 7f2fd659c640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb80099b0 legacy=0x7f2fb800a970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:58:32.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.753+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:58:32.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.754+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:58:32.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:32.754+0000 7f2fd659c640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb80099b0 legacy=0x7f2fb800a970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:58:33.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:33.155+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:58:33.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:33.155+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:58:33.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:33.155+0000 7f2fd659c640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb80099b0 legacy=0x7f2fb800a970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:58:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:33.956+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:58:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:33.956+0000 7f2fd659c640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb80099b0 legacy=0x7f2fb800a970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:58:33.985 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:33.956+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6789/0 2024-08-19T12:58:34.943 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/remote/smithi104/log 2024-08-19T12:58:34.944 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-19T12:58:35.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 10 monclient(hunting): tick 2024-08-19T12:58:35.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:58:35.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 1 monclient(hunting): continuing hunt 2024-08-19T12:58:35.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:58:35.562 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb800d660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb80099b0 legacy=0x7f2fb800a970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 1 -- >> v1:172.21.15.104:6789/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 10 monclient: _add_conns ranks=[6,2,7] 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-d con 0x7f2fb8004b70 addr v1:172.21.15.26:6792/0 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb8005660 addr v1:172.21.15.26:6790/0 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-i con 0x7f2fb80182f0 addr v1:172.21.15.104:6792/0 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 1 -- --> v1:172.21.15.26:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd809e3b0 con 0x7f2fb8005660 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 1 -- --> v1:172.21.15.26:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd8136790 con 0x7f2fb8004b70 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 1 -- --> v1:172.21.15.104:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd809e0d0 con 0x7f2fb80182f0 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fcffff640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2fb8005660 legacy=0x7f2fb8005a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f2fb8004b70 legacy=0x7f2fb8004f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:58:35.563 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.553+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f2fb80182f0 legacy=0x7f2fb801a6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:58:35.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.754+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f2fb8004b70 legacy=0x7f2fb8004f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:58:35.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.754+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2fb8005660 legacy=0x7f2fb8005a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-08-19T12:58:35.756 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:35.754+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f2fb80182f0 legacy=0x7f2fb801a6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:58:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:36.155+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2fb8005660 legacy=0x7f2fb8005a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-08-19T12:58:36.166 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:36.155+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f2fb8004b70 legacy=0x7f2fb8004f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:58:36.167 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:36.155+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f2fb80182f0 legacy=0x7f2fb801a6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:58:36.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:36.956+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2fb8005660 legacy=0x7f2fb8005a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6790/0 2024-08-19T12:58:36.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:36.956+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f2fb8004b70 legacy=0x7f2fb8004f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6792/0 2024-08-19T12:58:36.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:36.956+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f2fb80182f0 legacy=0x7f2fb801a6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6792/0 2024-08-19T12:58:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 10 monclient(hunting): tick 2024-08-19T12:58:38.559 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 10 monclient(hunting): _check_auth_tickets 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 1 monclient(hunting): continuing hunt 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 10 monclient(hunting): _reopen_session rank -1 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 1 -- >> v1:172.21.15.104:6792/0 conn(0x7f2fb80182f0 legacy=0x7f2fb801a6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 1 -- >> v1:172.21.15.26:6792/0 conn(0x7f2fb8004b70 legacy=0x7f2fb8004f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 1 -- >> v1:172.21.15.26:6790/0 conn(0x7f2fb8005660 legacy=0x7f2fb8005a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fd6d9d640 1 -- reap_dead start 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 10 monclient: _add_conns ranks=[0,3,8,5,2,1,7,6,4] 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb800a0a0 addr v1:172.21.15.26:6789/0 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-g con 0x7f2fb800a4e0 addr v1:172.21.15.104:6790/0 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 10 monclient(hunting): picked mon.noname-e con 0x7f2fb800a970 addr v1:172.21.15.26:6793/0 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb8011fa0 con 0x7f2fb800a0a0 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd809e3b0 con 0x7f2fb800a4e0 2024-08-19T12:58:38.560 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 1 -- --> v1:172.21.15.26:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd8136790 con 0x7f2fb800a970 2024-08-19T12:58:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fcffff640 10 monclient(hunting): _renew_subs 2024-08-19T12:58:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb8004f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:58:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb800a970 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:58:38.561 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.553+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800d640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:58:38.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.754+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb8004f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:58:38.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.754+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb800a970 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:58:38.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:38.754+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800d640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:58:39.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:39.155+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb800a970 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:58:39.164 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:39.155+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb8004f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:58:39.165 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:39.155+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800d640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:58:39.465 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-19T12:58:39.473 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 466, in task thrash_proc.stop_and_join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 178, in stop_and_join return self.join() File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/mon_thrash.py", line 170, in join self.stopping.set() AttributeError: 'bool' object has no attribute 'set' 2024-08-19T12:58:39.474 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-19T12:58:39.475 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-19T12:58:39.477 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-19T12:58:39.580 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-19T12:58:39.589 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-19T12:58:39.589 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T12:58:39.592 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-19T12:58:39.607 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-08-19T12:58:39.608 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-08-19T12:58:39.611 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T12:58:39.611 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-08-19T12:58:39.612 INFO:teuthology.orchestra.run.smithi026.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 +274us[ +280us] +/- 60ms 2024-08-19T12:58:39.612 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 -184us[ -178us] +/- 53ms 2024-08-19T12:58:39.612 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 -58us[ -53us] +/- 68ms 2024-08-19T12:58:39.612 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T12:58:39.613 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-19T12:58:39.613 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-08-19T12:58:39.613 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv01.front.sepia.ceph.com 3 6 377 30 +262us[ +256us] +/- 60ms 2024-08-19T12:58:39.614 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 26 -148us[ -148us] +/- 53ms 2024-08-19T12:58:39.614 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -37us[ -37us] +/- 68ms 2024-08-19T12:58:39.614 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-08-19T12:58:39.614 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-19T12:58:39.623 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-19T12:58:39.623 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-19T12:58:39.640 DEBUG:teuthology.orchestra.run.smithi026:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-19T12:58:39.715 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-19T12:58:39.768 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-19T12:58:39.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T12:58:39.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-19T12:58:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T12:58:39.827 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-19T12:58:39.836 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-19T12:58:39.858 INFO:teuthology.task.internal:Duration was 1498.358171 seconds 2024-08-19T12:58:39.858 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-19T12:58:39.877 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-19T12:58:39.877 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-19T12:58:39.880 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-19T12:58:39.920 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T12:58:39.921 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-19T12:58:39.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:39.956+0000 7f2fd6d9d640 1 -- >> v1:172.21.15.26:6793/0 conn(0x7f2fb800a970 legacy=0x7f2fb8011410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6793/0 2024-08-19T12:58:39.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:39.956+0000 7f2fd659c640 1 -- >> v1:172.21.15.26:6789/0 conn(0x7f2fb800a0a0 legacy=0x7f2fb8004f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.26:6789/0 2024-08-19T12:58:39.958 INFO:teuthology.orchestra.run.smithi026.stderr:2024-08-19T12:58:39.956+0000 7f2fd5d9b640 1 -- >> v1:172.21.15.104:6790/0 conn(0x7f2fb800a4e0 legacy=0x7f2fb800d640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.104:6790/0 2024-08-19T12:58:40.251 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-19T12:58:40.251 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2024-08-19T12:58:40.251 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-19T12:58:40.274 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-08-19T12:58:40.275 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-19T12:58:40.314 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-19T12:58:40.314 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-19T12:58:40.317 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-19T12:58:40.383 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-19T12:58:40.383 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-19T12:58:40.386 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-19T12:58:40.487 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-19T12:58:40.497 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-19T12:58:40.497 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-19T12:58:40.523 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-19T12:58:40.548 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-19T12:58:40.560 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-19T12:58:40.566 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-19T12:58:40.587 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2024-08-19T12:58:40.611 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-08-19T12:58:40.624 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-19T12:58:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T12:58:40.659 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-19T12:58:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-19T12:58:40.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-19T12:58:40.690 INFO:teuthology.task.internal:Transferring archived files... 2024-08-19T12:58:40.691 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/remote/smithi026 2024-08-19T12:58:40.691 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-19T12:58:40.733 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859520/remote/smithi104 2024-08-19T12:58:40.734 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-19T12:58:40.884 INFO:teuthology.task.internal:Removing archive directory... 2024-08-19T12:58:40.885 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-19T12:58:40.888 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-19T12:58:40.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-19T12:58:40.963 INFO:teuthology.task.internal:Not uploading archives. 2024-08-19T12:58:40.963 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-19T12:58:41.032 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-19T12:58:41.032 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-19T12:58:41.035 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-19T12:58:41.048 INFO:teuthology.orchestra.run.smithi026.stdout: 262218 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 19 12:58 /home/ubuntu/cephtest 2024-08-19T12:58:41.050 INFO:teuthology.orchestra.run.smithi104.stdout: 262218 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 19 12:58 /home/ubuntu/cephtest 2024-08-19T12:58:41.051 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-19T12:58:41.063 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-19T12:58:41.082 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: AttributeError: 'bool' object has no attribute 'set' 2024-08-19T12:58:41.085 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_api_tests} duration: 1498.3581712245941 failure_reason: '''bool'' object has no attribute ''set''' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-08-19T12:58:41.085 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-19T12:58:41.181 INFO:teuthology.run:FAIL